CS297 Proposal

Description:

People generally want opinions from others when buying shoes offline. Clicking and sending images of the shoe is not an ideal solution as it does not give the complete feel of the shoe. This problem can be solved using Shoe AR Reconstruction an app intended to make shoe shopping easier. The user clicks a photo/video of a shoe. This image data is converted to form a mesh that can be shared. On receiving a model the user can open it in the app and interact with it to get a better look and feel of the shoe. This can be particularly useful in something like Amazon where sellers can click images to build a mesh and customers can render this mesh for a better buying experience.

Schedule:

Deliverables:

The full project will be done when CS298 is completed. The following will be done by the end of CS297:

1. Implement a basic AR app for displaying teapots in different settings and allowing resizing interaction.
Also saving the rooms with the updated pot sizes.

2. Build an AR app implementing object capture to capture 2D image data for 3D modeling.
The object capture will be tested for increasingly complex objects starting from a tennis ball. Work on making two different types of meshes.

3. Try out different methods of sharing/compression for AR models.
Examine mesh sizes and explore sharing options. These can be peer-to-peer(iMessage) or server-to-device(S3 buckets).
Implement deep link for the shared file.

4. Render a 3D model on the user's app and try multiple versions of interaction with the object.

• Fix the model in space and the user moves the phone to examine the model
• The phone does not move and the user can spin the model and interact with gestures

5. Finish report.

References:

[2020] ARKit and ARCore in serve to augmented reality. Z. Oufqir, A. El Abderrahmani and K. Satori. International Conference on Intelligent Systems and Computer Vision (ISCV). 2020.

[2017] MagicToon: A 2D-to-3D creative cartoon modeling system with mobile AR. L. Feng, X. Yang and S. Xiao. IEEE Virtual Reality (VR). 2017.

[2018] Integrating YOLO Object Detection with Augmented Reality for iOS Apps. S. Mahurkar. IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference. 2018.

[2021] ARCritique: Supporting Remote Design Critique of Physical Artifacts through Collaborative Augmented Reality. Y. Li, D. Hicks, W. S. Lages, S. Won Lee, A. Sharma and D. A. Bowman. 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 2021