Abstract:

People generally want opinions from others when buying shoes offline. Clicking and sending images of the shoe is not an ideal solution as it does not give the complete feel of the shoe. This problem can be solved using Shoe AR Reconstruction an app intended to make shoe shopping easier. The user clicks a photo/video of a shoe. This image data is converted to form a mesh that can be shared. On receiving a model the user can open it in the app and interact with it to get a better look and feel of the shoe. This can be particularly useful in something like Amazon where sellers can click images to build a mesh and customers can render this mesh for a better buying experience. The solution to this is building an Augmented Reality app that build 3D mesh of shoes and allows users to make an informed decision about the product they are buying.

CS297 Results

• Built a demo Augmented Reality App using frameworks provided by Apple. Compared different ways to build an AR app.
• Explored different ways to build a 3D mesh from 2D image data. Concluded that the mesh building will happen on server side.
• Implemented deep linking to make sharing easier and pass data along with the link
• Built an app to view and interact with shoes in Augmented Reality. Tried different versions and decided on one.

Proposed Schedule

Key Deliverables:

• Software
  • Code which runs on server to build a 3D mesh from 2D image data
  • Build an app that allows users to capture images that will be used for building the 3D model
  • Implement a viewing experience for the shoe model in Augmented Reality
• Report
  • CS 298 Report
  • CS 298 Presentation

Innovations and Challenges

• Figuring out how to build a 3D mesh from 2D data
• Building an app that lets users interact with 3D objects and have a good experience

References:

[2020] ARKit and ARCore in serve to augmented reality. Z. Oufqir, A. El Abderrahmani and K. Satori. International Conference on Intelligent Systems and Computer Vision (ISCV). 2020.

[2017] MagicToon: A 2D-to-3D creative cartoon modeling system with mobile AR. L. Feng, X. Yang and S. Xiao. IEEE Virtual Reality (VR). 2017.

[2018] Integrating YOLO Object Detection with Augmented Reality for iOS Apps. S. Mahurkar. IEEE Annual Ubiquitous Computing, Electronics and Mobile Communication Conference. 2018.

[2021] ARCritique: Supporting Remote Design Critique of Physical Artifacts through Collaborative Augmented Reality. Y. Li, D. Hicks, W. S. Lages, S. Won Lee, A. Sharma and D. A. Bowman. 2021 IEEE Conference on Virtual Reality and 3D User Interfaces