CS298 Proposal
Shoe AR Reconstruction
Sneh Kothari (sneharvind.kothari@sjsu.edu)
Advisor: Dr. Chris Pollett
Committee Members: Dr. William Andreopoulos, Dr. Robert Chun
Abstract:
People generally want opinions from others when buying shoes offline.
Clicking and sending images of the shoe is not an ideal solution as it does not give the complete feel of the shoe.
This problem can be solved using Shoe AR Reconstruction an app intended to make shoe shopping easier.
The user clicks a photo/video of a shoe. This image data is converted to form a mesh that can be shared.
On receiving a model the user can open it in the app and interact with it to get a better look and feel of the shoe.
This can be particularly useful in something like Amazon where sellers can click images to build a mesh and customers can render this mesh for a better buying experience.
The solution to this is building an Augmented Reality app that build 3D mesh of shoes and allows users to make an informed decision about the product they are buying.
CS297 Results
- Built a demo Augmented Reality App using frameworks provided by Apple. Compared different ways to build an AR app.
- Explored different ways to build a 3D mesh from 2D image data. Concluded that the mesh building will happen on server side.
- Implemented deep linking to make sharing easier and pass data along with the link
- Built an app to view and interact with shoes in Augmented Reality. Tried different versions and decided on one.
Proposed Schedule
Week 1:
Jan 31 - Feb 7 | Organizational meeting with advisor |
Week 2:
Feb 8 - Feb 14 | Read blogs and research papers and come up with a plan for the model building |
Week 3:
Feb 15 - Feb 21 | Implement 3D model building on server |
Week 4:
Feb 22 - Feb 28 | Implement 3D model building on server |
Week 5:
Mar 1 - Mar 7 | Implement 3D model building on server |
Week 6:
Mar 8 - Mar 14 | Implement 3D model building on server |
Week 7:
Mar 15 - Mar 21 | Build app to capture images for building 3D model |
Week 8:
Mar 22 - Mar 28 | Build app to display 3D model |
Week 9:
Mar 29 - Apr 4 | Build app to display 3D model |
Week 10:
Apr 5 - Apr 11 | Integrate server image with app and connect using deep links |
Week 11:
Apr 12 - Apr 18 | Integrate server image with app and connect using deep links |
Week 12:
Apr 19 - Apr 25 | Start CS 298 Report and Presentation |
Week 13:
Apr 26 - May 2 | Work on CS 298 Report and Presentation |
Week 14:
May 3 - May 9 | Finish CS 298 Report and Presentation |
Key Deliverables:
- Software
- Code which runs on server to build a 3D mesh from 2D image data
- Build an app that allows users to capture images that will be used for building the 3D model
- Implement a viewing experience for the shoe model in Augmented Reality
- Report
- CS 298 Report
- CS 298 Presentation
Innovations and Challenges
- Figuring out how to build a 3D mesh from 2D data
- Building an app that lets users interact with 3D objects and have a good experience
References:
[2020] ARKit and ARCore in serve to augmented reality. Z. Oufqir, A. El Abderrahmani and K. Satori. International Conference on Intelligent Systems and Computer Vision (ISCV). 2020.
[2017] MagicToon: A 2D-to-3D creative cartoon modeling system with mobile AR. L. Feng, X. Yang and S. Xiao. IEEE Virtual Reality (VR). 2017.
[2018] Integrating YOLO Object Detection with Augmented Reality for iOS Apps. S. Mahurkar. IEEE Annual Ubiquitous Computing, Electronics and Mobile Communication Conference. 2018.
[2021] ARCritique: Supporting Remote Design Critique of Physical Artifacts through Collaborative Augmented Reality. Y. Li, D. Hicks, W. S. Lages, S. Won Lee, A. Sharma and D. A. Bowman. 2021 IEEE Conference on Virtual Reality and 3D User Interfaces
|