Chris Pollett > Students >
Ivan
    ( Print View)

    [Bio]

    [Blog]

    [C297 Proposal]

    [Deliverable 1]

    [Deliverable 2]

    [Deliverable 3]

    [Deliverable 4]

    [CS297 Report (PDF)]

    [Paper 1: A-Frame Virtual Reality (PDF)]

    [Paper 2: ChatGPT - Fundamentals and Applications (PDF)]

    [Paper 3: ChatGPT Consistency (PDF)]

    [Paper 4: VR Voice Interaction (PDF)]

    [C298 Proposal]

    [CS298 - Deliverable 1]

    [CS298 - Deliverable 2]

    [CS298 - Deliverable 3]

    [CS298 - Deliverable 4]

    [CS298 Report (PDF)]

    [CS298 Presentation (PDF)]

    [CS298 - Code (ZIP)]

CS297 Proposal

Integrating ChatGPT with A-Frame for User-Driven 3D Modeling

Ivan Hernandez (ivan.a.hernandez@sjsu.edu)

Advisor: Dr. Chris Pollett

Description:

The objective of this research is to develop a system that integrates ChatGPT into an A-Frame-based virtual reality (VR) environment, thus enabling users to create, modify, and interact with 3D models based on user-provided natural language descriptions. Currently, VR and modeling applications often rely on traditional input devices and graphical user interfaces, which can be complex and unintuitive for users. By leveraging ChatGPT's natural language understanding capabilities, this system can offer a more engaging and immersive experience for users to guide and modify the 3D modeling process in real-time.

Schedule:

Week 1: Aug 23 - Aug 29Search for research papers and finish the project proposal
Week 2: Aug 30 - Sep 5Get more familiar with A-Frame. Read Paper [1]
Week 3: Sep 6 - Sep 12Start Deliverable 1(A-Frame environment) and summarize Paper [1]
Week 4: Sep 13 - Sep 19Complete Deliverable 1
Week 5: Sep 20 - Sep 26Start Deliverable 2(ChatGPT Integration). Read Paper [2]
Week 6: Sep 27 - Oct 3Continue working on Deliverable 2 and summarize Paper [2]
Week 7: Oct 4 - Oct 10Complete Deliverable 2
Week 8: Oct 11 - Oct 17Start Deliverable 3(ChatGPT to A-Frame modeling commands). Read Paper [3]
Week 9: Oct 18 - Oct 24Continue working on Deliverable 3 and summarize Paper [3]
Week 10: Oct 25 - Oct 31Continue working on Deliverable 3
Week 11: Nov 1 - Nov 7Continue working on Deliverable 3
Week 12: Nov 8 - Nov 14Complete Deliverable 3. Read Paper [4]
Week 13: Nov 15 - Nov 21Start Deliverable 4(Voice command integration) and summarize paper [4]
Week 14: Nov 22 - Nov 28Complete Deliverable 4
Week 15: Nov 29 - Dec 5Complete CS 297 report

Deliverables:

The full project will be done when CS298 is completed. The following will be done by the end of CS297:

1. Set up an A-Frame VR environment and create simple 3D primitives in the scene

2. Integrate ChatGPT into the A-Frame environment

3. Develop an interpreter layer that translates ChatGPT's responses into A-Frame modeling commands

4. Implement voice interactions for easier communication with the system

5. CS297 Report.

References:

[1] Web-based Virtual Reality with A-Frame. S. G. Santos and J. C. S. Cardoso. 2019 14th Iberian Conference on Information Systems and Technologies (CISTI). 2019.

[2] ChatGPT: Fundamentals, Applications and Social Impacts. M. Abdullah, A. Madain and Y. Jararweh. 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS). 2022.

[3] Consistency Analysis of ChatGPT. Jang, Myeongjun and Thomas Lukasiewicz. ArXiv. 2023.

[4] Research on Voice Interaction Technology in VR Environment. C. Li and B. Tang. 2019 International Conference on Electronic Engineering and Informatics (EEI). 2019.