Chris Pollett > Students >
Ivan
    ( Print View)

    [Bio]

    [Blog]

    [C297 Proposal]

    [Deliverable 1]

    [Deliverable 2]

    [Deliverable 3]

    [Deliverable 4]

    [CS297 Report (PDF)]

    [Paper 1: A-Frame Virtual Reality (PDF)]

    [Paper 2: ChatGPT - Fundamentals and Applications (PDF)]

    [Paper 3: ChatGPT Consistency (PDF)]

    [Paper 4: VR Voice Interaction (PDF)]

    [C298 Proposal]

    [CS298 - Deliverable 1]

    [CS298 - Deliverable 2]

    [CS298 - Deliverable 3]

    [CS298 - Deliverable 4]

CS298 Proposal

Integrating ChatGPT with A-Frame for User-Driven 3D Modeling

Ivan Hernandez (ivan.a.hernandez@sjsu.edu)

Advisor: Dr. Chris Pollett

Committee Members: Dr. Genya Ishigaki, Prof. Kevin Smith

Abstract:

The objective of this project is to develop a system that integrates ChatGPT into an A-Frame-based virtual reality (VR) environment, thus enabling users to create, modify, and interact with 3D models based on user-provided natural language descriptions. Currently, VR and modeling applications often rely on traditional input devices and graphical user interfaces, which can be complex and unintuitive for users. By leveraging ChatGPT's natural language understanding capabilities, this system can offer a more engaging and immersive experience for users to guide and modify the 3D modeling process in real-time. Furthermore, we will extend this system’s capabilities by implementing scene serialization to save and load VR scenes, custom A-Frame component generation using JavaScript, and more advanced 3D modeling techniques (bevel, booleans, etc.) by directly accessing the underlying Three.js libraries.

CS297 Results

  • Created a demo VR environment with A-Frame and directly compared its workflow with that of Unity3D.
  • Successfully integrated ChatGPT with an A-Frame VR environment for real-time 3D modeling.
  • Developed an interpreter layer to translate ChatGPT's responses into A-Frame modeling commands.
  • Implemented voice interactions for easier communication with the system.

Proposed Schedule

Week 1: Jan 30 - Feb 6 Finish CS 298 proposal
Week 2: Feb 6 - Feb 13 Research serialization strategies and begin implementing A-Frame scene serialization
Week 3: Feb 13 - Feb 20 Finish serialization functionality and test different scenarios
Week 4-6: Feb 20 - Mar 12 Implement custom A-Frame component generation with Javascript
Week 7-10: Mar 12 - Apr 9 Implement Three.js integration for more 3D modeling features
Week 11-12: Apr 9 - Apr 23 Conduct comprehensive testing to evaluate the system's effectiveness.
Week 13: Apr 23 - Apr 30 Work on CS 298 Report and begin preparation for Master’s Defense
Week 14: Apr 30 - May 7 Continue working on CS298 Report and Presentation
Week 15: May 7 - May 14 Finish CS298 Report and Presentation

Key Deliverables:

  • Software
    • Implement an A-Frame serialization module for saving and loading A-Frame VR scenes.
    • Design a system that creates A-Frame Javascript components with customizable functionality which can be attached to A-Frame objects. Then, implement a parser that ensures that the component is functional and compatible within the VR environment.
    • Extend the project’s capabilities by implementing modeling features, such as bevel, extrude, and booleans using the underlying Three.js library that is not natively available in A-Frame.
    • Implement a structured test series covering simple to complex modeling scenarios to evaluate the effectiveness and usability of the system.
  • Report
    • CS 298 Report
    • CS 298 Presentation

Innovations and Challenges

  • One of the challenges for this project will be to effectively design a system that can generate A-Frame components that provide accurate behavior and functionality for objects in the scene. The generated Javascript components must use working code that adheres to the newest version of A-Frame standards and correctly interprets the users intentions.
  • There is little to no research that explores the combination of NLP with VR for 3D modeling. Therefore, as this project requires the integration of various technologies, including VR, NLP, and 3D Graphics, it will be challenging to develop a system that merges these fields effectively while ensuring a seamless and immersive 3D modeling experience.
  • Integrating the Three.js libraries into the A-Frame project will bring about more 3D modeling capabilities that not only need to be easy to use but also intuitive for the user to incorporate into their modeling process. Therefore, as more complexity is added to the system, it will make it more challenging to maintain a consistent level of accessibility and user-friendly interactions while modeling with voice commands.
  • Conducting comprehensive testing for this project will also prove to be challenging as it needs to cover a diverse case of modeling scenarios, including simple to complex cases as well as extreme and invalid test cases. A clear and balanced testing methodology needs to be established so that we can properly evaluate the quantitative performance metrics as well as the qualitative user interactions.

References:

[1] Web-based Virtual Reality with A-Frame. S. G. Santos and J. C. S. Cardoso. 2019 14th Iberian Conference on Information Systems and Technologies (CISTI). 2019.

[2] ChatGPT: Fundamentals, Applications and Social Impacts. M. Abdullah, A. Madain and Y. Jararweh. 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS). 2022.

[3] Consistency Analysis of ChatGPT. Jang, Myeongjun and Thomas Lukasiewicz. ArXiv. 2023.

[4] Research on Voice Interaction Technology in VR Environment. C. Li and B. Tang. 2019 International Conference on Electronic Engineering and Informatics (EEI). 2019.

[5] Collaboration between ChatGPT API and A-Frame. Takashi Yoshinaga. Youtube. 2023.