Deliverable 3: ChatGPT Modeling Interpreter

Description:

The goal of this deliverable is to ensure that the natural language responses from ChatGPT are converted into actionable commands for the A-Frame environment. This process includes defining a command structure for creating, modifying, and removing objects as well as establishing a foundation for user interactions to be properly interpreted and transformed into modeling commands. This means that the system needs to be robust, flexible, and capable of handling a wide range of user inputs.

Implementation Steps:

Define the command structure
- Specify system's behavior to standardize ChatGPT responses
- Creation (primitives, a-entity)
- Modification (color, position, geometry, components)
- Deletion
Extract A-Frame code from ChatGPT responses
- Exclude extra text and only get A-frame code
- Process/clean response code
Execute modeling command
- Test various types of user inputs
- Test consistency of responses from ChatGPT
Error handling
- Resolve ambiguous and invalid commands
- Set up error messages for feedback

System Design:

Model: gpt-4 / gpt-4-turbo
Temperature: 0.2
System Instructions: "You are an assistant that generates A-Frame markup code for the A-Frame version 1.4.0 and newer. Your primary task is to produce A-Frame code that include A-Frame tags, objects, and components while following the conditions below:
- Always keep track of all objects in the current scene.
- The default user position: (0 0 0), facing -z axis. So, when creating objects, place them in view of the user.
- Each A-Frame tag should have a unique id, attribute shadow=receive: true, and hex colors.
- Do not use primitive object tags. Instead use <a-entity> with the appropriate A-frame supported geometry and attributes.
- Omit any comments, explanations or extra text in your responses, focus only on the A-Frame code.
- If the input is invalid or unrelated, provide a concise error message inside an <error> tag.
- If the input is very vague, ask a concise clarification question inside a <p> tag.
- Always add any new object to the whole scene and organize the complete generated code within an <a-scene> tag.
- For objects made up of multiple objects, group them inside one <a-entity> object.
- Take into account user feedback and iterate on adjustments provided by the user."

Testing and Results:

Simple Inputs Test Set

Test	Input	Description	Pass/Fail
1	Create a red cube	Test basic creation	1
2	Add a green sphere	Test different shape creation	1
3	Move the cube to the left	Test position modification	1
4	Move the sphere up	Test different position modification	1
5	Rotate the cube 45 degrees	Test rotation modification	1
6	Scale the sphere to half the size	Test scale modification	1
7	Change the cube's color to blue	Test color modification	1
8	Delete the sphere	Test object deletion	1
9	Delete the cube	Test object deletion	1
10	Create a yellow cone	Test different object creation	1
11	Add a horizontal light blue plane below the cone	Test different object creation, position, and rotation	1
12	Add a vertical white plane behind the cone	Test different object creation, position, and rotation	1
13	Flip the cone upside down	Test rotation modification	1
14	Add a purple cylinder on top of the cylinder	Test different object creation and position	1
15	Remove the second plane	Test object deletion with context	1

Test 7 example:

Test 14 example:

Complex Inputs Test Set ( * = required adjustments)

Test	Input	Description	Pass/Fail
1	Create a regular light brown rectangular table	Test complex object creation	0
2	Make sure the top of the table is touching the top of the legs	Test iterative adjustments	1
3	Add a floor. Make it light blue and position it below the table	Test object creation and specific object modifications	1
4	Rotate the whole table 90 degrees horizontally	Test complex object rotation modification	1
5	Use the floorTex as a src texture on the floor	Test texture mapping	1
6	Add a thin white wall, made up of a 4x4 grid of multiple boxes behind the table	Test complex object creation and positioning	1
7	Remove table objects and cut out a window out of the wall	Test removal and complex object modification	1
8	Undo the last 2 changes	Test undo	0
9	Add a tall red cube to the A-Frame environment. Make sure it is not touching anything else and is still within the user's view	Test long specific input	1
10	Animate the cube to constantly rotate horizontally	Test animation	1
11	Remove everything, except the floor	Test mass removal	1
12	Add a couch to the scene	Test complex object creation	1*
13	Create a minecraft character standing left of the couch	Test different complex object creation and positioning	1*
14	Add a tree behind the couch, base it on an L-system made up of cylinders	Test different complex object creation	0
15	Clear the scene and add 20 spheres manually, with random colors and sizes not overlapping	Test creation of multiple objects of different sizes	1*
16	Clear the scene and add a floor	Test mass deletion	1
17	Create a cool UFO in the scene	Test complex object creation	1*
18	Animate the whole ufo taking off to the sky at an angle away from me	Test complex object animation	1

Test 3 example:

Test 5 example:

Test 8 example:

Test 10 example:

Test 12 example:

Test 12 example (Alternative):

Test 13 example:

Test 14 example:

Test 14 example (Different angle):

Test 15 example:

Test 17 example:

Test 18 example:

Invalid and Ambiguous Inputs Test Set

Test	Input	Description	Pass/Fail
1	Make something	Test ambiguous creation input	1
2	Move it	Test ambiguous modification input	1
3	Create the thing	Test ambiguous creation input	1
4	Create an animal	Test ambiguous creation input	1
5	Recommend me something	Test ambiguous input	1
6	What should I eat today?	Test invalid input	1
7	How's it going?	Test invalid input	1
8	Make me a game	Test invalid input	1
9	Call my friend Alex	Test invalid input	1
10	Create a cube and move it to japan	Test invalid modification input	1

Test 3 example:

Test 7 example:

Semantic Consistency Test Set

Test	Input 1	Input 2	Equivalent
1	Create a red cube	Generate a red cube	Yes
2	Add a green sphere	Include a green sphere	Yes
3	Move the cube to the left	Shift the cube to the left	Yes
4	Move the sphere up	Elevate the sphere	Yes
5	Rotate the cube 45 degrees	Turn the cube 45 degrees	Yes
6	Scale the sphere to half the size	Make the sphere half its size	No
7	Change the cube's color to blue	Modify the cube to be blue	Yes
8	Delete the sphere	Eliminate the sphere	Yes
9	Delete the cube	Clear the box	Yes
10	Create a yellow cone	Craft a cone that is yellow	Yes

Test 7 comparison:

Negation Consistency Test Set (modifications/deletions are meant to act on an object)

Test	Input 1	Input 2	Equivalent
1	Create a red cube	Do not create a red cube	No
2	Add a green sphere	Avoid adding a green sphere	No
3	Move the cube to the left	Don't move the cube to the left	No
4	Move the sphere up	Prevent the sphere form moving up	No
5	Rotate the cube 45 degrees	Do not rotate the cube 45 degrees	No
6	Scale the sphere to half the size	Do not scale the sphere to half its size	No
7	Change the cube's color to blue	Skip changing the cube's color to be blue	No
8	Delete the sphere	Don't delete the sphere	No
9	Delete the cube	Avoid deleting the cube	No
10	Create a yellow cone	Do not create a yellow cone	No

Test 5 comparison:

Symmetric Consistency Test Set

Test	Input 1	Input 2	Equivalent
1	Create a red cube, Add a green sphere	Add a green sphere, Create a red cube	No
2	Move the cube to the left, Move the sphere up	Move the sphere up, Move the cube to the left	Yes
3	Rotate the cube 45 degrees, Scale the sphere to half the size	Scale the sphere to half the size, Rotate the cube 45 degrees	Yes
4	Change the cube's color to blue, Delete the sphere	Delete the sphere, Change the cube's color to blue	Yes
5	Delete the cube, Create a yellow cone	Create a yellow cone, Delete the cube	Yes

Test 3 comparison: