Chris Pollett > Students >
Ivan

    ( Print View)

    [Bio]

    [Blog]

    [C297 Proposal]

    [Deliverable 1]

    [Deliverable 2]

    [Deliverable 3]

    [Deliverable 4]

    [CS297 Report (PDF)]

    [Paper 1: A-Frame Virtual Reality (PDF)]

    [Paper 2: ChatGPT - Fundamentals and Applications (PDF)]

    [Paper 3: ChatGPT Consistency (PDF)]

    [Paper 4: VR Voice Interaction (PDF)]

    [C298 Proposal]

Deliverable 3: ChatGPT Modeling Interpreter


Description:

The goal of this deliverable is to ensure that the natural language responses from ChatGPT are converted into actionable commands for the A-Frame environment. This process includes defining a command structure for creating, modifying, and removing objects as well as establishing a foundation for user interactions to be properly interpreted and transformed into modeling commands. This means that the system needs to be robust, flexible, and capable of handling a wide range of user inputs.

Implementation Steps:

  1. Define the command structure
    • Specify system's behavior to standardize ChatGPT responses
    • Creation (primitives, a-entity)
    • Modification (color, position, geometry, components)
    • Deletion
  2. Extract A-Frame code from ChatGPT responses
    • Exclude extra text and only get A-frame code
    • Process/clean response code
  3. Execute modeling command
    • Test various types of user inputs
    • Test consistency of responses from ChatGPT
  4. Error handling
    • Resolve ambiguous and invalid commands
    • Set up error messages for feedback

System Design:

  • Model: gpt-4 / gpt-4-turbo
  • Temperature: 0.2
  • System Instructions: "You are an assistant that generates A-Frame markup code for the A-Frame version 1.4.0 and newer. Your primary task is to produce A-Frame code that include A-Frame tags, objects, and components while following the conditions below:
    • Always keep track of all objects in the current scene.
    • The default user position: (0 0 0), facing -z axis. So, when creating objects, place them in view of the user.
    • Each A-Frame tag should have a unique id, attribute shadow=receive: true, and hex colors.
    • Do not use primitive object tags. Instead use <a-entity> with the appropriate A-frame supported geometry and attributes.
    • Omit any comments, explanations or extra text in your responses, focus only on the A-Frame code.
    • If the input is invalid or unrelated, provide a concise error message inside an <error> tag.
    • If the input is very vague, ask a concise clarification question inside a <p> tag.
    • Always add any new object to the whole scene and organize the complete generated code within an <a-scene> tag.
    • For objects made up of multiple objects, group them inside one <a-entity> object.
    • Take into account user feedback and iterate on adjustments provided by the user."

Testing and Results:

  1. Simple Inputs Test Set

    Test Input Description Pass/Fail
    1 Create a red cube Test basic creation 1
    2 Add a green sphere Test different shape creation 1
    3 Move the cube to the left Test position modification 1
    4 Move the sphere up Test different position modification 1
    5 Rotate the cube 45 degrees Test rotation modification 1
    6 Scale the sphere to half the size Test scale modification 1
    7 Change the cube's color to blue Test color modification 1
    8 Delete the sphere Test object deletion 1
    9 Delete the cube Test object deletion 1
    10 Create a yellow cone Test different object creation 1
    11 Add a horizontal light blue plane below the cone Test different object creation, position, and rotation 1
    12 Add a vertical white plane behind the cone Test different object creation, position, and rotation 1
    13 Flip the cone upside down Test rotation modification 1
    14 Add a purple cylinder on top of the cylinder Test different object creation and position 1
    15 Remove the second plane Test object deletion with context 1

    Test 7 example:

    Simple Example

    Test 14 example:

    Simple Example

  2. Complex Inputs Test Set ( * = required adjustments)

    Test Input Description Pass/Fail
    1 Create a regular light brown rectangular table Test complex object creation 0
    2 Make sure the top of the table is touching the top of the legs Test iterative adjustments 1
    3 Add a floor. Make it light blue and position it below the table Test object creation and specific object modifications 1
    4 Rotate the whole table 90 degrees horizontally Test complex object rotation modification 1
    5 Use the floorTex as a src texture on the floor Test texture mapping 1
    6 Add a thin white wall, made up of a 4x4 grid of multiple boxes behind the table Test complex object creation and positioning 1
    7 Remove table objects and cut out a window out of the wall Test removal and complex object modification 1
    8 Undo the last 2 changes Test undo 0
    9 Add a tall red cube to the A-Frame environment. Make sure it is not touching anything else and is still within the user's view Test long specific input 1
    10 Animate the cube to constantly rotate horizontally Test animation 1
    11 Remove everything, except the floor Test mass removal 1
    12 Add a couch to the scene Test complex object creation 1*
    13 Create a minecraft character standing left of the couch Test different complex object creation and positioning 1*
    14 Add a tree behind the couch, base it on an L-system made up of cylinders Test different complex object creation 0
    15 Clear the scene and add 20 spheres manually, with random colors and sizes not overlapping Test creation of multiple objects of different sizes 1*
    16 Clear the scene and add a floor Test mass deletion 1
    17 Create a cool UFO in the scene Test complex object creation 1*
    18 Animate the whole ufo taking off to the sky at an angle away from me Test complex object animation 1

    Test 3 example:

    Complex Example

    Test 5 example:

    Complex Example

    Test 8 example:

    Complex Example

    Test 10 example:

    Complex Example

    Test 12 example:

    Complex Example

    Test 12 example (Alternative):

    Complex Example

    Test 13 example:

    Complex Example

    Test 14 example:

    Complex Example

    Test 14 example (Different angle):

    Complex Example

    Test 15 example:

    Complex Example

    Test 17 example:

    Complex Example

    Test 18 example:

    Complex Example

  3. Invalid and Ambiguous Inputs Test Set

    Test Input Description Pass/Fail
    1 Make something Test ambiguous creation input 1
    2 Move it Test ambiguous modification input 1
    3 Create the thing Test ambiguous creation input 1
    4 Create an animal Test ambiguous creation input 1
    5 Recommend me something Test ambiguous input 1
    6 What should I eat today? Test invalid input 1
    7 How's it going? Test invalid input 1
    8 Make me a game Test invalid input 1
    9 Call my friend Alex Test invalid input 1
    10 Create a cube and move it to japan Test invalid modification input 1

    Test 3 example:

    Invalid Example

    Test 7 example:

    Invalid Example

  4. Semantic Consistency Test Set

    Test Input 1 Input 2 Equivalent
    1 Create a red cube Generate a red cube Yes
    2 Add a green sphere Include a green sphere Yes
    3 Move the cube to the left Shift the cube to the left Yes
    4 Move the sphere up Elevate the sphere Yes
    5 Rotate the cube 45 degrees Turn the cube 45 degrees Yes
    6 Scale the sphere to half the size Make the sphere half its size No
    7 Change the cube's color to blue Modify the cube to be blue Yes
    8 Delete the sphere Eliminate the sphere Yes
    9 Delete the cube Clear the box Yes
    10 Create a yellow cone Craft a cone that is yellow Yes

    Test 7 comparison:

    Semantic Example Semantic Example

  5. Negation Consistency Test Set (modifications/deletions are meant to act on an object)

    Test Input 1 Input 2 Equivalent
    1 Create a red cube Do not create a red cube No
    2 Add a green sphere Avoid adding a green sphere No
    3 Move the cube to the left Don't move the cube to the left No
    4 Move the sphere up Prevent the sphere form moving up No
    5 Rotate the cube 45 degrees Do not rotate the cube 45 degrees No
    6 Scale the sphere to half the size Do not scale the sphere to half its size No
    7 Change the cube's color to blue Skip changing the cube's color to be blue No
    8 Delete the sphere Don't delete the sphere No
    9 Delete the cube Avoid deleting the cube No
    10 Create a yellow cone Do not create a yellow cone No

    Test 5 comparison:

    Negation Example Negation Example

  6. Symmetric Consistency Test Set

    Test Input 1 Input 2 Equivalent
    1 Create a red cube, Add a green sphere Add a green sphere, Create a red cube No
    2 Move the cube to the left, Move the sphere up Move the sphere up, Move the cube to the left Yes
    3 Rotate the cube 45 degrees, Scale the sphere to half the size Scale the sphere to half the size, Rotate the cube 45 degrees Yes
    4 Change the cube's color to blue, Delete the sphere Delete the sphere, Change the cube's color to blue Yes
    5 Delete the cube, Create a yellow cone Create a yellow cone, Delete the cube Yes

    Test 3 comparison:

    Symmetric Example Symmetric Example