Outline

• Lenses, VR Headsets, and the Eye
• Cameras
• Quiz
• Gamepad API

Introduction

• Last week, we talked about optics and its relationship to VR.
• We gave the thin lens equation and showed how it works for both convex and concave lenses.
• We defined diopters and how to add them to figure out the effect of a sequence of lenses.
• We described different kinds of optical aberrations such as chromatic aberrations, spherical aberrations, barrel and pincushion distortion, astigmatism, coma, and flare.
• We then looked at how light passes through the different materials of the eye based on the indexes of refraction.
• Today, we start by continuing our discussion of the eye as an optical device.

The Eye as an Optical Device

• The outer diameter of the eye is 24mm which, from lens equation, would imply one needs a 40 diopter (D) lens to converge parallel rays onto the retina.
• The eye has two lens the cornea, which is about 40D, and the eye lens, which is about 20D.
• This gives the eye a combined power of 60D and allows it to focus parallel rays onto the retina, which, as it is inside the eye, is 17mm behind the outer cornea. (First image above).
• To bring nearer objects into focus, the eye muscles pull on the eye lens to stretch it, increasing its number of diopters through a process called accomodation. (Two right images above).
• In young children, this can be done to increase the number of diopters of the eye by 15D to 20D. Young adults can accommodate there eyes by 10D, adults over 50 have little or no ability to do eye lens accommodation, a condition known as presbyopia.

Vision Abnormalities

• For people with hyperopia (farsightedness) the relaxed eye does not have enough power to converge parallel rays onto the retina. This can be fixed by the placement of a convex lens in front of the eye.
• The opposite condition myopia (nearsightedness) is caused when the relaxed eye has too much converging power. It can be fixed using a concave lens.
• Astigmatism in the eye can be corrected with compensatory elliptically shaped lenses.
• Each eye on a human may or may not have a different problem and so the two lenses in glasses may be different from each other.
• Chromatic aberrations in the eye are often corrected for by the brain automatically based on the years of experience it has studying inputs with these kinds of aberrations.

VR Headset Model

• A typical VR headset has a screen placed less than 10cm from the eye -- a shorter distance than a young adult can accommodate.
• To fix this, we use a convex lens to magnify the screen so that it fills the field of view.
• The lens is usually placed at its focal length from the screen (figure above) so that rays of light emitted from a point on the screen are parallel on the other side of the lens.
• To the eye observing these rays, it is as if the light coming from the screen is placed infinitely far away.
• To account for people with vision problems, if the VR headset is too small to accommodate a person wearing their glasses, then a focusing knob for each eye might be used to adjust the distance of the lens to the given eye.
• Ideally, the lens should be centered perfectly in front of the cornea.
• For this reason, some headsets have an adjustment to vary the distance between the eyes (the interpupillary distance), a standard range might be between 55 to 75mm. This won't account for the fact that your eyes often slightly rotate in their sockets.

Cameras

• Cameras are engineered eyes which might be suitable for recording a scene.
• Camera obscuras (pinhole camera), which are centuries old but nowadays are used to safely observe solar eclipses, consist of a box the inside of which is often painted black with a pinhole in it through which light can enter and shine against the opposite side of the box on a hung white sheet (for larger boxes) or a piece of paper. (See left diagram.)
• The image on the sheet is upside down from the original image but may be traced (historically what painter's did) or observed by a person or person's head inside the box. A mirror can also be employed to uninvert the image.
• Since the 1820s, with photos like the View from the Window at Le Gras above by Nicephore Niepce, various chemical based processes were used to etch the image automatically onto a imaging surface.
• Film was popular throughout the 20th century until digital cameras were developed to allow the electronic capture of the image without the etching process.
• Two popular digital camera technologies ares Charged Coupled Device (CCD) arrays and CMOS active-pixel image sensors (Right pictures above).
• These technologies record the amount of light hitting a pixel location along the image to directly produce a captured image.
• Each pixel location has sensors which respond to different frequencies of light, say red, green, blue.

Shutters and Aperture

• Two ways to control the image recorded by our sensor are:
  1. Control how long the sensor is exposed to photons from the scene to be captured
  2. Control from how wide an angle those photons are allowed to come
• A shutter is an opaque object placed between the image source and the image sensor.
• To take a picture, the shutter is removed for some period (the shutter speed) before being reinserted between image source and the sensor.
• A longer exposure/lower shutter speed will result in more photons being captured, but will also cause moving objects to become blurry.
• CMOS sensors work by sending out the image line-by-line and so use a rolling shutter. I.e., Shuttering for line `i` of the image is done just before line `i` is read from the sensor, and before line `i+1` is read. This can yield weird effects for moving objects such as the helicopter with curved bladed above.
• CCDs on the other hand capture the whole image at once using a global shutter so don't suffer from this problem. Until recently though, they have been more expensive.
• The aperture is the size of the hole through which light comes into image aparatus. It is usually expressed as the ratio of the focal length to the diameter of the entrance hole
• The smaller the ratio the more crisp the image will be as light from fewer angles will hit a given area on the sensor, on other hand, the smaller the ratio the darker image will tend to be.

Quiz

Which of the following statements is true?

1. Red light is higher frequency than blue light.
2. The number of diopters of a lens is `1/f` where `f` is its focal length.
3. Fish-eye lenses used in VR headsets usually cause pin-cushion distortions.

Gamepad API

• The Gamepad API is a browser API that can be used to determine the state of any controllers connected to your computer.
• The first browser to support it was Chromium in 2011.
• For a VR Headset, it can be used to determine the state of the controller for the headset when viewing a WebVR page.
• In the case of the Oculus Go, the Gamepad API is only active when viewing a page in VR mode as far as I can tell.
• To demonstrate the Gamepad API, I [modified the spinning cube example] to print console messages of the current controller state.
• We can then view these message via: adb logcat -s chromium.

Handling Connecting/Disconnecting Gamepads.

• Two events are associated with gamepads and the browser:
  1. gamepadconnected
  2. gamepaddisconnected
• In the example, we set up callbacks for these events in our main() function from before:

  function gamepadHandler(e, connecting)
  {
      var gamepad = e.gamepad;
      if (connecting) {
         console.log("Gamepad connected at index " +
            gamepad.index + ": "+ gamepad.id + ". " +
            gamepad.buttons.length + " buttons," + 
            gamepad.axes.length + " axes.");
      } else {
         console.log("Gamepad disconnected from index "+ 
             gamepad.index + ": "+ gamepad.id);
      }
  }
  window.addEventListener("gamepadconnected", function(e) {
     gamepadHandler(e, true); }, false);
  window.addEventListener("gamepaddisconnected", function(e) {
     gamepadHandler(e, false); }, false);
  

• The connection message looks like:

  03-18 11:57:25.554  2233  2233 I chromium: [INFO:CONSOLE(215)] 
  "Gamepad connected at index 0: Oculus Go Controller. 2 buttons,
  2 axes.", source: https://www.cs.sjsu.edu/faculty/pollett/
  185c.1.19s/WebGamePad.html (215)
  

  in the log.

Monitoring Properties of the Gamepads.

• In a typical scenario, as part of the update phase of your game loop, you would cycle through the Gamepads connected and for each one find out its properties then use those properties to do stuff.
• In our cube example, we didn't really have a game loop, updating the scene was done as part of drawScene(), so we add some Gamepad polling there:

  gamepads = navigator.getGamepads();
  if (gamepads != {}) {
  console.log("We have at least one Gamepad!");
  for (var i = 0; i < gamepads.length; i++)
      if (gamepads[i] && gamepads[i].connected) {
          var buttons = gamepads[i].buttons;
          console.log("Num Buttons" + buttons.length);
          // asssuming Oculus Go so know button number should be 2.
          console.log("Button 0 pressed? " + gamepads[i].buttons[0].pressed);
          console.log("Button 1 pressed? " + gamepads[i].buttons[1].pressed);
          console.log("Left right trackpad value " + gamepads[i].axes[0]);
          console.log("Up down trackpad value " + gamepads[i].axes[1]);
          console.log("Hand controller is for " + gamepads[i].hand + " hand.");
          console.log("Controller supports pose " + 
              gamepads[i].pose.hasOrientation);
          console.log("Pos orientation quarternion " +
              gamepads[i].pose.orientation);
      }
  }
  

• This code generates log messages like:

  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(425)] "We have at least one Gamepad!", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (425)
  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(429)] "Num Buttons2", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (429)
  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(431)] "Button 0 pressed? false", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (431)
  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(432)] "Button 1 pressed? false", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (432)
  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(433)] "Left right trackpad value 0", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (433)
  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(434)] "Up down trackpad value 0", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (434)
  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(435)] "Hand controller is for right hand.", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (435)
  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(436)] "Controller supports pose true", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (436)
  03-18 12:10:10.533  2233  2233 I chromium: [INFO:CONSOLE(437)] "Pos orientation quarternion 0.0631496012210846,-0.12882277369499207,0.023658324033021927,0.9893720746040344", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (437)
  03-18 12:10:10.658  2233 11306 I chromium: [INFO:vr_shell_gl.cc(1493)] Left VR
  03-18 12:10:10.667  2233  2233 I chromium: [INFO:CONSOLE(219)] "Gamepad disconnected from index 0: Oculus Go Controller", source: https://www.cs.sjsu.edu/faculty/pollett/185c.1.19s/WebGamePad.html (219)
  

Some Remarks on the Code

• Warning: If you are lazy and try to do something like JSON.stringify(gamepad), you will never see any properties for your gamepads.
• Notice to get an array of Gamepad objects, we used navigator.getGamepads().
• If this isn't, empty we cycle over the array of gamepads (an Oculus Rift for example might have two).
• We use the connected property of the Gamepad to determine if it is available to send inputs to the browser.
• Each Gamepad has a buttons property. For an Oculus Go, this will be an array of length 2. One button is the trigger, the other is for clicking the track pad.
• Gamepad also has an axes property. The 0 coordinate of this is the left-right position on the track pad as a value between -1 and 1. The 1 coordinate of this is the up-down position on the track pad as a value between -1 and 1.
• The Gamepad hand property can be used to determine if the Gamepad is set up for the left or right hand.
• The Gamepad pose property of type GamepadPose has information about the position and orientation of the Gamepad
• Not all gamepads support orientation and position info. For example, the Oculus Go supports orientation but not position. The hasOrientation and hasPosition properties of pose let us find out whether or not these properties are present for the controller.
• If present, a GamepadPose has a position property with a 3D vector for the position, and a orientation with a quaternion for the orientation.
• On the Mozilla Development Page for GamepadPose one can see it also has properties for various kinds of linear and angular acceleration.

Drawing a Gamepad and Checking for Intersections

• The getPoseMatrix code from before:

  function getPoseMatrix(out, pose)
  {
      var orientation = pose.orientation;
      if (!orientation) {
          orientation = [0, 0, 0, 1];
      }
      mat4.fromQuat(out, orientation);
      mat4.invert(out, out);
  }
  

  which we used to get a matrix to convert into the headset orientation can also be used to convert the orientation of a Gamepad pose to a matrix.
• If we had a model that we wanted to draw for our gamepad in the scene, with an associated modelViewMatrix, we could use the above to calculate a poseMatrix for our gamepad, then compute:

  mat4.multiply(vrModelViewMatrix, poseMatrix, modelViewMatrix);
  gl.uniformMatrix4fv(programInfo.uniformLocations.modelViewMatrix,
     false, vrModelViewMatrix);
  // code as before to set up project matrix and viewport for given eye
  gl.drawElements(gl.TRIANGLES, vertexCount, type, offset);
  

  to draw the gamepad.
• The orientation property can be used to calculate the 3D axes of the orientation quaternion, a vector that is where the gamepad is pointing.
• We also know the position at which we have placed the gamepad in the scene as a vector.
• Together these give us the line that the gamepad is pointing the through in the scene and we can use our triangle intersection algorithms from earlier in the semester to determine if this hits any triangles of objects in the scene.
• Perhaps easier, one could also have a bounding sphere around each object in ones scene and then use a line sphere intersection test as per Wikipedia.

A More Advanced Gamepad Test

• GamepadTest [Download as GamepadTest.zip] is a more sophisticated program I wrote to test the Gamepad API.
• It factors our earlier WebGL code using Ecmascript 2015 classes and modules.
• The entry point into the code is VRExperience.js's startExperience() function.
• This sets up WebGL as well as the Scene class and adds a couple Entity's to it (Entity's look like boxes).
• The class Scene extends CompositeEntity which in turn extends Entity.
• A CompositeEntity has two arrays entities and collidables, the first contains all the sub-entities that are on the CompositeEntity, the second contains pairs of entities which may collide.
• In addition to the list of entities, a Scene has a view_entity, which specifies where the Scene should be viewed from.
• The default view_entity of a scene is a PlayerEntity which is a CompositeEntity with a GamepadEntity on it. This in turn is a CompositeEntity with the two Entities to specify the two boxes needed to draw the game pad.
• Any Entity has an update() and render() method.
• startExperience's init() method calls window.requestAnimationFrame(animate). animate calls the update() and render() methods of Scene which recursively calls these methods for its children and then calls window.requestAnimationFrame(animate).
• The update method of the GamepadEntity sets the orientation to draw the entity using the Gamepad API. It also detects if the trigger is pressed using the API and sets the laser to red in this case. If the laser pointer is pointed at another entity, during an update of the scene the collision will be detected and handled (box changes color and is deleted) when checkAndHandleCollision(entity) of GamepadEntity is called.
• The default update method of an Entity updates the position and orientation of the entity according to its velocity, axis, frequency and acceleration.