Outline

• Finish VR Implications of Human Vision
• Quiz
• Depth Perception and VR
• Unity

Introduction

• Last week, we were talking about human vision and its implications for VR.
• We had discussed that at 1.5in, we'd need a display to be about 2300 PPI for pixels not to be noticeable. This compares with around 600 PPI for most home use VR systems.
• We said humans have about a 270 degree field of vision; whereas, the best a flat screen can give is 180 degrees, and a device like the Oculus Go has a field of vision of about 110 degrees and AR sets like Hololen2 and Magic Leap are around 50-60 degrees.
• We are going to finish up talking about vision today, talk about perception, and then introduce how to compile Unity apps.

Foveated Rendering

• We have said that photoreceptor density decreases away from the fovea.
• Most VR systems render images at the same level of detail all across their screens.
• If we could track where the eye was looking, we could in theory render only the portion of the scene right in front of the eye in high detail, and thus save computational resources.
• This is called foveated rendering and currently is not in use in any commercial system, but has been shown to work.
• One key problem is its too costly to ensure that the delay in eye tracking is small enough not to cause issues such as discrepancies between eye movement and display updates.

VOR Gain Adaptation

• VOR Gain is a ratio that compares eye rotation rate (numerator) to counter the rotation and translation of the head (denominator) when tracking an object as your head moves.
• The head has six degrees of freedom (3 position, 3 rotation), so this number is often broken into 6 components.
• For pitch and yaw, VOR Gain is close to 1. I.e., if you yaw head to left 10 degrees/second, your eye yaws 10 degrees/second in the opposite way.
• VOR roll gain is very small because the eye does not have much roll range.
• Suppose you are wearing a VR headset that has an imperfect optical system, some tracking latency, and suppose there may be some imperfectly rendered objects on the screen.
• Your visual system could convince your brain, through its ability to adapt, that the headset is functioning correctly, and that you are stationary in some scene.
• Now when you take off the headset, your perception of stationarity will need to re-adapt, and you might actually do things like yaw your head in the real world to compensate for the appearance of stationary objects moving back and forth.

Display scanout

• Earlier in the semester, we mentioned that cameras either have a rolling or global shutter based on whether the sensing elements are scanned line-by-line (as in CMOS) or in parallel (as in CCD).
• Displays work the same way, most displays have a rolling scanout because they trace their output format to old school cathode ray tube (CRT) displays. It is also possible to have a global scanout.
• In a CRT, an electron gun is controlled by a magnet, and scans from the top left of the screen to the bottom right, line-by-line. When the gun is over a region that needs to be lit, it fires and causes a phosphor element region of the screen to light up. (For Color TVs, had three different colors of phosphor elements).
• Because of the slow response of our photoreceptors, under most circumstances, we would not see this scanning process.
• However, if our eyes and/or features in the scene are moving, then things like a vertical line can appear as if they are tilted. See the figures above.
• Another problem with displays is that pixels may have some delay before switching values (up to 20ms), this may cause blurring effects.

Retinal Image Slip

• Eye movements contribute both to maintaining a target in a fixed location on the retina (smooth pursuit, VOR) and also changing its location slightly to reduce perceptual fading (microsaccades).
• As the eye is always moving, during normal, non-VR activities, the image of a feature on the retina may move slightly due to these motions and optical distortions.
• This is called retinal image slip.
• When wearing a VR headset, it sometimes happens that the image slip one gets is not like one would get in normal circumstances.
• This is because of optical distortions in the VR lenses, tracking latency, and display scanout.
• This may cause the eye to do more microsaccades than in normal circumstances and cause it to become tired and contribute to VR sickness.

Vergence-accommodation mismatch

• We already said that vergence-accommodation mismatch occurs when the eyes need to change focus (accommodation) doesn't match the direction change the eyes needs to make to focus stereoscopically on an object (convergence).
• As in a VR headset you only need to accommodate, as the screen is always the same distance from the eye, you are essentially training a headset user to be able to separate these two kinds of motion. This can lead to fatigue.
• Some attempts have been made to reduce this mismatch by using eye tracking to estimate the amount of convergence and then reducing the power of the optical system to match.

Quiz

Which of the following statements is true?

1. MPEG4-MVC encodes 3D-180 video as a sequence of side-by-side 180 degree video frames.
2. Scotopic vision is a fancy name for color vision.
3. The six eye muscles can make the eye pitch and yaw, but can't easily make it roll.

Visual Perception

• We now switch our focus to VR related to perception.
• What we have been talking about so far is human vision from a physiology standpoint.
• Perception refers to how we perceive the world based on interpreting these physiological inputs.
• We are mainly interested in the perception of distance (we see a scene, how do we figure out how far away something is based on its scale?), perception of motion (how fast is something moving), and color (can we accurately fake all colors humans can perceive by using only three colors?)

Perception of Depth

• Perceived distance can be either metric, an estimate of how far something is (say in meters), or ordinal, this object is in front of some other object.
• The eye makes use of both single eye (monocular) and both eyes together (stereoscopic) sensory cues to determine distance. That is, pieces of information derived from sensory stimulation relevant to perception.
• For one eye, these cues might be things like image size for familiar objects (for example, humans, coins, basketballs.)
• This size should be proportional to `1/z`, the `z` distance of the object from the eye, and is called the retinal image size cue.
• Our brains automatically perform size constancy scaling of familiar objects by assuming that the distance to an object is changing rather than its size as it come closer.
• Optical illusions such as the images above show this scaling is not always correct.
• Size constancy falls under one area of general perception known as subjective constancy.
• The reverse process can also happen, if we don't know the size of an unfamiliar object, we might uses its depth to estimate its size. This is called size perception.

Other Monocular Depth Cues

• Visual angle near the horizon versus higher up in sky. Perceived angle on the horizon often seems to be larger than actual and might contribute to effect like the moon seeming bigger when near the horizon.
• Height in the visual field. Due to perspective projection, the horizon is a line that divides the view in half, between sky and ground. Objects closer to the horizon are often perceived as farther away.
• Accommodation. How much our eyes have had to accommodate to see an object clearly also tells us something about its size.
• Motion parallax. Seeing one object by the same eye from two different viewpoints can be a depth cue. Nearby objects moving the same amount in a giving time change more in visual position in the eye than far away objects moving the same amount. This relative difference in motions is called parallax.
• Shadows, Ordinal Depth (interposition), Image Blur. Lengths of shadows, an object being in front of another, and bluriness associated with depth can all also provide depth cues.

Stereo-Depth Cues

• When focusing both eyes on an object, humans perceive a single focused image over a surface in space called a horopter (see figure above).
• Motor control of the eye muscles provides information to the brain about the amount of convergence, providing a direct estimate of distance.
• Each eye has a slightly different viewpoint in a scene, and thus gets a different image on its retina, in a phenomena known as retinal disparity. Even though the eyes have rotated slightly to converge so they have close to the same image, this can be a powerful depth cue.
• When converging on an object at one depth, one often has double images at other depths, although we don't usually pay attention to it. This phenomena is called diplopia and can also be a depth cue.

VR and Depth Cues

• Incorrect Scale A VR world may be filled with unfamiliar objects that may or may not be similar to familiar ones. Given a lack of the familiar, it might be hard to judge sizes of things -- this can further be compounded by the lack of cues of ones own height in VR. For example, the above image shows a villa in Tuscany. It should be habitable by humans, but it lacks enough familiar objects to be able to answer questions like how big the tiles are or if the desk is too low. Another compounding problem is that users often sit to experience VR, but may be position in a scene as if standing. Further, often the interpupillary distance in the headset does not match the user's actual interpupillary distance.
• Mismatches. Depth cues work in concert. Vergence-accommodation mismatch, imperfect head tracking, latency, etc. can cause visual stimuli not to appear in the correct place at the correct time. Another issue is some tracking systems track the head orientation only, not position. This makes it impossible to use motion parallax cues if the user's head moves from side to side without any rotation.
• Monocular Cues. There tend to be more monocular depth cues than stereo ones. So we still perceive some 3D even if we only have monocular cues. This can be useful in saving computation resources. It means we might be able to to get away with easier, faster, 360 images and video for things faraway.

Unity

• I am going to briefly switch topics now, and talk about the Unity Game engine which you will need for HW4.
• We will talk about this engine for a bit before finishing Ch 6 next day.
• Unity is a cross-platform gaming engine developed by Unity Tecchnologies first released in 2005.
• It can be used to build games, for Windows, MacOS, Android, iOS, Playstation, Wii U, Steam, Oculus Go, Oculus Rift, HTC Vive, etc.
• You can developed in it and switch platforms relatively painlessly -- it sorta just works.
• Coding is done in C#, which is relatively easy to pick up if you know Java.
• The free Unity personal edition version of the SDK can be downloaded from the Unity store.
• I recommend downloading it, if you haven't, before this Wednesday's Lecture.

Building a Dummy App for the Oculus Go

• These instructions assume you've already installed the software needed to do native app development (you've got both the Java and Android SDKs, etc) the for Oculus Go.
• First, you want to launch Unity and create a new 3D project.
• Next, you want to configure the Build Settings and target the Android Platform:
  
  make sure to click switch platform.
• Next, still under Build Settings, click on Player Settings, and under XR settings, check Virtual Reality Supported, and click the + to choose the Oculus VR SDK.
  
• You then want to download and install, the Oculus Integration package from the Asset Store (on a Mac under Window file menu or click icon).
  
• We'll pick it up from here next day.