Outline

• Perception of Motion
• Quiz
• Perception of Color

Introduction

• Last week, we were talking about the Unity Game engine.
• Before that we had been talking about perception.
• Recall physiology tells us how inputs from our body get to our brain.
• Perception refers to how we perceive the world based on interpreting these physiological inputs.
• So far we've discussed the perception of depth based on monocular and stereoscopic cues and their VR implications.
• Today, we resume the discussion of perception by considering the perception of motion and color and how different sources of information get combined in what we perceive.

Perception of Motion

• The ability to perceive motion lets us separate moving figures from a stationary background.
• This can be beneficial if the moving figure might hurt us or eat us.
• Motion also lets us figure out the 3D structure of objects, for example, we can better understand if an apple is good to eat, or if it is severely bruised, by rotating it around and looking at it.
• Non-motion, the perception of stationarity, is also important. For example, if we are using our Vestibulo-Ocular Reflex (VOR) to track an object in VR, but there is some slight latency, then what should have been non-moving objects with respect to our brain might appear to move in a disorienting way.

Brain Mechanisms for Detecting Motion

• Several different mechanisms in the brain are used to detect kinds of motion:
  • A Reichardt Detector is a brain circuit which responds to directional motion. As shown in the left figure above, ganglion neurons and the LGN can detect simple features (such as vertical/horizontal lines) in different spots in the retinal image. Higher levels motion detection neurons exist that respond when the feature moves from one spot to another. Speed can be detected by neurons at this level by variations in the length of the input paths from the from the feature detection neurons. Odd spacing of features such as spokes on a wheel together with motion can confuse this circuitry, for example, to make the spokes look like they are going backwards even if going forwards.
  • Mechanisms for Local Data to Global Conclusions. Motion detectors are local in that only a small part of the visual field causes the detector to fire. Data from many detectors are integrated to give the sense of rigid body motion. Our brain anticipates this coordinated motion to match common expectations. It can be fooled, thus, if a large portion of the moving object is blocked as in the middle figures above. If all we could see was the content of the circular region, we couldn't tell the direction of motion. This is called the aperture problem. This problem can cause vection (illusion of self-motion) and VR sickness.
  • Mechanisms to Distinguish Object Motion from Observer Motion try to solve which of the two right figures above actually represent what is really happening. One such mechanism is saccadic suppression which hides visual signals during movement. This might help rule out the second case above. Another cue is proprioception of the eye muscles, this might indicate the second case. Finally, large scale motion of the whole scene is interpreted as the user moving.

Stroboscopic Apparent Motion

• Stroboscopic apparent motion is caused when a sequence of still images is shown quickly yielding the appearance of motion.
• It was first discovered in the early 19th century. If you go to the world's tallest building before the Eiffel tower, the Mole Antonelliana, you can see a movie museum with Zoetropes like the above one from the 1830s illustrating this effect.
• The effect can still be perceived at frame rates as low as 2 frames per second, even though images persist in the visual cortex for about 100ms.
• The phi phenomenon and beta movement are related effects where motion is perceived when consecutive lights in an array of lights are toggled off and on. If the toggling is more than 15 per second, then a blank dot appear to be moving along the lights. (Right figure above.)
• Both effects are likely caused by triggering/fooling the neural circuitry of the previous slide.

Frame Rates and Motion

• The above chart shows various frame rates at which still images might be displayed in rapid succession and a corresponding thing that can/or cannot be determined by the human visual system.
• Flicker discomfort is caused when humans perceive the alternation between the still image being shown and the black shutter between images. This effect goes away at around 60FPS.
• The effect is more pronounced for peripheral vision where between 70 and 90 FPS is needed to reduce this effect if you have a reasonably close to the eye computer monitor.
• Psychologist study the precise rate at which this effect goes away under the title of flicker fusion threshold.
• For LEDs connected to a pulse generator, flicker can be detected at much higher rates in what's known as the zipper effect. In the zipper effect, the pulsed LED when moved around appears as if it is a moving array of lights. This effect can still be perceived up to 5000FPS.

VR Implications of Motion

• We have already said that VR displays need to be of a much higher resolution so as to avoid effects like distinguishable pixels and aliasing.
• With regards to motion, frame rates of VR systems should be greater than 60FPS to handle various problems such as the perception of stationarity...
• In the real world, if one fixates on a nearby object and rotates ones head left, VOR kicks in rotating your eyes to the right to maintain focus.
• Doing the same thing wearing a VR headset requires the object to shift across the screen while you turn your head.
• Assuming pixels, instantly change at each new frame time, we get a lagging, wobbling drift in motion between frame called judder.
• One way to reduce this, is to have the image quickly turn off after being displayed so you can't see the judder motion as much.
• Doing this, is called a low persistence display mode. The short amount of time the display is on is still enough for the eye to collect light from the scene.
• Above 90FPS, the effects of juddering are less severe for people, and above 500FPS it is no longer perceptible.
• Low persistence requires fast pixel speeds. OLED are limited to about 0.1ms switching speeds, LCDs, which are used in most VR displays, on the other hand, take about 20ms to switch.

Quiz

Which of the following statements is true?

1. In all the examples where we scripted a game object in Unity, we directly invoked the constructor of that object within our script.
2. In our scripts, we used ray casting to determine which objects in a scene were in the direction we were looking.
3. The Unity Input class only works for Oculus devices.

Perception of Color

• Color perception is the result of our visual physiology and neural structure rather than something that can be easily measure in the real world.
• If two people have comparable color perception systems, then they can discuss colors using commonly agreed names.
• The dress color illusion illustrates this subjectivity. We can in the real world measure the pixel color values for the image above.
• However, the image lies at the boundaries of the overlaps between the frequency of lights observable by various people and so it is perceived as different colors by different people. For example, 57% perceive it as blue and black (correct), 30% perceive it as white and gold, 10% as blue and brown, and 10% as oscillating between these.
• There are many possible wavelengths of light, each of which gets reflected off and emitted by materials in different ways.
• The human eye is only sensitive to three distinct narrows ranges of wavelengths of light. So the whole color space is reduced to the 3D color space of the human eye.
• The color yellow provides one example of this dimensionality reduction. Yellow as one frequency of light has a wavelength of 580nm. We don't have a cone in our eye that directly responds to only this frequency, however, such light triggers our green cones (which respond to best 533nm light) and our red cones (which respond best to 564 nm). If instead we show a combination of the two wavelengths red and green, they actually will be perceived also as yellow. So we can write yellow in RGB as (255, 255, 0).

Color Spaces

• In computer graphics, a parameterized color space more closely corresponding to different perceived differences between colors is often used.
• One such scheme is the HSV scheme shown on the left above. It consists of:
  • Hue, the perceived color such as red or green.
  • Saturation, the purity of the color. I.e., how much energy is coming from wavelengths other than the wavelength of the hue. (0 is unpure and corresponds to white light).
  • Value corresponds to the brightness.
• An ideal representation would have distances between two points in it corresponding to the amount of perception difference. The CIE color standard which ignores brightness corresponds to this (Right figure above).
• Trichromatic theory says we can generate all perceived colors by adding light of three different primary color types. RGB is an example color space based on this idea.
• For print materials, light is reflected rather than directly emitted. One can imagine starting with pure white light and subtracting different colors to make a given perceived color. This is the idea behind the CMYK (cyan, magenta, yellow, black) color space used in printing.

Light Constancy

• The ability of humans to see the same color for an object over a variety of lighting situations is called color constancy.
• Different processes allow humans to do this.
  • One is chromatic adaption in which the perceived color shifts when one is exposed to a source of that color for a long time.
  • Another is color expectations from the colors of surrounding objects.
  • Further our mind makes use of expectations about how objects are usually colored.
  The image on the right above illustrate these expectations.
• Our brains also maintain lightness constancy so that the overall brightness levels appear to be unchanged, even if lighting conditions are dramatically altered (left figure).
• One theory for how this works, is the ratio principle which says the brain only cares about the ratio of the reflected intensities between objects, not the absolute reflected intensities.

Display Issues

• Displays usually use RGB lights to generate the palette of colors and brightness.
• Usually these values can be specified as numbers between 0 and 255.
• The human eye on the other hand can perceive seven orders of magnitude of perceptible light intensity.
• This is sometimes mitigated by doing things like gamma correction to handle nonlinearities in the way the monitors output the different colors and the ways humans can distinguish colors.
• As we can see from our CIE color chart one last problem humans can perceive colors outside of what can be generated using RGB.

Combining Sources of Information

• The brain synthesizes a variety of inputs to interpret a scene.
• Based on a variety of cues it might generate several hypotheses for what it is seeing, the most likely of which might rise to the level of awareness.
• For some situations, several hypotheses might be interpreted by the brain as being roughly equally likely, giving rise to what is known as multistable perception.
• Optical illusions such as the above images (which face is the front face on the left? is the right image a duck or a bunny?) are examples of this.
• When visual and audio cues are combined, expectation can alter what is heard or seen in what is known as the McGurk Effect. Ventriloquist rely on this by using sounds similar to the desired sound to say but which don't involve lip movement. For example, using `d` for `b`. This combined with the expectation of what's going to be said and the movement of the vent's (aka the ventriloquist's dummy) lip, convince the mind a `b` sound was said even though a `d` was used.
• As a VR developer, one should be aware of the hypothesis generating aspect of perception, as things like real-world, VR world size, mismatches, vection, and so on, can cause incorrect interpreation of scenes and unintended or intended, multistable perception of scenes.