Outline

• Interpolating Values Across Triangles
• Aliasing
• Culling
• VR and Rasterization
• Quiz
• Correcting Optical Distortions
• Improving Latency

Introduction

• Last week we were talking about visual rendering.
• This was the process of determining how pixels should be drawn on the screen based on a VR scene after we have already applied our matrix transformations to determine which triangles are in front of the screen.
• We said there were two approaches to doing this: image-order rendering (cycle over screen pixels determine which triangles should affect the color of pixels, compute color) and object order rendering (cycle over triangles, determine pixels covered by triangle, update them as necessary).
• For the first approach, we discussed how ray-tracing works, discussed different lighting models, and the implications of this approach to VR.
• We then said because of the sheer number of ray-casts needed for the first approach, that often the second approach is used for VR.
• We went over the second approach, talking about how to draw triangles (rasterization), and the use of a z-buffer.
• We begin today by describing the process of computing color and other values across a triangle.

Barycentric Coordinates

• Properties of a triangle are often described only at the vertices. For example, we give the colors of a triangle at the vertices. For texture mapping we might say the three vertices of our triangle correspond to these three pixels from some texture image.
• We then use linear interpolation to compute the values interior to the triangle.
• To represent a triangle given by vertices `vec{p}_1, vec{p}_2, vec{p}_3` in a way suitable for interpolation we often use barycentric coordinates. That is, we write an interior point as a weighted average:
  `vec{p} = alpha_1vec{p}_1 + alpha_2vec{p}_2 + alpha_3vec{p}_3`,
  subject to `0 \leq alpha_1, alpha_2, alpha_3 \leq 1` and `alpha_1 + alpha_2 + alpha_3 = 1`.
• Given a point `vec{p}` and a triangle with vertices `vec{p}_1, vec{p}_2, vec{p}_3`. Let `vec{e_1} = vec{p}_2 - vec{p}_1`, `vec{e_2} = vec{p}_3 - vec{p}_1`, and `vec{e_3} = vec{p} - vec{p}_1`. Define `d_{ij} = \vec{e}_i cdot vec{e}_j` . We can find `vec{p}` in barycentric coordinates via the following equations. Let
  `s = 1/(d_{11}d_{22} - d_{12}d_{12})`
  then
  `alpha_1 = s(d_{22}d_{31} - d_{12}d_{32})`
  `alpha_2 = s(d_{11}d_{32} - d_{12}d_{31})`
  `alpha_3 = 1 - alpha_1 - alpha_2`.

Interpolating Values Across Triangles

• One useful property of barycentric coordinates is that if we start with a point apply matrices to map it to screen coordinates and then compute in barycentric coordinates for the triangle it belongs to, then this will be the same as if we started by converting the point to barycentric coordinates first, applying the matrices to the triangles vertices, and then using our barycentric coordinates to figure out where the interior point goes.
• So in rendering a triangle first compute the screen coordinates of its vertices `vec{p}_1, vec{p}_2, vec{p}_3`. If the colors of vectex `vec{p}_i` is `(R_i, B_i, G_i)`, then color of the interior point with barycentric coordinates `(alpha_1, alpha_2, alpha_3)` will be:
  `R = alpha_1 R_1 + alpha_2 R_2 + alpha_3 R_3`.
  `G = alpha_1 G_1 + alpha_2 G_2 + alpha_3 G_3`.
  `B = alpha_1 B_1 + alpha_2 B_2 + alpha_3 B_3`.
• For texture mapping, if the vertices `vec{p}_1, vec{p}_2, vec{p}_3` correspond to points `vec{q}_1, vec{q}_2, vec{q}_3` from some 2D image. Let `vec{q}_{i} = (q_{i1}, q_{i2})`. Then an interior point `alpha_1 vec{p}_1 + alpha_2 vec{p}_2 + alpha_3 vec{p}_3` will get pixel value from: `alpha_1 vec{q}_1 + alpha_2 vec{q}_2 + alpha_3 vec{q}_3` in the image.
• Another use of interpolation, is to interpolate the normal of a surface approximated by triangles across the interiors of a triangle. This is called normal mapping and is important for realistic shading.
• For more complicated materials, we can make an "image" of normals for what the normals should be for a patch on the surface. Then, just a like a texture map we can use barycentric coordinates to map these normals across the interior of our triangle. This is called bump mapping.

Aliasing

• We said before that aliasing artifacts occur when a continuous scene is discretized to pixels on a screen.
• This can lead to lines becoming staircases, etc.
• If we look at the triangle above, you can imagine turning on pixels when drawing it based on whether the coordinate of the center of the pixel is interior to the triangle or not.
• Notice in the two pixel image above, a whole pixel might not be interior to the triangle, causing an artifact.
• One way to get better results is to do supersampling: Sample many points within the pixel and average their color to determine the output color for the pixel.
• When one is doing texture mapping, one can get spatial aliasing problems. These are caused when the viewing transformation greatly reduces the size and aspect ratio of the original texture when it is mapped onto the screen. This might leave insufficient resolution to represent a repeating pattern in the texture (See left checkerboard).
• One way to fix this issue is to use mutliple textures of different resolutions called a mipmap.

Culling

• Many triangles can be quickly eliminated from our list of triangles to cycle through in object-order rendering in a process called culling. This, of course, can improve rendering time.
• Culling is dependent on the data structures used to store triangles.
• Often a hierarchal strcture (BSP-tree) is used which can allow us to eliminate thousands of triangles from consideration for rendering with a single test.
• Different kinds of culling eliminate different triangles from render consideration.
• View volume culling eliminates all triangles that are wholly outside of the viewing frustum.
• Unlike traditional computer graphics, for VR, the frustum typically has a curved cross section for each eye lens, rather than a rectangular shape.
• If we replace the frustum with a truncated cone, a simple geometric test can quickly eliminate all objects oustide of the view: Given a point `(x,y, z)` and a viewing angle `theta`, we eliminates points with
  `sqrt(x^2 + y^2)/(-z) > tan theta`.
• Given that GPUs are optimized for rectangle views, often a stencil buffer is used to mark pixels within a rectangle which are out of view.
• Backface culling is another form of culling which removes triangles that have outward surface normals that point away from the focal point. These should not be rendered from behind if the model is consistently formed.
• Occlusion culling eliminates parts a model that might be hidden from view by a closer object. This tends to be a more complicated kind of culling to implement.

VR and Rasterization

• Aliasing effects, like the staircase problem, are made worse if one is moving one's head around looking at different parts of a scene: the staircase quickly begins to look like an escalator.
• Worse, if one has left and right viewpoints, then often the staricases or escalators of the two viewpoints don't match.
• This mismatch can cause problems when the mind tries to fuse the two images into one stereoscopic understanding of the scene.
• Anti-aliasing techniques and reducing contrast at edges can help. So too can using higher resolution scenes.
• Another issue is that many traditional computer graphics techniques rely on the fact that we are not looking at a scene stereoscopically.
• For example, stereoscopically a texture map is more likely to look like a painting than non-stereoscopically.
• Similar issues arise for normal and bump mapping.

Quiz

Which of the following statements is true?

1. A Reichardt Detector is a brain circuit which responds to directional motion.
2. Judder is an effect that occurs when a display's pixel-change-speeds are too fast.
3. Lambertian shading describes how ambient lighting should look in a scene.

Correcting Optical Distortions -- Hardware

• Recall pincushion and barrel distortions are common for an optical systems with a high field of view.
• If we see a VR screen through such a lens without any correction (usually for VR we want to do a pincushion correction), the virtual world appears incorrectly warped.
• If one yaws one's head back and forth, then fixed lines in the world, such as walls, appear to dynamically change their curvature because the distortion in the periphery is much stronger than in the center.
• This can actually interfere with the perception of stationarity.
• It can also add to VR sickness because objects will incorrectly appear to accelerate.
• Significant research is being done to try to solve this problem. For example, digital light processing technology avoids the lens issue by directly projecting the light into the eye without lenses.
• Another way to reduce the problem is to use a Fresnel lens (see above). Fresnel lenses are used in both the Oculus Go and HTC Vive. A Fresnel lens more accurately controls the bending of light rays by using a corrugated or sawtooth surface over a larger area.
• One drawback of a Fresnel lens is there are discontinuities in how it passes light at the sawtooths, which can cause glaring effects.

Correcting Optical Distortions -- Software

• Another way to handle optical distortion is to compute an inverse of it in software.
• Both pincushion and and barrel distortion are radially symmetric. That is, if we convert the screen to polar coordinates `(r, theta)` rather than `(x,y)` then the distortion only depends on `r` not `theta`.
• Both distortions can be approximated using a polynomial in `r`. For example, if `r_d` is the distorted `r` value and `r_u` the undistorted value,we might approximate the distortion as
  `r_d = f(r_u) = r_u + c_1r_u^3 + c_2r_u^5`
  for constants `c_1` and `c_2`. If `c_1 < 0`, one gets a barrel distortion, for `c_1 >0` a pincushion distortion.
• The process of computing `c_i` can be done via curve fitting done to images before and after the desired lens to determine a best fit.
• Then an inverse of `f` can be calculated offline using an approximation:
  `f^{-1}(r_d) approx (c_1r_d^2 + c_2r_d^4 + c_1^2r_d^4 + c_2^2r_d^8 + 2c_1c_2r_d^6) /(1 + c_1r_d^2 + c_2r_d^4)`
• Finally, the values for the pre-computed index can be stored in an array for fast access.
• This process is done once/headset design.
• Sometimes `f^{-1}` is worked directly into the perspective transformation as a post-processing step in the GPU, known as distortion shading.

Latency, Examples

• Motion-to-photons latency in a VR headset is the amount of time it takes to update the display in response to a change in head orientation and position.
• For example, if the user is fixating on a stationary feature in the VR world, as the head yaws to the right, the image of the feature must immediately shift to the left, or it won't appear stationary.
• To be more concrete, let `d` be the number of pixels/degree of the screen. Let `omega` be the angular velocity of the head in degress/second. Let `ℓ` be the latency in seconds. In this setup, `d omega ℓ` represents the incorrect shift perceived in the image.
• If `d=40` pixels, `omega=50` degrees/second, and `ℓ = 0.02` seconds then the incorrect displacement is 4 pixels.
• A very fast head turn might be 300 degrees per second, which would result in a 24 pixel error.

A Magic System Without Latency Issues

Let's consider a perfect VR system which has no latency issues...

• In a perfect VR system, as the head moves, the viewpoint must change for visual rendering.
• Such a system knows perfectly the head position and orientation at any time.
• The VWG continuously maintains the position and orientation of all objects in the world, and the visual rendering system maintains all perspective and viewpoint transformations, and the entire rasterization process continuously sets the RGB values on the display according to the shading model.
• The display continuously updates, taking no time to switch pixels.
• The display has retina-level resolution and a dynamic range of light output over seven orders of magnitude to match human perception.
• Visual stimulation in the VR world matches what would occur in a similar physical world in terms of geometry.
• There are no errors in time and space.

Historical Problems

• In practice, such a perfect system is not realizable because each of these operations require time to propagate information and perform computations.
• In early VR systems, the total motion-to-photons latency was over 100ms.
• By the 1990s, 60ms was considered an acceptable amount.
• As latency is perhaps the greatest cause of VR sickness, it is also one of the main obstructions to widespread adoption of VR.
• To some degree people can adapt to a fixed latency (for example, always 60ms), but then when they remove the headset that can suffer nausea.
• Variable latencies though can exacerbate VR sickness, because humans can't adapt to them.
• Nowadays, with the advent of GPUs and new display technology, latency is less of a problem with VR systems.
• Today, many systems have a latency of around 15-25ms, and this is further compensated for using predictive methods in the tracking system, giving an almost effectively zero latency.
• For this reason, other factors such as vection and optical aberrations are now the most common causes of VR sickness.
• Let's consider how this reduction in latency was achieved...

Overview of Latency Reduction Methods

• The following strategies are used to both reduce latency and to minimize the side effect of any remaining latency:
  1. Lower the complexity of the virtual world.
  2. Improve the rendering pipeline performance.
  3. Remove delays along the path from rendered image to switching pixels.
  4. Use prediction to estimate furture viewpoints and world states.
  5. Shift or distort the rendered image to compensate for last-moment viewpoint errors and missing frames.
• We will consider each of these in turn next day.