Introduction

On Monday, we finished talking about perception of depth, motion, and color and the importance of perception to VR.
Today, we will start by talking about visual rendering.
Visual rendering specifies what the visual display should show through an interface to the virtual world generator.
Earlier in the semester, using various matrix transformations, we described where objects should appear in the visual display, we are now interested in how they should appear at a pixel level based on light sources, reflectance properties of materials, etc. that exist in the virtual world.
We are also interested in how to quickly render these using the graphics hardware available.

Drawing Triangles - Visibility

Let's assume the VR world is defined in terms of triangle primitives.
Using our chain of matrix multiplications, we can determine the location of every triangle on the virtual screen.
We next want to figure out which screen pixels are covered by what triangles, and then what color to draw these pixels based on the information about the materials of covering triangles.
If more than one triangle is covering a pixel, we need to be able to determine their relative order, translucency, etc., in order to determine color.

Object-Order versus Image-order Rendering

In object-order rendering to resolve which pixel is visible, we iterate over the list of triangles and attempt to render each to the screen. For each triangle (outer for loop) in the field of view, the pixels (inner for-loop) are updated only if the corresponding part of the triangle is closer to the eye than any triangles rendered to far. If we assume we are looking in the z-direction:
```
initialize z-buffer
for all triangles
   for all pixels covered by the triangle
       compute color and z
       if z is closer than what is already in the z-buffer
           update the color and z of the pixel
```
In image-order rendering, the loop orders are reversed: For each pixel in the field of view, we cycle over the triangles to see which ones should influence its value. We will consider this method next. As pseudo-code:
```
for all pixels on the screen
   for all objects seen in this pixel
     if this is the closest object seen at the pixel
         compute color and z
         set the color of the pixel
```

Ray-Tracing

In Ray Tracing, to calculate the RGB value of a pixel, a viewing ray is drawn from the focal point through the center of the pixel on a virtual screen. We then do the following two steps:
1. Ray Casting. This consists of determining both the viewing ray and its nearest point of intersection among all triangles in the virtual world.
2. Shading. This consists of calculating the RGB pixel value based on lighting conditions and material properties at the intersection point.

Ray-Casting

The above figure shows what we need to do in ray-casting: Find the points of intersection of the cast ray with triangles in the scene.
For a triangle ABC, normalizing △(`\vec{A}\vec{B}\vec{C}`) := `(\vec{B}-\vec{A}) times (\vec{C}- \vec{A})` gives us a normal `\vec{N}` to this triangle.
If the sign of this is positive, we have a forward facing triangle, which are the ones we are interested in.
From `\vec{N}`, we use `\vec{N} cdot (\vec{X} - \vec{A}) = 0` to give us the plane the triangle is in.
If our ray is `R`, we next compute △(`\vec{A}\vec{B}\vec{R}`), △(`\vec{A}\vec{R}\vec{C}`), △(`\vec{R}\vec{B}\vec{C}`) and if they all have the same postive sign, we know the ray is inside the triangle.
We can then substitute the line equation of the ray equal into the plane equation to find where in the triangle the line intersects.
We can then calculate the distance of this point to the eye.
We then repeat this task for the next triangle.
The above process, although not too complicated, could be slow if there are a lot of triangles.
Typically, a 3D tree structure, such as a BSP-tree (binary-space-partitioning tree) or Octree is used to store the triangles and a query is done on this tree using the ray to find a smaller number of triangles that could potentially intersect the ray.
One also might use a faster triangle-ray algorithm such as Moller-Trumbore which avoids needing to compute the plane.

Models of Shading

Assume now we have determined the triangle that should be drawn over a pixel.
We next need to figure out where it is being illuminated from. It could be illuminated directly from a light source, or illunimated via reflections from other triangles.
A shading model tries to determine the color that the pixel covered by a triangle should appear, based on the angle of the view ray hitting the triangle and the properties of the triangle material and light source.
In Lambertian shading, it is assumed that the angle of the viewing ray is independent of the resulting pixel R,G,B value.
This corresponds to the case of a diffuse reflection of the light off the material. I.e., when the material is rough, and so the light source gets reflected off the material in all directions. Mathematically, the equation of this model is:
`vec{L} = (R,G, B) = vec{d} vec{I} max(0, n cdot ℓ)`.
Here `vec{L}` is the final R,G,B value fo the pixel, `vec{d} =(d_R, d_G, d_B)` are reflectance coefficients of the material, and `vec{I} = (I_R, I_G, I_B)` are the intensities of the incoming light, `n` and `ℓ` are from the left figure above. The product `vec{d} vec{I}` should be evaluated as a pointwise product of vectors (Hadamard product).
A second shading model, Blinn-Phong shading can be used to handle reflections off a shiny surface as well (specular lighting). Its equation is:
`vec{L} = (R,G, B) = vec{d} vec{I} max(0, n cdot ℓ) + vec{s} vec{I} max(0, n cdot b)^x.`
Here `vec{s}` represents `(s_r, s_G, s_B)` shinyness coefficients, `b` is the angle bisector between `ℓ` and `vec v` (right figure above, `b= (ℓ +v)/(||ℓ +v||)`) , and `x` is a material propery also controlling the global shinyness of the material usually its value ranges from 100 (mildly shiny) to 10000 (act like mirror).
If a material were a perfect mirror, then all of the light from the source would be reflected to the pixel only if they are perfectly aligned; otherwise, no light would arrive. Perfect reflectance occurs if `v` and `ℓ` form the same angle with respect to `n`. Blinn Phong proposes adding a fudge factor for this exact match based on how far the normal is from the bisector.
Often we assume there are random lighting photons coming from everywhere in the scene, so-called ambient light. To add ambient shading to our model we add one more term to our equation so far:
`vec{L} = (R,G, B) = vec{d} vec{I} max(0, n cdot ℓ) + vec{s} vec{I} max(0, n cdot b)^x + L_{a}.`
One reason to add an ambient term in our model is to prevent surefaces in shadows of light sources from appearing black.

In-Class Exercise

Suppose a scene is globally lit according to an ambient light source of `(.1,.1,.1)`. A pixel's color value comes from a triangle which acts as red diffusely and green specularly. Suppose we have a light source with intensities `(.7,.5,.3)`, and that `n cdot ℓ = .6` and `n cdot b = .9`. Let `x=5000`.
What would be the final color of the pixel?
Please post your solution to the May 1 In-Class Exercise Thread.

Multiple Light Sources, Reflectance, Refractance

The discussion so far allows us to compute how a pixel might be lit based on one light source.
If there were multiple light sources, we simply sum over the lighting values they would produce individually.
One thing we haven't mentioned, is how do we know the light source wasn't occluded by another triangle, preventing it from directly shining on the triangle we are trying to compute the color with respect to?
To handle this we can compute triangle interections for a shadow rays sent from the point we are considering on the current triangle and to each light source.
Another calculation that can be done is mirror reflections (and similarly, refraction -- this latter to handle transparency).
Given an incoming vector `vec w`, one can calculate the bounce direction as: `B(vec w) = 2(vec w cdot vec n)vec n - vec w`
where `vec n` is the normal at the intersection point.
We can then see where the bounce ray hits the geometry of the scene. The color of the point hit by this ray is then calculated and used to determine the color of the original point on the mirror.
So the basic color of a pixel is then the color of the first object intersected together with a brightness determined by how well it is lit, together with the attenuated colors coming from reflections, refractions etc.
Reflections themselves might come from earlier reflected sources so the whole process is usually recursive to some cut off depth.

BRDFs

Bidirectional Reflectance Distribution Function

The shading model above is often what is used in practice because it is relatively simple, however, it has only a minimal basis in real world physics.
To account for shading in a more physical and precise way, it is common to use a bidirectional reflectance distribution function.
In the left figure above, `theta_i` and `theta_r` represent the angles of the light source and viewing ray respectively with respect to the surface.
If we look straight down on the surface, then `phi_i` and `phi_r` represent the angles made by the light and viewing vector.
A BRDF is a function of the form:
`f(phi_i, psi_i, phi_r, psi_r) = mbox(radiance)/mbox(irradiance)`.
Here radiance is the light energy reflected from the surface in directions `phi_r` and `psi_r` and irradiance is the light light arriving at the surface from directions `phi_i` and `psi_i`.
Our previous models can be expressed as a simple type of BRDF. For example, Lambertian shading has a constanct BRDF because the surface reflects equally in all directions.
However, BRDF and extensions of it have been used to account for much more complex and physcially correct lighting effect with a variety of surface textures.

VR and Shading

Because of the increased resolution and frame rate requirement of VR, one has a strong pressure to reduce rendering complexity.
Our simplified lighting model can also have issues when rendered stereoscopically.
For example, as we can see from the above figure specular reflections are slightly different when computed for each eye using our model.
Given that our model is only semi-physical to begin with, this can actually be a source of VR sickness.

Rasterization

If one is trying to draw a 1080p image at 90Hz, then we need to be able to do 180 million raycasts/second to render our scene.
Even with a fast ray-triangle intersection algorithm, and using BSP-trees, this can be hard, so often it is more efficient to switch from image order-rendering to object order rendering.
The objects in our case are triangles, and the process of drawing them is called rasterization, one of the main functions of modern GPUs.
Typically, we sort the triangles according to depth. This is not always possible as illustrated by the above figure, so we might combine sorting together with triangle subdivision to get all our triangles absolutely ordered by depth.
Using the painter's algorithm, we then draw the triangles that appear in the view from furthest away to closest.

Depth Buffer

Another way to manage the depth problem on a pixel-by-pixel basis to use a depth buffer or z-buffer.
This buffer stores a positive real number at every pixel location. Before any triangle is rendered, a maximum value is stored at every location (a value than the z-value of any triangle).
Each triangle is rendered by calculating a rectangular part of the image (a bounding box) that fully contains it. This can be done quickly fromt he triangle vertices.
For each pixel in this box, one determines which ones lie in and which ones lie outside the triangle.
For points within the triangle, we check with the depth buffer and record its depth value there if it is nearer than the current value. If it is nearer, we also draw it.

Visual Rendering

Outline

Introduction

Drawing Triangles - Visibility

Object-Order versus Image-order Rendering

Ray-Tracing

Ray-Casting

Models of Shading

In-Class Exercise

Multiple Light Sources, Reflectance, Refractance

BRDFs

VR and Shading

Rasterization

Depth Buffer