Outline

• Viewing Transformations
• Quiz
• WebGL

Introduction 1

• Last week, we were discussing the computer graphics data structures and transformations needed to specify VR scenes.
• We discussed the half-edge data structure as a data structure useful to define 3D meshes.
• We then looked at different kinds of transformations to put objects in 3D scenes.
• In particular, we looked at translations, 2D and 3D rotations.
• We discussed 3D rotations described both via yaw, pitch, and roll and by unit quaternions.
• We also defined the homogeneous transformation matrix which can be used to specify both translations and rotations as a single 4x4 matrix.
• Although we didn't say, we can also do x, y, z scalings via the matrix: `((s_x, 0, 0, 0),(0,s_y, 0,0),(0,0,s_z,0), (0,0,0,1))`

Introduction 2 - Big Picture

• Models given by 3D meshes are often specified in some model coordinates which might be output in some format by a program like Blender or Maya.
• If a model is an assembly of smaller parts such as arms and legs, we might use rotations, scalings, and translations to specify how these parts local coordinates fit into the overall model coordinates.
• Doing rotations, scalings, and translations can then put whole models into world coordinates.
• To then draw a 2D image on a display, we next describe different kinds of projection transformation matrices.
• For VR headsets we typically have two displays, one for each eye, so we also need to say how to handle this.

Viewing Transformations

• Eventually, our VR scene is going to be drawn on a physical display.
• We can think of what appears on this display as the result of taking a picture of the scene using a virtual camera placed in the scene and pointing in some direction.
• In a VR setting, we typically have two such cameras, one for each eye, placed eye-distance apart.
• We would like to figure out the matrices needed to simulate these cameras?

Specifying an Eye what an eye is Looking At

• In a VR setting, the position and orientation of an eye are usually given by a tracking system and possibly controller inputs.
• To keep track of this, we need to say where the eye is in the VR scene and what it is looking at. I.e., the quantities:
  1. Position of the eye: `vec e`
  2. A unit vector pointing in the direction the eye is looking: `hat c` (we use `c` as supposed to be center of screen).
  3. A unit vector pointing in the up direction for the eye: `hat u`.
• Given an arbitrary point `vec p` in our scene we can compute `hat c` as:
  `hat c = (vec p - vec e)/(||vec p - vec e||)`
• A transformation to eye coordinates (as if the eye was in the z axis looking at the origin) will consist of a translation to the point `vec e` together with rotation to look in the direction `c`. i.e.,
  `T_{eye} = ((hat x_1, hat x_2, hat x_3,0),(hat y_1, hat y_2, hat y_3,0),(hat z_1, hat z_2, hat z_3,0), (0,0,0,1))((1,0,0,-e_1),(0,1,0, -e_2),(0,0,1, -e_3), (0,0,0,1))`
• Here each of `hat z`, `hat x`, `hat y` can be calculated using cross-product as:
  `hat z = - hat c`
  `hat x = hat u times hat z`
  `hat y = hat z times hat x`.

Orthographic Projection

• Let `(x,y,z)` denote any point after `T_{eye}` is applied.
• We could imagine directly projecting it onto a screen in the `xy` plane via the map: `(x,y,z) mapsto (x, y,0)`.
• Such a map is called an orthographic projection.
• If we imagine the `xy` plane as a virtual display of the models, then such a projection has several problems:
  1. A jumble of objects would appear superimposed, rather than hiding parts of a model that are in front of another.
  2. The display would extend infinitely in all directions (except z). Ideally, we would like to eliminate from the scene all but a rectangle corresponding to the actual screen size
  3. Objects closer should appear larger than those further away as in the real world.
• To fix these problems, rather than use an orthographic projection, we will instead use perspective projections.

Perspective Projection

• To understand perspective projection, for each point `(x,y,z)` consider the line through this point and the origin. I.e., the set of points:
  `(lambda x, lambda y, lambda z)`
  for some `lambda in RR`.
• If we place a virtual screen, which we imagine as a plane, in our world, there will be a choice of `lambda`, where the line through `(x,y,z)` intersects this plane.
• The collection of such point drawn own our virtual screen is the perspective projection.
• The figure above illustrates this for the three points making up a triangle and for the plane `z=-1`.
• Given a point `(x,y,z)` and a screen at `d` in the `z` direction parallel to the x-y plane. We have `d = lambda z`, and so `(x,y,z)` gets mapped to `(dx/z, dy/z, d)` via this transformation.

Quiz

Which of the following statements is true?

1. We said any 3D-translation of a point `vec{p}=(x,y,z)` can be specified using a `3 times 3`-matrix multiplied against `vec{p}^T`.
2. Rotations in 2D can be viewed as a multiplication with a complex number of unit length; any rotation in 3D can be specified using a unit quaternion `q` and the operation `vec v mapsto q vec v q^{-1}`.
3. It is possible to have a 3D rotation that does not fix an axis of rotation.

Chaining Transformations

• We can combine the transformations we have described so far to place an object in a scene as:
  `T= T_{vp}T_{can}T_{eye}T_{rb}`
• In the above, `T_{rb}` is the rigid body transformation (translation and rotation) used to put a point in world coordinates.
• `T_{eye}` is the transformation that maps the virtual world into the coordinates of the eye.
• So far we are pretending a user has only one eye midway between our two virtual eyes.
• `T_{can}` performs the perspective transformation as a unitless `4 times 4` matrix called the canonical view transform.
• `T_{vp}` is called the viewport transformation. It converts the points with x,y,z real values between -1 and 1 of our canonical transform into pixel units for a physical display.
• We now go into `T_{can}` and `T_{vp}` in more detail before saying how to handle left and right virtual eyes.

Canonical View Transform

• The above figure illustrates a viewing frustum (a pyramid-like figure with top chopped off) based on the four corners of a rectangular virtual screen.
• It is delimited by two planes perpendicular to the `z`-direction at `z=n` and `z=f` called the near plane and far plane respectively. Sometimes people tack in the word clipping. I.e., near clipping plane instead of near plane.
• Note `n,f < 0` as the `z`-axis points in the opposite direction.
• The virtual screen is centered in the near plane, so `d=n` in our perspective transformation from before.
• Objects within the viewing frustum we want to make sure appear on our screen. Objects further away than `f` or nearer than `n` we can imagine as being clipped out of the scene.

Getting a Canonical View Matrix

• We want to reproduce our perspective transformation as a `4 times 4` matrix.
• Consider the following matrix operation:
  `((n,0,0,0),(0,n,0,0),(0,0,n,0),(0,0,1,0))((x),(y),(z),(1)) = ((nx),(ny),(nz),(z))`
• In the first two coordinates, we get the numerator of our perspective transformation, but we don't get the nonlinear `1/z` factor .
• To handle this, we let the 4th component of our `4`-vectors be `z` rather than 1.
• We then always interpret a `4`-vector as `(v_1, v_2, v_3, v_4)` as representing the 3D point `(v_1/v_4, v_2/v_4, v_3/v_4)`
• Hence, the result of the above transformation would be interpreted as `((nx)/z, (ny)/z, n)` which is the desired perspective transformation where `d=n`.

Keeping Track of Depth

• Often the matrix of the previous slide is modified as follows:
  `T_p = ((n,0,0,0),(0,n,0,0),(0,0,n + f, -fn),(0,0,1,0))`
• This is the same as before, except in how the `z` coordinate is transformed.
• In terms of `z` on a screen we know `z=n`, so we co-opt the `z`-coordinate to keep of which objects are in front of which other objects.
• `T_p` calculates the `z` coordinate as `(n+f)z - fn`.
• Although this doesn't preserve exact distances, but it does preserve the ordering.
• If we were using the painter's algorithm to draw points on the screen, we would use this ordering to draw points from furthest to nearest.
• Values bigger than `f^2` or less than `n^2` we could discard.

Additional Translation and Scaling

• After `T_p` is applied, the eight corners of the frustum are transformed into the corners of a rectangular box as shown in the above image.
• We can normalize this box via a re-scaling and translation so that it is centered at the origin and the coordinates of its corners are `(\pm 1, \pm 1, \pm 1)` using the following matrix:
  `T_{st} = ((2/(r-l),0,0,-(r+l)/(r-l)),(0,2/(t-b),0,-(t+b)/(t-b)),(0,0,2/(n-f), -(n+f)/(n-f)),(0,0,0,1))`
• We define `T_{can}` using `T_p` and `t_{st}` as:
  `T_{can} = T_{st} T_p`.

Viewport Transform

• After `T_{can}` the `x` and `y` coordinates range between `-1` and `1`. We want to convert this into the pixel units of the physical display.
• Let `m` be the number of horizontal pixels and `n` the number of vertical pixels. For example, maybe `m=1920` and `n=1080`.
• Suppose these are indexed from `0` to `m-1` and `0` to `n-1` respectively and `(0,0)` is in the lower left corner.
• The transform we want will be the following combination of a scaling and a translation:
  `T_{vp} = ((m/2,0,0,(m-1)/2),(0,n/2,0,(n-1)/2),(0,0,1,0),(0,0,0,1))`.

Left and Right Eyes

• So far, our matrix chain does not handle the fact that humans have two eyes, a left and right one.
• To handle this we change our chain of matrices to the following two chains: `T= T_{vp}T_{can}T_{l\e\ft}T_{eye}T_{rb}`
  `T= T_{vp}T_{can}T_{right}T_{eye}T_{rb}`
  one for each eye display.
• Here `T_{l\e\ft}` and `T_{right}` are simply translation matrices by half the average an eye separation distance, `t=0.064m`, about the `x` axis:
  `T_{l\e\ft} = ((1,0,0,t/2),(0,1,0,0),(0,0,1,0),(0,0,0,1))`
  `T_{right} = ((1,0,0,-t/2),(0,1,0,0),(0,0,1,0),(0,0,0,1))`
  

WebGL and WebVR

• We are now going to look at how to apply some of geometric modeling concepts we have learned to draw 3D scenes on the web.
• We are going to make use of two Javascript APIs, WebGL and WebVR, which were being used under the hood when we were creating A-Frame projects.
• WebGL evolved out of Canvas3D experiments by Vladimir Vukicevic at Mozilla carried out in 2006.
• It is based on the embedded systems version of OpenGL.
• The standards for WebGL are maintained by the non-profit Khronos Group containing Intel, nVidia, AMD, etc.
• WebVR is another Javascript API which extends basic WebGL with additional features useful for VR headsets.
• It first appeared in 2014 and was initially developed by Vladimir Vukicevic.

WebGL Cube Demo

[Test Out Cube Demo]

• We are going to learn how to code WebGL before WebVR.
• Above is an image of our first example, which was modified from a collection of sample WebGL projects available from Mozilla.
• We'll go over the code for this example in detail next day.