Introduction

On Monday, we began talking about sampling.
Recall that when we do the conversion from a continuous image space of windows coordinates to the discrete space of our screen, we can get various aliasing artifacts like jaggies, moire patterns, and flickering.
We described how to solve this problem by blending the values for a pixel location `I[i][j]` using a filter functions `F_(i,j)(x,y)` by computing some kind of integral
`I[i][j] leftarrow int int_(Omega) dx dy I(x,y) F_(i,j)(x,y).`
We gave an example where `F_(i,j)(x,y)` was a box filter restricting the region to average, `Omega`, to floating points samples within the pixel `Omega_(i,j)`.
Rather than actually compute the integral, we instead render the scene to a larger image first, then when we compute the final image, we take some fixed set of discrete samples from the larger image for each pixel and average them to get a value for the pixel:
`I[i][j] leftarrow 1/n sum_(k=1)^nI(x_k,y_k)`
If samples are chosen according to a grid pattern then we called this technique super sampling.
Another technique we described was multisampling where we render to a high res scene with z buffer and figure what fraction of each triangle covers a pixel and use this weighted sum to give the pixels final value.
Today, we start by trying to answer the question of why we don't typically get aliasing effects with digital cameras or the eye.

Anti-Aliasing in Cameras and the Eye

In digital cameras, anti-aliasing is achieved by the spatial integration over the extent of each pixel sensor, as well as by the optical blurring that happens due to the lens
Some cameras also include additional optical elements to blur continuous image data before it is sampled at the sensors.
Scenes with regular patterns such as tweed jackets, plaids, etc can still exhibit aliasing effects in digital camera photos.
The human eye doesn't typically see aliasing artifacts when it looks around at a scene.
Recall the central (foveal) region of vision is the most sensitive. Typically, your eyes rapidly bounces around a scene in what are called saccades changing what the foveal region is looking at.
Your mind then reconstructs a single image by stitching together these smaller high resolution images, using memory, and filling in gaps with lower quality, but wider field of view, peripheral vision data.
The book cites a paper by Williams (1985) which appears to show that optical blurring of light through the lens of the eye before it hits the fovea receptors in the back tends to prevent aliasing.
Also, the saccades introduce a randomness to the visual sampling process which tends to counteract aliasing.

Compositing Images

Suppose we want to draw an image of a weather man in front of a weather map where the video for these two things comes from separate sources but both screen sizes are the same.
If the weather man was in front of a blue/green screen we could imagine digitally cutting him out and then just drawing this portion of the image over the corresponding part of the map image.
This kind of compositing often leads to a jagged boundary in the weatherman/weather map transition area.
In a real photograph or an image rendered with a box filter, the pixel color on the man/map boundary would be set using an average of man and map colors.
With discrete image layers though, we have only one saved color value per pixel, and this can no longer calculate the correct composite pixel colors.

Alpha Blending

To solve this problem we associate to each pixel in each image layer a value `alpha[i][j]` that describes the overall opacity or coverage of the image layer at that pixel.
A value of 1 represents a fully opaque pixel and a value of 0 represents a transparent pixel.
A value in between these extreme represents a partially transparent pixel.
To composite two image layers, at each pixel we use these alpha values to determine how to combine the foregound and background colors.
Let `I(x,y)` be a continuous image, and let `C(x,y)` be a binary valued coverage function over the continuous domain. So a value of `1` is a point "occupied" by the image, a value of 0 indicates a point not occupied by the image.
Store into our discrete image and alpha array the values:
`I[i][j] leftarrow int int_(Omega_(i,j)) dx dy I(x,y) C(x,y)`
`alpha[i][j] leftarrow int int_(Omega_(i,j)) dx dy C(x,y).`
We imagine we calculate `I^(f)[i][j]` and `I^(b)[i][j]` for the foreground and background images. The fact that we multiply the coverage function before trying to do a combination of the images is called using premultiplied colors.
To compose `I^(f)[i][j]` over `I^(b)[i][j]`, we compute the composite image color `I^(c)[i][j]` as
`I^(c)[i][j] leftarrow I^(f)[i][j] + I^(b)[i][j](1 - alpha^(f)[i][j])`

Computing composite alpha and non-symmetry

To compute alpha for the resulting image so that we could then use it for further compositing, we calculate
`alpha^(c)[i][j] leftarrow alpha^(f)[i][j] + alpha^(b)[i][j](1 - alpha^(f)[i][j])`
Often the background layer is completely opaque for all pixels, so the composite image has the property as well.
Our two formulas, for blended intensity and for blended alpha, are called discrete binary over operations.
One can verify these operations are associative but not commutative:
`I^a \text(over) (I^b \text(over) I^c) = (I^a \text(over) I^b) \text(over) I^c,`
but in general,
`I^a \text(over) I^b ne I^b \text(over) I^a`.

Comparison with Continuous Composition

How does alpha blending compare with a correctly anti-aliased continuously composited image?
Let `C^f`, `C^b` be the coverage functions for continuous images `I^f`, `I^b` representing a foreground and background image.
We assume if `C^(f)(x,y) = 1` we observe the foreground color, and, if `C^(f)(x,y) = 0` we observe the background color.
The box-filtered anti-aliased discrete image at pixel `(i,j)` should be
`I^(c)[i][j] leftarrow int int_(Omega_(i,j)) dx dy [I^(f)(x,y) C^(f)(x,y) + I^(b)(x,y) C^(b)(x,y) - I^(b)(x,y) C^(b)(x,y)C^(f)(x,y)]`
`= int int_(Omega_(i,j)) dx dy I^(f)(x,y) C^(f)(x,y) + int int_(Omega_(i,j)) dx dy I^(b)(x,y) C^(b)(x,y)`
`- int int_(Omega_(i,j)) dx dy I^(b)(x,y) C^(b)(x,y)C^(f)(x,y)`
On the other hand, alpha blending would give
`I^(c)[i][j] leftarrow I^(f)[i][j] + I^(b)[i][j](1 - alpha^(f)[i][j])`
`= int int_(Omega_(i,j)) dx dy I^(f)(x,y) C^(f)(x,y)`
`+ (int int_(Omega_(i,j)) dx dy I^(b)(x,y) C^(b)(x,y))(1 - int int_(Omega_(i,j)) dx dy C^(f)(x,y))`
`= int int_(Omega_(i,j)) dx dy I^(f)(x,y) C^(f)(x,y) + int int_(Omega_(i,j)) dx dy I^(b)(x,y) C^(b)(x,y)`
` - int int_(Omega_(i,j)) dx dy I^(b)(x,y) C^(b)(x,y) cdot int int_(Omega_(i,j)) dx dy C^(f)(x,y)`.

Comparison continued.

The difference between the two results is
`int int_(Omega_(i,j)) I^(b)(x,y) C^(b)(x,y)C^(f)(x,y) - int int_(Omega_(i,j)) dx dy I^(b)(x,y) C^(b)(x,y) cdot int int_(Omega_(i,j)) dx dy C^(f)(x,y)`
The error is thus the difference between the integral of a product and a product of integrals.
We can think of this error as representing the amount of "correlation" between the distribution of foreground coverage in some pixel and and the distribution of the background data within that pixel.
If we assume that the foreground coverage is uniform and random then we can safely ignore this error.

Non-premultiplied Colors

In some uses of alpha, instead of storing `I` and `alpha`, we store `I/alpha` and `alpha`.
This is called "non-premultiplied" format.
This can improve data precision. When a pixel has a small coverage value, then `I` will be a small number and only use a a few of the low-order bits in a fixed point representation.
In the non-premultiplied form, one can use all of the bits in the stored file format to describe the color independent of the coverage.
In non-premultiplied form the "over" operation becomes more complicated
`alpha^(c)[i][j] leftarrow alpha^(f)[i][j] + alpha^(b)[i][j](1 - alpha^(f)[i][j])`
`I'^(c)[i][j] leftarrow 1/(alpha^(c)[i][j])(alpha^(f)[i][j]I'^(f)[i][j] + alpha^(b)[i][j]I'^(b)[i][j](1 - alpha^(f)[i][j])).`
When all the pixels in the background have alpha values of `1`, this becomes
`alpha^(c)[i][j] leftarrow 1`
`I'^(c)[i][j] leftarrow alpha^(f)[i][j]I'^(f)[i][j] +I'^(b)[i][j](1 - alpha^(f)[i][j])`
Premultiplied format although losing precision, does have the advantage that it is easier to scale the image. For example to scale an image down by a factor of two (as might be done in some texture mapping techniques), we can simply average the four appropriate `I^f` values. This is harder to do in the non-premultiplied format.

Uses of Alpha

In OpenGL, alpha is used not just for image processing but also as a tool modeling transparency and blending color values.
We can assign an alpha value as the fourth component of fragColor in a fragment shader and use this to combine fragColor with whatever is currently present in the framebuffer.
For example, alpha can be used as a way of modeling (monochromatic) transparency.
To enable blending in OpenGL, one can call glEnable(BLEND)
Then to set how colors are combined one calls glBlendFuncSeparate(GLenum srcRGB, GLenum dstRGB, GLenum srcAlpha, GLenum dstAlpha).
The book has an example of fur modeling using OpenGL alpha blending.
The bunny was drawn using a series of "shells", each textured with an image representing a slice of fur. The textures includes alpha values, so the spaces between the fur was semi-transparent.

Reconstruction

Next week we will start talking about the topic of reconstruction.
Given a discrete image `I[i][j]`, how do we reconstruct a continuous image `I(x,y)`.
This problem is important for the problem of resizing texture images.
For example, in our fragment shader, we might want to fetch a color from a texture coordinate that falls in between the texture's pixels.
Figuring out what texture color to use is called reconstruction.

Finish Sampling, Reconstruction

Outline

Introduction

Anti-Aliasing in Cameras and the Eye

Compositing Images

Alpha Blending

Computing composite alpha and non-symmetry

Comparison with Continuous Composition

Comparison continued.

Non-premultiplied Colors

Uses of Alpha

Reconstruction