Outline

• FFMPEG
• Quiz
• Human Vision

Introduction

• Before the break, we finished up talking about light and optics.
• We had given a simple optical model of a VR headset.
• We then talked about the Javascript Gamepad API, how to determine if controllers were connected, and if so, how to determine their orientation, state of their triggers, and trackpad.
• We went over how to compile Native Oculus Go apps which extends the Android Activity class and make use of the C++ API's: VrApi.h for the visual aspects of a VR native app and VrApi_Input.h for the controller aspects.
• Today, we begin by talking about how to handle resource management in VR apps and also how to manipulate video resources using FFMPEG so that they can be used by VR apps.
• We then look at human vision and visual perception as it pertains to VR.

Singleton Pattern and Resource Management

• When writing an application with lots of multimedia resources such as a game or a VR app, it is useful to keep track of which resources we have already read into memory and which ones we might still need to read in.
• We also might want to do some simple memory management.
• So that all parts of our app have a standard place they can go to get resources, we probably want to use the Singleton design pattern, which forces all instance of the class we are using as the resource manager to be the same.
• In Java this might look like:

  public final class ResourceManager {
  
      private static final int NUM_IMAGES = 100;
      private Image[] image = new Image[NUM_IMAGES];
  
      private static final ResourceManager RMANAGER = new ResourceManager();
  
      private ResourceManager() {} // private so can't call externally
  
      public static ResourceManager getInstance() {
          return RMANAGER;
      }
  
      public Image getImageByName(String name) {
          // check if already loaded image in image array
          // if yes, return image, if no, load and return
      }
  }
  

• Notice the only external-to-the-class way of making ResourceManager's is via getInstance(), and these all return the same instance.
• We can get and use instances of the resource manager with code like:

  ResourceManager rm = ResourceManager.getInstance();
  Image im = rm.getImageByName("bob.png");
  

• If you are running code in multiple threads, you can sup up the above idea using synchronized blocks/methods.

FFMPEG

• For VR applications, one particularly useful class of resources are 3D images and video, so I'd like to briefly mention one tool for manipulating 3D video: FFMPEG.
• Moving Picture Experts Group (MPEG) is responsible for several of the most common video formats: MPEG-1 (used in VCD), MPEG-2 (used in DVDs), and MP4 (used in Digital Video Broadcasting, and on the internet generally).
• The name FFMPEG come from a play on this: Fast Forward MPEG.
• FFMPEG was created in 2000 by Fabrice Bellard (aka Gerard Lantau) and is a command line tool for manipulating video.
• It can be installed from the command-line by using brew, apt, choco depending on the OS you are using. For example, on my Mac, I did:

  brew install ffmpeg
  

Common Use Cases of FFMPEG 1

• Below we list some common uses of FFMPEG. One point of FFMPEG is that since it is a command line tool, you can easily script it, allowing you to manipulate many files in the same way. These files might be in some folder, etc.
• Convert video to MP4. Unlike images, where nowadays pretty much every browser supports all the popular image formats (JPG, PNG, BMP, GIF, (maybe not HEIC)), there is still a fair bit of variability in video support. This is often because of patent issues with the underlying technology. With the publication of OpenH264 by Cisco in 2013, all the major browsers can play MP4 video containing the H264 codec, so it is useful to be able to convert to this codec. In FFMPEG one could do this with:

  ffmpeg -i movie.avi some_conversion_parameters movie.mp4
  

  some_conversion_parameters can be blank or you can set some parameter on the conversion. Example parameters might be:

  -vcodec h264 -acodec aac  -b:a 192k -crf 28 -pix_fmt yuv420p 
     -strict -2 -preset veryfast
  

• Extract an image from a video. This can be done with:

  ffmpeg -i movie.mp4 my_image.png
  

  We can use the -ss flag to specify some number of seconds into the video.

Common Use Cases of FFMPEG 2

• Extract a portion of a video. You can do this again with the -ss tag:

  ffmpeg -ss 00:00:45 -i orginalfile.mp4 -t 00:00:10 -vcodec copy -acodec copy newfile.mp4
  

  The above extracts 10 seconds from the original file starting at the 45 second mark.
• Concatenate videos. To do this, we can make a file, say list.txt, with a list of videos to be concatenated:

  /some/path/file1.mp4
  /some/path/file2.mp4
  ...
  

  Then we can run the command:

  ffmpeg -f concat -safe 0 -i list.txt -c copy output.mp4
  

• Resize a video:

  ffmpeg -i movie.mp4  -vf scale=some_width:some_height -q:v 1 \
    out_movie.png
  

  For example, -vf scale=4096:-1 would preserve the aspect ratio and make the width be 4096. Here -q:v 1 says preserve quality.
• Flip a video. Sometimes you can use video flipping to fake a change of point of view. I.e., flipping vertically, you get a little bit of a role reversal effect.

  ffmpeg -i movie.mp4 -vf vflip -c:a copy out_movie.mp4
  

  Here -c:a copy says to just copy the audio.

Video Formats for VR

• Video and or photos in VR is often projected onto the sky or some other surface using a projection mapping that we will discuss in a later lecture.
• It is common to use 180 degree 360 degree paroramas, which may or may not be in 3D.
• MPEG4-MVC (multi-view coding), used in Blu-ray, is a backward compatible stream format which allows one to encode 3D in the same video, so that non 3D players see the video as normal 2D.
• It, however, is not particularly popular as a format for creating 3D video resources as it won't work with phones being used in headsets.
• Instead, two images either side-by-side for 180 3D or top-bottom for 360 3D are used.
• If we are taking video using a pair of cameras about eye-width apart you often want to combine them into a single, side-by-side video.
• You can do this using ffmpeg's -filter-complex field as illustrated by the following php script I wrote.

Making SBS Video From Two Video Sources

<?php
$flip = false;
$front = true;
$back = false;
$lenscorrection = false;
$letterbox = false;
foreach ($argv as $index => $arg) {
    if (in_array($arg, ['--flip', '--front', '--back', '--lenscorrection',
        '--letterbox'])) {
        $cmd_switch = ltrim($arg, "-");
        $$cmd_switch = true;
        unset($argv[$index]);
    }
}
$argv = array_values($argv);
if (count($argv) < 4) {
    echo <<< EOD
    To use:
This program should be run from the command line with a line like:
php vr180tool.php [option list]  left_eye_file right_eye_file output_file

Here [option list] can be any sequence of the following options
 --flip //flip images upside down
 --front // make 180 3d from just front camera results
 --back // make 180 3d from just back camera results
 --lenscorrection // apply fisheye len correction
 --letterbox // apply letterbox to top and bottom of video

Here left_eye_file, right_eye_file, output_file should be a file that ffmpeg
can manipulate. For the Insta 360 One camera, left_eye_file and right_eye_file
would typically be .insv files for the two different perspectives needed to
make a stereoscopic 3d movie.

EOD;
    exit();
}

$lens_correction = "lenscorrection=cx=0.5:cy=0.5:k1=-0.34:k2=-0.045";
$left_correction = ($lenscorrection) ? "[lensleft];[lensleft]$lens_correction" :
    "";
$right_correction = ($lenscorrection) ?
    "[lensright];[lensright]$lens_correction" : "";
$flip_cmd = ($flip) ? "[to_flip];[to_flip]vflip" : "";
$box = ($letterbox) ? '[stacked];' .
    '[stacked]scale=(iw*sar)*min(2560/(iw*sar)\,1440/ih):'.
    'ih*min(2560/(iw*sar)\,1440/ih), pad=2560:1440:'.
    '(2560-iw*min(2560/iw\,1440/ih))/2:(1440-ih*min(2560/iw\,1440/ih))/2' :
    "";
list(, $left_mov, $right_mov, $output) = $argv;
$is_dir = false;
$is_file = false;
foreach([$left_mov, $right_mov] as $mov) {
    if (!file_exists($mov)) {
        echo "File: $mov does not exist!\n";
        exit();
    }
    if (is_dir($mov) || is_link($mov)) {
        $is_dir = true;
    }
    if (is_file($mov)) {
        $is_file = true;
    }
    if ($is_dir && $is_file) {
        echo "Inputs must be both files or both folders\n";
        exit();
    }
}
if ($is_file) {
    $movs = [0 => [$left_mov, $right_mov, $output]];
} else {
    if (!file_exists($output)) {
        mkdir($output);
    }
    if (is_file($output)) {
        echo "If inputs are folders, output must be folder\n";
        exit();
    }
    $pre_left_movs = glob("$left_mov/*.{insv,INSV,mp4,MP4,m4v,M4V}",
        GLOB_BRACE);
    $movs = [];
    foreach ($pre_left_movs as $mov) {
        $pre_right_mov = str_replace($left_mov, $right_mov, $mov);
        if (file_exists($pre_right_mov)) {
            $path_info = pathinfo($mov);
            $movs[] = [$mov, $pre_right_mov, "$output/" .
                $path_info['filename'] . ".mp4"];
        }
    }
}
echo "Making 180 degree 3D Output \n";
foreach ($movs as $mov) {
    list($left_mov, $right_mov, $output) = $mov;
    echo "Making movie from $left_mov and $right_mov with command:\n";
    if ($back) {
        $ffmpeg =
            "ffmpeg -i \"$right_mov\" -i \"$left_mov\" -filter_complex ".
            "\"[0]crop=1920:1920:0:0{$left_correction}[cropl];" .
            "[1]crop=1920:1920:0:0{$right_correction}[cropr];" .
            "[cropl][cropr]hstack$flip_cmd$box\" \"$output\"";
    } else {
        $ffmpeg =
            "ffmpeg -i \"$right_mov\" -i \"$left_mov\" -filter_complex ".
            "\"[0]crop=1920:1920:1920:0{$left_correction}[cropl];" .
            "[1]crop=1920:1920:1920:0{$right_correction}[cropr];" .
            "[cropl][cropr]hstack$flip_cmd$box\" \"$output\"";
    }
    echo $ffmpeg . "\n";
    exec($ffmpeg);
}

Remarks on FFMPEG Script

• The script on the previous slide first determines the command line arguments and then sets up some PHP variables it will use via string-intepolation to make an FFMPEG command line string.
• We see one such FFPEG command string in the line:

  $ffmpeg =
      "ffmpeg -i \"$right_mov\" -i \"$left_mov\" -filter_complex ".
      "\"[0]crop=1920:1920:0:0{$left_correction}[cropl];" .
      "[1]crop=1920:1920:0:0{$right_correction}[cropr];" .
      "[cropl][cropr]hstack$flip_cmd$box\" \"$output\"";
  

• This string is then exec'd:

  exec($ffmpeg);
  

• There are two interesting things about this string: First, it has two input -i, one for the left and right video for the stereoscopic output. The second is the -filter-complex command.
• The filter-complex command consists of a sequence of sub-commands of the form [in_label_1]...[in_label_n]ffmpeg_operation[out_label].
• Labels of the form [n] correspond to the `n`th input so [0] is the right video [1] is for the left video above. So [0]crop=1920:1920:0:0[cropl] would crop the portion of this input to a 1920x1920 square beginning in the (0,0) coordinate and call the output channel for later steps [cropl].
• A typical, raw 360 camera image contains the output of two fish eye lenses next to each other (front and back):
  
• Our script here is designed for handling two 360 camera video sources by cropping out the front camera images from each camera and producing an output side-by-side 180 degree 3D video.
• To make a crude 360 video of parorama from one camera, we might crop out the front and back videos from a single 360 input video, apply lens corrections to them, and horizontally stack them back together. The filter complex command might look like:

  [0]crop=1920:1920:0:0[lens_left];[lens_left]lenscorrection=cx=0.5:cy=0.5:k1=-0.34:k2=-0.045[out_left];
  [0]crop=1920:1920:1920:0[lens_right];[lens_right]lenscorrection=cx=0.5:cy=0.5:k1=-0.34:k2=-0.045[out_right];
  [out_left][out_right]hstack
  

• To make a 360 3D video we can use vstack to stack two such 360 videos.

Quiz

Which of the following statements is true?

1. Nicephore Niepce took photos in the 1820s using a Charged Couple Device.
2. Javascript Gamepad object cannot be used to find the state of the Oculus Go trackpad.
3. app->GetSystemProperty can be used to figure out which is the dominant hand for an Oculus Go controller in a Native app.

The Physiology of Human Vision

• Often it seems that we see the world with a large field of view, with perfect clarity, speed, accuracy, and resolution.
• In reality, our mind uses neural structures to fill in and help us understand the world consistently with our visual expectations.
• As VR often co-opts these processes, it is useful to understand how the mind does what it does, in order to improve our VR systems.

Parts of the Eye

• Before the break, we had looked at the eye as an optical device.
• We now look at the parts of the eye that actually receive light and send signals to the brain.
• This is done in the back of the eye in the retina by cells called photoreceptors. You can think of them a little as input visual pixels to the mind.
• The most sensitive part of the retina is called the fovea and is responsible for the portion of the world we see with the greatest visual acuity.
• In the center of the retina there is a small hole called the optic disk through which neural impulses are sent from the eye to the brain along the optic nerve.

Photoreceptors and Wavelength

• The retina has two kinds of photoreceptors: (1) rods which can be triggered by very low levels of light, and (2) cones which require more light to be triggered and which are responsible for color vision.
• The above left figure shows the wavelengths of light each is sensitive to.
• The eye is sensitive to photons hitting from 1 photon/100 seconds to 100,000 photons/second. In the former setting, only rods will be triggered. Low light vision with just the rods is called scotopic vision. This is what we have at night.
• The smallest cone cells are about 1000nm across. There are about 120 million rods in the eye and 6 million cones.
• The above right figure shows the angular density of rods and cones in the back of the eye.
• The book gives an experiment you can conduct to see that you have a blindspot in your retina.

From Photoreceptors to the Visual Cortex

• Photoreceptors transduce light-energy into an electrical signal called a neural impulse.
• Such signals are propagated in a hierarchal manner to the visual cortex (see figure above).
• As the levels traversed by signals increases in the cortex, the numbers of neurons influenced by a single initiating photoreceptor increases (red in above).
• Similarly, a single neuron at a deeper layer receives inputs from many photoreceptors.
• Neurons at lower levels might make comparisons of signals from neighboring neurons; whereas, deeper layers might respond to input over a larger field of view.
• Eventually, the signals reach the highest levels of the brain where they are fused with past experiences, etc.
• It takes significant time for the brain to do the processing of say a face such that it reaches consciousness (on the order of 100ms).
• Next day, we go into more detail about what happens at various stages doing this kind of processing.