Outline

• Multimedia HTML 5
• AndroidPlaying Audio and Video
• Using the Camera

Introduction

• On Monday, we were talking about Touch Events on iPhone and Android
• We start today by going over how to do some simple 2D graphics in HTML 5
• We then talk about handling touch in HTML 5
• We then look at the video and audio tags
• The remainder of the lecture will be about multimedia in Android

Two-Dimensional Graphics in HTML

• Since Netscape 2, which introduced Javascript in 1996, there have been different techniques to get crude 2D graphics in browsers -- the very earliest techniques were very hackish involving drawing single pixel images over and over again.
• Starting with IE 5 in 1998, an XML, tag-based language approach to 2D graphics called VML became available to web designers.
• About a year later another tag-based, this time non-Microsoft language, called SVG appeared, and this eventually became supported by all major browsers.
• Tag-based (vector/scene-based) approaches to 2D-graphics can often be inconvenient for scripting, writing games, etc as they are a little higher level and so slower.
• In 2004, Apple introduced the canvas tag in its Safari browser to support a more low-level, raster, programmatic approach to graphics. This is now supported by all major browsers, is a WHATWG standard, and it is this tag that we will briefly introduce.

Canvas Tag

• A basic canvas tag usage might look like:
  

  <canvas id="drawpad" />
  <style type="text/css">
  #drawpad {
      width: 200px;
      height: 100px;
      background-color: light-gray;
  }
  </style> 
  

• We can then refer to this object in Javascript, using document.getElementById("drawpad") or in jQuery using $("#drawpad"), use it to get a context and then start drawing:

  <script type="text/javascript"*gt;
     drawpad_object = document.getElementById("drawpad");
     drawpad_context = drawpad_object.getContext("2d");
     drawpad_context.strokeStyle = "blue";
     drawpad_context.strokeRect(10,10,50,50);
     drawpad_context.fillStyle = "green";
     drawpad_context.font = "32pt Arial";
     max_width = 100;
     drawpad_context.rotate(3.14/3); //pi/3
     drawpad_context.fillText("hello", 80, -10, max_width);
   
  </script>
  

• In the above, besides 2d we could also have used webgl if we wanted a more OpenGL-like, 3D-type canvas.
• Notice coordinates for the fillText are relative to the rotated canvas.
• Context also supports methods for clearing areas clearRect, drawing images using drawImage. To specify more complex shapes, you call drawpad_context.beginPath(), give your path instructions, then drawpad_context.closePath(), and finally call fill(path) or stroke(path). Here path instructions can be moveTo, lineTo, arc, quadraticCurveTo, bezierCurveTo, etc.
• To do double buffering with canvas, make a page with two canvas tags. Style them so they are the same size, but have one with visibility hidden. Draw your graphics into this one, then toggle which one is hidden.

Touch Events HTML5

• In HTML 5 there are four main touch events which can be applied to any tag: touchStart, touchMove, touchEnd, touchCancel
• The meanings are similar to the corresponding iOS events we've already discussed
• As an example of setting up such events, consider

  <canvas id="drawpad"
  
    ontouchstart="startHandler(event);"
  
    ontouchmove="moveHandler(event);"
  
    ontouchend="endHandler(event);"
  
    ontouchcancel="cancelHandler(event);"
  
  />
  

• You can also add events programmatically with:

  drawpad_object.addEventListener("touchstart", startHandler2, false);
  

• These events are easier to test on a mobile device or using the iPhone Simulator / Android Emulator.

Video and Audio Tags

• A basic video tag in HTML 5 looks like:

  <video id="myvideo" width="320" height="240" controls="controls">
    <source src="some_movie.mp4" type="video/mp4" />
    <!-- firefox does not support mp4-->
    <source src="some_movie.ogg" type="video/ogg" />
    <!-- firefox does support ogg but others don't-->
    Your browser does not support the video tag.
  </video>
  

• Currently, WebM (Google) (mimetype video/webm,) and Ogg seem like the most cross-browser supported formats.
• The HTML 5 audio tag is almost identical to the above except the source tags should be audio files.
• If we did myvideo_object = document.getElementById("myvideo"), the object would support load(), play(), pause() methods, and have a currentSrc, currentTime, duration and playbackRate properties. The tag also can supports loadedmetadata, timeupdate, pause, play, and ended events.
• Each browser maker also supports several extensions to these events.
• The following slides http://blog.gingertech.net/wp-content/uploads/2011/01/LCA_MM_AVProc2011/ seem like a cool demo of what you can do with the audio tag.

Multimedia on Android

• The foundation of Android's multimedia platform is based on OpenCore from PacketVideo.
• We will begin looking at these libraries by looking at how to play an MP3 file.
• The steps to write such a program involve:
  1. Put the mp3 file in the res/raw directory of the project. You will have to create this folder.
  2. Create a new instance of MediaPlayer and reference your MP3 by calling MediaPlayer.create().
  3. Call the MediaPlayer method start().
• We will make a simple app with a single button which will cause an mp3 file with the radio broadcast of Orson Welles' War of the Worlds to play. (Public domain from the 1930s).

main.xml

• Here is the main.xml for our project (this is where we define the button):

  <?xml version="1.0" encoding="utf-8"?>
  <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
      android:orientation="vertical"
      android:layout_width="fill_parent"
      android:layout_height="fill_parent"
      >
  <TextView  
      android:layout_width="fill_parent" 
      android:layout_height="wrap_content" 
      android:text="Test Media Player"
      />
   <Button
      android:id="@+id/playwarworlds" 
      android:layout_width="fill_parent" 
      android:layout_height="wrap_content" 
      android:text="Play Orson Welles"
      />
  </LinearLayout>
  

AudioTestActivity.java

• Here is the source code of our activity:

  import android.view.View;
  import android.view.View.OnClickListener;
  import android.media.MediaPlayer;
  import android.media.MediaPlayer.*;
  
  public class AudioTestActivity extends Activity {
  
      public void onCreate(Bundle savedInstanceState) {
          super.onCreate(savedInstanceState);
          setContentView(R.layout.main);
          View button = findViewById(R.id.playwarworlds);
          button.setOnClickListener( new OnClickListener() {
              public void onClick(View v) 
              {
                  MediaPlayer player = MediaPlayer.create(AudioTestActivity.this, R.raw.warworlds);
                  player.start();
                  player.setOnCompletionListener(new OnCompletionListener() 
                  {
                      public void onCompletion(MediaPlayer m) 
                      {
                      }
                  });
              }
          });
      }
  }
  

• A MediaPlayer enters a prepare state before playing which gets the media ready to be played (might be remote).
• MediaPlayer also has pause(), resume() and stop() (returns to before prepare was called) methods which could be hooked up to buttons

Playing Video

• We next test out playing video in Android.
• To do this we will create a small app which will play the first copyrighted film ever (Fred Ott's sneeze).
• We will put a file fred_ott.mp4 on the sdcard of the emulator using the Emulator's File Explorer.
• Our app will then play this file.

main.xml

• Here is our main.xml file:

  <?xml version="1.0" encoding="utf-8"?>
  <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
      android:orientation="vertical"
      android:layout_width="fill_parent"
      android:layout_height="fill_parent"
      >
  <VideoView 
      android:id="@+id/video" 
      android:layout_width="320px" 
      android:layout_height="240px" 
  />
  <Button
      android:id="@+id/play"
      android:text="Play Fred Ott" 
      android:layout_width="fill_parent" 
      android:layout_height="wrap_content" 
      />
  </LinearLayout>
  

VideoTestActivity.java

• The activity is reasonably straightforward:

  package org.pollett;
  
  import android.app.Activity;
  import android.graphics.PixelFormat;
  import android.os.Bundle;
  import android.view.View;
  import android.widget.MediaController;
  import android.widget.VideoView;
  
  public class VideoTestActivity extends Activity 
  {
      @Override
      public void onCreate(Bundle savedInstanceState) 
      {
          super.onCreate(savedInstanceState);
          getWindow().setFormat(PixelFormat.TRANSLUCENT);
          setContentView(R.layout.main);
          video = (VideoView)findViewById(R.id.video);
          video.setVideoPath("/mnt/sdcard/fred_ott.mp4");
          mc = new MediaController(this);
          mc.setMediaPlayer(video);
          video.setMediaController(mc);
          video.requestFocus();
  
          View button = findViewById(R.id.play);
          button.setOnClickListener(new View.OnClickListener() 
          {
              public void onClick(View v)
              {
                  VideoTestActivity.this.mc.show();
              }
          }
          );
  
      }
      
      VideoView video;
      MediaController mc;
  }
  

Android's Camera

• To use an Android phone's camera you make use of the Camera class.
• This class has a method takePicture(Camera.ShutterCallback shutter, Camera.PictureCallback raw, Camera.PictureCallback jpeg) which actually takes a picture
• Notice you provide three callbacks to the method to control how the picture is taken: the first has a method that is called when the shutter is opened, the second two have methods to get the reaw and jpeg data from the camera.
• Let's look at an app that takes a picture. To begin, we need to add the following lines to the AndroidManifest.xml file in order for anything to work:

  <uses-permission android:name="android.permission.CAMERA" />
  <uses-feature android:name="android.hardware.camera" />
  <uses-feature android:name="android.hardware.camera.autofocus" />
  

main.xml

• Our layout just has a SurfaceView in which the camera data will live:

  <?xml version="1.0" encoding="utf-8"?>
  <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
      android:orientation="vertical"
      android:layout_width="fill_parent"
      android:layout_height="fill_parent"
      >
  	<SurfaceView android:id="@+id/surface"
  		android:layout_weight="1" android:layout_width="fill_parent"
  		android:layout_height="fill_parent">
  	</SurfaceView>
  </LinearLayout>
  

• The Java Code for our app will consist of an extension of Activity, CameraTest, and a class implementing Camera.PictureCallback, called ImageCaptureCallback which we will use to save JPEG files.

CameraTest.java

package org.pollett;

import java.text.SimpleDateFormat;
import java.util.Date;

import android.app.Activity;
import android.content.ContentValues;
import android.content.Intent;
import android.graphics.PixelFormat;
import android.hardware.Camera;
import android.net.Uri;
import android.os.Bundle;
import android.provider.MediaStore.MediaColumns;
import android.provider.MediaStore.Images.ImageColumns;
import android.provider.MediaStore.Images.Media;
import android.view.KeyEvent;
import android.view.MenuItem;
import android.view.SurfaceHolder;
import android.view.SurfaceView;

public class CameraTest extends Activity implements SurfaceHolder.Callback {

    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        getWindow().setFormat(PixelFormat.TRANSLUCENT);
        setContentView(R.layout.main);
        
        /*
           the surface holder is used to manage what is actually displayed 
           on the surface 
         */
        surfaceView = (SurfaceView)findViewById(R.id.surface);
        surfaceHolder = surfaceView.getHolder();
        surfaceHolder.addCallback(this);
        surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);            
    }

    /**
       Adds menu to view the photos that have already been taken.
     */
    public boolean onCreateOptionsMenu(android.view.Menu menu)
    {
        MenuItem item = menu.add(0, 0, 0, "View Photos?");
        item.setOnMenuItemClickListener(
            new MenuItem.OnMenuItemClickListener() {
                public boolean onMenuItemClick(MenuItem item) {
                    Intent intent = new Intent(Intent.ACTION_VIEW,
                            CameraTest.this.targetResource);
                    startActivity(intent);
                    return true;
                }
            }
        );
        return true;
    }

    /**
        Pushing the center button takes a picture,
        and sets up the callback for saving it.
     */
    public boolean onKeyDown(int keyCode, KeyEvent event)
    {
        ImageCaptureCallback camDemo = null;
        if(keyCode == KeyEvent.KEYCODE_DPAD_CENTER) {
            try {
                String filename= this.timeStampFormat.format(new Date());
                ContentValues values = new ContentValues();
                values.put(MediaColumns.TITLE, filename);
                values.put(ImageColumns.DESCRIPTION, "Image from emulator");
                Uri uri = getContentResolver().insert(
                        Media.EXTERNAL_CONTENT_URI, values);
                camDemo = new ImageCaptureCallback(
                        getContentResolver().openOutputStream(uri)
                        );
            } catch (Exception e) {
                
            }
        }
        if(keyCode == KeyEvent.KEYCODE_BACK) {
            return super.onKeyDown(keyCode, event);
        }
        if(keyCode == KeyEvent.KEYCODE_DPAD_CENTER) {
            this.camera.takePicture(this.mShutterCallback, 
                    this.mPictureCallbackRaw, camDemo);
            return true;
        }
        return false;
    }
  
    /**
       Start the camera after the surface is created
     */
    public void surfaceCreated(SurfaceHolder holder) {
                this.camera = Camera.open();
    }
    
    /**
       Actually, display the camera data in the surface
     */
    public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
        if(this.isPreviewRunning) {
            this.camera.stopPreview();
        }
        Camera.Parameters p = this.camera.getParameters();
        p.setPreviewSize(w, h);
        try {
            this.camera.setPreviewDisplay(holder);
            this.camera.startPreview();
            this.isPreviewRunning = true;
        } catch (Exception e) {
            
        }
    }


    /**
       Shut things down when surface goes away
     */
    public void surfaceDestroyed(SurfaceHolder holder) {
        this.camera.stopPreview();
        this.isPreviewRunning = false;
        this.camera.release();
    }

    //Handles raw data when picture taken
    Camera.PictureCallback mPictureCallbackRaw = new Camera.PictureCallback() {
        public void onPictureTaken(byte[] data, Camera c) {
            CameraTest.this.camera.startPreview();
        }
    };

    Camera.ShutterCallback mShutterCallback = new Camera.ShutterCallback() {
        public void onShutter() {
            //could be used to play a sound
        }
    };
   
    Camera camera;
    boolean isPreviewRunning = false;
    SimpleDateFormat timeStampFormat = new SimpleDateFormat("yyyyMMddHHmmssSS");
    SurfaceView surfaceView;
    SurfaceHolder surfaceHolder;
    Uri targetResource = Media.EXTERNAL_CONTENT_URI;
    
    
}

ImageCaptureCallback.java

package org.pollett;

import java.io.OutputStream;
import android.hardware.Camera;
import android.hardware.Camera.PictureCallback;

public class ImageCaptureCallback implements PictureCallback {
    
    public ImageCaptureCallback(OutputStream fileoutputStream)
    {
        this.fileoutputStream = fileoutputStream;
    }
    
    public void onPictureTaken(byte[] data, Camera camera) {
        try {
            this.fileoutputStream.write(data);
            this.fileoutputStream.flush();
            this.fileoutputStream.close();
            
        } catch(Exception e) {
            e.printStackTrace();
        }
    }
    OutputStream fileoutputStream;
}

Demo

• In the emulator when you run the app, it shows a moving test pattern:
  
• The picture gallery that you can get to using the menu looks like:
• In File Explorer you can actually see the jpeg files in a DCIM folder on the sdcard.