Finish iOS Touch Events, Multimedia HTML 5 and Android

CS175

Chris Pollett

Nov 19, 1014

Outline

Finish iOS Touch Events
Multimedia HTML 5
AndroidPlaying Audio and Video
Using the Camera

Introduction

On Monday, we started talking about touch events for iOS.
We start today by going over a code example that we demo'd on Monday. It basically tracks the number of taps and number of touches on the screen.
We then go over how to do some simple 2D graphics in HTML 5
We will folllow this by talking about how to handle touch in HTML 5
We then look at the video and audio tags
The remainder of the lecture will be about multimedia in Android

Touch Application

We will now create a simple app to experiment with touch events on the iPhone.
It will display a message saying what kind of touch event has occurred, and it will also say how many touches or taps there were.
The touch events are better demo'd on an actual device than in the simulator.
For this new project we use the Single-View App template.

View Controller Header

We then modify the view controller header as follows:

#import <UIKit/UIKit.h>

@interface ViewController : UIViewController
@property (weak, nonatomic) IBOutlet UILabel *messageLabel;
@property (weak, nonatomic) IBOutlet UILabel *tapsLabel;
@property (weak, nonatomic) IBOutlet UILabel *touchesLabel;

-(void)updateLabelsFromTouches: (NSSet *)touches;
@end

Changes to TouchApp Storyboard file

We next modify the view controller's storyboard file. We turn off Use Autolayout for the controller and set it be just for iPhone. The we add three labels.
We center the labels and change the text to Messages, Taps, Touches.
We control drag from the File's Owner to each of the labels and either select, the messagesLabel outlet, tapsLabel outlet, or touchesLabel outlet.
Finally, we click on the view in interface builder and click on the View Attributes tab.
We make sure that under the Interaction settings both User Interaction Enabled and Multiple Touch are checked.

View Controller Code

The view controller's code is then set up as follows:

#import "TouchAppViewController.h"

@implementation TouchAppViewController

-(void) updateLabelsFromTouches:(NSSet *)touches {
    NSUInteger numTaps = [[touches anyObject] tapCount];
    NSString *tapsMessage = [[NSString alloc]
                             initWithFormat:@"%lu taps detected", (unsigned long)numTaps];
    _tapsLabel.text = tapsMessage;

    NSUInteger numTouches = [touches count];
    NSString *touchMsg = [[NSString alloc] initWithFormat:
                          @"%lu touches detected", (unsigned long)numTouches];
    _touchesLabel.text = touchMsg;
}
- (void)viewDidLoad {
    [super viewDidLoad];
    // Do any additional setup after loading the view, typically from a nib.
}

- (void)didReceiveMemoryWarning {
    [super didReceiveMemoryWarning];
    // Dispose of any resources that can be recreated.
}


#pragma mark -
- (void) touchesBegan:(NSSet *)touches withEvent:(UIEvent *)event {
    _messageLabel.text = @"Touches Began";
    [self updateLabelsFromTouches:touches];
}

- (void) touchesCancelled:(NSSet *)touches withEvent:(UIEvent *)event {
    _messageLabel.text = @"Touches Cancelled";
    [self updateLabelsFromTouches:touches];
}

- (void) touchesEnded:(NSSet *)touches withEvent:(UIEvent *)event {
    _messageLabel.text = @"Touches Ended";
    [self updateLabelsFromTouches:touches];
}

- (void) touchesMoved:(NSSet *)touches withEvent:(UIEvent *)event {
    _messageLabel.text = @"Drag detected";
    [self updateLabelsFromTouches:touches];
}

@end

Touch App Screenshot

Screenshot of the touch app running showing a message that a touch event just ended

Handling More Complicated Touch Events

To handle more complicated events on the iPhone such as swipes, pinches, and proper handling of multiple taps we use the basic structure we've already created and make a note of where things happen on the screen.
We say something is a swipe in some direction based on the coordinate where the finger first touched the screen and based on the coordinate on which it was released from the screen.
Similarly, for a pinch we check that two touches were registered on the screen and their starting coordinates are farther apart then their ending coordinates.

To check if something is a double tap, but not a triple tap, you can use methods like:


case 2: //double tap
[NSObject cancelPreviousPerformRequestWithTarget:self
   selector:@selector(singleTap) object:nil];

[self performSelector:@selector(doubleTap) withObject:nil afterDelay: .4]
break;

case 3: //triple tap
[NSObject cancelPreviousPerformRequestWithTarget:self
   selector:@selector(doubleTap) object:nil];

[self performSelector:@selector(tripleTap) withObject:nil afterDelay: .4]
break;

in touchesBegan.

More complicated gestures like making a check-mark on the screen are similarly constructed out of the basic touch events
We conclude our discussion of iPhone Touch Events by giving an example of how to do swipes.

Swipe Example

To begin we create a new single view application.
This application will display in a label whether a vertical or horizontal swipe was detected.

We modify the boiler-plate view controller header to look like:

#import <UIKit/UIKit.h>

#define kMinimumGestureLength  25
#define kMaximumVariance 5

@interface SwipeAppViewController : UIViewController 
@property (weak, nonatomic) IBOutlet UILabel *label;
@property CGPoint gestureStartPoint;
-(void)eraseText;
@end

Next we edit the view controller's stroyboard file and add a label and ctrl-click from the File's Owner to the label and connect it to our outlet.

Swipe Controller Code

#import "SwipeAppViewController.h"

@implementation SwipeAppViewController

-(void)eraseText {
   _label.text = @"";
}


- (void)didReceiveMemoryWarning {
	// Releases the view if it doesn't have a superview.
    [super didReceiveMemoryWarning];
	
	// Release any cached data, images, etc that aren't in use.
}

- (void)viewDidUnload {
	// Release any retained subviews of the main view.
	// e.g. self.myOutlet = nil;
}



#pragma mark -
-(void)touchesBegan:(NSSet *)touches withEvent:(UIEvent *)event {
	UITouch *touch = [touches anyObject];
	gestureStartPoint = [touch locationInView:self.view];
}

-(void)touchesMoved:(NSSet *)touches withEvent:(UIEvent *)event {
	UITouch *touch = [touches anyObject];
	CGPoint currentPosition = [touch locationInView:self.view];
	
	CGFloat deltaX = fabsf(gestureStartPoint.x - currentPosition.x);
	CGFloat deltaY = fabsf(gestureStartPoint.y - currentPosition.y);
	
	if(deltaX >= kMinimumGestureLength && deltaY <= kMaximumVariance) {
		_label.text = @"Horizontal swipe detected";
		[self performSelector:@selector(eraseText) withObject:nil afterDelay: 2];
		
	} else 	if(deltaY >= kMinimumGestureLength && deltaX <= kMaximumVariance) {
		_label.text = @"Vertical swipe detected";
		[self performSelector:@selector(eraseText) withObject:nil afterDelay: 2];
	}
}

@end

Android Key and Touch Events

We have already seen that in Android we can create a class which implements View.OnClickListener and then use a View's setOnClickListener method to get this class to handle click events for that view.
Similarly, you can set a View's OnKeyListener with an object of a class implementing View.OnKeyListener to handle KeyEvents.
To implement this interface you have to define a method:
```
boolean onKey(View v, int keyCode, KeyEvent event)
```
the key code tells you what key was pressed.
KeyEvent's have predefined constants for the different keyCode's as well as methods to figure out keyboard state (was shift pressed? etc).
A View also can have a OnTouchListener which is an Object of a class implementing View.OnTouchListener.
To implement this class, you need to implement the method:
```
boolean onTouch(View v, MotionEvent event)
```
MotionEvent has a method getAction() which returns an int from a list of constants which says what kind of touch event it was.
The event actually holds an array of x, y coordinates associated with the event and has member functions getHistoricalX(int i), getHistoricalY(int i), getHistorySize() for accessing these.
The most recent points can be obtained with getX() and getY().

Two-Dimensional Graphics in HTML

Since Netscape 2, which introduced Javascript in 1996, there have been different techniques to get crude 2D graphics in browsers -- the very earliest techniques were very hackish involving drawing single pixel images over and over again.
Starting with IE 5 in 1998, an XML, tag-based language approach to 2D graphics called VML became available to web designers.
About a year later another tag-based, this time non-Microsoft language, called SVG appeared, and this eventually became supported by all major browsers.
Tag-based (vector/scene-based) approaches to 2D-graphics can often be inconvenient for scripting, writing games, etc as they are a little higher level and so slower.
In 2004, Apple introduced the canvas tag in its Safari browser to support a more low-level, raster, programmatic approach to graphics. This is now supported by all major browsers, is a WHATWG standard, and it is this tag that we will briefly introduce.

Canvas Tag

A basic canvas tag usage might look like:

<canvas id="drawpad" />
<style type="text/css">
#drawpad {
    width: 200px;
    height: 100px;
    background-color: light-gray;
}
</style>

We can then refer to this object in Javascript, using document.getElementById("drawpad") or in jQuery using $("#drawpad"), use it to get a context and then start drawing:

<script type="text/javascript"*gt;
   drawpad_object = document.getElementById("drawpad");
   drawpad_context = drawpad_object.getContext("2d");
   drawpad_context.strokeStyle = "blue";
   drawpad_context.strokeRect(10,10,50,50);
   drawpad_context.fillStyle = "green";
   drawpad_context.font = "32pt Arial";
   max_width = 100;
   drawpad_context.rotate(3.14/3); //pi/3
   drawpad_context.fillText("hello", 80, -10, max_width);
 
</script>

In the above, besides 2d we could also have used webgl if we wanted a more OpenGL-like, 3D-type canvas.
Notice coordinates for the fillText are relative to the rotated canvas.
Context also supports methods for clearing areas clearRect, drawing images using drawImage. To specify more complex shapes, you call drawpad_context.beginPath(), give your path instructions, then drawpad_context.closePath(), and finally call fill(path) or stroke(path). Here path instructions can be moveTo, lineTo, arc, quadraticCurveTo, bezierCurveTo, etc.
To do double buffering with canvas, make a page with two canvas tags. Style them so they are the same size, but have one with visibility hidden. Draw your graphics into this one, then toggle which one is hidden.

Touch Events HTML5

In HTML 5 there are four main touch events which can be applied to any tag: touchStart, touchMove, touchEnd, touchCancel
The meanings are similar to the corresponding iOS events we've already discussed

As an example of setting up such events, consider

<canvas id="drawpad"

  ontouchstart="startHandler(event);"

  ontouchmove="moveHandler(event);"

  ontouchend="endHandler(event);"

  ontouchcancel="cancelHandler(event);"

/>

You can also add events programmatically with:

drawpad_object.addEventListener("touchstart", startHandler2, false);

These events are easier to test on a mobile device or using the iPhone Simulator / Android Emulator.

Video and Audio Tags

A basic video tag in HTML 5 looks like:

<video id="myvideo" width="320" height="240" controls="controls">
  <source src="some_movie.mp4" type="video/mp4" />
  <!-- firefox does not support mp4-->
  <source src="some_movie.ogg" type="video/ogg" />
  <!-- firefox does support ogg but others don't-->
  Your browser does not support the video tag.
</video>

Currently, WebM (Google) (mimetype video/webm,) and Ogg seem like the most cross-browser supported formats.
The HTML 5 audio tag is almost identical to the above except the source tags should be audio files.
If we did myvideo_object = document.getElementById("myvideo"), the object would support load(), play(), pause() methods, and have a currentSrc, currentTime, duration and playbackRate properties. The tag also can supports loadedmetadata, timeupdate, pause, play, and ended events.
Each browser maker also supports several extensions to these events.

Multimedia on Android

The foundation of Android's multimedia platform is based on OpenCore from PacketVideo.
We will begin looking at these libraries by looking at how to play an MP3 file.
The steps to write such a program involve:
1. Put the mp3 file in the res/raw directory of the project. You will have to create this folder.
2. Create a new instance of MediaPlayer and reference your MP3 by calling MediaPlayer.create().
3. Call the MediaPlayer method start().
We will make a simple app with a single button which will cause an mp3 file with the radio broadcast of Orson Welles' War of the Worlds to play. (Public domain from the 1930s).

main.xml

Here is the main.xml for our project (this is where we define the button):

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    >
<TextView  
    android:layout_width="fill_parent" 
    android:layout_height="wrap_content" 
    android:text="Test Media Player"
    />
 <Button
    android:id="@+id/playwarworlds" 
    android:layout_width="fill_parent" 
    android:layout_height="wrap_content" 
    android:text="Play Orson Welles"
    />
</LinearLayout>

AudioTestActivity.java

Here is the source code of our activity:

import android.view.View;
import android.view.View.OnClickListener;
import android.media.MediaPlayer;
import android.media.MediaPlayer.*;

public class AudioTestActivity extends Activity {

    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
        View button = findViewById(R.id.playwarworlds);
        button.setOnClickListener( new OnClickListener() {
            public void onClick(View v) 
            {
                MediaPlayer player = MediaPlayer.create(AudioTestActivity.this, R.raw.warworlds);
                player.start();
                player.setOnCompletionListener(new OnCompletionListener() 
                {
                    public void onCompletion(MediaPlayer m) 
                    {
                    }
                });
            }
        });
    }
}

A MediaPlayer enters a prepare state before playing which gets the media ready to be played (might be remote).
MediaPlayer also has pause(), resume() and stop() (returns to before prepare was called) methods which could be hooked up to buttons

Playing Video

We next test out playing video in Android.
To do this we will create a small app which will play the first copyrighted film ever (Fred Ott's sneeze).
We will put a file fred_ott.mp4 on the sdcard of the emulator using the Emulator's File Explorer.
Our app will then play this file.

main.xml

Here is our main.xml file:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    >
<VideoView 
    android:id="@+id/video" 
    android:layout_width="320px" 
    android:layout_height="240px" 
/>
<Button
    android:id="@+id/play"
    android:text="Play Fred Ott" 
    android:layout_width="fill_parent" 
    android:layout_height="wrap_content" 
    />
</LinearLayout>

VideoTestActivity.java

The activity is reasonably straightforward:

package org.pollett;

import android.app.Activity;
import android.graphics.PixelFormat;
import android.os.Bundle;
import android.view.View;
import android.widget.MediaController;
import android.widget.VideoView;

public class VideoTestActivity extends Activity 
{
    @Override
    public void onCreate(Bundle savedInstanceState) 
    {
        super.onCreate(savedInstanceState);
        getWindow().setFormat(PixelFormat.TRANSLUCENT);
        setContentView(R.layout.main);
        video = (VideoView)findViewById(R.id.video);
        video.setVideoPath("/mnt/sdcard/fred_ott.mp4");
        mc = new MediaController(this);
        mc.setMediaPlayer(video);
        video.setMediaController(mc);
        video.requestFocus();

        View button = findViewById(R.id.play);
        button.setOnClickListener(new View.OnClickListener() 
        {
            public void onClick(View v)
            {
                VideoTestActivity.this.mc.show();
            }
        }
        );

    }
    
    VideoView video;
    MediaController mc;
}

Android's Camera

To use an Android phone's camera you make use of the Camera class.
This class has a method takePicture(Camera.ShutterCallback shutter, Camera.PictureCallback raw, Camera.PictureCallback jpeg) which actually takes a picture
Notice you provide three callbacks to the method to control how the picture is taken: the first has a method that is called when the shutter is opened, the second two have methods to get the reaw and jpeg data from the camera.

Let's look at an app that takes a picture. To begin, we need to add the following lines to the AndroidManifest.xml file in order for anything to work:

<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />

main.xml

Our layout just has a SurfaceView in which the camera data will live:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    >
	<SurfaceView android:id="@+id/surface"
		android:layout_weight="1" android:layout_width="fill_parent"
		android:layout_height="fill_parent">
	</SurfaceView>
</LinearLayout>

The Java Code for our app will consist of an extension of Activity, CameraTest, and a class implementing Camera.PictureCallback, called ImageCaptureCallback which we will use to save JPEG files.

CameraTest.java

package org.pollett;

import java.text.SimpleDateFormat;
import java.util.Date;

import android.app.Activity;
import android.content.ContentValues;
import android.content.Intent;
import android.graphics.PixelFormat;
import android.hardware.Camera;
import android.net.Uri;
import android.os.Bundle;
import android.provider.MediaStore.MediaColumns;
import android.provider.MediaStore.Images.ImageColumns;
import android.provider.MediaStore.Images.Media;
import android.view.KeyEvent;
import android.view.MenuItem;
import android.view.SurfaceHolder;
import android.view.SurfaceView;

public class CameraTest extends Activity implements SurfaceHolder.Callback {

    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        getWindow().setFormat(PixelFormat.TRANSLUCENT);
        setContentView(R.layout.main);
        
        /*
           the surface holder is used to manage what is actually displayed 
           on the surface 
         */
        surfaceView = (SurfaceView)findViewById(R.id.surface);
        surfaceHolder = surfaceView.getHolder();
        surfaceHolder.addCallback(this);
        surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);            
    }

    /**
       Adds menu to view the photos that have already been taken.
     */
    public boolean onCreateOptionsMenu(android.view.Menu menu)
    {
        MenuItem item = menu.add(0, 0, 0, "View Photos?");
        item.setOnMenuItemClickListener(
            new MenuItem.OnMenuItemClickListener() {
                public boolean onMenuItemClick(MenuItem item) {
                    Intent intent = new Intent(Intent.ACTION_VIEW,
                            CameraTest.this.targetResource);
                    startActivity(intent);
                    return true;
                }
            }
        );
        return true;
    }

    /**
        Pushing the center button takes a picture,
        and sets up the callback for saving it.
     */
    public boolean onKeyDown(int keyCode, KeyEvent event)
    {
        ImageCaptureCallback camDemo = null;
        if(keyCode == KeyEvent.KEYCODE_DPAD_CENTER) {
            try {
                String filename= this.timeStampFormat.format(new Date());
                ContentValues values = new ContentValues();
                values.put(MediaColumns.TITLE, filename);
                values.put(ImageColumns.DESCRIPTION, "Image from emulator");
                Uri uri = getContentResolver().insert(
                        Media.EXTERNAL_CONTENT_URI, values);
                camDemo = new ImageCaptureCallback(
                        getContentResolver().openOutputStream(uri)
                        );
            } catch (Exception e) {
                
            }
        }
        if(keyCode == KeyEvent.KEYCODE_BACK) {
            return super.onKeyDown(keyCode, event);
        }
        if(keyCode == KeyEvent.KEYCODE_DPAD_CENTER) {
            this.camera.takePicture(this.mShutterCallback, 
                    this.mPictureCallbackRaw, camDemo);
            return true;
        }
        return false;
    }
  
    /**
       Start the camera after the surface is created
     */
    public void surfaceCreated(SurfaceHolder holder) {
                this.camera = Camera.open();
    }
    
    /**
       Actually, display the camera data in the surface
     */
    public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
        if(this.isPreviewRunning) {
            this.camera.stopPreview();
        }
        Camera.Parameters p = this.camera.getParameters();
        p.setPreviewSize(w, h);
        try {
            this.camera.setPreviewDisplay(holder);
            this.camera.startPreview();
            this.isPreviewRunning = true;
        } catch (Exception e) {
            
        }
    }


    /**
       Shut things down when surface goes away
     */
    public void surfaceDestroyed(SurfaceHolder holder) {
        this.camera.stopPreview();
        this.isPreviewRunning = false;
        this.camera.release();
    }

    //Handles raw data when picture taken
    Camera.PictureCallback mPictureCallbackRaw = new Camera.PictureCallback() {
        public void onPictureTaken(byte[] data, Camera c) {
            CameraTest.this.camera.startPreview();
        }
    };

    Camera.ShutterCallback mShutterCallback = new Camera.ShutterCallback() {
        public void onShutter() {
            //could be used to play a sound
        }
    };
   
    Camera camera;
    boolean isPreviewRunning = false;
    SimpleDateFormat timeStampFormat = new SimpleDateFormat("yyyyMMddHHmmssSS");
    SurfaceView surfaceView;
    SurfaceHolder surfaceHolder;
    Uri targetResource = Media.EXTERNAL_CONTENT_URI;
    
    
}

ImageCaptureCallback.java

package org.pollett;

import java.io.OutputStream;
import android.hardware.Camera;
import android.hardware.Camera.PictureCallback;

public class ImageCaptureCallback implements PictureCallback {
    
    public ImageCaptureCallback(OutputStream fileoutputStream)
    {
        this.fileoutputStream = fileoutputStream;
    }
    
    public void onPictureTaken(byte[] data, Camera camera) {
        try {
            this.fileoutputStream.write(data);
            this.fileoutputStream.flush();
            this.fileoutputStream.close();
            
        } catch(Exception e) {
            e.printStackTrace();
        }
    }
    OutputStream fileoutputStream;
}

Demo

In the emulator when you run the app, it shows a moving test pattern:
The picture gallery that you can get to using the menu looks like:
In File Explorer you can actually see the jpeg files in a DCIM folder on the sdcard.