Importing/Exporting to/from an Array

You can convert your data from an array to a variety of formats:

a = np.array([1, 2, 3], int)
a.tolist() # [1, 2, 3]
list(a) # [1, 2, 3] # same as above
tuple(a) # (1, 2, 3)
a.tofile("bob.txt") #writes to the file bob.txt
c = a.tobytes() # '\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00
    # \x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00'

We've already seen we can create array from a variety of object like lists and tuples. We can also read in arrays from files or strings:
```
b = np.fromfile("bob.txt",int)
d = np.frombuffer(c)
```

Pillow

Pillow (derived from PIL, Python Image Library) is another useful Python library.
PIL was created by Fredrik Lundh in 1995.
Pillow is a drop in replacement for PIL that also provided Python 3 support.
It can be used to read in and manipulate images.
It can also be used to write out new images.
To install Pillow you might need to write the line:
```
pip install pillow
```
To import PIL, you simply add to your code:
```
from PIL import Image
```

Image Reading, Creation, Capturing

Once imported you can read in an image with a statement like:

im = Image.open("myphoto.jpg") #Pillow supports many formats
    #png, ppm, gif, tiff, etc
im.show() #this would display the image in a new window

You can also create a blank image:

im = Image.new('RGBA', (100, 100), (255,255,255,255)) 
   #100x100 size image with white opaque background
im.show();

Besides RGBA color mode, there are the following other modes: 1 (black white 1 pixel/byte), L (black white 8 pixels/byte), 8 (8-bit color palette, i.e., there is a palette of 256 colors where each color is specified as a 32 bit number. The 8 bit number in a pixel says which of the colors to choose), RGB, CMYK, YCbCr, LAB, HSV, I (32-bit integer pixels), F (32-bit floating pixels). You can use im.convert(new_mode) to make an image based on im in a different mode.

On Mac and Windows you can create an image from a screen capture:

from PIL import Image, ImageGrab
im = ImageGrab.grab() # capture whole screen
im.show()
im = ImageGrab.grab((10,10,250,250)) 
    #captures from (10,10) to (250,250) of screen
im.show()

Image Manipulation With Pillow

Pillow can be used to resize images:

out_im = im.resize((50,200)) #out_im is a rescaled image of size 50x 200
  #can also specify as a second argument how to resample from:
  #Image.NEAREST , BILINEAR, BICUBIC, LANCZOS

Pillow has methods for rotations, transformations, and filters:

from PIL import Image, ImageFilter
im = Image.open("myphoto.jpg")
out_im = im.rotate(30)
out_im = im.transpose(Image.FLIP_LEFT_RIGHT) #other possibilities are
  # FLIP_TOP_BOTTOM, ROTATE_90, ROTATE_180, ROTATE_270
out_im = im.filter(ImageFilter.BLUR) #other possibilities are:
  # CONTOUR, DETAIL, EDGE_ENHANCE, EDGE_ENHANCE_MORE, EMBOSS, 
  # FIND_EDGES, SMOOTH, SMOOTH_MORE, and SHARPEN
  # MinFilter(some_num), MaxFilter(some_num), MedianFilter(some_num),
  # ModeFilter(some_num), Kernel(size, kernel, scale=None, offset=0)
  # RankFilter(size, rank)

4-tuples can be used to define rectangles for cropping and pasting:

box = (20, 20, 80, 80)
out_im = im.crop(box) #new image given by rectangle with top left (20,20)
   #bottom left (80, 80) 
box2 = (0, 0, 60, 60)
im.paste(out_im, box2) # paste image back into old image at the given coords
   #rectangle in box2 has to be of same size as dimension of image

Images of the same size and mode can be blended:

im1 = Image.open("myphoto1.jpg")
im2 = Image.open("myphoto2.jpg")
q = 0.5
out = Image.blend(im1, im2, q) #adds images to make output image
   #with weight 1-q to im1 and q to im2.

Applying Operations to a Whole Image

If our image is not black and white, we can use split to break it into components:

r, g, b, a = 0, 1, 2, 3
source = im1.split(); # creates 4 images one for each color component

We can then manipulate a component by applying a point transform:

blue_mask = source[b].point(lambda i: i > 50  and 255) 
   # if i > 50 output 255; otherwise 0
mod_green = source[g].point(lambda i: i * 0.4)
source[g].paste(mod_green, None, blue_mask) 
   #mod_green is pasted back only where mask says
   #None component could have been a rectangle to restrict paste to

Finally, we can merge results back together:
```
im1 = Image.merge(im1.mode, source)
```

Drawing to an Image

ImageDraw in Pillow supports several methods for drawing into an image:

from PIL import Image, ImageDraw, ImageFont
im = Image.new('RGB', (100, 100), (0,0,0))
draw = ImageDraw.Draw(im)
draw.point(((5, 5), (85, 5), (85, 6)), fill=(255, 0, 0)) #draw three red points
draw.line((10,15,80,90), fill=(255,255,255)) #draw a white line
draw.ellipse((30,30,40,50), fill=(0,0,255)) # draw a blue ellipse
draw.text((20, 55), "hello world", font=ImageFont.load_default())
del draw
im.show()

Other methods of Draw include: arc, bitmap, chord, multiline_text, pieslice, polygon, rectangle, shape

Getting Data, Mapping Image Data to Numpy, Writing Images

To get the value of a single pixel we can use getpixel:

im = Image.new('RGB', (100, 100), (0,0,0))
draw = ImageDraw.Draw(im)
draw.point((5, 5), fill=(255, 0, 0))
del draw
im.getpixel((5,5)) # (255, 0, 0)

To get all of the data in one go one can use getdata or tobytes:

list(im.getdata()) # Since im was in rgb, this will output a 
    # sequence of triples [(0, 0, 0), (0, 0, 0), ...
import numpy as np
im_array = np.frombuffer(im.tobytes(), dtype=np.uint8) #1-dimensional
im_array.reshape((im.size[1], im.size[0], 3)) 
   # make into 3d array where 3rd dimension color

When we are done manipulating an image, we can use its save() method to write it back to a desired target:
```
im.save("out_image.png", "PNG") #so you can choose image file format when save
```

In-Class Exercise

Write a Python program using Pillow that takes a screenshot of the top left quadrant of your screen, and then rescales this to a 100 x 100 grayscale image.
Post your code and an example image to the Sep 29 In-Class Exercise Thread.

Neural Net Experiments

People use neural nets for tasks where it would be hard for a human to come up with an exact algorithm for that task. For example, handwriting recognition.
Not everyone uses the same hardware or even neural network software.
On the other hand, people do want to be able to share their neural net algorithms.
A surprising number of neural net paper results are entirely unreproducible.
To share in this context means to describe the architecture of ones network and the training algorithm used in sufficient detail that someone on different hardware and using different neural net software has some hope of still producing a similarly trained network.
Since for neural networks it might be hard to figure out exactly what a trained model is doing, it is hard to give a formal proof of correctness for the trained model.
Neural network experiments which show how well a model generalizes, how accurate it is, etc. are used therefore in lieu of such a proof.
They serve as an argument as to why or why not use a particular architecture and training algortihm to solve some problem.

What should a Neural Network Experiment contain?

A statement of a hypothesis which could be proven true or false.
- The hypothesis should be settling something interesting to know about your neural net. For example, how well it generalizes.
- Your hypothesis should be robust under tweakings of hyperparameters. I.e., if you can change the value of the answer of your hypothesis too easily by tweaking hyperparameters, then it suggests your results don't generalize.
A description of your neural net and training set ups for the experiment conducted. This should be in sufficient details so that someone else could try to replicate your experiment.
The results of your experiments. I.e., what was the output of your experiments.
A conclusion from your results with regard to your initial hypothesis.

Things to Measure with Experiments 1 - Accuracy , Confusion Matrix

What kind of hypothesis we have is dependent on the things we can actually measure with an NN experiment. On the next few slides I give some examples of the kinds of things we can measure. After this, I give some example hypothesis which could be tested using these kind of experiments.

Accuracy = Number of times model correct/Number of test examples.
We can refine this a little by asking for the following four different rates which can be arranged into 2x2 table known as a confusion matrix (here I am using rates for the entries, people often use raw counts (the numerators of the below)):

TPR FPR

FNR TNR
1. True Positive Rate (TPR, aka sensitivity)= number of times model output true when test example was true/number of times model output true.
2. False Positive Rate(FPR) = number of times model output true when test example was false/number of times model output true.
3. False Negative Rate (FNR) = number of times model output false when test example was true/number of times model output false.
4. True Negative Rate (TNR, aka specificity) = number of times model output false when test example was false/number of times model output false.
If we use raw counts rather than rates, we can generalize the confusion matrix to the multi-label setting. Below is an example of doing this for a situation with three labels A,B,C:

Observed Label

A B C

Predicted Label A 5 1 2

B 3 7 4

C 0 2 9

		Observed Label
		A	B	C
Predicted Label	A	5	1	2
B	3	7	4
C	0	2	9

Things to Measure with Experiments 2 - ROC Curve

Receiver Operator Curve (ROC) curve were developed during WW II for analysis of radar signals.
An ROC curve plots the TPR versus the FPR for a binary classifier.
To create it:
1. Compute a scatter plot of classifier scores versus training items.
2. For each possible score value `v` from largest to smallest:
  1. Choose it as a classifier threshold and figure out what the `TPR` and `FPR` would be for that threshold, `TPR_v`, `FPR_v`.
  2. Plot `(FPR_v, TPR_v)` as a point on the ROC curve.
3. The curve is then made by joining the points plotted.
In the figure above, for the threshold v, we get the point `(2/6 = 1/3, 4/6=2/3)`.
The curve will always be above the line y= x.
A larger area under this curve indicates a better classifier.

Things to Measure with Experiments 3 - Performance

Runtime of training as a function of data set size.
Memory usage during training as a function of data set size.
Number of specific function calls as a function of data set size.
If your training algorithm involved random numbers, one should repeat the experiments several times and calculate a standard deviation for the rates above.

Example Hypotheses

Neural Net Architecture A is more accurate than Neural Net Architecture B.
Training Algorithm A converges to a more accurate model than Training Algorithm B.
Neural Net Architecture A requires less memory, less neurons, etc to train than Neural Net Architecture B
Training Algorithm A converges faster than Training Algorithm B.
Accuracy of model using Architecture A and Training Algorithm B is such and such function of training time/some other parameter.
The best value for hyperparameter `lambda` is `c`.
The model will generalize to never before seen data if hyperparameter `lambda` is set to value `c`.

Representing Our Results - Graphing and Charting

Often plotting the results of our experiments on a graph or chart can make them easier to understand.
When we use a graph, it is important to give the reader enough detail so that someone else can understand what it represents.
At a minimum, this means:
1. Give the graph/chart a title or caption
2. Label for a graph each of the axes, each function plotted, and specifying units.
3. Label for a chart each of the items being chartted.
In python, we can use matplotlib (created by John Hunter, 2003) to do a variety of plotting and charting tasks.
It can be installed via the command:
```
pip install matplotlib
```
The matplotlib Gallery has a wide variety of plotting examples.

matplotlib - Basic Graphing

If we have two numpy arrays of points we can graph them together as follows:

import matplotlib.pyplot as plt
import numpy as np
a = np.arange(0, 10,.5, dtype=float);
b = a * a
plt.title("The function y=x^2")
plt.xlabel("x-axis")
plt.ylabel("y-axis")
plt.plot(a,b)
plt.show()

After the show() call, the plot is reset. A second call to plt.show() would output nothing: We need to make new calls to set up the another plot.
We can specify as part of the plot() a variety of attributes of the points to be plot such as color, linetype, and linewidth.
We can call plot() multiple times to plot multiple functions on the same graph.

Below we do this and illustrate different linetypes and how to make a legend:

import matplotlib.pyplot as plt
import numpy as np
a = np.arange(0, 10,.5, dtype=float);
plt.title("Growth rates y=x, y=x^2, and y=x^3")
plt.xlabel("x-axis")
plt.ylabel("y-axis")
id_line, = plt.plot(a,a, color="blue",
    label="y=x", linestyle='dashed', linewidth=2) 
quad_line, = plt.plot(a, a**2,
    color="red", label="y=x^2", linestyle='dotted') 
cube_line, = plt.plot(a, a**3,
    color="green", label="y=x^2",
    linestyle='dashdot')
plt.legend(handles=[id_line, quad_line, cube_line],
    loc=2) 
    #loc can be a number 1-4, number represents
    #which corner
plt.show()

matplotlib - Scatter Plots, Bar, Histograms, Pie Charts

matplotlib can be used to make scatter plots:

import matplotlib.pyplot as plt
import numpy as np
a = np.arange(0, 10,.5, dtype=float);
plt.title("Various polynomials")
plt.xlabel("x-axis")
plt.ylabel("y-axis")
plt.scatter(a, a**2)
plt.scatter(a, a**3, 
   marker="+", color="green");
plt.scatter(a, a**4, 200, marker="o");
   # 200 is size in pixels
plt.show()

matplotlib can be used to plot histograms:

import matplotlib.pyplot as plt
import numpy as np
a = np.array([1,1,2,2,2,2,3,4,4,4,4,
    5,5,5,5,6,6,7,8,8,9,9,9,10,10,10])
plt.hist(a, bins=5)
plt.show();

matplotlib can be used to plot bar charts:

import matplotlib.pyplot as plt
import numpy as np
plt.title("Product Comparison")
plt.bar(["Product A", "Product B"],
    [50, 100])
plt.show();

matplotlib can be used to plot pie charts:

import matplotlib.pyplot as plt
import numpy as np
plt.title("Marketshare Comparison")
companies = ["Company A", "Company B",
   "Company C"]
shares = [20, 50, 30]
colors = ['red', 'green', 'blue']
plt.pie(shares, labels=companies, 
    colors=colors, startangle=100)
plt.show();

matplotlib - Saving Images

Rather than using show() to display images to a window, we can use draw() and savefig() to write an a graph or chart to a file:

import matplotlib.pyplot as plt
plt.title("Product Comparison")
plt.bar(["Product A", "Product B"],
    [50, 100])
plt.draw();
plt.savefig("product_comparison.png");

Pillow, Neural Net Experiments

Outline

Introduction