Introduction

Last class, we began talking about TensorFlow.
We described how to build computational graphs using TensorFlow from constant, placeholder, or Variable leaves using operations like +, *, log_sigmoid, etc.
Base datatypes that such nodes in TensorFlow can manipulate are float32, float64, int32, int64, uint8, int16, int8, uint16, half
Nodes in a TensorFlow computation graph are Tensor objects.
We showed how to create a TensorFlow Session and use it to initialize Variable's and compute the outputs of a computational graph when placeholders have been given input values.
We then showed how a loss function could be specified for such a graph
We talked about tensorflow.train.GradientDescentOptimizer, a subclass of tensorflow.train.Optimizer, which could be used to update the Variables of a graph according to a specified loss node in our graph using a gradient descent operation and training data fed into placeholders.
We gave a complete example of this and showed how to read off adjusted variables after training had been done.
The trained graph could also be used to determine values on new inputs.
Today, we start by looking at how to build more complicated networks using TensorFlow.

Creating a Lot of Leaf Nodes at Once - placeholders and constants

Last day, we created a simple placeholder using the statement:
```
x = tf.placeholder(tf.float32)
```
The function tensorflow.placeholder() actually takes three arguments: a dtype (datatype), a shape (in numpy sense), and a name for the operations (useful for debugging with TensorFlows graphing tools).

For example, if we wanted to create a 3x3 array of placeholders we could do:

import tensorflow as tf
x = tf.placeholder(tf.float32, shape=(3,3))
session = tf.Session()
session.run(x, {x:[[1,2,3],[4,5,6],[7,8,9]]})
#outputs
#array([[ 1.,  2.,  3.],
#       [ 4.,  5.,  6.],
#       [ 7.,  8.,  9.]], dtype=float32)

The function tensorflow.constant() takes fives arguments value (a value or list of values of dtype), a dtype, a shape, a name, and verify_shape a flag to check whether the supplied value is of correct shape. So for example we can create a 3x2 array of constants using:
```
x = tf.constant([[1,2],[3,4],[5,6]], tf.float32, shape=(3,2))
session = tf.Session()
session.run(x)
#outputs
#array([[ 1.,  2.],
#       [ 3.,  4.],
#       [ 5.,  6.]], dtype=float32)
```

Creating a Lot of Leaf Nodes at Once - Variables

A Variable is a subclass of Tensor which has methods and properties suitable for training. The constructor Variable() is a little more complicated than placeholder() or constant(). The key thing though is the first argument, initial-value, can take a list of initial values which by default specifies the shape of the Variable:
```
import tensorflow as tf
W = tf.Variable([[0,1,0],[1,-1,1]], dtype=tf.float32)
session = tf.Session()
init = tf.global_variables_initializer()
session.run(init)
session.run(W)
#outputs
# array([[ 0.,  1.,  0.],
#       [ 1., -1.,  1.]], dtype=float32)
```

The tensorflow module itself has a function tensorflow.get_variables() which can be used to create variables as well. Its first four arguments are: a name, a shape, a dtype, and an initializer. Here is an example using it:

W = tf.get_variable("W", shape=[2,3], initializer = tf.zeros_initializer)
init = tf.global_variables_initializer()
session = tf.Session()
session.run(init)
session.run(W)
#outputs
#array([[ 0.,  0.,  0.],
#       [ 0.,  0.,  0.]], dtype=float32)

Rather than use the tf.zeros_initializer, we can supply anything that can be converted to the class tensorflow.Initializer. This includes using a list of initial values or using one of the other initilizer functionss such as:
- tf.constant_initializer(some_value, dtype=tf.float32, verify_shape=False) - initialize all entries to some value
- tf.ones_initializer() - initialize all entries to 1
- tf.random_normal_initializer(mean=0.0, stddev=1.0, seed=None, dtype=tf.float32) - initialized all entries according to values randomly chosen from a normal distribution.
- tf.random_uniform_initializer(minval=0, maxval=None, seed=None, dtype=tf.float32) - initialized all entries according to values randomly chosen from a uniform distribution.

Building a Perceptron Layer

We can build a function that outputs a Tensor object corresponding to computing a layer of perceptrons using Variables for weights and biases, and placeholders for inputs as follows:

import tensorflow as tf
def perceptron(weights, inputs, biases, activation):
    print(tf.shape(weights))
    print(tf.shape(inputs))
    nodes = tf.matmul(weights, inputs) + biases
    return activation(nodes)

def step(nodes):
    return tf.ceil(tf.clip_by_value(nodes, 0, 1))

x = tf.placeholder(tf.float32, shape=(2,1))
uniform_init = tf.random_uniform_initializer(0, 1) 
W1 = tf.get_variable("W1", shape=(2,2), initializer = uniform_init)
b1 = tf.get_variable("b1", shape=(2,1), initializer = uniform_init)
my_layer1 = perceptron(W1, x, b1, step)
session = tf.Session()
init = tf.global_variables_initializer()
session.run(init)
session.run(my_layer1, {x:[[1],[1]]})
#outputs
#array([[ 1.],
#       [ 1.]], dtype=float32)

If I wanted to use my_layer1 as inputs to an additional perceptron layer, I could do (assuming I hadn't done the session lines above first):

W2 = tf.get_variable("W2", shape=(2,2), initializer = uniform_init)
b2 = tf.get_variable("b2", shape=(2,1), initializer = uniform_init)
my_layer2 = perceptron(W2, my_layer1, b2, step)

In-Class Exercise

Recall the two layer network we created for `PAR_3` we created for the Sep 20 In-Class Exercise.
Implement this network in TensorFlow.
Post your solution code to the Oct 25 In-Class Exercise Thread.

The nn and layers Submodules

Rather than write your own code to create a Neural Net layer, you can instead make use of the pre-built layers in the tensorflow.nn submodule.
For example, if I wanted to make a softmax layer using the nodes from my_layer2 I could do:
```
my_layer3 = tf.nn.softmax(my_layer2)
```
The nn has a wide-variety of layer types such xw_plus_b, relu, relu_layer, max_pool, raw_rnn, various convolution layer types, etc.
Another modules with a collection of functions to build layers is the tensorflow.layers submodule.

For example, we could from my_layer2 also create a dense layer with 4 outputs with step as our activation using:

tf.layers.dense(my_layer2,4, step)
#
# or we could add a dense layer to the inputs
x = tf.placeholder(tf.float32, shape=(2,1))
tf.layers.dense(x,4, step)
#outputs
#<tf.Tensor 'dense/Ceil:0' shape=(2, 4) dtype=float32>

TensorFlow Sessions

Sessions are used to specify the environment under which Tensor objects are evaluate and Operator are executed.

After creating a Session, we have used the run command to control the current state of the session:

init = tf.global_variables_initializer()
session = tf.Session()
session.run(init) #set the function to be used to initialize variables
session.run(W) # compute the value of Tensor Object W according to the
   # current environment

The Session constructor actually takes three arguments: a target, the execution engine should use if different from current process (can use to do distributed computing); a graph, the session will execute; config options for the Session.
One configuration option that is useful, is to know where TensorFlow is going to run our graphs: the CPU or a GPU.
We can get TensorFlow to log this information using:
```
session = tf.Session(config=tf.ConfigProto(log_device_placement=True))
```
If this says: Device mapping: no known devices, then for sure we are not using the GPU.
If TensorFlow knows of GPUs, then after the above when run() is called, it will also output where placeholders, constants, etc are mapped.
To get a list of devices the session is aware of, we can do:
```
devices = session.list_devices()
```

Saving and Restoring the Results of Training

Recall last day, we built an optimizer for training:

epsilon = 0.01 # the learning rate
optimizer = tf.train.GradientDescentOptimizer(epsilon)

We then created a loss function, initialized some training data, and trained:

loss = tf.reduce_sum(tf.square(linear_model - y))
train = optimizer.minimize(loss)
x_train = [1, 2, 3, 4]
y_train = [0, -1, -2, -3]
init = tf.global_variables_initializer()
session = tf.Session()
session.run(init)
for i in range(1000):
    session.run(train, {x: x_train, y: y_train})

If we want to save the current state of the session after training, we can do:
```
saver = tf.train.Saver()
saver.save(session, "my_trained_model.ckpt")
```

If at some later date we want to restore this session, we can do:

session = tf.Session()
saver = tf.train.Saver()
saver.restore(session, "my_trained_model.ckpt")

That's it for our first pass of TensorFlow.
For this first homework with it you can do reading and writing of test data directly using python rather than use TensorFlow Dataset manipulaton classes.

Regularization for Deep Learning

We would like our trained neural nets to perform well not only on the training data, but also on new inputs.
Strategies to reduce the test data (generalization) error are known collectively as regularization.
Regularization strategies might involve putting constraints or penalties on parameter (weight) values.
These might improve training performance, but might also be used to encode prior information we have about what the trained values should look like.

Parameter Norm Penalties

Parameter Norm Penalties are a popular technique for regularization that comes from linear regression in statistics.
Suppose we have a loss function: `J(vec{theta}, vec{mathbb(x)}, mathbb{y})`.
We create a new loss function where we add on to this a function that increase with the weights: `\tilde{J}(vec{theta}, vec{mathbb(x)}, mathbb{y}) = J(vec{theta}, vec{mathbb(x)}, mathbb{y}) + \alpha Omega(vec{theta})`.
`alpha` in the above is a hyperparameter controlling the amount of regularization.
Some popular choices for `Omega` are:
- `1/2||vec{theta}||_2^2` - using this is called `L_2`-parameter regularization also known as ridge regression or Tikhonov regularization.
- `||vec{theta}||_1` - using this is called `L_1`-parameter regularization.
In both cases, using these regularizations will make the resulting loss function larger as the length of the weight vector gets larger.
When all the components of the weight vector all small, the regularization term will tend to be small compared to the loss function.
So we are favoring small weight solutions to our training problem. This will tend to avoid overfitting from large weights with weird cancellations.
`L_1`-parameter regularization tends to result in more of the weights getting 0 values after training. I.e., the weight vector becomes more sparse.
On the other hand, the behavior of this kind of regularization is harder to analyse.

More Tensorflow, Regularization

Outline