Functions

The keyword def is used to create functions in Python:

def smaller_value( a, b):
   if a < b: return a # notice single line if can be same line
   else: return b

Function invocation is similar to most languages:
```
print(smaller_value(8, 4))
```

Functions can be recursive and are allowed to take as arguments and return tuples, lists, etc:

def reverse_list(list):
    if list == []: return []
    return reverse_list(list[1:]) + [list[0]]
print(reverse_list([1, 2, 3, 4])) # prints [4, 3, 2, 1]

def divide(a,b):
   q = a // b
   r = a - q*b
   return (q, r)
quotient, remainder = divide(2373, 16)

We can also supply default values to functions:

def expify(a, b=2):
   return b**a #if don't give a second argument b will be 2

Function Scoping, Functions as Variables

By default variables used in a function definition are local.

i = 5
def printi():
   i=4
   print(i)
printi() # outputs 4
print(i) #outputs 5
#note without the i=4 assignment would get i=5

To be able to modify a variable in global scope we must use the keyword global:
```
def assign_i():
  global i
  i=3
assign_i()
print i #now get 3
```
Function names can be treated as references to code in memory, so we can assign them to variables. i.e., We have function pointers and functions can be passed to other functions, etc:
```
a = printi
a() # prints 4
```

Python also supports anonymous function definitions, functions as arguments, and functions returning functions:

Consider:
def f(x): return x**3
f(10) #returns 1000

g = lambda x: x**3 #notice don't use return with lambda
g(10) #returns 1000 # lambda only works if  after : have an expression

#we can supply a function as arguments to other functions
def a(x):
    if x >= 0: return 1
    else: return 0;
def threshold (w, x, activation) :
    l = len(x)
    inner = 0;
    for i in range(0, l):
       inner += w[i] * x[i]
    return activation(inner)
w = [1, 2]
x = [1, 0]
threshold (w, x, a) # returns 1

# we can also return functions from functions
def make_adder (n): return lambda x: x + n
f = make_adder(2)
g = make_adder(6)
print (f(42), g(42))

Anonymous function are often used to write "one-off" functions like callback handlers in GUI's and things.

Objects and Classes

We have already seen that lists, dicts, etc are objects and we can invoke methods on them much as we do in other languages. For instance, my_list.append("hello")

Given an object we can see its methods by calling dir(my_obj). For example, a=[]; dir(a); gives:

['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', 
'__delslice__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', 
'__getitem__', '__getslice__', '__gt__', '__hash__', '__iadd__', '__imul__', 
'__init__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', 
'__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', 
'__setattr__', '__setitem__', '__setslice__', '__sizeof__', '__str__', 
'__subclasshook__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 
'remove', 'reverse', 'sort']

You can see append in the list. Special methods such as operator overloads have '__' around them. For example, __add__ in the above corresponds to the + operator. We can actually use the + operator in a cumbersome way as: a.__add__([1, 2]).
If you were writing your own class overriding __add__ would give you operator overloading of "+" for operations of the form my_obj + something_else. To override something_else + my_obj you would need to implement __radd__. To override += use overrider __iadd__, etc.
__init__ represents the constructor.

To define new classes we using the keyword class. For example,

class Stack(object): #this says stack inherits from object
    a_class_variable = 5 # this var behaves like a Java static var
    def __init__(self): #self = this in Java
        self.stack = [] #now stack is a field variable of Stack
        #in general using self.field_var is how we declare and 
        #instantiate a instance variable
    def push(self, object): #the first argument of any method 
        self.stack.append(object) # is the object itself
    def pop(self):
        return self.stack.pop()
    @property #properties are computed attributes
    def length(self):
        return len(self.stack) 
    #(where an attribute is like a field) of a class
    # Can use dot notation with or without parentheses to invoke
    #@property is an example of a Python decorator which is a syntactic
    #which says this function object immediately after its definition should be
    #passed to the property() function to get additional features

To instantiate an instance of this class and use it one could do:

my_instance = Stack()
my_instance.push("hello")
print(my_instance.length)
print(isinstance(my_instance, object)) #returns True
print(issubclass(Stack, object)) #returns True
#type(my_instance) returns something containing the word Stack
...
#etc

In-Class Exercise

Define a Python class Queue extending object that implements a (first-in first out) queue.
Write a couple lines showing you know how to use it.
Please post your solution to the Sep 15 In-Class Exercise Thread.

Inheritance, static methods, abstract classes.

Stack above is almost identical to the built-in list object except for the push method. We could have instead inherited from list using:

class Stack(list):
    def push(self, object):
        #could refer to parent by using syntax list.some_method_of_list
        #or use super(list, self).some_method_of_list
        self.append(object)

Python supports multiple inheritance: class MyClass(A, B, ...):
We can use the super mechanism to distinguish between parent members. Python also uses name mangling on methods beginning with "__". The method __foo in class A would be mangled to _A__foo and so would be different from a __foo in class B. This is sometimes used to fake private members of classes.

Static methods of classes can be created and used using the syntax:

class MyClass:
   @staticmethod
   def my_method():
      #some code

MyClass.my_method() #similar to Java

Similarly, abstract methods and properties can be declared as:

#load in abstract class module
from abc import ABCMeta, abstractmethod, abstractproperty
class MyClass:
    __metaclass__ = ABCMeta # a metaclass is a class object that knows how to
                            # create other class objects. The default metaclass
                            # is type (Python 3, types.ClassType in Python 2). 
                            # Here we are assigning ABCMeta
                            # to be used rather than type
                            # In Python 3, write MyClass(metaclass=ABCmeta)
    @abstractmethod
    def my_abstract_method(self):
       pass
    @abstractproperty
    def my_abstract_property(self):
       pass

Exceptions

Like Java, Python has a syntax for exception handling, try-except-finally statements, which have the following syntax:

    try:
        statement_block_0
    except SomeError1 as error_name1:
        statement_block_1 #executed if a SomeError1 occurs
    except SomeError2 as error_name2:
        statement_block_2
    ...
    finally:
        statement_block_n # always gets executed

For example,

    try:
      f = open("file.txt", "r")
    except IOError as e:
      print(e)

To signal an exception you use the raise keyword:

    raise RuntimeError("Something bad just happened")

You define new exceptions by extending exception:

class MyException(exception): pass

#now could use as:
raise MyException("Whoa! A MyException occurred!")

#more control can be had by overriding __init__
#here we define an exception taking two arguments
class MyException2(exception):
    def __init__(self, errno, msg):
        self.args = (errno, msg)
        self.errno = errno
        self.errmsg = msg

raise MyException2(403, "Access Forbidden")

An exception block for FooError will also respond to any subclasses of FooError

Modules

Typically, when you code larger projects you split them into several files.
If you want to use code in one file in another file in Python you use import.
For example, in one file div.py you might have a collection of functions about division. This would be your module.

To use this module in another file, you could put:

import div #notice not div.py
a, b = div.divide(198, 15) #notice function in div.py have to be prefixed with div.

What if we want to use a different prefix then div? We could do:
```
import div as foo #now foo is the prefix
```

What if we don't want to keep writing div.some_function ?

from div import divide #from div import *; would import all functions
print(divide(198, 15))

Modules can be bundled together in so-called packages. Basically, if you put all the files of a module bob in a directory, foo. Then foo would act as a package name and we'd refer to the module bob as foo.bob. To work as a package, foo has to have a special file __init__.py, which may be empty, used to mark the folder as a package. I won't say more about packages unless we really need it.

Documentation Strings and Help

The first statement of a module, class or function definition can be a string called a documentation string.

For example,

def fact(n):
    "This function computes a factorial" #can use triple quoted strings
    if(n <= 1): return 1
    else: return n * fact(n - 1)

The documentation string of such an object is associated with its __doc__ property which can be printed, etc:
```
print(fact.__doc__)
```
Python has the built-in function help() which can be run from Python in interactive mode to get information about modules.
For example, I could put the above fact code into a file test.py. Then in interpretative mode type:
```
import test
help(test.fact)
```
This would go to a screen that prints the documentation string.
Python also comes with a pydoc command that can also give documentation information. For example, at a Unix prompt (not in Python interpretative mode), I could type:
```
pydoc test.fact
```
and get the same result as help(test.fact) before

What Single Perceptrons Cannot Do

For now, let's assume the activation function used by our perceptrons is a step function.
Its relatively easy to see a single perceptron cannot compute certain functions.
A single perceptron outputs 1 when we view input `vec{x}` as a point it is in an upper halfspace, that is above a line, plane, or hyperplane, and 0 otherwise.
Consider the two input function `phi(x,y) = { (1 mbox( if x=y)), (0 mbox( otherwise.)) :}`
We can plot this function as:

0 1

1 0

from which visually we can see there is no single line in the plane that separates the 1's from the 0's.

Single Perceptrons Don't Do Equality - Proofyized Version

Let's try to see this more formally...
Given a predicate `psi(\vec{x})`, let `|~psi(\vec{x})~|` denote the function which is 1 when the predicate is true, and 0 otherwise.
Suppose `phi(x,y) = |~ ax + by \geq theta ~|` for some `a`, `b`, `theta` real numbers.
Then:
`phi(1, 0) = 0 => a < theta`
`phi(0, 1) = 0 => b < theta`
`phi(1, 1) = 1 => a+ b \geq theta`
`phi(0, 0) = 1 => 0 \geq theta`
Adding the first two conditions gives:
`a+b < 2theta`.
Combine with the third condition gives:
`theta < 2theta`.
Hence, `theta` must be positive.
But as by the fourth condition `theta` is at most `0`, it can't be positive. Therefore, `phi` can't be computed by a perceptron. Q.E.D.

Two-Layer Networks

In the case of `phi(x,y)`, one can imagine drawing a line to either side of the diagonal and defining the 1 region to be the intersection of half-spaces given by these lines.
This idea could be used to give a two layer network for equality.
In it, both the perceptron at the top and in the bottom layers depend on all of their inputs.
Notice `phi(x,y)` is a so-called symmetric function: Given an permutation of its inputs, it output is the same. I.e., `phi(x,y) = phi(y,x)`.
As an example, let `PAR_n(x_1, ..., x_n)` be the function which is `1` if the number of its inputs which are `1` is odd. Notice for any permutation of `pi:[n] -> [n]`, we have `PAR_n(x_1, ..., x_n) = PAR_n(x_{pi(1)}, ..., x_{pi(n)})`, so it is symmetric.
In a similar fashion to equality, `PAR_n` can be shown to be computable as intersection and unions of half spaces.
When trying to design a neural network architecture, we generally want to find the simplest architecture with the fewest weights that could possible learn the desired function.
One could ask: Can we simplify the neural network architecture, where we restrict the first level of network to perceptrons that only depend on at most `K` of their inputs being 1?
We will show (Minsky and Papert 1969's result) that if `K < n` where `n` is the number of input bits, `PAR_n` cannot be computed by this simpler architecture.

Example of Two Layer Network for Perceptron

How could we determine the perceptrons needed for a two layer perceptron network computing `PAR_3(x_1, x_2, x_3)`?

Answer. First, notice `PAR_3` is 1 for the boolean values `(1, 0, 0)`, `(0,1,0)`, `(0,0,1)`, and `(1,1,1)`. It is 0 for `(0,0,0)`, `(1, 1, 0)`, `(0,1,1)`, and `(1,0,1)`. We will build a perceptron circuit with four perceptron gates `G_1, ..., G_4`, which computes the function `G_4(G_1(x_1, x_2, x_3),G_2(x_1, x_2, x_3), G_3(x_1, x_2, x_3))`. `G_1` checks if `x+y+z ge 1/2`. The equation `x+y+z = 1/2` determines the plane that cuts the `x`, `y`, and `z` axis at a 1/2, so it is above the point `(0,0,0)`. So if `G_1` is `1` then we know the input isn't `(0,0,0)`, although it might be any of the other elements of the boolean cube. `G_2` checks if `x+y+z le 3/2`. The points `(1, 0, 0)`, `(0,1,0)`, `(0,0,1)` all satisfy this inequality, but `(1, 1, 0)`, `(0,1,1)`, and `(1,0,1)` do not. So `G_1` and `G_2` can both be 1 on the boolean cube only for the points `(1, 0, 0)`, `(0,1,0)`, `(0,0,1)`. We let `G_3` check if `x+y+z ge 2.5`. The only point on the boolean cube which satisfies this inequality is the point `(1,1,1)`. Notice if either `G_2` or `G_3` is 1, the other is 0. Finally, we let `G_4` check `Out_{G_1} + Out_{G_2} + 2Out_{G_3} ge 2`. Since `G_2` and `G_3` can't simultaneously be `1`, the only way for `G_4` to output `1` is if either `Out_{G_1}` and `Out_{G_2}` are both 1, or `Out_{G_1}` and `Out_{G_3}` are both 1. The first case corresponds only to `(1, 0, 0)`, `(0,1,0)`, `(0,0,1)` on the boolean cube, the latter to `(1,1,1)`.

Simplifying Symmetric Perceptrons

Given a boolean vector `vec{x}`, let `#(vec{x})` be the function which returns the number of 1 bits in `vec{x}`.

Lemma. Suppose a symmetric function `f(vec{x})` is computed as `|~ vec{w}\cdot vec{x} \geq theta ~|`, where `vec{w} in RR^n` and `theta in RR`. Then there exists a `gamma in RR` such that `f(vec{x}) = |~ gamma cdot #(vec{x}) \geq 1 ~|` for all `vec{x} in {0,1}^n`.

Proof. Since `f` is symmetric either all of the following `n` inequalities hold, or none of them do:
\begin{eqnarray*} w_1x_1 + w_2x_2 + w_3x_3 + \cdots + w_{n-1} x_{n-1} + w_n x_n &\geq& \theta\\ w_1x_2 + w_2x_3 + w_3x_4 + \cdots + w_{n-1} x_{n} + w_n x_1 &\geq& \theta\\ ...&&\\ w_1x_{n-1} + w_2x_n + w_3x_1 \cdots + w_{n-1} x_{n-3} + w_n x_{n-2} &\geq& \theta`\\ w_1x_{n} + w_2x_1 + w_3x_2 \cdots + w_{n-1} x_{n-2} + w_n x_{n-1} &\geq& \theta \end{eqnarray*} Adding these equations and grouping by the `x_i`'s implies `f(vec{x}) = 1` iff
`(sum_{i=1}^n w_i) x_1 + (sum_{i=1}^n w_i) x_2 + cdots + (sum_{i=1}^n w_i) x_n \geq n \cdot theta`.
Thus, `f(vec{x}) = 1` iff `(sum_{i=1}^n w_i)(sum_{i=1}^n x_i) \geq n\cdot theta` and as `#(\vec{x}) = (sum_{i=1}^n x_i)`, dividing both sides by `n \cdot theta` and defining `gamma = 1/(n\cdot theta) (sum_{i=1}^n w_i)` gives the result. Q.E.D.

Finish First Pass Python - Go/No-Go Perceptron Results

Outline

Introduction