Outline

• Insertion Sort as Code
• Insertion Sort Analysis - Worst Case and Average Case
• Orders of Growth

Insertion Sort as Code

• On Monday, we went over Insertion Sort as pseudo-code and I explained the books format for pseudo-code.
• Before we begin today, I want to show that this code is close to something that would work in a modern programming language.
• To see this we can look at the original pseudo-code below and a Python program which uses it as a basis.
• As you can see they are quite similar, the main difference is that we need driver code before we get to the code that actually does insertion sort.

Original Pseudo-Code

1 for j=2 to A.length
2     key = A[j]
3     // Insert A[j] into the sorted sequence A[1..j-1]
4     i = j - 1
5     while i > 0 and A[i] > key
6         A[i+1] = A[i]
7         i = i - 1
8     A[i + 1] = key

Working Python Code

import sys

def insertion_sort(A):
    for j in range(1, len(A)):
        key = A[j]
        # Insert A[j] into the sorted sequence A[0:j-2]
        i = j - 1
        while i >= 0 and A[i] > key:
            A[i + 1] = A[i]
            i = i - 1
        A[i + 1] = key

if len(sys.argv) < 2:
    print "Enter some numbers to sort"
    raise SystemExit(1)
A = sys.argv[1:];
for i in range(0,len(A)):
    try:
        A[i] = int(A[i])
    except ValueError:
        print "Items to sort must be integers"
        raise SystemExit(1)
insertion_sort(A);
print A;

Insertion Sort in Java

• With a little more set up we can write the same code in Java.
• Notice most of the new code is just to set things up so that we can call the sort function.
• After showing the Java code below in class I had below work on modifying it to do `k`-selection. Here is the k-selection code we got in class.

public class InsertionSort
{
    public static void sort(int[] A)
    {
        for(int j = 0; j < A.length; j++)
        {
            int key = A[j];
            // Insert A[j] into the sorted sequence A[0], A[1], ... A[j-1]
            int i = j - 1;
            while(i >= 0 && A[i] > key)
            {
                A[i + 1] = A[i];
                i = i - 1;
            }
            A[i + 1] = key;
        }
    }

    public static void main(String args[])
    {
        if(args.length < 1)
        {
            System.out.println("Enter some numbers to sort");
            System.exit(1);
        }

        int[] A = new int[args.length];
        try
        {
            for(int j = 0; j < args.length; j++)
            {
                A[j] = Integer.parseInt(args[j]);
            }
        }
        catch (NumberFormatException e)
        {
            System.out.println("Items to sort must be integers");
            System.exit(1);
        }
        sort(A);

        String space = "";
        for(int key: A)
        {
            System.out.print(space + key);
            space = " ";
        }
        System.out.print("\n");
    }
}

Analyzing Algorithms

• Analyzing an algorithm means predicting the resources that the algorithm requires:
  • memory
  • communication
  • hardware
  • time
• To analyze an algorithm, we must have a model of the technology on which it will be implemented.
• For this semester, we will assume our algorithms will run on a generic one-core processor called a Random-Access Machine (RAM).
• We will also assume that our algorithms will be implemented as computer programs for such a machine.
• In a class like CS154, we would be very precise in exactly what the instruction set of our machine was, but for this class we will keep things on a more intuitive level.
• We assume it have instructions for basic control (if-else, subroutine call and return), arithmetic (add, subtract, multiply, divide, remainder, floor, ceiling), data movement (load, store, copy), and that each of these operations takes constant time.
• We will assume that our machine supports integer and floating point numbers, but we are not concerned with their precision.
• We will assume a maximum constant amount of space for the word size of our machine.

Measuring the time of an Algorithm

• The time taken by INSERTION-SORT depends on the input.
• Sorting a thousand numbers takes longer than sorting three numbers.
• Insertion sort can also take a different amount of time even if the inputs have the same size, depending on how sorted the inputs already are.
• What we mean by the input size of a problem depends on the problem being studied.
• For many problems such as sorting or computing a discrete Fourier transform, the most natural measure of input size is the number of items in the input.
• For other problems, like multiplying two integers, the best measure of input size is the total number of bits.
• Sometimes the input will consist of two numbers rather than one; or in general, k-tuples or lists instead of single items.
• So when we analyze an algorithm we need to say how we are measuring this size.
• The running time of an algorithm on a particular input is the number of primitive operations (steps or RAM instructions) executed before the algorithm halts.
• We will generally assume that each line of our pseudo-code takes a constant `c` time to execute. I.e., line `i` takes at most `c_i >0` time to execute.

Analyzing INSERTION-SORT -- Best Case

• For each `j=2,3,..., n` where `n=A.text{length}`, let `t_j` denote the number of times the while loop test in line 5 is executed for that value of `j`.
• When a for or while loops exits in the usual way, the test is executed one time more than the loop body.
• We assume comments take no time to execute.
• The above image has a table. The second column is the run time for that line, the third column indicates how many times that line runs. The product of these two things gives us the total time spent on that line. If we sum up the total times spent on each line, we get the total time of the program:
  `T(n) = c_1n + c_2(n-1) + c_4(n-1) +c_5 sum_(j=2)^nt_j + c_6 sum_(j=2)^n(t_j -1) + c_7sum_(j=2)^n(t_j -1) +c_8(n-1)`.
• For INSERTION-SORT, the best case occurs when the array is already sorted. In this case, `t_j =1`, and the best case run time reduces to:
  `T(n) = c_1n + c_2(n-1) + c_4(n-1) +c_5(n-1) + c_8(n-1)`
  `= (c_1 + c_2 +c_4 + c_5 +c_8)n -(c_2+c_4 +c_5+c_8)`
• We can express this as `an+b` for constants `a` and `b` that depend on the statement costs `c_i`. So it is a linear function of `n`.

Worst Case analysis

• If the array is in reverse sorted order -- the worst case result happens. For each `A[j]` we need to compare it with the entire subarray `A[1..j-1]`, so `t_j = j`, for `j=2,3,...,n`.
• Recall `sum_(j=2)^n j = (n(n+1))/2 -1` and `sum_(j=2)^n (j -1) = (n(n-1))/2`.
• So in the worst case the running time is:
  `T(n) = c_1n + c_2(n-1) + c_4(n-1) +c_5((n(n+1))/2 -1) + c_6((n(n-1))/2) + c_7((n(n-1))/2)`
  `+c_8(n-1)`
  `=((c_5+c_6+c_7)/2)n^2 +(c_1 + c_2 +c_4 +(c_5 -c_6 -c_7)/2 +c_8)n - (c_2 +c_4 +c_5 + c_8).`
• We can express this worst case running time as `an^2 + bn +c` for constants `a`, `b`, and `c` that depend of the `c_i`'s. So we say that it is a quadratic function of `n`.

What case should we look at?

• So far we have looked at the best and worst case performance of INSERTION-SORT.
• The best case gives a lower bound on the runtime (it must take at least that long), the worst case gives us an upper bound (it takes at most that amount of time.)
• For this semester we will usually focus on the worst case running time of algorithms. Three reasons for doing so are:
  1. The worst case serves as a upper bound on the runtime so we are guaranteed the algorithm won't take longer.
  2. For some algorithms the worst case is actually also the case that occurs for most instances. For example, in searching a database for a particular piece of information, the searching algorithm's worst case occurs when the information is not present. In some application, search for absent information may be frequent.
  3. The "average case" is often roughly as bad as the worst case. Suppose that we randomly choose `n` numbers and apply insertion sort. How long does it take to determine where in subarray `A[1..j-1]` to insert `A[j]`. On average we will have to look through half of the elements. So on average `t_j` is about `j/2`. So the average-case running time turns out to be a quadratic function of the input size just like the worst-case.
• We will sometimes be able to work out the average case running time. This will typically involve a probabilistic analysis, in which case we can talk about the expected running time of an algorithm.

Orders of Growth

• We simplified the running time of INSERTION-SORT to a function of the form `an^2 +bn+c` for constants `a,b,c`.
• We will frequently want to simplify this even further and only consider the leading term in the formula. In this case, `an^2`, which we will call a rate of growth or order of growth.
• This is because the lower order terms are less important as the instance size grows.
• We will write the high order term as `Theta(n^2)` (pronounced theta of `n^2`) because the constant `a` is less important than the growth `n^2`.
• For large enough inputs a `Theta(n^2)` algorithm will perform better than a `Theta(n^3)` algorithm, etc. We will make this notion more precise next week.