Outline

• Finish Compilation Overview
• In-Class Exercise

Introduction

• On Monday, we went over different steps in and kinds of compilation such as preprocessing, linking, post-compilation assembly, source-to-source translation, etc.
• We then gave a diagram showing the basic phases of compilation and went over the first two stages: lexical analysis (scanning for tokens), and syntax analysis (building a parse tree using a rules from a context free grammar).
• Today, we begin by giving an overview of the remaining phases in compilation beginning with semantic analysis.

Semantic Analysis and Symbol Tables

• Semantic analysis is the discovery of meaning in a program. I.e., what does the program say to do.
• To do this the semantic analyzer needs to recognize and track values of identifiers used by the program. I.e., so things like variables retain their types between usages and have slots for their values when code is generated.
• Information about the identifiers discovered in analyzing a program is usually maintained in a symbol table.
• Using a symbol table, the compiler can enforce a large variety of rules. For example for a C program we might enforce:
  • Every identifier is declared before it is used.
  • No identifier is used in an inappropriate context (calling an integer as a subroutine, adding a string to an integer, referencing a field of the wrong type of struct, etc.).
  • Subroutine calls provide the correct number and types of arguments. Labels on the arms of a switch statement are distinct constants.
  • Any function with a non-void return type returns a value explicitly.

Runtime versus Compile-time semantic rules

• Semantic rules that can be checked at compile time, such as those of the previous slide, give a language its static semantics.
• Some rules cannot be check until the program is actually running. These rules give a programming language its dynamic semantics.
• Some examples of possible dynamic semantic rules are:
  • Variables are never used in an expression unless they have been given a value.
  • Pointers are never dereferenced unless they refer to a valid object.
  • Array subscript expressions lie within the bounds of the array.
  • Arithmetic operations do not overflow.
• When the compiler cannot do the check at compile time, it often outputs code to do the dynamic check along with the code to actually run the program.
• You can think of try{}catch(){} blocks of Java as a language construct to facilitate the generation of this runtime code.

Abstract Syntax Trees

• A parse tree (aka concrete syntax tree) shows how a sequence of tokens can be derived from the rules of a context-free grammar.
• As part of checking static semantic rules, the semantic analyzer typically transforms the parse tree into an abstract syntax tree (AST) by removing non-code generating "artificial nodes" in the tree's interior and annotating the remaining nodes with information such as pointers from identifiers to their symbol table entries.
• Many interpreters use the abstract/annotated syntax tree to represent the running program and execution amounts to tree traversal.
• For compilers, the AST constitutes an intermediate form that is passed from the compiler front-end to the back-end. Sometimes in addition to creating an AST, the compiler-front end does an additional tree traversal to make a control-flow graph whose nodes resemble fragments of assembly language for an idealized machine.

Target Code Generation

• The code generation phase of a compiler translates the intermediate form of the previous slide into the target language.
• To generate assembly or machine language the code generator traverses the intermediate representation, generating LOAD's and STORE's for variable references, interspersing these with appropriate arithmetic operators, tests, and branches.
• The above shows the code generated by a compiler used by the book's author for teaching purposes.
• Often the symbol table is stored along with the program so as to facilitate symbolic debuggers. This would be included in a nonexecutable part of the target code.

Code Improvement

• In addition to code generation, the compiler might have an optional additional phase where it tries to improve the output machine code.
• This can happen prior to code generation by doing machine independent manipulation of the AST.
• For example, one might be able to verify that some store operations are unnecessary as the relevant variables can be kept in registers for the duration of a loop. This might be checked in the AST by looking if we need to call any subroutines during the loop.
• Some optimization are done after initial code generation and may or may not be machine dependent.

Syntax in Detail

• That concludes our initial pass at how compilers work.
• We now return to our study of programming languages by looking in more detail at the tools used to specify the syntax of a language, and at the algorithms needed for performing lexical and syntactic analysis.
• To specify syntax, we need techniques that allow us to specify tokens and to specify rules used by the parser.
• We'll use regular expressions for tokens.
• For the rules used by the parser, we will use Backus-Naor notation for context-free grammars.

Tokens and Regular Expressions

• Tokens are the shortest strings of characters in a programming language with individual meaning.
• Some example kinds of tokens include: keywords, identifiers, symbols, and constants of various types.
• Symbols like {, }, +, -, ++,-- in Java or C are also tokens.
• The most common notation for specifying the tokens of a programming language is using regular expression.
• This notation is simple, expressive, and there are linear time algorithms for recognizing when a string matches a regular expression.
• A regular expression is one of the following:
  1. A character
  2. The empty string, denoted `epsilon`.
  3. Two regular expressions next to each other, meaning any string generated by the first one followed by (concatenated with) any string generated by the second one.
  4. Two regular expressions separated by a vertical bar ( | ), meaning any string generated by the first one or any string generated by the second one
  5. A regular expression followed by a Kleene star (`star`), meaning the concatenation of zero or more strings generated by the expression in front of the star.

Regular Expression Example

• When writing regular expressions, people often use abbreviations for subexpressions of a regular expression.
• For example,
  `mbox{number} -> mbox{integer}| mbox{real}`
  says that `mbox{number}` is an abbreviation for the regular expression `mbox{integer}| mbox{real}`.
• As a larger example, we could expand the following out into a single regular expression for numeric constants in some language (a number):
  `mbox{number} -> mbox{integer} | mbox{real}`
  `mbox{integer} -> mbox{digit}\ mbox{digit} star`
  `mbox{real} -> mbox{integer}\ mbox{exponent} | mbox{decimal} ( mbox{exponent} | epsilon )`
  `mbox{decimal} -> mbox{digit} star ( . mbox{digit} | mbox{digit} . ) mbox{digit} star`
  `mbox{exponent} -> (e|E)(+|-|epsilon) mbox{integer}`
  `mbox{digit} -> 0|1|2|3|4|5|6|7|8|9`
  
• As an example of using the above regular expression... 1234 matches the number regular expression as by the first line an integer can be a number, and then by the second line an integer is one or more digits, and finally by the last line each of 1,2,3,4 are digits.

In-Class Exercise

• Expand out the definition of integer from the previous slide as a single regular expression.
• Show that 1E-4 is a number.
• Please post your solution to the Sep 1 In-Class Exercise Thread.

More on Tokens and Regular Expressions

• Tokens in some programming languages (SQL) are allowed to be case insensitive, but for other languages (Java) are case sensitive.
• Some languages allow more general, non-ASCII characters (say Unicode) for identifiers.
• White space is treated as important in some languages (Python), but not in others.
• Many programming languages have restrictions on the lengths of tokens.
• As regular expressions are useful for general string processing, not just compiler construction, many languages have a built in facility for handling regular expressions. For example, Perl, PHP, Javascript, Python, etc.
• In these language, the simple regular expression notation we have given has been augmented to make it easier to handle the above cases.
• For example, /reg_exp/i might be used to say match a regular expression in a case insensitive way.
• Even with these extensions the underlying algorithm for recognizing regular expression are generally linear time (some regex extension allowed by Perl break this guarantee).
• We will talk about these next day.
• We conclude today with a quick intro to C and Unix shell...

Introduction to C and Unix Environment

• We'll soon talk about C based tools for writing lexers and parsers for programming languages. In order for these talks to make sense, we need to make sure everyone is on the same page as far as programming in C.
• Since it was first created at Bell Labs along with Unix in the early 70's, C has been ported to many OS's and environment.
• The particular OS/platform in which you program can have a significant effect on the kind of code you write. The same is true of Java. Although, the basic language is the same, the available libraries and stuff are different if you are using Java SE versus Android for instance.
• We'll focus in this class on using C in a Unix environment, and we will do things from the command line.

Unix Environment Remarks

• If you are using a Mac or Linux, there is nothing much involved in getting a command line environment -- just launch a terminal.
• If you are using a Windows machine, it is recommended that you install Cygwin and make sure to also install the devel library. Then you can launch the Cygwin terminal and you should be able to do the same commands, as Mac and Linux users.
• I am most familiar with csh/tcsh as my Unix shell. If you prefer bash/zsh/etc that is okay, just try to write your make files so they don't rely on shell specific code.

More Unix

For those of you unfamiliar with Unix we next look at some of the most important shell concepts/commands to get started with:

• man -- the Unix manual.

  man ls -- would give the Unix manual entry for the ls command.
  man -k some_keyword -- lists all commands whose description has that keyword.
  

• Paths are used to refer to documents somewhere on the computer. All directories are subdirectory of / . A path might be absolute and look like /foo/bar/ which refers to the bar subdirectory of the foo directory of the root dir. Or relative foo/bar -- bar subdir of foo dir of current directory. There are also some abbreviations for special paths: . -- current dir, .. -- parent dir, ~ -- user's dir, ~bob -- user bob's dir.

Still more Unix

• ls path -- list contents of directory given by path.
• cd path -- switches the current directory to be path
• rm filename -- deletes filename. Can have flags like: rm -rf foo -- force deletes foo directory and all its children
• cp f1 f2 -- copies file f1 to f2. cp -r dir1 dir2 copies contents of dir1 to dir2
• mv f1 f2 -- moves/renames f1 to f2.
• set -- lists your environment settings. This can be particularly useful for checking if your path environment variable is set correctly. The path variable says which directories to look for the executables of the currently typed command.
• You can use whatever editors you like for editing your programs. Unix comes with a few built-in editors such as vi, pico (nano), emacs, but you can feel free to use a GUI editor such as Atom or Visual Studio Express, Xcode, etc.

A Basic C Program: hello.c

#include <stdio.h> //for printf statement
int main()
{
   printf("Hello World\n");
   return 0;
}

To compile and run this program:

gcc hello.c -o hello
./hello 

Remarks on hello.c

• Notice the main function is the main entry into a C program.
• #include is a preprocessor directive which says include the contents of the given header file at that location in the current document.
• stdio.h has the function prototype for printf, so that we can print to screen.

Similarities b/w Java and C

• As of C99 you can use both /* */ or // for comments. ANSI C only had /* */ .
• Variable declarations are pretty much the same and similar types are available:

  int a,b; // declares two int's a and b.

• Have types float, double, char, etc.
• Array declarations also work similarly.

  int a[10]; //don't use a variable rather than 10 in ANSI C.

• Variable assignment also works like you would expect: a=5; a++; a--; ++a; --a; a+=100; //etc if-statements, while statements, for statements, switch, etc. have the same syntax as Java.

Important differences

You don't have a class construct. You can pretend --if it helps -- as if everything is declared in the same class. Although, these rules were relaxed somewhat in C99 from ANSI C, you should have all your variable declarations at the start of your functions:

void foo()
{
	int a=1;
    printf("%d", a);
    int b; // ANSI C's head explodes
}

Similarly, you either need to have a function prototype, or the function itself before the first place you use it -- See next slide.

Function examples

//the following is okay:
char foo();
void bar()
{
     printf("%c", foo());
}
char foo( )
{
     return 'a';
}