Outline

• Preprocessors, Linkers, Assemblers
• Programming Environments
• Quiz
• How compilation works

Introduction

• Last week, we defined a programming language as a a notation system for describing computation in both a machine readable and human readable format.
• We then explained what each of the three parts of that definition meant.
• We next looked at program language design and what makes a language successful.
• Then we discussed the important kinds of programming languages: functional, dataflow, logic-based, von Neumann, Object-oriented, and scripting.
• At the end of last day, we talked about what compilers do, what interpreters do, and what virtual machines are.
• We begin today by looking more at the details of interpretation and compilation.

Preprocessors

• Many interpreters and compilers employ an initial translator called a preprocessor before interpretation or compilation begins.
• The preprocessor might remove comments and white space.
• It might group characters into tokens such as keywords, identifiers, numbers, and symbols.
• It might expand certain abbreviations such as #define's for C.
• Finally, it may identify higher-level syntactic structures such as loops and subroutines.
• For compilers, the preprocessor might be used to specify what portions of the code should be compiled and which should be skipped (conditional compilation). This allows for different version of a program to be built from the same source.
• One reason for an interpreter to use preprocessing is to produce an intermediate form that mirrors the structure of the source, but can be interpreted more efficiently.

Linking, Loading, Overlays

• Often code in a program relies on functions that have been previously written and grouped together called a library.
• A program called a linker is used to merge your compiled program with previous compiled libraries. (The ld function in *nix does this.)
• If you compile a C program with a printf, you can think of the compiler as leaving a placeholder in a table for where to go to execute the printf. When the compiled stdio library is linked to your code the value in this table is filled in.
• To run a program after it is compiled, the OS uses a loader program to read in the compiled code, copy the command line arguments onto the stack, initialize registers, and call the starting function of the program.
• The loader might also be responsible for loading libraries dynamically at run time (dynamically linked library (DLL)), as opposed to statically at compile time.
• In early computer systems and some embedded system, one often doesn't have enough RAM to read an entire compiled program into memory. Code in such settings is often split into a graph where certain groups of functions that are often executed together are read into memory executed and then, depending on the outcome an edge in the graph, would be followed to the next group of functions to be read and executed. Such a graph is called an overlay.

  Example OS/360 Overlay
   INCLUDE SYSLIB(MOD1)
   INCLUDE SYSLIB(MOD2)
   OVERLAY A
     INCLUDE SYSLIB(MOD3)
       OVERLAY AA
         INCLUDE SYSLIB(MOD4)
         INCLUDE SYSLIB(MOD5)
       OVERLAY AB
          INCLUDE SYSLIB(MOD6)
   OVERLAY B
      INCLUDE SYSLIB(MOD7)
  

Quiz

Which of the following statements is true?

1. It is impossible to create a programming language that is not Turing complete.
2. Delaying decisions about program implementation until run time is known as late binding.
3. von Neumann languages employ a computational model based on the recursive definition of functions.

Post-Compilation Assembly

• Many compilers generate assembly language instead of machine language.
• This can make it easier for debugging as assembler is easier to read.
• This also isolates the compiler from changes in the format of machine language files that some times occur with new releases of the OS.
• In the latter case, only the assembler must be changed, and it can be shared by many compilers.

Source-to-Source Translation and Bootstrapping

• When first creating a new language or extending an existing one, it is often easier to write a compiler that starts from the extended language and maps to the unextended/some other existing high level language.
• This is called source-to-source translation and is illustrated by the left diagram above.
• For example, this is how the original C++ compiler from AT&T worked.
• This compiler was a true compiler in the sense that it performed a complete analysis of the syntax of the C++ program and generated most of the error messages a programmer would see prior to running the program.
• Since it did analysis rather than just perform simple pattern matching transformations, this can be considered compilation rather than preprocessing.
• Some compilers are self-hosting -- they are written in the language they compile. For example, Ada compilers in Ada or C compilers written in C.
• The process of how such a compiler can be compiled is called bootstrapping and is illustrated by the Pascal diagram to the right above. The starting place is one writes a P-Code interpreter (let's callit PI) by hand, then a Pascal to P-Code compiler (let's call is PPC) in P-Code, then uses this to compile a Pascal to machine language compiler (PMLC).
• The last part of the above is: we run PPC on the source PMLC to get a P-code program PMLCP. We then run PMLCP using the P-code interpreter on the input source PMLC. This outputs a machine code compilation of PMLC.

Just-in-time Compilation

• Some programming system deliberately delay compilation until the last possible moment.
• For example, Java defines a machine independent, intermediate Java bytecode language.
• Programs compiled to this bytecode can be easily distributed over the internet and run on any system with a Java runtime.
• Originally, bytecode was interpreted.
• Modern Java implementation use a just-in-time compiler to quickly translate the bytecode into machine language immediately before each execution of the program.
• C# takes a similar JIT approach with its Common Intermediate Language (CIL).

Microcode

• For some machines (typically designed before the mid-1980s), the assembly level instruction set is not actually implemented in hardware, but in fact runs on an interpreter.
• The interpreter is written using a low-level instruction set called microcode that is stored in read-only memory and executed by hardware.

Programming Environments

• Compilers and interpreters do not exist in isolation.
• Programmers are assisted by a host of other tools: Assemblers, debuggers, preprocessors, linkers, editors, etc.
• Editors might be augmented with cross-referencing facilities that allow the programmer to find the point at which an object is defined and where it is used.
• Style checkers and converters (for example, lint) can be used to enforce syntactic or semantic conventions more tightly that those enforced by the compiler.
• Configuration management tools help track dependencies among different versions of different modules of large software systems.
• Profilers and other performance analysis tools might be used to identify pieces of a program that consume the most resources.
• In older system, each of the above facilities might be a separate command line tool.
• More modern program environment collect these tools into a GUI based Integrated Development Environment.

How compilation works

• Let's look briefly at how a compiler works.
• Compilation is typically broken into series of phases as in the diagram above.
• The first few phases, typically together called the front end, serve to figure out the meaning of the source program.
• The last few phases, called the backend, serve to construct an equivalent target program.
• Compilation is often split into passes (originally because a pass would involve scanning through the whole program once).
• Passes typically group together a collection of phases that are serialized with respect to compilation. I.e., a given pass needs to complete before the next one begins.
• Each pass might be handled by a separate program.
• The program for a given pass might be shared between different compilers.
• Let's briefly look at some of the steps listed above...

Lexical and Syntax Analysis

• Consider a C GCD program that gets two int's and computes their gcd:

  int main() {
    int i = getint(), j = getint(); 
    while (i != j) {
      if (i > j) i = i - j;
      else j = j - i;
    }
    putint(i);
  }
  

• Scanning and parsing serve to recognize the structure of the program without regard to its meaning.
• Scanning is sometimes called lexical analysis.
• The scanner reads the sequence of characters in the program 'i', 'n', 't', ' ', 'm', 'a', 'i', 'n' ... and groups them into tokens:
  
• Scanning also typically removes extraneous characters and comments. Tokens might be annotated/tagged with their line number and other diagnostic info for debugging.
• Parsing organizes tokens into a parse tree that represents higher level constructs: statements, expressions, subroutines, and so on.
• The root of this tree is the whole program.

Example Parse Tree

More on Parsing

• The structure of a parse tree relies on a set of potentially recursive rules known as a context-free grammar.
• Each rule has the name of a construct on its left hand side, followed by an arrow or a :=, followed by a possible expansion on the right. Below are some examples:

  iteration-statement := while ( expression ) statement
  statement := compound-statement
  compound-statement := { block-item-list opt }
  block-item-list_opt := block-item-list
  block-item-list_opt := empty-string
  block-item-list := block-item
  block-item-list := block-item-list block-item 
  block-item := declaration
  block-item := statement
  

• As a context-free grammar defines the syntax of the language; parsing is therefore known as syntax-analysis.