Outline

• Finish up Lexical Analysis
• In-Class Exercise
• Parsing

Introduction

• Recall we were talking about regular expressions as a method that programming language designers can use to specify what constitutes a legal keyword, identifier, symbol, or constants in the language they are designing.
• On Monday, we presented the Thompson 1968/McNaughton-Yamada 1960 construction of a NFA from a regular expression.
• This algorithm runs in time `O(m)` given a length `m` regular regular expression.
• We then showed how given an `m` state NFA for a set of tokens we can check if a length n string `w` matches a token of the NFA in time `O(nm^2)` (this is an old result (mentioned in Hopcroft Ullmann (1968), not sure who due to)).
• Hence, we have fast algorithms for recognizing if a string matches a set of tokens specified by a regular expression.
• Given a regular expression `R` for a set of tokens let S=(a|b ... |z ...) be a regex (regular expression) for the symbols that might occur in a program. Then `S^\starRS^\star` is a regex for strings containing `R` as opposed to exact matching `R`.
• Also, it is not hard to modify our notion of accept state to a finite list of types of accept states: say accept_variable state, accept_keyword_states, etc. So we can have a fast algorithm not only to detect the next token from a string but also the kind of token it is.

Lexical Errors

• Given we have algorithms for scanning for the next token. What happens if the next thing we scan from a program doesn't match any of the tokens from our programming language?
• In that case our program contains a lexical error.
• Lexical errors can be handled by:
  1. Throwing away the current, invalid token;
  2. Skipping forward until a character is found that can legitimately begin a new token;
  3. Restarting the scanning algorithm
  4. Counting on the error-recovery mechanism of the parser to cope with any cases in which the resulting sequence of tokens is not syntactically valid.

Pragmas

• Some programming language allow the program to send directives or hints to the compiler known as pragmas. (C uses #pragma, Java use annotations, C# has attributes).
• As some examples,

  //For C:
  #pragma once // says include a header only once, can use instead of #ifndef..
  #pragma warning (disable : 1337 ) //says not to generate 1337 warnings
  
  //For Java
  @Override //compiler should complain if not overriding something from parent class
  void mySuperMethod() { ... }
  @Readonly List<@Readonly T> list; // list's value should not change
  

• Pragmas are at the border between things that might be handled by the lexer and those handled by the parser.
• Pragmas which act as hints (for example suggest that a register be used to hold a variable) often can appear anywhere in a pragma and can be handle as part of the lexer.
• Pragma which serve as compiler directives such as some of the examples above are better handled by the parser.

Parsing - CFGs

• When we did our overview of compilation, we said the parser's job is to use tokens given to it by the scanner/lexer and build up a parse tree representing statements, expressions, subroutines, and so on of the whole program.
• To specify how a parse tree should be constructed, a programming language specification, usually gives rules (aka productions) for what constitutes a legal statement, expression, etc. using some kind of context free grammar.
• Below is an example of a part of a context-free grammar for English:

  sentence → noun-phrase verb-phrase .
  noun-phrase → article noun
  article → a | the
  noun → girl | dog
  verb-phrase → verb noun-phrase
  verb → sees | pets
  

• A grammar such as the above serve as a generator for a language.
• There should be a derivation sequence for each string in the language.
• For example, the English sentence "the girl pets a dog". Can be generated as:

  sentence → noun-phrase verb-phrase
  →  article noun verb-phrase → the noun verb-phrase
  → the girl verb-phrase → the girl verb noun-phrase
  → the girl pets noun-phrase → the girl pets article noun
  → the girl pets a noun → the girl pets a dog 
  

More on Context Free Grammars

• Our rules distinguish between non-terminals or variables and terminals. A terminal cannot be changed by our rules.
• We often use so-called Backus Naur Form for our rules (BNF). In this notation, non-terminals are written as string between < and >, ::= is used to separate the left and right hand side of rules, and | is used to mean 'or'.
• A rule is context-free, if the left hand side of a rule contains exactly one non-terminal.
• For example, the rules of the last slide are context free, but the rules:

  <sentence> ::= <start-of-sentence> <noun-phrase> <verb-phrase>
  <start-of-sentence> <article> ::= The | A
  <start-of-sentence> ::= empty-string
  

  are not (second rule violates it).
• If a set of rules are all context free, then we have a context free grammar.
• The first rule in our grammar's left side is usually considered a starting non-terminal.
• A derivation begins with the starting non-terminal and at each step replaces in the current string one non-terminal with what it maps to on the right hand side of a rule. The process usually continues until we have a string of only terminals.

Parse Trees

• By using the starting non-terminal as the root and hanging trees off non-terminals with what they were replaced by, it is straightforward to go from a derivation to a parse tree:

        sentence
           |
  noun-phrase verb-phrase
      |           |
  article noun    verb noun-phrase
    |      |        |        |
   the    girl     pets    article noun
                              |      |
                              a     dog
  

In-Class Exercise

• Show the derivation of a second sentence from the English grammar we just gave.
• Give a parse tree for your derivation.
• Please post your solution to the Sep 15 In-Class Exercise Thread.

Parsers

• A parser's job is to be a language recognizer, i.e., to determine if a string could have been generated by a CFG.
• Given a string of length `n`, there are `O(n^3)` algorithms that can determine if a string could have been generated by a CFG. For example, the dynamic programming Cocke-Younger-Kasami (CYK) algorithm (1965,1967) and the Earley Algorithm (1970) can do this.
• However, even for moderate sized programs these algorithms would be much too slow.
• Instead, language designers usually try to specify their language in terms of special subclasses of CFGs, namely LL and LR, for which linear time algorithms are known.
• Here LL stands for left-to-right leftmost derivation, and LR stands for left-to-right rightmost derivation.
• Left-to-right means we scan the program from left to right during parsing (i.e., from the beginning of the program to the end rather than from the end to the beginning).
• A derivation is leftmost (resp. rightmost) if the non-terminal that is expanded at each step is always the leftmost (resp. rightmost) one.

LL and LR Parser

• The above figure illustrates the LL (left hand side) and LR (right hand side) parsing of the same string A, B, C; using the grammar rules in the inset.
• Notice it is the lexical analysis phase that determine A B C are identifiers (id).
• We will consider both LL and LR parsing on Monday.
• As LL is simpler, we will begin with it, however, the class of LR grammars is larger and can express some programming language concepts more easily.
• LL Parsers sometime also called "recursive descent" or "top-down" parsers or "predictive" parsers. This is because they start with the start rule left hand side and predict what it goes to, then predict what its left most non-terminal goes to, etc.
• LR parser can be thought of as shifting tokens on the stack and looking at the symbols on the top of the stack to see if they match the right hand side of a rule, if so they reduce the right hand side of the rule to its left hand -side. So such parsers are sometimes called shift-reduce parsers.
• Each of LL and LR Grammars have several important subclasses SLL and full LL for LL , and SLR and LALR for LR.
• You sometimes see the notations `LL(n)` and `LR(n)` here `n` refers to the number of tokens ahead one must scan to either predict the rule to use, or decide whether to reduce.
• LL(1) parsing was first described by Lewis and Stearns 1968, LR(k) parsing was invented by Knuth 1965.

Recursive Descent Parsing Example

• To illustrate recursive descent parsing, we are going to make use of the following LL(1) grammar for a simple calculator language:

  program ::= stmt_list $$
  stmt_list::=  stmt stmt_list | `epsilon`
  stmt ::=  id := expr | read id | write expr 
  expr ::= term term_tail
  term_tail ::=  add op term term_tail | `epsilon`
  term ::=  factor factor_tail
  factor_tail ::=  mult op factor factor_tail | `epsilon` 
  factor ::= ( expr ) | id | number
  add op ::=  + | -
  mult op ::=  * | /
  

• As example, program in this language might be:

  read A
  read B
  sum := A + B
  write sum
  write sum / 2
  $$
  

Parse Tree for Calculator Example Program

Recursive Descent Example Parser

Remarks on Example Parser

• The main entry point into the parser of the previous slide is program().
• The parser roughly consists of one procedure/non-terminal in the grammar.
• Since the grammar is LL(1), we can look at the current input token before matching it and determine which of potentially several rules should be the rule to match, then we recursively call the procedures fro the right hand side of that rule.