Introduction

Last week, we introduced two techniques for parsing: top-down (LL) and bottom-up (LR).
Recall top down parsing corresponds to leftmost derivations. We begin with the start rule of our grammar's left hand side and predict what it goes to, then predict what its left most non-terminal goes to, etc.
On the other hand, bottom up parsing corresponds to rightmost derivations using our grammar. It begins by shifting tokens on the stack and looking at the symbols on the top of the stack to see if they match the right hand side of a rule, if so they reduce the right hand side of the rule to its left hand -side.
On Wednesday, we gave a context-free grammar for a simple calculator example language, then gave a top-down parser for it.
That parser only needed to look at the next symbol returned from the lexer to determine which rule was being handled as the grammar in question was LL(1).
We begin today by talking about writing LL(1) grammars.

Writing LL(1) Grammars

As we have seen it is not too hard to write a parser if a context free grammar is LL(1), so what problems can a grammar have that prevent it from being LL(1)?
The two most common problems are: left recursion and common prefixes.
A grammar is left recursive if there is a nonterminal A such that `A =>^+ A \alpha` for some derivation from its rules and sequence alpha of terminals and non-terminals. For example, the grammar:
```
id_list ::= id_list_prefix ;
id_list_prefix ::= id_list_prefix , id | id
```
Notice the second rule has id_list_prefix right after the ::=. Because of this a recursive descent parser cannot tell if it should call the id_list_prefix() procedure recursively or just match id.
A grammar suffers from common prefixes when two productions for the same symbol have right hand sides that begin the same. For example:
```
stmt ::= id := expr
stmt ::= id (argument_list)
```
Again, by looking at a single next symbol we cannot choose between them.

Eliminating Left Recursion and Common Prefixes

Both Left Recursion and Common Prefixes can be eliminated mechanically.
For left recursion, we can replace the left recursive rules with right recursive equivalents. For example for the rules of the last slide:
```
id_list ::= id id_list_tail
id_list_tail ::= , id id_list_tail | ;
```
To handle common prefixes, we can use a technique called left factoring as illustrated by how we modify the code from the previous slide:
```
stmt ::= id stmt_list_tail
stmt_list_tail ::= := expr | (argument_list)
```

Ambiguity and the Dangling Else Problem

A language might not suffer from either left recursion or common prefixes, but still not be LL.
One problem might be ambiguity in the grammar. I.e., two different leftmost derivations for the same string.

For example, consider the grammar:

stmt ::= if condition then_clause else_clause | other_stmt 
then_clause ::= then stmt
else_clause ::= else stmt | empty-string

We could parse

if C_1 then if C_2 then S_1 else S_2

as either:

if C_1 then
   if C_2 then S_1
   else S_2

if C_1 then
   if C_2 then S_1
else S_2

Solving the Dangling Else Problem

Call a statement balanced if it has the same number of then's as else's, and unbalanced otherwise.

The following grammar can be parsed bottom up, but not top-down:

stmt ::= balanced_stmt | unbalanced_stmt
balanced_stmt ::= if condition then balanced_stmt else balanced_stmt
    | other_stmt
unbalanced_stmt ::= if condition then stmt
    | if condition then balanced_stmt else unbalanced_stmt

A more common approach to issues of ambiguity in a grammar is to combine the grammar with meta disambiguating rule which says what to do in the case of a conflict between two productions.
For example, we could always choose the first applicable production. So in
```
else_clause ::= else stmt | empty-string
```
we would prefer else stmt over empty-string which would mean we pairs else's with the nearest then.
Another possible fix is to change the grammar to avoid the issue, for example by requiring begin/end blocks around the then and else clauses .

Quiz

Which of the following statements is true?

Whitespace doesn't matter for the make utility.
There exists a regular expression that cannot be converted to an NFA.
LL stands for left-to-right leftmost derivation.

Table-Driven Top-Down Parsing

In a recursive descent parser, the procedure for a given non-terminal has a case statement for each of the possible right hand sides of a rule for that non-terminal. It has match calls for the symbols.
At any given point, during parsing, we consider recursive calls that have yet to occur, we obtain a list of the symbols that the parser expects to see between the given location and the end of the program.
A table driven top-down parser maintains an explicit stack (rather than use the run time stack) containing this list of symbols and uses this stack and parse and production tables to do parsing.
The advantage of this approach is we can build a single driver parser, and to make something that works for a particular language we just need to generate the parse and production tables from the language grammar.
The left hand image above shows an example driver program for such a parser, the right hand side shows an example parse table which could be generated from the grammar for the simple calculator language.

Generating Parse and Production Tables From a Grammar

In order, to figure out which rule to predict based on the current token, we make use of two auxiliary sets:
- FIRST(A) is the set of all tokens that could be the start of an A.
- FOLLOW(A) is the set of all tokens that could come after an A in a valid program.
We next describe two conditions that will ensure we can predict the next rule we are parsing.

More on Predictive Parsing

We extend BNF so that {<non-terminal>} means 0 or more <non-terminal> and [<non-terminal>] means <non-terminal> is optional. This doesn't effect the grammars we can create.
Consider a rule of the form A → α₁| α₂| α₃|...| α_n.
The first condition is that the first symbols of each of the rules must be distinct.

For example, for the grammar:

<factor> ::= (<expr>)|<number>
<number> ::= <digit> {<digit>}
<digit> ::= 0|1|2|3|4|5|6|7|8|9

We need First((<expr>)) and First(<number>) to be disjoint.

First((<expr>)) = {(} and First (<number>) = First(<digit>) = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} so the condition holds.

Second Criteria for Predictive Parsing

If we have rule of the form A→α[β]γ (I.e., we have an optional β), then the set of first tokens β can go to must be distinct from the set of tokens that could immediately follow β.

For example, consider:

 
A → B [C] D. 
C→ aE | bF. 
D→ cG.

Then First( C ) = {a, b} and Follow( C ) ={c}. So the criteria would hold.

First, Follow for Calculator Example

First and Follow sets for calculator example

Once, we have built these sets we can then use then to make our parse table from before.
In the above, next to each rule in {} is a list of symbols which when we see the left hand non-terminal at the top of the stack should cause us to predict that rule.
For a rule `A ::= alpha`, this set consists of the union of FIRST(`\alpha`), together with FOLLOW(A), if it is possible to derive `alpha =>^\star epsilon`

Tools for Scanners and Parsers.

Lex is a tool for writing scanners; Yacc is a tool for writing parsers.
Both originated at Bell Labs in 1970s (Lex - Mike Lesk and Eric Schmidt 1975, Yacc - Johnson 1975).
Flex/Bison are their roughly equivalent GNU counterparts.
Both are C pre-processors. That is, we write C code with Lex/Yacc directives in them, run the Lex/Yacc preprocessor and get a pure C program which can then be compiled.

The basic structure of both a Lex and a Yacc program is as follows:

%{// C code to insert verbatim at start of program
%}
/* Lex/Yacc Definitions */
%%
//Lex/Yacc code
%%
// more C code.

A Simple Lex Example

%{
#include <stdio.h>
int wordCount = 0;
%}
word [^ \t\n]+ /* make an abbreviation word for the expr 
    [^ \t\n]+ (make sure to delete this comment before runninf lex)*/
%%
[\t\n ]+ {printf("I see whitespace\n");} //what to do if see pattern
{word} {wordCount++;}
%%
int main()
{
   yylex(); //call the lexer. Gets input from command line until ^D
   printf("word count: %d", wordCount); return 0;
}
To compile:
lex  -o lextest.c lextest.l #default output is lex.yy.c
gcc  lextest.c -o lextest -ll  #-ll not needed if use flex.

A Yacc Example

Suppose we put this file in yacctest.y

%{
#include <stdio.h>
int yylex();
int yyerror();
%}
%token ARTICLE NORMAL_NOUN PROPER_NOUN
%%
noun_phrase : PROPER_NOUN { printf("Proper Noun\n"); }
    | ARTICLE NORMAL_NOUN {printf("Usual Noun\n"); }
%%
int main(int argc, char **argv)
{
  extern FILE *yyin;
  yyin = fopen(argv[1], "r"); //sets up lexer to use this file as input
  yyparse();  
  fclose(yyin);  
}

More on Yacc Example

To compile the above you could use the line:
```
yacc -d yacctest.y
```
This will produce two files: y.tab.c and y.tab.h. If you use bison you'd get yacctest.tab.c and yacctest.tab.h
The .h file contains #defines for the tokens ARTICLE, NORMAL_NOUN, etc.
You would then need to write a lex program which includes y.tab.h.
It might have a rule like:
```
A|a|The|the {return ARTICLE;}
```

For example, here is a minimal lex program which we could put in lex.yy.l:


%{
#include <stdio.h>
#include "y.tab.h"
%}

%%
A|a|The|the {return ARTICLE;}

dog|cat {return NORMAL_NOUN;}

Chris|Mary {return PROPER_NOUN;}
%%

We could run lex on this as:
```
lex -o lex.yy.c lex.yy.l
```

Still More on Yacc Example

Once you have run yacc and lex on the above grammar and its corresponding scanner. To compile the whole thing you would type:
```
gcc -o yacctest y.tab.c lex.yy.c -ly -ll
```
Suppose we had a file eng.txt:
```
The cat
```
Then running the program might look like:
```
>./yacctest eng.txt
 Usual Noun
```

Finish Parsing

Outline