Language Processing

Language Processors

A simple language processor is a pipeline consisting of two phases: parsing and execution:

A more complex language processor might include scanning, type checking and translation:

Parsing

A parser uses a formal grammar to transform an input expression from a string into a tree (unless a syntax error is detected).

The tree tells the execution module in which order the sub-expressions should be executed.

For example, here's a formal grammar for a simple language called SOP (Sums of Products):

1.     A sum is one or more products separated by plus signs:

sum ::= product ~ ("+" ~ product)*

2.     A product is one or more terms separated by multiply signs:

product ::= term ~ ("*" ~ term)*

3.     A term is a number or a sum in parentheses:

term ::= number | "(" ~ sum ~ ")"

4.     A number is an optional sign followed by one or more digits:

number ::= ("+" | "-")? ~ [0-9]+

Notes

a* means iterate a 0 or more times

a+ means iterate a 1 or more times

a? means a is optional

a ~ b means a is followed by b

::= means "consists of"

a | b means a or b

[0-9] means "0"|"1"|"2"|"3"|"4"|"5"|"6"|"7"|"8"|"9"

Parsing "2 * 3 + 5" using the SOP grammar:

parse("2 * 3 + 5", SOP) = (+ (* 2 3) 5)

produces the tree:

Execution

Parsing uses a grammar to transform an expression in the form of a string into an expression in the form of a tree.

Execution uses an environment to transform an expression in the form of a tree into a value.

The Expression-Value Dichotomy is one of the most important distinctions in programming languages. It is the distinction between expressions (programmer input) and values (computer output).

                                           

For now, we can think of an environment as a symbol table that associates names to values:

For example:

execute((* x (+ pi z)), env) = 814.0

Like the parser, the processor is recursive. Only it works from the leaves of the tree to the root. Here's how the tree above might be executed:

1.     The value of a number is itself:

2.     The value of a product is the product of the values of its terms:

3.     The value of a sum is the sum of the values of its products:

Terminology

Meta-language refers to the implementation language of a language processor, while the language being processed is called the object language.

Syntax refers to the grammatic structure of an expression, while semantics refers to the behavior of an expression when it is executed.

An interpreter is a language processor that typically uses a read-execute-print loop that reads an expression entered by the user, parses it, executes it, then prints the value.

A compiler is a language processor that reads a program from one or more files, parses it, then translates it into one or more files of assembly language for some particular hardware platform.

An assembler translates a file of assembly language instructions into an excutable file containing binary machine language instructions.

Other Phases

Scanning

During the scanning phase the expression in string form is translated into a stream of tokens.

Examples of tokens include numbers, names, operators, and punctuation.

Scanners are analogous to spell checkers in word processors while parsers are analogous to grammar checkers.

For example:

scan("2 + 3 * 5") = ("2", "+", "3", "*", "5")

Type Checking

Like parsing, type checking uses rules such as:

1.     f(a) has type B if a has type A and f is a function with domain A and range B

f(a): A :- a: A, f: A=>B

2.     a has type B if a has type A and A is a subtype of B

a: B :- a: A, A <: B

Note

a: A means "a has type A"

:- means "if"

, means "and"

=> means "maps to"

<: means "subtype of"

For example:

sin(PI): Double because PI: Double and sin: Double => Double

"hello": Object because "hello": String and String extends Object

Translation

The translate phase translates the expression into a sequence of assembly language instructions, which can be executed by hardware.