Backus-Naur Form, BNF, is a notation for the formal description of programming languages. While most commonly used to specify the syntax of "conventional" programming languages such as Pascal and C, BNF is also of value in command language interpreters and other language processing.
This paper describes a Forth extension which transforms BNF expressions to executable Forth words. This gives Forth a capability equivalent to YACC or TMG, to produce a working parser from a BNF language description.
This article first appeared in ACM SigFORTH Newsletter vol. 2 no. 2. Since then the code has been updated from the original by staff at MPE, and this documentation has been derived from the article supplied by Brad Rodriguez, whose original implementation is a model of Forth programming.
The source code is compiled by the second stage build and is in the file SOURCES\VFXBASE\BNF.FTH.
BNF expressions or productions are written as follows:
|
This example indicates that the given production may be formed in one of three ways. Each alternative is the concatenation of a series of terms. A term in a production may be either another production, called a nonterminal, or a fundamental token, called a terminal.
A production may use itself recursively in its definition. For example, an unsigned integer can be defined with the productions
|
which says that a number is either a single digit, or a single digit followed by another number of one or more digits.
We will use the conventions of a vertical bar | to separate alternatives, and angle brackets to designate the name of a production. Unadorned ASCII characters are terminals, the fundamental tokens.
The logic of succession and alternation can be implemented in two "conditional execution" operators, && and ||. These correspond exactly to the "logical connectives" of the same names in the C language (although their use here was actually inspired by the Unix "find" command). They are defined:
|
|| given a true value on the stack, exits the colon definition immediately with true on the stack. This can be used to string together alternatives: the first alternative which is satisfied (returns true) will stop evaluation of further alternatives.
&& given a false value on the stack, exits the colon definition immediately with false on the stack. This is the "concatenation" operator: the first term which fails (returns false) stops evaluation and causes the entire sequence to fail.
We assume that each "token" (terminal) is represented by a Forth word which scans the input stream and returns a success flag. Productions (nonterminals) which are built with such tokens, ||, and &&, are guaranteed to return a success flag.
So, assuming the "token" words '0' thru '9' have been defined, the previous example becomes:
|
Neglecting the problem of forward referencing for the moment, this example illustrates three limitations:
a) we need an explicit operator for concatenation, unlike BNF.
b) && and || have equal precedence, which means we can't mix && and || in the same Forth word and get the equivalent BNF expression. We needed to split the production <NUMBER> into two words.
c) we have made no provision for restoring the scan pointer if a BNF production fails.
We will address these next.
Several improvements can be made to this "rough" BNF parser, to remove its limitations and improve its "cosmetics."
a) Concatenation by juxtaposition. We can cause the action of && to be performed "invisibly" by enforcing this rule for all terms (terminals and nonterminals): Each term examines the stack on entry. if false, the word exits immediately with false on the stack. Otherwise, it parses and returns a success value.
To illustrate this: consider a series of terms
|
Let <ONE> execute normally and return "false." The <TWO> is entered, and exits immediately, doing nothing. Likewise, <THREE> and <FOUR> do nothing. Thus the remainder of the expression is skipped, without the need for a return-stack exit.
b) Precedence. By eliminating the && operator in this manner, we make it possible to mix concatenation and alternation in a single expression. A failed concatenation will "skip" only as far as the next operator. So, our previous example becomes:
|
c) Backtracking. If a token fails to match the input stream, it does not advance the scan pointer. Likewise, if a BNF production fails, it must restore the scan pointer to the "starting point" where the production was attempted, since that is the point at which alternatives must be tried. We therefore enforce this rule for all terminals and nonterminals: Each term saves the scan pointer on entry. If the term fails, the scan pointer is restored; otherwise, the saved value is discarded.
We will later find it useful to "backtrack" an output pointer, as well.
d) Success as a variable. An examination of the stack contents during parsing reveals the surprising fact that, at any time, there is only one success flag on the stack! (This is because flags placed on the stack are immediately "consumed.") We can use a variable, SUCCESS, for the parser success flags, and thereby simplify the manipulations necessary to use the stack for other data. All BNF productions accept, and return, a truth value in success.
<BNF is used at the beginning of a production. If SUCCESS is false, it causes an immediate exit. Otherwise, it saves the scan pointer on the return stack.
BNF> is used at the end of a production. If SUCCESS is false, it restores the scan position from the saved pointer. In any case, it removes the saved pointer from the return stack.
<BNF and BNF> are "run-time" logic, compiled by the words ::= and ;; respectively.
::= name starts the definition of the BNF production name.
;; ends a BNF definition.
| separates alternatives. If SUCCESS is true, it causes an immediate exit and discards the saved scan pointer. Otherwise, it restores the scan position from the saved pointer.
{ ... } denotes a production statement which is issued when successful evaluation of the preceding checks has been performed.. The alternative form { _ }{ _ } allows conditional productions in which the first part is produced when SUCCESS is true, and the second part is produced when SUCCESS is false.
[[ ... ]] denotes a block that must be performed 0 or more times, and thus is totally optional. Alternatives are not permitted.
<< ... >> denotes a block that must be performed at least once. Alternatives are not permitted.
There are four words which simplify the definition of token words and other terminals:
@TOKEN fetches the current token from the input.
+TOKEN advances the input scan pointer.
=TOKEN compares the value on top of stack to the current token, following the rules for BNF parsing words.
nn TOKEN name builds a "terminal" name, with the ASCII value nn.
The parser uses the Forth >IN as the input pointer, and the dictionary pointer DP as the output pointer. These choices were made strictly for convenience; there is no implied connection with the Forth compiler.
The syntax of a BNF definition in Forth resembles the "traditional" BNF syntax:
Traditional:
|
Forth:
|
The first example below is a simple pattern recognition problem, to identify text having balanced left and right parentheses. Several aspects of the parser are illustrated by this example:
a) Three tokens are defined on line 4. To avoid name conflicts, they are named with enclosing quotes. <EOL> matches the end-of-line character in the Forth Terminal Input Buffer.
b) A recursive production, <S>, is shown.
c) The definition of <S> also shows a null alternative. This is often encountered in BNF. The null alternative parses no tokens, and is always satisfied.
d) Not all parsing words need be written as BNF productions. The definition of <CHAR> is Forth code to parse any ASCII character, excluding parentheses and nulls. Note that ::= and ;; are used, not to create a production, but as an easy way to create a conditionally-executing (per SUCCESS) Forth word.
e) PARSE shows how to invoke the parser: SUCCESS is initialized to "true," and the "topmost" BNF production is executed. on its return, SUCCESS is examined to determine the final result.
f) PARSE also shows how end-of-input is indicated to the BNF parser: the sequence is defined as the desired BNF production, followed by end-of-line.
The second example parses algebraic expressions with precedence. This grammar is directly from [AH077], p. 138. The use of the productions <T'> and <E'> to avoid the problem of left-recursion is described on p. 178 of that book. Note also:
a) <DIGIT> is defined "the hard way." It would be better to do this with a Forth word.
b) <ELEMENT> requires a forward reference to <EXPRESSION>. We must patch this reference manually.
The third example shows how this algebraic parser can be modified to perform code generation, coincident with the parsing process. Briefly: each alternative of a BNF production includes Forth code to compile the output which would result from that alternative. If the alternative succeeds, that output is left in the dictionary. If it fails, the dictionary pointer is "backtracked" to remove that output. Thus, as the parser works its way, top-down, through the parse tree, it is constantly producing and discarding trial output.
This example produces Forth source code for the algebraic expression.
a) The word ," appends a text string to the output.
b) We have chosen to output each digit of a number as it is parsed. (DIGIT) is a subsidiary word to parse a valid digit. <DIGIT> picks up the character from the input stream before it is parsed, and then appends it to the output. If it was not a digit, SUCCESS will be false and ;BNF will discard the appended character.
If we needed to compile numbers in binary, <NUMBER> would have to do the output. <NUMBER> could start by placing a zero on the stack as the accumulator. <DIGIT> could augment this value for each digit. Then, at the end of <NUMBER>, the binary value on the stack could be output.
c) After every complete number, we need a space. We could factor <NUMBER> into two words, like <DIGIT>. But since <NUMBER> only appears once, in <ELEMENT>, we append the space there.
d) In <PRIMARY>, MINUS is appended after the argument is parsed. In <FACTOR>, POWER is appended after its two arguments are parsed. <T'> appends * or / after the two arguments, and likewise <E'> appends + or -.
In all of these cases, an argument may be a number or a sub-expression. If the latter, the entire code to evaluate the sub-expression is output before the postfix operator is output. (Try it. It works.)
e) PARSE has been modified to TYPE the output from the parser, and then to restore the dictionary pointer.
This parser is susceptible to the Two Classic Mistakes of BNF expressions. Both of these cautions can be illustrated with the production <NUMBER>:
|
a) Order your alternatives carefully. If <NUMBER> were written
|
then all numbers would be parsed as one and only one digit! This is because alternative #1 -- which is a subset of alternative #2 -- is always tested first. In general, the alternative which is the subset or the "easier to-satisfy" should be tested last.
b) Avoid "left-recursion." If <NUMBER> were written
|
then you will have an infinite recursive loop of <NUMBER> calling <NUMBER>! To avoid this problem, do not make the first term in any alternative a recursive reference to the production being defined. (This rule is somewhat simplified; for a more detailed discussion of this problem, refer to [AH077], pp. 177 to 179.)
In the jargon of compiler writers, this parser is a "top-down parser with backtracking." Another such parser, from ye olden days of Unix, was TMG. Top-down parsers are among the most flexible of parsers; this is especially so in this implementation, which allows Forth code to be intermixed with BNF expressions.
Top-down parsers are also notoriously inefficient. Predictive parsers, which look ahead in the input stream, are better. Bottom-up parsers, which move directly from state to state in the parse tree according to the input tokens, are better still. Such a parser, YACC (a table-driven LR parser), has entirely supplanted TMG in the Unix community.
Still, the minimal call-and-return overhead of Forth alleviates the speed problem somewhat, and the simplicity and flexibility of the BNF Parser may make it the parser of choice for many applications. Experience at MPE shows that BNF parsers are actually quite fast.
Compilers. The obvious application of a BNF parser is in writing translators for other languages. This should certainly strengthen Forth's claim as a language to write other languages.
Command interpreters. Complex applications may have an operator interface sufficiently complex to merit a BNF description. For example, this parser has been used in an experimental lighting control system; the command language occupied 30 screens of BNF.
Pattern recognition. Aho & Ullman [AH077] note that any construct which can be described by a regular expression, can also be described by a context-free grammar, and thus in BNF. [AH077] identifies some uses of regular expressions for pattern recognition problems; such problems could also be addressed by this parser.
An extension of these parsing techniques has been used to implement a Snobol4-style pattern matcher [ROD89a]. Goal directed evaluation. The process of searching the parse tree for a successful result is essentially one of "goal-directed evaluation." Many problems can be solved by goal-directed techniques.
For example, a variation of this parser has been used to construct an expert system [ROD89b].
[AH077] Alfred Aho and Jeffrey Ullman, Principles of Compiler Design, Addison-Wesley, Reading, MA (1977), 604 pp.
[ROD89a] B. Rodriguez, "Pattern Matching in Forth," presented at the 1989 FORML Conference, 14 pp.
|
|
|
|
|
This article first appeared in ACM SigFORTH Newsletter vol. 2 no. 2. Since then the code has been updated from the original by staff at MPE.
Bradford J. Rodriguez
T-Recursive Technology
115 First St. #105
Collingwood, Ontario L9Y 4W3 Canada
bj@forth.org
variable success \ -- addr
This variable is set true if the last BNF statement succeded,
otherwise it is false.
variable skipspace \ -- addr
Controls space skipping. When set true, following spaces
are skipped.
variable BNF-ignore-lines \ -- addr
Controls line break handling. When set true, line breaks
are ignored by REFILLing the input buffer.
: | \ -- ; performs OR function
Performs the OR function inside a BNF definition.
: ?bnf-error \ --
Produce an error message on parsing failure.
: save-success \ -- ; R: -- success
Save the SUCCESS flag on the return stack.
: check-success \ -- ; R: success --
Generate an error if the value of SUCCESS previously saved
on the return stack was true but now isn't. Useful to provide
sensible source error messages inside deeply nested definitions.
: ::= \ -- sys ; defines a BNF definition
Start a BNF definition of the form:
::= <name> ... ;;
: ;; \ sys -- ; marks end of ::= <name> ... ;; definition
Ends a BNF definition of the form:
::= <name> ... ;;
: { \ -- sys
Marks the tart of production output if SUCCESS is true.
Use in the form: "{ ... }{ ... }" which generates the code
for "SUCCESS @ IF ... ELSE ... THEN".
: }{ \ sys -- sys'
Allows an ELSE clause for production output.
: } \ sys --
End of production output
: [[ \ -- addr1 addr2 ; start of [[ ... ]] block, loop end inline
Starts an optional block (0 or more repetitions) of the form:
[[ ... ]]
Note that alternatives using | are not permitted.
: ]] \ addr1 addr2 -- ; end of [[ ... ]] block, loop start inline
Ends an optional block of the form:
[[ ... ]]
Note that alternatives using | are not permitted.
: << \ -- addr1 addr2 ; start of << ... >> block, loop end inline
Starts a block (1 or more repetitions) of the form:
<< ... >>
Note that alternatives using | are not permitted.
: >> \ addr1 addr2 -- ; end of [[ ... ]] block, loop start inline
Ends a block (1 or more repetitions) of the form:
<< ... >>
Note that alternatives using | are not permitted.
variable last-token \ -- addr
Holds the last token (or 0) retrieved by @TOKEN.
: token \ n --
Use in the form "<char> TOKEN <name>" to define a word <name>
which succeeds if the next token (character) is <char>.
: +spaces \ --
Enables space skipping.
If +SPACES does not call nextNonBL then it has to appear BEFORE the
last word for which spaces will not be skipped, which is confusing.
This way the final word which does not discard its following spaces
appears in the source code before the +SPACES, which looks more logical.
: -spaces \ --
Disables space skipping.
: string \ -- ; string <name> text ; e.g. string 'CAP' WS_CAPTION
Used in the form "STRING <name> text" to create a word <name>
which succeeds when space delimited text is next in the input
stream. Note that text may not contain spaces. Because of some
parsing requirements, e.g. some BASICs and FORTRAN, a superset
of text will succeed, leaving the residue in the input stream.
This means that for "STRING <name> abcd" the strings "a", "ab",
and "abc" will also succeed. Thus if you need to test a set
of strings, you should test the longest first, e.g:
|
Because the BNF parser is a top-down recursive descent parser, when a rule fails, it backtracks to the previous successful position, both in terms of output and source file position. Because of this, the reported error position may be some way before the actual location that triggered the error.