A BNF Parser in Forth

Introduction

Backus-Naur Form, BNF, is a notation for the formal description of programming languages. While most commonly used to specify the syntax of "conventional" programming languages such as Pascal and C, BNF is also of value in command language interpreters and other language processing.

This paper describes a Forth extension which transforms BNF expressions to executable Forth words. This gives Forth a capability equivalent to YACC or TMG, to produce a working parser from a BNF language description.

This article first appeared in ACM SigFORTH Newsletter vol. 2 no. 2. Since then the code has been updated from the original by staff at MPE, and this documentation has been derived from the article supplied by Brad Rodriguez, whose original implementation is a model of Forth programming.

The source code is compiled by the second stage build and is in the file Sources\VFXBase\bnf.fth.

BNF Expressions

BNF expressions or productions are written as follows:


production  ::=  term ... term    \ alternate #1
              |  term ... term    \ alternate #2
              |  term ... term    \ alternate #3

This example indicates that the given production may be formed in one of three ways. Each alternative is the concatenation of a series of terms. A term in a production may be either another production, called a nonterminal, or a fundamental token, called a terminal.

A production may use itself recursively in its definition. For example, an unsigned integer can be defined with the productions


<digit> ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
<number> ::= <digit> | <digit> <number>

which says that a number is either a single digit, or a single digit followed by another number of one or more digits.

We will use the conventions of a vertical bar | to separate alternatives, and angle brackets to designate the name of a production. Unadorned ASCII characters are terminals, the fundamental tokens.

A Simple Solution through Conditional Execution

The logic of succession and alternation can be implemented in two "conditional execution" operators, && and ||. These correspond exactly to the "logical connectives" of the same names in the C language (although their use here was actually inspired by the Unix "find" command). They are defined:


: || IF R> DROP 1 THEN ;    ( exit on true)
: && 0= IF R> DROP 0 THEN ; ( exit on false)

|| given a true value on the stack, exits the colon definition immediately with true on the stack. This can be used to string together alternatives: the first alternative which is satisfied (returns true) will stop evaluation of further alternatives.

&& given a false value on the stack, exits the colon definition immediately with false on the stack. This is the "concatenation" operator: the first term which fails (returns false) stops evaluation and causes the entire sequence to fail.

We assume that each "token" (terminal) is represented by a Forth word which scans the input stream and returns a success flag. Productions (nonterminals) which are built with such tokens, ||, and &&, are guaranteed to return a success flag.

So, assuming the "token" words '0' thru '9' have been defined, the previous example becomes:


: <DIGIT>   '0' || '1' || '2' || '3' || '4'
                || '5' || '6' || '7' || '8' || '9' ;
: <NUMBER1>   <DIGIT> && <NUMBER> ;
: <NUMBER>    <DIGIT> || <NUMBER1> ;

Neglecting the problem of forward referencing for the moment, this example illustrates three limitations:

  1. we need an explicit operator for concatenation, unlike BNF.
  2. && and || have equal precedence, which means we can't mix && and || in the same Forth word and get the equivalent BNF expression. We needed to split the production <NUMBER> into two words. BNF production fails.

We will address these next.

A Better Solution

Several improvements can be made to this "rough" BNF parser, to remove its limitations and improve its "cosmetics."

a)) Concatenation by juxtaposition. We can cause the action of && to be performed "invisibly" by enforcing this rule for all terms (terminals and nonterminals): Each term examines the stack on entry. if false, the word exits immediately with false on the stack. Otherwise, it parses and returns a success value.

To illustrate this: consider a series of terms


<ONE> <TWO> <THREE> <FOUR>

Let <ONE> execute normally and return "false." The <TWO> is entered, and exits immediately, doing nothing. Likewise, <THREE> and <FOUR> do nothing. Thus the remainder of the expression is skipped, without the need for a return-stack exit.

b)) Precedence. By eliminating the && operator in this manner, we make it possible to mix concatenation and alternation in a single expression. A failed concatenation will "skip" only as far as the next operator. So, our previous example becomes:


: <NUMBER> <DIGIT> || <DIGIT> <NUMBER> ;

c)) Backtracking. If a token fails to match the input stream, it does not advance the scan pointer. Likewise, if a BNF production fails, it must restore the scan pointer to the "starting point" where the production was attempted, since that is the point at which alternatives must be tried. We therefore enforce this rule for all terminals and nonterminals: Each term saves the scan pointer on entry. If the term fails, the scan pointer is restored; otherwise, the saved value is discarded.

We will later find it useful to "backtrack" an output pointer, as well.

d)) Success as a variable. An examination of the stack contents during parsing reveals the surprising fact that, at any time, there is only one success flag on the stack! (This is because flags placed on the stack are immediately "consumed.") We can use a variable, SUCCESS, for the parser success flags, and thereby simplify the manipulations necessary to use the stack for other data. All BNF productions accept, and return, a truth value in success.

Notation

<BNF is used at the beginning of a production. If SUCCESS is false, it causes an immediate exit. Otherwise, it saves the scan pointer on the return stack.

BNF> is used at the end of a production. If SUCCESS is false, it restores the scan position from the saved pointer. In any case, it removes the saved pointer from the return stack.

<BNF and BNF> are "run-time" logic, compiled by the words ::= and ;; respectively.

::= name starts the definition of the BNF production name.

;; ends a BNF definition.

| separates alternatives. If SUCCESS is true, it causes an immediate exit and discards the saved scan pointer. Otherwise, it restores the scan position from the saved pointer.

{ ... } denotes a production statement which is issued when successful evaluation of the preceding checks has been performed.. The alternative form { _ }{ _ } allows conditional productions in which the first part is produced when SUCCESS is true, and the second part is produced when SUCCESS is false.

[[ ... ]] denotes a block that must be performed 0 or more times, and thus is totally optional. Alternatives are not permitted.

<< ... >> denotes a block that must be performed at least once. Alternatives are not permitted.

There are four words which simplify the definition of token words and other terminals:

@TOKEN fetches the current token from the input.

+TOKEN advances the input scan pointer.

=TOKEN compares the value on top of stack to the current token, following the rules for BNF parsing words.

nn TOKEN name builds a "terminal" name, with the ASCII value nn.

The parser uses the Forth >IN as the input pointer, and the dictionary pointer DP as the output pointer. These choices were made strictly for convenience; there is no implied connection with the Forth compiler.

Examples and Usage

The syntax of a BNF definition in Forth resembles the "traditional" BNF syntax:

Traditional:


prod ::= term term | term term

Forth:


::= prod term term | term term ;;

The first example below is a simple pattern recognition problem, to identify text having balanced left and right parentheses. Several aspects of the parser are illustrated by this example:

  1. Three tokens are defined on line 4. To avoid name conflicts, they are named with enclosing quotes. <EOL> matches the end-of-line character in the Forth Terminal Input Buffer. often encountered in BNF. The null alternative parses no tokens, and is always satisfied.
  2. Not all parsing words need be written as BNF productions. The definition of <CHAR> is Forth code to parse any ASCII character, excluding parentheses and nulls. Note that ::= and ;; are used, not to create a production, but as an easy way to create a conditionally-executing (per SUCCESS) Forth word. "true," and the "topmost" BNF production is executed. on its return, SUCCESS is examined to determine the final result.
  3. PARSE also shows how end-of-input is indicated to the BNF parser: the sequence is defined as the desired BNF production, followed by end-of-line.

The second example parses algebraic expressions with precedence. This grammar is directly from [AH077], p. 138. The use of the productions <T'> and <E'> to avoid the problem of left-recursion is described on p. 178 of that book. Note also:

The third example shows how this algebraic parser can be modified to perform code generation, coincident with the parsing process. Briefly: each alternative of a BNF production includes Forth code to compile the output which would result from that alternative. If the alternative succeeds, that output is left in the dictionary. If it fails, the dictionary pointer is "backtracked" to remove that output. Thus, as the parser works its way, top-down, through the parse tree, it is constantly producing and discarding trial output.

This example produces Forth source code for the algebraic expression.

Cautions

This parser is susceptible to the Two Classic Mistakes of BNF expressions. Both of these cautions can be illustrated with the production <NUMBER>:


::= <NUMBER>
    <DIGIT> <NUMBER> | <DIGIT> ;;

a)) Order your alternatives carefully. If <NUMBER> were written


::= <NUMBER>
<DIGIT> | <DIGIT> <NUMBER> ;;

then all numbers would be parsed as one and only one digit! This is because alternative #1 -- which is a subset of alternative #2 -- is always tested first. In general, the alternative which is the subset or the "easier to-satisfy" should be tested last.

b)) Avoid "left-recursion." If <NUMBER> were written


::= <NUMBER>
    <NUMBER> <DIGIT> | <DIGIT> ;;

then you will have an infinite recursive loop of <NUMBER> calling <NUMBER>! To avoid this problem, do not make the first term in any alternative a recursive reference to the production being defined. (This rule is somewhat simplified; for a more detailed discussion of this problem, refer to [AH077], pp. 177 to 179.)

Comparison to "traditional" work

In the jargon of compiler writers, this parser is a "top-down parser with backtracking." Another such parser, from ye olden days of Unix, was TMG. Top-down parsers are among the most flexible of parsers; this is especially so in this implementation, which allows Forth code to be intermixed with BNF expressions.

Top-down parsers are also notoriously inefficient. Predictive parsers, which look ahead in the input stream, are better. Bottom-up parsers, which move directly from state to state in the parse tree according to the input tokens, are better still. Such a parser, YACC (a table-driven LR parser), has entirely supplanted TMG in the Unix community.

Still, the minimal call-and-return overhead of Forth alleviates the speed problem somewhat, and the simplicity and flexibility of the BNF Parser may make it the parser of choice for many applications. Experience at MPE shows that BNF parsers are actually quite fast.

Applications and Variations

Compilers. The obvious application of a BNF parser is in writing translators for other languages. This should certainly strengthen Forth's claim as a language to write other languages.

Command interpreters. Complex applications may have an operator interface sufficiently complex to merit a BNF description. For example, this parser has been used in an experimental lighting control system; the command language occupied 30 screens of BNF.

Pattern recognition. Aho & Ullman [AH077] note that any construct which can be described by a regular expression, can also be described by a context-free grammar, and thus in BNF. [AH077] identifies some uses of regular expressions for pattern recognition problems; such problems could also be addressed by this parser.

An extension of these parsing techniques has been used to implement a Snobol4-style pattern matcher [ROD89a].

Goal directed evaluation. The process of searching the parse tree for a successful result is essentially one of "goal-directed evaluation." Many problems can be solved by goal-directed techniques.

For example, a variation of this parser has been used to construct an expert system [ROD89b].

References

[AH077] Alfred Aho and Jeffrey Ullman, Principles of Compiler Design, Addison-Wesley, Reading, MA (1977), 604 pp.

[ROD89a] B. Rodriguez, "Pattern Matching in Forth," presented at the 1989 FORML Conference, 14 pp.

Example 1 - balanced parentheses


\ Example #1 - from Aho & Ullman, Principles of Compiler Design, p137
\ This grammar recognises strings having balanced parentheses

hex

ascii ( token '('
ascii ) token ')'
0 token <eol>

::= <char>
  @token
  dup 02A 07F within?
  swap 1 027 within?  or
  dup success !
  +token
;;

::= <s>
        '(' <s> ')' <s>
     |  <char> <s>
     |
;;

: parse
  1 success !
  <s> <eol>
  cr  success @
  if    ." Successful
  else  ." Failed"
  endif
;

Example 2 - Infix notation


ascii + token '+'       ascii - token '-'
ascii * token '*'       ascii / token '/'
ascii ( token '('       ascii ) token ')'

ascii ^ token '^'

ascii 0 token '0'       ascii 1 token '1'
ascii 2 token '2'       ascii 3 token '3'
ascii 4 token '4'       ascii 5 token '5'
ascii 6 token '6'       ascii 7 token '7'
ascii 8 token '8'       ascii 9 token '9'

0 token <eol>

::= <digit>
          '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'
;;

::= <number>
          <digit> <number>
        | <digit> { }
;;

defer <expression>              \ needed for a recursive definition

::= <element>
          '(' <expression> ')'
        | <number>
;;

::= <primary>
          '-' <primary>
        | <element>
;;

::= <factor>
          <primary> '^' <factor>
        | <primary>
;;

::= <t'>
          '*' <factor> <t'>
        | '/' <factor> <t'>
        |
;;

::= <term>
          <factor> <t'>
;;

::= <e'>
          '+' <term> <e'>
        | '-' <term> <e'>
        |
;;

::= <<expression>>
          <term> <e'>
;;
assign <<expression>> to-do <expression>

: parse
  1 success !
  <expression> <eol>
  cr  success @
  if    ." Successful
  else  ." Failed"
  endif
;

Example 3 - infix notation again with on-line calculation


: x^y           \ x y -- n
  dup 0< abort" can't deal with negative powers"
  1 swap 0                              \ -- x 1 y 0
  ?do  over *  loop                     \ -- x x^i
  nip                                   \ -- x^y
;

1 constant mul
2 constant div
3 constant add
4 constant sub

: dyadic        \ x op y -- n
  case  swap
    mul of  *  endof
    div of  /  endof
    add of  +  endof
    sub of  -  endof
            1 abort" invalid operator"
  endcase
;

decimal

ascii + token '+'       ascii - token '-'
ascii * token '*'       ascii / token '/'
ascii ( token '('         ascii ) token ')'

ascii ^ token '^'

ascii 0 token '0'       ascii 1 token '1'
ascii 2 token '2'       ascii 3 token '3'
ascii 4 token '4'       ascii 5 token '5'
ascii 6 token '6'       ascii 7 token '7'
ascii 8 token '8'       ascii 9 token '9'

bl token <sp>
9 token <tab>
0 token <eol>

::= <whitespacechar>  \ -- ; could be expanded to refill input buffer
  <sp> | <tab>
;;

::= <whitespace>
  [[ <whitespacechar> ]]
;;


::= <digit>
          '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'
;;

::= <number>
          { 0 }                               \ initial accumulator
          <whitespace>
          << <digit>
             { 10 *  last-token @ ascii 0 -  +  }
          >>
;;

defer <expression>              \ needed for a recursive definition

::= <element>
          '(' <expression> ')'
        | <number>
;;

::= <primary>
          '-' <primary>  { negate }
        | <element>
;;

::= <factor>
          <primary> '^' <factor> { x^y }
        | <primary>
;;

::= <term-op>
          '*' { mul }
        | '/' { div }
;;

::= <term>
          <factor> [[ <whitespace> <term-op> <factor> { dyadic } ]]
;;

::= <exp-op>
          '+' { add }
        | '-' { sub }
;;

::= <<expression>>
          <term> [[ <whitespace> <exp-op> <term> { dyadic } ]]
;;  assign <<expression>> to-do <expression>

: parse
  1 success !
  <expression> <whitespace> <eol>
  cr  success @
  if    ." Successful" cr ." Result = " .
  else  ." Failed"
  endif
;

Acknowledgements

This article first appeared in ACM SigFORTH Newsletter vol. 2 no. 2. Since then the code has been updated from the original by staff at MPE.

Bradford J. Rodriguez
T-Recursive Technology
115 First St. #105
Collingwood, Ontario L9Y 4W3 Canada
bj@forth.org

Glossary

variable success        \ -- addr
This variable is set true if the last BNF statement succeded, otherwise it is false.

variable skipspace      \ -- addr
Controls space skipping. When set true, following spaces are skipped.

variable BNF-ignore-lines       \ -- addr
Controls line break handling. When set true, line breaks are ignored by REFILLing the input buffer.

vocabulary bnf-voc      \ --
Holds BNF internal words.

: |             \ -- ; performs OR function
Performs the OR function inside a BNF definition.

: ?bnf-error    \ --
Produce an error message on parsing failure.

: save-success  \ -- ; R: -- success
Save the SUCCESS flag on the return stack.

: check-success \ -- ; R: success --
Generate an error if the value of SUCCESS previously saved on the return stack was true but now isn't. Useful to provide sensible source error messages inside deeply nested definitions.

: ::=           \ -- sys ; defines a BNF definition
Start a BNF definition of the form:

::= <name> ... ;;

: ;;            \ sys -- ; marks end of ::= <name> ... ;; definition
Ends a BNF definition of the form:

::= <name> ... ;;

: {             \ -- sys
Marks the start of production output if SUCCESS is true. Use in the form: "{ ... }{ ... }" which generates the code for "SUCCESS @ IF ... ELSE ... THEN". Note that this notation conflicts with the use of { ... } for locals, but not with {: ... :}.

: }{            \ sys -- sys'
Allows an ELSE clause for production output.

: }             \ sys --
End of production output

The operators [[ ... ]] define a sequence that may be performed 0 or more times. The operators << ... >> define a sequence that must be performed 1 or more times.

The point of this is to allow repetition to be defined more easily without constant recourse to recursive definitions.

The block may not include | operators.

: [[            \ -- addr1 addr2 ; start of [[ ... ]] block, loop end inline
Starts an optional block (0 or more repetitions) of the form:

  [[ ... ]]

Note that alternatives using | are not permitted.

: ]]            \ addr1 addr2 -- ; end of [[ ... ]] block, loop start inline
Ends an optional block of the form:

  [[ ... ]]

Note that alternatives using | are not permitted.

: <<            \ -- addr1 addr2 ; start of << ... >> block, loop end inline
Starts a block (1 or more repetitions) of the form:

  << ... >>

Note that alternatives using | are not permitted.

: >>            \ addr1 addr2 -- ; end of [[ ... ]] block, loop start inline
Ends a block (1 or more repetitions) of the form:

  << ... >>

Note that alternatives using | are not permitted.

: token         \ n --
Use in the form "<char> TOKEN <name>" to define a word <name> which succeeds if the next token (character) is <char>.

: +spaces       \ --
Enables space skipping. If +SPACES does not call nextNonBL then it has to appear BEFORE the last word for which spaces will not be skipped, which is confusing. This way the final word which does not discard its following spaces appears in the source code before the +SPACES, which looks more logical.

: -spaces       \ --
Disables space skipping.

: string        \ -- ; string <name> text ; e.g. string 'CAP' WS_CAPTION
Used in the form "STRING <name> text" to create a word <name> which succeeds when space delimited text is next in the input stream. Note that text may not contain spaces. Because of some parsing requirements, e.g. some BASICs and FORTRAN, a superset of text will succeed, leaving the residue in the input stream. This means that for "STRING <name> abcd" the strings "a", "ab", and "abc" will also succeed. Thus if you need to test a set of strings, you should test the longest first, e.g:


String str1 abcd
String str2 abc
String str3 ab
\ WRONG because abcd will match str3
::= test  str3 | str2 | str1  ;;
\ RIGHT
::= test  str1 | str2 | str3  ;;

Error reporting

Because the BNF parser is a top-down recursive descent parser, when a rule fails, it backtracks to the previous successful position, both in terms of output and source file position. Because of this, the reported error position may be some way before the actual location that triggered the error.