330552ea49
Currently when a push parser finishes its parsing (i.e., it did not
return YYPUSH_MORE), it also clears its state. It is therefore
impossible to see if it had parse errors.
In the context of autocompletion, because error recovery might have
fired, the parser is actually already in a different state. For
instance on `(1 + + <TAB>` in the bistromathic, because there's a
`exp: "(" error ")"` recovery rule, `1 + +` tokens have already been
popped, replaced by `error`, and autocompletions think we are ready
for the closing ")". So here, we would like to see if there was a
syntax error, yet `yynerrs` was cleared.
In the case of a successful parse, we still have a problem: if error
recovery succeeded, we won't know it, since, again, `yynerrs` is
clearer.
It seems much more natural to leave the parser state available for
analysis when there is a failure.
To reuse the parser, we should either:
1. provide an explicit means to reinitialize a parser state for future
parses.
2. automatically reset the parser state when it is used in a new
parse.
Option 2 requires to check whether we need to reinitialize the parser
each time we call `yypush_parse`, i.e., each time we give a new token.
This seems expensive compared to Option 1, but benchmarks revealed no
difference. Option 1 is incompatible with the documentation
("After `yypush_parse` returns a status other than `YYPUSH_MORE`, the
parser instance `yyps` may be reused for a new parse.").
So Option 2 wins, reusing the private `yynew` member to record that a
parse was finished, and therefore that the state must reset in the
next call to `yypull_parse`.
While at it, this implementation now reuses the previously enlarged
stacks from one parse to another.
* data/skeletons/yacc.c (yypstate_new): Set up the stacks in their
initial configurations (setting their bottom to the stack array), and
use yypstate_clear to reset them (moving their top to their bottom).
(yypstate_delete): Adjust.
(yypush_parse): At the beginning, clear yypstate if needed, and at the
end, record when yypstate needs to be clearer.
* examples/c/bistromathic/parse.y (expected_tokens): Do not propose
autocompletion when there are parse errors.
* examples/c/bistromathic/bistromathic.test: Check that case.
Examples in C
This directory contains simple examples of Bison grammar files in C.
Some of them come from the documentation, which should be installed together with Bison. The URLs are provided for convenience.
rpcalc - Reverse Polish Notation Calculator
The first example is that of a simple double-precision Reverse Polish Notation calculator (a calculator using postfix operators). This example provides a good starting point, since operator precedence is not an issue.
Extracted from the documentation: Reverse Polish Notation Calculator.
calc - Simple Calculator
This example is slightly more complex than rpcalc: it features infix
operators (1 + 2, instead of 1 2 + in rpcalc), but it does so using a
unambiguous grammar of the arithmetic instead of using precedence
directives (%left, etc.).
mfcalc - Multi-Function Calculator
A more complete C example: a multi-function calculator. More complex than the previous example. Using precedence directives to support infix operators.
Extracted from the documentation: Multi-Function Calculator: mfcalc.
lexcalc - calculator with Flex and Bison
The calculator with precedence directives and location tracking. It uses Flex to generate the scanner.
reccalc - recursive calculator with Flex and Bison
This example builds on top of the previous one to provide a reentrant
parser. Such parsers can be called concurrently in different threads, or
even recursively. To demonstrate this feature, expressions in parentheses
are tokenized as strings, and then recursively parsed from the parser. So
(((1)+(2))*((3)+(4))) uses eight parsers, with a depth of four.
pushcalc - calculator implemented with a push parser
All the previous examples are so called "pull parsers": the user invokes the parser once, which repeatedly calls the scanner until the input is drained.
This example demonstrates the "push parsers": the user calls the scanner to fetch the next token, passes it to the parser, and repeats the operation until the input is drained.
This example is a straightforward conversion of the 'calc' example to the push-parser model.
bistromathic - all the bells and whistles
This example demonstrates best practices when using Bison.
- Its hand-written scanner tracks locations.
- Its interface is pure.
- It uses the
errortoken to get error recovery. - Its interface is "incremental", well suited for interaction: it uses the push-parser API to feed the parser with the incoming tokens.
- It features an interactive command line with completion based on the
parser state, based on
yyexpected_tokens. - It uses Bison's standard catalogue for internationalization of generated messages.
- It uses a custom syntax error with location, lookahead correction and token internationalization.
- It supports debug traces with semantic values.
- It uses named references instead of the traditional $1, $2, etc.