fix: crash when redefining the EOF token

Reported by Agency for Defense Development.
https://lists.gnu.org/r/bug-bison/2020-08/msg00008.html

On an empty such as

    %token FOO
           BAR
           FOO 0
    %%
    input: %empty

we crash because when we find FOO 0, we decrement ntokens (since FOO
was discovered to be EOF, which is already known to be a token, so we
increment ntokens for it, and need to cancel this).  This "works well"
when EOF is properly defined in one go, but here it is first defined
and later only assign token code 0.  In the meanwhile BAR was given
the token number that we just decremented.

To fix this, assign symbol numbers after parsing, not during parsing,
so that we also saw all the explicit token codes.  To maintain the
current numbers (I'd like to keep no difference in the output, not
just equivalence), we need to make sure the symbols are numbered in
the same order: that of appearance in the source file.  So we need the
locations to be correct, which was almost the case, except for nterms
that appeared several times as LHS (i.e., several times as "foo:
...").  Fixing the use of location_of_lhs sufficed (it appears it was
intended for this use, but its implementation was unfinished: it was
always set to "false" only).

* src/symtab.c (symbol_location_as_lhs_set): Update location_of_lhs.
(symbol_code_set): Remove broken hack that decremented ntokens.
(symbol_class_set, dummy_symbol_get): Don't set number, ntokens and
nnterms.
(symbol_check_defined): Do it.
(symbols): Don't count nsyms here.
Actually, don't count nsyms at all: let it be done in...
* src/reader.c (check_and_convert_grammar): here.  Define nsyms from
ntokens and nnterms after parsing.
* tests/input.at (EOF redeclared): New.

* examples/c/bistromathic/bistromathic.test: Adjust the traces: in
"%nterm <double> exp %% input: ...", exp used to be numbered before
input.
This commit is contained in:
Akim Demaille
2020-08-05 08:50:48 +02:00
parent 89e42ffb4b
commit b7aab2dbad
4 changed files with 53 additions and 28 deletions

View File

@@ -251,29 +251,29 @@ err: Next token is token ) (1.4: )
err: Shifting token ) (1.4: )
err: Entering state 20
err: Stack now 0 2 10 20
err: Reducing stack by rule 15 (line 151):
err: Reducing stack by rule XX (line XXX):
err: $1 = token ( (1.1: )
err: $2 = token error (1.2-3: )
err: $3 = token ) (1.4: )
err: -> $$ = nterm exp (1.1-4: 666)
err: Entering state 7
err: Stack now 0 7
err: Entering state 8
err: Stack now 0 8
err: Return for a new token:
err: Reading a token
err: Now at end of input.
err: LAC: initial context established for end of file
err: LAC: checking lookahead end of file: R2 G8 S19
err: Reducing stack by rule 2 (line 126):
err: LAC: checking lookahead end of file: R2 G7 S14
err: Reducing stack by rule XX (line XXX):
err: $1 = nterm exp (1.1-4: 666)
err: -> $$ = nterm input (1.1-4: )
err: Entering state 8
err: Stack now 0 8
err: Entering state 7
err: Stack now 0 7
err: Now at end of input.
err: Shifting token end of file (1.5: )
err: LAC: initial context discarded due to shift
err: Entering state 19
err: Stack now 0 8 19
err: Stack now 0 8 19
err: Entering state 14
err: Stack now 0 7 14
err: Stack now 0 7 14
err: Cleanup: popping token end of file (1.5: )
err: Cleanup: popping nterm input (1.1-4: )' -p