When a pstate is used for multiple successive parses, some state may
leak from one run into the following one. That was introduced in
330552ea49 "yacc.c: push: don't clear
the parser state when accepting/rejecting".
Reported by Ryan <[email protected]>
https://lists.gnu.org/r/bug-bison/2021-03/msg00000.html
* data/skeletons/yacc.c (yypush_parse): We reusing a pstate from a
previous run, do behave as if it were the first run.
* tests/push.at (Pstate reuse): Check this.
The right-shift added in c22902e360
("tables: fix handling for useless tokens") is incorrect. In
particular, we need to reset the "new" bits.
Reported by Balázs Scheidler.
https://github.com/akimd/bison/issues/74
* src/tables.c (pos_set_set): Fix the right-shift.
In some rare conditions, the generated parser can be wrong when there
are useless tokens.
Reported by Balázs Scheidler.
https://github.com/akimd/bison/issues/72
Balázs managed to prove that the bug was introduced in
commit af1c6f973a
Author: Theophile Ranquet <[email protected]>
Date: Tue Nov 13 10:38:49 2012 +0000
tables: use bitsets for a performance boost
Suggested by Yuri at
<http://lists.gnu.org/archive/html/bison-patches/2012-01/msg00000.html>.
The improvement is marginal for most grammars, but notable for large
grammars (e.g., PosgreSQL's postgre.y), and very large for the
sample.y grammar submitted by Yuri in
http://lists.gnu.org/archive/html/bison-patches/2012-01/msg00012.html.
Measured with --trace=time -fsyntax-only.
parser action tables postgre.y sample.y
Before 0,129 (44%) 37,095 (99%)
After 0,117 (42%) 5,046 (93%)
* src/tables.c (pos): Replace this set of integer coded as an unsorted
array of integers with...
(pos_set): this bitset.
which was implemented long ago, but that I installed only recently
(March 2019), first published in v3.3.90.
That patch introduces a bitset to represent a set of integers. It
managed negative integers by using a (fixed) base (the smallest
integer to represent). It avoided negative accesses into the bitset
by ignoring integers smaller than the base, under the asumption that
these cases correspond to useless tokens that are ignored anyway.
While it turns out to be true for all the test cases in the test suite
(!), Balázs' use case demonstrates that it is not always the case.
So we need to be able to accept negative integers that are smaller
than the current base.
"Amusingly" enough, the aforementioned patch was visibly unsure about
itself:
/* Store PLACE into POS_SET. PLACE might not belong to the set
of possible values for instance with useless tokens. It
would be more satisfying to eliminate the need for this
'if'. */
This commit needs several improvements in the future:
- support from bitset for bit assignment and shifts
- amortized resizing of pos_set
- test cases
* src/tables.c (pos_set_base, pos_set_dump, pos_set_set, pos_set_test):
New.
Use them instead of using bitset_set and bitset_test directly.
There were several bugs in pruning that would leave the state-item
graph in an inconsistent state which could cause crashes later on:
- Pruning now happens in one pass instead of two.
- Disabled state-items no longer prune the state-items they transition
to if that state-item has other states that transition to it.
- State-items that transition to disabled state-items are always
pruned even if they have productions.
Reported by Michal Bartkowiak <[email protected]>
https://lists.gnu.org/r/bug-bison/2021-01/msg00000.html
and Zartaj Majeed
https://github.com/akimd/bison/issues/71
* src/state-item.c (prune_forward, prune_backward): Fuse into...
(prune_state_item): this.
Adjust callers.
Currently each time we meet %merge we record this location as the
defining location (and symbol). Instead, record the first definition.
In the generated code we go from
yy0->A = merge (*yy0, *yy1);
to
yy0->S = merge (*yy0, *yy1);
where S was indeed the first symbol, and in the diagnostics we go from
glr-regr18.y:30.18-24: error: result type clash on merge function 'merge': <type2> != <type1>
30 | sym2: sym3 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
glr-regr18.y:29.18-24: note: previous declaration
29 | sym1: sym2 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
glr-regr18.y:31.13-19: error: result type clash on merge function 'merge': <type3> != <type2>
31 | sym3: %merge<merge> { $$ = 0; } ;
| ^~~~~~~
glr-regr18.y:30.18-24: note: previous declaration
30 | sym2: sym3 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
to
glr-regr18.y:30.18-24: error: result type clash on merge function 'merge': <type2> != <type1>
30 | sym2: sym3 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
glr-regr18.y:29.18-24: note: previous declaration
29 | sym1: sym2 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
glr-regr18.y:31.13-19: error: result type clash on merge function 'merge': <type3> != <type1>
31 | sym3: %merge<merge> { $$ = 0; } ;
| ^~~~~~~
glr-regr18.y:29.18-24: note: previous declaration
29 | sym1: sym2 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
where both duplicates are reported against definition 1, rather than
using definition 1 as a reference when diagnosing about definition 2,
and then 2 as a reference for 3.
* src/reader.c (record_merge_function_type): Keep the first definition.
* tests/glr-regression.at: Adjust.
Reported by Jot Dot.
https://lists.gnu.org/r/help-bison/2020-12/msg00014.html
* data/skeletons/glr.c, data/skeletons/glr2.cc (b4_call_merger): Use
the symbol's slot, not its type.
* examples/c/glr/c++-types.y: Use explicit per-symbol typing together
with api.value.type=union.
(yylex): Use yytoken_kind_t.
Don't generate C code from bison, leave that to the skeletons.
* src/output.c (merger_output): Emit invocations to b4_call_merger.
* data/skeletons/glr.c (b4_call_merger): New.
Symbols are richer than types, and in M4 it is my simpler (and more
common) to deal with symbols rather than types. So let's associate
mergers to a symbol rather than a type name.
* src/reader.h (merger_list): Replace the 'type' member by a symbol
member.
* src/reader.c (record_merge_function_type): Take a symbol as
argument, rather than a type name.
* src/output.c (merger_output): Adjust.
This macro is not exposed to users, make start it with 'YY_'.
* data/skeletons/bison.m4, data/skeletons/c.m4, data/skeletons/glr.c,
* data/skeletons/glr.cc, data/skeletons/lalr1.cc,
* src/parse-gram.c, tests/actions.at, tests/c++.at, tests/headers.at,
* tests/local.at (YYUSE): Rename as...
(YY_USE): this.
When comparing traces from different machines, the mixture of
stdout/stderr in the output are making things uselessly difficult.
* src/lssi.c, src/state-item.c: Output debug traces on stderr.
When using glr.cc, the C function yyparse is an internal detail that
should not be exposed. Users might call it by accident (I did).
* data/skeletons/glr.c (yyparse): When used for glr.cc, rename as yy_parse_impl.
* data/skeletons/glr.cc: Adjust.
The yydefgoto table uses -1 as an invalid for an impossible case (we
never use yydefgoto[0], since it corresponds to the reduction to
$accept, which never happens). Since yydefgoto is a table of state
numbers, this -1 forces a signed type uselessly, which (1) might
trigger compiler warnings when storing a value from yydefgoto into a
state number (nonnegative), and (2) wastes bits which might result in
using a int16 where a uint8 suffices.
Reported by Jot Dot <[email protected]>.
https://lists.gnu.org/r/bug-bison/2020-11/msg00027.html
* src/tables.c (default_goto): Use 0 rather than -1 as invalid value.
* tests/regression.at: Adjust.
Before:
YY_ASSERT (tok == token::YYEOF || tok == token::YYerror || tok == token::YYUNDEF || tok == 120 || tok == 49 || tok == 50 || tok == 51 || tok == 52 || tok == 53 || tok == 54 || tok == 55 || tok == 56 || tok == 57 || tok == 97 || tok == 98);
After:
YY_ASSERT (tok == token::YYEOF
|| (token::YYerror <= tok && tok <= token::YYUNDEF)
|| tok == 120
|| (49 <= tok && tok <= 57)
|| (97 <= tok && tok <= 98));
Clauses are now also wrapped on several lines. This is nicer to read
and diff, but also avoids pushing Visual C++ to its arbitrary
limits (640K and lines of 16380 bytes ought to be enough for anybody,
otherwise make an C2026 error).
The useless parens are there for the dummy warnings about
precedence (in the future, will we also have to put parens in
`1+2*3`?).
* data/skeletons/variant.hh (_b4_filter_tokens, b4_tok_in, b4_tok_in):
New.
(_b4_token_constructor_define): Use them.
Working on the previous commit I realized that YY_ASSERT was used in
the generated headers, so must follow api.prefix to avoid clashes when
multiple C++ parser with variants are used.
Actually many more macros should obey api.prefix (YY_CPLUSPLUS,
YY_COPY, etc.). There was no complaint so far, so it's not urgent
enough for 3.7.4, but it should be addressed in 3.8.
* data/skeletons/variant.hh (b4_assert): New.
Use it.
* tests/local.at (AT_YYLEX_RETURN): Fix.
* tests/headers.at: Make sure variant-based C++ parsers are checked
too.
This test did find that YY_ASSERT escaped renaming (before the fix in
this commit).
In some extreme situations (about 800 tokens), we generate a
single-line assertion long enough for Visual C++ to discard the end of
the line, thus falling into parse ends for the missing `);`. On a
shorter example:
YY_ASSERT (tok == token::TOK_YYEOF || tok == token::TOK_YYerror || tok == token::TOK_YYUNDEF || tok == token::TOK_ASSIGN || tok == token::TOK_MINUS || tok == token::TOK_PLUS || tok == token::TOK_STAR || tok == token::TOK_SLASH || tok == token::TOK_LPAREN || tok == token::TOK_RPAREN);
Whether NDEBUG is used or not is irrelevant, the parser dies anyway.
Reported by Jot Dot <[email protected]>.
https://lists.gnu.org/r/bug-bison/2020-11/msg00002.html
We should avoid emitting lines so long.
We probably should also use a range-based assertion (with extraneous
parens to pacify fascist compilers):
YY_ASSERT ((token::TOK_YYEOF <= tok && tok <= token::TOK_YYUNDEF)
|| (token::TOK_ASSIGN <= tok && ...)
But anyway, we should simply not emit this assertion at all when not
asked for.
* data/skeletons/variant.hh: Do not define, nor use, YY_ASSERT when it
is not enabled.
When generating a C parser, YYEMPTY is present in enum yytokentype but
there is no corresponding #define like there is for the other values.
There is a special case for YYEMPTY in b4_token_enums but no
corresponding case in b4_token_defines.
* data/skeletons/c.m4 (b4_token_defines): Do define YYEMPTY.
Commit af000bab11 ("doc: work around
Texinfo 6.7 bug"), published in 3.4.91, added a dependency on the
"all" target.
This is a super bad idea, since "make all" will run this
target *before* "all", which builds bison. It turns out that this new
dependency actually needed bison to be built. So all the regular
process (i) build $(BUILT_SOURCES) and then (ii) build bison, was
wrecked since some of the $(BUILT_SOURCES) depended on bison...
It was "easy" to see in the logs of "make V=1" because we were
building bison files (such as src/files.o) *before* displaying the
banner for "all-recursive". With this fix, we finally get again the
proper sequence:
rm -f examples/c/reccalc/scan.stamp examples/c/reccalc/scan.stamp.tmp
/opt/local/libexec/gnubin/mkdir -p examples/c/reccalc
touch examples/c/reccalc/scan.stamp.tmp
flex -oexamples/c/reccalc/scan.c --header=examples/c/reccalc/scan.h ./examples/c/reccalc/scan.l
mv examples/c/reccalc/scan.stamp.tmp examples/c/reccalc/scan.stamp
rm -f lib/fcntl.h-t lib/fcntl.h && \
{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
...
} > lib/fcntl.h-t && \
mv lib/fcntl.h-t lib/fcntl.h
...
mv -f lib/alloca.h-t lib/alloca.h
make all-recursive
Reported by Mingli Yu <[email protected]>.
https://github.com/akimd/bison/issues/31https://lists.gnu.org/r/bison-patches/2020-05/msg00055.html
Reported by Claudio Calvelli <[email protected]>.
https://lists.gnu.org/r/bug-bison/2020-09/msg00001.htmlhttps://bugs.gentoo.org/716516
* doc/local.mk (all): Rename as...
(all-local): this.
So that we don't compete with BUILT_SOURCES.
Reported by Dennis Clarke.
https://lists.gnu.org/r/bug-bison/2020-08/msg00013.html
* examples/d/local.mk, examples/java/calc/local.mk,
* examples/java/simple/local.mk: Pass bison's options before its
argument, in case we're in POSIX mode.
An assertion failed when the last character is a '\' and we're in a
character or a string.
Reported by Agency for Defense Development.
https://lists.gnu.org/r/bug-bison/2020-08/msg00009.html
* src/scan-gram.l: Catch unterminated escapes.
* tests/input.at (Unexpected end of file): New.
Reported by Agency for Defense Development.
https://lists.gnu.org/r/bug-bison/2020-08/msg00008.html
On an empty such as
%token FOO
BAR
FOO 0
%%
input: %empty
we crash because when we find FOO 0, we decrement ntokens (since FOO
was discovered to be EOF, which is already known to be a token, so we
increment ntokens for it, and need to cancel this). This "works well"
when EOF is properly defined in one go, but here it is first defined
and later only assign token code 0. In the meanwhile BAR was given
the token number that we just decremented.
To fix this, assign symbol numbers after parsing, not during parsing,
so that we also saw all the explicit token codes. To maintain the
current numbers (I'd like to keep no difference in the output, not
just equivalence), we need to make sure the symbols are numbered in
the same order: that of appearance in the source file. So we need the
locations to be correct, which was almost the case, except for nterms
that appeared several times as LHS (i.e., several times as "foo:
..."). Fixing the use of location_of_lhs sufficed (it appears it was
intended for this use, but its implementation was unfinished: it was
always set to "false" only).
* src/symtab.c (symbol_location_as_lhs_set): Update location_of_lhs.
(symbol_code_set): Remove broken hack that decremented ntokens.
(symbol_class_set, dummy_symbol_get): Don't set number, ntokens and
nnterms.
(symbol_check_defined): Do it.
(symbols): Don't count nsyms here.
Actually, don't count nsyms at all: let it be done in...
* src/reader.c (check_and_convert_grammar): here. Define nsyms from
ntokens and nnterms after parsing.
* tests/input.at (EOF redeclared): New.
* examples/c/bistromathic/bistromathic.test: Adjust the traces: in
"%nterm <double> exp %% input: ...", exp used to be numbered before
input.
* cfg.mk (_space_before_paren_exempt): Be less laxist.
* src/output.c, src/reader.c: Fix space before paren issues.
Pacify the warnings where applicable.
Older versions of GCC (4.1.2 here) don't like repeated typedefs.
CC src/bison-parse-simulation.o
src/parse-simulation.c:61: error: redefinition of typedef 'parse_state'
src/parse-simulation.h:74: error: previous declaration of 'parse_state' was here
make: *** [Makefile:7876: src/bison-parse-simulation.o] Error 1
Reported by Nelson H. F. Beebe.
* src/parse-simulation.c (parse_state): Don't typedef,
parse-simulation.h did it already.
This reverts commit d0bec3175f (which
should have read "We have a clash...", not "With have a clash...").
Now that `max()` was renamed `max_int()`, we can use `max` again, as
elsewhere in the code.
* src/counterexample.c (visited_hasher): Alpha reconversion.
Reported by Maarten De Braekeleer.
https://lists.gnu.org/r/bison-patches/2020-07/msg00080.html
We don't want to use gnulib's min and max macros, since we use
function calls in min/max arguments.
* src/location.c (max_int, min_int): Move to...
* src/system.h: here.
* src/counterexample.c, src/derivation.c: Use max_int instead of max.
From
foo.y:1.7-11: error: %type redeclaration for bar
1 | %type <foo> bar bar
| ^~~~~
foo.y:1.7-11: note: previous declaration
1 | %type <foo> bar bar
| ^~~~~
to
foo.y:1.17-19: error: %type redeclaration for bar
1 | %type <foo> bar bar
| ^~~
foo.y:1.13-15: note: previous declaration
1 | %type <foo> bar bar
| ^~~
* src/symlist.h, src/symlist.c (symbol_list_type_set): There's no need
for the tag's location, use that of the symbol.
* src/parse-gram.y: Adjust.
* tests/input.at: Adjust.
We crash if the input contains a string containing a NUL byte.
Reported by Suhwan Song.
https://lists.gnu.org/r/bug-bison/2020-07/msg00051.html
* src/flex-scanner.h (STRING_FREE): Avoid accidental use of
last_string.
* src/scan-gram.l: Don't call STRING_FREE without calling
STRING_FINISH first.
* tests/input.at (Invalid inputs): Check that case.
The original Texinfo macros introducing colors were made for
diagnostics, which are printed in bold. So by copy-paste accident the
styles we introduced for counterexamples were also in bold. They
should not.
* doc/bison.texi: Separate the styling of diagnostics from the styling
for counterexamples.
Don't use bold in the latter case.
With have a clash with the "max" function.
src/counterexample.c: In function 'visited_hasher':
src/counterexample.c:720:48: error: declaration of 'max' shadows a global declaration [-Werror=shadow]
src/counterexample.c:116:12: error: shadowed declaration is here [-Werror=shadow]
* src/counterexample.c (visited_hasher): Alpha conversion.
Fix 6b78e50cef, "cex: make "rerun with
'-Wcex'" a note instead of a warning"
* tests/conflicts.at (-W versus %expect and %expect-rr): Fix
expectations.
Unfortunately I found no way to use the ↳ glyph in Texinfo, so I used
@arrow{} instead, which has a different width, so we have to have all
the examples doubled, once for TeX, another for the rest of the world.
* doc/bison.texi: Use the current display in the examples.
* doc/calc.y, doc/ids.y, doc/if-then-else.y, doc/sequence.y: New.
Currently the suggestion to rerun is a -Wother warning:
warning: 2 shift/reduce conflicts [-Wconflicts-sr]
warning: rerun with option '-Wcounterexamples' to generate conflict counterexamples [-Wother]
Instead, let's attach it as a subnote of the diagnostic (in the
current case, -Wconflicts-sr):
warning: 2 shift/reduce conflicts [-Wconflicts-sr]
note: rerun with option '-Wcounterexamples' to generate conflict counterexamples
* src/conflicts.c (conflicts_print): Do that.
Adjust the test suite.
From
input.y: warning: reduce/reduce conflict on token $end [-Wcounterexamples]
Example: A b .
First derivation
a
`-> A b .
Second derivation
a
`-> A b
`-> b .
to
input.y: warning: reduce/reduce conflict on token $end [-Wcounterexamples]
Example: A b .
First reduce derivation
a
`-> A b .
Second reduce derivation
a
`-> A b
`-> b .
* src/counterexample.c (print_counterexample): here.
Compute the width of the labels to properly align the values.
* tests/conflicts.at, tests/counterexample.at, tests/diagnostics.at,
* tests/report.at: Adjust.
Now that the derivation is no longer printed on one line, aligning the
example and the derivation is no longer useful. It can actually be
harmful, as it makes the overall structure less clear.
* src/derivation.h, src/derivation.c (derivation_print_leaves): Remove
the `prefix` argument.
* src/counterexample.c (print_counterexample): Put the example next to
its label.
* tests/conflicts.at, tests/counterexample.at, tests/diagnostics.at,
* tests/report.at: Adjust.
Now that we use complain, the "sections" are clearer.
* src/counterexample.c (print_counterexample): Use the empty line only
in reports.
* tests/counterexample.at, tests/diagnostics.at, tests/report.at: Adjust.
This is more consistent, and brings benefits: users know that these
diagnostics are attached to -Wcounterexamples, and they can also click
on the hyperlink if permitted by their terminal.
We go from
warning: 1 reduce/reduce conflict [-Wconflicts-rr]
Reduce/reduce conflict on token $end:
Example A b .
First derivation a -> [ A b . ]
Second derivation a -> [ A b -> [ b . ] ]
to
warning: 1 reduce/reduce conflict [-Wconflicts-rr]
input.y: warning: reduce/reduce conflict on token $end [-Wcounterexamples]
Example A b .
First derivation a -> [ A b . ]
Second derivation a -> [ A b -> [ b . ] ]
with an hyperlink on -Wcounterexamples.
* src/counterexample.c (counterexample_report_reduce_reduce):
Use complain.
* tests/counterexample.at, tests/diagnostics.at, tests/report.at:
Adjust.
* src/complain.c (begin_hyperlink, end_hyperlink): New.
(warnings_print_categories): Use them.
* tests/local.at (AT_SET_ENV): Disable hyperlinks in the tests, they
contain random id's, and brackets (which is not so nice for M4).
Unfortunately Texinfo somewhat mangles anchors such as `-Werror` into
`g_t_002dWerror`, so let's not include the dash.
* doc/bison.texi (Diagnostics): here.
The code was written on top of buffers of `char[26]`, and then was
changed to use `char *`, yet was still using `sizeof buf`, which
became `sizeof (char *)` instead of `sizeof (char[26])`.
Reported by Dagobert Michelsen.
https://lists.gnu.org/r/bug-bison/2020-07/msg00023.html
* src/glyphs.h, src/glyphs.c: Get rid of uses of `char *`, use only
glyph_buffer_t.
Sometimes, understanding the derivations is difficult, because they
are serialized to fit in one line. For instance, the example taken
from the NEWS file:
%token ID
%%
s: a ID
a: expr
expr: expr ID ',' | "expr"
gave
First example expr • ID ',' ID $end
Shift derivation $accept → [ s → [ a → [ expr → [ expr • ID ',' ] ] ID ] $end ]
Second example expr • ID $end
Reduce derivation $accept → [ s → [ a → [ expr • ] ID ] $end ]
Printing as trees, it gives:
First example expr • ID ',' ID $end
Shift derivation
$accept
↳ s $end
↳ a ID
↳ expr
↳ expr • ID ','
Second example expr • ID $end
Reduce derivation
$accept
↳ s $end
↳ a ID
↳ expr •
* src/glyphs.h, src/glyphs.c (down_arrow, empty, derivation_separator):
New.
* src/derivation.c (derivation_print, derivation_print_impl): Rename
as...
(derivation_print_flat, derivation_print_flat_impl): These.
(fputs_if, derivation_depth, derivation_width, derivation_print_tree)
(derivation_print_tree_impl, derivation_print): New.
* src/counterexample.c (print_counterexample): Adjust.
* tests/conflicts.at, tests/counterexample.at, tests/diagnostics.at,
* tests/report.at: Adjust.
When reporting counterexamples for s/r conflicts, put the shift first.
This is more natural, and displays the default resolution first, which
is also what happens for r/r conflicts where the smallest rule number
is displayed first, and "wins".
* src/counterexample.c (counterexample): Add a shift_reduce member.
(new_counterexample): Adjust.
Swap the derivations when this is a s/r conflict.
(print_counterexample): For s/r conflicts, prefer "Shift derivation"
and "Reduce derivation" rather than "First/Second derivation".
* tests/conflicts.at, tests/counterexample.at, tests/report.at: Adjust.
* NEWS, doc/bison.texi: Ditto.
Currently we use both names. Let's stick to the short one.
* src/AnnotationList.c, src/conflicts.c, src/counterexample.c,
* src/getargs.c, src/getargs.h, src/graphviz.c, src/ielr.c,
* src/lalr.c, src/print-graph.c, src/print-xml.c, src/print.c,
* src/state-item.c, src/state.c, src/state.h, src/tables.c:
s/lookahead_token/lookahead/gi.
* src/counterexample.c, src/parse-simulation.c: It is more usual in
Bison to use sizeof on expressions than on types, especially for
allocation.
Let the compiler do it's job instead of calling memcpy ourselves.
There are too many gl_list_t in there, it's hard to understand what is
going on. Introduce and use more precise types. I sure can be wrong
in some places, it's hard to tell without proper tool support.
* src/counterexample.c, src/lssi.c, src/lssi.h, src/parse-simulation.c,
* src/parse-simulation.h, src/state-item.c, src/state-item.h
(si_bfs_node_list, search_state_list, ssb_list, lssi_list)
(state_item_list): New.
It does not make a lot of sense to use ::= in our counterexamples,
that's not something that belongs to the Bison "vocabulary". Using
the colon makes sense, but it's too discreet. Let's use the arrow,
which we already use in some reports (HTML and Dot).
* src/gram.h (print_dot_fallback): Generalize into...
(print_fallback): this.
(print_arrow): New.
* src/derivation.c: Use it.
* NEWS, tests/conflicts.at, tests/counterexample.at,
* tests/diagnostics.at, tests/report.at: Adjust.
* doc/bison.texi: Ditto.
Unfortunately the literal `→` is output as `↦`. So we need to use
@arrow.
Prefer `&foos[i]` to `foos + i` when `foos` is an array. IMHO, it
makes the semantics clearer.
* src/counterexample.c, src/lssi.c, src/parse-simulation.c,
* src/state-item.c: With arrays, prefer the array notation rather than
the pointer one.
* examples/c/bistromathic/parse.y (user_context): We need the current
line.
(yyreport_syntax_error): Quote the guilty line, with squiggles.
* examples/c/bistromathic/bistromathic.test: Adjust.
Currently autocompletion on a line with errors leaks the error
messages. It can be useful to let the user know, but GNU Readline
does not provide us with an nice way to display the error. So we
actually break into the current line of the user.
So instead, do not show these errors.
* examples/c/bistromathic/parse.y (user_context): New.
Use %param to pass it to the parser and scanner.
Keep quiet when in computing autocompletion.
That quite defeats the whole point of locations... But anyway, we
should not see these messages at all.
* examples/c/bistromathic/parse.y (expected_tokens): Fix (useless)
location tracking.
Currently when we output useless rules, they appear before the
grammar, but using the same invocation. As a result, the anchor is
defined twice, and the wrong one, being first, is honored.
* data/xslt/xml2xhtml.xsl (rule): Take a new 'anchor' parameter to
decide whether being an anchor, or a target.
Let it be true when output the grammar.
* tests/report.at: Adjust.
The text and Dot reports are expected to be identical when generated
directly (--report, --graph) or indirectly (via XML). The xml
testsuite had not be run for ages, let it catch up a bit.
* src/print-xml.c: Pass the type of the symbols.
* data/xslt/xml2text.xsl
Catch up with the new layout.
Display the symbol types.
Use '•', not '.'
* tests/local.at: Smash '•' to '.' when matching against the direct
text report.
* tests/report.at: Adjust XML expectations.
* data/skeletons/c++.m4, data/skeletons/glr.c, data/skeletons/lalr1.d,
* data/skeletons/lalr1.java, data/skeletons/yacc.c:
Be more accurate about yychar and yytoken.
Don't name local variables as if they were members.
* examples/c/bistromathic/parse.y, examples/c/lexcalc/parse.y,
* examples/c/reccalc/parse.y: here.
Add some comments.
* src/parse-gram.y (api_version): Pull out of handle_require.
Bump to 3.7.
* data/bison-default.css: Cobalt does not seem to be supported.
* doc/bison.texi (Counterexamples): A new section.
(Understanding): Show the counterexamples as it shows in the report:
with its items.
(Bison Options): Document -Wcex and -rcex.
In commit c80cdf2db2 ("doc: simplify
uses of @ref", Jan 27 2020, released in Bison 3.6), I broke the
references to the Bibliography. For instance:
For a more detailed exposition of the mysterious behavior in LALR parsers
-and the benefits of IELR, @pxref{Bibliography,,Denny 2008 March}, and
-@ref{Bibliography,,Denny 2010 November}.
+and the benefits of IELR, @pxref{Bibliography}, and
+@ref{Bibliography}.
which results in "see Bibliography" twice, instead of the more precise
reference.
* doc/bison.texi (@pcite, @tcite): New.
Use them instead of @ref to Bibliography.
Cite only the first author (that's what we did for the other entries).
Currently when a push parser finishes its parsing (i.e., it did not
return YYPUSH_MORE), it also clears its state. It is therefore
impossible to see if it had parse errors.
In the context of autocompletion, because error recovery might have
fired, the parser is actually already in a different state. For
instance on `(1 + + <TAB>` in the bistromathic, because there's a
`exp: "(" error ")"` recovery rule, `1 + +` tokens have already been
popped, replaced by `error`, and autocompletions think we are ready
for the closing ")". So here, we would like to see if there was a
syntax error, yet `yynerrs` was cleared.
In the case of a successful parse, we still have a problem: if error
recovery succeeded, we won't know it, since, again, `yynerrs` is
clearer.
It seems much more natural to leave the parser state available for
analysis when there is a failure.
To reuse the parser, we should either:
1. provide an explicit means to reinitialize a parser state for future
parses.
2. automatically reset the parser state when it is used in a new
parse.
Option 2 requires to check whether we need to reinitialize the parser
each time we call `yypush_parse`, i.e., each time we give a new token.
This seems expensive compared to Option 1, but benchmarks revealed no
difference. Option 1 is incompatible with the documentation
("After `yypush_parse` returns a status other than `YYPUSH_MORE`, the
parser instance `yyps` may be reused for a new parse.").
So Option 2 wins, reusing the private `yynew` member to record that a
parse was finished, and therefore that the state must reset in the
next call to `yypull_parse`.
While at it, this implementation now reuses the previously enlarged
stacks from one parse to another.
* data/skeletons/yacc.c (yypstate_new): Set up the stacks in their
initial configurations (setting their bottom to the stack array), and
use yypstate_clear to reset them (moving their top to their bottom).
(yypstate_delete): Adjust.
(yypush_parse): At the beginning, clear yypstate if needed, and at the
end, record when yypstate needs to be clearer.
* examples/c/bistromathic/parse.y (expected_tokens): Do not propose
autocompletion when there are parse errors.
* examples/c/bistromathic/bistromathic.test: Check that case.
Currently, completion when there is a syntax error shows broken
locations.
* examples/c/bistromathic/parse.y (expected_tokens): Initialize the
location.
* examples/c/bistromathic/bistromathic.test: Check that.
The previous commit ("yacc.c: declare and initialize and the same
time") made b4_initialize_parser_state_variables useless.
* data/skeletons/yacc.c (b4_initialize_parser_state_variables): Inline
into...
(yypstate_clear): here.
In order to factor the code of push and pull parsers, the declaration
of the parser's state variable was common (being local variable in
pull parsers, and struct members in push parsers). This result in
rather poor style in pull parser, with first variable declarations,
and then their initializations.
The initialization is about to differ between push and pull parsers,
so it is no longer worth keeping both cases together.
* data/skeletons/yacc.c (b4_declare_parser_state_variables): Accept an
argument, and when it is set, initialize the variables.
Adjust dependencies.
Currently yypull_parse takes a yypstate* as argument, and accepts it
to be NULL. This does not seem to make a lot of sense: rather it is
its callers that should do that.
I believe this is historical: yypull_parse was introduced
first (c3d503425f), with yyparse being a
macro. So yyparse could hardly deal with memory allocation properly.
In 7172e23e8f that yyparse was turned
into a genuine function. At that point, it should have allocated its
own yypstate*, which would have left yypull_parse deal with only one
single non-null ypstate* argument.
Fortunately, it is nowhere documented that it is valid to pass NULL to
yypull_parse. It is now forbidden.
* data/skeletons/yacc.c (yypull_parse): Don't allocate a yypstate.
Needs a location to issue the error message.
(yyparse): Allocate the yypstate.
* README-hacking.md (Working from the Repository): Make it first to
make it easier to find the instructions to build from the repo.
(Implementation Notes): New.
* README: Provide more links.
Reported by Martin Blais and Yuriy Solodkyy.
https://lists.gnu.org/r/help-bison/2020-05/msg00011.htmlhttps://lists.gnu.org/r/bug-bison/2020-06/msg00038.html
While at it, modernize filename_type as api.filename.type and document
it properly.
* data/skeletons/c++.m4 (filename_type): Rename as...
(api.filename.type): this.
Default to const std::string.
* data/skeletons/location.cc (position, location): Expose the
filename_type type.
Use api.filename.type.
* doc/bison.texi (%define Summary): Document api.filename.type.
(C++ Location Values): Document position::filename_type.
* src/muscle-tab.c (muscle_percent_variable_update): Ensure backward
compatibility.
* tests/c++.at: Check that using const file names is ok.
tests/input.at: Check backward compat.
Reported by Dwight Guth.
https://lists.gnu.org/r/bug-bison/2020-06/msg00037.html
* src/AnnotationList.c (AnnotationList__computePredecessorAnnotations):
Beware that SBITSET__FOR_EACH nests _two_ for-loops, so "break" does
not actually break out of it.
That was the only occurrence in the code.
* src/Sbitset.h (SBITSET__FOR_EACH): Warn passersby.
* src/AnnotationList.c, src/ielr.c: Fix include order.
Prefer `res` to `result`.
Reduce scopes.
Be free of the oldish 76 cols limitation when it clutters too much the
code.
Denest when possible (we're starving for horizontal width).
AFAICT, "dotted rule" is a more frequent synonym of "item" than
"pointed rule". So let's migrate to using "dot" only.
* doc/bison.texi: Use dot/'•' rather than point/'.'.
* src/print-xml.c (print_core): Use dot rather than point. This is
not backward compatible, but AFAICT, we don't have actual user of the
XML output (but ourselves). So...
* data/xslt/xml2dot.xsl, data/xslt/xml2text.xsl,
* data/xslt/xml2xhtml.xsl, tests/report.at: ... adjust.
It makes no sense, and is actually confusing, to display twice the
same example with no visible difference.
* src/complain.h, src/complain.c (is_styled): New.
* src/counterexample.c (print_counterexample): Display the unified
example a second time only if it makes a difference.
* tests/conflicts.at, tests/counterexample.at, tests/report.at: Adjust.
* tests/diagnostics.at: Make sure we do display the unifying examples
twice when colors are enabled. And check those colors.
I implemented this to print A ::= [ ], but A ::= [ %empty ] might be
clearer.
* src/parse-simulation.c (nullable_closure): Don't generate null
nonterminal derivations as leaves.
* src/derivation.c (derivation_print_impl): Don't print seperator
spaces for null nonterminal.
* tests/counterexample.at: Update test results.
This was a hack to make it easier for people to migrate from yacc.c to
lalr1.cc and from glr.c to glr.cc: when set, YYSTYPE and YYLTYPE were
`#defined`. It was never documented (just mentioned in NEWS for Bison
2.2, 2006-05-19), but was used to simplify the test suite. Stop that:
adjust the test suite to the skeletons, not the converse.
In C++ use yy::parser::semantic_type, yy::parser::location_type, and
yy::parser::token::MY_TOKEN, instead of YYSTYPE, YYLTYPE and MY_TOKEN.
* data/skeletons/glr.cc, data/skeletons/lalr1.cc: Remove its support.
* tests/actions.at, tests/c++.at, tests/calc.at: Adjust.
Use of print_unicode_char suggested by Bruno Haible.
https://lists.gnu.org/r/bug-gettext/2020-06/msg00012.html
* src/gram.h (print_dot_fallback, print_dot): New.
* src/gram.c, src/derivation.c: Use it.
* tests/counterexample.at, tests/report.at: Adjust the test suite.
* .travis.yml, README-hacking.md: Adjust.
And let --report=all include the counterexamples.
* src/getargs.h, src/getargs.c (report_cex): New.
* src/main.c: Compute counterexamples when -rcex is specified.
* src/print.c: Include the counterexamples when -rcex is specified.
* tests/conflicts.at, tests/existing.at, tests/local.at: Adjust.
Instead of
Shift/reduce conflict on token D:
Example A a • D
First derivation s ::=[ A a a ::=[ b ::=[ c ::=[ • ] ] ] d ::=[ D ] ]
Example A a • D
Second derivation s ::=[ A a d ::=[ • D ] ]
display
Shift/reduce conflict on token D:
Example A a • D
First derivation s ::=[ A a a ::=[ b ::=[ c ::=[ • ] ] ] d ::=[ D ] ]
Example A a • D
Second derivation s ::=[ A a d ::=[ • D ] ]
* src/counterexample.c (print_counterexample): Indent.
* tests/counterexample.at: Adjust.
Showing the items (with the state numbers) is really something we
should restrict to the report.
* src/counterexample.c (counterexample_report_shift_reduce)
(counterexample_report_reduce_reduce): Don't show the pointed rules,
we will do that in the report.
* tests/counterexample.at: Adjust.
* src/conflicts.c (find_state_item_number, report_state_counterexamples):
Move to...
* src/counterexample.h, src/counterexample.c (find_state_item_number)
(counterexample_report_state): this.
Add support for `out` as an argument.
(counterexample_report_reduce_reduce, counterexample_report_shift_reduce):
Accept an `out` argument, and be static.
* upstream/maint:
maint: post-release administrivia
version 3.6.4
glr.cc: don't leak glr.c/glr.cc scaffolding to the user
Some fixes were needed to adjust to recent changes in glr.cc and
glr.c.
* data/skeletons/glr.cc: Stop messing with the user's epilogue to
insert glr.cc code. We need that code to be inserted _before_ the
user's epilogue, not after. So define b4_glr_cc_pre_epilogue.
* data/skeletons/glr.c: Use it.
Until we have a decent reimplementation of glr.cc, we have to use
tricks to shoehorn C++ symbols to the C engine of glr.c. Some of them
are done via #define. Unfortunately in Bison 3.6 some of these we
done in the header file, which broke valid user code.
Reported by Egor Pugin.
https://lists.gnu.org/r/bug-bison/2020-06/msg00003.html
* data/skeletons/glr.cc: Stop playing tricks with b4_pre_epilogue.
(b4_glr_cc_setup, b4_glr_cc_cleanup): New.
Much cleaner way to instal glr.cc's scaffolding around glr.c.
* data/skeletons/glr.c: Adjust to use them.
From
"number" shift, and go to state 1
"Ñùṃéℝô" shift, and go to state 2
to
"number" shift, and go to state 1
"Ñùṃéℝô" shift, and go to state 2
* src/print.c: Use mbswidth, not strlen, to compute visual columns.
* tests/report.at: Adjust.
Currently we use "quotearg" to escape the strings output in Dot. As a
result, if the user's locale is C for instance, all the non-ASCII are
escaped. Unfortunately graphviz does not interpret this style of
escaping.
For instance:
5 -> 2 [style=solid label="\"\303\221\303\271\341\271\203\303\251\342\204\235\303\264\""]
was displayed as a sequence of numbers. We now output:
5 -> 2 [style=solid label="\"Ñùṃéℝô\""]
independently of the user's locale.
* src/system.h (obstack_backslash): New.
* src/graphviz.h, src/graphviz.c (escape): Remove, use
obstack_backslash instead.
* src/print-graph.c: Likewise.
* tests/report.at: Adjust.
Currently our scanner decodes all the escapes in the strings, and we
later reescape the strings when we emit them.
This is troublesome, as we do not respect the user input. For
instance, when the user writes in UTF-8, we destroy her string when we
write it back. And this shows everywhere: in the reports we show the
escaped string instead of the actual alias:
0 $accept: . exp $end
1 exp: . exp "\342\212\225" exp
2 | . exp "+" exp
3 | . exp "+" exp
4 | . "number"
5 | . "\303\221\303\271\341\271\203\303\251\342\204\235\303\264"
"number" shift, and go to state 1
"\303\221\303\271\341\271\203\303\251\342\204\235\303\264" shift, and go to state 2
This commit preserves the user's exact spelling of the string aliases,
instead of interpreting the escapes and then reescaping. The report
now shows:
0 $accept: . exp $end
1 exp: . exp "⊕" exp
2 | . exp "+" exp
3 | . exp "+" exp
4 | . "number"
5 | . "Ñùṃéℝô"
"number" shift, and go to state 1
"Ñùṃéℝô" shift, and go to state 2
Likewise, the XML (and therefore HTML) outputs are fixed.
* src/scan-gram.l (STRING, TSTRING): Do not interpret the escapes in
the resulting string.
* src/parse-gram.y (unquote, parser_init, parser_free, unquote_free)
(handle_defines, handle_language, obstack_for_unquote): New.
Use them to unquote where needed.
* tests/regression.at, tests/report.at: Update.
This is to record the current state of the report, which escapes the
UTF-8 characters (as parse.error="verbose" does), but shouldn't (as
parse.error="detailed" does).
* tests/report.at: here.
Suggesting -Wcounterexamples when there are conflicts is probably not
what the user wants. If she knows her conflicts and has set
%expect/%expect-rr appropriately, we shouldn't warn.
The commit also swaps the counterexamples and the report of conflicts,
into, IMHO, a more natural order: from
Shift/reduce conflict on token B:
1: 3 a: A .
1: 8 y: A . B
Example A • B C
First derivation s ::=[ a ::=[ A • ] x ::=[ B C ] ]
Example A • B C
Second derivation s ::=[ y ::=[ A • B ] c ::=[ C ] ]
input.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
input.y:4.4: warning: rule useless in parser due to conflicts [-Wother]
to
input.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
Shift/reduce conflict on token B:
1: 3 a: A .
1: 8 y: A . B
Example A • B C
First derivation s ::=[ a ::=[ A • ] x ::=[ B C ] ]
Example A • B C
Second derivation s ::=[ y ::=[ A • B ] c ::=[ C ] ]
input.y:4.4: warning: rule useless in parser due to conflicts [-Wother]
* src/conflicts.c (rule_conflicts_print): Rename as...
(report_rule_expectation_mismatches): this.
Move the handling of report_counterexamples to...
(conflicts_print): Here.
Display this warning when applicable.
Plural vs. singular is always a problem...
But we already have conflicts-sr and conflicts-rr, so counterexamples
makes more sense than counterexample. Besides, -Wcounterexample will
still be accepted as an unambiguous prefix of -Wcounterexamples.
Add -Wcex as a convenient alias.
While at it, use only "counterexample", never "counter example".
* src/complain.h, src/complain.c
(Wcounterexample, warning_counterexample): Rename as...
(Wcounterexamples, warning_counterexamples): these.
(argmatch_warning_docs): Rename -Wcounterexample as -Wcounterexamples.
(argmatch_warning_args): Likewise.
Add support for -Wcex.
Adjust dependencies.
While defining api.header.include worked as expected, its default
value was incorrectly defined. As a result, by default, the generated
parsers still duplicated the content of the generated header instead
of including it.
* data/skeletons/yacc.c (api.header.include): Fix its default value.
* tests/output.at: Check it.
* doc/bison.texi (%define Summary): Document api.header.include.
While at it, move the definition of api.namespace at the proper
place.
Use colors to show the counterexamples and the derivations in color,
to highlight their structure. Align the outputs, and add i18n
support. Reduce width by using a one-space separator instead of
two-space.
From
Example A • B C
First derivation s ::=[ a ::=[ A • ] x ::=[ B C ] ]
Second derivation s ::=[ y ::=[ A • B ] c ::=[ C ] ]
to
Example A • B C
First derivation s ::=[ a ::=[ A • ] x ::=[ B C ] ]
Example A • B C
Second derivation s ::=[ y ::=[ A • B ] c ::=[ C ] ]
with colors.
* data/bison-default.css (cex-dot, cex-0, cex-1, cex-2, cex-3, cex-4)
(cex-5, cex-6, cex-7, cex-step, cex-leaf): New.
* src/derivation.c (derivation_print_styled_impl): New.
(derivation_print, derivation_print_leaves): Use it.
* src/counterexample.c: Reformat the output.
* tests/counterexample.at: Adjust.
It's unfortunate that the traditions between formal language theory
and Yacc differs, but here, tokens should be upper case, and
nonterminals should be lower case.
* tests/counterexample.at: Comply with this.
In Bison we refer to "shift/reduce" conflicts, not "shift-reduce" (in
Bison 3.6.3 186 occurrences vs 15). Enforce consistency on this.
Instead of "spending" a second line for each conflict to report the
lookaheads, put that on the same line as the type of conflict. Also,
prefer "token" to "symbol". Maybe we should even prefer "lookahead".
While at it, enable internationalization, with plurals where
appropriate.
As a consequence, instead of
Shift-Reduce Conflict:
6: 3 b: . %empty
6: 6 d: c . A
On Symbol: A
display
Shift/reduce conflict on token A:
6: 3 b: . %empty
6: 6 d: c . A
* NEWS, doc/bison.texi, src/conflicts.c: Spell it "shift/reduce", not
"shift-reduce".
* src/counterexample.c (counterexample_report_shift_reduce)
(counterexample_report_reduce_reduce): Reformat and internationalize
output.
* tests/counterexample.at: Adjust expectations.
For instance, in the case of Bison's own parser:
- case 40:
+ case 40: /* grammar_declaration: "%code" "identifier" "{...}" */
{
muscle_percent_code_grow ((yyvsp[-1].ID), (yylsp[-1]),
translate_code_braceless ((yyvsp[0].BRACED_CODE), (yylsp[0])),
(yylsp[0]));
code_scanner_last_string_free ();
}
break;
* data/skeletons/c.m4: Modified.
* data/skeletons/d.m4: Modified.
* data/skeletons/java.m4: Modified.
* src/output.c (output_escaped): New.
(quoted_output): Use it, and rename as...
(output_quoted): this.
Adjust dependencies.
(rule_output): New.
(user_actions_output): Use it.
* data/skeletons/c.m4, data/skeletons/d.m4, data/skeletons/java.m4
(b4_case): Add support for $3, an optional comment.
We were still using GCC9, because GCC10 was failing.
* .travis.yml (PPC64le): Use GCC10.
While at it, use -O2 instead of -O3: it's certainly nicer for the
CPUs, and allows to test different sets of compiler flags (we use -O3
in several other configurations).
* upstream/maint:
maint: post-release administrivia
version 3.6.3
build: check -Wmissing-prototypes
tests: show logs
c++: fix printing of state number on streams
First, we should avoid code such as
gl_list_iterator_t it = gl_list_iterator (deriv->children);
derivation *child = NULL;
while (gl_list_iterator_next (&it, (const void **) &child, NULL))
{
derivation_print (child, f);
because of -Wstrict-aliasing (whose job is to catch type-punning
issues). See https://lists.gnu.org/r/bug-bison/2020-05/msg00039.html.
Rather we need
gl_list_iterator_t it = gl_list_iterator (deriv->children);
const void **p = NULL;
while (gl_list_iterator_next (&it, &p, NULL))
{
derivation *child = (derivation *) p;
derivation_print (child, f);
Second, list iterators actually have destructors. Even though they
are noop in the case of linked-lists, we should use them.
Let's address both issues with typed wrappers (such as
derivation_list_next) that take care of both issues, and besides allow
to scope the iterators within the loop:
derivation *child;
for (gl_list_iterator_t it = gl_list_iterator (deriv->children);
derivation_list_next (&it, &child);
)
{
derivation_print (child, f);
* src/derivation.h, src/derivation.c (derivation_list_next): New.
Use it where appropriate.
* src/counterexample.c (search_state_list_next): New.
Use it where appropriate.
* src/parse-simulation.h, src/parse-simulation.c
* src/state-item.h (state_item_list_next): New.
Use it where appropriate.
pstate_clear is lacking a prototype.
Reported by Ryan
https://lists.gnu.org/r/bug-bison/2020-05/msg00101.html
Besides, none of the C examples were compiled with the warning flags.
* configure.ac (warn_c): Add -Wmissing-prototypes.
* data/skeletons/yacc.c (pstate_clear): Make it static.
* examples/local.mk (TEST_CFLAGS): New.
* examples/c/bistromathic/local.mk, examples/c/calc/local.mk,
* examples/c/lexcalc/local.mk, examples/c/mfcalc/local.mk,
* examples/c/pushcalc/local.mk, examples/c/reccalc/local.mk,
* examples/c/rpcalc/local.mk:
Use it.
GCC's warn_unused_result is not silenced by a cast to void, so we have
to "use" scanf's result.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25509https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66425
Flex generated code produces too many warnings, including things such
as, with ICC:
examples/c/lexcalc/scan.c(1088): error #1682: implicit conversion
of a 64-bit integral type to a smaller integral type (potential portability problem)
2259 YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),
2260 ^
2261
2262
I am tired of trying to fix Flex's output. The project does not seem
maintained. We ought to avoid it. So, for the time being, don't try
to enable warnings with Flex.
* examples/c/bistromathic/parse.y, examples/c/reccalc/scan.l: Fix
warnings.
* doc/bison.texi: Discard scanf's return value to defeat
-Werror=unused-result.
item_number is used for elements of ritem as well as indices into
ritem which is fairly confusing. Introduce item_index to represent
indices into ritem.
* src/gram.h (item_index): Introduce it for ritem indices.
* src/closure.h, src/closure.c, src/ielr.c, src/lr0.c,
* src/print-graph.c, src/state.h, src/state.h:
Replace uses of item_number with item_index where appropriate.
Teaches bison about a new command line option, --file-prefix-map OLD=NEW
(based on the -ffile-prefix-map option from GCC) which causes it to
replace and file path of OLD in the text of the output file with NEW,
mainly for header guards and comments. The primary use of this is to
make builds reproducible with different input paths, and in particular
the debugging information produced when the source code is compiled. For
example, a distro may know that the bison source code will be located at
"/usr/src/bison" and thus can generate bison files that are reproducible
with the following command:
bison --output=/build/bison/parse.c -d --file-prefix-map=/build/bison/=/usr/src/bison/ parse.y
Importantly, this will change the header guards and #line directives
from:
#ifndef YY_BUILD_BISON_PARSE_H
#line 100 "/build/bison/parse.h"
to
#ifndef YY_USR_SRC_BISON_PARSE_H
#line 100 "/usr/src/bison/parse.h"
which is reproducible.
See https://lists.gnu.org/r/bison-patches/2020-05/msg00016.html
Signed-off-by: Joshua Watt <[email protected]>
* src/files.h, src/files.c (spec_mapped_header_file)
(mapped_dir_prefix, map_file_name, add_prefix_map): New.
* src/getargs.c (-M, --file-prefix-map): New option.
* src/output.c (prepare): Define b4_mapped_dir_prefix and
b4_spec_header_file.
* src/scan-skel.l (@ofile@): Output the mapped file name.
* data/skeletons/glr.c, data/skeletons/glr.cc,
* data/skeletons/lalr1.cc, data/skeletons/location.cc,
* data/skeletons/yacc.c:
Adjust.
* doc/bison.texi: Document.
* tests/input.at, tests/output.at: Check.
Instead of `On Symbols: {b,c,}`, display `On Symbols: b, c`.
* src/counterexample.c (counterexample_report_reduce_reduce): We don't
need braces.
Use commas as a separator, not a terminator.
* tests/counterexample.at: Adjust.
Instead of generating switch statements with numbers, let's use the
symbol kinds. Not only is this more readable, it also makes reading
diff easier, as a change in symbol numbers won't have such a large
effect on the implementation of symbol actions.
* data/skeletons/bison.m4 (_b4_symbol_case): Use the symbol kind
rather than its number.
This should have been done in 3.6, but I wanted to avoid introducing
conflicts into Vincent's work on counterexamples. It turns out it's
completely orthogonal.
* data/README.md, data/skeletons/bison.m4, data/skeletons/c++.m4,
* data/skeletons/c.m4, data/skeletons/glr.c, data/skeletons/java.m4,
* data/skeletons/lalr1.d, data/skeletons/lalr1.java,
* data/skeletons/variant.hh, data/skeletons/yacc.c, src/conflicts.c,
* src/derives.c, src/gram.c, src/gram.h, src/output.c,
* src/parse-gram.c, src/parse-gram.y, src/print-xml.c, src/print.c,
* src/reader.c, src/symtab.c, src/symtab.h, tests/input.at,
* tests/types.at:
s/user_token_number/code/g.
Plus minor changes.
* src/parse_simulation.c: Fix bug in parse_state_free.
Free new_root when simulate_reduction generates zero states.
* src/parse-simulation.c, src/parse-simulation.h
(parse_state_list, parse_state_list_append): New.
* src/parse-simulation.c, src/parse-simulation.h,
* src/counterexample.c: Replace all uses of lists of parse states and
appends to parse_state_lists with the new API.
* src/derivation.h, src/derivation.c: Make derivation struct opaque.
Add derivation_list type for clarity.
(derivation_list_new): New.
(derivation_list_append): New.
(derivation_list_prepend): New.
(derivation_new_leaf): New constructor for derivations with no
children.
* src/counterexample.c, src/parse-simulation.c,
* src/parse-simulation.h: Replace uses of gl_list_t containing
derivations with derivation_list and its API.
Replace calls of dervation_new using null children with
derivation_new_leaf.
* src/parse-simulation.c: replace ps_chunk and its API with typed
versions si_chunk and deriv_chunk.
* src/parse-simlation.h, src/parse-simulation.c: Remove
parse_state_retain_deriv in favor of derivation reference counting.
* src/counterexample.c: Remove search_state_retain_deriv.
There's no need to go for the heap when using gnulib's hash module.
* src/state-item.c (hash_pair_lookup, hash_pair_remove,
state_sym_lookup): Use the heap Luke.
That removes a leak from hash_pair_lookup.
(init_prods): Use hash_pair_insert instead of duplicating it.
* src/state-item.h, src/state-item.c (state_item): Make the state
const.
(state_item_set): Make it clearer that it works in the state_items
global array.
* src/counterexample.c: Use 'res' as a variable name for returned
value, as elsewhere.
Avoid uninitialized variables, especially pointers.
Avoid assignment where possible.
The CI has "failures" such as (253, "Null nonterminals"):
@@ -21,7 +21,7 @@
3: 3 b: . %empty
3: 4 c: . %empty
On Symbols: {A,}
-time limit exceeded: 6.000000
+time limit exceeded: 11.000000
First Example c • c A A $end
First derivation $accept ::=[ a ::=[ c d ::=[ a ::=[ b ::=[ • ] d ::=[ c A A ] ] ] ] $end ]
Second Example c • A $end
* tests/counterexample.at (AT_BISON_CHECK_CEX): New.
Use it to neutralize differences in timeout values.
* src/parse-simulation.c: Replace reference counting with
parse_state_retain everywhere.
(free_parse_state): Make this function iterative instead of
recursive. Long parse_state chains were causing stack exhaustion.
* tests/counterexample.at: Fix expectations.
Fixes the SEGV in test 247 (counterexample.at:195): "S/R after first
token".
* src/counterexample.c: here.
* tests/counterexample.at: Fix expectations.
* src/counterexample.c, src/derivation.c:
Do not output diagnostics on stdout, that's the job of stderr, and the
testsuite heavily depend on this.
Do not leave trailing spaces in the output.
* tests/counterexample.at: Use AT_KEYWORDS.
Specify the expected outputs.
* tests/local.mk: Add counterexample.at.
* src/complain.h, src/complain.c: Add support for -Wcounterexample.
* src/conflicts.c (report_counterexamples): New.
(rule_conflicts_print): Use it when -Wcounterexample is given.
* src/getargs.h, src/getargs.c: Add support for --trace=cex.
* src/main.c (main): Init and deinit counterexample generation.
In Bison 3.6.2, the comments with brackets lose their brackets, for
improper m4 quotation.
* data/skeletons/bison.m4 (b4_gsub): New.
* data/skeletons/c-like.m4 (_b4_comment): Use it.
* tests/m4.at: Check b4_gsub.
Let --trace=m4-early dump all the logs from the start (as --trace=m4
used to do), and have --trace=m4 now start traces only when actually
working of the user's grammar.
Can make a big difference in the case of small inputs. E.g.
$ bison -S tests/testsuite.dir/001/input.m4 tests/testsuite.dir/001/input.y --trace=m4 |& wc
3952 19446 251068
$ bison -S tests/testsuite.dir/001/input.m4 tests/testsuite.dir/001/input.y --trace=m4-early |& wc
19491 131904 1830495
* data/skeletons/traceon.m4: New.
* src/getargs.h, src/getargs.c: Introduce --trace=m4-early.
* src/output.c (output_skeleton): Adjust for --trace=m4 and --trace=m4-early.
Commit a4ede8f85b ("package: make bison
a relocatable package") made Bison relocatable, but in fact it still
contains one absolute reference: the M4 variable, which points to the
M4 program. Let's fix that by using relocate(), see if an M4 binary is
available at the relocated location, and otherwise fallback to the
original M4 location.
See https://lists.gnu.org/r/bison-patches/2020-05/msg00078.html,
and https://lists.gnu.org/r/bison-patches/2020-05/msg00087.html.
* src/files.h, src/files.c (m4path): New.
* src/output.c: Use it.
* build-aux/update-test: When given a directory, use the testsuite.log
which it contains.
Do not accept empty "from"s, as substituting the empty string with
something is rarely a good idea.
With input such as
%token<fl> yVL_CLOCK "/*verilator sc_clock*/"
we generate
yVL_CLOCK = 610, /* "/*verilator sc_clock*/" */
which is invalid since the comment will actually be closed on the
first "*/". Let's turn "*/" into "*\/" to avoid this. But GCC will
also warn about "/*" inside a comment, so let's "escape" it too.
Reported by Huang Rui.
https://github.com/akimd/bison/issues/38
* data/skeletons/c-like.m4 (_b4_comment): Escape comment delimiters in
comments.
* tests/input.at (Torturing the Scanner): Check thes cases.
* tests/m4.at: New.
c.m4 contains a definition of _Noreturn which is modeled after
gnulib's lib/_Noreturn.h. The latter was recently
changed (b61bf2f0e8) to not using
[[noreturn]] at all, because the uses of _Noreturn in gnulib are
sometimes incompatible with the rules of [[noreturn]].
As a result glr.cc started to use _Noreturn in C++, which clang
refuses (all the glr.cc tests currently fail with Clang++).
* data/skeletons/c.m4 (b4_attribute_define): Restore the definition of
_Noreturn as [[noreturn]] in modern C++.
The generated code uses _Noreturn in places where [[noreturn]] is
valid.
* etc/bench.pl.in, examples/c/bistromathic/parse.y,
* examples/c/calc/calc.y, examples/c/pushcalc/calc.y: Check scanf's
return value.
* doc/bison.texi: Likewise, but only for the second example, to avoid
cluttering the very simple case.
* build-aux/update-test: When given a directory, use the testsuite.log
which it contains.
Do not accept empty "from"s, as substituting the empty string with
something is rarely a good idea.
* maint:
news: update
maint: post-release administrivia
version 3.6.1
c++: style: reorder generated code
c++: provide yy::parser::symbol_type::name
c++: make parser::symbol_name public
examples: beware of ~/.inputrc
build: also provide lzip compressed tarballs
style: minor fixes
yacc.c: restore ansi-c compatibility
Reported by Paul Eggert.
* src/getargs.c: We don't need it anyway, since we use _Noreturn.
* data/skeletons/c.m4: While at it, update the definition of _Noreturn
stolen from gnulib.
The implementation of yy::parser::symbol_name is emitted even before
the implementation of yy::parser::parser. This makes little sense.
* data/skeletons/lalr1.cc (symbol_name): Move its implementation in
the same place as in the class definition: after "error" and before
"context".
Reported by Martin Blais <[email protected]>.
https://lists.gnu.org/r/help-bison/2020-05/msg00005.html
* data/skeletons/lalr1.cc (symbol_name): Make it public.
Add a private hidden hook to enable testing of private parts.
* tests/local.at (AT_DATA_GRAMMAR_PROLOGUE): Help Emacs find the right
language mode.
* tests/c++.at (C++ Variant-based Symbols Unit Tests): Check that we
can read symbol_name.
Suggested by Matias Fonzo <[email protected]>.
* cfg.mk: Post announcements to bison-announce.
* configure.ac: Build lzip packages.
* .travis.yml: Build only xz, we don't care about the other formats
here.
* gnulib: Update.
* bootstrap.conf: Use attribute.
* src/system.h: Remove macros for attributes.
Adjust dependencies.
* src/scan-gram.l (DEPRECATED): Rename as...
(DEPRECATED_DIRECTIVE): this, to avoid the clash with the DEPRECATED macro.
Suggested by Matias Fonzo <[email protected]>.
* cfg.mk: Post announcements to bison-announce.
* configure.ac: Build lzip packages.
* .travis.yml: Build only xz, we don't care about the other formats
here.
I don't think it's fair to have yypstate_new, yypstate_delete,
yypush_parse and yypull_parse to have their own section, on par with
yyparse and yylex. Let them be in a single section about push
parsers. And show new/delete first.
* doc/bison.texi (Push Parser Interface): New.
Fuse the aforementioned sections into it.
The user gives yyexpected_tokens a limit: the max number of tokens she
wants to hear about. That's because an error message that reports a
bazillion of possible tokens is useless.
In that case yyexpected_tokens returned 0, so the user would not know
if there are too many expected tokens or none (yes, that's possible).
There are several ways to tell the user in which situation she's in:
- return some E2MANY, a negative value. Then it makes the pattern
int argsize = yypcontext_expected_tokens (ctx, arg, ARGS_MAX);
if (argsize < 0)
return argsize;
no longer valid, as for E2MANY (i) the user must generate the error
message anyway, and (ii) she should not return E2MANY
- return ARGS_MAX + 1. Then it makes it dangerous for the user, as
she has to iterate update `min (ARGS_MAX, argsize)`.
Returning 0 is definitely simpler and safer for the user, as it tells
her "this is not an error, just generate your message without a list
of expecting tokens". So let's still return 0, but set arg[0] to the
empty token when the list is really empty.
* data/skeletons/glr.c, data/skeletons/lalr1.cc, data/skeletons/lalr1.java
* data/skeletons/yacc.c (yyexpected_tokens): Put the empty symbol
first if there are no possible tokens at all.
* examples/c/bistromathic/parse.y: Demonstrate how to use that.
The shell grammar does not allow empty statements in then/else part of
an if, but examples/test failed to catch the syntax errors from the
script it ran. So exited with success anyway.
You would expect 'set -e' to suffice, but with bash 3.2 actually it
does not. As a matter of fact, I could find a way to have this behave
properly:
$ cat test.sh
set -e
cleanup ()
{
status=$?
echo "cleanup: $status"
exit $status
}
trap cleanup 0 1 2 13 15
. $1
s=$?
echo "test.sh: $s"
exit $s
$ cat bistro.test
if true; then
fi
$ /bin/sh ./test.sh ./bistro.test
./bistro.test: line 2: syntax error near unexpected token `fi'
cleanup: 0
$ echo $?
0
Remove the set -e (or the trap), and tada, it works... So we have to
deal with the error by hand.
* examples/test ($exit): Replace with...
($status): this.
Preserve the exit status of the test case.
* examples/c/bistromathic/bistromathic.test: Fix syntax error.
For instance test 386, "glr.cc api.value.type={double}":
types.at:366: $CXX $CXXFLAGS $CPPFLAGS $LDFLAGS -o test test.cc $LIBS
stderr:
test.cc: In function 'ptrdiff_t yysplitStack(yyGLRStack*, ptrdiff_t)':
test.cc:490:4: error: 'PTRDIFF_MAX' was not declared in this scope
(PTRDIFF_MAX < SIZE_MAX ? PTRDIFF_MAX : YY_CAST (ptrdiff_t, SIZE_MAX))
^
test.cc:1805:37: note: in expansion of macro 'YYSIZEMAX'
ptrdiff_t half_max_capacity = YYSIZEMAX / 2 / state_size;
^~~~~~~~~
test.cc:490:4: note: suggested alternative: '__PTRDIFF_MAX__'
(PTRDIFF_MAX < SIZE_MAX ? PTRDIFF_MAX : YY_CAST (ptrdiff_t, SIZE_MAX))
^
test.cc:1805:37: note: in expansion of macro 'YYSIZEMAX'
ptrdiff_t half_max_capacity = YYSIZEMAX / 2 / state_size;
^~~~~~~~~
The failing tests are using glr.cc only, which I don't understand, the
problem is rather in glr.c, so I would expect glr.c tests to also fail.
Reported by Bruno Haible.
https://lists.gnu.org/archive/html/bug-bison/2020-05/msg00053.html
* data/skeletons/yacc.c: Move the block that defines
YYPTRDIFF_T/YYPTRDIFF_MAXIMUM, YYSIZE_T/YYSIZE_MAXIMUM, and
YYSIZEOF to...
* data/skeletons/c.m4 (b4_sizes_types_define): Here.
(b4_c99_int_type): Also take care of the #undefinition of short.
* data/skeletons/yacc.c, data/skeletons/glr.c: Use
b4_sizes_types_define.
* data/skeletons/glr.c: Adjust to use YYPTRDIFF_T/YYPTRDIFF_MAXIMUM,
YYSIZE_T/YYSIZE_MAXIMUM.
One function missing on Solaris 10 Sparc:
CCLD examples/c/bistromathic/bistromathic
Undefined first referenced
symbol in file
strnlen examples/c/bistromathic/bistromathic-parse.o
ld: fatal: symbol referencing errors. No output written to examples/c/bistromathic/bistromathic
Reported by Dagobert Michelsen.
https://lists.gnu.org/r/bug-bison/2020-05/msg00048.html
* examples/c/bistromathic/parse.y (xstrndup): Don't use strnlen.
xstrndup is assembled from gnulib's xstrndup, strndup and strnlen...
AIX 7.1 supports diff -u, but its output does not match the expected
one.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00049.html
* tests/atlocal.in (DIFF_U_WORKS): New.
* tests/local.at (AT_DIFF_U_CHECK): New.
* tests/existing.at (_AT_TEST_EXISTING_GRAMMAR): Use AT_DIFF_U_CHECK.
Readline may emit escape sequences before the prompt.
Reported by Bruno Haible.
https://lists.gnu.org/r/platform-testers/2020-05/msg00001.html.
* examples/c/bistromathic/bistromathic.test: Trust readline _only_ if
we get what we expect on some reference computation.
Appearing on IRIX with gcc -mabi=n32.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00039.html
* examples/c++/variant-11.yy, examples/c/bistromathic/parse.y: Don't
give chars to isdigit, cast them to unsigned char before.
* src/complain.c: Use c_isdigit.
* src/fixits.c (fixits_run): Avoid casts.
* src/lalr.c (goto_print): Use %zu for a size_t.
Don't try to build bistromathic if we don't have readline.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00028.html
* configure.ac (ENABLE_BISTROMATHIC): New.
* examples/c/bistromathic/local.mk: Use it.
* examples/c/bistromathic/bistromathic.test: Exit 77 for skip.
"foo || bar" does not invoke bar on AIX 7.2 when foo does not exist.
It just dies.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00029.html
* examples/c/reccalc/reccalc.test: Check for seq in a subshell.
* data/skeletons/lalr1.java (Location): Make it a static class.
(Lexer.yylex, Lexer.getLVal, Lexer.getStartPos, Lexer.getEndPos):
These are not needed in push parsers.
* examples/java/calc/Calc.y: Demonstrate push parsers in the Java.
* doc/bison.texi: Push parsers have been supported for a long time,
remove incorrect statements stating the opposite.
Reported by Horst von Brand.
https://lists.gnu.org/r/bug-bison/2020-04/msg00033.html
* examples/c/bistromathic/Makefile: libintl might not be needed, but
libm probably is.
* examples/c/bistromathic/parse.y: Include locale.h.
On OpenBSD 6.5, the prompt is repeated, but not the actual command
line... Don't try to cope with that.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00015.html
* examples/c/bistromathic/bistromathic.test: Skip when readline behave
this way.
The option --header was introduced in version 2.5.6.
The option --header-file was introduced in version 2.6.4.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00013.html
So use --header, and do bother with versions that don't support it.
* m4/flex.m4: Check whether flex supports --header.
* configure.ac (FLEX_WORKS, FLEX_CXX_WORKS): Set to false if it doesn't.
* * examples/c/reccalc/local.mk, examples/c/reccalc/Makefile:
Use --header rather than --header-file.
To write unit tests for their scanners, some users depended on
symbol_type::token():
Lexer lex("12345");
symbol_type t = lex.nextToken();
assert(t.token() == token::INTLIT);
assert(t.value.as<int>() == 12345);
But symbol_type::token() was removed in Bison 3.5 because it relied on
a conversion table. So users had to find other patterns, such as
assert(t.type_get() == by_type(token::INTLIT).type_get());
which relies on several private implementation details.
As part of transitioning from "token type" to "token kind", and making
this a public and documented interface, "by_type" was renamed
"by_kind" and "type_get()" was renamed as "kind()". The latter had
backward compatibility mechanisms, not the former.
In Bison 3.6 none of this should be used, but rather
assert(t.kind() == symbol_kind::S_INTLIT);
Reported by Pramod Kumbhar.
https://lists.gnu.org/r/bug-bison/2020-05/msg00012.html
* data/skeletons/c++.m4 (by_type): Make it an alias to by_kind.
I don't plan to fix everything in one go. But this was in the way of
the next commit.
* data/skeletons/lalr1.java: Avoid space before parens.
* tests/java.at: Adjust.
Bison emits strings to translate in the generated code, for builtin
tokens. So they appear only in generated parsers, which are not
shipped, so they are not in the src tree, so we cannot use them in our
POTFILE.
Except src/parse-gram.c, which is in the source tree. And even in the
git repo. But to avoid useless diffs in the repo, we do not keep the
src/parse-gram.c _with_ the #lines. This is done in a dist-hook which
regenerates src/parse-gram.c when we run "make dist".
Unfortunately, then, update-po traverses the whole tree and sees that
the location of the strings to translate in src/parse-gram.c have
changed, so the bison.pot is to be updated. And that is not possible
in the "make dist" which is run within "make distcheck"
(not the one preparing the dist for distcheck, the one run by
distcheck to check that a distributed tarball can build a tarball)
because then the src tree is read-only.
So let's not put src/parse-gram.c in the POTFILE, and expose these
strings to gettextize by hand.
* src/i18n-strings.c: New.
* po/POTFILES.in: Add it, and remove src/parse-gram.c.
Some people have been using that symbol. Some even have #defined it
themselves.
https://lists.gnu.org/r/bison-patches/2020-04/msg00138.html
Let's provide backward compatibility, having it point to YYUNDEF, so
that an error message is generated.
* data/skeletons/yacc.c (YYERRCODE): New, at the exact same location
it was defined before.
These are internal details. `type_get ()` is still there to ensure
backward compatibility, `kind ()` being the modern way.
* data/skeletons/c++.m4 (by_type, by_type::type): Rename as...
(by_kind, by_kind::kind_): this.
Adjust dependencies.
From
public interface Lexer {
/* Token kinds. */
/** Token number, to be returned by the scanner. */
static final int YYEOF = 0;
/** Token number, to be returned by the scanner. */
static final int YYERRCODE = 256;
/** Token number, to be returned by the scanner. */
static final int YYUNDEF = 257;
/** Token number, to be returned by the scanner. */
static final int BANG = 258;
...
/** Deprecated, use b4_symbol(0, id) instead. */
public static final int EOF = YYEOF;
to
public interface Lexer {
/* Token kinds. */
/** Token "end of file", to be returned by the scanner. */
static final int YYEOF = 0;
/** Token error, to be returned by the scanner. */
static final int YYerror = 256;
/** Token "invalid token", to be returned by the scanner. */
static final int YYUNDEF = 257;
/** Token "!", to be returned by the scanner. */
static final int BANG = 258;
...
/** Deprecated, use YYEOF instead. */
public static final int EOF = YYEOF;
* data/skeletons/java.m4 (b4_token_enum): Display the symbol's tag in
comment.
* data/skeletons/lalr1.java: Address overquotation issue.
* examples/java/calc/Calc.y, examples/java/simple/Calc.y: Use YYEOF,
not EOF.
On an invalid character literal such as "'\777'" we used to produce
two errors:
input.y:2.9-12: error: invalid number after \-escape: 777
input.y:2.8-13: error: empty character literal
Get rid of the second one.
* src/scan-gram.l (STRING_GROW_ESCAPE): New.
* tests/input.at: Adjust.
I'm quite pleased to see that the tricky case of glr.c was already
prepared by the changes to support syntax_error exceptions. Better
yet, it is actually syntax_error that becomes a special case of the
general pattern: make yytoken be YYERRCODE.
* data/skeletons/glr.c (YYFAULTYTOK): Remove the now useless (Basil)
Faulty token.
Instead, use the error token.
* data/skeletons/lalr1.d, data/skeletons/lalr1.java: When computing
the action, first check the case of the error token.
* tests/calc.at: Check cases for the error token symbols before and
after it.
* data/skeletons/yacc.c (yyparse): When the scanner returns YYERRCODE,
go directly to error recovery (yyerrlab1).
However, don't keep the error token as lookahead, that token is too
special.
* data/skeletons/lalr1.cc: Likewise.
* examples/c/bistromathic/parse.y (yylex): Use that feature to report
nicely invalid characters.
* examples/c/bistromathic/bistromathic.test: Check that.
* examples/test: Neutralize gratuitous differences such as rule
position.
* tests/calc.at: Check that case in C only.
The other case seem to be working, but that's an illusion that the
next commit will address (in fact, they can enter endless loops, and
report the error several times anyway).
* examples/c/bistromathic/parse.y: here.
* examples/c/bistromathic/bistromathic.test: Check it.
Included a stupid case where the error is actually ignored.
When the user ctrl-d the line, we left the cursor not at col 0.
Let's fix that.
This revealed a few short-comings in the testing framework.
* examples/test (run): Also display the diffs.
And support -n.
* examples/c/bistromathic/bistromathic.test
* examples/c/bistromathic/parse.y
We will not keep YYERRCODE anyway, it causes backward compatibility
issues. So as a first step, let all the skeletons use that name,
until we have a better one.
* data/skeletons/bison.m4, data/skeletons/glr.c,
* data/skeletons/glr.cc, data/skeletons/lalr1.cc,
* data/skeletons/lalr1.d, data/skeletons/lalr1.java,
* data/skeletons/yacc.c, doc/bison.texi, tests/headers.at,
* tests/input.at:
here.
* etc/bench.pl.in: Don't force parse.error=detailed
Use a simpler way to display the pseudo %bison directive.
(&bench_with_gbenchmark): Give details about the compiler.
We could try to avoid the weird "#if 1", but then the indentation of
the inner #if would be wrong. Let' keep it this way.
* data/skeletons/yacc.c: here.
Also, avoid sticking the comment to the directive.
Currently it was only using stubs. Let's actually translate the
strings using gettext.
* examples/c/bistromathic/local.mk: Define LOCALEDIR, BISON_LOCALEDIR
and link with libintl.
* examples/c/bistromathic/parse.y: Use them.
Remove useless includes.
Take ENABLE_NLS into account.
(error_format_string): New.
(yyreport_syntax_error): Rewrite to rely on a format string, which is
more appropriate for internationalization.
* examples/c/bistromathic/Makefile: We no longer use Flex.
We need readline and intl.
* doc/bison.texi: Point to bistromathic for a better option for
internationalization.
* po/POTFILES.in: Add bistromathic.
In C/C++, N_ is a no-op. Define it if the user didn't.
Suggested by Frank Heckenbach.
https://lists.gnu.org/r/bug-bison/2020-04/msg00010.html
* src/output.c (prepare_symbol_names): Rename has_translations as
has_translations_flag.
* data/skeletons/bison.m4 (b4_has_translations_if): New.
* data/skeletons/java.m4 (b4_trans): Use it.
* data/skeletons/glr.c, data/skeletons/lalr1.cc, data/skeletons/yacc.c
(N_): Provide a default definition.
* src/output.c (prepare_symbol_names): Don't play tricks with the
symbols, it's quite too late.
(has_translations): Move to...
* src/symtab.c: here.
(symbols_pack): Use it to enable translation for special symbols.
On macOS, wc -l always prepends the result with a tab, even when fed
by stdin. But anyway, we should have used `grep -c -v`, which appears
to be portable according to Autoconf's "Limitations of Usual Tools"
section.
Reported by Denis Excoffier.
https://lists.gnu.org/r/bug-bison/2020-04/msg00009.html
* tests/calc.at (_AT_CHECK_CALC): Use grep's -c instead.
macOS' version of readline does not repeat stdin on stdout in
non-interactive mode, contrary to the current version of GNU readline.
* examples/test: Add support for strip_prompt.
* examples/c/bistromathic/bistromathic.test (strip_prompt): Set it
when needed.
Early exit when needed.
symbol_type::token () was removed: it returned the token kind of a
symbol. To do that, one needs to convert from the symbol kind to the
token kind, which requires a table.
This broke some users' unit tests for scanners, see
https://lists.gnu.org/r/bug-bison/2020-01/msg00001.htmlhttps://lists.gnu.org/r/bug-bison/2020-03/msg00020.htmlhttps://lists.gnu.org/r/help-bison/2020-04/msg00005.html
Instead of making this possible again, let's check the symbol's kind
instead. So give proper access to a symbol's kind.
That feature existed, undocumented, as 'type_get()'. Let's rename
this as 'kind()'.
* data/skeletons/c++.m4, data/skeletons/glr.cc,
* data/skeletons/lalr1.cc (type_get): Rename as...
(kind): This.
(type_get): Install a backward compatibility alias.
* doc/bison.texi (Complete Symbols): Document symbol_type and
symbol_type::kind.
* data/skeletons/c++.m4: Define the old names in terms on the new
ones, instead of the converse.
* doc/bison.texi (C++ Parser Interface): Be more extensive about
token_kind_type.
Not all the symbols have a fixed symbol code. UNDEF's one is fixed:
-2.
* data/skeletons/glr.c, data/skeletons/lalr1.cc, data/skeletons/lalr1.d,
* data/skeletons/yacc.c: here.
Why didn't I think about this before??? symbolName should be a method
of SymbolKind.
* data/skeletons/lalr1.java (YYParser::yysymbolName): Move as...
* data/skeletons/java.m4 (SymbolKind::getName): this.
Make the table a static final table, not a local variable.
Adjust dependencies.
* doc/bison.texi (Java Parser Interface): Document i18n.
(Java Parser Context Interface): Document SymbolKind.
* examples/java/calc/Calc.y, tests/local.at: Adjust.
* doc/bison.texi (C++ Parser Context): New.
* data/skeletons/lalr1.cc (parser::yysymbol_name): Rename as...
(parser::symbol_name): this.
(A Complete C++ Example): Promote LAC, now that we have it.
Promote parse.error detailed over verbose.
* examples/c++/calc++/calc++.test, tests/local.at: Adjust.
* NEWS (Deep overhaul of the symbol and token kinds): New.
* doc/bison.texi: Promote YYEOF over "0" in scanners.
(Token Decl): No longer show YYEOF here, it now works by default.
(Token I18n): More details about YYEOF here.
(Calc++): Just use YYEOF.
I have been hesitating a lot before doing it ---after all the user
must not use this kind, so what's the point of showing it in
yytoken_kind_t. And eventually I chose to play it safe with the
typing system and make it possible to use yytoken_kind_t for all the
tokens, even the "empty token".
* data/skeletons/c.m4: Give an id and a tag to YYEMPTY.
(b4_token_enums): Define YYEMPTY.
* data/skeletons/c++.m4 (b4_token_enums): Define YYEMPTY.
* data/skeletons/glr.c, data/skeletons/glr.cc, data/skeletons/yacc.c:
(YYEMPTY): Remove.
Use b4_symbol(-2, id) instead.
The user should think of yypcontext fields as accessible only via
yypcontext_* functions. So let's rename yyexpected_tokens to reflect
that.
Let's _not_ rename yyreport_syntax_error, as the user may define this
function, and is not allowed to access directly the fields of
yypcontext_t: she *must* use the "accessors". This is comparable to
the case of C++/Java where the user defines
parser::report_syntax_error, not parser::context::report_syntax_error.
* data/skeletons/glr.c, data/skeletons/yacc.c (yyexpected_tokens):
Rename as...
(yypcontext_expected_tokens): this.
Adjust dependencies.
The name "$end" is nice in the report, in particular it avoids that
pointed-rules (aka items) be too long. It also helps keeping them
"standard".
But it is bad in error messages, we should report "end of file" (or
maybe "end of input", this is debatable). So, unless the user already
defined the alias for the error token herself, make it "end of file".
It should even be translated if the user already translated some
tokens, so that there is now no strong reason to redefine the $end
token.
* src/output.c (prepare_symbol_names): Issue "end of file" instead of
"$end".
* data/skeletons/lalr1.java (yytnamerr_): Remove the renaming hack.
* build-aux/update-test: Accept files with names containing a "+",
such as c++.at.
* tests/actions.at, tests/c++.at, tests/conflicts.at,
* tests/glr-regression.at, tests/regression.at, tests/skeletons.at:
Adjust.
Yet, don't change the structure identifier to avoid introducing
conflicts in Vincent Imbimbo's PR (which, amusingly enough, is about
conflicts).
* src/symtab.c: here.
* tests/diagnostics.at, tests/input.at: Adjust.
yy::parser features a parse() function, not a yyparse() one.
* data/skeletons/lalr1.cc (yyreport_syntax_error)
(context::yyexpected_tokens): Rename as...
(report_syntax_error, context::expected_tokens): these.
Currently EOF is handled in an adhoc way, with a #define YYEOF 0 in
the implementation file. As a result, the user has to define her own
EOF token if she wants to use it, which is a pity.
Give the $end token a visible kind name, YYEOF. Except that in C,
where enums are not scoped, we would have collisions between all the
definitions of YYEOFs in the header files, so in C, make it
<api.PREFIX>EOF.
* data/skeletons/c.m4 (YYEOF): Override its name to avoid collisions.
Unless the user already gave it a different name.
* data/skeletons/glr.c (YYEOF): Remove.
Use ]b4_symbol(0, [id])[ instead.
Add support for "pre_epilogue", for glr.cc.
* data/skeletons/glr.cc: Remove dead code (never emitted #undefs).
* data/skeletons/yacc.c
* src/parse-gram.c
* src/reader.c
* src/symtab.c
* tests/actions.at
* tests/input.at
* data/skeletons/bison.m4 (b4_symbol_token_kind): Give a definition to
$undefined.
(b4_token_visible_if): $undefined has an id.
* src/output.c (prepare_symbol_definitions): Stop lying: $undefined
_is_ a token.
* tests/input.at: Adjust.
There are people out there that do use YYERRCODE (the token kind of
the error token). See for instance
https://github.com/borbolla-automation/SPC_Machines/blob/3812012bb782bfdfe7b325950a35cd337925fcad/unixODBC-2.3.2/Drivers/nn/yylex.c.
Currently, YYERRCODE is defined by yacc.c in an adhoc way as a #define
in the *.c file only. It belongs with the other token kinds.
YYERRCODE is not a nice name, it does not fit in our naming scheme.
YYERROR would be more logical, but it collides with the YYERROR macro.
Shall we keep the same name in all the skeletons? Besides, to avoid
collisions in C, we need to apply the api prefix: YYERRCODE is
actually <PREFIX>ERRCODE. This is not needed in the other languages.
* data/skeletons/bison.m4 (b4_symbol_token_kind): New.
Map the error token to "YYERRCODE".
* data/skeletons/yacc.c (YYERRCODE): Don't define it, it's handled by...
* src/output.c (prepare_symbol_definitions): this.
* tests/input.at (Redefining the error token): Check it.
* data/skeletons/bison.m4 (b4_symbol_kind): Dispatch on the UNDEF
token number rather than its name.
* data/skeletons/c++.m4, data/skeletons/c.m4, data/skeletons/java.m4:
Comment changes.
A forthcoming commit (tokens: properly define the "error" token kind)
revealed a problem in the C++ generated headers: they are not
self-contained. With this file:
%language "c++"
%define api.value.type variant
%code {
static int yylex (yy::parser::semantic_type *lvalp);
}
%token <int> X
%%
exp:
X { printf ("x\n"); }
;
%%
void
yy::parser::error (const std::string& m)
{
std::cerr << m << '\n';
}
static
int yylex (yy::parser::semantic_type *lvalp)
{
static int const input[] = {yy::parser::token::X, 0};
static int toknum = 0;
return input[toknum++];
}
int
main (int argc, char const* argv[])
{
yy::parser p;
return p.parse ();
}
the generated header fails to compile cleanly (foo.cc just #includes
the generated header):
$ clang++-mp-9.0 -c -Wundefined-func-template foo.cc
In file included from foo.cc:1:
bar.tab.hh:550:12: warning: instantiation of function 'yy::parser::basic_symbol<yy::parser::by_type>::basic_symbol' required here, but no definition is available
[-Wundefined-func-template]
struct symbol_type : basic_symbol<by_type>
^
bar.tab.hh:436:7: note: forward declaration of template entity is here
basic_symbol (basic_symbol&& that);
^
bar.tab.hh:550:12: note: add an explicit instantiation declaration to suppress this warning if 'yy::parser::basic_symbol<yy::parser::by_type>::basic_symbol' is explicitly instantiated
in another translation unit
struct symbol_type : basic_symbol<by_type>
^
1 warning generated.
* data/skeletons/c++.m4 (b4_public_types_define): Move the
implementation of the basic_symbol move-ctor to...
(b4_public_types_define): here, its declaration.
* tests/headers.at (Sane headers): Use a declared token so that the
corresponding token constructor is declared. Which triggers the
aforementioned issue.
That's one nice benefit from using enums.
* data/skeletons/lalr1.java (YYSYMBOL_YYEMPTY): No longer define it.
Use 'null' instead.
* examples/java/calc/Calc.y, tests/local.at: Adjust.
* data/skeletons/bison.m4, data/skeletons/c++.m4, data/skeletons/c.m4,
* data/skeletons/glr.cc, data/skeletons/lalr1.cc,
* data/skeletons/lalr1.d, data/skeletons/lalr1.java:
Refer to the "kind" of a symbol, not its "type", where appropriate.
* doc/bison.texi: Replace occurrences of "token type" with "token
kind".
Stop referring to the "macro definitions" of the token kinds, just
name them "definitions".
* maint:
maint: post-release administrivia
version 3.5.4
examples: reccalc: really compile cleanly in C99
news: announce that Bison 3.6 drops YYERROR_VERBOSE
news: update for 3.5.4
style: fix spellos
typo: succesful -> successful
package: improve the readme
java: check and fix support for api.token.raw
java: style: prefer 'int[] foo' to 'int foo[]'
build: fix syntax-check issues
tests: recheck: work properly when the test suite was interrupted
doc: c++: promote api.token.raw
build: fix compatibility with old compilers
examples: reccalc: compile cleanly in C99
The previous fix does not suffice, and actually managed to make things
worse by defining yyscan_t twice in parse.y...
Reported by kencu.
https://trac.macports.org/ticket/59927#comment:29
* examples/c/reccalc/parse.y (yyscan_t): Define it with the same
guards as used by Flex.
The first name is too long. We already have `yypstate`, so
`yypcontext` is ok. We are also migrating to using `*_t` for our
types.
* NEWS, data/skeletons/glr.c, data/skeletons/yacc.c, doc/bison.texi,
* examples/c/bistromathic/parse.y, src/parse-gram.y, tests/local.at:
(yyparse_context_t, yyparse_context_location, yyparse_context_token):
Rename as...
(yypcontext_t, yypcontext_location, yypcontext_token): these.
The Java enums are very different from the C model. As a consequence,
one cannot "build" an enum directly from an integer, we must retrieve
it. That's the purpose of the SymbolType.get class method.
* data/skeletons/java.m4 (b4_symbol_enum, b4_case_code_symbol)
(b4_declare_symbol_enum): New.
* data/skeletons/lalr1.java: Use SymbolType,
SymbolType.YYSYMBOL_YYEMPTY, etc.
* examples/java/calc/Calc.y, tests/local.at: Adjust.
* tests/local.at (AT_LANG_MATCH, AT_YYERROR_DECLARE(java))
(AT_YYERROR_DECLARE_EXTERN(java), AT_PARSER_CLASS): New.
(AT_MAIN_DEFINE(java)): Use AT_PARSER_CLASS.
* tests/scanner.at: Add a test for Java.
* data/skeletons/lalr1.java (yytranslate_): Cast the result.
* data/skeletons/d.m4 (b4_symbol_enum, b4_declare_symbol_enum): New.
* data/skeletons/lalr1.d: Use them.
Use SymbolType, SymbolType.YYSYMBOL_YYEMPTY etc. where appropriate.
(undef_token_, token_number_type, yy_error_token_): Remove.
* tests/local.mk (recheck): Look at the per-test logs, not the overall
log, which, when interrupted, contains only information about... the
tests that passed.
GCC 4.2 dies with
src/InadequacyList.c: In function 'InadequacyList__new_conflict':
src/InadequacyList.c:37: error: #pragma GCC diagnostic not allowed inside functions
src/InadequacyList.c:37: error: #pragma GCC diagnostic not allowed inside functions
src/InadequacyList.c:40: error: #pragma GCC diagnostic not allowed inside functions
Reported by Evan Lavelle.
See https://lists.gnu.org/r/bug-bison/2020-03/msg00021.html
and https://trac.macports.org/ticket/59927.
* src/system.h (GCC_VERSION): New.
Use it to control IGNORE_TYPE_LIMITS_BEGIN and
IGNORE_TYPE_LIMITS_END.
Because of the insane current implementation of glr.cc, things are a
bit nasty. We will rename symbol_number_type as symbol_type_type
later, to keep this commit small.
* data/skeletons/c++.m4 (b4_declare_symbol_enum): New.
Also define YYNTOKENS to avoid type clashes when yyntokens_ was
actually defined in another enum.
Use it.
(symbol_number_type): Be an alias of symbol_type_type.
Use YYSYMBOL_YYEMPTY and the like.
Use symbol_number_type where appropriate.
(empty_symbol): Remove.
(yytranslate_): Use symbol_number_type, not token_number_type.
* data/skeletons/lalr1.cc: Use symbol_number_type where appropriate.
Adjust to the replacement of empty_symbol by YYSYMBOL_YYEMPTY.
(yy_error_token_, yy_undef_token_, yyeof_, yyntokens_): Remove.
Adjust dependencies.
* data/skeletons/glr.cc: Use symbol_number_type where appropriate.
Forward definitions of YYSYMBOL_YYEMPTY, etc. to glr.c.
* tests/headers.at: Accept YYNTOKENS and other YYSYMBOL_*.
* tests/local.at (AT_YYERROR_DEFINE(c++)): Use symbol_number_type.
Now that yacc.c and glr.c both know yysymbol_type_t, convert the
common routines.
* data/skeletons/c.m4 (yydestruct, yy_symbol_value_print)
(yy_symbol_print): Use yysymbol_type_t instead of int.
* data/skeletons/glr.c: Use yySymbol where appropriate.
* data/skeletons/yacc.c (YY_ACCESSING_SYMBOL): New wrapper around
yystos.
Use it.
* tests/local.at (yyreport_syntax_error): Use yysymbol_type_t where
appropriate.
Apply the same changes as in yacc.c. Now yySymbol and yysymbol_type_t
are aliases. We will remove the former later, to avoid cluttering
this commit.
* data/skeletons/glr.c: Use b4_declare_symbol_enum.
Use YYSYMBOL_YYEOF etc. where appropriate.
(YYUNDEFTOK, YYTERROR): Remove.
(YYTRANSLATE, yySymbol, yyexpected_tokens, yysyntax_error_arguments):
Adjust.
(yy_accessing_symbol): New.
Use it where appropriate.
This triggers warnings with several compilers. For instance ICC fills
the logs with pages and pages of
input.c(477): error: a value of type "int" cannot be used to initialize an entity of type "const yysymbol_type_t={yysymbol_type_t}"
0, 2, 2, 2, 2, 2, 2, 2, 2, 2,
^
input.c(477): error: a value of type "int" cannot be used to initialize an entity of type "const yysymbol_type_t={yysymbol_type_t}"
0, 2, 2, 2, 2, 2, 2, 2, 2, 2,
^
And so does G++9 when compiling yacc.c's (C) output
input.c:545:8: error: invalid conversion from 'int' to 'yysymbol_type_t' [-fpermissive]
545 | 0, 5, 9, 2, 2, 2, 2, 2, 2, 2,
| ^
| |
| int
input.c:545:15: error: invalid conversion from 'int' to 'yysymbol_type_t' [-fpermissive]
545 | 0, 5, 9, 2, 2, 2, 2, 2, 2, 2,
| ^
| |
| int
Clang++ is no exception
input.c:545:8: error: cannot initialize an array element of type 'const yysymbol_type_t' with an rvalue of type 'int'
0, 5, 9, 2, 2, 2, 2, 2, 2, 2,
^
input.c:545:15: error: cannot initialize an array element of type 'const yysymbol_type_t' with an rvalue of type 'int'
0, 5, 9, 2, 2, 2, 2, 2, 2, 2,
^
At some point we could use yysymbol_type_t's enumerators to define
yytranslate. Meanwhile...
* data/skeletons/yacc.c (yytranslate): Use the original integral type
to define it.
(YYTRANSLATE): Cast the result into yysymbol_type_t.
Currently we define enumerators only for symbols that have an
identifier. That rules out tokens such as '+', and nonterminals such
as foo-bar and foo.bar. As a consequence we are taking chances: the
compiler might compile yysymbol_type_t as too small an integral type
for some symbol codes.
* data/skeletons/bison.m4 (b4_symbol_sid): Forge a unique symbol
identifier for symbols that don't have an ID.
This is not only cleaner, it also protects us from mixing signed
values (YYEMPTY is #defined as -2) with unsigned types (the
yysymbol_type_t enum is typically compiled as a small unsigned).
For instance GCC 9:
input.c: In function 'yyparse':
input.c:1107:7: error: conversion to 'unsigned int' from 'int'
may change the sign of the result
[-Werror=sign-conversion]
1107 | yyn += yytoken;
| ^~
input.c:1107:10: error: conversion to 'int' from 'unsigned int'
may change the sign of the result
[-Werror=sign-conversion]
1107 | yyn += yytoken;
| ^~~~~~~
input.c:1108:47: error: comparison of integer expressions of
different signedness:
'yytype_int8' {aka 'const signed char'} and
'yysymbol_type_t' {aka 'enum yysymbol_type_t'}
[-Werror=sign-compare]
1108 | if (yyn < 0 || YYLAST < yyn || yycheck[yyn] != yytoken)
| ^~
input.c:702:25: error: operand of ?: changes signedness from 'int'
to 'unsigned int' due to unsignedness of
other operand [-Werror=sign-compare]
702 | #define YYEMPTY (-2)
| ^~~~
input.c:1220:33: note: in expansion of macro 'YYEMPTY'
1220 | yytoken = yychar == YYEMPTY ? YYEMPTY : YYTRANSLATE (yychar);
| ^~~~~~~
input.c:1220:41: error: unsigned conversion from 'int' to
'unsigned int' changes value
from '-2' to '4294967294'
[-Werror=sign-conversion]
1220 | yytoken = yychar == YYEMPTY ? YYEMPTY : YYTRANSLATE (yychar);
| ^
Eventually, it might be interesting to move away from -2 (which is the
only possible negative symbol number) and use the next available
number, to save bits. We could actually even simply use "0" and shift
the rest, which would allow to write "!yytoken" to mean really
"yytoken != YYEMPTY".
* data/skeletons/c.m4 (b4_declare_symbol_enum): Define YYSYMBOL_YYEMPTY.
* data/skeletons/yacc.c: Use it.
* src/parse-gram.y (yyreport_syntax_error): Use YYSYMBOL_YYEMPTY, not
YYEMPTY, when dealing with a symbol.
* tests/regression.at: Adjust.
Now that we have a proper type for internal symbol numbers, let's use
it. More code needs conversion, e.g., printers and destructors, but
they are shared with glr.c, which is not ready yet for this change.
It will also help us deal with warnings such as (GCC9 on GNU/Linux):
input.c: In function 'int yyparse()':
input.c:475:37: error: enumeral and non-enumeral type in conditional expression [-Werror=extra]
475 | (0 <= (YYX) && (YYX) <= YYMAXUTOK ? yytranslate[YYX] : YYSYMBOL_YYUNDEF)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
input.c:1024:17: note: in expansion of macro 'YYTRANSLATE'
1024 | yytoken = YYTRANSLATE (yychar);
| ^~~~~~~~~~~
* data/skeletons/yacc.c (yytranslate, yysymbol_name)
(yyparse_context_t, yyexpected_tokens, yypstate_expected_tokens)
(yysyntax_error_arguments):
Use yysymbol_type_t instead of int.
There's a number of advantage in exposing the symbol (internal)
numbers:
- custom error messages can use them to decide how to represent a
given symbol, or a set of symbols.
- we need something similar in uses of yyexpected_tokens. For
instance, currently, bistromathic's completion() reads:
int ntokens = expected_tokens (line, tokens, YYNTOKENS);
[...]
for (int i = 0; i < ntokens; ++i)
if (tokens[i] == YYTRANSLATE (TOK_VAR))
[...]
else if (tokens[i] == YYTRANSLATE (TOK_FUN))
[...]
else
[...]
- now that it's a compile-time expression, we can easily build static
tables, switch, etc.
- some users depended on the ability to get the token number from a
symbol to write test cases for their scanners. But Bison 3.5
removed the table this feature depended upon (a reverse
yytranslate). Now they can check against the actual symbol number,
without having pay (space and time) a conversion.
See https://lists.gnu.org/r/bug-bison/2020-01/msg00001.html, and
https://lists.gnu.org/archive/html/bug-bison/2020-03/msg00015.html.
- it helps us clearly separate the internal symbol numbers from the
external token numbers, whose difference is sometimes blurred in the
code when values coincide (e.g. "yychar = yytoken = YYEOF").
- it allows us to get rid of ugly macros with inconsistent names such
as YYUNDEFTOK and YYTERROR, and to group related definitions
together.
- similarly it provides a clean access to the $accept symbol (which
proves convenient in a current experimentation of mine with several
%start symbols).
Let's declare this type as a private type (in the *.c file, not
the *.h one). So it does not need to be influenced by the api prefix.
* data/skeletons/bison.m4 (b4_symbol_sid): New.
(b4_symbol): Use it.
* data/skeletons/c.m4 (b4_symbol_enum, b4_declare_symbol_enum): New.
* data/skeletons/yacc.c: Use b4_declare_symbol_enum.
(YYUNDEFTOK, YYTERROR): Remove.
Use the corresponding symbol enum instead.
* tests/local.mk (recheck): Look at the per-test logs, not the overall
log, which, when interrupted, contains only information about... the
tests that passed.
* data/skeletons/lalr1.java (yysyntaxErrorArguments): Move it from the
context, to the parser object.
Generate only for detailed and verbose error messages.
* tests/local.at (AT_YYERROR_DEFINE(java)): Use yyexpectedTokens
instead.
We could just "inline yysyntax_error_arguments back" in the routines
it was originally extracted from, but I think the code is nicer to
read this way.
* data/skeletons/glr.c (yysyntax_error_arguments): Generate only for
detailed and verbose error messages.
* data/skeletons/yacc.c: Likewise.
* data/skeletons/lalr1.cc (parser::context::yysyntax_error_arguments):
Move as...
(parser::yysyntax_error_arguments_): this.
And only for detailed and verbose error messages.
Because glr.c shares the same testing routines, we also need to
convert it.
* data/skeletons/glr.c (yyparse_context_token): New.
* tests/local.at (yyreport_syntax_error): here.
Suggested by Adrian Vogelsgesang.
https://lists.gnu.org/archive/html/bison-patches/2020-02/msg00069.html
* data/skeletons/lalr1.java (Context.EMPTY, Context.getToken): New.
(Context.yyntokens): Rename as...
(Context.NTOKENS): this.
Because (i) all the Java coding styles recommend upper case for
constants, and (ii) the Java Skeleton exposes Lexer.EOF, not
Lexer.YYEOF.
* data/skeletons/yacc.c (yyparse_context_token): New.
* examples/c/bistromathic/parse.y (yyreport_syntax_error): Don't use
yysyntax_error_arguments.
* examples/java/calc/Calc.y (yyreportSyntaxError): Likewise.
yyparse returns 0, 1, 2 since ages (accept, reject, memory exhausted).
Some of our auxiliary functions such as yy_lac and
yyreport_syntax_error also need to return error codes and also use 0,
1, 2. Because it uses yy_lac, yyexpected_tokens also needs to return
"problem", "memory exhausted", but in case of success, it needs to
return the number of tokens, so it cannot use 1 and 2 as error code.
Currently it uses -1 and -2, which is later converted into 1 and 2 as
yacc.c expects it.
Let's simplify this and use consistently -1 and -2 for auxiliary
functions that are not exposed (or not yet exposed) to the user. In
particular this will save the user from having to convert
yyexpected_tokens's -2 into yyreport_syntax_error's 2: both return -1
or -2.
* data/skeletons/yacc.c (yy_lac, yyreport_syntax_error)
(yy_lac_stack_realloc): Return -1, -2 for errors instead of 1, 2.
Adjust callers.
* examples/c/bistromathic/parse.y (yyreport_syntax_error): Do take
error codes into account.
Issue a syntax error message even if we ran out of memory.
* src/parse-gram.y, tests/local.at (yyreport_syntax_error): Adjust.
The cost of the file layer is large and makes benchmarks too coarse,
as seen for in following example, first with a file, then with a
literal string:
0. %skeleton "yacc.c" %define parse.lac full
1. %skeleton "yacc-v1.c" %define nofinal %define parse.lac full
2. %skeleton "yacc-v2.c" %define nofinal %define parse.lac full
3. %skeleton "yacc-v3.c" %define nofinal %define parse.lac full
4. %skeleton "yacc.c"
5. %skeleton "yacc-v1.c" %define nofinal
6. %skeleton "yacc-v2.c" %define nofinal
7. %skeleton "yacc-v3.c" %define nofinal
--------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------
BM_y0 32558 ns 32537 ns 21228
BM_y1 32400 ns 32369 ns 21233
BM_y2 33485 ns 33464 ns 20625
BM_y3 32139 ns 32125 ns 21446
BM_y4 31343 ns 31329 ns 21747
BM_y5 31344 ns 31317 ns 22035
BM_y6 31287 ns 31255 ns 22039
BM_y7 31387 ns 31373 ns 22178
--------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------
BM_y0 10642 ns 10634 ns 63601
BM_y1 10657 ns 10654 ns 63625
BM_y2 10441 ns 10432 ns 65957
BM_y3 10558 ns 10554 ns 64546
BM_y4 9521 ns 9516 ns 72011
BM_y5 9179 ns 9157 ns 75028
BM_y6 9360 ns 9356 ns 73770
BM_y7 9365 ns 9359 ns 72609
Of course, at the same time it is less realistic: most users read
files rather that strings, so it might lead to us to pay attention to
costs most people don't see.
* etc/bench.pl.in (&calc_input): Output into a file given as argument.
Output in C syntax.
(&generate_grammar_calc): Use it.
Simplify the grammar: remove operators we don't care about.
Rewrite the scanner to work on a char* instead of a FILE*.
* etc/bench.pl.in (&bench_with_timethese): Also use y$i, as in
&bench_with_gbenchmark.
(&generate_grammar_calc): Don't add a prefix, let the callers do it.
* etc/bench.pl.in (&compiler): New, extracted from...
(&compile): here.
Don't link when using gbm.
(&calc_input): Don't make massive input for micro
benchmarks.
(&generate_grammar_calc): When using gbm, use api.prefix to avoid name
collisions.
Be ready to issue BENCHMARKS instead of a main.
(&bench): Rename as...
(&bench_with_timethese): this.
(&bench_with_gbenchmark): New.
(&bench): New.
Dispatch on these two.
In push parsers, when asking for the list of expected tokens at some
point, it makes no sense to build a yyparse_context_t: the yypstate
alone suffices (the only difference being the lookahead). Instead of
forcing the user to build a useless shell around yypstate, let's offer
yypstate_expected_tokens.
See https://lists.gnu.org/r/bison-patches/2020-03/msg00025.html.
* data/skeletons/yacc.c (yypstate): Declare earlier, so that we can
use it for...
(yypstate_expected_tokens): this new function, when in push parsers.
Adjust dependencies.
* examples/c/bistromathic/parse.y: Simplify: use
yypstate_expected_tokens.
Style fixes.
Reduce scopes (reported by Joel E. Denny).
* upstream/maint:
maint: post-release administrivia
version 3.5.3
news: update for 3.5.3
yacc.c: make sure we properly propagated the user's number for error
diagnostics: don't crash because of repeated definitions of error
style: initialize some struct members
diagnostics: beware of zero-width characters
diagnostics: be sure to close the styling when lines are too short
muscles: fix incorrect decoding of $
code: be robust to reference with invalid tags
build: fix typo
doc: update recommandation for libtextstyle
style: comment changes
examples: use consistently the GFDL header for readmes
style: remove useless declarations
typo: succesful -> successful
README: point to tests/bison, and document --trace
gnulib: update
maint: post-release administrivia
According to https://www.unix.com/man-page/POSIX/1posix/yacc/, the
user is allowed to specify her user number for the error token:
The token error shall be reserved for error handling. The name
error can be used in grammar rules. It indicates places where the
parser can recover from a syntax error. The default value of error
shall be 256. Its value can be changed using a %token
declaration. The lexical analyzer should not return the value of
error.
I think this feature is useless, the user should not have to deal with
that. The intend is probably to give the user a means to use 256 if
she wants to, but provided "error" cleared the path first by being
assigned another number. In the case of Bison, 256 is assigned to
"error" at the end if the user did not use it for a token of hers. So
this feature is useless.
Yet it is valid, and if the user assigns twice a token number to
"error", then the second time we want to complain about it and want to
show the original definition. At this point, we try to display the
built-in definition of "error", whose location is NULL, and we crash.
Rather, the location of the first user definition of "error" should
become its defining location.
Reported byg Ahcheong Lee.
https://lists.gnu.org/r/bug-bison/2020-03/msg00007.html
* src/symtab.c (symbol_class_set): If this is a declaration and the
symbol was not declared yet, keep this as defining location.
* tests/input.at (Redefining the error token): New.
Currenly we rely on (visual) width of the characters to decide where
to open and close the styling of the quoted lines. This breaks when
we deal with zero-width characters: we cannot just rely on (visual)
columns, we need to know whether we are before, inside, or after the
highlighted portion.
* src/location.c (location_caret): col_end: no longer add 1, "regular"
characters have a width of 1, only 0-width characters have 0-width.
opened: replace with 'state', a three-valued enum.
Don't reopen the style if we already did.
* tests/diagnostics.at (Zero-width characters): New.
bar.y:4.12-17: <error>error:</error> redefining user token number of foo
- 4 | %token foo <error>123
+ 4 | %token foo <error>123</error>
| <error>^~~~~~</error>
* src/location.c (location_caret): Be sure to close.
* tests/diagnostics.at (Line is too short, and then you die): New.
Because we want to support $<a->b>$, we must accept -> in type tags,
and reject $<->$, as it is unfinished.
Reported by Ahcheong Lee.
* src/scan-code.l (yylex): Make sure "tag" does not end with -, since
-> does not close the tag.
* tests/input.at (Stray $ or @): Check this.
Currently pstate_new does not set up its variables, this task is left
to yypush_parse. This was probably to share more code with usual pull
parsers, where these (local) variables are indeed initialized by
yyparse.
But as a consequence yyexpected_tokens crashes at the very beginning
of the parse, since, for instance, the stacks are not even set up.
See https://lists.gnu.org/r/bison-patches/2020-03/msg00001.html.
The fix could have very simple, but the documentation actually makes
it very clear that we can reuse a pstate for several parses:
After yypush_parse returns a status other than YYPUSH_MORE, the
parser instance yyps may be reused for a new parse.
so we need to restore the parser to its pristine state so that (i) it
is ready to run the next parse, (ii) it properly supports
yyexpected_tokens for the next run.
* data/skeletons/yacc.c (b4_initialize_parser_state_variables): New,
extracted from the top of yyparse/yypush_parse.
(yypstate_clear): New.
(yypstate_new): Use it when push parsers are enabled.
Define after the yyps macros so that we can use the same code as the
regular pull parsers.
(yyparse): Use it when push parsers are _not_ enabled.
* examples/c/bistromathic/bistromathic.test: Check the completion on
the beginning of the line.
Currently completion on "at" proposes only "atan", but does not
actually complete "at" into "atan".
* examples/c/bistromathic/parse.y (completion): Install the lcp in
matches[0].
* examples/c/bistromathic/bistromathic.test: Check that case.
Currently "(1+<TAB>" does not work as expected, because "+" is not a
word breaking character.
* examples/c/bistromathic/parse.y (init_readline): Specify our word
breaking characters.
* examples/c/bistromathic/bistromathic.test: Avoid trailing spaces.
These macros have been extremely useful when we had to support K&R C,
which we dropped long ago. Now, they merely make the code uselessly
hard to read.
* data/skeletons/c.m4, data/skeletons/glr.c, data/skeletons/glr.cc,
* data/skeletons/yacc.c:
Stop using b4_function_define.
Let's use GNU readline and its TAB autocompletion to demonstrate the
use of yyexpected_tokens.
This shows a number of weaknesses in our current approach:
- some macros (yyssp, etc.) from push parsers "leak" in user code, we
need to undefine them
- the context needed by yyexpected_tokens does not need the token,
yypstate actually suffices
- yypstate is not properly setup when first allocated, which results
in a crash of yyexpected_tokens if fired before a first token was
read. We should move initialization from yypush_parse into
yypstate_new.
* examples/c/bistromathic/parse.y (yylex): Take input as a string, not
a file.
(EXIT): New token.
(input): Adjust to work only on a line.
(line): Remove.
(symbol_count, process_line, expected_tokens, completion)
(init_readline): New.
* examples/c/bistromathic/bistromathic.test: Adjust expectations.
This example will soon use GNU readline, so its scanner should be easy
to use (concurrently) on strings, not streams. This is not a place
where Flex shines, and anyway, these are examples of Bison, not Flex.
There's already lexcalc and reccalc that demonstrate the use of Flex.
* examples/c/bistromathic/scan.l: Remove.
* examples/c/bistromathic/parse.y (yylex): New.
Adjust dependencies.
The bistromathic example should not use Flex, it makes it too complex.
But it was the only example to show location tracking with Flex.
* examples/c/lexcalc/lexcalc.test, examples/c/lexcalc/parse.y,
* examples/c/lexcalc/scan.l: Demonstrate location tracking as is done
in bistromathic.
The current implementation of parser::context keeps a copy of the
lookahead. This is troublesome since we support move-only types.
Besides, while GCC is happy with the current implementation, Clang
complains that the ctor it needs to build the copy of the lookahead is
not yet available.
461. calc.at:1120: testing Calculator C++ %defines %locations parse.error=verbose %name-prefix "calc" %verbose ...
calc.at:1120: COLUMNS=1000; export COLUMNS; bison --color=no -fno-caret -Wno-deprecated -o calc.cc calc.y
calc.at:1120: $CXX $CXXFLAGS $CPPFLAGS $LDFLAGS -o calc calc.cc calc-lex.cc calc-main.cc $LIBS
stderr:
In file included from calc-lex.cc:7:
calc.hh:351:12: error: instantiation of function 'calc::parser::basic_symbol<calc::parser::by_type>::basic_symbol' required here, but no definition is available [-Werror,-Wundefined-func-template]
struct symbol_type : basic_symbol<by_type>
^
calc.hh:273:7: note: forward declaration of template entity is here
basic_symbol (const basic_symbol& that);
^
calc.hh:351:12: note: add an explicit instantiation declaration to suppress this warning if 'calc::parser::basic_symbol<calc::parser::by_type>::basic_symbol' is explicitly instantiated in another translation unit
struct symbol_type : basic_symbol<by_type>
^
1 error generated.
In file included from calc-main.cc:7:
calc.hh:351:12: error: instantiation of function 'calc::parser::basic_symbol<calc::parser::by_type>::basic_symbol' required here, but no definition is available [-Werror,-Wundefined-func-template]
struct symbol_type : basic_symbol<by_type>
^
calc.hh:273:7: note: forward declaration of template entity is here
basic_symbol (const basic_symbol& that);
^
calc.hh:351:12: note: add an explicit instantiation declaration to suppress this warning if 'calc::parser::basic_symbol<calc::parser::by_type>::basic_symbol' is explicitly instantiated in another translation unit
struct symbol_type : basic_symbol<by_type>
^
1 error generated.
stdout:
calc.at:1120: exit code was 1, expected 0
461. calc.at:1120: 461. Calculator C++ %defines %locations parse.error=verbose %name-prefix "calc" %verbose (calc.at:1120): FAILED (calc.at:1120)
* data/skeletons/lalr1.cc (context::yyla_): Make it a const-ref.
Move the implementation out of the declaration.
Address compiler warnings such as
warning: declaration of 'yyla' shadows a member of 'yy::parser::context' [-Wshadow]
* data/skeletons/lalr1.cc (context): Don't use the same names for
variables and members.
Use foo_ for private members, as in parser.
Also, use the + trick in array accesses to please ICC and provide it
with an int.
* data/skeletons/lalr1.cc: added support here
* tests/calc.at: added test cases
* tests/local.at: added yyreport_syntax_error implementation
for C++ test cases
Prefer b4_parse_error_case over the adhoc solution
`m4_case + b4_percent_define_get`. Same for b4_parse_error_bmatch.
* data/skeletons/glr.c: here
* data/skeletons/yacc.c: here
* etc/bench.pl.in: Adjust to the current use of %define's values.
Don't use %error-verbose.
Prefer Bison to CPP (e.g., api.value.type).
Avoid returning characters directly, so that %define api.token.raw works.
* doc/bison.texi (Parser Internationalization): Move most of its
content into...
(Enabling I18n): this new node.
(Token I18n): New.
(Token Decl): Refer to token internationalization.
(Error Reporting Function): Promote parse.error detailed.
Since Bison 2.7, output was indented four spaces for explanatory
statements. For example:
input.y:2.7-13: error: %type redeclaration for exp
input.y:1.7-11: previous declaration
Since the introduction of caret-diagnostics, it became less clear.
Remove the indentation and display submessages as in GCC:
input.y:2.7-13: error: %type redeclaration for exp
2 | %type <float> exp
| ^~~~~~~
input.y:1.7-11: note: previous declaration
1 | %type <int> exp
| ^~~~~
* src/complain.h (SUB_INDENT): Remove.
(warnings): Add "note" to the enum.
* src/complain.h, src/complain.c (complain_indent): Replace by...
(subcomplain): this.
Adjust all dependencies.
* tests/actions.at, tests/diagnostics.at, tests/glr-regression.at,
* tests/input.at, tests/named-refs.at, tests/regression.at:
Adjust expectations.
bison 3.5.2
* tag 'v3.5.2':
version 3.5.2
news: 3.5.2
gnulib: update
doc: update Doxygen template
java: avoid trailing white spaces
m4: fix b4_token_format
doc: clearly state that %yacc only makes sense with yacc.c
doc: spell check
examples: be more robust to spaces in paths
larlr1.cc: Reject unsupported values for parse.lac
We used to emit:
/** Token number,to be returned by the scanner. */
static final int NUM = 258;
/** Token number,to be returned by the scanner. */
static final int NEG = 259;
with no space after the comma. Fix that.
* data/skeletons/bison.m4 (b4_token_format): Quote where appropriate.
This revealed a number of things I had not realized:
- the Java location tracking was aliasing the same pair of positions
for all the symbols (see previous commit).
- in impure parsers, it's quite easy to use incorrect locations for
diagnostics, since yyerror uses yylloc, which is the location of the
lookahead, not that of the current lhs. So we need something like
{
YYLTYPE old_yylloc = yylloc;
yylloc = @$;
yyerror (]AT_PARAM_IF([result, count, nerrs, ])[buf);
yylloc = old_yylloc;
}
Maybe we should do that little yylloc dance in the skeleton instead
of leaving it to the user? It might be costly... But that's only
for users of the impure parsers, which are asking for trouble
anyway.
- in glr.cc invoking yyerror is somewhat cumbersome: the C++ interface
is not available as we are in yyparse (which in C), and yyerror is
used by glr.cc itself to bind it to the user's parser::error. If we
call yyerror, we need:
yyerror (]AT_LOCATION_IF([[&@$, ]])[yyparser, ]AT_PARAM_IF([result, count, nerrs, ])[msg);
However calling yy::parser::error is easier, once we know that the
current parser object is available as 'yyparser'. Which also saves
us from having to pass the parse-params ourselves:
yyparser.error (]AT_LOCATION_IF([[@$, ]])[msg);
* tests/calc.at: Invoke yyerror by hand, instead of using fprintf etc.
Adjust expectations.
* data/skeletons/lalr1.java (Context): Make data members private.
(Context.getLocation): New.
* examples/java/calc/Calc.y, tests/java.at, tests/local.at: Adjust.
* doc/bison.texi (Tokens from Literals): Move to code using
%token-table to...
(Decl Summary: %token-table): here.
* data/skeletons/bison.m4: Implement mutual exclusion.
* tests/input.at: Check it.
* doc/local.mk: Be robust to the removal of doc/.
With texinfo.tex 2019-09-24.13, node names with + are not properly
handled.
https://lists.gnu.org/r/bug-texinfo/2020-02/msg00004.html
* doc/bison.texi: Always use the three-argument form for references to
node with a + in the name.
This reverts commit ebab1ffca8.
This commit removed "useless" initializers, going from
/* YYPACT[STATE-NUM] -- Index in YYTABLE of the portion describing
STATE-NUM. */
private static final byte yypact_[] = yypact_init ();
private static final byte[] yypact_init ()
{
return new byte[]
{
25, -7, -8, 37, -8, 40, -8, 20, -8, 61,
-8, -8, 3, 9, 51, -8, -8, -2, -2, -2,
-2, -2, -2, -8, -8, -8, 1, 66, 66, 3,
3, 3
};
}
to
/* YYPACT[STATE-NUM] -- Index in YYTABLE of the portion describing
STATE-NUM. */
private static final byte[] yypact_ =
{
25, -7, -8, 37, -8, 40, -8, 20, -8, 61,
-8, -8, 3, 9, 51, -8, -8, -2, -2, -2,
-2, -2, -2, -8, -8, -8, 1, 66, 66, 3,
3, 3
};
But it turns out that this was on purpose, to work around the 64KB
limitation in JVM methods. It was introduced on the 2008-11-10 by
Di-an Jan in 09ccae9b18: "Work around
Java's ``code too large'' problem for parser tables". See
https://lists.gnu.org/r/help-bison/2008-11/msg00004.html. A real
test, where we would hit the JVM limitation, would be nice.
To avoid further regressions, add comments.
parse.error has more than two possible values.
* data/skeletons/bison.m4 (b4_error_verbose_if, b4_error_verbose_flag):
Remove.
(b4_parse_error_case, b4_parse_error_bmatch): New.
Adjust dependencies.
* data/skeletons/lalr1.java (yyexpectedTokens)
(yysyntaxErrorArguments): Make them methods of Context.
(Context.yysymbolName): New.
* tests/local.at: Adjust.
The error format should be translated, but contrary to the case of
C/C++, we cannot just depend on macros to adapt on the
presence/absence of '_'. Let's consider that the message format is to
be translated iff there are some internationalized tokens.
* src/output.c (prepare_symbol_names): Define b4_has_translations.
* data/skeletons/java.m4 (b4_trans): New.
* data/skeletons/lalr1.java: Use it to emit translatable or not the
format string.
And allow syntax error messages for 'detailed' and 'verbose' to be
translated.
* data/skeletons/lalr1.java (Context, yyexpectedTokens)
(yysyntaxErrorArguments): New.
(yysyntax_error): Use it.
In Java there is no need for N_ and yytranslate_. So instead of
hard-coding the use of N_ in the table of the symbol names, rely on
b4_symbol_translate.
* src/output.c (prepare_symbol_names): Use b4_symbol_translate instead
of N_.
* data/skeletons/c.m4 (b4_symbol_translate): New.
* data/skeletons/lalr1.java (yysymbolName): New.
Use it.
* examples/java/calc/Calc.y: Use parse.error=detailed.
* tests/calc.at: Check parse.error=detailed.
We used to emit:
/** Token number,to be returned by the scanner. */
static final int NUM = 258;
/** Token number,to be returned by the scanner. */
static final int NEG = 259;
with no space after the comma. Fix that.
* data/skeletons/bison.m4 (b4_token_format): Quote where appropriate.
Unfortunately in the Java skeleton the user cannot override the way
locations are displayed, and locations don't know the structure of the
positions. So they cannot implement the tricks used in the C/C++
skeletons to display "1.1" instead of "1.1-1.2".
* tests/local.at (Java): Add support for column tracking in the
locations, as we did in examples/java/calc.
* tests/calc.at: Use AT_CALC_YYLEX.
Soon calculator tests for Java will move from java.at to calc.at.
Which implies improving the Java testing infrastructure in
local.at (for instance really tracking columns in positions, not just
token number). Detach java.at from local.at.
* tests/java.at (AT_JAVA_POSITION_DEFINE_OLD): New.
Use it.
* examples/java/calc/Calc.y: The StreamTokenizer cannot "peek" for the
next character, it reads it, and keeps it for the next call. So the
current location is one passed the end of the current token. To avoid
this, keep the previous position, and use it to end the current token.
* examples/java/calc/Calc.test: Adjust.
The C, C++ and D skeletons used to show the stack right after popping
the stack during the reduction. Now that the stack is printed after
reaching a new state, that has become useless:
Entering state 1
Stack now 0 1
Reducing stack by rule 5 (line 83):
$1 = token "number" (1)
-> $$ = nterm exp (1)
Stack now 0
Entering state 8
Stack now 0 8
Remove the "Stack now 0" line.
* data/skeletons/lalr1.cc, data/skeletons/lalr1.d,
* data/skeletons/lalr1.java, data/skeletons/yacc.c:
Here.
Currently, if we have long rules and series of shift, we stack states
without showing stack. Let's be more incremental, and do how the Java
skeleton does.
* data/skeletons/lalr1.cc, data/skeletons/lalr1.d,
* data/skeletons/yacc.c:
Here.
Adjust test cases.
* tests/torture.at (AT_DATA_STACK_TORTURE): Disable stack traces: this
test produces a very large stack, and showing the stack each time we
shift a token goes quadatric.
The Java skeleton displays
Reading a token:
Next token is token "number" (1)
while the other display
Reading a token: Next token is token "number" (1)
When generating logs in the scanner, the first part is separated from
the second, and the end of the scanner logs have the second part
pasted in. So let's propagate the Java way, but with the colon.
* data/skeletons/glr.c, data/skeletons/lalr1.cc, data/skeletons/lalr1.d,
* data/skeletons/lalr1.java, data/skeletons/yacc.c: Do it.
Adjust test cases and doc.
* tests/local.at (AT_LANG_MATCH): New.
(AT_YYERROR_DECLARE(java), AT_YYERROR_DECLARE_EXTERN(java)): New.
* tests/calc.at: The grammar file for Java is quite different for the
others, and continuing to assemble it from pieces makes the grammar
file hard to understand. Let's also dispatch on the language to
assemble it, and isolate Java from the others.
Most of this comes from java.at.
This example, so far, was tracking the current token number, not the
current column number. This is not nice for an example...
* examples/java/Calc.y (PositionReader): New.
Use it.
* examples/java/Calc.test: Check the output.
AFAICT, autoboxing/unboxing was added in Java 5 (September 30, 2004).
I think we can afford to use it. It should help us merge some Java
tests with the main ones.
However, beware that != does not unbox: it compares the object
addresses.
* examples/java/Calc.y, tests/java.at: Simplify.
* examples/java/Calc.test, tests/java.at: Improve tests.
When building the test cases, emitting code in the epilogue is very
constraining. Let's make it simpler thanks to %code epilogue.
However, I don't want to document this: it is bad style to use it (we
should avoid having too many ways to write the same thing,
TI!MTOWTDI), just put your code in the true epilogue section.
* data/skeletons/glr.c, data/skeletons/lalr1.d, data/skeletons/lalr1.java,
* data/skeletons/yacc.c: Implement support for %code epilogue.
Remove useless comments.
* tests/calc.at, tests/java.at: Simplify.
Modeled after what was done in yacc.c, yet somewhat different since
yyGLRStack play the role of the full yyparse_context_t. It will
probably be defined as a alias, to make interfaces compatible.
LAC is not supported here. And is somewhat tricky.
* data/skeletons/glr.c (yyexpected_tokens, yysyntax_error_arguments): New.
* data/skeletons/glr.c: Move the handling of error messages after the
definition of types. Especially after yyGLRStack, which we will need
in forthcoming patches.
Simplify: yyGLRStack is defined at this point.
The PDF output is more consistent: some nodes were not pointed to with
their title. The HTML output becomes "see section Foo" instead of
"see Foo", but this should be addressed in the next Texinfo release.
Info output is simplified, as it uses only the node name and not its
title. But it's considered easier to read this way.
See https://lists.gnu.org/r/help-texinfo/2020-01/msg00031.html.
* doc/bison.texi: Set @xrefautomaticsectiontitle on.
Simplify all uses of ref.
Add an example to demonstrate the use of push parser. I'm pleasantly
surprised: parse.error=detailed works like a charm with push parsers.
* examples/c/local.mk, examples/c/pushcalc/Makefile
* examples/c/pushcalc/README.md, examples/c/pushcalc/calc.test,
* examples/c/pushcalc/calc.y, examples/c/pushcalc/local.mk:
New.
* examples/c/calc/calc.y: Restore to its original state, with
parse.error=detailed instead of parse.error=custom (this example
should be simple).
* examples/c/calc/calc.test: Check syntax errors.
* examples/c/lexcalc/parse.y: Add comments.
We used to display the unexpected token first:
$ bison foo.y
foo.y:1.8-13: error: syntax error, unexpected %token, expecting character literal or identifier or <tag>
1 | %token %token
| ^~~~~~
GCC uses a different format:
$ gcc-mp-9 foo.c
foo.c:1:5: error: expected identifier or '(' before ')' token
1 | int()()()
| ^
and so does Clang:
$ clang-mp-9.0 foo.c
foo.c:1:5: error: expected identifier or '('
int()()()
^
1 error generated.
They display the unexpected token last (or not at all). Also, they
don't waste width with "syntax error". Let's try that. It gives, for
the same example as above:
$ bison foo.y
foo.y:1.8-13: error: expected character literal or identifier or <tag> before %token
1 | %token %token
| ^~~~~~
* src/complain.h, src/complain.c (syntax_error): New.
* src/parse-gram.y (yyreport_syntax_error): Use it.
As a test case, support translations in Bison itself.
* src/parse-gram.y: Mark the translatable tokens.
While at it, use clearer names.
* tests/input.at: Adjust expectations.
Don't force callers of location_caret to have to deal with flags that
disable it.
* src/location.h, src/location.c (location_caret)
(location_caret_suggestion): Early return if disabled.
* src/complain.c: Simplify.
Just as the yacc.c skeleton, the lalr1.cc skeleton should reject
invalid values for parse.lac.
* data/skeletons/lalr1.cc: check validity of parse.lac
* tests/input.at: new test cases
Just as the yacc.c skeleton, the lalr1.cc skeleton should reject
invalid values for parse.lac.
* data/skeletons/lalr1.cc: check validity of parse.lac
* tests/input.at: new test cases
Some users would like to avoid having to "parse" the *.y file to find
the strings to translate. Let's issue the translatable tokens with N_
to allow "parsing" the generated parsers instead.
See
https://lists.gnu.org/archive/html/bison-patches/2019-01/msg00015.html
* src/output.c (prepare_symbol_names): Issue symbol_names with N_()
markup.
In addition to
%token NUM "number"
accept
%token NUM _("number")
in which case the token will be translated in error messages.
Do not use _() in the output if there are no translatable tokens.
* src/symtab.h, src/symtab.c (symbol): Add a 'translatable' member.
* src/parse-gram.y (TSTRING): New token.
(string_as_id.opt): Replace with...
(alias): this.
Use it.
* src/scan-gram.l (SC_ESCAPED_TSTRING): New start conditions, to match
TSTRINGs.
* src/output.c (prepare_symbols): Define b4_translatable if there are
translatable strings.
* data/skeletons/glr.c, data/skeletons/lalr1.cc,
* data/skeletons/yacc.c (yytnamerr): Receive b4_translatable, and use it.
* src/output.c (escape_trigraphs, xescape_trigraphs): New.
(prepare_symbol_names): Use it.
* tests/regression.at: Check the handling of trigraphs with
parse.error = detailed.
"detailed" error messages are almost like "verbose", except that we
don't double escape them, they don't get inner quotes, we don't use
yytnamerr, and we hide the table.
"custom" is exposed with the "detailed" tokens, not the "verbose"
ones: they are not double-quoted.
Because there's a risk that some people use yytname even without
"verbose", let's keep yytname (instead of yys_name) in "simple"
parse.error.
* src/output.c (prepare_symbol_names): Be ready to output symbol names
unquoted.
(prepare_symbol_names): Output both the old tname table, and the new
symbol_names one.
* data/skeletons/bison.m4: Accept 'detailed'.
* data/skeletons/yacc.c: When parse.error is 'detailed', don't emit
yytname and yytnamerr, just yysymbol_name with the table inside.
* tests/calc.at: Adjust.
Only parse.error verbose and simple will get the original yytname: the
other options will rely on a different table. So let's move on top of
the yysymbol_name function.
* data/skeletons/c.m4 (yy_symbol_print): Use yysymbol_name.
* data/skeletons/glr.c (yytokenName): Rename as...
(yysymbol_name): this.
The change of naming scheme is unfortunate, but it's definitely glr.c
which is "wrong".
Currently yy_symbol_print is defined before yytokenName, although it
should use it instead of read yytname directly. Move blocks around to
avoid this.
* data/skeletons/glr.c (yy_symbol_print): Move its definition after
that of yytokenName.
See https://github.com/nemequ/icc-travis/issues/15.
Thanks to Jeff Hammond and Evan Nemerson for their help.
* configure.ac (warn_common): Disable dubious warnings.
* .travis.yml: Use ICC again.
Be robust to newer versions of Autoconf where the package URL defaults
to https instead of http.
* configure.ac (AC_INIT): Use https.
* tests/report.at: Adjust expected output s/http/https/
to match updated URL.
Currently we get warnings with GCC 4.8 when running the
maintainer-check-g++ tests:
143. skeletons.at:85: testing Installed skeleton file names ...
../../tests/skeletons.at:120: COLUMNS=1000; export COLUMNS; bison --color=no -fno-caret --skeleton=yacc.c -o input-cmd-line.c input-cmd-line.y
../../tests/skeletons.at:121: $CC $CFLAGS $CPPFLAGS $LDFLAGS -o input-cmd-line input-cmd-line.c $LIBS
stderr:
input-cmd-line.c: In function 'int yysyntax_error(long int*, char**, const yyparse_context_t*)':
input-cmd-line.c:977:52: error: conversion to 'int' from 'long int' may alter its value [-Werror=conversion]
YYSIZEOF (yyarg) / YYSIZEOF (*yyarg));
^
cc1plus: all warnings being treated as errors
stdout:
../../tests/skeletons.at:121: exit code was 1, expected 0
and
429. calc.at:823: testing Calculator parse.error=custom %locations api.prefix={calc} ...
../../tests/calc.at:823: COLUMNS=1000; export COLUMNS; bison --color=no -fno-caret -Wno-deprecated -o calc.c calc.y
../../tests/calc.at:823: $CC $CFLAGS $CPPFLAGS $LDFLAGS -o calc calc.c $LIBS
stderr:
calc.y: In function 'int yyreport_syntax_error(const yyparse_context_t*)':
calc.y:157:58: error: conversion to 'int' from 'long unsigned int' may alter its value [-Werror=conversion]
int n = yysyntax_error_arguments (ctx, arg, sizeof arg / sizeof *arg);
^
cc1plus: all warnings being treated as errors
stdout:
../../tests/calc.at:823: exit code was 1, expected 0
We could use a cast to avoid the warning, but it becomes too
cluttered. We can also use YYPTRDIFF_T, but that forces the user to
use YYPTRDIFF_T too, although this is an array of tokens, which is
limited by YYNTOKENS, an int. So let's completely avoid this warning.
* data/skeletons/yacc.c, tests/local.at (yyreport_syntax_error): Avoid
relying on sizeof to compute the array capacity.
Enhance the calculator tests: show that passing arguments to yyerror
works.
* tests/calc.at: Add a new parse-param, nerrs, which counts the number
of syntax errors in a run.
* tests/local.at: Adjust to handle the new 'nerrs' argument, when
present.
The custom error reporting function show sees the user's additional
arguments. Let's experiment with passing them as arguments to
yyreport_syntax_error, but maybe storing them in the context would be
a bettter alternative.
* data/skeletons/yacc.c (yyreport_syntax_error): Handle the
parse-params.
* tests/calc.at, tests/local.at: Adjust.
Currently the parse-params are tested in calc.at by checking that the
global variable and the parse-params have the same value. But it does
not check that value, that could remain being 0 just as well.
* tests/calc.at: Don't define the params when they are not used.
Check the final value of result and count.
Also, do count the number of line of logs.
Provide users with a means to query for the currently allowed tokens.
Could be used for autocompletion for instance.
* data/skeletons/yacc.c (yyexpected_tokens): New, extracted from
yysyntax_error_arguments.
* examples/c/calc/calc.y (PRINT_EXPECTED_TOKENS): New.
Use it.
We use a different format to check parse.error custom. Compute the
"verbose" one from it instead of forcing the test author to provide
the various formats of expected error messages.
* tests/calc.at (_AT_CHECK_CALC_ERROR): Handle this transformation
when needed.
Simplify callers.
* tests/local.at (AT_ERROR_CUSTOM_IF, AT_ERROR_VERBOSE_IF)
(AT_ERROR_SIMPLE_IF): New.
(AT_YYERROR_DEFINE(c)): Generate yyreport_syntax_error.
* tests/calc.at (_AT_CHECK_CALC_ERROR): Accept custom error messages
as additional test case.
Use it.
Add a new test case for %define parse.error custom.
When parse.error is custom, let users define a yyreport_syntax_error
function, and use it.
* data/skeletons/bison.m4 (b4_error_verbose_if): Accept 'custom'.
* data/skeletons/yacc.c: Implement it.
* examples/c/calc/calc.y: Experiment with it.
That allows users to cover more cases, such as easily filtering some
arguments they don't want to expose. But they now have to call
yysymbol_name explicitly.
* data/skeletons/yacc.c (yysyntax_error_arguments, yysyntax_error):
Deal with symbol numbers instead of symbol names.
Isolate a function that returns the list of expected and unexpected
tokens. It will be exposed to users willing to customize their error
messages.
* data/skeletons/yacc.c (yyparse_context_t): New.
(yyerror_message_arguments): New, extracted from yysyntax_error.
When not empty, AT_PARSE_PARAMS was guaranteed to end with a comma.
Remove the trailing comma, so that we can use AT_PARSE_PARAMS at the
end of the arguments, not only at the beginning.
* tests/local.at: here.
Unfortunately, m4_append relies on the macro not being defined whereas
we would have preferred it to check for emptiness. So use
m4_define/m4_undefine instead of m4_pushdef/m4_popdef.
Let's have C be the reference, and match it elsewhere. Maybe C is too
verbose and some adjustments are needed, but then that would be done
in another batch of patches.
* data/skeletons/lalr1.cc: Print the stack once we popped after
YYERROR, and before emptying the stack at the end of parsing.
Currently the C and C++ parse traces differ in the order in which the
stack is displayed: bottom up in C, top down in C++. Let's stick to
the C order.
* data/skeletons/stack.hh (stack::iterator, stack::const_iterator)
(begin, end): Be forward, not backward.
Provide the users with a public API to get the name of the tokens. A
thin wrapper around yytname.
* data/skeletons/yacc.c (yysymbol_name): New.
Use it.
I was hoping it would help us catch warnings when char is
unsigned (see 78bb152a63), but it does
not seem to help. It's a pity that the compiler is the same all over
the place, I would have preferred testing others.
* .travis.yml: here.
Supporting YYERROR_VERBOSE via cpp is a nuisance: m4 is in charge of
handling alternatives. When adding more options for %define
parse.error, supporting both CPP and M4 is too complex. Anyway,
YYERROR_VERBOSE was deprecated long ago.
* data/skeletons/yacc.c: Use m4 only to handle verbose/simple error
messages.
I would like to offer new ways to build the error message. As a first
step, let's simplify yysyntax_error whose first loop does two things
at the same time: (i) collect the tokens to be reported in the error
message, and (ii) accumulate their sizes and possibly return
"overflow". Let's pull (ii) in a second step.
Then test 525 (regression.at:1193: parse.error=verbose overflow)
failed. This test checks that we correctly report "memory overflow"
when the error message is too large. However the test is mistaken: it
is triggered in a place where there are five (large) expected tokens,
so anyway we would not display them, so there is no (memory) overflow
here! Transform this test to (i) check that indeed there is no
overflow, and (ii) create syntax_error3 which does check the intended
behavior, but with four expected tokens.
* data/skeletons/yacc.c (yysyntax_error): First compute the list of
arguments, then compute yysize.
* tests/regression.at (parse.error=verbose overflow): Enhance and fix.
GCC's -Wchar-subscripts may report issues on platforms where char is
unsigned. Unfortunately the current CI does not reproduce the
problem. But that would allow contributors to report issues if the
warning appears somewhere.
See 139d065594.
Problem reported by Andy Fiddaman in:
https://lists.gnu.org/r/bug-bison/2019-12/msg00021.html
* configure.ac (warn_common): Add -Wchar-subscripts.
Problem reported by Andy Fiddaman in:
https://lists.gnu.org/r/bug-bison/2019-12/msg00021.html
* data/skeletons/yacc.c (yy_reduce_print, yy_lac, yysyntax_error)
(yyreturn): If I might be a char, write a[+I] instead of a[I],
so that ‘gcc -Wchar-subscripts’ does not complain.
Another breakage revealed by vcsn.
* data/skeletons/c++.m4 (yytranslate_): Do not hard code "yy" and
"parser", both can be changed by the user.
Actually, since we are in the parser itself, there's really no need to
qualify the type.
* data/skeletons/glr.c (YYASSERT): Rename as...
(YY_ASSERT): this, for consistency with yacc.c, and also to emphasize
the fact that this is not for the end user (YY_ prefix).
* tests/glr-regression.at: Define parse.assert.
Now that we use small integral types, possibly unsigned (e.g.,
unsigned char), to store state numbers, using -1 to denote an empty
state (i.e., a state that stores no semantical value) is very
dangerous: it will be confused with state 255, which might be
non-empty.
Rather than allocating a larger range of state numbers to keep the
empty-state apart, let's use the number of a state known to store no
value. The initial state, numbered 0, seems to fit perfectly the job.
Reported by Frank Heckenbach.
https://lists.gnu.org/archive/html/bug-bison/2019-11/msg00016.html
* data/skeletons/lalr1.cc (empty_state): Be 0.
It is not used. And its implementation was wrong when api.token.raw
was defined, as it was still mapping to the external token numbers,
instead of the internal ones. Besides it was provided only when
api.token.constructor is defined, yet always declared.
* data/skeletons/c++.m4 (by_type::token): Remove, useless.
Reported by Frank Heckenbach.
https://lists.gnu.org/archive/html/bug-bison/2019-11/msg00016.html
The cast is needed when yytranslate_'s argument type is token_type,
i.e., when api.token.constructor is defined.
373. types.at:138: testing lalr1.cc api.value.type=variant api.token.constructor ...
======== Testing with C++ standard flags: ''
../../tests/types.at:138: bison --color=no -fno-caret -o test.cc test.y
../../tests/types.at:138: $CXX $CXXFLAGS $CPPFLAGS $LDFLAGS -o test test.cc $LIBS
stderr:
test.cc:966:16: error: result of comparison of constant 257 with
expression of type 'yy::parser::token_type'
(aka 'yy::parser::token::yytokentype') is always true
[-Werror,-Wtautological-constant-out-of-range-compare]
else if (t <= user_token_number_max_)
~ ^ ~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
It is because it is expected that when api.token.constructor is
defined, only symbol constructors will be used, that yytranslate_ then
takes a token_type. But it is wrong: we still allow literal
characters in this case, as demonstrated by test 373 for instance.
%define api.value.type variant
%define api.token.constructor
%token <std::pair<int, int>> '1' '2';
[...]
static yy::parser::symbol_type yylex ()
{
static char const input[] = "12";
int res = input[toknum++];
typedef yy::parser::symbol_type symbol;
if (res)
return symbol (res, std::make_pair (res - '0', res - '0' + 1));
else
return symbol (res);
}
So let yytranslate_ always take an int, which makes the cast truly
useless.
* data/skeletons/c++.m4, data/skeletons/lalr1.cc (yytranslate_): here.
The C++ implementation of LAC did not skip the $undefined token,
probably because it was not exposed. Expose it, and use clearer
names.
* data/skeletons/c++.m4: Don't define undef_token_ in yytranslate_,
but...
* data/skeletons/lalr1.cc (yy_undef_token_): here.
Use a more precise type to define yy_undef_token_ and yy_error_token_.
Unfortunately we move from a compile-time value defined via an enum to
a static const member. Eventually we should make it constexpr.
Make LAC implementation more alike yacc.c's one.
* data/skeletons/lalr1.d, data/skeletons/lalr1.java: Don't expose
yyuser_token_number_max_ and yyundef_token_. Do as in C++: scope them
into yytranslate_, and only when api.token.raw is not defined.
(yyterror_): Rename as...
(yy_error_token_): this.
* data/skeletons/lalr1.d (token_number_type): New.
Use it.
Can't be done in the Java backend, as Java does not have type aliases.
* data/skeletons/lalr1.d, data/skeletons/lalr1.java (yytoken_number_):
Remove, useless.
Was used in ancient C skeletons to support YYPRINT, long obsoleted by
%printer.
It is not used at all. We will remove it also from yacc.c, but
later (see TODO).
* data/skeletons/lalr1.cc, data/skeletons/lalr1.d,
* data/skeletons/lalr1.java (yyerrcode_):
Remove.
Reported by Frank Heckenbach.
https://lists.gnu.org/archive/html/bug-bison/2019-11/msg00016.html
* data/skeletons/c++.m4 (b4_yytranslate_define): Don't use yyeof_ as
if it had two different types.
It is used once against the input argument, which is the value
returned by yylex, which is an "external token number", typically an
int. It is also used as output type, an "internal symbol number".
It turns out that in both cases we mean "0", but let's keep yyeof_
only for the case "internal symbol number", i.e., _after_ conversion
by yytranslate.
This frees us from one cast.
The current code for yysyntax_error for %define parse.error verbose is
fishy (given that YYEMPTY is -2, invalid argument for yytname[]):
static int
yysyntax_error ([...])
{
YYPTRDIFF_T yysize0 = yytnamerr (YY_NULLPTR, yytname[yytoken]);
[...]
if (yytoken != YYEMPTY)
A nearby comment reports
The only way there can be no lookahead present (in yychar) is if
this state is a consistent state with a default action. Thus,
detecting the absence of a lookahead is sufficient to determine
that there is no unexpected or expected token to report. In that
case, just report a simple "syntax error".
So it _is_ possible to call yysyntax_error with yytoken == YYEMPTY,
albeit quite difficult when meaning to, so virtually impossible by
accident (after all, there was never a bug report about this).
I failed to produce a test case, but Joel E. Denny provided me with
one (added to the test suite below). The yacc.c skeleton fails on
this, and once fixed dies on a second problem. The glr.c skeleton was
also dying, but immediately of this second problem.
Indeed we were not allocating space for the error message's final \0.
This was hidden by the fact that we only had error messages with at
least an unexpected token displayed, so with at least one "%s" in the
format string, whose size (2) was included (incorrectly) in the final
size of the message (where the %s have been replaced by the actual
content).
* data/skeletons/glr.c, data/skeletons/yacc.c (yysyntax_error):
Do not invoke yytnamerr on YYEMPTY.
Clarify the computation of the length of the _final_ error message,
with the NUL terminator but without the '%s's.
* tests/conflicts.at (Syntax error in consistent error state):
New, contributed by Joel E. Denny.
Having a file named "exception" is risky: the compiler might use that
file in #include.
Reported by 马俊 <[email protected]>.
* tests/local.at (AT_SKIP_IF_EXCEPTION_SUPPORT_IS_POOR): Generate
'exceptions', not 'exception'.
String literals as tokens serve two distinct purposes: freeing from
having to implement the keyword matching in the scanner, and improving
error messages. Most of the time both can be achieved at the same
time, but on occasions, it does not work so well.
We promote their use for error messages. We will also still support
the former case, but it is _not_ the recommended approach.
* doc/bison.texi (Tokens from Literals): Clearly state that we don't
recommend looking up the token types in the list of token names.
String literals, which allow for better error messages, are (too)
liberally accepted by Bison, which might result in silent errors. For
instance
%type <exVal> cond "condition"
does not define “condition” as a string alias to 'cond' (nonterminal
symbols do not have string aliases). It is rather equivalent to
%nterm <exVal> cond
%token <exVal> "condition"
i.e., it gives the type 'exVal' to the "condition" token, which was
clearly not the intention.
Introduce -Wdangling-alias to catch this.
* src/complain.h, src/complain.c: Add support for -Wdangling-alias.
(argmatch_warning_args): Sort.
* src/symtab.c (symbol_check_defined): Complain about dangling
aliases.
* doc/bison.texi: Document it.
* tests/input.at (Dangling aliases): New test.
On
%token TOKEN1
%type <ival> TOKEN1 TOKEN2 't'
%token TOKEN2
%%
expr:
bison -Wyacc gives
input.y:2.15-20: warning: POSIX yacc reserves %type to nonterminals [-Wyacc]
2 | %type <ival> TOKEN1 TOKEN2 't'
| ^~~~~~
input.y:2.29-31: warning: POSIX yacc reserves %type to nonterminals [-Wyacc]
2 | %type <ival> TOKEN1 TOKEN2 't'
| ^~~
input.y:2.22-27: warning: POSIX yacc reserves %type to nonterminals [-Wyacc]
2 | %type <ival> TOKEN1 TOKEN2 't'
| ^~~~~~
The messages appear to be out of order, but they are emitted when the
error is found.
* src/symtab.h (symbol_class): Add pct_type_sym, used to denote
symbols appearing in %type.
* src/symtab.c (complain_pct_type_on_token): New.
(symbol_class_set): Check that %type is not applied to tokens.
(symbol_check_defined): pct_type_sym also means undefined.
* src/parse-gram.y (symbol_decl.1): Set the class to pct_type_sym.
* src/reader.c (grammar_current_rule_begin): pct_type_sym also means
undefined.
* tests/input.at (Yacc's %type): New.
As an extension to POSIX Yacc, Bison's %type accepts tokens.
Unfortunately with string literals as implicit tokens, this is
misleading, and led some users to write
%type <exVal> cond "condition"
believing that "condition" would be associated to the 'cond'
nonterminal (see https://github.com/apache/httpd/pull/72).
* doc/bison.texi: Promote %nterm rather than %type to declare the type
of nonterminals.
* README: A few fixes.
Explain how to install color support.
* README-hacking: Rename as...
* README-hacking.md: this, and convert to Markdown.
Improve typography.
Improve explanations about update-test.
* src/gram.c (grammar_dump): Print terminals likewise non terminals.
* tests/sets.at (Reduced Grammar): Update test case to catch up the
change and add a test case where prec and assoc are used.
* tests/diagnostics.at (Locations from M4, Tabulations and multibyte
characters from M4): These tests are actually checking a message
coming from C, not from M4. Replace with...
(Complaints from M4): This.
We still have a few old C casts in lalr1.cc, let's get rid of them.
Reported by Frank Heckenbach.
Actually, let's monitor all our casts using easy to grep macros.
Let's use these macros to use the C++ standard casts when we are in
C++.
* data/skeletons/c.m4 (b4_cast_define): New.
* data/skeletons/glr.c, data/skeletons/glr.cc,
* data/skeletons/lalr1.cc, data/skeletons/stack.hh,
* data/skeletons/yacc.c:
Use it and/or its casts.
* tests/actions.at, tests/cxx-type.at,
* tests/glr-regression.at, tests/headers.at, tests/torture.at,
* tests/types.at:
Use YY_CAST instead of C casts.
* configure.ac (warn_cxx): Add -Wold-style-cast.
* doc/bison.texi: Disable it.
* data/skeletons/location.cc: Remove the u (for unsigned) suffix from
the initial line and column.
* NEWS: AFAICT, only C++ backends have their location types changed.
We have too many global variables, adding structure would help. For a
start, let's hide some of the variables closer to their usage.
* src/getargs.c, src/files.h (current_file): Move to...
* src/scan-gram.c: here.
* src/scan-gram.h (gram_in, gram__flex_debug): Remove, make them
private to the scanner.
* src/reader.h, src/reader.c (reader): Take a grammar file as argument.
Move the handling of scanner variables to...
* src/scan-gram.l (gram_scanner_open, gram_scanner_close): here.
(gram_scanner_initialize): Remove, replaced by gram_scanner_open.
* src/main.c: Adjust.
* src/parse-gram (%initial-action): here.
(handle_skeleton): Don't depend on the current file name to look for
"local" skeletons (subject to changes coming from "#lines"): depend
only on the initial file name, the one given on the command line.
Currently there are two globals denoting the input file: grammar_file
is the one from the command line, and current_file which might change
because of #line. Use only the former.
* src/complain.c (error_message): here.
* tests/diagnostics.at: Adjust.
This skeleton uses a single stack of state structures, so it is less
likely to benefit from a stack size reduction than yacc.c (which uses
several stacks: state number, value and location). But it will reduce
the size of the LAC stack.
This skeleton was already using int for state numbers, so, contrary to
yacc.c, this brings nothing for large automata.
Overall, it is still nicer to make the skeletons alike.
* data/skeletons/lalr1.cc (state_type): Here.
Reported by Thomas Petazzoni.
https://lists.gnu.org/archive/html/bug-bison/2019-08/msg00000.html
* examples/c/reccalc/local.mk: Complete dependencies, including for
earlier versions of Automake (for sake of our CI, on top of Ubuntu
Xenial/Bionic, which feature only Automake 1.15).
(%D%/scan.c %D%/scan.h): Upgrade to the full version provided in
Automake's documentation.
See https://lists.gnu.org/archive/html/bug-bison/2019-10/msg00061.html
to https://lists.gnu.org/archive/html/bug-bison/2019-10/msg00073.html.
Paul Eggert's changes in gnulib do fix the issue for modern GCCs (7,
8, 9) on macOS. Unfortunately these warnings are back on the
CI (GNU/Linux) with GCC 4.6, 4.7, (not 4.8) and 4.9.
Disable the warning locally.
* configure.ac (warn_common, warn_tests): Remove -Wtype-limits.
* src/system.h (IGNORE_TYPE_LIMITS_BEGIN, IGNORE_TYPE_LIMITS_END): New.
* src/InadequacyList.c, src/parse-gram.c, src/parse-gram.y,
* src/symtab.c: Use it.
This is really weird: GCC points to the LHS of the assignment...
260. headers.at:184: testing Sane headers: api.pure api.push-pull=both ...
tests/headers.at:184: COLUMNS=1000; export COLUMNS; bison --color=no -fno-caret -d -o input.c input.y
tests/headers.at:184: $CC $CFLAGS $CPPFLAGS -c -o input.o input.c
stderr:
input.c: In function 'yyparse':
input.c:1276:16: error: 'yylval' may be used uninitialized in this function [-Werror=maybe-uninitialized]
1276 | yylval = *yypushed_val;
| ~~~~~~~^~~~~~~~~~~~~~~
input.c: In function 'yypull_parse':
input.c:1276:16: error: 'yylval' may be used uninitialized in this function [-Werror=maybe-uninitialized]
1276 | yylval = *yypushed_val;
| ~~~~~~~^~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
stdout:
tests/headers.at:184: exit code was 1, expected 0
See also d87c8ac79a
and 9645a2b20e.
* tests/headers.at (Several parsers, Several parsers): Disable these
warnings when in push parser.
The documentation for Oracle Solaris Studio 12.3 (Sun C++ 5.12
2011/11/16) says it supports C++03. This compiler rejects the
location.cc use of std::max for some reason; I don’t know why
since I don’t use C++ as a rule. The simplest workaround is to
open-code ‘max’.
* data/skeletons/location.cc (add_):
Do max by hand rather than relying on std::max.
Don’t include <algorithm.h>; no longer needed.
As documented in the Autoconf manual, Solaris 10 sed rejects
script labels contianing more than 7 characters. POSIX requires
support for at least 8 characters, but we might as well be portable
to Solaris 10 which is still supported.
* tests/local.at (AT_SETS_CHECK): Use only the first 7 characters
in sed labels.
* src/symtab.c: Include intprops.h
(symbol_user_token_number_set): Don’t allow user_token_number ==
INT_MAX because too much other code adds 1 to the user token number.
(symbols_token_translations_init): Complain on integer overflow
instead of indulging in undefined behavior.
* src/scan-gram.l: Include errno.h, for errno.
(scan_integer, handle_syncline): Check for integer overflow.
* tests/input.at (too-large.y): Adjust to match new diagnostics.
* src/parse-gram.y: Include intprops.h.
(handle_require): Don’t indulge in undefined behavior if the major
or minor number is out of range. Instead, check that the
resulting value is nonnegative, fits in int, and that the minor
number is less than 100. Also, check that a number was parsed.
Sun C 5.12 defines __SUNPRO_C to 0x5120 but diagnoses
‘__attribute__ ((__unused__))’. Change the ifdefs to use
the same method as Gnulib in this area.
* data/skeletons/c.m4 (YY_ATTRIBUTE): Remove, since
not all attributes were added in the same compiler version.
(YY_ATTRIBUTE_PURE, YY_ATTRIBUTE_UNUSED):
Use specific GCC version for each attribute.
Pay no attention to __SUNPRO_C.
* tests/headers.at (Several parsers): Tighten tests accordingly.
Oracle Solaris Studio 12.3 (Sun C 5.12 2011/11/16) by default does
not conform to C99; it defines __STDC_VERSION__ to be 199409L, so
the Bison code does not include <stdint.h> (not required by C89
amendment 1) even though this compiler does have <stdint.h>. On
this platform <limits.h> defines INT_LEAST8_MAX (POSIX allows
this) so the skeleton got confused and thought that <stdint.h> had
been included even though it wasn’t.
* data/skeletons/c.m4 (b4_c99_int_type_define) [!__PTRDIFF_MAX__]:
Always include <limits.h>.
(YY_STDINT_H): Define when <stdint.h> was included.
All uses of expressions like ‘defined INT_LEAST8_MAX’ changed to
‘defined YY_STDINT_H’, since Sun C 5.12 <limits.h> defines macros
like INT_LEAST8_MAX but does not declare types like int_least8_t.
Let's make a difference between places where Perl is required for the
test (AT_PERL_REQUIRE), and the places where it's used to run the
test, but it's not not to run the test (AT_PERL_CHECK).
* tests/local.at (AT_REQUIRE): New.
(AT_PERL_CHECK, AT_PERL_REQUIRE): New.
Use them where appropriate.
* tests/local.mk ($(TESTSUITE)): Beware not to start the line with
'-pi' if Perl is empty, as Make understands this as "it's ok to fail".
Which it is not.
My previous tests (with ./configure PERL=false) have been fooled by
configure, that managed to find perl anyway. This time, I ran this on
a Fedora in Docker, without Perl.
* tests/calc.at, tests/diagnostics.at, tests/headers.at,
* tests/input.at, tests/local.at, tests/named-refs.at,
* tests/output.at, tests/regression.at, tests/skeletons.at,
* tests/synclines.at, tests/torture.at: Don't require Perl.
* data/skeletons/c.m4 (b4_c99_int_type_define): Reorder to put the
signed types first, since they’re simpler and this keeps similar
code closer. For signed types, don’t bother checking whether the
type promotes to int since the type must be signed anyway. For
unsigned types, protect a test like ‘UCHAR_MAX <= INT_MAX’ with
‘!defined __UINT_LEAST8_MAX__’, as otherwise the logic is wrong
for oddball platforms; and once we do that, there should no need
for ‘defined INT_MAX’ so remove that.
Currently we face test suite failures in different environments,
because of a conflict between the definitions of isnan by gnulib, and
by the C++ library:
262. headers.at:186: testing Sane headers: %locations %debug c++ ...
./headers.at:186: COLUMNS=1000; export COLUMNS; bison --color=no -fno-caret -d -o input.cc input.y
./headers.at:186: $CXX $CXXFLAGS $CPPFLAGS -c -o input.o input.cc
stderr:
In file included from /usr/include/c++/4.8.2/cmath:44:0,
from /usr/include/c++/4.8.2/random:38,
from /usr/include/c++/4.8.2/bits/stl_algo.h:65,
from /usr/include/c++/4.8.2/algorithm:62,
from location.hh:41,
from input.hh:90,
from input.cc:50:
/u/cs/fac/eggert/src/gnu/bison/lib/math.h: In function 'bool isnan(double)':
/u/cs/fac/eggert/src/gnu/bison/lib/math.h:2849:1: error: new declaration 'bool isnan(double)'
_GL_MATH_CXX_REAL_FLOATING_DECL_2 (isnan, isnan, bool)
^
In file included from /usr/include/features.h:375:0,
from /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/os_defines.h:39,
from /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/c++config.h:2097,
from /usr/include/c++/4.8.2/cstdlib:41,
from input.hh:48,
from input.cc:50:
/usr/include/bits/mathcalls.h:235:1: error: ambiguates old declaration 'int isnan(double)'
__MATHDECL_1 (int,isnan,, (_Mdouble_ __value)) __attribute__ ((__const__));
^
There might be something to do in gnulib about this, but I believe
that gnulib should not be used in the test suite in the first place.
The test suite should work with other compilers than the one used to
compile the package. For a start, Bison sources are more
demanding (C99) than the generated parsers. Last time I tried, tcc
for example, was not able to compile Bison, yet our generated parsers
should compile cleanly with it.
Besides the problem at hand is with the C++ compiler, with is not the
one used to set up gnulib at configuration-time (config.h is mainly
built from probing the C compiler).
We should really not depend on gnulib in tests.
This was introduced in 2001 to check whether including
stdlib.h/string.h is safe thanks to STDC_HEADERS
(2ce1014469). Today, we assume at least
a C90 compiler, it should be safe enough.
* tests/local.at, tests/testsuite.h: Do not include config.h.
* tests/atlocal.in (conftest.cc): Likewise.
(CPPFLAGS): Do not expose lib/, as because of this we might picked up
gnulib replacement headers for system headers.
* tests/input.at: Use int instead of ptrdiff_t, for easier portability
(some machine on the CI did not find ptrdiff_t).
* tests/c++.at: Add missing include for getchar.
A number of portability issues with GCC 4.6 .. 4.9 (inclusive):
input.c:184:7: error: "UCHAR_MAX" is not defined [-Werror=undef]
#elif UCHAR_MAX <= INT_MAX
^
input.c:184:20: error: "INT_MAX" is not defined [-Werror=undef]
#elif UCHAR_MAX <= INT_MAX
^
input.c:202:7: error: "USHRT_MAX" is not defined [-Werror=undef]
#elif USHRT_MAX <= INT_MAX
^
input.c:202:20: error: "INT_MAX" is not defined [-Werror=undef]
#elif USHRT_MAX <= INT_MAX
^
* data/skeletons/c.m4 (b4_c99_int_type_define): Don't assume they are
defined.
Flex should not be required to build Bison or run the test suite (of
course it is needed for maintaining Bison). Yet the Automake
conditional FLEX_WORKS does not work.
* m4/flex.m4 (_AC_PROG_LEX_YYTEXT_DECL): Since this is called
conditionally, don't define LEX_IS_FLEX here, but rather...
(AC_PROG_LEX): here.
* configure.ac: Be more cautious about possibly undefined variables.
That way, glr.c can use it too.
* data/skeletons/c.m4 (b4_int_type):
Do not special-case ‘char’; it’s not worth the trouble,
as clang complains about char subscripts.
(b4_c99_int_type, b4_c99_int_type_define): New macros,
taken from yacc.c.
* data/skeletons/glr.c: Use b4_int_type_define.
* data/skeletons/yacc.c (b4_int_type): Remove, since there’s
no longer any need to redefine it.
Use b4_c99_int_type_define rather than its body.
This changes the Yacc skeleton to use “least” integer types to
keep tables smaller on some platforms, which should lessen cache
pressure. Since Bison uses the Yacc skeleton, it follows suit.
* data/skeletons/yacc.c: Include limits.h and stdint.h if this
seems to be needed.
(yytype_uint8, yytype_int8, yytype_uint16, yytype_int16):
If available, use GCC predefined macros __INT_MAX__ etc. to select
a “least” type, as this avoids namespace hassles. Otherwise, if
available fall back on selecting a “least” type via the C99 macros
INT_MAX, INT_LEAST8_MAX, etc. Otherwise, fall further back on one of
the builtin C99 types signed char, short, and int. Make sure that
any selected type promotes to int. Ignore any macros YYTYPE_INT16,
YYTYPE_INT8, YYTYPE_UINT16, YYTYPE_UINT8 defined by the user.
(ptrdiff_t, PTRDIFF_MAX): Simplify in the light of the above.
(yytype_uint8, yytype_uint16): Do not assume that unsigned char
and unsigned short promote to int, as this isn’t true on some
platforms (e.g., TI TMS320C55x).
* src/parse-gram.y (YYTYPE_INT16, YYTYPE_INT8, YYTYPE_UINT16)
(YYTYPE_UINT8): Remove, as these are no longer effective.
* data/skeletons/yacc.c (YYPTRDIFF_T, YYPTRDIFF_MAXIMUM):
Default to long, not int.
(yy_lac_stack_realloc, yy_lac, yytnamerr, yyparse):
Avoid casts to YYPTRDIFF_T that were masking the problem.
Because the checking of the grammar is made by phases after the whole
grammar was read, we sometimes have diagnostics that look weird. In
some case, within one type of checking, the entities are not checked
in the order in which they appear in the file. For instance, checking
symbols is done on the list of symbols sorted by tag:
foo.y:1.20-22: warning: symbol BAR is used, but is not defined as a token and has no rules [-Wother]
1 | %destructor {} QUX BAR
| ^~~
foo.y:1.16-18: warning: symbol QUX is used, but is not defined as a token and has no rules [-Wother]
1 | %destructor {} QUX BAR
| ^~~
Let's sort them by location instead:
foo.y:1.16-18: warning: symbol 'QUX' is used, but is not defined as a token and has no rules [-Wother]
1 | %destructor {} QUX BAR
| ^~~
foo.y:1.20-22: warning: symbol 'BAR' is used, but is not defined as a token and has no rules [-Wother]
1 | %destructor {} QUX BAR
| ^~~
* src/location.h (location_cmp): Be robust to empty file names.
* src/symtab.c (symbol_cmp): Sort by location.
* tests/input.at: Adjust expectations.
From
input.y:1.17-19: warning: symbol baz is used, but is not defined as a token and has no rules [-Wother]
1 | %printer {} foo baz
| ^~~
to
input.y:1.17-19: warning: symbol 'baz' is used, but is not defined as a token and has no rules; did you mean 'bar'? [-Wother]
1 | %printer {} foo baz
| ^~~
| bar
* bootstrap.conf: We need fstrcmp.
* src/symtab.c (symbol_from_uniqstr_fuzzy): New.
(complain_symbol_undeclared): Use it.
* tests/diagnostics.at (Suggestions): New.
* data/bison-default.css (insertion): Rename as...
(fixit-insert): this, as this is what GCC uses.
* src/symtab.c (complain_symbol_undeclared): New.
Use it.
Use quote on the guilty symbol (like GCC does, and we also do
elsewhere).
* tests/input.at: Adjust.
Both are stored in a hash, and back in the days, we used to iterate
over these tables using hash_do_for_each. However, the order of
traversal was not deterministic, which was a nuisance for
deterministic output (and therefore also a problem for tests). So at
some point (83b60c97ee) we generated a
sorted list of these symbols, and symbols_do actually iterated on that
list. But we kept the constraints of using hash_do_for_each, which
requires a lot of ceremonial code, and makes it hard/unnatural to
preserve data between iterations (see the next commit).
Alas, this is C, not C++.
Let's remove this abstraction, and directly iterate on the sorted
tables.
* src/symtab.c (symbols_do): Remove.
Adjust callers to use a simple for-loop instead.
(table_sort): New.
(symbols_check_defined): Use it.
(symbol_check_defined_processor, symbol_pack_processor)
(semantic_type_check_defined_processor, symbol_translation_processor):
Remove.
Simplify the corresponding functions (that no longer need to return a
bool).
This commit adds the suggestion in green, on the line below the
caret-and-tildes.
foo.y:1.1-14: warning: deprecated directive: '%error-verbose', use '%define parse.error verbose' [-Wdeprecated]
1 | %error-verbose
| ^~~~~~~~~~~~~~
| %define parse.error verbose
The current approach, with location_caret_suggestion, is fragile:
there's a protocol of calls to the complain functions which is strict.
We should rather have a richer structure describing the diagnostics,
including with submessages such as the suggestions, passed in the end
to the routines in charge of formatting and printing them.
* src/location.h, src/location.c (location_caret_suggestion): New.
* src/complain.c (deprecated_directive): Use it.
* tests/diagnostics.at, tests/input.at: Adjust expectations.
* src/location.c (caret_set_file): New.
Store the current line's length in caret_info.line_len.
Pay attention to fseek's return value.
Extracted from...
(location_caret): here.
I no longer agree with that item, there are indeed two things to
report: lack of definition, and being useless. We could have either
one without the other, they are not directly related.
input.c: In function 'int yyparse()':
input.c: error: conversion to 'long int' from 'long unsigned int'
may change the sign of the result [-Werror=sign-conversion]
yyes_capacity = sizeof yyesa / sizeof *yyes;
^
cc1plus: all warnings being treated as errors
* data/skeletons/yacc.c: here.
GCC 4.8 reports:
input.y:57:33: error: conversion to 'int' from 'long unsigned int'
may alter its value [-Werror=conversion]
int input_elts = sizeof input / sizeof input[0];
^
* tests/local.at (AT_YYLEX_DEFINE(c)): Add a cast (sorry, Paul!).
* src/location.c (caret_set_file):
* src/scan-code.l (contains_dot_or_dash):
Do not quietly convert pointer to bool, as Oracle Developer Studio
12.6 complains and it is arguably confusing style anyway.
For instance with GCC 4.9 and --enable-gcc-warnings:
25. input.at:1201: testing Torturing the Scanner ...
../../tests/input.at:1344: $CC $CFLAGS $CPPFLAGS -c -o input.o input.c
stderr:
input.c:239:18: error: "__STDC_VERSION__" is not defined [-Werror=undef]
# elif 199901 <= __STDC_VERSION__
^
input.c:256:18: error: "__STDC_VERSION__" is not defined [-Werror=undef]
# elif 199901 <= __STDC_VERSION__
^
* data/skeletons/yacc.c: Check that __STDC_VERSION__ is defined before
using it.
* doc/bison.texi (Mfcalc Symbol Table, Mfcalc Lexer):
Don’t abort on memory allocation failure or integer overflow.
Instead, comment that these things aren’t checked for.
On the CI with GCC 6:
examples/c++/calc++/parser.cc:845:5: error: 'ptrdiff_t' was not declared in this scope
ptrdiff_t yycount = 0;
^~~~~~~~~
examples/c++/calc++/parser.cc:845:5: note: suggested alternatives:
/usr/include/x86_64-linux-gnu/c++/6/bits/c++config.h:202:28: note: 'std::ptrdiff_t'
typedef __PTRDIFF_TYPE__ ptrdiff_t;
^~~~~~~~~
* data/skeletons/lalr1.cc: Qualify ptrdiff_t and size_t with std::.
* tests/torture.at (Big horizontal): Adjust to recent changes with
integers. If there are states 0..256, Bison now uses a signed
rather than an unsigned 16-bit integer.
This patch contains more fixes to prefer signed to unsigned
integer types, as modern tools like 'gcc -fsanitize=undefined'
can check for signed integer overflow but not unsigned overflow.
* NEWS: Document the API change.
* boostrap.conf (gnulib_modules): Add intprops.
* data/skeletons/glr.c: Include stddef.h and stdint.h,
since this skeleton can assume C99 or later.
(YYSIZEMAX): Now signed, and the minimum of SIZE_MAX and PTRDIFF_MAX.
(yybool) [!__cplusplus]: Now signed (which is how bool behaves).
(YYTRANSLATE): Avoid use of unsigned, and make the macro
safe even for values greater than UINT_MAX.
(yytnamerr, struct yyGLRState, struct yyGLRStateSet, struct yyGLRStack)
(yyaddDeferredAction, yyinitStateSet, yyinitGLRStack)
(yyexpandGLRStack, yymarkStackDeleted, yyremoveDeletes)
(yyglrShift, yyglrShiftDefer, yy_reduce_print, yydoAction)
(yyglrReduce, yysplitStack, yyreportTree, yycompressStack)
(yyprocessOneStack, yyreportSyntaxError, yyrecoverSyntaxError)
(yyparse, yy_yypstack, yypstack, yypdumpstack):
* tests/input.at (Torturing the Scanner):
Prefer ptrdiff_t to size_t.
* data/skeletons/c++.m4 (b4_yytranslate_define):
* src/AnnotationList.c (AnnotationList__computePredecessorAnnotations):
* src/AnnotationList.h (AnnotationIndex):
* src/InadequacyList.h (InadequacyListNodeCount):
* src/closure.c (closure_new):
* src/complain.c (error_message, complains, complain_indent)
(complain_args, duplicate_directive, duplicate_rule_directive):
* src/gram.c (nritems, ritem_print, grammar_dump):
* src/ielr.c (ielr_compute_ritem_sees_lookahead_set)
(ielr_item_has_lookahead, ielr_compute_annotation_lists)
(ielr_compute_lookaheads):
* src/location.c (columns, boundary_print, location_print):
* src/muscle-tab.c (muscle_percent_define_insert)
(muscle_percent_define_check_values):
* src/output.c (prepare_rules, prepare_actions):
* src/parse-gram.y (id, handle_require):
* src/reader.c (record_merge_function_type, packgram):
* src/reduce.c (nuseless_productions, nuseless_nonterminals)
(inaccessable_symbols):
* src/relation.c (relation_print):
* src/scan-code.l (variant, variant_table_size, variant_count)
(variant_add, get_at_spec, show_sub_message, show_sub_messages)
(parse_ref):
* src/scan-gram.l (<SC_ESCAPED_STRING,SC_ESCAPED_CHARACTER>)
(scan_integer, convert_ucn_to_byte, handle_syncline):
* src/scan-skel.l (at_complain):
* src/symtab.c (complain_symbol_redeclared)
(complain_semantic_type_redeclared, complain_class_redeclared)
(symbol_class_set, complain_user_token_number_redeclared):
* src/tables.c (conflict_tos, conflrow, conflict_table)
(conflict_list, save_row, pack_vector):
* tests/local.at (AT_YYLEX_DEFINE(c)):
Prefer signed to unsigned integer.
* data/skeletons/lalr1.cc (yy_lac_check_):
* tests/actions.at (_AT_CHECK_PRINTER_AND_DESTRUCTOR):
* tests/local.at (AT_YYLEX_DEFINE(c)):
Omit now-unnecessary casts.
* data/skeletons/location.cc (b4_location_define):
* doc/bison.texi (Mfcalc Lexer, C++ position, C++ location):
Prefer int to unsigned for line and column numbers.
Change example to abort explicitly on memory exhaustion,
and fix an off-by-one bug that led to undefined behavior.
* data/skeletons/stack.hh (stack::operator[]):
Also allow ptrdiff_t indexes.
(stack::pop, slice::slice, slice::operator[]):
Index arg is now ptrdiff_t, not int.
(stack::ssize): New method.
(slice::range_): Now ptrdiff_t, not int.
* data/skeletons/yacc.c (b4_state_num_type): Remove.
All uses replaced by b4_int_type.
(YY_CONVERT_INT_BEGIN, YY_CONVERT_INT_END): New macros.
(yylac, yyparse): Use them around conversions that -Wconversion
would give false alarms about. Omit unnecessary casts.
(yy_stack_print): Use int rather than unsigned, and omit
a cast that doesn’t seem to be needed here any more.
* examples/c++/variant.yy (yylex):
* examples/c++/variant-11.yy (yylex):
Omit no-longer-needed conversions to unsigned.
* src/InadequacyList.c (InadequacyList__new_conflict):
Don’t assume *node_count is unsigned.
* src/output.c (muscle_insert_unsigned_table):
Remove; no longer used.
* NEWS: Mention this.
* data/skeletons/c.m4 (b4_int_type):
Prefer char if it will do, and prefer signed types to unsigned if
either will do.
* data/skeletons/glr.c (yy_reduce_print): No need to
convert rule line to unsigned long.
(yyrecoverSyntaxError): Put action into an int to
avoid GCC warning of using a char subscript.
* data/skeletons/lalr1.cc (yy_lac_check_, yysyntax_error_):
Prefer ptrdiff_t to size_t.
* data/skeletons/yacc.c (b4_int_type):
Prefer signed types to unsigned if either will do.
* data/skeletons/yacc.c (b4_declare_parser_state_variables):
(YYSTACK_RELOCATE, YYCOPY, yy_lac_stack_realloc, yy_lac)
(yytnamerr, yysyntax_error, yyparse): Prefer ptrdiff_t to size_t.
(YYPTRDIFF_T, YYPTRDIFF_MAXIMUM): New macros.
(YYSIZE_T): Fix "! defined YYSIZE_T" typo.
(YYSIZE_MAXIMUM): Take the minimum of PTRDIFF_MAX and SIZE_MAX.
(YYSIZEOF): New macro.
(YYSTACK_GAP_MAXIMUM, YYSTACK_BYTES, YYSTACK_RELOCATE)
(yy_lac_stack_realloc, yyparse): Use it.
(YYCOPY, yy_lac_stack_realloc): Cast to YYSIZE_T to pacify GCC.
(yy_reduce_print): Use int instead of unsigned long when int
will do.
(yy_lac_stack_realloc): Prefer long to unsigned long when
either will do.
* tests/regression.at: Adjust to these changes.
Currently we properly use the "best" integral type for tables,
including those storing state numbers. However the variables for
state numbers used in yyparse (and its dependencies such as
yy_stack_print) still use int16_t invariably. As a consequence, very
large models overflow these variables.
Let's use the "best" type for these variables too. It turns out that
we can still use 16 bits for twice larger automata: stick to unsigned
types.
However using 'unsigned' when 16 bits are not enough is troublesome
and generates tons of warnings about signedness issues. Instead,
let's use 'int'.
Reported by Tom Kramer.
https://lists.gnu.org/archive/html/bug-bison/2019-09/msg00018.html
* data/skeletons/yacc.c (b4_state_num_type): New.
(yy_state_num): Be computed from YYNSTATES.
* tests/linear: New.
* tests/torture.at (State number type): New.
Use it.
Instead of
#define YYPACT_NINF -130
#define yypact_value_is_default(Yystate) \
(!!((Yystate) == (-130)))
generate
#define YYPACT_NINF (-130)
#define yypact_value_is_default(Yyn) \
((Yyn) == YYPACT_NINF)
* data/skeletons/c.m4 (b4_table_value_equals): Add support for $4.
* data/skeletons/glr.c, data/skeletons/yacc.c: Use it.
Also, use shorter macro argument names, the name of the macro is clear
enough.
* upstream/maint:
c++: add copy ctors for compatibility with the IAR compiler
CI: show git status
CI: disable ICC
tests: pass -jN from Make to the test suite
quotearg: avoid leaks
maint: post-release administrivia
Reported by Andreas Damm.
https://savannah.gnu.org/support/?110032
* data/skeletons/lalr1.cc (stack_symbol_type::operator=): New
overload, const, to please the IAR C++ compiler (version ca 2013).
* bootstrap.conf: We need winsz-ioctl and winsz-termios.
* src/location.c (columns): Use winsize to get the number of
columns.
Code taken from the GNU Coreutils.
* src/location.h, src/location.c (caret_init): New.
* src/complain.c (complain_init): Call it.
* tests/bison.in: Export COLUMNS so that users of tests/bison can
enjoy proper line truncation.
* src/location.c (min_int, columns): New.
(location_caret): Compute the line width. Based on it, compute how
many columns must be skipped before the quoted location and truncated
after, to fit the sceen width.
* tests/local.at (AT_QUELL_VALGRIND): Transform into...
(AT_SET_ENV_IF, AT_SET_ENV): these.
Define COLUMNS to protect the test suite from the user's environment.
So far diagnostics were cheating: in addition to the 'column' field of
locations (based on actual screen width per multibyte characters and
on tabulation expansion), the scanner sets the 'byte' field.
Diagnostics used this byte count to decide where to insert (color)
style.
We want to be able to truncate the quoted lines when there are too
wide to fit the screen. This requires that the diagnostics learn how
to count columns, the byte-in-boundary trick no longer works.
Bytes are still used for fix-its.
* bootstrap.conf: We need mbfile for mbf_getc.
* src/location.c (caret_info): We need an mbfile.
(caret_set_file): Initialize it.
(caret_getc): Convert to mbfile.
(location_caret): Instead of relying on the byte position to decide
where to insert the color style, count the current column using
boundary_compute.
* src/location.c (caret_info): Replace file and line with pos, a
boundary. This will allow us to use features of the boundary type,
such as boundary_compute.
The handling of the contributions of the tabulations in the columns is
burried inside location_compute. We will soon be willing to use the
boundary part of the computation (to compute the current column number
each time we read a multibyte char).
* src/location.c (boundary_compute): New, extracted from...
(location_compute): here.
This changes the traces from
Reading a token:
Now at end of input.
to
Reading a token:
Next token is token $end (7FFEE56E6474)
which is ok. Actually it is even better, as it gives the location
when locations are enabled, and is clearer when rules explicitly use
the EOF token.
* data/skeletons/lalr1.d (yytranslate_): Handle eof here, as is done
in lalr1.cc.
Bison used to feature %raw, documented as follows:
@item %raw
The output file @file{@var{name}.h} normally defines the tokens with
Yacc-compatible token numbers. If this option is specified, the
internal Bison numbers are used instead. (Yacc-compatible numbers start
at 257 except for single character tokens; Bison assigns token numbers
sequentially for all tokens starting at 3.)
Unfortunately, as far as I can tell, it never worked: token numbers
are indeed changed in the generated tables (from external token number
to internal), yet the code was still applying the mapping from
external token numbers to internal token numbers.
This commit reintroduces the feature as it was expected to be.
* data/skeletons/bison.m4 (b4_token_format): When api.token.raw is
enabled, use the internal token number.
* data/skeletons/yacc.c (yytranslate): Don't emit if api.token.raw is
enabled.
(YYTRANSLATE): Adjust.
We used to treat lone CRs (\r, aka ^M) as regular NLs (\n), probably
to please Classic MacOS. As of today, it makes more sense to treat \r
like a plain white space character.
https://lists.gnu.org/archive/html/bison-patches/2019-09/msg00027.html
* src/scan-gram.l (no_cr_read): Remove. Instead, use...
(eol): this new abbreviation denoting end-of-line.
* src/location.c (caret_getc): New.
(location_caret): Use it.
* tests/diagnostics.at (Carriage return): Adjust expectations.
(CR NL): New.
When the input file contains lone CRs (aka, ^M, \r), the locations see
a new line. Diagnostics look only at \n as end-of-line, so sometimes
there is an offset in diagnostics. Worse yet: sometimes we loop
endlessly waiting for \n to come from a continuous stream of EOF.
Fix that:
- check for EOF
- beware not to call end_use_class if begin_use_class was not
called (which would abort). This could happen if the actual
line is shorter that the expected one.
Prompted by a (private) report from Marc Schönefeld.
* src/location.c (location_caret): here.
* tests/diagnostics.at (Carriage return): New.
When the input file contains lone CRs (aka, ^M, \r), the locations see
a new line. Diagnostics look only at \n as end-of-line, so sometimes
there is an offset in diagnostics. Worse yet: sometimes we loop
endlessly waiting for \n to come from a continuous stream of EOF.
Fix that:
- check for EOF
- beware not to call end_use_class if begin_use_class was not
called (which would abort). This could happen if the actual
line is shorter that the expected one.
Prompted by a (private) report from Marc Schönefeld.
* src/location.c (location_caret): here.
* tests/diagnostics.at (Carriage return): New.
With
%token EOF 0 EOF 0
we get
input.y:3.14-16: warning: symbol EOF redeclared [-Wother]
3 | %token EOF 0 EOF 0
| ^~~
input.y:3.8-10: previous declaration
3 | %token EOF 0 EOF 0
| ^~~
Assertion failed: (nsyms == ntokens + nvars), function check_and_convert_grammar,
file /Users/akim/src/gnu/bison/src/reader.c, line 839.
Reported by Marc Schönefeld.
* src/symtab.c (symbol_user_token_number_set): Register only the
first definition of the end of input token.
* tests/input.at (Symbol redeclared): Check that case.
The CI, with CC='gcc-7 -fsanitize=undefined,address
-fno-omit-frame-pointer', reports:
calc.cc:1652:50: runtime error: load of value 190, which is not a valid value for type 'bool'
../../tests/calc.at:867: cat stderr
--- expout 2019-09-05 20:30:37.887257545 +0000
+++ /home/travis/build/bison-3.4.1.72-79a1-dirty/_build/tests/testsuite.dir/at-groups/438/stdout 2019-09-05 20:30:37.887257545 +0000
@@ -1 +1,2 @@
syntax error
+calc.cc:1652:50: runtime error: load of value 190, which is not a valid value for type 'bool'
438. calc.at:867: 438. Calculator glr.cc (calc.at:867): FAILED (calc.at:867)
The problem is that yylookaheadNeeds is not initialized in
yyinitStateSet, and when it is copied, the value is not 0 or 1.
* data/skeletons/glr.c (yylookaheadNeeds): Initialize yylookaheadNeeds.
hash_initialize returns NULL when out of memory. Check for it, and
die cleanly instead of crashing.
Reported by 江 祖铭 (Zu-Ming Jiang).
https://lists.gnu.org/archive/html/bug-bison/2019-08/msg00015.html
* src/muscle-tab.c, src/state.c, src/symtab.c, src/uniqstr.c:
Check the value returned by hash_initialize.
https://lists.gnu.org/archive/html/bison-patches/2019-08/msg00007.html
When Bison is started with a flag that suppresses warning messages, the
error_message() function can produce a few gigabytes of indentation
because of a dangling pointer.
* src/complain.c (error_message): Don't reset indent_ptr here, but...
(complain_indent): here.
* tests/diagnostics.at (Indentation with message suppression): Check
this case.
* .travis.yml: Bionic is now available, with GCC8.
GCC7 sanitizers work, but they are too longer: cover only part 1.
Redefine part 1 and part 2 so that part 1 is really the core of the
tests: not playing with POSIX and C++ compiler for C code.
Travis answered favorably to my suggestion to provide a means to
disable git clone on some jobs (issue 7542). See
https://docs.travis-ci.com/user/customizing-the-build/#disabling-git-clone.
* .travis.yml: Disable git globally, enable it for i. the compile job,
and ii. the test job on ICC which needs the install-icc.sh script.
With
%token EOF 0 EOF 0
we get
input.y:3.14-16: warning: symbol EOF redeclared [-Wother]
3 | %token EOF 0 EOF 0
| ^~~
input.y:3.8-10: previous declaration
3 | %token EOF 0 EOF 0
| ^~~
Assertion failed: (nsyms == ntokens + nvars), function check_and_convert_grammar,
file /Users/akim/src/gnu/bison/src/reader.c, line 839.
Reported by Marc Schönefeld.
* src/symtab.c (symbol_user_token_number_set): Register only the
first definition of the end of input token.
* tests/input.at (Symbol redeclared): Check that case.
The CI, with CC='gcc-7 -fsanitize=undefined,address
-fno-omit-frame-pointer', reports:
calc.cc:1652:50: runtime error: load of value 190, which is not a valid value for type 'bool'
../../tests/calc.at:867: cat stderr
--- expout 2019-09-05 20:30:37.887257545 +0000
+++ /home/travis/build/bison-3.4.1.72-79a1-dirty/_build/tests/testsuite.dir/at-groups/438/stdout 2019-09-05 20:30:37.887257545 +0000
@@ -1 +1,2 @@
syntax error
+calc.cc:1652:50: runtime error: load of value 190, which is not a valid value for type 'bool'
438. calc.at:867: 438. Calculator glr.cc (calc.at:867): FAILED (calc.at:867)
The problem is that yylookaheadNeeds is not initialized in
yyinitStateSet, and when it is copied, the value is not 0 or 1.
* data/skeletons/glr.c (yylookaheadNeeds): Initialize yylookaheadNeeds.
* .travis.yml: Bionic is now available, with GCC8.
GCC7 sanitizers work, but they are too longer: cover only part 1.
Redefine part 1 and part 2 so that part 1 is really the core of the
tests: not playing with POSIX and C++ compiler for C code.
hash_initialize returns NULL when out of memory. Check for it, and
die cleanly instead of crashing.
Reported by 江 祖铭 (Zu-Ming Jiang).
https://lists.gnu.org/archive/html/bug-bison/2019-08/msg00015.html
* src/muscle-tab.c, src/state.c, src/symtab.c, src/uniqstr.c:
Check the value returned by hash_initialize.
https://lists.gnu.org/archive/html/bison-patches/2019-08/msg00007.html
When Bison is started with a flag that suppresses warning messages, the
error_message() function can produce a few gigabytes of indentation
because of a dangling pointer.
* src/complain.c (error_message): Don't reset indent_ptr here, but...
(complain_indent): here.
* tests/diagnostics.at (Indentation with message suppression): Check
this case.
Implement lookahead correction (LAC) for the C++ skeleton. LAC is a
mechanism to make sure that we report the correct list of expected
tokens if a syntax error occurs. So far, LAC was only supported for
the C skeleton "yacc.c".
* data/skeletons/lalr1.cc: Add LAC support.
* doc/bison.texi: Update.
* data/skeletons/yacc.c (yysyntax_error): Change the nesting of `m4`
conditions slightly to make it more readable.
The generated C code stays unchanged.
* tests/regression.at: Use %expect to avoid warnings.
Set the keywords to facilitate running specific tests.
Use macros such as AT_YYLEX_DECLARE to facilitate tests for other
languages.
Likewise for AT_FULL_COMPILE.
In Texinfo. empty lines in multitable rows generate empty lines in the
output. Avoid them altogether.
With help from Gavin Smith.
https://lists.gnu.org/archive/html/bug-texinfo/2019-07/msg00000.html
* build-aux/cross-options.pl: Separate rows with empty lines.
So, to be more readable, generate a single line for each row.
Use Perl format to this end.
See the previous commit. This option should be removed, -o suffices.
* src/getargs.c (FIXED_OUTPUT_FILES): New.
Add support for it.
(getargs): Define loc, and use it.
This is safer when we need to pass a pointer to a location.
The name fixed-output-files is pretty clear: generate y.tab.c, as Yacc
does. So let's detach this from %yacc which does more: it requires
POSIX Yacc behavior.
This directive is obsolete since December 29th 2001
8c9a50bee1. It does not show in the
doc. I don't want to spend more time on improving its diagnostics, it
could be removed just as well as far as I'm concerned.
* src/scan-gram.l, src/parse-gram.y (%fixed-output-files): Detach from
%yacc.
Years ago we moved from 'look-ahead' to 'lookahead', and that alias
was kept for backward compatibility. But now that we use argmatch to
generate the documentation, that value clutters the doc.
* src/getargs.c (argmatch_report_args): Remove the
--report=look-aheads alias.
The doc says that -Dfoo=bar is the same as %define foo "bar". It is
not: the quotes are not added (and it makes a difference).
* doc/bison.texi (Tuning the Parser): Fix the definition of -D/-F
* src/getargs.c (usage): Likewise.
Let's clarify --help: use clearer "section" names, as in the doc.
Move --yacc to where it belongs.
* src/getargs.c (usage): Rename "Parser" as "Tuning the Parser", as in
the doc.
Rename "Output" as "Output Files"
Move --yacc to "Tuning the Parser".
* doc/bison.texi: Likewise.
It can now generate the usage message.
* src/complain.h (feature_fixit_parsable): Rename as...
(feature_fixit): this, for column economy.
Adjust dependencies.
(warning_usage): New.
Use it.
* src/complain.h, src/complain.c, src/getargs.h, src/getargs.c:
Use ARGMATCH_DEFINE_GROUP instead of the older interface.
The code is inconsistent: sometimes we pass by value, sometimes by
reference. Let's stick to the last, more conventional for large
values in C.
* src/scan-code.l: Pass locations by reference.
* configure.ac (DCFLAGS): Pass -g.
* data/skeletons/d.m4 (b4_locations_if): Remove, let bison.m4's one do
its job.
* data/skeletons/lalr1.d (position): Leave filename empty by default.
(position::toString): Don't print empty file names.
(location::this): New ctor.
(location::toString): Match the implementations of C/C++.
(yy_semantic_null): Leave undefined, the previous implementation does
not compile.
* tests/calc.at: Improve the implementation for D.
Enable more checks, in particular using locations.
* tests/local.at (AT_YYERROR_DEFINE(d)): Fix its implementation.
Sadly enough, AFAIK, there were never answers to the "More user
feedback will help to stabilize it" sentences. Remove them.
* src/getargs.c: IELR, canonical LR and XML output are here to stay,
and they are no more experimental than some other features.
* doc/bison.texi: Likewise.
Also remove "experimental" warning for Java, LAC, LR tuning options,
and named references.
* data/skeletons/lalr1.java: Use more conventional function names for
Java.
Prefer < and <= to => and >.
Use the same approach for m4 quotation as in the other skeletons.
Fix indentation issues.
* tests/calc.at, tests/java.at, tests/javapush.at: Fix quotation style.
(main): Use 'args', not 'argv', the former seems more conventional and
is used elsewhere in Bison.
Prefer character literals to integers to denote characters.
* examples/java/Calc.y: Likewise.
Travis answered favorably to my suggestion to provide a means to
disable git clone on some jobs (issue 7542). See
https://docs.travis-ci.com/user/customizing-the-build/#disabling-git-clone.
* .travis.yml: Disable git globally, enable it for i. the compile job,
and ii. the test job on ICC which needs the install-icc.sh script.
This is an experiment. Maybe more styles will be used (in which case
a short-hand function will be useful), maybe it will be just reverted.
* data/bison-default.css (.traces0): New.
* src/lalr.c (lalr): Use it.
See e3fdc37049: in C++ we generate
explicitly the code for the default action instead of simply copying
blindly the semantic value buffer. This is important when copying
raw memory is not enough, as exemplified by move-only types.
This is currently tested by examples/c++/variant.yy and variant-11.yy.
But it is safer to also have a test in the main test suite.
* tests/local.at (AT_REQUIRE_CXX_STD): Fix.
(AT_BISON_OPTION_PUSHDEFS, AT_BISON_OPTION_POPDEFS): Define/undefine
AT_BISON_OPTIONS.
* tests/c++.at (Default action): New.
It will be convenient to check sets elsewhere.
* tests/sets.at (AT_EXTRACT_SETS): Transform into...
* tests/local.at (AT_SETS_CHECK): this.
* tests/sets.at: Adjust.
The final gain is small: 2h2min instead 2h9min. But that is still an
improvement.
* .travis.yml (git.depth): Make the clone very shallow.
(git.submodules): Don't clone gnulib in test jobs.
(jobs.include.compile.script): Do it here.
* data/diagnostics.css: Rename as...
* data/bison-default.css: this.
Add the GPL header.
This is the convention followed by Bruno Haible in gettext.
Adjust dependencies.
* src/complain.c (complain_init_color): Use BISON_STYLE instead of
BISON_DIAGNOSTICS_STYLE.
Build the tarball in one job, check it in many.
Unfortunately no real gain in overall duration.
With help from Clément Démoulins.
* .travis.yml: here.
Remove all the tricks that were used to be able to boostrap on old
distros.
(before_install): Merge into 'script', because before_install applies
to all the jobs, and we don't want to run it for the 'compile' job.
In preparation for Bison 3.4, revert to the version before this
commit:
commit 03752516b21091cf3c4beea7e8b9bcad462d50ed
Author: John Darrington <[email protected]>
Date: Sun May 12 00:42:36 2019 +0200
version-etc: Ease translation.
* lib/version-etc.c (version_etc_arn, emit_bug_reporting_address):
Move URLs and formatting newlines out of translatable string.
because it changes the messages for --version, translated in
gnulib.po. These changes are not yet available on the translation
project, so we would have a regression in the set of translated
strings.
It works only in implicit rules (or with GNU Make, but not with
Solaris Make).
Reported by Bruno Haible.
http://lists.gnu.org/archive/html/bug-bison/2019-05/msg00009.html
Diagnosed thanks to Kiyoshi Kanazawa.
* examples/c/reccalc/local.mk, examples/d/local.mk,
* examples/java/local.mk: Don't use $< in non implicit rules.
# $SSH_PRIVATE_KEY is multiline. Use $'...' to register its value: $'-----BEGIN OPENSSH PRIVATE KEY-----\nXXXXX...\n...==\n-----END OPENSSH PRIVATE KEY-----'.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Akim Demaille
Paul EggertandAkim Demaille
Kaz KylhekuandAkim Demaille
Vincent ImbimboandAkim Demaille
Martin RehakandAkim Demaille
Todd C. MillerandAkim Demaille
Nick GassonandAkim Demaille
Maarten De BraekeleerandAkim Demaille
Akim DemailleandAkim Demaille
Joshua WattandAkim Demaille
Thomas PetazzoniandAkim Demaille
Bruno HaibleandAkim Demaille
Adrian VogelsgesangandAkim Demaille
Victor Morales CayuelaandAkim Demaille
Jim MeyeringandAkim Demaille
kaneko yandAkim Demaille
László VáradyandAkim Demaille