In c2ba260487 ("glr.c: fix signature
when using custom error messages"), I meant to add a test case for C,
not C++. It does not work in C++.
* tests/calc.at: Run for glr.c, not glr.cc.
GCC 4.2 on macOS claims to support C++98, but does not feature it.
input.cc: In member function 'void state_stack::yycompressStack()':
input.cc:1774: error: 'class std::vector<glr_stack_item, std::allocator<glr_stack_item> >' has no member named 'data'
Reported by Christopher Nielsen <[email protected]>.
<https://trac.macports.org/raw-attachment/ticket/59927/bison-test-results-20210811-95b72.log.xz>.
* m4/bison-cxx-std.m4 (_BISON_CXXSTD_98_snippet): Check for it.
* m4/cxx.m4 (BISON_TEST_FOR_WORKING_CXX_COMPILER): Ditto.
* tests/atlocal.in: It was an error for tests to depend on gnulib:
they must not, as gnulib would hide portability issues that we want to
catch. So this piece of code is no longer useful, and must not be
useful.
In POSIX Yacc mode, declare yyerror and yylex unless already #defined,
or if YYERROR_IS_DECLARED/YYLEX_IS_DECLARED are defined (for
consistency with Bison's YYSTYPE_IS_DECLARED/YYLTYPE_IS_DECLARED).
See <https://austingroupbugs.net/view.php?id=1388#c5220>.
* data/skeletons/c.m4 (b4_function_declare): Resurect.
(b4_lex_formals): Since we will possibly expose this prototype
in the header, take the prefix into account.
* data/skeletons/yacc.c (b4_declare_yyerror_and_yylex): New.
(b4_shared_declarations): Use it.
* tests/local.at (AT_YACC_IF): New.
When in Yacc mode, set the `yacc` Autotest keyword.
(AT_YYERROR_DECLARE(c)): Don't declare in Yacc mode,
to avoid clashes (since this signature is static).
(AT_YYERROR_DEFINE(c)): Don't define as static in Yacc mode.
* tests/regression.at (Early token definitions with --yacc): Specify
that we are in Yacc mode.
Unfortunately it seems to be quite difficult to have "LAC: Exploratory
stack" run for D.
* data/skeletons/lalr1.d: We need File when traces are enabled.
* tests/local.at (AT_YYLEX_DEFINE(d)): New.
* tests/regression.at: Prepare for D, but don't run it, it does not
work.
* tests/local.at (AT_FOR_EACH_SKEL): New.
Use where appropriate.
* data/skeletons/lalr1.d: Reject -d.
* tests/input.at, tests/scanner.at: Also check D.
Check in m4's output if there are sequences such as m4_foo or b4_foo,
which are probably resulting from incorrect m4 processing.
It actually already is useful:
- it caught a leaking b4_lac_if leaking from glr.c, where LAC is not
supported, hence b4_lac_if is not defined.
- it also caught references to location.hh in position.hh when
location.hh does not exist.
- while making "Code injection" robust to these new warnings (it is
its very purpose to let b4_canary pass unevaluated), I saw that it
did not check lalr1.d, and when adding lalr1.d, it revealed it did
underquote ocurrences of token value types.
* src/scan-skel.l (macro): New abbreviation.
Use it.
* data/skeletons/glr.c: Don't use b4_lac_if, we don't have it.
* data/skeletons/location.cc: Don't generate position.hh when we don't
generate location.hh.
* data/skeletons/d.m4 (b4_basic_symbol_constructor_define): Fix
underquotation.
* data/skeletons/bison.m4 (b4_canary): New.
* tests/input.at (Code injection): Use it, and check lalr1.d too.
* examples/c++/Makefile, examples/c++/calc++/Makefile,
* examples/c++/glr/Makefile, examples/c/bistromathic/Makefile,
* examples/c/calc/Makefile, examples/c/glr/Makefile,
* examples/c/lexcalc/Makefile, examples/c/mfcalc/Makefile,
* examples/c/pushcalc/Makefile, examples/c/reccalc/Makefile,
* examples/c/rpcalc/Makefile, examples/d/calc/Makefile,
* examples/d/simple/Makefile, examples/java/calc/Makefile,
* examples/java/simple/Makefile:
Use --html to generate *.html directly.
No longer demonstrate --xml.
No longer show rules for xml to html.
Use --header, not --defines.
Use --graph without specifying the output file now that we
generate *.gv by default.
We had:
```
-mbchar ...|\xF0[\x\90-\xBF]([\x80-\xBF]{2})|...
+mbchar ...|\xF0[\x90-\xBF]([\x80-\xBF]{2})|...
```
so a precise sequence that matches the incorrect regex can let NUL
bytes pass through, which triggers an assertion violation downstream.
It is a pity that Flex does not report an error for such input.
Reported by Ahcheong Lee <[email protected]>.
<https://lists.gnu.org/r/bug-bison/2021-04/msg00003.html>
* src/scan-gram.l (mbchar): Fix the bad regex.
* tests/input.at (Invalid inputs): Check that case.
* doc/bison.texi: Use @samp{...}, not "..." for pieces of code.
Use @samp{...}, not @command{...} for command lines.
Promote %header/--header over %defines/--defines.
Spellcheck.
The union of the values is handled by the backend.
In D, unions can hold classes, structs, etc., so this is more similar
to the C++ api.value.type variant.
* data/skeletons/d.m4, data/skeletons/lalr1.d: Here.
* tests/calc.at, tests/local.at: Test it.
When a pstate is used for multiple successive parses, some state may
leak from one run into the following one. That was introduced in
330552ea49 "yacc.c: push: don't clear
the parser state when accepting/rejecting".
Reported by Ryan <[email protected]>
https://lists.gnu.org/r/bug-bison/2021-03/msg00000.html
* data/skeletons/yacc.c (yypush_parse): We reusing a pstate from a
previous run, do behave as if it were the first run.
* tests/push.at (Pstate reuse): Check this.
The right-shift added in c22902e360
("tables: fix handling for useless tokens") is incorrect. In
particular, we need to reset the "new" bits.
Reported by Balázs Scheidler.
https://github.com/akimd/bison/issues/74
* src/tables.c (pos_set_set): Fix the right-shift.
Don't repeatedly call malloc/free for each call to map_file_name.
* bootstrap.conf: We need hash-map.
* src/files.h, src/files.c (map_file_name): The caller must not free
the result.
Adjust callers.
(mapped_dir_prefix, spec_mapped_header_file): Remove.
* src/files.c
(map_file_name): Rename as...
(map_file_name_alloc): this.
(mapped_files, map_file_name, string_equals, string_hash, string_free):
New.
* examples/c/bistromathic/parse.y, examples/c/glr/c++-types.y,
* examples/c/lexcalc/parse.y: Use const where appropriate.
Avoid `yy` prefixes where it does not make sense.
Avoid the `p` prefix for pointers.
* examples/c/bistromathic/parse.y, examples/c/glr/c++-types.y,
* examples/c/lexcalc/parse.y: Don't use the private internal detail
`YY_LOCATION_PRINT`, use `YYLOCATION_PRINT`.
It is very helpful to be able to look at the locations in the debug
traces, let's provide the users with (i) a means to do that for their
location types, and (ii) a public macro to print locations when debug
traces are enabled.
* data/skeletons/c.m4 (b4_yy_location_print_define): Rename as...
(b4_yylocation_print_define): this.
Define YYLOCATION_PRINT instead of YY_LOCATION_PRINT.
Ensure backward compatibility for those who might have defined/used
YY_LOCATION_PRINT in spite the warnings.
Adjust dependencies.
* data/skeletons/glr2.cc: We don't use YYLOCATION_PRINT here.
That change was started in Bison 3.4. The announcement for 3.7 stated
that in Bison 3.8 we would use *.gv by default.
* src/files.c (compute_output_file_names): spec_graph_file defaults
too *.gv.
* doc/bison.texi, examples/c++/calc++/local.mk, tests/output.at:
Adjust.
Its removal was annonced several times in NEWS (for Bison 3.5, 3.6, 3.7).
* data/skeletons/c.m4, data/skeletons/yacc.c: Remove support for YYPRINT.
* NEWS: Fix the mess introduced by the merge.
Document the removal of YYPRINT.
* doc/bison.texi (The YYPRINT Macro): Remove.
Using #define YYSTYPE has always been strongly discouraged in C++.
Macros are dangerous and can result in subtle bugs.
https://lists.gnu.org/r/bug-bison/2020-12/msg00007.html
Maybe some people are currently using #define YYSTYPE. Instead of
dropping support right now, first issue a warning. Bison can "see" if
YYDEBUG is defined (it could even be on the command line), only the
compiler knows. Unfortunately `#warning` is non-portable, and
actually GCC even dies on it when `-pedantic` is enabled. So we need
to use `#pragma message`. We must make it conditional as some
compilers might not support it, but it doesn't matter if only _some_
compilers emit the warning: it should be enough to catch the attention
of the developers.
* data/skeletons/c++.m4: Issue a warning when the user defined
YYSTYPE.
* tests/actions.at: Don't #define YYSTYPE.
* tests/headers.at (Several parsers): Ignore the YYSTYPE in the
warning.
We have been accepting this for years, but it is deprecated: people
are expecting to define api.value.type instead.
* doc/bison.texi: Make it clear that YYSTYPE and YYLTYPE are for C
only.
* data/skeletons/glr2.cc, data/skeletons/lalr1.cc (b4_lex): Don't
refer to YYSTYPE, but to value_type.
Note that this is "symbolic" in the sense that these values are never
used by b4_function_call.
* data/skeletons/bison.m4 (b4_union_if): New.
Remove support for "%define variant", which was deprecated long ago.
* tests/local.at (AT_UNION_IF): New.
* src/main.c (main): When traces are enabled, display the Bison
version.
* tests/conflicts.at, tests/report.at, tests/sets.at:
Use AT_PACKAGE_VERSION (for package.m4) instead of post-processing the
output.
Automake appears to not recognize "check-local" when it is not a
isolated target. Fixes 77a8b84fc8.
* tests/local.mk (check-local): Make it visible to Automake.
Currently glr2.cc does not use 'symbol's everywhere, in various places
it also uses yykind, yyval and yyloc "by hand". So we need two
different calls for user-defined constructors: once for ~symbol,
another for yy_destroy_. Both need to call the user destructors with
different calling conventions.
* data/skeletons/glr2.cc (b4_symbol_action): Rename as...
(b4_symbol_action_for_yyval): this.
(b4_symbol_action): New, taken from lalr1.cc.
(yy_destroy_, yy_symbol_value_print_): Use b4_symbol_action_for_yyval.
Three new commits:
commit 8358090292
Author: Paul Eggert <[email protected]>
Date: Wed Jan 20 18:30:16 2021 -0800
c: port to HP-UX 11.23
commit 2c294c1325
Author: Vincent Imbimbo <[email protected]>
Date: Sat Jan 23 13:25:18 2021 -0500
cex: fix state-item pruning
commit c22902e360
Author: Akim Demaille <[email protected]>
Date: Sat Jan 23 18:40:15 2021 +0100
tables: fix handling for useless tokens
In some rare conditions, the generated parser can be wrong when there
are useless tokens.
Reported by Balázs Scheidler.
https://github.com/akimd/bison/issues/72
Balázs managed to prove that the bug was introduced in
commit af1c6f973a
Author: Theophile Ranquet <[email protected]>
Date: Tue Nov 13 10:38:49 2012 +0000
tables: use bitsets for a performance boost
Suggested by Yuri at
<http://lists.gnu.org/archive/html/bison-patches/2012-01/msg00000.html>.
The improvement is marginal for most grammars, but notable for large
grammars (e.g., PosgreSQL's postgre.y), and very large for the
sample.y grammar submitted by Yuri in
http://lists.gnu.org/archive/html/bison-patches/2012-01/msg00012.html.
Measured with --trace=time -fsyntax-only.
parser action tables postgre.y sample.y
Before 0,129 (44%) 37,095 (99%)
After 0,117 (42%) 5,046 (93%)
* src/tables.c (pos): Replace this set of integer coded as an unsorted
array of integers with...
(pos_set): this bitset.
which was implemented long ago, but that I installed only recently
(March 2019), first published in v3.3.90.
That patch introduces a bitset to represent a set of integers. It
managed negative integers by using a (fixed) base (the smallest
integer to represent). It avoided negative accesses into the bitset
by ignoring integers smaller than the base, under the asumption that
these cases correspond to useless tokens that are ignored anyway.
While it turns out to be true for all the test cases in the test suite
(!), Balázs' use case demonstrates that it is not always the case.
So we need to be able to accept negative integers that are smaller
than the current base.
"Amusingly" enough, the aforementioned patch was visibly unsure about
itself:
/* Store PLACE into POS_SET. PLACE might not belong to the set
of possible values for instance with useless tokens. It
would be more satisfying to eliminate the need for this
'if'. */
This commit needs several improvements in the future:
- support from bitset for bit assignment and shifts
- amortized resizing of pos_set
- test cases
* src/tables.c (pos_set_base, pos_set_dump, pos_set_set, pos_set_test):
New.
Use them instead of using bitset_set and bitset_test directly.
Currently each time we meet %merge we record this location as the
defining location (and symbol). Instead, record the first definition.
In the generated code we go from
yy0->A = merge (*yy0, *yy1);
to
yy0->S = merge (*yy0, *yy1);
where S was indeed the first symbol, and in the diagnostics we go from
glr-regr18.y:30.18-24: error: result type clash on merge function 'merge': <type2> != <type1>
30 | sym2: sym3 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
glr-regr18.y:29.18-24: note: previous declaration
29 | sym1: sym2 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
glr-regr18.y:31.13-19: error: result type clash on merge function 'merge': <type3> != <type2>
31 | sym3: %merge<merge> { $$ = 0; } ;
| ^~~~~~~
glr-regr18.y:30.18-24: note: previous declaration
30 | sym2: sym3 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
to
glr-regr18.y:30.18-24: error: result type clash on merge function 'merge': <type2> != <type1>
30 | sym2: sym3 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
glr-regr18.y:29.18-24: note: previous declaration
29 | sym1: sym2 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
glr-regr18.y:31.13-19: error: result type clash on merge function 'merge': <type3> != <type1>
31 | sym3: %merge<merge> { $$ = 0; } ;
| ^~~~~~~
glr-regr18.y:29.18-24: note: previous declaration
29 | sym1: sym2 %merge<merge> { $$ = $1; } ;
| ^~~~~~~
where both duplicates are reported against definition 1, rather than
using definition 1 as a reference when diagnosing about definition 2,
and then 2 as a reference for 3.
* src/reader.c (record_merge_function_type): Keep the first definition.
* tests/glr-regression.at: Adjust.
Reported by Jot Dot.
https://lists.gnu.org/r/help-bison/2020-12/msg00014.html
* data/skeletons/glr.c, data/skeletons/glr2.cc (b4_call_merger): Use
the symbol's slot, not its type.
* examples/c/glr/c++-types.y: Use explicit per-symbol typing together
with api.value.type=union.
(yylex): Use yytoken_kind_t.
Don't generate C code from bison, leave that to the skeletons.
* src/output.c (merger_output): Emit invocations to b4_call_merger.
* data/skeletons/glr.c (b4_call_merger): New.
Symbols are richer than types, and in M4 it is my simpler (and more
common) to deal with symbols rather than types. So let's associate
mergers to a symbol rather than a type name.
* src/reader.h (merger_list): Replace the 'type' member by a symbol
member.
* src/reader.c (record_merge_function_type): Take a symbol as
argument, rather than a type name.
* src/output.c (merger_output): Adjust.
This macro is not exposed to users, make start it with 'YY_'.
* data/skeletons/bison.m4, data/skeletons/c.m4, data/skeletons/glr.c,
* data/skeletons/glr.cc, data/skeletons/lalr1.cc,
* src/parse-gram.c, tests/actions.at, tests/c++.at, tests/headers.at,
* tests/local.at (YYUSE): Rename as...
(YY_USE): this.
Akim Demaille