When expanding the GLR stack, none of the pointers were updated to
reflect the new location of the displaced objects.
This fixes
748: Incorrect lookahead during nondeterministic GLR: glr2.cc
* data/skeletons/glr2.cc (yyexpandGLRStack): Update the split point
and the stack tops.
(reduceToOneStack): Factor a bit.
This test fails:
748: Incorrect lookahead during nondeterministic GLR: glr2.cc
It consumes lots of stack space, so at some point we need to expand
it. Because of Boolean logic mistakes, we then claim
memory-exhausted (first error). Hence we jump to cleaning the
stack (popall_), calling all the destructors, and at some point we
crash with heap-use-after-free (second error).
This commit fixes the first error. Unfortunately, even though we now
do expand the stack, we crash again with (another)
heap-use-after-free, not addressed here.
Eventually, we should make sure popall_() properly works.
* data/skeletons/glr2.cc (yyexpandGLRStackIfNeeded): Return true iff
success (i.e., memory not exhausted).
From a debugger, it is easier to pass a file name than working on
stdin.
* examples/c++/glr/c++-types.yy: Reduce scopes.
Avoid YYSTYPE/YYLTYPE: use the C++ types.
(input, process): New.
(main): Use them.
And also, remove the incorrect indentation of these comments:
- /* YYR2[YYN] -- Number of symbols on the right hand side of rule YYN. */
+/* YYR2[RULE-NUM] -- Number of symbols on the right-hand side of rule RULE-NUM. */
static const yytype_int8 yyr2[] =
{
0, 2, 4, 0, 2, 1, 1, 1, 3, 2,
I don't remember why this indentation was added (in
0991e29b75), but it seems wrong,
at least for yacc.c. I suspect this was done with lalr1.cc (where
this is embeded in the class definition, so it should be indented),
but today lalr1.cc uses other routines to output these comments.
* data/skeletons/bison.m4 (b4_integral_parser_tables_map): Improve the
wording of the comments of some tables.
* data/skeletons/c.m4 (b4_integral_parser_table_define): Remove
indentation.
A glr_stack_item has "raw" memory to store either a glr_state or a
semantic_option. glr_stack_item::setState stores a state using a copy
assignment. However, this is more like a construction: we are
starting from "raw" memory, so use the placement new operator instead.
While it probably makes no difference when parse.assert is disabled,
it does make one when it is: the constructor properly initialize the
magic number, the assignment does not. So without these changes, the
next commit (which stores genuine objects in semantic values) fails
tests 712 and 730 because of incorrect magic numbers.
* data/skeletons/glr2.cc (glr_stack_item::setState): Build the state,
don't just copy it.
ast.hh:24:7: error: 'Node' has no out-of-line virtual method definitions; its vtable will be emitted in every translation unit [-Werror,-Wweak-vtables]
class Node
^
ast.hh:57:7: error: 'Nterm' has no out-of-line virtual method definitions; its vtable will be emitted in every translation unit [-Werror,-Wweak-vtables]
class Nterm : public Node
^
ast.hh:102:7: error: 'Term' has no out-of-line virtual method definitions; its vtable will be emitted in every translation unit [-Werror,-Wweak-vtables]
class Term : public Node
^
* examples/c++/glr/ast.hh: Define the destructors out of the class
definition.
This does not change anything, it is still in the header, but that
does pacify clang.
When debugging these parsers, we really need debug traces.
Enable them, and bind them to $YYDEBUG.
* tests/glr-regression.at: Support the YYDEBUG envvar.
As a consequence, now that syntactic ambiguities are reported, adjust
the expected output.
(No users destructors if stack 0 deleted): Don't return 0 on memory
exhaustion, really return the parser's status, and adust expectations.
Currently the example really looks like C. Instead of a union of
structs to implement the AST, use a hierarchy. It would be nice to
feature a C++17 version with std variants.
* examples/c++/glr/c++-types.yy (Node, free_node, new_nterm)
(new_term): Move into...
* examples/c++/glr/ast.hh: here, a proper C++ hierarchy.
Amusingly enough, glr2.cc still had its core function, yyparse, being
a free function instead of a member function.
* data/skeletons/glr2.cc (yyparse): Remove this free function called
from yyparser::parse. Inline its body into...
(yyparser::parse): this member function.
This requires moving a bit the yychar, etc. macros.
Access to token can be simplified (the
b4_namespace_ref::b4_parser_class prefix is no longer needed).
Remove the useless conditional b4_pure_if: the skeleton is, of course,
pure (no global variables). Make glr2.cc intrinsically pure.
* data/skeletons/glr2.cc (b4_pure_if): Remove definition and uses.
(b4_lex): New.
Stolen from lalr1.cc to avoid needing to use the one from c.m4.
Currently, yycompressStack expects the free items to be states only.
That's not the case.
Fixes 712 and 730 pass. 748 still fails, but later and
differently (heap-use-after-free).
* data/skeletons/glr2.cc (glr_stack_item::setState): New.
(glr_stack_item::yycompressStack): Use it.
* tests/glr-regression.at: Adjust.
A glr_state keeps tracks of its predecessor using an offset relative
to itself (i.e., pointer subtraction). Unfortunately we sometimes
have to compute offsets for pointers that live in different
containers, in particular in yyfillin. In that case there is no
reason for the distance between the two objects to be a multiple of
the object size (0x40 on my machine), and the resulting ptrdiff_t may
be "wrong", i.e., it does allow to recover one from the other. We
cannot use "typed" pointer arithmetics here, the Euclidean division
has it wrong. So use "plain" char* pointers.
Fixes 718 (Duplicate representation of merged trees: glr2.cc) and
examples/c++/glr/c++-types.
Still XFAIL:
712: Improper handling of embedded actions and dollar(-N) in GLR parsers: glr2.cc
730: Incorrectly initialized location for empty right-hand side in GLR: glr2.cc
748: Incorrect lookahead during nondeterministic GLR: glr2.cc
* data/skeletons/glr2.cc (glr_state::as_pointer_): New.
(glr_state::pred): Use it.
* examples/c++/glr/c++-types.test: The test passes.
* tests/glr-regression.at (Duplicate representation of merged trees:
glr2.cc): Passes.
The use of YY_IGNORE_NULL_DEREFERENCE_BEGIN/END in `check_` is to
please GCC 10:
glr-regr8.cc: In member function 'YYRESULTTAG glr_stack::yyresolveValue(glr_state&)':
glr-regr8.cc:1433:21: error: potential null pointer dereference [-Werror=null-dereference]
1433 | YYASSERT (this->magic_ == MAGIC);
| ~~~~~~^~~~~~
glr-regr8.cc:905:40: note: in definition of macro 'YYASSERT'
905 | # define YYASSERT(Condition) ((void) ((Condition) || (abort (), 0)))
| ^~~~~~~~~
* data/skeletons/glr2.cc (glr_state::check_): New.
Use it in the member functions.
We obviously have broken pointer arithmetics that hands us
glr_stack_items that are not glr_stack_items. Have a simple check for
this, to have earlier failures.
* data/skeletons/glr2.cc (glr_stack_item::check_): New.
Use it.
(glr_stack_item::contents): Avoid the useless struct.
Fix minor stylistic issues.
When installed on master as of 2020-12-05 (on top of "glr2.cc: fix
when the stack is not expandable", almost all the GLR regression tests
fail (with a SEGV):
709: Badly Collapsed GLR States: glr2.cc FAILED (glr-regression.at:130)
712: Improper handling of embedded actions and dollar(-N) in GLR parsers: glr2.cc FAILED (glr-regression.at:275)
715: Improper merging of GLR delayed action sets: glr2.cc FAILED (glr-regression.at:404)
718: Duplicate representation of merged trees: glr2.cc FAILED (glr-regression.at:502)
721: User destructor for unresolved GLR semantic value: glr2.cc FAILED (glr-regression.at:566)
724: User destructor after an error during a split parse: glr2.cc FAILED (glr-regression.at:624)
727: Duplicated user destructor for lookahead: glr2.cc FAILED (glr-regression.at:724)
730: Incorrectly initialized location for empty right-hand side in GLR: glr2.cc FAILED (glr-regression.at:823)
733: No users destructors if stack 0 deleted: glr2.cc FAILED (glr-regression.at:911)
736: Corrupted semantic options if user action cuts parse: glr2.cc FAILED (glr-regression.at:974)
739: Undesirable destructors if user action cuts parse: glr2.cc FAILED (glr-regression.at:1042)
742: Leaked semantic values if user action cuts parse: glr2.cc FAILED (glr-regression.at:1173)
748: Incorrect lookahead during nondeterministic GLR: glr2.cc FAILED (glr-regression.at:1546)
751: Leaked semantic values when reporting ambiguity: glr2.cc FAILED (glr-regression.at:1639)
754: Leaked lookahead after nondeterministic parse syntax error: glr2.cc FAILED (glr-regression.at:1710)
757: Uninitialized location when reporting ambiguity: glr2.cc FAILED (glr-regression.at:1794)
766: Predicates: glr2.cc FAILED (glr-regression.at:2045)
These pass:
745: Incorrect lookahead during deterministic GLR: glr2.cc ok
760: Missed %merge type warnings when LHS type is declared later: glr2.cc ok
763: Ambiguity reports: glr2.cc ok
With Valentin Tolmer's "glr2.cc: Fix memory corruption bug" commit,
these test fail "gracefully":
712: Improper handling of embedded actions and dollar(-N) in GLR parsers: glr2.cc FAILED (glr-regression.at:268)
730: Incorrectly initialized location for empty right-hand side in GLR: glr2.cc FAILED (glr-regression.at:816)
748: Incorrect lookahead during nondeterministic GLR: glr2.cc FAILED (glr-regression.at:1539)
And these do not end:
709: Badly Collapsed GLR States: glr2.cc FAILED (glr-regression.at:123)
715: Improper merging of GLR delayed action sets: glr2.cc FAILED (glr-regression.at:397)
718: Duplicate representation of merged trees: glr2.cc FAILED (glr-regression.at:495)
751: Leaked semantic values when reporting ambiguity: glr2.cc FAILED (glr-regression.at:1632)
With "tests: glr2.cc: run the glr-regression tests", none loop, and
709, 715, and 751 pass. Only 718 still fails.
* tests/glr-regression.at: Run all the tests with glr2.cc.
* tests/local.at (AT_GLR2_CC_IF): New.
When "tests: glr2.cc: run the glr-regression tests" tests are run,
before this commit the following tests used to loop endlessly:
709: Badly Collapsed GLR States: glr2.cc FAILED (glr-regression.at:123)
715: Improper merging of GLR delayed action sets: glr2.cc FAILED (glr-regression.at:397)
718: Duplicate representation of merged trees: glr2.cc FAILED (glr-regression.at:495)
751: Leaked semantic values when reporting ambiguity: glr2.cc FAILED (glr-regression.at:1632)
After this commit, no test loops and 709, 715, and 751 pass. Only 718
still fails.
* data/skeletons/glr2.cc (yyresolveValue): Add missing incrementation
of the iteration variable.
* tests/glr-regression.at: Adjust the tests to be more independent of
the language, and run them with glr.cc.
Some tests relied on yychar, yylval and yylloc being global variables:
pass arguments instead.
* tests/glr-regression.at: Instead of using AT_BISON_CHECK and
AT_COMPILE, use AT_FULL_COMPILE. This is shorter, and makes it easier
to add support for other programming languages.
* tests/glr-regression.at: Use %expect and %expect-rr in the grammar
files, rather than accepting diagnostics.
This will make it easier to support other programming languages.
When using glr.cc, the C function yyparse is an internal detail that
should not be exposed. Users might call it by accident (I did).
* data/skeletons/glr.c (yyparse): When used for glr.cc, rename as yy_parse_impl.
* data/skeletons/glr.cc: Adjust.
src/reader.c: In function 'grammar_start_symbols_add':
src/reader.c:67:24: error: declaration of 'dup' shadows a global declaration [-Werror=shadow]
* src/reader.c (grammar_start_symbols_add): Rename dup as dupl.
The yydefgoto table uses -1 as an invalid for an impossible case (we
never use yydefgoto[0], since it corresponds to the reduction to
$accept, which never happens). Since yydefgoto is a table of state
numbers, this -1 forces a signed type uselessly, which (1) might
trigger compiler warnings when storing a value from yydefgoto into a
state number (nonnegative), and (2) wastes bits which might result in
using a int16 where a uint8 suffices.
Reported by Jot Dot <[email protected]>.
https://lists.gnu.org/r/bug-bison/2020-11/msg00027.html
* src/tables.c (default_goto): Use 0 rather than -1 as invalid value.
* tests/regression.at: Adjust.
We were declaring foo_parse_expr but implementing yyparse_expr.
* data/skeletons/yacc.c (_b4_define_sub_yyparse): Use api.prefix for
the parse function name.
* maint:
c++: shorten the assertions that check whether tokens are correct
c++: don't glue functions together
lalr1.cc: YY_ASSERT should use api.prefix
c++: don't use YY_ASSERT at all if parse.assert is disabled
c++: style: follow the Bison m4 quoting pattern
yacc.c: provide the Bison version as an integral macro
regen
style: make conversion of version string to int public
%require: accept version numbers with three parts ("3.7.4")
yacc.c: fix #definition of YYEMPTY
gnulib: update
doc: fix incorrect section title
doc: minor grammar fixes in counterexamples section
Before:
YY_ASSERT (tok == token::YYEOF || tok == token::YYerror || tok == token::YYUNDEF || tok == 120 || tok == 49 || tok == 50 || tok == 51 || tok == 52 || tok == 53 || tok == 54 || tok == 55 || tok == 56 || tok == 57 || tok == 97 || tok == 98);
After:
YY_ASSERT (tok == token::YYEOF
|| (token::YYerror <= tok && tok <= token::YYUNDEF)
|| tok == 120
|| (49 <= tok && tok <= 57)
|| (97 <= tok && tok <= 98));
Clauses are now also wrapped on several lines. This is nicer to read
and diff, but also avoids pushing Visual C++ to its arbitrary
limits (640K and lines of 16380 bytes ought to be enough for anybody,
otherwise make an C2026 error).
The useless parens are there for the dummy warnings about
precedence (in the future, will we also have to put parens in
`1+2*3`?).
* data/skeletons/variant.hh (_b4_filter_tokens, b4_tok_in, b4_tok_in):
New.
(_b4_token_constructor_define): Use them.
Working on the previous commit I realized that YY_ASSERT was used in
the generated headers, so must follow api.prefix to avoid clashes when
multiple C++ parser with variants are used.
Actually many more macros should obey api.prefix (YY_CPLUSPLUS,
YY_COPY, etc.). There was no complaint so far, so it's not urgent
enough for 3.7.4, but it should be addressed in 3.8.
* data/skeletons/variant.hh (b4_assert): New.
Use it.
* tests/local.at (AT_YYLEX_RETURN): Fix.
* tests/headers.at: Make sure variant-based C++ parsers are checked
too.
This test did find that YY_ASSERT escaped renaming (before the fix in
this commit).
In some extreme situations (about 800 tokens), we generate a
single-line assertion long enough for Visual C++ to discard the end of
the line, thus falling into parse ends for the missing `);`. On a
shorter example:
YY_ASSERT (tok == token::TOK_YYEOF || tok == token::TOK_YYerror || tok == token::TOK_YYUNDEF || tok == token::TOK_ASSIGN || tok == token::TOK_MINUS || tok == token::TOK_PLUS || tok == token::TOK_STAR || tok == token::TOK_SLASH || tok == token::TOK_LPAREN || tok == token::TOK_RPAREN);
Whether NDEBUG is used or not is irrelevant, the parser dies anyway.
Reported by Jot Dot <[email protected]>.
https://lists.gnu.org/r/bug-bison/2020-11/msg00002.html
We should avoid emitting lines so long.
We probably should also use a range-based assertion (with extraneous
parens to pacify fascist compilers):
YY_ASSERT ((token::TOK_YYEOF <= tok && tok <= token::TOK_YYUNDEF)
|| (token::TOK_ASSIGN <= tok && ...)
But anyway, we should simply not emit this assertion at all when not
asked for.
* data/skeletons/variant.hh: Do not define, nor use, YY_ASSERT when it
is not enabled.
IELR needs to rule out the successors of the kernel items of the
initial state (`$accept: input • $end`). In the case of multistart,
this condition must be expressed differently: the mere item index does
not suffice.
* src/ielr.c (ielr_item_has_lookahead, ielr_compute_lookaheads): Don't
rely on the item index to check whether is_successor_of_initial_item.
It is certainly more costly than just checking the item index, but (i)
we need to compute the rule anyway, so it's not very much more costly,
and (ii) in ielr_item_has_lookahead, this situation is actually
impossible, so an optimizing compiler reading the assertions should
actually avoid this computation.
Checking that an item index is > 1 means ruling out `$accept: • input
$end` and `$accept: input • $end`. But actually only the latter is
possible there, i.e., we're checking whether this item is about a
successor of a (kernel) item of the initial state ($accept: input •
$end).
* src/ielr.c (is_successor_of_initial_item): Use a variable to name
this condition.
* src/gram.h (rule_is_initial): New.
* src/graphviz.c, src/print-xml.c, src/print.c, src/lalr.c: Use it.
Some of these occurrences were incorrect (checking whether this is
rule 0), and not behaving properly in the case of multistart.
* src/lalr.c: Remove incorrect comment (subsumed anyway by the
(correct) one in the header.
(set_goto_map): More debug traces.
(map_goto): Add an assertion.
This should have been part of commit "symbols: stop dealing with YYEMPTY
as b4_symbol(-2, ...)" (cd40ec9526).
Give names to all the special symbols: "eof", "error" and "undef".
* data/skeletons/bison.m4 (b4_symbol): Let `b4_symbol(eof, ...)` mean
`b4_symbol(0, ...)`, `b4_symbol(error, ...)` mean `b4_symbol(1, ...)`,
and , `b4_symbol(undef, ...)` mean `b4_symbol(2, ...)`..
* data/skeletons/c.m4, data/skeletons/glr.c, data/skeletons/glr.cc,
* data/skeletons/glr2.cc, data/skeletons/lalr1.cc,
* data/skeletons/lalr1.d, data/skeletons/lalr1.java,
* data/skeletons/yacc.c:
Prefer symbols to numbers.
examples/java/calc/Calc.java:1531: warning: [deprecation] Integer(String) in Integer has been deprecated
yylval = new Integer(st.sval);
^
* examples/java/calc/Calc.y, examples/java/simple/Calc.y,
* tests/calc.at, tests/scanner.at: Use Integer.parseInt.
Vector is synchronized, which is completely useless in our case (and
not even relevant when concurrency matters). No seasoned Java
programmer would use it.
Reported by Uxio Prego.
* data/skeletons/lalr1.java: Replace Vector with ArrayList.
Unfortunately its API is not as rich, and lacks lastElement and
setSize.
* tests/calc.at: Add tests for LAC in pull and push parsers.
Skip LAC: line from the logs.
* tests/local.at (reportSyntaxError): Output the error message in a
single call, to avoid having the error message on stderr be
interrupted by the debug traces of LAC in getExpectedTokens.
Shamelessly stolen from Adrian Vogelsgesang's implementation in
lalr1.cc.
* data/skeletons/lalr1.java (yycdebugNnl, yyLacCheck, yyLacEstablish)
(yyLacDiscard, yylacStack, yylacEstablished): New.
(Context): Use it.
* examples/java/calc/Calc.y: Enable LAC.
* upstream/maint:
doc: fix typo
maint: post-release administrivia
version 3.7.3
build: don't link bison against libreadline
gnulib: update
glr.cc: fix: use symbol_name
build: fix a concurrent build issue in examples
When printing items, it is clearer to put the dot after %emtpy rather
than before:
0 $accept: . unit "end of file"
1 unit: . assignments exp
- 2 assignments: . %empty
+ 2 assignments: %empty .
3 | . assignments assignment
Also, use the Unicode characters if they are supported.
* src/gram.c (item_print): Put the dot after %emtpy.
* tests/conflicts.at, tests/reduce.at, tests/report.at: Adjust.
Currently `b4_percent_define_ifdef([foo])` assigns a default value to
`foo` when invoked. As a consequence, skeletons such as lalr1.d
cannot specify their specific default values: `foo` was defined in
bison.m4.
Instead, provide `foo` with a default value when `b4_foo_if` is
invoked.
I could not measure a runtime difference between both cases.
* data/skeletons/bison.m4 (_b4_percent_define_define): New.
Helps getting rid of spurious indentation that resulted in spurious
white space in the output.
(b4_percent_define_if_define): Move the definition to...
(_b4_percent_define_if_define): when the defined macros is called.
* tests/local.at (AT_TOKEN_TRANSLATE_IF): New, moved from...
* tests/calc.at: here.
Instead of sorting per feature (main, yylex, calc.y) and then by
language, do the converse, so that C bits are together, etc.
This is something that has always bothered me: with pure parsers (and
they all should be) the user does not have an (easy) access to yynerrs
at the end of the parse. In the case of error recovery, that's the
only direct means to know if there were errors. The usual approach
being having the user maintain a counter incremented each time yyerror
is called.
So here, also capture yynerrs in the return value of the start-symbol
parsing functions.
* data/skeletons/yacc.c (yy_parse_impl_t): New.
(yy_parse_impl): Use it.
(b4_accept): Fill it.
* examples/c/lexcalc/parse.y, examples/c/lexcalc/scan.l: No longer
pass nerrs as lex- and parse-param, just use the resulting yynerrs.
bistromathic and reccalc both demonstrate %param.
Akim Demaille