This is something that has always bothered me: with pure parsers (and
they all should be) the user does not have an (easy) access to yynerrs
at the end of the parse. In the case of error recovery, that's the
only direct means to know if there were errors. The usual approach
being having the user maintain a counter incremented each time yyerror
is called.
So here, also capture yynerrs in the return value of the start-symbol
parsing functions.
* data/skeletons/yacc.c (yy_parse_impl_t): New.
(yy_parse_impl): Use it.
(b4_accept): Fill it.
* examples/c/lexcalc/parse.y, examples/c/lexcalc/scan.l: No longer
pass nerrs as lex- and parse-param, just use the resulting yynerrs.
bistromathic and reccalc both demonstrate %param.
After all, why not?
* src/reader.c (switching_token): Use symbol_id_get.
(check_start_symbols): Require that the start symbol is a token only
if it's the only one.
* examples/c/lexcalc/parse.y: Let NUM be a start symbol.
For each start symbol, generate a parsing function with a richer
return value than the usual of yyparse. Reserve a place for the
returned semantic value, in order to avoid having to pass a pointer as
argument to "return" that value. This also makes the call to the
parsing function independent of whether a given start-symbol is typed.
For instance, if the grammar file contains:
%type <int> expression
%start input expression
(so "input" is valueless) we get
typedef struct
{
int yystatus;
} yyparse_input_t;
yyparse_input_t yyparse_input (void);
typedef struct
{
int yyvalue;
int yystatus;
} yyparse_expression_t;
yyparse_expression_t yyparse_expression (void);
This commit also changes the implementation of the parser termination:
when there are multiple start symbols, it is the initial rules that
explicitly YYACCEPT. They do that after having exported the
start-symbol's value (if it is typed):
switch (yyn)
{
case 1: /* $accept: YY_EXPRESSION expression $end */
{ ((*yyvalue).TOK_expression) = (yyvsp[-1].TOK_expression); YYACCEPT; }
break;
case 2: /* $accept: YY_INPUT input $end */
{ YYACCEPT; }
break;
I have tried several ways to deal with termination, and this is the
one that appears the best one to me. It is also the most natural.
* src/scan-code.h, src/scan-code.l (obstack_for_actions): New.
* src/reader.c (grammar_rule_check_and_complete): Generate the actions
of the rules for each start symbol.
* data/skeletons/bison.m4 (b4_symbol_slot): New, with safer semantics
than type and type_tag.
* data/skeletons/yacc.c (b4_accept): New.
Generates the body of the action of the start rules.
(_b4_declare_sub_yyparse): For each start symbol define a dedicated
return type for its parsing function.
Adjust the declaration of its parsing function.
(_b4_define_sub_yyparse): Adjust the definition of the function.
* examples/c/lexcalc/parse.y: Check the case of valueless symbols.
* examples/c/lexcalc/lexcalc.test: Check start symbols.
Currently this example crashes on input such as "T (x) + y;".
The same example with glr.c works properly.
* examples/c++/glr/Makefile, examples/c++/glr/README.md,
* examples/c++/glr/c++-types.test, examples/c++/glr/c++-types.yy,
* examples/c++/glr/local.mk, examples/c++/local.mk: New.
Based on examples/c/glr/c++-types.y.
* data/skeletons/lalr1.d: Change the return value.
* examples/d/calc/calc.y, examples/d/simple/calc.y: Adjust.
* tests/scanner.at: Adjust.
* tests/calc.at (_AT_DATA_CALC_Y(d)): New, extracted from...
(_AT_DATA_CALC_Y(c)): here.
The two grammars have been sufficiently different to be separated.
Still trying to be them together results in a maintenance burden. For
the same reason, instead of specifying the results for D and for the
rest, compute the expected results with D from the regular case.
Based on the test case 668 (cxx-type.at:437) "GLR: Merge conflicting
parses, pure, locations".
* examples/c/glr/Makefile, examples/c/glr/README.md,
* examples/c/glr/c++-types.test, examples/c/glr/c++-types.y,
* examples/c/glr/local.mk: New.
The name "defines" is incorrect, the generated file contains far more
than just #defines.
* src/getargs.h, src/getargs.c (-H, --header): New option.
With optional argument, just like --defines, --xml, etc.
(defines_flag): Rename as...
(header_flag): this.
Adjust dependencies.
* data/skeletons/bison.m4, data/skeletons/c.m4, data/skeletons/glr.c,
* data/skeletons/glr.cc, data/skeletons/glr2.cc, data/skeletons/lalr1.cc,
* data/skeletons/yacc.c:
Adjust.
* examples, doc/bison.texi: Adjust.
* tests/headers.at, tests/local.at, tests/output.at: Convert most
tests from using --defines to using --header.
Reported by Dennis Clarke.
https://lists.gnu.org/r/bug-bison/2020-08/msg00013.html
* examples/d/local.mk, examples/java/calc/local.mk,
* examples/java/simple/local.mk: Pass bison's options before its
argument, in case we're in POSIX mode.
Reported by Agency for Defense Development.
https://lists.gnu.org/r/bug-bison/2020-08/msg00008.html
On an empty such as
%token FOO
BAR
FOO 0
%%
input: %empty
we crash because when we find FOO 0, we decrement ntokens (since FOO
was discovered to be EOF, which is already known to be a token, so we
increment ntokens for it, and need to cancel this). This "works well"
when EOF is properly defined in one go, but here it is first defined
and later only assign token code 0. In the meanwhile BAR was given
the token number that we just decremented.
To fix this, assign symbol numbers after parsing, not during parsing,
so that we also saw all the explicit token codes. To maintain the
current numbers (I'd like to keep no difference in the output, not
just equivalence), we need to make sure the symbols are numbered in
the same order: that of appearance in the source file. So we need the
locations to be correct, which was almost the case, except for nterms
that appeared several times as LHS (i.e., several times as "foo:
..."). Fixing the use of location_of_lhs sufficed (it appears it was
intended for this use, but its implementation was unfinished: it was
always set to "false" only).
* src/symtab.c (symbol_location_as_lhs_set): Update location_of_lhs.
(symbol_code_set): Remove broken hack that decremented ntokens.
(symbol_class_set, dummy_symbol_get): Don't set number, ntokens and
nnterms.
(symbol_check_defined): Do it.
(symbols): Don't count nsyms here.
Actually, don't count nsyms at all: let it be done in...
* src/reader.c (check_and_convert_grammar): here. Define nsyms from
ntokens and nnterms after parsing.
* tests/input.at (EOF redeclared): New.
* examples/c/bistromathic/bistromathic.test: Adjust the traces: in
"%nterm <double> exp %% input: ...", exp used to be numbered before
input.
* examples/c/bistromathic/parse.y (user_context): We need the current
line.
(yyreport_syntax_error): Quote the guilty line, with squiggles.
* examples/c/bistromathic/bistromathic.test: Adjust.
Currently autocompletion on a line with errors leaks the error
messages. It can be useful to let the user know, but GNU Readline
does not provide us with an nice way to display the error. So we
actually break into the current line of the user.
So instead, do not show these errors.
* examples/c/bistromathic/parse.y (user_context): New.
Use %param to pass it to the parser and scanner.
Keep quiet when in computing autocompletion.
That quite defeats the whole point of locations... But anyway, we
should not see these messages at all.
* examples/c/bistromathic/parse.y (expected_tokens): Fix (useless)
location tracking.
* examples/c/bistromathic/parse.y, examples/c/lexcalc/parse.y,
* examples/c/reccalc/parse.y: here.
Add some comments.
* src/parse-gram.y (api_version): Pull out of handle_require.
Bump to 3.7.
Currently when a push parser finishes its parsing (i.e., it did not
return YYPUSH_MORE), it also clears its state. It is therefore
impossible to see if it had parse errors.
In the context of autocompletion, because error recovery might have
fired, the parser is actually already in a different state. For
instance on `(1 + + <TAB>` in the bistromathic, because there's a
`exp: "(" error ")"` recovery rule, `1 + +` tokens have already been
popped, replaced by `error`, and autocompletions think we are ready
for the closing ")". So here, we would like to see if there was a
syntax error, yet `yynerrs` was cleared.
In the case of a successful parse, we still have a problem: if error
recovery succeeded, we won't know it, since, again, `yynerrs` is
clearer.
It seems much more natural to leave the parser state available for
analysis when there is a failure.
To reuse the parser, we should either:
1. provide an explicit means to reinitialize a parser state for future
parses.
2. automatically reset the parser state when it is used in a new
parse.
Option 2 requires to check whether we need to reinitialize the parser
each time we call `yypush_parse`, i.e., each time we give a new token.
This seems expensive compared to Option 1, but benchmarks revealed no
difference. Option 1 is incompatible with the documentation
("After `yypush_parse` returns a status other than `YYPUSH_MORE`, the
parser instance `yyps` may be reused for a new parse.").
So Option 2 wins, reusing the private `yynew` member to record that a
parse was finished, and therefore that the state must reset in the
next call to `yypull_parse`.
While at it, this implementation now reuses the previously enlarged
stacks from one parse to another.
* data/skeletons/yacc.c (yypstate_new): Set up the stacks in their
initial configurations (setting their bottom to the stack array), and
use yypstate_clear to reset them (moving their top to their bottom).
(yypstate_delete): Adjust.
(yypush_parse): At the beginning, clear yypstate if needed, and at the
end, record when yypstate needs to be clearer.
* examples/c/bistromathic/parse.y (expected_tokens): Do not propose
autocompletion when there are parse errors.
* examples/c/bistromathic/bistromathic.test: Check that case.
Currently, completion when there is a syntax error shows broken
locations.
* examples/c/bistromathic/parse.y (expected_tokens): Initialize the
location.
* examples/c/bistromathic/bistromathic.test: Check that.
* upstream/maint:
maint: post-release administrivia
version 3.6.3
build: check -Wmissing-prototypes
tests: show logs
c++: fix printing of state number on streams
pstate_clear is lacking a prototype.
Reported by Ryan
https://lists.gnu.org/r/bug-bison/2020-05/msg00101.html
Besides, none of the C examples were compiled with the warning flags.
* configure.ac (warn_c): Add -Wmissing-prototypes.
* data/skeletons/yacc.c (pstate_clear): Make it static.
* examples/local.mk (TEST_CFLAGS): New.
* examples/c/bistromathic/local.mk, examples/c/calc/local.mk,
* examples/c/lexcalc/local.mk, examples/c/mfcalc/local.mk,
* examples/c/pushcalc/local.mk, examples/c/reccalc/local.mk,
* examples/c/rpcalc/local.mk:
Use it.
GCC's warn_unused_result is not silenced by a cast to void, so we have
to "use" scanf's result.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25509https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66425
Flex generated code produces too many warnings, including things such
as, with ICC:
examples/c/lexcalc/scan.c(1088): error #1682: implicit conversion
of a 64-bit integral type to a smaller integral type (potential portability problem)
2259 YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]),
2260 ^
2261
2262
I am tired of trying to fix Flex's output. The project does not seem
maintained. We ought to avoid it. So, for the time being, don't try
to enable warnings with Flex.
* examples/c/bistromathic/parse.y, examples/c/reccalc/scan.l: Fix
warnings.
* doc/bison.texi: Discard scanf's return value to defeat
-Werror=unused-result.
* etc/bench.pl.in, examples/c/bistromathic/parse.y,
* examples/c/calc/calc.y, examples/c/pushcalc/calc.y: Check scanf's
return value.
* doc/bison.texi: Likewise, but only for the second example, to avoid
cluttering the very simple case.
* maint:
news: update
maint: post-release administrivia
version 3.6.1
c++: style: reorder generated code
c++: provide yy::parser::symbol_type::name
c++: make parser::symbol_name public
examples: beware of ~/.inputrc
build: also provide lzip compressed tarballs
style: minor fixes
yacc.c: restore ansi-c compatibility
The user gives yyexpected_tokens a limit: the max number of tokens she
wants to hear about. That's because an error message that reports a
bazillion of possible tokens is useless.
In that case yyexpected_tokens returned 0, so the user would not know
if there are too many expected tokens or none (yes, that's possible).
There are several ways to tell the user in which situation she's in:
- return some E2MANY, a negative value. Then it makes the pattern
int argsize = yypcontext_expected_tokens (ctx, arg, ARGS_MAX);
if (argsize < 0)
return argsize;
no longer valid, as for E2MANY (i) the user must generate the error
message anyway, and (ii) she should not return E2MANY
- return ARGS_MAX + 1. Then it makes it dangerous for the user, as
she has to iterate update `min (ARGS_MAX, argsize)`.
Returning 0 is definitely simpler and safer for the user, as it tells
her "this is not an error, just generate your message without a list
of expecting tokens". So let's still return 0, but set arg[0] to the
empty token when the list is really empty.
* data/skeletons/glr.c, data/skeletons/lalr1.cc, data/skeletons/lalr1.java
* data/skeletons/yacc.c (yyexpected_tokens): Put the empty symbol
first if there are no possible tokens at all.
* examples/c/bistromathic/parse.y: Demonstrate how to use that.
The shell grammar does not allow empty statements in then/else part of
an if, but examples/test failed to catch the syntax errors from the
script it ran. So exited with success anyway.
You would expect 'set -e' to suffice, but with bash 3.2 actually it
does not. As a matter of fact, I could find a way to have this behave
properly:
$ cat test.sh
set -e
cleanup ()
{
status=$?
echo "cleanup: $status"
exit $status
}
trap cleanup 0 1 2 13 15
. $1
s=$?
echo "test.sh: $s"
exit $s
$ cat bistro.test
if true; then
fi
$ /bin/sh ./test.sh ./bistro.test
./bistro.test: line 2: syntax error near unexpected token `fi'
cleanup: 0
$ echo $?
0
Remove the set -e (or the trap), and tada, it works... So we have to
deal with the error by hand.
* examples/test ($exit): Replace with...
($status): this.
Preserve the exit status of the test case.
* examples/c/bistromathic/bistromathic.test: Fix syntax error.
One function missing on Solaris 10 Sparc:
CCLD examples/c/bistromathic/bistromathic
Undefined first referenced
symbol in file
strnlen examples/c/bistromathic/bistromathic-parse.o
ld: fatal: symbol referencing errors. No output written to examples/c/bistromathic/bistromathic
Reported by Dagobert Michelsen.
https://lists.gnu.org/r/bug-bison/2020-05/msg00048.html
* examples/c/bistromathic/parse.y (xstrndup): Don't use strnlen.
xstrndup is assembled from gnulib's xstrndup, strndup and strnlen...
Readline may emit escape sequences before the prompt.
Reported by Bruno Haible.
https://lists.gnu.org/r/platform-testers/2020-05/msg00001.html.
* examples/c/bistromathic/bistromathic.test: Trust readline _only_ if
we get what we expect on some reference computation.
Appearing on IRIX with gcc -mabi=n32.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00039.html
* examples/c++/variant-11.yy, examples/c/bistromathic/parse.y: Don't
give chars to isdigit, cast them to unsigned char before.
* src/complain.c: Use c_isdigit.
* src/fixits.c (fixits_run): Avoid casts.
* src/lalr.c (goto_print): Use %zu for a size_t.
Don't try to build bistromathic if we don't have readline.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00028.html
* configure.ac (ENABLE_BISTROMATHIC): New.
* examples/c/bistromathic/local.mk: Use it.
* examples/c/bistromathic/bistromathic.test: Exit 77 for skip.
"foo || bar" does not invoke bar on AIX 7.2 when foo does not exist.
It just dies.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00029.html
* examples/c/reccalc/reccalc.test: Check for seq in a subshell.
* data/skeletons/lalr1.java (Location): Make it a static class.
(Lexer.yylex, Lexer.getLVal, Lexer.getStartPos, Lexer.getEndPos):
These are not needed in push parsers.
* examples/java/calc/Calc.y: Demonstrate push parsers in the Java.
* doc/bison.texi: Push parsers have been supported for a long time,
remove incorrect statements stating the opposite.
Reported by Horst von Brand.
https://lists.gnu.org/r/bug-bison/2020-04/msg00033.html
* examples/c/bistromathic/Makefile: libintl might not be needed, but
libm probably is.
* examples/c/bistromathic/parse.y: Include locale.h.
On OpenBSD 6.5, the prompt is repeated, but not the actual command
line... Don't try to cope with that.
Reported by Bruno Haible.
https://lists.gnu.org/r/bug-bison/2020-05/msg00015.html
* examples/c/bistromathic/bistromathic.test: Skip when readline behave
this way.
