* data/skeletons/yacc.c (yyparse): When the scanner returns YYERRCODE,
go directly to error recovery (yyerrlab1).
However, don't keep the error token as lookahead, that token is too
special.
* data/skeletons/lalr1.cc: Likewise.
* examples/c/bistromathic/parse.y (yylex): Use that feature to report
nicely invalid characters.
* examples/c/bistromathic/bistromathic.test: Check that.
* examples/test: Neutralize gratuitous differences such as rule
position.
* tests/calc.at: Check that case in C only.
The other case seem to be working, but that's an illusion that the
next commit will address (in fact, they can enter endless loops, and
report the error several times anyway).
* examples/c/bistromathic/parse.y: here.
* examples/c/bistromathic/bistromathic.test: Check it.
Included a stupid case where the error is actually ignored.
When the user ctrl-d the line, we left the cursor not at col 0.
Let's fix that.
This revealed a few short-comings in the testing framework.
* examples/test (run): Also display the diffs.
And support -n.
* examples/c/bistromathic/bistromathic.test
* examples/c/bistromathic/parse.y
Currently it was only using stubs. Let's actually translate the
strings using gettext.
* examples/c/bistromathic/local.mk: Define LOCALEDIR, BISON_LOCALEDIR
and link with libintl.
* examples/c/bistromathic/parse.y: Use them.
Remove useless includes.
Take ENABLE_NLS into account.
(error_format_string): New.
(yyreport_syntax_error): Rewrite to rely on a format string, which is
more appropriate for internationalization.
* examples/c/bistromathic/Makefile: We no longer use Flex.
We need readline and intl.
* doc/bison.texi: Point to bistromathic for a better option for
internationalization.
* po/POTFILES.in: Add bistromathic.
macOS' version of readline does not repeat stdin on stdout in
non-interactive mode, contrary to the current version of GNU readline.
* examples/test: Add support for strip_prompt.
* examples/c/bistromathic/bistromathic.test (strip_prompt): Set it
when needed.
Early exit when needed.
Why didn't I think about this before??? symbolName should be a method
of SymbolKind.
* data/skeletons/lalr1.java (YYParser::yysymbolName): Move as...
* data/skeletons/java.m4 (SymbolKind::getName): this.
Make the table a static final table, not a local variable.
Adjust dependencies.
* doc/bison.texi (Java Parser Interface): Document i18n.
(Java Parser Context Interface): Document SymbolKind.
* examples/java/calc/Calc.y, tests/local.at: Adjust.
* doc/bison.texi (C++ Parser Context): New.
* data/skeletons/lalr1.cc (parser::yysymbol_name): Rename as...
(parser::symbol_name): this.
(A Complete C++ Example): Promote LAC, now that we have it.
Promote parse.error detailed over verbose.
* examples/c++/calc++/calc++.test, tests/local.at: Adjust.
The user should think of yypcontext fields as accessible only via
yypcontext_* functions. So let's rename yyexpected_tokens to reflect
that.
Let's _not_ rename yyreport_syntax_error, as the user may define this
function, and is not allowed to access directly the fields of
yypcontext_t: she *must* use the "accessors". This is comparable to
the case of C++/Java where the user defines
parser::report_syntax_error, not parser::context::report_syntax_error.
* data/skeletons/glr.c, data/skeletons/yacc.c (yyexpected_tokens):
Rename as...
(yypcontext_expected_tokens): this.
Adjust dependencies.
The name "$end" is nice in the report, in particular it avoids that
pointed-rules (aka items) be too long. It also helps keeping them
"standard".
But it is bad in error messages, we should report "end of file" (or
maybe "end of input", this is debatable). So, unless the user already
defined the alias for the error token herself, make it "end of file".
It should even be translated if the user already translated some
tokens, so that there is now no strong reason to redefine the $end
token.
* src/output.c (prepare_symbol_names): Issue "end of file" instead of
"$end".
* data/skeletons/lalr1.java (yytnamerr_): Remove the renaming hack.
* build-aux/update-test: Accept files with names containing a "+",
such as c++.at.
* tests/actions.at, tests/c++.at, tests/conflicts.at,
* tests/glr-regression.at, tests/regression.at, tests/skeletons.at:
Adjust.
That's one nice benefit from using enums.
* data/skeletons/lalr1.java (YYSYMBOL_YYEMPTY): No longer define it.
Use 'null' instead.
* examples/java/calc/Calc.y, tests/local.at: Adjust.
* doc/bison.texi: Replace occurrences of "token type" with "token
kind".
Stop referring to the "macro definitions" of the token kinds, just
name them "definitions".
* maint:
maint: post-release administrivia
version 3.5.4
examples: reccalc: really compile cleanly in C99
news: announce that Bison 3.6 drops YYERROR_VERBOSE
news: update for 3.5.4
style: fix spellos
typo: succesful -> successful
package: improve the readme
java: check and fix support for api.token.raw
java: style: prefer 'int[] foo' to 'int foo[]'
build: fix syntax-check issues
tests: recheck: work properly when the test suite was interrupted
doc: c++: promote api.token.raw
build: fix compatibility with old compilers
examples: reccalc: compile cleanly in C99
The previous fix does not suffice, and actually managed to make things
worse by defining yyscan_t twice in parse.y...
Reported by kencu.
https://trac.macports.org/ticket/59927#comment:29
* examples/c/reccalc/parse.y (yyscan_t): Define it with the same
guards as used by Flex.
The first name is too long. We already have `yypstate`, so
`yypcontext` is ok. We are also migrating to using `*_t` for our
types.
* NEWS, data/skeletons/glr.c, data/skeletons/yacc.c, doc/bison.texi,
* examples/c/bistromathic/parse.y, src/parse-gram.y, tests/local.at:
(yyparse_context_t, yyparse_context_location, yyparse_context_token):
Rename as...
(yypcontext_t, yypcontext_location, yypcontext_token): these.
The Java enums are very different from the C model. As a consequence,
one cannot "build" an enum directly from an integer, we must retrieve
it. That's the purpose of the SymbolType.get class method.
* data/skeletons/java.m4 (b4_symbol_enum, b4_case_code_symbol)
(b4_declare_symbol_enum): New.
* data/skeletons/lalr1.java: Use SymbolType,
SymbolType.YYSYMBOL_YYEMPTY, etc.
* examples/java/calc/Calc.y, tests/local.at: Adjust.
Now that we have a proper type for internal symbol numbers, let's use
it. More code needs conversion, e.g., printers and destructors, but
they are shared with glr.c, which is not ready yet for this change.
It will also help us deal with warnings such as (GCC9 on GNU/Linux):
input.c: In function 'int yyparse()':
input.c:475:37: error: enumeral and non-enumeral type in conditional expression [-Werror=extra]
475 | (0 <= (YYX) && (YYX) <= YYMAXUTOK ? yytranslate[YYX] : YYSYMBOL_YYUNDEF)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
input.c:1024:17: note: in expansion of macro 'YYTRANSLATE'
1024 | yytoken = YYTRANSLATE (yychar);
| ^~~~~~~~~~~
* data/skeletons/yacc.c (yytranslate, yysymbol_name)
(yyparse_context_t, yyexpected_tokens, yypstate_expected_tokens)
(yysyntax_error_arguments):
Use yysymbol_type_t instead of int.
Suggested by Adrian Vogelsgesang.
https://lists.gnu.org/archive/html/bison-patches/2020-02/msg00069.html
* data/skeletons/lalr1.java (Context.EMPTY, Context.getToken): New.
(Context.yyntokens): Rename as...
(Context.NTOKENS): this.
Because (i) all the Java coding styles recommend upper case for
constants, and (ii) the Java Skeleton exposes Lexer.EOF, not
Lexer.YYEOF.
* data/skeletons/yacc.c (yyparse_context_token): New.
* examples/c/bistromathic/parse.y (yyreport_syntax_error): Don't use
yysyntax_error_arguments.
* examples/java/calc/Calc.y (yyreportSyntaxError): Likewise.
yyparse returns 0, 1, 2 since ages (accept, reject, memory exhausted).
Some of our auxiliary functions such as yy_lac and
yyreport_syntax_error also need to return error codes and also use 0,
1, 2. Because it uses yy_lac, yyexpected_tokens also needs to return
"problem", "memory exhausted", but in case of success, it needs to
return the number of tokens, so it cannot use 1 and 2 as error code.
Currently it uses -1 and -2, which is later converted into 1 and 2 as
yacc.c expects it.
Let's simplify this and use consistently -1 and -2 for auxiliary
functions that are not exposed (or not yet exposed) to the user. In
particular this will save the user from having to convert
yyexpected_tokens's -2 into yyreport_syntax_error's 2: both return -1
or -2.
* data/skeletons/yacc.c (yy_lac, yyreport_syntax_error)
(yy_lac_stack_realloc): Return -1, -2 for errors instead of 1, 2.
Adjust callers.
* examples/c/bistromathic/parse.y (yyreport_syntax_error): Do take
error codes into account.
Issue a syntax error message even if we ran out of memory.
* src/parse-gram.y, tests/local.at (yyreport_syntax_error): Adjust.
In push parsers, when asking for the list of expected tokens at some
point, it makes no sense to build a yyparse_context_t: the yypstate
alone suffices (the only difference being the lookahead). Instead of
forcing the user to build a useless shell around yypstate, let's offer
yypstate_expected_tokens.
See https://lists.gnu.org/r/bison-patches/2020-03/msg00025.html.
* data/skeletons/yacc.c (yypstate): Declare earlier, so that we can
use it for...
(yypstate_expected_tokens): this new function, when in push parsers.
Adjust dependencies.
* examples/c/bistromathic/parse.y: Simplify: use
yypstate_expected_tokens.
Style fixes.
Reduce scopes (reported by Joel E. Denny).
Currently pstate_new does not set up its variables, this task is left
to yypush_parse. This was probably to share more code with usual pull
parsers, where these (local) variables are indeed initialized by
yyparse.
But as a consequence yyexpected_tokens crashes at the very beginning
of the parse, since, for instance, the stacks are not even set up.
See https://lists.gnu.org/r/bison-patches/2020-03/msg00001.html.
The fix could have very simple, but the documentation actually makes
it very clear that we can reuse a pstate for several parses:
After yypush_parse returns a status other than YYPUSH_MORE, the
parser instance yyps may be reused for a new parse.
so we need to restore the parser to its pristine state so that (i) it
is ready to run the next parse, (ii) it properly supports
yyexpected_tokens for the next run.
* data/skeletons/yacc.c (b4_initialize_parser_state_variables): New,
extracted from the top of yyparse/yypush_parse.
(yypstate_clear): New.
(yypstate_new): Use it when push parsers are enabled.
Define after the yyps macros so that we can use the same code as the
regular pull parsers.
(yyparse): Use it when push parsers are _not_ enabled.
* examples/c/bistromathic/bistromathic.test: Check the completion on
the beginning of the line.
Currently completion on "at" proposes only "atan", but does not
actually complete "at" into "atan".
* examples/c/bistromathic/parse.y (completion): Install the lcp in
matches[0].
* examples/c/bistromathic/bistromathic.test: Check that case.
Currently "(1+<TAB>" does not work as expected, because "+" is not a
word breaking character.
* examples/c/bistromathic/parse.y (init_readline): Specify our word
breaking characters.
* examples/c/bistromathic/bistromathic.test: Avoid trailing spaces.
Let's use GNU readline and its TAB autocompletion to demonstrate the
use of yyexpected_tokens.
This shows a number of weaknesses in our current approach:
- some macros (yyssp, etc.) from push parsers "leak" in user code, we
need to undefine them
- the context needed by yyexpected_tokens does not need the token,
yypstate actually suffices
- yypstate is not properly setup when first allocated, which results
in a crash of yyexpected_tokens if fired before a first token was
read. We should move initialization from yypush_parse into
yypstate_new.
* examples/c/bistromathic/parse.y (yylex): Take input as a string, not
a file.
(EXIT): New token.
(input): Adjust to work only on a line.
(line): Remove.
(symbol_count, process_line, expected_tokens, completion)
(init_readline): New.
* examples/c/bistromathic/bistromathic.test: Adjust expectations.
This example will soon use GNU readline, so its scanner should be easy
to use (concurrently) on strings, not streams. This is not a place
where Flex shines, and anyway, these are examples of Bison, not Flex.
There's already lexcalc and reccalc that demonstrate the use of Flex.
* examples/c/bistromathic/scan.l: Remove.
* examples/c/bistromathic/parse.y (yylex): New.
Adjust dependencies.
The bistromathic example should not use Flex, it makes it too complex.
But it was the only example to show location tracking with Flex.
* examples/c/lexcalc/lexcalc.test, examples/c/lexcalc/parse.y,
* examples/c/lexcalc/scan.l: Demonstrate location tracking as is done
in bistromathic.
