This avoids heap allocation and gives minimal costs for the
creation and destruction of the YYParser.Symbol struct if
the location tracking is active.
Suggested by H. S. Teoh.
* data/skeletons/lalr1.d: Here.
* doc/bison.texi: Document it.
* examples/d/calc/calc.y: Adjust.
* tests/calc.at: Test it.
The complete symbol approach was deemed to be the right approach for Dlang.
Now, the user can return from yylex() an instance of YYParser.Symbol structure,
which binds together the TokenKind, the semantic value and the location. Before,
the last two were reported separately to the parser.
Only the user API is changed, Bisons's internal structure is kept the same.
* data/skeletons/d.m4 (struct YYParser.Symbol): New.
* data/skeletons/lalr1.d: Change the return value.
* doc/bison.texi: Document it.
* examples/d/calc/calc.y, examples/d/simple/calc.y: Demonstrate it.
* tests/calc.at, tests/scanner.at: Test it.
When using lookahead correction, the method YYParser.Context.getExpectedTokens
is not annotated with const, because the method calls yylacCheck, which is not
const. Also, because of yylacStack and yylacEstablished, yylacCheck needs to
be called from the context of the parser class, which is sent as parameter to
the Context's constructor.
* data/skeletons/lalr1.d (yylacCheck, yylacEstablish, yylacDiscard,
yylacStack, yylacEstablished): New.
(Context): Use it.
* doc/bison.texi: Document it.
* tests/calc.at: Check it.
Parser.Context class returns a const YYLocation, so Lexer's method
yyerror() needs to receive the location as a const parameter.
Internal error reporting flow is changed to be similar to that of
the other skeletons. Before, case YYERRLAB was calling yyerror()
with the result of yysyntax_error() as the string parameter. As the
custom error message lets the user decide if they want to use
yyerror() or not, this flow needed to be changed. Now, case YYERRLAB
calls yyreportSyntaxError(), that builds the error message using
yysyntaxErrorArguments(). Then yyreportSyntaxError() passes the
error message to the user defined syntax_error() in case of a custom
message, or to yyerror() otherwise.
In the tests in tests/calc.at, the order of the tokens needs to be
changed in order of precedence, so that the D program outputs the
expected tokens in the same order as the other parsers.
* data/skeletons/lalr1.d: Add the custom error message feature.
* doc/bison.texi: Document it.
* examples/d/calc/calc.y: Adjust.
* tests/calc.at, tests/local.at: Test it.
In D's case, yyerrok() is a private method of the Parser class.
It can be called directly as `yyerrok()` from the grammar rules section.
* data/skeletons/lalr1.d: Add yyerrok().
* examples/d/calc/calc.y, examples/d/simple/calc.y: Demonstrate yyerrok().
* tests/calc.at: Update D tests to use yyerrok().
examples/java/calc/Calc.java:1531: warning: [deprecation] Integer(String) in Integer has been deprecated
yylval = new Integer(st.sval);
^
* examples/java/calc/Calc.y, examples/java/simple/Calc.y,
* tests/calc.at, tests/scanner.at: Use Integer.parseInt.
* tests/calc.at: Add tests for LAC in pull and push parsers.
Skip LAC: line from the logs.
* tests/local.at (reportSyntaxError): Output the error message in a
single call, to avoid having the error message on stderr be
interrupted by the debug traces of LAC in getExpectedTokens.
* tests/local.at (AT_TOKEN_TRANSLATE_IF): New, moved from...
* tests/calc.at: here.
Instead of sorting per feature (main, yylex, calc.y) and then by
language, do the converse, so that C bits are together, etc.
* data/skeletons/lalr1.d: Change the return value.
* examples/d/calc/calc.y, examples/d/simple/calc.y: Adjust.
* tests/scanner.at: Adjust.
* tests/calc.at (_AT_DATA_CALC_Y(d)): New, extracted from...
(_AT_DATA_CALC_Y(c)): here.
The two grammars have been sufficiently different to be separated.
Still trying to be them together results in a maintenance burden. For
the same reason, instead of specifying the results for D and for the
rest, compute the expected results with D from the regular case.
The D skeleton was not properly supporting @1 etc.
Reported by Adela Vais.
https://lists.gnu.org/r/bison-patches/2020-09/msg00049.html
* data/skeletons/d.m4 (b4_rhs_location): Fix it.
* tests/calc.at: Check the support of @n for all the skeletons.
This is consistent with --defines being deprecated in favor of
--header. The directive %defines is also too similar to %define.
And %header matches nicely with api.header.name.
* src/scan-gram.l (%defines): Deprecate to %header.
(%header): Scan it.
* src/parse-gram.y (PERCENT_DEFINES): Replace with...
(PERCENT_HEADER): this.
* data/skeletons/lalr1.java
* doc/bison.texi
* tests/actions.at, tests/c++.at, tests/calc.at, tests/conflicts.at,
* tests/input.at, tests/java.at, tests/local.at, tests/output.at,
* tests/synclines.at, tests/types.at:
Convert most tests to check %header instead of %defines.
This is a fork of glr.cc to be c++-first instead of a wrapper around
glr.c.
* data/skeletons/glr2.cc: New.
* data/skeletons/bison.m4, data/skeletons/c++.m4: Adjust.
* data/skeletons/c.m4 (b4_user_args_no_comma): New.
* src/reader.c (grammar_rule_check_and_complete): glr2.cc is C++.
* tests/actions.at, tests/c++.at, tests/calc.at, tests/conflicts.at,
* tests/input.at, tests/local.at, tests/regression.at, tests/scanner.at,
* tests/synclines.at, tests/types.at: Also check glr2.cc.
This was a hack to make it easier for people to migrate from yacc.c to
lalr1.cc and from glr.c to glr.cc: when set, YYSTYPE and YYLTYPE were
`#defined`. It was never documented (just mentioned in NEWS for Bison
2.2, 2006-05-19), but was used to simplify the test suite. Stop that:
adjust the test suite to the skeletons, not the converse.
In C++ use yy::parser::semantic_type, yy::parser::location_type, and
yy::parser::token::MY_TOKEN, instead of YYSTYPE, YYLTYPE and MY_TOKEN.
* data/skeletons/glr.cc, data/skeletons/lalr1.cc: Remove its support.
* tests/actions.at, tests/c++.at, tests/calc.at: Adjust.
From
public interface Lexer {
/* Token kinds. */
/** Token number, to be returned by the scanner. */
static final int YYEOF = 0;
/** Token number, to be returned by the scanner. */
static final int YYERRCODE = 256;
/** Token number, to be returned by the scanner. */
static final int YYUNDEF = 257;
/** Token number, to be returned by the scanner. */
static final int BANG = 258;
...
/** Deprecated, use b4_symbol(0, id) instead. */
public static final int EOF = YYEOF;
to
public interface Lexer {
/* Token kinds. */
/** Token "end of file", to be returned by the scanner. */
static final int YYEOF = 0;
/** Token error, to be returned by the scanner. */
static final int YYerror = 256;
/** Token "invalid token", to be returned by the scanner. */
static final int YYUNDEF = 257;
/** Token "!", to be returned by the scanner. */
static final int BANG = 258;
...
/** Deprecated, use YYEOF instead. */
public static final int EOF = YYEOF;
* data/skeletons/java.m4 (b4_token_enum): Display the symbol's tag in
comment.
* data/skeletons/lalr1.java: Address overquotation issue.
* examples/java/calc/Calc.y, examples/java/simple/Calc.y: Use YYEOF,
not EOF.
I'm quite pleased to see that the tricky case of glr.c was already
prepared by the changes to support syntax_error exceptions. Better
yet, it is actually syntax_error that becomes a special case of the
general pattern: make yytoken be YYERRCODE.
* data/skeletons/glr.c (YYFAULTYTOK): Remove the now useless (Basil)
Faulty token.
Instead, use the error token.
* data/skeletons/lalr1.d, data/skeletons/lalr1.java: When computing
the action, first check the case of the error token.
* tests/calc.at: Check cases for the error token symbols before and
after it.
* data/skeletons/yacc.c (yyparse): When the scanner returns YYERRCODE,
go directly to error recovery (yyerrlab1).
However, don't keep the error token as lookahead, that token is too
special.
* data/skeletons/lalr1.cc: Likewise.
* examples/c/bistromathic/parse.y (yylex): Use that feature to report
nicely invalid characters.
* examples/c/bistromathic/bistromathic.test: Check that.
* examples/test: Neutralize gratuitous differences such as rule
position.
* tests/calc.at: Check that case in C only.
The other case seem to be working, but that's an illusion that the
next commit will address (in fact, they can enter endless loops, and
report the error several times anyway).
On macOS, wc -l always prepends the result with a tab, even when fed
by stdin. But anyway, we should have used `grep -c -v`, which appears
to be portable according to Autoconf's "Limitations of Usual Tools"
section.
Reported by Denis Excoffier.
https://lists.gnu.org/r/bug-bison/2020-04/msg00009.html
* tests/calc.at (_AT_CHECK_CALC): Use grep's -c instead.
* data/skeletons/lalr1.cc: added support here
* tests/calc.at: added test cases
* tests/local.at: added yyreport_syntax_error implementation
for C++ test cases
This revealed a number of things I had not realized:
- the Java location tracking was aliasing the same pair of positions
for all the symbols (see previous commit).
- in impure parsers, it's quite easy to use incorrect locations for
diagnostics, since yyerror uses yylloc, which is the location of the
lookahead, not that of the current lhs. So we need something like
{
YYLTYPE old_yylloc = yylloc;
yylloc = @$;
yyerror (]AT_PARAM_IF([result, count, nerrs, ])[buf);
yylloc = old_yylloc;
}
Maybe we should do that little yylloc dance in the skeleton instead
of leaving it to the user? It might be costly... But that's only
for users of the impure parsers, which are asking for trouble
anyway.
- in glr.cc invoking yyerror is somewhat cumbersome: the C++ interface
is not available as we are in yyparse (which in C), and yyerror is
used by glr.cc itself to bind it to the user's parser::error. If we
call yyerror, we need:
yyerror (]AT_LOCATION_IF([[&@$, ]])[yyparser, ]AT_PARAM_IF([result, count, nerrs, ])[msg);
However calling yy::parser::error is easier, once we know that the
current parser object is available as 'yyparser'. Which also saves
us from having to pass the parse-params ourselves:
yyparser.error (]AT_LOCATION_IF([[@$, ]])[msg);
* tests/calc.at: Invoke yyerror by hand, instead of using fprintf etc.
Adjust expectations.
In Java there is no need for N_ and yytranslate_. So instead of
hard-coding the use of N_ in the table of the symbol names, rely on
b4_symbol_translate.
* src/output.c (prepare_symbol_names): Use b4_symbol_translate instead
of N_.
* data/skeletons/c.m4 (b4_symbol_translate): New.
* data/skeletons/lalr1.java (yysymbolName): New.
Use it.
* examples/java/calc/Calc.y: Use parse.error=detailed.
* tests/calc.at: Check parse.error=detailed.
Unfortunately in the Java skeleton the user cannot override the way
locations are displayed, and locations don't know the structure of the
positions. So they cannot implement the tricks used in the C/C++
skeletons to display "1.1" instead of "1.1-1.2".
* tests/local.at (Java): Add support for column tracking in the
locations, as we did in examples/java/calc.
* tests/calc.at: Use AT_CALC_YYLEX.
The C, C++ and D skeletons used to show the stack right after popping
the stack during the reduction. Now that the stack is printed after
reaching a new state, that has become useless:
Entering state 1
Stack now 0 1
Reducing stack by rule 5 (line 83):
$1 = token "number" (1)
-> $$ = nterm exp (1)
Stack now 0
Entering state 8
Stack now 0 8
Remove the "Stack now 0" line.
* data/skeletons/lalr1.cc, data/skeletons/lalr1.d,
* data/skeletons/lalr1.java, data/skeletons/yacc.c:
Here.
Currently, if we have long rules and series of shift, we stack states
without showing stack. Let's be more incremental, and do how the Java
skeleton does.
* data/skeletons/lalr1.cc, data/skeletons/lalr1.d,
* data/skeletons/yacc.c:
Here.
Adjust test cases.
* tests/torture.at (AT_DATA_STACK_TORTURE): Disable stack traces: this
test produces a very large stack, and showing the stack each time we
shift a token goes quadatric.
The Java skeleton displays
Reading a token:
Next token is token "number" (1)
while the other display
Reading a token: Next token is token "number" (1)
When generating logs in the scanner, the first part is separated from
the second, and the end of the scanner logs have the second part
pasted in. So let's propagate the Java way, but with the colon.
* data/skeletons/glr.c, data/skeletons/lalr1.cc, data/skeletons/lalr1.d,
* data/skeletons/lalr1.java, data/skeletons/yacc.c: Do it.
Adjust test cases and doc.
* tests/local.at (AT_LANG_MATCH): New.
(AT_YYERROR_DECLARE(java), AT_YYERROR_DECLARE_EXTERN(java)): New.
* tests/calc.at: The grammar file for Java is quite different for the
others, and continuing to assemble it from pieces makes the grammar
file hard to understand. Let's also dispatch on the language to
assemble it, and isolate Java from the others.
Most of this comes from java.at.
When building the test cases, emitting code in the epilogue is very
constraining. Let's make it simpler thanks to %code epilogue.
However, I don't want to document this: it is bad style to use it (we
should avoid having too many ways to write the same thing,
TI!MTOWTDI), just put your code in the true epilogue section.
* data/skeletons/glr.c, data/skeletons/lalr1.d, data/skeletons/lalr1.java,
* data/skeletons/yacc.c: Implement support for %code epilogue.
Remove useless comments.
* tests/calc.at, tests/java.at: Simplify.
