Bitten by macros, again.
See 680b715518.
* data/variant.hh (_b4_symbol_constructor_declare)
(_b4_symbol_constructor_define): Do not use user types, which can
include commas as in `std::pair<int, int>`, to macros.
* tests/local.at: Adjust the lex related macros to support the
case of token constructors.
* tests/types.at: Also check token constructors on types with commas.
Prompted by Rici Lake.
http://lists.gnu.org/archive/html/bug-bison/2018-10/msg00000.html
We have four classes of directives that declare symbols: %nterm,
%type, %token, and the family of %left etc. Currently not all of them
support the possibility to have several type tags (`<type>`), and not
all of them support the fact of not having any type tag at all
(%type). Let's unify this.
- %type
POSIX Yacc specifies that %type is for nonterminals only. However,
some Bison users want to use it for both tokens and nterms
(actually, Bison's own grammar does this in several places, e.g.,
CHAR). So it should accept char/string literals.
As a consequence cannot be used to declare tokens with their alias:
`%type foo "foo"` would be ambiguous (are we defining foo = "foo",
or are these two different symbols?)
POSIX specifies that it is OK to use %type without a type tag. I'm
not sure what it means, but we support it.
- %token
Accept token declarations with number and string literal:
(ID|CHAR) NUM? STRING?.
- %left, etc.
They cannot be the same as %token, because we accept to declare the
symbol with %token, and to then qualify its precedence with %left.
Then `%left foo "foo"` would also be ambiguous: foo="foo", or two
symbols.
They cannot be simply a list of identifiers, but POSIX Yacc says we
can declare token numbers here. I personally think this is a bad
idea, precedence management is tricky in itself and should not be
cluttered with token declaration issues.
We used to accept declaring a token number on a string literal here
(e.g., `%left "token" 1`). This is abnormal. Either the feature is
useful, and then it should be supported in %token, or it's useless
and we should not support it in corner cases.
- %nterm
Obviously cannot accept tokens, nor char/string literals. Does not
exist in POSIX Yacc, but since %type also works for terminals, it is
a nice option to have.
* src/parse-gram.y: Avoid relying on side effects. For instance, get
rid of current_type, rather, build the list of symbols and iterate
over it to assign the type.
It's not always possible/convenient. For instance, we still use
current_class.
Prefer "decl" to "def", since in the rest of the implementation we
actually "declare" symbols, we don't "define" them.
(token_decls, token_decls_for_prec, symbol_decls, nterm_decls): New.
Use them for %token, %left, %type and %nterm.
* src/symlist.h, src/symlist.c (symbol_list_type_set): New.
* tests/regression.at b/tests/regression.at
(Token number in precedence declaration): We no longer accept
to give a number to string literals.
After having spent quite some time on cleaning the handling of symbol
declarations in the grammar files, I believe we should keep it.
It looks like it's a duplicate of %type, but it is not. While POSIX
Yacc requires %type to apply only to nonterminal symbols, it appears
that both byacc and bison accept it for tokens too. And some
experienced users do actually expect this feature to group
symbols (terminal or not) by type ("On the other hand, it is generally
more useful IMHO to group terminals and non-terminals with the same
type tag together",
http://lists.gnu.org/archive/html/bug-bison/2018-10/msg00000.html).
Even Bison's own parser does this today (see CHAR).
Basically reverts 7928c3e6fb.
* src/scan-gram.l (%nterm): Dedeprecate, but issue a Wyacc warning.
* tests/input.at: Adjust expectations.
(Yacc warnings on symbols): New.
* src/symtab.c (symbol_class_set): Fix error introduced in
20b0746793.
I personally prefer 'non terminal', or 'non-terminal', but
'nonterminal' is the common spelling.
* data/glr.c, src/parse-gram.y, src/symtab.c, src/symtab.h,
* tests/input.at, doc/refcard.tex: here.
Currently our error messages include both "symbol redeclared" and
"symbol redefined", and they mean something different. This is
obscure, let's make this clearer.
I think the idea between 'definition' vs. 'declaration' is that in the
case of the nonterminals, the actual definition is its set of rules,
so %nterm would be about declaration. The case of %token is less
clear.
* src/symtab.c (complain_class_redefined): New.
(symbol_class_set): Use it.
Simplify the logic of this function to clearly skip its body when the
preconditions are not met.
* tests/input.at (Symbol class redefinition): New.
Convert some of the READMEs to Markdown, which is now more common, and
nicely displayed in some git hosting services.
Add missing READMEs and Makefiles. Generate XML, HTML and Dot files. Be
sure to ship the test files. Complete CLEANFILES to remove all generated
files.
* examples/calc++: Move into...
* examples/c++: here.
* examples/mfcalc, examples/rpcalc: Move into...
* examples/c: here.
* examples/README.md, examples/c++/calc++/Makefile, examples/c/local.mk,
* examples/c/mfcalc/Makefile, examples/c/rpcalc/Makefile,
* examples/d/README.md, examples/java/README.md:
New files.
* examples/test (medir): Be robust to deeper directory nesting.
Revamping the handling of the symbols is the grammar is much more
delicate than I anticipated. Let's first move things around for
clarity.
* src/symtab.c (symbol_make_alias): Don't accept to alias
non-terminals.
(symbol_user_token_number_set): Don't accept user token numbers
for non-terminals.
Don't do anything in case of redefinition, instead of trying to
update. The flow is eaier to follow this way.
When we extract the examples from the documentation, %require
"@value{VERSION}" is replaced with the current version. If we change
the git branch, without changing the documentation, the generated
examples will %require a version of Bison that differs from the actual
version.
* examples/local.mk (extracted.stamp): Depend on doc/version.texi.
* data/README: Convert to Markdown.
Start documenting some of the macros used in all our skeletons.
Simplify and fix the documentation of the macros in the skeletons.
Currently it is the front end that passes the symbol types to the
backend. For instance:
%token <ival> NUM
%type <ival> exp1 exp2
exp1: NUM { $$ = $1; }
exp2: NUM { $<ival>$ = $<ival>1; }
In both cases, $$ and $1 are passed to the backend as having type
'ival' resulting in code like `val.ival`. This is troublesome in the
case of api.value.type=union, since in that the case the code this:
%define api.value.type union
%token <int> NUM
%type <int> exp1 exp2
exp1: NUM { $$ = $1; }
exp2: NUM { $<int>$ = $<int>1; }
because in this case, since the backend does not know the symbol being
processed, it is forced to generate casts in both cases: *(int*)(&val)`.
This is unfortunate in the first case (exp1) where there is no reason
at all to use a cast instead of `val.NUM` and `val.exp1`.
So instead delegate the computation of the actual value type to the
backend: pass $<ival>$ as `symbol-number, ival` and $$ as
`symbol-number, MULL`, instead of passing `ival` before.
* src/scan-code.l (handle_action_dollar): Find the symbol the action
is about, not just its tyye. Pass both symbol-number, and explicit
type tag ($<tag>n when there is one) to b4_lhs_value and b4_rhs_value.
* data/bison.m4 (b4_symbol_action): adjust to the new signature to
b4_dollar_pushdef.
* data/c-like.m4 (_b4_dollar_dollar, b4_dollar_pushdef): Accept the
symbol-number as new argument.
* data/c.m4 (b4_symbol_value): Accept the symbol-number as new
argument, and use it.
(b4_symbol_value_union): Accept the symbol-number as new
argument, and use it to prefer ready a union member rather than
casting the union.
* data/yacc.c (b4_lhs_value, b4_rhs_value): Accept the new
symbol-number argument.
Adjust uses of b4_dollar_pushdef.
* data/glr.c (b4_lhs_value, b4_rhs_value): Adjust.
* data/lalr1.cc (b4_symbol_value_template, b4_lhs_value): Adjust
to the new symbol-number argument.
* data/variant.hh (b4_symbol_value, b4_symbol_value_template): Accept
the new symbol-number argument.
* data/java.m4 (b4_symbol_value, b4_rhs_data): New.
(b4_rhs_value): Use them.
* data/lalr1.java: Adjust to b4_dollar_pushdef, and use b4_rhs_data.
* data/bison.m4, data/c++.m4, data/glr.c, data/java.m4, data/lalr1.cc,
* data/yacc.c, src/scan-code.l:
Fix comments.
Prefer POS to denote the position of a symbol in a rule, since NUM
is also used to denote symbol numbers.
Instead of defining yy::variant<S> and then alias
yy::parser::semantic_type to variant<sizeof (union_type)>, directly
define yy::parser::semantic_type.
This model is more appropriate if we want to sit the storage on top of
unions in C++11.
* data/variant.hh (b4_variant_define): Specialize and inline the
definition into...
(b4_value_type_declare): Here.
Define union_type here.
* data/lalr1.cc: Adjust.
* examples/java/Calc.y: New, based on test 495: "Calculator
parse.error=verbose %locations".
* examples/java/Calc.test, examples/java/local.mk: New.
* configure.ac (ENABLE_JAVA): New.
* examples/test (prog): Be ready to run Java programs.
This was demanded several times. See for instance:
- David M. Warme
https://lists.gnu.org/archive/html/help-bison/2011-04/msg00003.html
- box12009
http://lists.gnu.org/archive/html/bug-bison/2016-10/msg00001.html
Basically, this reverts:
- commit 3d3bc1fe30
Get rid of (yy)rhs and (yy)prhs
- commit d333175f63
Avoid compiler warning.
Note that since these tables are not needed in the generated parsers,
no skeleton requests them. This change only brings back their
definition to M4, making it possible to user-defined skeletons to use
these tables.
* src/output.c (muscle_insert_item_number_table): Define.
(prepare_rules): Generate the rhs and prhs tables.
Reported by Rici Lake.
http://lists.gnu.org/archive/html/bug-bison/2018-10/msg00000.html
* src/complain.h: Formatting change.
* src/parse-gram.y (id): Reject character literals used in a context
for non-terminals.
* tests/input.at (Invalid %nterm uses): Check that.
Reported by Uxio Prego.
https://lists.gnu.org/archive/html/help-bison/2018-11/msg00031.html
We also need to move the unreachable 'goto' to a reachable place,
otherwise clang complains about the code being unreachable anyway.
See also https://bugs.llvm.org/show_bug.cgi?id=39736.
Interestingly, we don't have to apply that trick to
`#define YYCDEBUG if (false) std::cerr`, clang does not warn when the
code comes from macro expansion.
* configure.ac: Use -Wunreachable-code when supported.
* data/lalr1.cc, data/yacc.c: Pacify clang's warning about `if (0)`
by using a macro.
Another possibility was to move this statement to a reachable place.
* tests/actions.at, tests/c++.at: Avoid generating unreachable code.
We should probably introduce some struct and functions to deal with
stack management, rather than open coding it. yyparse would be much
nicer to read, and a better model for possible other skeletons.
* data/yacc.c (yyparse::yysetstate): Avoid generating code when
neither yyoverflow nor YYSTACK_RELOCATE is defined.
Currently on a grammar such as
exp : a '1' | a '2' | a '3' | b '1' | b '2' | b '3'
a:
b:
we count only one rr-conflict on the `b:` rule, i.e., we expect:
b: %expect-rr 1
although there are 3 conflicts in total. That's because in the
conflicted state we count only a single conflict, not three (one for
each of the lookaheads: '1', '2', '3').
State 0
0 $accept: . exp $end
1 exp: . a '1'
2 | . a '2'
3 | . a '3'
4 | . b '1'
5 | . b '2'
6 | . b '3'
7 a: . %empty ['1', '2', '3']
8 b: . %empty ['1', '2', '3']
'1' reduce using rule 7 (a)
'1' [reduce using rule 8 (b)]
'2' reduce using rule 7 (a)
'2' [reduce using rule 8 (b)]
'3' reduce using rule 7 (a)
'3' [reduce using rule 8 (b)]
$default reduce using rule 7 (a)
exp go to state 1
a go to state 2
b go to state 3
See https://lists.gnu.org/archive/html/bison-patches/2013-02/msg00106.html.
* src/conflicts.c (rule_has_state_rr_conflicts): Rename as...
(count_rule_state_sr_conflicts): this.
DWIM.
(count_rule_rr_conflicts): Adjust.
* tests/conflicts.at (%expect-rr in grammar rules)
(%expect-rr too much in grammar rules)
(%expect-rr not enough in grammar rules): New.
On a grammar such as
exp: "num" | "num" | "num"
we currently report only one RR conflict, instead of two.
This bug is present since the origins of Bison
commit 08089d5d35
Author: David MacKenzie <[email protected]>
Date: Tue Apr 20 05:42:52 1993 +0000
Initial revision
and was preserved in
commit 676385e29c
Author: Paul Hilfinger <[email protected]>
Date: Fri Jun 28 02:26:44 2002 +0000
Initial check-in introducing experimental GLR parsing. See entry in
ChangeLog dated 2002-06-27 from Paul Hilfinger for details.
See
https://lists.gnu.org/archive/html/bison-patches/2018-11/msg00011.html
* src/conflicts.h, src/conflicts.c (count_state_rr_conflicts)
(count_rr_conflicts): Use only the correct count of conflicts.
* tests/glr-regression.at: Fix expectations.
Currently on a grammar such as
exp: "number" | exp "+" exp | exp "*" exp
we count only one sr-conflict for both binary rules, i.e., we expect:
exp: "number" | exp "+" exp %expect 1 | exp "*" exp %expect 1
although there are 4 conflicts in total. That's because in the states
in conflict, for instance that for the "+" rule:
State 6
2 exp: exp . "+" exp
2 | exp "+" exp . [$end, "+", "*"]
3 | exp . "*" exp
"+" shift, and go to state 4
"*" shift, and go to state 5
"+" [reduce using rule 2 (exp)]
"*" [reduce using rule 2 (exp)]
$default reduce using rule 2 (exp)
we count only a single conflict, although there are two (one on "+"
and another with "*").
See https://lists.gnu.org/archive/html/bison-patches/2013-02/msg00106.html.
* src/conflicts.c (rule_has_state_sr_conflicts): Rename as...
(count_rule_state_sr_conflicts): this.
DWIM.
(count_rule_sr_conflicts): Adjust.
* tests/conflicts.at (%expect in grammar rules): New.
We also lack a consistent naming for directive implementations.
`directive_skeleton` is too long, `percent_skeleton` is not very nice
looking, `process_skeleton` looks ambiguous, `do_skeleton` is somewhat
ambiguous too, but seems a better track.
* src/parse-gram.y (version_check): Rename as...
(do_require): this.
(do_skeleton): New.
Use it.
We may generate code such as
basic_symbol (typename Base::kind_type t, YY_RVREF (std::pair<int,int>) v);
which, of course, breaks, because YY_RVREF sees two arguments. Let's
not play tricks with _VA_ARGS__, I'm unsure about it portability.
Anyway, I plan to change more things in this area.
Reported by Sébastien Villemot.
http://lists.gnu.org/archive/html/bug-bison/2018-11/msg00014.html
* data/variant.hh (b4_basic_symbol_constructor_declare)
(b4_basic_symbol_constructor_define): Don't use macro on user types.
* tests/types.at: Check that we support pairs.
In C++, [[noreturn]] must not be between "static" and the rest of the
function signature, it must precede it. C's _Noreturn does not seem
to have such a constraint, but it is therefore compatible with the C++
constraint. Since we #define _Noreturn as [[noreturn]] is modern C++,
be sure to push the _Noreturn first.
Unfortunately this was not caught by the test suite, because it always
loads config.h first, and config.h contains another definition of
_Noreturn that does not use [[noreturn]], and hides ours. That's
probably a sign we should avoid always loading config.h.
* data/glr.c (yyFail, yyMemoryExhausted): here.
* tests/local.at (AT_LANG_FOR_EACH_STD): New.
(AT_REQUIRE_CXX_VERSION): Rename as...
(AT_REQUIRE_CXX_STD): this.
Accept an argument for what to do when the requirement is not met.
* tests/types.at (api.value.type): Check all the C++ stds.
The following commit introduce even more compilations/runs than
before, and with ASAN on, we go beyond to 50min credit from Travis.
* .travis.yml (Clang 7 libc++ and ASAN): Split in two.
Akim Demaille