This should have been done in 3.6, but I wanted to avoid introducing
conflicts into Vincent's work on counterexamples. It turns out it's
completely orthogonal.
* data/README.md, data/skeletons/bison.m4, data/skeletons/c++.m4,
* data/skeletons/c.m4, data/skeletons/glr.c, data/skeletons/java.m4,
* data/skeletons/lalr1.d, data/skeletons/lalr1.java,
* data/skeletons/variant.hh, data/skeletons/yacc.c, src/conflicts.c,
* src/derives.c, src/gram.c, src/gram.h, src/output.c,
* src/parse-gram.c, src/parse-gram.y, src/print-xml.c, src/print.c,
* src/reader.c, src/symtab.c, src/symtab.h, tests/input.at,
* tests/types.at:
s/user_token_number/code/g.
Plus minor changes.
To write unit tests for their scanners, some users depended on
symbol_type::token():
Lexer lex("12345");
symbol_type t = lex.nextToken();
assert(t.token() == token::INTLIT);
assert(t.value.as<int>() == 12345);
But symbol_type::token() was removed in Bison 3.5 because it relied on
a conversion table. So users had to find other patterns, such as
assert(t.type_get() == by_type(token::INTLIT).type_get());
which relies on several private implementation details.
As part of transitioning from "token type" to "token kind", and making
this a public and documented interface, "by_type" was renamed
"by_kind" and "type_get()" was renamed as "kind()". The latter had
backward compatibility mechanisms, not the former.
In Bison 3.6 none of this should be used, but rather
assert(t.kind() == symbol_kind::S_INTLIT);
Reported by Pramod Kumbhar.
https://lists.gnu.org/r/bug-bison/2020-05/msg00012.html
* data/skeletons/c++.m4 (by_type): Make it an alias to by_kind.
These are internal details. `type_get ()` is still there to ensure
backward compatibility, `kind ()` being the modern way.
* data/skeletons/c++.m4 (by_type, by_type::type): Rename as...
(by_kind, by_kind::kind_): this.
Adjust dependencies.
symbol_type::token () was removed: it returned the token kind of a
symbol. To do that, one needs to convert from the symbol kind to the
token kind, which requires a table.
This broke some users' unit tests for scanners, see
https://lists.gnu.org/r/bug-bison/2020-01/msg00001.htmlhttps://lists.gnu.org/r/bug-bison/2020-03/msg00020.htmlhttps://lists.gnu.org/r/help-bison/2020-04/msg00005.html
Instead of making this possible again, let's check the symbol's kind
instead. So give proper access to a symbol's kind.
That feature existed, undocumented, as 'type_get()'. Let's rename
this as 'kind()'.
* data/skeletons/c++.m4, data/skeletons/glr.cc,
* data/skeletons/lalr1.cc (type_get): Rename as...
(kind): This.
(type_get): Install a backward compatibility alias.
* doc/bison.texi (Complete Symbols): Document symbol_type and
symbol_type::kind.
* data/skeletons/c++.m4: Define the old names in terms on the new
ones, instead of the converse.
* doc/bison.texi (C++ Parser Interface): Be more extensive about
token_kind_type.
I have been hesitating a lot before doing it ---after all the user
must not use this kind, so what's the point of showing it in
yytoken_kind_t. And eventually I chose to play it safe with the
typing system and make it possible to use yytoken_kind_t for all the
tokens, even the "empty token".
* data/skeletons/c.m4: Give an id and a tag to YYEMPTY.
(b4_token_enums): Define YYEMPTY.
* data/skeletons/c++.m4 (b4_token_enums): Define YYEMPTY.
* data/skeletons/glr.c, data/skeletons/glr.cc, data/skeletons/yacc.c:
(YYEMPTY): Remove.
Use b4_symbol(-2, id) instead.
* data/skeletons/bison.m4 (b4_symbol_kind): Dispatch on the UNDEF
token number rather than its name.
* data/skeletons/c++.m4, data/skeletons/c.m4, data/skeletons/java.m4:
Comment changes.
A forthcoming commit (tokens: properly define the "error" token kind)
revealed a problem in the C++ generated headers: they are not
self-contained. With this file:
%language "c++"
%define api.value.type variant
%code {
static int yylex (yy::parser::semantic_type *lvalp);
}
%token <int> X
%%
exp:
X { printf ("x\n"); }
;
%%
void
yy::parser::error (const std::string& m)
{
std::cerr << m << '\n';
}
static
int yylex (yy::parser::semantic_type *lvalp)
{
static int const input[] = {yy::parser::token::X, 0};
static int toknum = 0;
return input[toknum++];
}
int
main (int argc, char const* argv[])
{
yy::parser p;
return p.parse ();
}
the generated header fails to compile cleanly (foo.cc just #includes
the generated header):
$ clang++-mp-9.0 -c -Wundefined-func-template foo.cc
In file included from foo.cc:1:
bar.tab.hh:550:12: warning: instantiation of function 'yy::parser::basic_symbol<yy::parser::by_type>::basic_symbol' required here, but no definition is available
[-Wundefined-func-template]
struct symbol_type : basic_symbol<by_type>
^
bar.tab.hh:436:7: note: forward declaration of template entity is here
basic_symbol (basic_symbol&& that);
^
bar.tab.hh:550:12: note: add an explicit instantiation declaration to suppress this warning if 'yy::parser::basic_symbol<yy::parser::by_type>::basic_symbol' is explicitly instantiated
in another translation unit
struct symbol_type : basic_symbol<by_type>
^
1 warning generated.
* data/skeletons/c++.m4 (b4_public_types_define): Move the
implementation of the basic_symbol move-ctor to...
(b4_public_types_define): here, its declaration.
* tests/headers.at (Sane headers): Use a declared token so that the
corresponding token constructor is declared. Which triggers the
aforementioned issue.
* data/skeletons/bison.m4, data/skeletons/c++.m4, data/skeletons/c.m4,
* data/skeletons/glr.cc, data/skeletons/lalr1.cc,
* data/skeletons/lalr1.d, data/skeletons/lalr1.java:
Refer to the "kind" of a symbol, not its "type", where appropriate.
Because of the insane current implementation of glr.cc, things are a
bit nasty. We will rename symbol_number_type as symbol_type_type
later, to keep this commit small.
* data/skeletons/c++.m4 (b4_declare_symbol_enum): New.
Also define YYNTOKENS to avoid type clashes when yyntokens_ was
actually defined in another enum.
Use it.
(symbol_number_type): Be an alias of symbol_type_type.
Use YYSYMBOL_YYEMPTY and the like.
Use symbol_number_type where appropriate.
(empty_symbol): Remove.
(yytranslate_): Use symbol_number_type, not token_number_type.
* data/skeletons/lalr1.cc: Use symbol_number_type where appropriate.
Adjust to the replacement of empty_symbol by YYSYMBOL_YYEMPTY.
(yy_error_token_, yy_undef_token_, yyeof_, yyntokens_): Remove.
Adjust dependencies.
* data/skeletons/glr.cc: Use symbol_number_type where appropriate.
Forward definitions of YYSYMBOL_YYEMPTY, etc. to glr.c.
* tests/headers.at: Accept YYNTOKENS and other YYSYMBOL_*.
* tests/local.at (AT_YYERROR_DEFINE(c++)): Use symbol_number_type.
Another breakage revealed by vcsn.
* data/skeletons/c++.m4 (yytranslate_): Do not hard code "yy" and
"parser", both can be changed by the user.
Actually, since we are in the parser itself, there's really no need to
qualify the type.
It is not used. And its implementation was wrong when api.token.raw
was defined, as it was still mapping to the external token numbers,
instead of the internal ones. Besides it was provided only when
api.token.constructor is defined, yet always declared.
* data/skeletons/c++.m4 (by_type::token): Remove, useless.
Reported by Frank Heckenbach.
https://lists.gnu.org/archive/html/bug-bison/2019-11/msg00016.html
The cast is needed when yytranslate_'s argument type is token_type,
i.e., when api.token.constructor is defined.
373. types.at:138: testing lalr1.cc api.value.type=variant api.token.constructor ...
======== Testing with C++ standard flags: ''
../../tests/types.at:138: bison --color=no -fno-caret -o test.cc test.y
../../tests/types.at:138: $CXX $CXXFLAGS $CPPFLAGS $LDFLAGS -o test test.cc $LIBS
stderr:
test.cc:966:16: error: result of comparison of constant 257 with
expression of type 'yy::parser::token_type'
(aka 'yy::parser::token::yytokentype') is always true
[-Werror,-Wtautological-constant-out-of-range-compare]
else if (t <= user_token_number_max_)
~ ^ ~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
It is because it is expected that when api.token.constructor is
defined, only symbol constructors will be used, that yytranslate_ then
takes a token_type. But it is wrong: we still allow literal
characters in this case, as demonstrated by test 373 for instance.
%define api.value.type variant
%define api.token.constructor
%token <std::pair<int, int>> '1' '2';
[...]
static yy::parser::symbol_type yylex ()
{
static char const input[] = "12";
int res = input[toknum++];
typedef yy::parser::symbol_type symbol;
if (res)
return symbol (res, std::make_pair (res - '0', res - '0' + 1));
else
return symbol (res);
}
So let yytranslate_ always take an int, which makes the cast truly
useless.
* data/skeletons/c++.m4, data/skeletons/lalr1.cc (yytranslate_): here.
The C++ implementation of LAC did not skip the $undefined token,
probably because it was not exposed. Expose it, and use clearer
names.
* data/skeletons/c++.m4: Don't define undef_token_ in yytranslate_,
but...
* data/skeletons/lalr1.cc (yy_undef_token_): here.
Use a more precise type to define yy_undef_token_ and yy_error_token_.
Unfortunately we move from a compile-time value defined via an enum to
a static const member. Eventually we should make it constexpr.
Make LAC implementation more alike yacc.c's one.
Reported by Frank Heckenbach.
https://lists.gnu.org/archive/html/bug-bison/2019-11/msg00016.html
* data/skeletons/c++.m4 (b4_yytranslate_define): Don't use yyeof_ as
if it had two different types.
It is used once against the input argument, which is the value
returned by yylex, which is an "external token number", typically an
int. It is also used as output type, an "internal symbol number".
It turns out that in both cases we mean "0", but let's keep yyeof_
only for the case "internal symbol number", i.e., _after_ conversion
by yytranslate.
This frees us from one cast.
This patch contains more fixes to prefer signed to unsigned
integer types, as modern tools like 'gcc -fsanitize=undefined'
can check for signed integer overflow but not unsigned overflow.
* NEWS: Document the API change.
* boostrap.conf (gnulib_modules): Add intprops.
* data/skeletons/glr.c: Include stddef.h and stdint.h,
since this skeleton can assume C99 or later.
(YYSIZEMAX): Now signed, and the minimum of SIZE_MAX and PTRDIFF_MAX.
(yybool) [!__cplusplus]: Now signed (which is how bool behaves).
(YYTRANSLATE): Avoid use of unsigned, and make the macro
safe even for values greater than UINT_MAX.
(yytnamerr, struct yyGLRState, struct yyGLRStateSet, struct yyGLRStack)
(yyaddDeferredAction, yyinitStateSet, yyinitGLRStack)
(yyexpandGLRStack, yymarkStackDeleted, yyremoveDeletes)
(yyglrShift, yyglrShiftDefer, yy_reduce_print, yydoAction)
(yyglrReduce, yysplitStack, yyreportTree, yycompressStack)
(yyprocessOneStack, yyreportSyntaxError, yyrecoverSyntaxError)
(yyparse, yy_yypstack, yypstack, yypdumpstack):
* tests/input.at (Torturing the Scanner):
Prefer ptrdiff_t to size_t.
* data/skeletons/c++.m4 (b4_yytranslate_define):
* src/AnnotationList.c (AnnotationList__computePredecessorAnnotations):
* src/AnnotationList.h (AnnotationIndex):
* src/InadequacyList.h (InadequacyListNodeCount):
* src/closure.c (closure_new):
* src/complain.c (error_message, complains, complain_indent)
(complain_args, duplicate_directive, duplicate_rule_directive):
* src/gram.c (nritems, ritem_print, grammar_dump):
* src/ielr.c (ielr_compute_ritem_sees_lookahead_set)
(ielr_item_has_lookahead, ielr_compute_annotation_lists)
(ielr_compute_lookaheads):
* src/location.c (columns, boundary_print, location_print):
* src/muscle-tab.c (muscle_percent_define_insert)
(muscle_percent_define_check_values):
* src/output.c (prepare_rules, prepare_actions):
* src/parse-gram.y (id, handle_require):
* src/reader.c (record_merge_function_type, packgram):
* src/reduce.c (nuseless_productions, nuseless_nonterminals)
(inaccessable_symbols):
* src/relation.c (relation_print):
* src/scan-code.l (variant, variant_table_size, variant_count)
(variant_add, get_at_spec, show_sub_message, show_sub_messages)
(parse_ref):
* src/scan-gram.l (<SC_ESCAPED_STRING,SC_ESCAPED_CHARACTER>)
(scan_integer, convert_ucn_to_byte, handle_syncline):
* src/scan-skel.l (at_complain):
* src/symtab.c (complain_symbol_redeclared)
(complain_semantic_type_redeclared, complain_class_redeclared)
(symbol_class_set, complain_user_token_number_redeclared):
* src/tables.c (conflict_tos, conflrow, conflict_table)
(conflict_list, save_row, pack_vector):
* tests/local.at (AT_YYLEX_DEFINE(c)):
Prefer signed to unsigned integer.
* data/skeletons/lalr1.cc (yy_lac_check_):
* tests/actions.at (_AT_CHECK_PRINTER_AND_DESTRUCTOR):
* tests/local.at (AT_YYLEX_DEFINE(c)):
Omit now-unnecessary casts.
* data/skeletons/location.cc (b4_location_define):
* doc/bison.texi (Mfcalc Lexer, C++ position, C++ location):
Prefer int to unsigned for line and column numbers.
Change example to abort explicitly on memory exhaustion,
and fix an off-by-one bug that led to undefined behavior.
* data/skeletons/stack.hh (stack::operator[]):
Also allow ptrdiff_t indexes.
(stack::pop, slice::slice, slice::operator[]):
Index arg is now ptrdiff_t, not int.
(stack::ssize): New method.
(slice::range_): Now ptrdiff_t, not int.
* data/skeletons/yacc.c (b4_state_num_type): Remove.
All uses replaced by b4_int_type.
(YY_CONVERT_INT_BEGIN, YY_CONVERT_INT_END): New macros.
(yylac, yyparse): Use them around conversions that -Wconversion
would give false alarms about. Omit unnecessary casts.
(yy_stack_print): Use int rather than unsigned, and omit
a cast that doesn’t seem to be needed here any more.
* examples/c++/variant.yy (yylex):
* examples/c++/variant-11.yy (yylex):
Omit no-longer-needed conversions to unsigned.
* src/InadequacyList.c (InadequacyList__new_conflict):
Don’t assume *node_count is unsigned.
* src/output.c (muscle_insert_unsigned_table):
Remove; no longer used.
Currently, with --no-lines, instead of "#line file line\n", we emit
"\n". Let's emit nothing.
* data/skeletons/bison.m4 (b4_syncline): Emit at end-of-line when enabled.
* data/skeletons/bison.m4, data/skeletons/c.m4, data/skeletons/glr.cc,
* data/skeletons/lalr1.cc, src/output.c: Use dnl after b4_syncline to
avoid spurious empty lines.
* tests/synclines.at (Sync Lines): Make sure that --no-lines is like
grep -v #line.
* tests/calc.at: Make sure that a rich grammar file behaves properly
with %no-lines.
Reported by Derek Clegg
http://lists.gnu.org/archive/html/bug-bison/2019-01/msg00021.html
aux/parser-internal.h:429:12: error: 'syntax_error' has no out-of-line virtual
method definitions; its vtable will be emitted in every translation unit
[-Werror,-Wweak-vtables]
struct syntax_error : std::runtime_error
To avoid this warning, we need syntax_error to have a virtual function
defined in a compilation unit. Let it be the destructor. To comply
with C++98, this dtor should be 'throw()'. Merely making YY_NOEXCEPT
be 'throw()' in C++98 triggers
errors (http://lists.gnu.org/archive/html/bug-bison/2019-01/msg00022.html),
so let's introduce YY_NOTHROW and flag only ~syntax_error with it.
Also, since we now have an explicit dtor, we need to provide an copy
ctor.
* configure.ac (warn_cxx): Add -Wweak-vtables.
* data/skeletons/c++.m4 (YY_NOTHROW): New.
(syntax_error): Declare the dtor, and define the copy ctor.
* data/skeletons/glr.cc, data/skeletons/lalr1.cc (~syntax_error):
Define.
Reported by Derek Clegg.
http://lists.gnu.org/archive/html/bug-bison/2019-01/msg00006.html
Clang does not like this:
template <typename D>
struct basic_symbol : D
{
basic_symbol();
};
struct by_type {};
struct symbol_type : basic_symbol<by_type>
{
symbol_type(){}
};
It gives:
$ clang++-mp-7.0 -Wundefined-func-template foo.cc -c
foo.cc:11:3: warning: instantiation of function 'basic_symbol<by_type>::basic_symbol'
required here, but no definition is available [-Wundefined-func-template]
symbol_type(){}
^
foo.cc:4:3: note: forward declaration of template entity is here
basic_symbol();
^
foo.cc:11:3: note: add an explicit instantiation declaration to suppress this warning
if 'basic_symbol<by_type>::basic_symbol' is explicitly instantiated in
another translation unit
symbol_type(){}
^
1 warning generated.
The same applies for the basic_symbol's destructor and `clear()`.
* configure.ac (warn_cxx): Add -Wundefined-func-template.
This triggered one failure in the test suite:
* tests/headers.at (Sane headers): here, where we check that we can
compile the generated headers in other compilation units than the
parser's.
Add a variant type to make sure that basic_symbol and symbol_type are
properly generated in this case.
* data/skeletons/c++.m4 (basic_symbol): Inline the definitions of the
destructor and of `clear` in the class definition.
The previous name was historical and inconsistent.
* src/muscle-tab.c (define_directive): Use the proper value passing
syntax, based on the muscle kind.
(muscle_percent_variable_update): Use the right value passing syntax.
Migrate from parser_class_name to api.parser.class.
* data/skeletons: Migrate from parser_class_name to api.parser.class.
* doc/bison.texi (%define Summary): Document both parser_class_name
and api.parser.class.
Promote the latter over the former.
This way, it is easier to make sure its implementation is available in
glr.cc too, which is not the case currently.
* data/skeletons/c++.m4 (b4_public_types_define): Move the
implementation of syntax_error...
(b4_public_types_declare): here.
We used to create a short definition of yy::parser with all the
implementations of its member functions outside. But yy::parser is no
longer short and simple to read. Maintaining each function twice is
painful: a lot of redundancy but different indentation levels, output
which depends on whether we are in a header or not (see
d132c2d545), etc.
Let's simplify this and put the implementations into the class
definition itself.
Discussed in this monologue:
https://lists.gnu.org/archive/html/bison-patches/2018-12/msg00058.html.
* data/skeletons/c++.m4, data/skeletons/lalr1.cc,
* data/skeletons/variant.hh (b4_basic_symbol_constructor_define)
(_b4_token_constructor_declare, b4_token_constructor_declare)
Merge into...
(b4_basic_symbol_constructor_define, _b4_token_constructor_define)
(b4_token_constructor_define): these.
