`startCapture` did not initialize `lexerState->atLineStart`;
its final value is a consequence of the separate but similar
behaviors within `lexer_CaptureRept` and `lexer_CaptureMacroBody`.
Fixes#881 by moving the task from the lexer to the parser.
This both alleviates the need for backtracking in the lexer,
removing what is (was) arguably a hack, and causes tokenization
boundaries to be properly respected, fixing the issue mentioned above.
Co-authored-by: Rangi <remy.oukaour+rangi42@gmail.com>
- `out_PushSection` should not set `currentSection` to NULL because
PUSHS, PUSHC, and PUSHO consistently keep the current section,
charmap, and options, even though the stack has been pushed.
- `Callback__FILE__` does not need to assert that `fileName` is not
empty because `__FILE__`'s value is quoted, and can safely be empty.
- `YY_FATAL_ERROR` and `YYLMAX` are not needed since the lexer is
not generated with flex.
Expanding empty strings is valid but pointless;
macro args already skipped doing so, now other
`beginExpansion` calls do too.
This also fixes failed interpolations (which were
already non-fatal) to continue reading characters,
not evaluate to their initial '{' character.
Bison 3.1 introduces "typed midrule values", which would write
`<captureTerminated>{ ... }` and `$$` instead of `{ ... }` and
`$<captureTerminated>[1-9]`, but rgbds supports 3.0 or even lower.
Previously a '.' could be past the truncation limit but still
cause the identifier to be marked as local, violating an
assertion in `sym_AddLocalLabel`.
Fixes#832
This shortens the lexer by 100 lines and simplifies
access to expansion contents, since it usually needs the
deepest one, not the top-level one.
Fixes#813
lexerState->expansionOfs is always either set to 0, or updated by
adding a positive quantity:
if (distance > lexerState->expansions->distance) {
lexerState->expansionOfs += distance - lexerState->expansions->distance;
...
}
so it will always be positive or zero.
Only two sites were for distances greater than 1:
a `shiftChars(2)`, trivial to just do two `shiftChar()`s;
and `shiftChars(size)` in `reportGarbageChar`, which
can be a `for` loop, and should be fixed anyway to
"avoid having to peek further than 0".
Instead of defining `LOOKUP_PRE_NEST` and `LOOKUP_POST_NEST`,
pass a variable name and a block to `lookupExpansion`; it
will assign successive looked-up expansions to the variable
and use them in the block.
The technique of using `__VA_ARGS__` to allow commas within a
block passed to a macro is not original, and should be stable.
This macro was only used twice, in `beginExpansion` and
`lexer_DumpStringExpansions`, with `getExpansionAtDistance`
already containing an inlined and slightly modified version
of `lookupExpansion` (retaining the `LOOKUP_PRE_NEST` and
`LOOKUP_POST_NEST` macros, but with both of them doing nothing).
Not using an X macro here makes the actual control flow in both
places more obvious, and I think the repeated code is acceptable
for the same reasons as the similar-but-distinct implementations
of `readString`, `appendStringLiteral`, `yylex_NORMAL`, and
`yylex_RAW`.
This decoding required high lookahead, and was not even
consistently useful (the `garbage_char` test case was not
valid UTF-8 and so did not benefit from `reportGarbageChar`).
This limits UTF-8 handling to the `STRLEN` and `STRSUB`
built-in functions, and to charmap conversion.
Macro args were already handled by `peek`, and character escapes
do not exist outside of string literals.
This only affects the error message printed when a non-whitespace
character comes after the backslash. Instead of "Illegal character
escape '%s'", it will print "Begun line continuation, but
encountered character '%s'".
