Fixes#881 by moving the task from the lexer to the parser.
This both alleviates the need for backtracking in the lexer,
removing what is (was) arguably a hack, and causes tokenization
boundaries to be properly respected, fixing the issue mentioned above.
Co-authored-by: Rangi <remy.oukaour+rangi42@gmail.com>
