Skip to content

Commit 872132f

Browse files
committed
Rollup merge of rust-lang#25137 - carols10cents:fix-token-declaration, r=alexcrichton
I know this is the most trivial thing since it's *just* the model lexer, not the real lexer, but [it is simpler to read](rust-lang#15883 (comment)) and it'd be great if it was up to date but [it's been rotting](rust-lang#22379) and this is a tiny bit of that. Thanks!!!!!!
2 parents f59f07d + 9c7d5ae commit 872132f

File tree

2 files changed

+5
-5
lines changed

2 files changed

+5
-5
lines changed

src/grammar/RustLexer.g4

+4-4
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@ lexer grammar RustLexer;
88

99

1010
tokens {
11-
EQ, LT, LE, EQEQ, NE, GE, GT, ANDAND, OROR, NOT, TILDE, PLUT,
11+
EQ, LT, LE, EQEQ, NE, GE, GT, ANDAND, OROR, NOT, TILDE, PLUS,
1212
MINUS, STAR, SLASH, PERCENT, CARET, AND, OR, SHL, SHR, BINOP,
1313
BINOPEQ, AT, DOT, DOTDOT, DOTDOTDOT, COMMA, SEMI, COLON,
1414
MOD_SEP, RARROW, FAT_ARROW, LPAREN, RPAREN, LBRACKET, RBRACKET,
15-
LBRACE, RBRACE, POUND, DOLLAR, UNDERSCORE, LIT_CHAR,
15+
LBRACE, RBRACE, POUND, DOLLAR, UNDERSCORE, LIT_CHAR, LIT_BYTE,
1616
LIT_INTEGER, LIT_FLOAT, LIT_STR, LIT_STR_RAW, LIT_BINARY,
17-
LIT_BINARY_RAW, IDENT, LIFETIME, WHITESPACE, DOC_COMMENT,
18-
COMMENT, SHEBANG
17+
LIT_BINARY_RAW, QUESTION, IDENT, LIFETIME, WHITESPACE, DOC_COMMENT,
18+
COMMENT, SHEBANG, UTF8_BOM
1919
}
2020

2121
import xidstart , xidcontinue;

src/grammar/verify.rs

+1-1
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ fn parse_token_list(file: &str) -> HashMap<String, token::Token> {
111111
"LIT_BINARY_RAW" => token::Literal(token::BinaryRaw(Name(0), 0), None),
112112
"QUESTION" => token::Question,
113113
"SHEBANG" => token::Shebang(Name(0)),
114-
_ => continue,
114+
_ => panic!("Bad token str `{}`", val),
115115
};
116116

117117
res.insert(num.to_string(), tok);

0 commit comments

Comments
 (0)