Function extract_tokens

Source
pub fn extract_tokens(input: String) -> Vec<Token>
Expand description

Extracts all tokens from the input string using the lexer

ยงPanics

Panics if two non-whitespace tokens are found without a valid separator between them.