Enum Token

pub enum Token {
    Integer(i64),
    Whitespace,
    Identifier(String),
    Decimal(f64),
    Keyword(Keyword),
    Operator(Operator),
}

Represents all possible tokens that can be produced by the lexer

Variants

Integer(i64)

Integer literals (e.g. 42)

Whitespace

Whitespace characters (e.g. space, tab, newline)

Identifier(String)

Identifiers (e.g. variable or function names)

Decimal(f64)

Floating point literals (e.g. 3.14)

Keyword(Keyword)

Language keywords (e.g. while, if)

Operator(Operator)

Operators (e.g. +, !=)

Trait Implementations

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Token