Tokenizing

The tokenizer:

        sub tokens {
          my @tokens = 
            split m{(
                     \*\* | :=  # ** or := operator 
                    |
                     [-+*/^()=]  # some other operator
                    |
                     [A-Za-z]\w+ # Identifier
                    |
                     \d*\.\d+(?:[Ee]\d+)?  # Decimal number
                    |
                     \d+         # Integer
                    )}x, shift();
          grep /\S/, @tokens;
        }

Easy to understand and to change, efficient, predictable.
Behaves very much like similar lex-generated parsers