Tokenizing

We can get rid of that grep:

        sub tokens {
            split m{(
                     \*\* | :=  # ** or := operator 
                    |
                     [-+*/^()=]  # some other operator
                    |
                     [A-Za-z]\w+ # Identifier
                    |
                     \d*\.\d+(?:[Ee]\d+)?  # Decimal number
                    |
                     \d+         # Integer
                    )
                    |
                    \s+
                   }x, shift();
        }

(Thanks to Andy Wardley.)