Tokenizing

In C, you use programs like lex to convert a description of the legal tokens into a tokenizer program.
Or you write a program to read the input character-by-character and run a state machine
That is not very Perl-like.
It is also not very efficient.