Subject: Re: A Proper Lexer
From: (Rob Warnock)
Date: Sat, 13 Sep 2008 21:51:16 -0500
Newsgroups: comp.lang.lisp
Message-ID: <>
Paul Tarvydas  <> wrote:
| akopa wrote:
| > I was a little surprised to find no equivalent of Lex available for
| > Common Lisp.  There is but
| > it is string oriented, not stream oriented.
| There is little incentive to use Lex in lisp, because the lisp reader *is*
| a lexer (tokenizer), hence, lisp is the language least likely to have a
| Lex-like library.  I've built many small languages and just used what lisp
| gives me, e.g. if you can design it so that the little language has a space
| between each token, you don't have to write a lexer.  I suppose you could
| get fancier by employing read macros, et al.

In that regard, the OP might be interested in META-style parsers,
originally developed by D. V. Schorre circa 1962:

     Schorre, D.V. "META II: A Syntax-Oriented Compiler Writing Language".
     Proc. 19'th Nat'l. Conf. of the ACM (Aug. 1964),D1.3-1-D1.3-11.

[I used Michael Green's implementation of Schorre's META II on the
DEC PDP-10 (part of Green's ALGOL-W compiler) in 1970 to write a tiny
BLISS compiler in a weekend.]

In 1991, Henry Baker did a very loose re-interpretation (or compilation)
of the META style into CL functions, macros, & readmacros to make META-
based parsers *very* concisely embeddable within a CL propram:
    "Pragmatic Parsing in Common Lisp"

Jochen Schmidt typed in and reorganised Baker's code, and made it
available here:

Another version by Fare Rideau is linked from here:

And, finally, yet a different version of Baker's code, assembled
by Henrik Motakef, is also available as a FreeBSD "port" under
the name "cl-meta".


Rob Warnock			<>
627 26th Avenue			<URL:>
San Mateo, CA 94403		(650)572-2607