Subject: Re: lists of lists (newbie)
From: (Rob Warnock)
Date: Sun, 21 Aug 2005 23:10:34 -0500
Newsgroups: comp.lang.lisp
Message-ID: <>
Joe Marshall  <> wrote:
| As it turned out, a list of strings wasn't the best representation
| anyway.  A single string was better.  Since this was a bottleneck, it
| was worthwhile to do a serious hack.  Using the FFI, I called mmap to
| map the file into the address space, then I faked up an array header
| so it looked like a lisp object and returned a displaced array to it.
| (The GC was ok with this, but it really depended on the details of how
| the GC was coded.)  This was six orders of magnitude faster than the
| original program.

In some implementations [e.g., CMUCL], READ-SEQUENCE will call the
underlying operating system's "read()" routine more-or-less directly,
which is why I use the following for this kind of thing:

    (defun file-string (path)
      "Sucks up an entire file from PATH into a freshly-allocated string,
      returning two values: the string and the number of bytes read."
      (with-open-file (s path)
	(let* ((len (file-length s))
	       (data (make-string len)))
	  (values data (read-sequence data s)))))

It's... uh... FAST:

    > (defvar *data* nil)	; avoid printing many megabytes

    > (time (multiple-value-bind (data len)
		(file-string "")
	      (setf *data* data)
    ; Compiling LAMBDA NIL: 
    ; Compiling Top-Level Form: 
    ; Evaluation took:
    ;   0.05 seconds of real time
    ;   0.00165 seconds of user run time
    ;   0.047575 seconds of system run time
    ;   101,130,862 CPU cycles
    ;   0 page faults and
    ;   11,638,960 bytes consed.
    > (subseq *data* 0 100)	; prove that it worked

    "From  Sun Jan  2 01:02:57 2005
    Return-Path: <>
    X-Original-To: lis"


Rob Warnock			<>
627 26th Avenue			<URL:>
San Mateo, CA 94403		(650)572-2607