Subject: Re: CMUCL18d on Alpha?
From: Erik Naggum <erik@naggum.net>
Date: Fri, 19 Apr 2002 17:11:49 GMT
Newsgroups: comp.lang.lisp
Message-ID: <3228225108884071@naggum.net>

* "Pierre R. Mai" <pmai@acm.org>
| Actually, since you've got millions of floats, I think that they
| should be read directly into a specialized vector, otherwise the list
| and boxing overhead is going to be huge, and might be the reason you
| are running into the 18b heap limit.  On the x86 using a fairly
| current CMUCL (post 18c, but pre 18d), I can read 2 million random
| single floats (3 per line, making a 20MB file) in around 16-18MB of
| heap space, doubles should be double that amount (not tested), with:
| 
| (defun read-floats-slow (stream)
|   (do ((result (make-array 1000 :element-type 'single-float :adjustable t
| 			   :fill-pointer 0))
|        (float (read stream nil stream) (read stream nil stream)))
|       ((eq stream float)
|        result)
|     (vector-push-extend float result)))
| 
| Reading takes 80s for the file on an AMD K6-2/550.

  If you start off with a vector that is pretty close in size to what you
  expect, you will also avoid all the overhead of copying the vector you
  have just extended several times.  Depending on the implementation, you
  may either extend the vector a million times (Allegro CL, with a default
  extension of 20) or about 15-20 times (CMUCL doubles the vector's size),
  or something entirely different; neither CLISP nor LWL provide reasonably
  readable disassembly to understand what they are doing.  In any case, if
  you have any means of waiting until you know how many floats to allocate
  space for, you will be better off.  One way to do this is actually to
  allocate, fill, and collect some fairly large "chunks" of the total, like
  1024-element vectors.  At the end of the read phase, you know how much
  space you need, and you can write a specialized accessor that uses the
  1024-element vectors directly in a two-level vector (note the simple
  10-bit shift instead of an expensive division), or you can copy them all
  into one vector of known size.  This _should_ have a significant effect
  on the time it takes to read all the data, but my guess is that it will
  all pale in comparison to what you wil do with all this stuff in memory.

///
-- 
  In a fight against something, the fight has value, victory has none.
  In a fight for something, the fight is a loss, victory merely relief.

  Post with compassion: http://home.chello.no/~xyzzy/kitten.jpg