I don't have much experience with the sort of problem you are
describing, but I thought of a couple things that might be helpful.

> I was able to get all of the output by calling (sleep 1) before checking
> the stream, but in my application, waiting a second for this is often
> more time than it takes to process a unit of work. I guess that some other
> function that forces a unix context switch probably would do the trick.

CLtL2 says that SLEEP will take "any non-negative, non-complex
number", ie, 0.001 might very well work for you.

READ-CHAR-NO-HANG may be useful to you.  Here's an example:

(defun get-waiting-stream-string (output-stream
				  &optional (eof-error-p nil)
					    (eof-val nil))
  "returns all waiting characters on the OUTPUT-STREAM and returns them
as a string.  If encounters EOF and EOF-ERROR-P is non-nil, returns
EOF-VAL (defaults to NIL)."
  (let ((output-string (make-array 128 :element-type 'character
				   :adjustable t :fill-pointer 0)))
    (loop
      (let ((new-ch (read-char-no-hang output-stream nil t)))
	(case new-ch
	  ;; nothing waiting
	  ((nil) (return output-string))
	  ;; eof
	  ((t) (if eof-error-p
		   (return eof-val)
		 (return output-string)))
	  ;; anything else
	  (t (vector-push-extend new-ch output-string
				 (length output-string))))))))

> It would also be nice to be able to get the return code of the subprocess!

OS-WAIT should give you this information.  Check the manual.  Eg,

#+ALLEGRO
(defun wait-for-p-id (p-id)
  "waits for the process named by P-ID to die and returns its exit status"
  (loop
    (multiple-value-bind
	(status dead-p-id) (sys:os-wait nil p-id)
      (when (and (numberp dead-p-id)
		 (= dead-p-id p-id))
	(return status)))))


Best of luck,

Wheeler
-- 
Wheeler Ruml, Eng Sci Lab Rm 420, <eecs.harvard.edu, at ruml> (617) 495-2081 (voice)
http://www.eecs.harvard.edu/~ruml/                       (617) 496-1066 (fax)