Build some menus and the next thing you know, you're supporting them for the rest of your life...

Feature request, huh? Well, that didn't take long...was using the new menubar/menus scheme with Ed Vecto commands, inserting Vecto commands into a lisp function when I decided to extend the Ed Vecto app to add my Vecto extensions and customizations.

Ok, so now the Hemlock apps menus would need to be open so another app could append a menu to a menubar. Not a problem. (Could be more than append. I'm taking a kiss approach to coding these days, so I'll let the interactions define the scope.)

Before adding the code to modify a defmenubar, I built some defmenu/definsert forms for my vecto libs. Got bored walking my code a line at a time, copying and pasting function calls and their args. So I added definsert to the set of menus and made inserting definsert calls easier. But it was still tedious.

That got me thinking about how nice it would be if there was a menu-builder app that would parse my lisp forms for the files in an asdf package. The app would, after parsing, make the forms available (with args) so I could just pick and choose which lisp forms to make available to add to a menubar or menu or associate with a function (eg. definsert.)

I'll do some sketching on the train today and see how much work that'd be to build.   #July 22, 2008 04:36 PM

I guess everyone has heard of "the editor wars" (usually simplified to "emacs vs. vi(m)".

So have I. I have one editor I use almost daily, but there are three editors I can actuall use (well, OK, two of them are more "editor families" than distinct editors).

First off, I self-identify as an emacs user. I've used one or another emacs as my primary editing environment for the last 20 years. A new editor would have to be pretty darn astoundingly good for me to change that.

However, if you don't have a favourite editor already, I suggest you try out several different editors. Use them for at least 3-4 weeks and spend a day or two coming to grips with your editing environment. When it comes to editing comfort, first impressions can actually lure you into ditching something that is good and instead make you choose something that is less good. After you become sufficiently used to your editing environment it will be hard to change.

So, why did I end up with emacs-like editors (one of the "editor families")? Primarily thanks to the fact that emacs had a built-in tutorial. Secondarily becaiuse of emacs lisp. I've always liked lisp and lisp-like languages (something that may be obvious, looking back through my Advogato diary entries).

I said I was proficient with three different editors, I better explain taht too. I can (and do) use vi (and vi- heritage) editor(s) (primarily nvi and vim, with a distinct preference of nvi over vim), because I used to earn my living by being a unix sysadmin, occasionally being sent out to clients. You can't really expect to be able to install and compile emacs whenever you hit a new site, it just wastes time. So, learn to use the tools available.

I can also use ed. Admittedly, I mostly use ed as part of shell scripting, but that's because it can trivially be driven via stdin (build edit command using echo, pipe these into ed filename and make sure to finish with wq, stick this in a loop and you're set to do things in sequence, with externally kept state; not pretty, but it does the job).

Somewhat amusingly, I actually used vi before I used emacs, but never really continued with vi (and from there derived editors). I don't know if I'd kept with vi, had there been a good in-editor tutorial (or at least a tutorial docuiment I dcould've played around with). It felt like a struggle, initially. I had some rudimentary ed when I started with vi, but the latter was sufficiently different to cause impedance mismatch between what I knew and the capabilities available in the new environment.

Emacs, on the other hand, was nothing like ed, came with in- editor tutorials, in-editor documentation and a lisp environment to boot. I was willing to invest the necessary time to becxome proficient and have, as they say, never looked back really hard.

Clearly, I have a favourite editor, but I don't think there is such a thing as "one true editor, best for all possible things". I guess the fact that my .emacs contains (setq wq "Emacs, not vi!") to be sufficient proof of that...   #July 22, 2008 02:51 PM

Yesterday, while writing up a function, I used a combination of anaphoric macro and menu selection. A short time ago, I'd changed this same idiom from this:

(let ((res (select-menu items)))
  (when res
     (menu-item-dispatch res)))

To this:

(awhen (select-menu items)
  (menu-item-dispatch it))

And I looked at the anaphoric form and said, that's not (awhen (select-menu ...)) that's (amenu ...)

(defmacro amenu (items &body; body)
  `(awhen (select-menu ,items)
     ,@body))

(amenu items
   (menu-item-dispatch it))

That's better.   #July 22, 2008 01:38 PM

It took more time to write the last blog entry than it did to make the menus work. Added all the Ed Vecto menus and tested them. All forms insert fine - with a couple of exceptions; the forms that have keywords or body parameters aren't parsed yet. Going to add my Vecto extensions and charting calls.

Added some lisp forms and ran into limitations. The menubar/menu/insert forms work for simple, straightforward data entry. More complex forms don't work.

For some Lisp forms, eg. defclass, it'll be necessary to construct a dialog interface that provides the means to grow forms when there can be multiple entries. Or, for more complex structures like loop, the interface will have to accommodate the myriad options.

Already have an app/interface (data entry code) for Lisp forms. The app simply makes it easy to select a lisp function/macro call for insertion at the point without args. While it speeds up data entry, only a few forms have the extra juice (custom functions with interface code) to add more structure to a call. Will reconcile the new definsert and defmenu's with Lisp forms and, where needed, extend the interface to make inserting complex forms easy.

Won't update all the Lisp forms right away. Will bring over those that I frequently use and gradually bring the rest in over time.   #July 22, 2008 05:34 AM

hmmm. It's a bit dusty in here.

After being poked by a few people to revive this thing, I thought I'd give it a shot. For various uninteresting reasons, the original purpose (of this blog) was not realized.

Anyway, I think I'll use to do a couple of things. One is as a place to put/discuss snippits of code, etc. that people ask me for. Another to discuss issues I've run into, and lastly, if I can find the time for a couple of lispy projects that are deeply on the back burner at the moment, I'll probably discuss them here. Wish I had more time for messing about with lisp at the moment, but such is life.

I'll try and keep it at least somewhat lisp-related, but I've also got a couple of (simple) maths topics I keep getting asked about, so may well put primers here, with code, if I can figure out a decent way to put some maths symbols in blogger (anyone know a good approach)....


Ok, so just to have some lisp content, something I was asked for the other day (and next time I'll just link here): Here is a rejection sampling algorithm for generating Gaussian samples. There are faster methods, but speed is fine on sbcl with the given declares (some of which may be unneeded, I wrote this ages ago). Should work fine anywhere, but good output is dependent a decent implementation of random. I won't bother describing the algorithm just now. Output is easily mapped to any other Gaussian you might want, this basic function doesn't do that for you.

(defun randn (&optional; (state *random-state*))
  "returns two double-floats sampled from zero-mean, variance 1.0 
   Gaussian distribution using polar Box-Muller method"
  (declare (optimize speed))
  (loop with w of-type (double-float 0.0d0)
     for x1 of-type (double-float 0.0d0) = (- (* 2 (random 1.0d0 state)) 1)
     for x2 of-type (double-float 0.0d0) = (- (* 2 (random 1.0d0 state)) 1)
     for s  of-type (double-float 0.0d0) = (+ (* x1 x1) (* x2 x2))
     while (>= s 1)
     finally 
       (setf w (sqrt (the (double-float 0.0d0) (/ (* -2 (log s)) s))))
       (return (values (* x1 w) (* x2 w)))))

Often you want an array of these, so you can easily mimic a matlab-like randn function and take advantage of the need to sample two at a time:

(defun randn-array (dims &optional; (state *random-state*)) 
  (declare (optimize speed))
  (loop with res = (make-array dims :element-type 'double-float)
     with bound = (- (array-total-size res) 1) 
     for n below bound
     do (multiple-value-bind (a b) (randn)
          (setf (row-major-aref res n) a (row-major-aref res (incf n)) b))
     finally 
       (when (evenp bound)
         (setf (row-major-aref res n) (randn)))
       (return res)))

With sbcl this takes about 1/3 of a second to generate a million element array, so I've never bothered tweaking it further. I haven't done any careful analysis of the output, but it inherits from the Mersenne-twister a decent underlying distribution. Here's the sort of thing you get plotting it against itself, pretty reasonable:



[update: as evan points out in comments, you probably want an optional state argument to this. My fault for typing this up quickly this morning; the version I actually use is part of a package with some dependencies, so I cleaned it up --- a bit too much! Updated now]   #July 21, 2008 05:39 PM

In my last post, I documented some interface code that I was playing with for Hemlock, the Common Lisp Editor. I'd written two macros, 'definsert and 'defchoice for the purpose of creating functions that provided a means to quickly insert lisp forms at the point while making it possible to present custom options for selection on a per form basis.

Friday night, I had a little time to play with the code and thought it was a good first-pass. I didn't get much time to code this weekend, but I kept thinking about what I'd written.

Saturday, I gave some thought to creating a set of menu items for the Hemlock Vecto editor/utility (I'm calling Ed Vecto.) It was easy to add code to definsert to construct a menu. But, the menu was just a blob that screamed for better structure (categories.)

On top of the weak menu structure, defchoice and definsert were disjoint and defchoice wouldn't work if more than one slot in a definsert needed customization. Better to use the same approach as defstruct or defclass and create a set of slots with customizable behavior, eg. initform/initarg forms. That way, defchoice could be eliminated.

Want the means to make a menu selection and dispatch to a function - in this case to perform data collection and insertion of a form.

While Hemlock apps had menus, the structure wasn't well defined - simply trees. Now was as good a time as any to update the menu structure. Created defmenubar and defmenu. Defmenubar defines the menubar categories. Defmenu maps categories to subcategories.

The basic structure for menus and insert commands looks like:

(defmenubar canvas graphics-state paths text) ; Vecto categories.
(defmenu canvas with-canvas clear-canvas save-png save-png-file) ; Vecto canvas funcs.

(definsert with-canvas (&key width height) &body body) ; this form isn't parsed yet.
(definsert clear-canvas)
(definsert save-png file)
(definsert save-png-stream stream)

(defmenu graphics-state set-line-cap set-fill ...) ; more options available.
(definsert set-line-cap (style :initform '(:miter :bevel :round)))
(definsert set-fill (color :initform choose-color))

Not sure about whether the keyword :initform will be used, but it's a reasonable placeholder for this article.

Will have to provide a mapping from defmenu slots to the definsert forms - probably use a keyword in defmenu (since defmenu creates the two elements of a menu the name and dispatch function and definsert creates the function I want to call when a menu is selected in Ed Vecto.) Adding a keyword option will make defmenu look like:

(defmenu graphics-state (:prefix 'insert) set-line-cap set-fill)

The new additions will make Hemlock apps cleaner and eliminate the trees I've been relying on for menus. Wanted to address Hemlock app menus for a while now, so it's good to take a shot at cleaning them up. Hopefully, this'll be a decent approach.

Will be upgrading Vecto to the latest revision (1.3.1) and adding the new calls to Ed Vecto (and playing with the new features - gradients and text-paths.) After testing Ed Vecto, will get back to the charting app.   #July 21, 2008 04:23 AM

I started a new job a little over a week ago wherein I have to write Java code, mostly writing tests in JUnit.  I will state up front that I do not have anything against Java.  I think it's a decently designed language considering when it was designed and its target audience.  I should also say that I'm impressed with Eclipse. I doubt my experience with Java would be as enjoyable without it. Kudos to the Eclipse team. That said, I sorely miss the qualities of Lisp when trying to do my work.

Now, I haven't done extensive work with Java yet since I am still getting up to speed on many things at my new place of employment.  Thus, it may be the case I don't know how to do certain things effectively in Java/JUnit.  Still, here are some things I really miss in Lisp when moving to Java.

Special variables: Debugging in the system I'm using involves changing various static constants in classes.  This is fine if you have write access to things, but when you don't or when obtaining it may cause conflicts with the version control software, it's a pain.  I do not like that it is a compile-time constant.  (let ((*debug* t)) ...) would be so much nicer.

CLOS and its MOP: You don't realize the power of generic functions until you don't have them.  Some of the code I'm working on is old and contains, shall we say, questionable design decisions.  (To be fair, the old code would never pass current code reviews, but it's now production code, so we can't change it.)  Testing it is problematic because I have to extend existing (mock!) classes to get at the objects I need to examine at the right time.  Adding an around method to a particular method would be much cleaner.  Even better, adjusting the methods programatically while debugging would be great.

Multiple inheritance: The desire to use multiple inheritance in Java may pass as I get more accustomed to Java, but certain common methods would be better served if implemented as a mixin or something similar.  Right now, there's more code repetition than there should be, in my opinion.  However, it can't be factored out into separate classes very easily.

Overall, what I dislike about using Java is the inability to poke and prod a running system to learn something about how it works.  Sure, you can use a debugger with breakpoints, but when an error occurs, I can't drop down to Java code and start making calls. It's the interaction that's lovely about Lisp and Java just doesn't have it in the same way. Partial information is powerful. Interacting with a failure is powerful. Being able to navigate the stack after an exception has been thrown while the system is still loaded is powerful. They are powerful because they provide information that may otherwise be hard to obtain. Instead of tracing through code, I can interact with data. The latter is considerably more productive, in my experience.

  #July 20, 2008 01:59 PM

  #July 20, 2008 11:23 AM

In my last post, I wrote a macro 'definsert to insert a lisp form in Hemlock. At first, I used it only for Vecto commands, but now it's working for some lisp forms, too. That's a nice bonus.

Later, I was working with the Hemlock Vecto app. I wrote all the definsert's for all the Vecto commands. There were two functions that took a single keyword argument out of a set of three choices. Seemed to make sense to present the keyword choices when inserting those two commands so as not to have to type them. What to do, what to do...

Ooh, ooh! I know, write a macro that creates a function that handles a dialog-style selection mechanism such that, when an insert-foo command is evaluated, it'll first check to see if a 'menu-foo function exists. If it does, it'll call that function instead of asking the default set of questions (use default-args, new var names, or enter values.)

(defmacro defmenu (name &rest items)
 `(defun (intern (format nil "~:@(~menu-~A~) ()
    (select-menu-item ',items)))

Saweet! Now, the commands providing custom UI choices look like:

(definsert set-line-join style)
(defmenu set-line-join :miter :bevel :round)

So now, when my ui calls 'insert-set-line-join, the app finds the 'menu-set-line-join function, calls it and presents the equivalent of a set of radio buttons with the three choices '(:miter :bevel :round). Select one and the following is inserted at the point:

(set-line-join :miter)

Similar problem with colors. Only a little different because colors come in rgb and rgba. Plus, there are color tables (I currently have two - x11-rgb and another color palette from a paint company.) Soon, there will be palettes composed of complementary colors. But I digress.

Early on, while working with Vecto, I wrote the methods 'set-stroke, 'set-fill and 'to-alpha to dispatch on types '(RGB RGBA String Symbol) and wrap the set-*-rgb and set-*-rgba functions from Vecto. That way, I can pollute my code with color names as strings if I want to. [Note to self, get rid of the strings.] The idea is that colors with names can be called by said name, and colors that are one-offs can use rgb/a values:

(set-fill 'red)
(set-fill "RED")
(set-stroke 'darkslateblue)

So, the 'defmenu declaration, in the colors case, will have to call a function. The declaration will look like:

(definsert set-fill color)
(defmenu #'choose-color)

This change requires 'defmenu to check if the args variable holds a function. If it does, it's funcalled. Otherwise, the default case holds:

(defmacro defmenu (name &rest items)
 `(defun (intern (format nil "~:@(~menu-~A~) ()
    (if (functionp ,(car items))
        (funcall ,(car items))
        (select-menu-item ',items)))

The function 'choose-color will present '(rgb rgba) as UI choices. Regardless of the choice, the color table is selected, then the color name. If the color is to be converted to alpha, the alpha channel value is entered. Finally, the collected data is formatted and output at the point as either an rgb color or call to convert the rgb color to rgba:

(set-fill "RED")
(set-fill (to-alpha "RED" 0.4))

Now to build the high-level interface. While it may seem odd to provide one interface option via 'defmenu, for the way I'm working with Hemlock it's fine.   #July 19, 2008 04:03 AM

What's in a name? That which we call a rose,
By any other name would smell as sweet,

But it would suck if you had to type |rose| all the time.

It's the things you don't use often that trip you up. It's been a while since I wrote a macro that defined a function and I was scratching my head when the definition came back in bars.

(defmacro definsert (name &rest args)
`(defun ,(intern (format nil "insert-~A" ,name)) ()
   (apply #'insert-indenting ,name ',args)))

(definsert rectangle x y w h) => |insert-rectangle|

Bleah. I want to call it without the bars.

Thanks to this post on c.l.l, I was able to convert the macro to:

(defmacro definsert (name &rest args)
`(defun ,(intern (format nil "~:@(insert-~A~)" ,name)) ()
   (apply #'insert-indenting ,name ',args)))

(definsert rectangle x y w h) => INSERT-RECTANGLE

Phew. Case sensitive - hidden in the format call of the second version is a cryptic command "~:@(text~)" to upcase the entire expression. The resulting string is interned as a symbol sans bars.

The thing I love about Lisp (and c.l.l) is that 99.99% of all questions have been asked and answered over the years. It's easy to find excellent answers to everything from the basic to the arcane.   #July 18, 2008 08:03 PM

  #July 17, 2008 02:36 PM

  #July 17, 2008 09:05 AM

Well, it has been just a bit over 2 weeks since LispForum went live. We've had over 160 people register and some good discussions break out on everything from Lisp web programming to GUI toolkits for Lisp to what everybody is working on that involves Lisp.

If you haven't checked out LispForum yet, drop in and see us. You'll find a small but growing community of Lispers sharing ideas and having fun.

  #July 14, 2008 03:33 PM

Interesting advice targeted at systems research but — I think — generally applicable to Lisp and the new in general:

• Go back to thinking about and building systems. Narrowness is irrelevant; breadth is relevant: it’s the essence of system.

• Work on how systems behave and work, not just how they compare. Concentrate on interfaces and architecture, not just engineering.

• Be courageous. Try different things; experiment. Try to give a cool demo.

• Funding bodies: fund more courageously, particularly long-term projects. Universities, in turn, should explore ways to let students contribute to long-term projects.

• Measure success by ideas, not just papers and money. Make the industry want your work.

I especially like the last.

  #July 14, 2008 11:11 AM

NB1: I apologize for the late announcement and the short notice.

NB2: ITA Software, a fine employer of Lisp hackers (disclosure: I work there), is kindly purchasing a buffet to accompany our Monthly Boston Lisp Meeting. Anyone who attends is welcome to partake. We appreciate it if you let us know you're coming, and what food taboos you have, so that we can order the right amount of food. Tell us by sending email to boston-lisp-meeting-register at common-lisp.net. We won't send any acknowledgment unless requested; importantly, we'll keep your identity and address confidential and won't communicate any such information to anyone, not even to our sponsors.

*

Jay McCarthy will give a 25' talk about Cryptographic Protocol Explication and End-Point Projection.

Cryptographic protocols are useful for engineering trust in transactions. There are several languages for describing these protocols, but these tend to capture the communications from the perspective of an individual role. In contrast, traditional protocol descriptions as found in a state of nature tend to employ a whole-protocol description, resulting in an impedance mismatch.

In this talk we present two results to address this gap between human descriptions and deployable specifications. The first is an end-point projection technique that consumes an explicit whole-protocol description and generates specifications that capture the behavior of each participant role. In practice, however, many whole-protocol descriptions contain idiomatic forms of implicit specification. We therefore present our second result, a transformation that identifies and eliminates these implicit patterns, thereby preparing protocols for end-point projection.

Concretely, our tools consume protocols written in our whole-protocol language, WPPL, and generate role descriptions in the cryptographic protocol programming language, CPPL. We have formalized and established properties of the transformations using the Coq proof assistant. We have validated our transformations by applying them successfully to almost all the protocols in the SPORE repository.

This talk will be based off a paper co-authored that has been accepted at ESORICS, a joint work with Shriram Krisnamurthi.

Jay McCarthy http://jay.teammccarthy.org/ is completing his PhD at Brown University under Shriram Krisnamurthi, and will be a professor of Computer Science next year at Brigham Young University.

* *

There may or may not be a second speaker (to be announced), or some other community activity. Suggestions sought.

* * *

Please note that the meeting is taking place not at NEU (like previous times) but at MIT (like the times before that). The location is confirmed - the same room as before at MIT. Please also note that the date is next week. My apologies once again for the short notice.

The Lisp Meeting will take place on Monday July 21st at MIT, Room 34-401B.

As the numbers indicate, this is in Building 34, on the 4th floor.

MIT map: http://whereis.mit.edu/bin/map?selection=34

Google map: http://maps.google.com/maps?q=50+Vassar+St,+Cambridge,+MA+02139,+USA

Many thanks go to Alexey Radul for arranging for the room, and to MIT for welcoming us.

* * * *

The previous Boston Lisp Meeting on June 25th was a success despite only 34 participants. Those who didn't come missed two very interesting presentations, followed by great food and intense discussion.

We're always looking for more speakers. The call for speakers and all the other details are at http://fare.livejournal.com/120393.html

Please forward this information to people you think would be interested. Please accept my apologies for your receiving this message multiple times.

For more information, see our new web site boston-lisp.org. For posts related to the Boston Lisp meetings in general, follow this link: http://fare.livejournal.com/tag/boston-lisp-meeting or subscribe to our RSS feed: http://fare.livejournal.com/data/rss?tag=boston-lisp-meeting

  #July 14, 2008 02:44 AM

ringbark writes:

2. I knowingly travelled on the longest escalator in Europe, at Angel. (Pedants please explain if it isn't.)

It seems to be the longest escalator in Western Europe. But there's one in Moscow, at the Park Pobedy station, that's almost twice the length. Moscow qualifies as being in Europe, I believe.

In NOCtool-related news, the network code has been written, integrated and lightly tested. I believe Jim Prewett is working on at least one UI at the moment. Once I've finished off a couple of other things, I'll look at writing something to integrate Nagios-monitors into NOCtool's infrastructure.   #July 11, 2008 10:31 AM

  #July 11, 2008 05:58 AM

By now we’ve all heard of monkey-patching, and I propose a new variant: stealthy-patching.  This is the act of patching a currently running process with zero downtime.  This is similar to monkey patching, but doesn’t have the negative connotations.

I just connected to the Gainesville-Green app and evaluated a few forms to add intelligent handling of the enter button on search pages, along with some new parenscript for more IE compatibility.  All told I redefined 1 UCW component and 2 defmethods, all without any active users getting interrupted.

On the other side of the coin, I also compiled and uploaded a new SBCL core containing those fixes, so next time the application restarts (in a suprise reboot, for example), my patches will still be in place.

Ahhh… so much flexibility.  It’s like getting into a hot-tub after a long day of frigid C#.

  #July 11, 2008 01:10 AM

CL-USER> (defun ~ (name) (merge-pathnames name (user-homedir-pathname)))
~
CL-USER> (~ "foo")
#P"/home/hefner/foo"
CL-USER> (~"cl/foo/myhack.lisp")
#P"/home/hefner/cl/foo/myhack.lisp"
CL-USER> 

Not as nice as Zach Beane's approach, of course.   #July 9, 2008 10:04 PM

I released new versions of Salza2 and ZPNG today.

The Salza2 updates are minor tweaks the defsystem to fix build problems on a few Lisps. Thanks to Chaitanya Gupta for bringing it to my attention. Get the new Salza2 here.

ZPNG changes are more extensive. I added COPY-PNG and PNG= functions from Tobias C. Rittweiler (thanks, tcr!). I also added a new class and protocol for writing out PNG files incrementally, row-by-row, so you don't have to create or convert all your image data to a big ZPNG array to write it out to a file. Doing it row-by-row will save memory. In theory you could create a PNG with overall image data much too big to fit in memory, as long as you can fit a single row. Get the new ZPNG here.

If you're into git, you can check out both projects from git.xach.com.

Enjoy!   #July 9, 2008 07:53 PM

Every now and then I want to use leading tildes in SBCL pathnames. That is, something like this:

(probe-file "~/.emacs") =>
   #P"/home/xach/.emacs" 
OR #P"/Users/xach/.emacs"
OR #P"/usr/u/xach/.emacs"

I know about user-homedir-pathname, but it's not quite convenient for interactive REPL use.

So I put together a short file that patches SBCL internals slightly so leading tildes in pathnames go through the POSIX getpw interface to look up home directories. It's not something I'll ever use in source files, but it's pretty convenient interactively.

To use it yourself, just download tilde.lisp and add this to your ~/.sbclrc:

(load "/path/to/tilde.lisp")
(tilde:install-tilde-expander)

After that, you can use "~/foo" and "~bob/foo" in SBCL pathnames.

update Here's a git repo for tilde.lisp, with an ASDF system file too.   #July 9, 2008 05:17 PM

I recently noticed that ASDF-Install can’t install cl-yacc because the yacc system puts its definition in yacc.asd instead of cl-yacc.asd (the latter being where it’s supposed to go). The right fix for this is (obviously) to fix cl-yacc but it reminded me how it might be nice to have nicknames for systems the same way that we can have nicknames for packages. At one level, this code is all you need:

(defvar *system-nicknames* (make-hash-table :test #'equal))

(defun register-system-nickname (system-name nickname)
  (let* ((use-nickname (coerce-name nickname))
     (use-system-name (coerce-name system-name))
     (nickname-system (gethash use-nickname *system-nicknames*))
     (system (find-system use-system-name nil)))
    (when nickname-system
      (format t "~a was a nickname ~a" use-nickname nickname-system))
    (setf (gethash use-nickname *system-nicknames*) use-system-name)
    (register-system
     use-nickname (or system (list :nickname use-system-name)))))

(defun unregister-system-nickname (system-name nickname)
  (let* ((use-nickname (coerce-name nickname))
     (use-system-name (coerce-name system-name))
     (nickname-system (gethash use-nickname *system-nicknames*))
     (system (find-system use-system-name)))
    (unless nickname-system
      (error "`~a' is not a nickname for any system." nickname))
    (remhash use-nickname *system-nicknames*)
    (remhash use-nickname *defined-systems*)))

Of course, having this be either a separate macro like defsystem-nicknames would be nice and having this be allowed as part of a system definition would be even nicer.

All is not well with the world, however, because the code above lets you assign nicknames to a system before that system is loaded by asdf and this is a feature that feels essential to me (if I used this feature, I’d want to assign my nicknames in my preference files whether or not the system has been brought into Lisp.). This isn’t hard to work but it requires changes to ASDF which I don’t want to make unless I can find another reason to make them or if others can convince me that system nicknames are more than just a one off idea to fix cl-yacc! So wadayathink?

  #July 9, 2008 02:48 PM

I just attended the 5th European Lisp Workshop at ECOOP 2008, organized by Didier Verna. As Didier pointed out in his introduction at the workshop, it's the third Lisp meeting in Europe in a short period of time demonstrating that Lisp is alive and well.

The workshop started with a keynote by Mark Tarver, "Lisp for the 21st century". His basic thesis is that Lisp's problems are mostly social and outlined ten qualities work future work in Lisp needs in order to be relevant and noticable. He then described his Qi language (pronounced "Q-I", something I didn't know), which brings strong type checking to Lisp. The end of his talk focussed on what he calls "computational adequacy" (one of his ten qualities) and how Qi fails in this regard. Essentially, computational adequacy means a language having enough stuff to easily get things done that people want to do. The main argument was that Lisp lags in this respect. It wasn't as inflammatory as you might think since Mark seems genuinely concerned with Lisp.

Michael Wessel then gave a talk about descriptional logics and a way to use them in Lisp that is simpler than current approaches. I found the talk a little confusing and dense, but managed to get some clarification from Michael afterwards. He works on semantic web tools and we seem to be in agreement that multiple, specialized ontologies will probably be the norm and there need to be tools to create the ontologies, since no one wants to do it manually.

Next, Pascal Costanza and Charlotte Herzeel talked about work done by their student Leonardo Uribe with QLisp, a Lisp for the simulation of quantum computation. They went into the history of parallelism in Lisp dialects, namely CmLisp, *Lisp and Paralation Lisp, and a little on how QLisp was implemented using them. Charlotte talked a bit about adding parallelizing constructs to Lisp and why it is important. I rather liked the talk, since I considered doing work on this a couple years ago, but got onto other things instead.

After Pascal and Charlotte, I gave my talk, "Adaptive Libraries and Interactive Code Generation for Common Lisp". I showed one way to realize a library for objects representing many types and how to evaluate code using it. Since there are ambiguities, I use interaction to resolve them and remember the context of the operation. Then I generate code specializing the use of the library using the contexts so that the ambiguities are removed and just for fun, I generate the code interactively. Didier asked for a demo, which I couldn't give for a variety of reasons, the main one being that I didn't have access to my code(*). Once I submit my thesis, I'll make a demo and put the code out there. I like to think my talk went well, but I'm not the best judge of such things.

In the afternoon session, Rich Hickey presented his work on Clojure. I like what he's done with it, especially his use of interfaces. I think I'll play around with it for some of my work instead of trying to use Java directly. My short synopsis: Check out Clojure.

Lastly, Pascal gave a great talk entitled "make-method-lambda Considered Harmful". It boiled down to this: make-method-lambda is a macro-like facility the depends on runtime values, so you get unintuitive results when compiling versus interpreting — sometimes. The proposed solution was to deprecate make-method-lambda and use keyword arguments to call-method in method combination. The bonus is that you can now use closures for the lambda forms of methods programmatically.

Overall, I thoroughly enjoyed the workshop and commend Didier for his efforts. The intimacy of the workshop allowed for better conversation than I have found at larger meetings. Discussions at the breaks and in the evening were enjoyable as well. My only regret is that I couldn't stick around to attend the Dynamic Languages Symposium today; I have to head back home so I can start a new job.

As for the venue, I didn't care for Cyprus all that much. The conference website claimed it was 3 km from my hotel to the conference venue, but it's more like 5 or 6. I walked there on Monday morning and learned of the error the hard way. There was lots of garbage on the side of the road and the cab ride back from the conference to my hotel seemed excessively expensive. Drivers there seem to totally ignore the concept of a lane.  I also wasn't impressed by Larnaca airport. It wasn't awful, but it isn't making me want to come back.

I'm looking forward to ILC 2009.

(*) It's a long story, but basically, I took my wife's laptop instead of mine and didn't have time to get my environment up and running on it before leaving.

  #July 8, 2008 12:58 PM

A week ago, our UCW-based website (Gainesville-Green) got some local news coverage (see “Gainesville-Green.com Segment Aired on WCJB TV20” on the project’s blog), and we spent the day trying to prep for an increased load.

Our setup is fairly straightforward:

1. apache and mod_proxy to serve static content and get dynamic content from…
2. a lisp (sbcl) http server listening on 127.0.0.1:3xxx talking via CL-SQL to…
3. a postgresql database on a separate, more powerful server

When 5pm rolled around, we had:

• long-lived caches of some expensive queries in lisp
• lisp generating proper Cache-Control and Last-Modified headers to let our mostly static content be cached aggressively (using a new UCW dispatcher)
• some visual / usability tweaks
• cron job to automatically restart the lisp application server if it fails to respond

Over the next couple of days we added some more:

• used mod_disk_cache on Apache’s end to put a cache server directly in front of our lisp server
• reworked some entry points to work nicer with the cache
• referenced our javascript libraries from Google’s AJAX Libraries API, which will serve the big js files off their content distribution network, gzipped and minified

So, not too much optimization was done inside lisp itself besides some basic memoization of database queries.  Hans Hübner made a post on Building Resilient Web Servers last week after our sprint, but our needs were much simpler.   We have basically static content that we want built on-demand.  From what I’ve read we could probably be faster by replacing Apache with nginx or some such, but for now it meets our needs nicely.

For a few days our cron job was reporting the site as regularly down and restarting it, and we finally tracked that down to a bug in the restarting script.  Turns out the HTTP “Not Modified” response code is 304, and we were looking for something in the 2xx range, so restarting needlessly.

The burst of traffic that Friday night was the most we’ve seen on any of our lisp projects, we got over 4000 hits that day on a weaker test server.  We’re serving 500-1000 pages per day now, and most of the bandwidth is coming from google’s CDN, so our load is nice and light, averaging around 15K per request.

I LOVE how straightforward and composable all these tools are.  Now that our “stay-alive” cron job isn’t randomly killing our lisp, we should have a nice long-lived process and better uptime.

  #July 7, 2008 06:37 PM

  #July 6, 2008 10:29 AM

  #July 6, 2008 06:59 AM

One of the bigger challenges with deploying dynamic web servers is being able to cope with load peaks that happen when the URL is communicated to larger communities in a short time. I have seen several sites being slashdotted and collapse under the load, and it is common to blame the underlying server technology for crashes; being Lisp lovers, we don't want to see that happen for our sites.

Using a caching frontend to reduce backend load

Following industry practice, we have been using a caching reverse proxy in front of our Lisp based dynamic web servers from the beginning. Our choice was Squid, as I had prior experience configuring it and found it to work reliably once the configuration was in place. I was never quite happy with that choice, though, because Squid's main operation mode is forward proxying. It has gazillions of configuration options that are not needed for a reverse proxy, which made us feel that Squid would not be a perfect match for our demands.

This is not to say that Squid has hurt us in any way. It has served us well during a load peak on the create-rainforest web site, but as reverse proxying has become very common in the last few years, we found it about time to go shopping for a solution that might be a better match to our needs.

We hoped to find a frontend proxy having the following features:

• Caching - We want the frontend to cache all content that is not session dependent in order to reduce the load on our Lisp backend.
• Scalable - The frontend must be able to handle loads much larger than what we see normally in order to accomodate for peaks.
• Request queueing - Ideally, we'd like the frontend to handle all concurrency issues and send requests to the backend one after the other using a single persistent http connection, maybe with pipelining.

This set of requirements reduced the available options dramatically, and we ended up giving varnish serious consideration.

Evaluating varnish

varnish seems to have most of the features that we require: It supports caching, has been tested under very high loads and supports FreeBSD which is our deployment platform. Varnish has been written by Poul-Henning Kamp of FreeBSD fame, so we also found it to be culturally compatible.

Our evaluation revealed that varnish is under active development and support, and we found the developers be very responsive to our bug reports and requests. Its architecture looks well thought out, and the configuration language (even if underdocumented) makes the request handling process very transparent.

There are a number of downsides with varnish, though:

Eager cache cleanup policy: varnish removes objects from its cache as soon as they expire. This means that expired objects are always fetched from the backend, even if the previous cached copy was still up to date. It is possible to work around this by either hacking varnish to revalidate cached objects before expiring them or by not expiring them automatically, but rather by explicit purge requests sent from the backend to varnish. Both options, while doable, require substantial work.

No high load safeguards: varnish offers no control over the number of backend connections established and does not support any queuing or backend resource control mechanisms. This means that it will unconditionally try to establish a connection to the backend if an object can't be served from the cache. Once the backend reaches saturation, varnish will aggravate the situation by sending even more requests to the backend, increasing the time to recover from the backend saturation.

Threading: varnish uses threading to schedule requests. While this in principle should not be something negative, threaded programs are much harder to debug in practice, and many concurrency bugs that threaded code is susceptible to only show up after a long time or under certain load patterns which may be hard to reproduce. Admittedly, I am a threads hater, but I am writing this here because we found a serious threading race condition on the second day of our evaluation which prevented varnish from even starting up. My trust in the code was seriously affected by this.

When talking to the varnish developers, the problems were acknowledged, help to fixable problems was very quick and they told us that more advanced features will be develop after the upcoming 2.0 release of varnish.

We would have liked to switch to varnish because of the very good support and because it is meant to be a web frontend, nothing else. Yet, at the current point in time, it does not seem to be mature enough to serve our needs. After having evaluated it, we turned back to squid as it seemed to be the only other option. We found that squid meets our requirements for cache cleaning and revalidation of cached objects very well.

Making objects cacheable

Chosing a front end software is only part of what needs to be done to make a web system fast and robust enough to withstand high loads. The most important factor is to make the frontend serve a large percentage of the incoming requests from its cache and only consult the backend server for content that is really dynamic. The HTTP/1.1 protocol provides for request and response headers that control how content can be cached, and there is little need for explicit configuration if these headers are used correctly.

Using If-modified-since

One way to limit the traffic to the backend is implement the If-modified-since mechanism. It is supported by Hunchentoot for static files by default, and can also be used for dynamic handlers that can check whether a resource has changed since it had previously been requested. Varnish will set the If-modified-since header when it requests resources that it already has in the cache, so every cacheable resource will normally be transfered from the backend to Varnish only once.

Controlling cache refreshing

Often, resources are dynamic, yet it is not crucial that every client sees the absolutely latest version of the resource. For example, in our square meter sales application, visitors should always see the current number of square meters sold, but it is not vital that this information is accurate to the second. Thus, we want the cache to refresh such pages only at a certain interval. The HTTP/1.1 provides for the cache-control directive and in particular the max-age parameter. It specifies how long a cache may consider a cached resource be valid without revalidating with the originating server. By setting this parameter in responses sent by the backend, we can effectively limit the maximum refresh rate for dynamic resources that do not need completely up to date every time.

Testing realistically

In order to test the performance of a Web system, it needs better tools than the often-used ApacheBench tool, which only tests response times and throughput of a single URL. For meaningful results, one should simulate a user load that reflects the load that real users create. I have been using SIEGE for informal testing often, as it is easy to use, but I have also found it a little flakey and prune to random crashes, which made us look for more reliable solutions.

In the FreeBSD ports collection, we found tsung. Tsung is an open-source multi-protocol distributed load testing tool written in Erlang, and it has good support for HTTP. In addition to running load tests against web servers and generating statistics reports, it supports a session recorder that can be used to create a log of the URLs that a user visits when browsing a web server which can then be directly used to simulate many simultaneous users. As an added bonus, it is possible to capture dynamically generated information from web server responses into variables and use them in subsequents requests in a session. This feature can be used to generate sale transactions or user registrations in a load simulation.

Using tsung and squid, we were able to tune our Lisp backend so that all non-dynamic content is served from the cache, pinpoint a serious performance problem and simulate realistic loads. We are now confident that our setup can withstand the next rush of users without crashing.

  #July 3, 2008 11:04 AM

There is a new version of Allegro CL 8.1 Express available for download. Allegro CL Express is the free version of Allegro CL. Current users should just update their installed copy. The new version has a larger allowable heap size and the newlicense program now retrieves the initial license in addition to updating licenses. On Windows, the update.exe program now gets updates as well as (as before) updating existing images.   #July 1, 2008 09:45 PM

I'll be in San Francisco from July 5th to 14th. I have a feeling that a lot of online acquaintances and people with similar interests (Erlang, Lisp, Smalltalk, etc) are based there. Let's meet up! Please tell me about any fun meetings, user groups, pub nights, etc that I should try to make it to.   #June 30, 2008 09:59 PM

I pushed out Vecto 1.3.1. New in this version:

• stroke-to-paths will convert stroke outlines of the current path to a new set of paths you can fill
• string-paths will add paths for a string in the current font, instead of painting the text
• set-gradient-fill will let you fill paths with a simple axial gradient. It's not much, but it's a start...

Many thanks to Ben Deane for providing a functional fill patch that made adding gradients much easier, and for suggesting the text-to-paths functionality. Thanks also to Jakub Higersberger for showing me how easy gradients could be. (I wound up using a very simple formula straight from the Adobe PDF Reference.)

This uses a simple white-to-transparent gradient:

Here's some code that uses all three new features:

(defun text-paths (file)
  (with-canvas (:width 400 :height 100)
    (set-font (get-font "/tmp/font.ttf") 70)
    (centered-string-paths 200 15 "Hello, world!")
    (set-line-join :round)
    (set-line-cap :round)
    (set-line-width 3)
    (set-dash-pattern #(0 5) 0)
    (stroke-to-paths)
    (set-gradient-fill 0 0   1 0 0 0.5
                       0 100 1 1 1 1)
    (fill-path)
    (save-png file)))

I fell short of exporting and documenting the functional fill option that enabled gradients, because I've run out of time. Maybe next release!   #June 30, 2008 08:36 PM