Subject: Re: Common Lisp programmer productivity
From: Erik Naggum <erik@naggum.net>
Date: Mon, 11 Feb 2002 09:22:19 GMT
Newsgroups: comp.lang.lisp
Message-ID: <3222408140336006@naggum.net>

* Software Scavenger
| I think most of you agree that learning Common Lisp and using it for real
| work should generally make a programmer more productive at that work.

  It depends largely on the goal of the programming task.  If the goal is
  to make something "work", you grab the tool that is already almost there
  and just whip something together to get all there, and Common Lisp is not
  "almost there" for a large set of common tasks today.  (Those who want
  Common Lisp to be "almost there" with a huge lot of stupid tasks go off
  to whine about libraries and small syntax issues and go create Arc or
  whatever, instead of seeing that being "almost there" is not an asset at
  all, because the more "almost there" you get, the fewer things are within
  reach at the same distance, and those things that are much farther away
  than others are curiously considered "unsuitable" for the language once
  it has succeeded in getting close enough for comfort for other tasks.)

  If the goal is to make something of lasting value, which does only what
  it is supposed to do and does not fail randomly, but has tightly
  controlled and predictable failure modes with graceful recovery, then
  most other languages and tools are so lacking in support for those
  problems they were not specifically created to handle well, that _they_
  are unsuited for top quality software and can only achieve this at great
  expense.  This is where I think Common Lisp really excels.

  Those who argue that Common Lisp should be "almost there" for a lot of
  common tasks, would like people to learn to use it with simple tasks and
  then move on to more complex tasks.  I have never seen any supporting
  evidence of this transition, and much evidence of the contrary: That if
  you use some tool for the simple tasks, it becomes so much harder to
  write truly robust code and solve complex problems.  I think this has two
  important reasons that have to do with us being human beings:  (1) If we
  have too much comfort and predictabilty, abandoning the comfort zone is
  very hard to do and we lose the predictability in an area where we think
  we have sufficient expertise to predict most developments, and that means
  that if you are good at slapping things together, you will not have the
  experience to feel comfortable in writing robust code even in you "home
  language".  (2) Learning how to write robust code and solving complex
  problems is a difficult process of its own, and _habituality_ dictates
  that the languages and tools we use for such purposes should be very
  different from what we do for simple tasks, like the difference between
  cheap ball-point pen used to jot down a shopping list and the calligraphy
  equipment used on diplomas, invitations, or like the difference between a
  newspaper on recycled trash paper that yellows and crumbles in three
  months and a book on acid-free virgin paper that is expected to last.  I
  think we do not use the same tools, languages, and equipment for that
  which we want to last and that which we want only in the short term for
  accidental reasons: I think they communicate intent simply in that
  choice: Using a calligraphic pen and expensive ink to sign a check in
  your grocery store would be way too weird.  For Common Lisp, this means
  that writing small programs that solve common problems is mere "training"
  for the real thing, getting used to the language, experimenting, etc, and
  that is all well and good, but it must be recognized to be "training",
  not the end result and purpose, like it is with crud like Perl.

  So while it is crucially important to be fluent in Common Lisp, it must
  have the purpose of being used in programming tasks that are more complex
  and more difficult than what you can do with "almost there" tools.
  Looking back at my varied carreer, I see that I have always been drawn to
  these kinds of things: Ada, SGML, Common Lisp; I recognize that I am not
  a "tinkerer" who is satisfied with all the one-liners that stood between
  me and some "really" interesting problem, I took the problems I faced
  seriously enough to commit myself to solving them for good.  This is why
  I love Common Lisp so much -- it lets me take problems seriously and
  solve them seriously.  (As for the Open Source element here, I am quite
  happy about "sharing" bits and pieces, but I invest too much in what I
  really want to do to want to give it away, _especially_ to "tinkerers".)

| But the big questions are how much and how fast.

  In my view, the big question is how.  Common Lisp can make programmers
  productive in a very different way than, say, Java can, because when
  faced with the same superficial problem, Common Lisp programmers see a
  different problem than Java programmers, and therefore very different
  solutions.  I think Common Lisp programmers think more of creating a
  system in which the problem is solved, while Java programmers think more
  of fitting the problem into the system of solutions provided by their
  language.  This means that a Common Lisp programmer would generally spend
  much more time thinking about the problem and its solution than a Java
  programmer, who applies his restricted "solution world" to the problem
  instead of applying his intelligence and knowledge of the real world to
  the problem.  In that thinking phase, a Common Lisp programmer is much
  more likely to make serious mistakes, go down dead ends and bakctrace,
  etc, than a Java programmer would.  In the end, I think Java is so hard
  to program in that a Java programmer and a Common Lisp programmer will
  emerge with a solution at the same time, but the Common Lisp solution
  will provide the means to solve the next hundred problems in the system
  with very little effort, while the Java solution will require repeating
  the same development process for ever one of those problems, and may find
  that they made serious mistakes and went down dead ends _much_ later than
  the Common Lisp solution did, leading to full reimplementations in the
  face of failure to predict future demands, while the Common Lisp solution
  would have been much better thought out and have prepared for those
  future changes.

  In summary, the first solution in Common Lisp and Java will probably be
  completed with differences in implementation costs that drown in the
  statistical noise, but the value of Common Lisp will show over time, with
  te changes that are required to keep it working well, with changing ideas
  about what the system should "really" be solving, etc, because the design
  has been made with an eye to systems-building.

| If you spend several years gaining CL experience and building a personal
| library of CL stuff, what are the productivity results likely to be over
| those several years, and how fast are those results likely to be
| changing?  And how big a factor is your personal library, e.g. your
| functions and macros which you reuse in different applications?

  I think each area in which you work requires its own libraries of such
  stuff, and that the "almost there" property of, say, Java, is that it is
  such a large common library of stuff that its immediate appeal for those
  tasks is quite strong, but every business that bases its operation on a
  large system of solutions, will bring much more to the programming task
  that is unique to that business than it can accept from the outside,
  anyway, so there is a certain balance that can be struck here.

  I think Java has some really huge advantages over Common Lisp that it
  would be completely futile to fight, but Common Lisp also has some
  amazing differences from other languages that Java in particular has no
  chance ever of adopting.  The question, then, is whether you model your
  programming tasks according to what is easy to do in Java, or whether you
  focus on your business and user needs and develop a system of your own.

  Please note that the preceding supports, e.g., Franz's pricing model.

  However, when it comes to building a large software system today, we have
  a serious problem: (1) You cannot build a large system from small pieces
  without getting into a huge logistical problem.  (2) People only know
  small pieces, and have experience only with small pieces (which you can
  see immediately if you try to study some of the larger Open Source
  projects).  So, (3) arguing for tackling the inherent cost of complexity
  up front is futile with today's managers and programmers because they are
  simply ignorant of complexity theory and even management of complexity.
  This means that a Common Lisp system in our time must start off as some
  silly component-based crufty glued-together crap, and while Common Lisp
  was exceptionally good for prototyping in days when billions of lines of
  code were not readily available, it is not good for prototyping today.
  That is, we must learn from building far smaller systems before we can
  embark upon the large ones, for no better reason than that we have lost
  the skills necessary to run large software projects.  (Industry reports
  seem to indicate that the rate of software project failures has been
  increasing and that this is one of the reasons for the negative trend in
  new software development projects.)  So how do we _start_ using Common
  Lisp today when the cost of smaller projects is relatively higher than
  smaller projects in "almost there" languages, when most projects are
  believed to much smaller than they actually _should_ be?

| I know these questions are too vague and general for good specific
| answers, but I would like to get some discussion of these issues and
| whatever vague hints, anecdotes, etc., might throw some light on such
| issues in different ways.

  I hope the above has been useful.

///
-- 
  In a fight against something, the fight has value, victory has none.
  In a fight for something, the fight is a loss, victory merely relief.