Subject: Re: O'reilly subjugated to the Lisp juggenaut (well, almost ;-)
From: Erik Naggum <>
Date: 18 Jan 2004 03:17:14 +0000
Newsgroups: comp.lang.lisp
Message-ID: <>

* Frode Vatvedt Fjeld
| Could you expand on this observation?  I mean, how were they too
| successful, and how did this cause their deaths?

  Being too successful is one of life's biggest risks, but a risk that
  has received relatively little attention, primarily because the too
  successful simply die off after a very brief period, leaving few, if
  any, traces of their existence.

  What makes natural selection work is that regardless of which random
  factor constitutes the crucial advantage that allows breeding, a large
  number of incidental factors are inherited from both parents by random
  but without perfect fidelity, so even after a long chain of successful
  breeding of the advantageous factor, all sorts of incidental factors
  show variation, which means that when the conditions that made the
  advantageous factor advantageous change, there will be a large number
  of previously non-winning individuals who are suddenly better adapted
  than the previously winning individuals.  Over time, conditions always
  change, so various factors are selected for, and over sufficient time,
  a large number of advantageous factors are present in the population.

  If, however, one factor is too successful, it will continue to be the
  winning factor regardless of the variation in the other factors over
  the range of variation in the conditions, and therefore will stifle
  the development of other advantageous factors until the conditions
  change sufficiently that it no longer is the winning factor.  At this
  point, the whole population is ill prepared for the change, and may
  well perish entirely if the winning factor accidentally becomes the
  matching factor for a disease or a predator.

  For human optimization of winning factors, we have another problem:
  The more we optimize a particular solution for a particular condition,
  the most costly it will be to acquire the same optimized match for a
  changed condition, for we will not tolerate that somebody else just
  happens to be better at it while we perish.  Therefore, as conditions
  change and competition drives us to optimize, people will voluntarily
  become too successful in the sense that they resist change and work to
  maintain the advantage by presenting the necessary adaption as a cost
  that they cannot afford.

  The Lisp Machines were heavily optimized for their particular (if not
  peculiar) conditions and were effectively much more dependent on those
  conditions than less optimized solutions, which could replace parts of
  the system without incurring large development costs to regain the
  advantages.  The tight coupling between software and hardware became a
  problem when cheaper and faster hardware arrived but which would have
  required massive development effort to maintain the advantages of the
  proprietary hardware, which was, after all, developed under intense
  pressure to make the software run fast enough.

  Software developers know better than most people how destructive to
  the core design intense optimization pressure can be and how the cost
  of increase in performance rises.  We still run software that was
  designed several decades ago, and although the optimization criteria
  of modern Intel processors are vastly different from early processors,
  we find that most optimizers of Intel code still optimize for sometime
  in the early to mid-1990s.

  Optimization is generally detrimental to future success, but it is the
  only way to accomplish present success in competition with others who
  are equally interested in short-term results.  In fact, when just one
  of the competitors becomes interested in short-term results and hopes
  to profit sufficiently to offset the risk of future profits, it takes
  more guts than most people can muster to stick with marathon runners
  as others rush to support and profit from sprinters.  It doesn't take
  a genius to figure out that optimizing for short-term profit will be
  the death of long-term profitability, but people have made short-term
  decisions for decades now and they still wonder why the future is less
  bright and much less certain.

  In the Lisp Machine case, being too successful meant that they failed
  to adapt in time when the external conditions changed.  Nothing in the
  success of the Lisp Machines indicated that they were on the wrong
  track, quite the contrary, until they were eclipsed by much cheaper
  hardware that took advantage of a few of their incidental features and
  dropped the crucial features because of the cost.  Depending too much
  on their relatively few winning factors and focusing too much on their
  development made it harder for other factors to evolve properly at the
  same time, and when these other factors were suddenly advantageous in
  the market, the previous winning factors became liabilities.

  Put another way, a company that produces one excellent product has a
  much, much smaller chance of winning in the long run than one that has
  a lot of crappy products that each manages to have a minor advantage
  over its competitors.  When the crapware producer par excellence keeps
  whining about "innovation", they really mean that their advantage over
  their competition is materially insignificant and that the only way
  they can maintain an advantage at all is by competing with themselves,
  i.e., the previous version of each product.  Over time, however, this
  process necessarily produces high quality products in a large number
  of areas, but only as long as their /competitors/ are better than they
  are at every single one of them some of the time.  When they actually
  win over their competitors, as a permanent condition, they, too, will
  be too successful and will keep doing what made them successful, which
  by the very nature of life, is not what will make them successful in
  the future, for /which/ of many incidental factors turned out to be
  the winning factor under some conditions is not only unpredictable,
  but entirely random.  All you know is that /some/ of your incidental
  factors /may/ turn out to be advantageous, but once you have found one
  of them, it is time to nourish all the /other/ incidental factors, for
  that is what your present and future competition is doing.  The old
  adage that if you find something that works, you should do more of it,
  is sound for an individual in a non-competitive environment, but it is
  extremely dangerous in a competitive environment, where you won only
  because you did something that the previous winner did /not/ do.  So,
  if you keep doing what made you successful, you will be too successful
  in a very short time, and then you just vanish when a competitor gains
  ground, like the Lisp Machines or like Digital Research.

Erik Naggum | Oslo, Norway

Act from reason, and failure makes you rethink and study harder.
Act from faith, and failure makes you blame someone and push harder.