Subject: Re: Lisp's future
From: Erik Naggum <erik@naggum.no>
Date: 31 Jan 2004 01:48:29 +0000
Newsgroups: comp.lang.lisp
Message-ID: <2004-031-ATW-KL2065E@naggum.no>

* Erann Gat
| What this debate is really about, it seems to me, is no different
| from the old deliberative vs. reactive debate that took place in AI
| robotics during the 80s.

  This debate is really about whether the act of programming produces or
  consumes information.  Of course it does both.

  This odd discussion is similar to the age-old discussion about how to
  write an essay.  Some people believe that you should think carefully
  about everything, construct a skeleton and a plan and proceed to flesh
  it out, in the belief that you should have all the information you
  need to write the essay up front and that the writing itself should
  not produce more information, indeed that thinking of something you
  had not thought of to begin with is some sort of failure condition,
  like lack of planning or lack of concentration on the task at hand.
  Other people believe that formulating an idea in words /creates/ the
  idea and that you should only define a purpose, a goal condition, and
  then proceed to write your ideas down while monitoring that you are on
  the right course.  (Monitoring is how consciousness works to begin
  with and with some training becomes automatic and emotional: you feel
  good when you go in the right direction and bad when you go in the
  wrong direction.  People who plan compulsively are usually the kind
  who think that emotions have no business in directing attention or
  focusing effort.)  Creative people also argue that premature planning
  solidifies unformed ideas that should be expressed fully if they are
  allowed to form at all, because almost all ideas appear good until put
  to the test, and the best test is not to think about what you would
  do, but to actually do it.  (The word «essay» itself is curious -- it
  originally meant an attempt at expressing something, but the written
  attempt has become such a standard model that the word has come to
  mean a particular style of exposition and the history of the word is
  all but forgotten.)

  Most of the arguments in favor of one or the other of these positions
  (when both are clearly necessary and correct) are made for the sole
  purpose of pushing a pendulum that has swung too far in one direction
  back in the other direction, so the context of each argument is that
  the antithetical position has become too strong, for whatever reason
  and by whatever measure.  We have not arrived at the stable position
  that is satisfactory to all parties, so we keep pushing the pendulum
  too much in our direction when we perceive that it has been pushed too
  much in the other direction by others.  In all likelihood, the stable
  position is one where different people exhibit different capacities
  and tendencies and where the belief in one answer has been replaced by
  a more pluralistic view.  In my view, the planners are not only wrong,
  but so averse to feeling bad that they prevent themselves from feeling
  good about what they do and therefore have no direct experience of the
  pleasure of thinking properly and so resort to their plans to avoid
  the pain of realizing that they do not think well.  Herbert Dreyfus
  has argued that it is extremely important to be emotionally attached
  to the success of one's work if one is to progress beyond advanced
  beginner to competent or even proficient, and point out that students
  who work very hard to avoid feeling the pain of failure never become
  competent enough to feel the joy of competence and success.  It is no
  accident that those who recoil from the pain of failure are those who
  are most likely to ask for «rules» or reach for the «answer book».

  Some people are very good at predicting what will happen and plan for
  both the likely and the unlikely, but then become nervous wrecks when
  their planning turns out to be faulty and they have to re-plan, which
  they resist until it is already too late.  Others are not so good at
  prediction and planning, but are very good at setting goals, and they
  try things out, compare it with their desired results, learn from each
  failure, accept that neither other people nor the world in general are
  obliged to do their bidding, and they never become nervous wrecks in
  the face of hardship, even disasters, but frequently set new goals as
  they find that some goals were unreachable at a reasonable cost.

  In the real world, it takes all kinds to make a good society, and the
  only track record we know from people who believe that only one kind
  of person should exist is incredibly murderous tyrannies.  A lot of
  (dysfunctional) people believe that all those who do not obey every
  rule should be punished or killed, and they spend their time blocking
  other views, with such phraseology as «unconditionally bad» about them
  while they misrepresent them in order to destroy the ability of others
  to present them in a rational discussion.  Typical behavioral patterns
  with these people include hostility towards any violation of their
  plans and rules, and when they see some person doing something they do
  not like, they believe that the proper course of action is to do what
  they do not like on purpose and in some demented idea of retaliation
  and retribution, even though they are never directly affected, instead
  of working to rectify or improve the global situation.  Breaking rules
  and upsetting plans cannot be mistakes to these people, only willful
  acts of destruction, which only severe punishment can cure, and they
  are never sufficiently humble to even consider the possibility that
  they are the first to break any rules on purpose.  People who plan too
  much are also unable to enjoy or cope intelligently with surprises,
  for any surprise is evidence of their insufficient planning.

  The important distinction between planners and creators is whether an
  action taken for a given purpose does or does not provide feedback to
  the decision machinery that determines the next action.  Listening to
  feedback is not an option if you want to retain your mental health,
  but listening to feedback will frequently mean that any plans you had
  must be discarded.  The more energy you have poured into planning, the
  less likely you are to listen to feedback that it is junk.  If you
  make up your mind that the world is different than it is, that it
  «ought» to be different, you will run into psychological problems, the
  larger the more you insist.  A lot of people believe that standing
  fast in their beliefs is the hallmark of integrity and that it is the
  duty of all people of true belief to force other people to obey their
  beliefs.  These days, we call them fundamentalists and terrorists, but
  in the world of software development, we have no such useful labels
  for those of deranged psychology and highly irrational beliefs, but
  they are nonetheless present in the industry.  The most effective test
  they can be put to is whether /all/ actions taken by programmers are
  able to provide feedback to the planning and scheduling of the whole
  project, or only /some/ actions (of the selection of which they demand
  to be the final arbiter).

  If /all/ actions taken by programmers are used for feedback, the best
  way to view the act of programming is that of producing information
  about the problem to be solved.  Some of it may be wrong, some of it
  may be of no use, some of it may be unexpectedly true and need work to
  be proven, but all of it is used as feedback.  This is obviously not
  the dichotomous relationship with planning that some people think it
  is -- if you are both intelligent and of sound mental health, you do
  not plan your actions, you plan your results and the acceptable costs,
  and you trust both your own competence and that of others in actually
  accomplishing those goals without detailed instructions.  Some of the
  time, however, an action may produce information that will affect the
  results and the costs, and it is at this point that a rational manager
  listens and an irrational manager ignores the available corrective
  feedback.  When all actions are used for feedback, the consequence of
  making mistakes will be a corrective and appropriate response, because
  everything everybody does /matters/.

  If only /some/ actions taken by a programmer are used for feedback,
  the result will necessarily be that programmers will perform actions
  that are not used for feedback with reckless abandon.  In this regime,
  the already tyrannical manager will write ever more detailed plans to
  enforce his model of reality, the stronger the more wrong it is, just
  like people who have to be taken care of by society in general because
  they have stopped accepting input from the real world.  In business,
  however, those who keep pushing for something they believe in tend to
  be remembered when they have had a string of successes, but there is
  now a growing literature on the stupidity of the highly intelligent in
  both managing their lives and their money.  (E.g., Why Smart People
  Can Be So Stupid, edited by Robert J. Sternberg, ISBN 0-300-09033-1.)
  It is quite possible for people who watch others who have succeeded by
  pushing for something they believed in to believe that it was pushing
  for something they believed in that produced the success, but in logic
  this is known as the post hoc ergo propter hoc fallacy.  In the real
  world, pushing for something you believe in generally causes massive
  and lasting failure, but if you accidentally get it right, pushing may
  magnify the success.  This is like performing an objectively insane
  action which by sheer accident happens to associate itself in the mind
  of the deranged with some random success condition that has nothing
  whatsoever to do with the action.  You could have worn the same tie
  the last three times you won a contract or did a good presentation,
  and then suddenly it becomes your «lucky tie».  Lots of people do
  this, and such things as astrology and numerology or the importance of
  winning New Hampshire because past presidents did, find a receptive
  audience in the highly unscientific population.  The more /selective/
  you are in the feedback you accept, the more insane your reasoning
  will become as you will necessarily reject corrective feedback that
  would have led to better reasoning.

  The point at which it becomes rational and productive to impose one's
  will on reality or other people is perhaps the most interesting aspect
  of the philosophy of ethics.  It is not understood to any significant
  degree how to determine this point in advance, so ethical thinking is
  best thought of as a systematic approach to respond to objections, but
  this imposes a requirement to object, and to do so rationally and only
  constructively, which is not as well-defined as it ought to be.  The
  history of modern societies provide us with the evidence that we have
  swerved aimlessly in this regard and are both unsystematic and highly
  irrational in our objections and the expression thereof, especially in
  cases where the severely misguided notions of «good» and «evil» are
  involved, and people conjure up feelings of suffering «injustice»
  instead of coping intelligently with the situation at hand.  There are
  people so devoid of rational ethical thinking that when they observe
  something they object to, they do exactly the same thing on purpose in
  order to express their objections, and then fail to understand that
  they have done something objectionable.  Human rights, for instance,
  is an expression of a fuzzy line of demarcation around an area where
  people are not allowed to impose their will on any other person for
  any reason whatsoever.  The U.S. Constitution was also thought to have
  protected certain rights even under serious threats, but turned out to
  be completely irrelevant to the American people.  Social security is
  an attempt to draw up an area where every citizen must accept the cost
  of sustaining other citizens who are unwilling or unable to sustain
  themselves, and those who object to this system tend to ignore the
  fact that the absence creates slavery and people so desperate that no
  law can bind them.  The political debates are mostly about which areas
  of public and personal life must yield to the imposition of the will
  of others.  This is far from the obvious and well-known that the mark
  of the highly irrational and tyrannical pretends it is: /Nothing/ is
  «unconditionally bad»; the will of some people who have made up their
  mind about both reality and people /must not/ be imposed or accepted
  to be imposed on others to the detriment of the ability to object both
  rationally and constructively.

  Being intelligent and reasonable human beings with the particular kind
  of brain that we possess, we are still not only fallible, but very far
  from omniscient.  We are also naturally inclined to commit a fallacy
  known as conformance bias, which makes us want to prove that what we
  already believed was true and to believe faint affirmations even in
  the presence of rejection.  Another dangerous problem with puny human
  brains is that counter-information tends to be rejected because opening
  every decision up for re-evaluation would consume infinite resources,
  and it also takes too much time and energy to re-evaluate enough to be
  able to know whether some counter-information is important or not.  For
  these reasons, we have set up the extremely inhuman methodology known
  as «science»: Figuring out that a particular causal relationship is
  for real independent of our desire for it to be is really hard work.
  Most people much prefer to believe that what they would like to be
  true is also actually true than to do all the hard work to determine
  what is and what is not actually true, not the least because most of
  what people believe is actually dramatically false.  Given what we
  know about human fallibility and society-wide myths and beliefs, it
  should be obvious that we simply cannot afford to be selective in what
  feedback we are willing to accept.  Every single relevant, corrective
  feedback we refuse to consider will come back to harm us.  So, too,
  with every rational, constructive objection to what we have done, but
  those who are least able to accept corrections are unsurprisingly also
  those who are least able to express corrections and objections in a
  way that other people can make use of, and they are the first to jump
  to conclusions about other people's motives and rush to accusations
  and all sorts of counterproductive idiocy.  Instead of thinking when
  they meet corrective feedback, they defend themselves and stick to
  their plans precisely /because/ those plans are in danger of proving
  insufficient or wrong, and this is much more likely to happen if the
  plans have been very costly to produce.

  To strike a balance between planning and creativity, we need to zoom
  out and look at the available measurements of the success we have with
  our various approaches.  To do this, we need to be aware of the goals
  we have for our work at a sufficiently abstract level.  The more we
  plan, the more concrete our goals become, however, and the danger of
  planning to avoid costly mistakes in achieving our goals is that we no
  longer remember what we were originally heading out to accomplish.

  The most productive plan, then, must be the plan that is not concrete,
  but abstract, yet has definite test criteria.  The distinction between
  a plan and a vision is therefore only in the availability of the tests
  we can employ to know if we have gotten where we wanted to get.  This
  is the kind of plan that is incommunicable to computers at present, so
  we are in no danger of confusing a program with its intention, just as
  we are never in danger of confusing a test harness with the objective
  of the whole endeavor.

  The notion that planning at the detail level is a necessary product of
  good planning at the abstract goal level is itself a product of the
  kind of personality disorder that prevent creative people and experts
  from producing the desirable results.  This kind of planning is only a
  good thing if you plan to hire monkeys who respond well to electric
  shocks when they deviate from their instructions.  Human beings are
  not computers or physical devices and planning for intelligent people
  is different from planning for morons.  Most planners also believe
  that they are vastly more intelligent than the people who are going to
  implement the plan in practice.  More often than not, however, they
  are not noticeably more intelligent and the belief that one person can
  plan what another person should do is a very good way of making the
  person who plans unable to accept negative feedback from the stupider
  person who implements it and the person who performs the planned steps
  is unlikely to invoke their intelligence when the plans are bad, so
  excessive planning is a very good way of destroying productivity and
  /avoiding/ the abstract goals of an enterprise entirely.

  However, we must take the planners' word for it -- they could never
  employ their creativity and intelligence in the face of unexpected
  situations and their fear of failure is probably well-founded, so if a
  person demands detailed plans, we should feel a profound pity for
  their damaged psychology and inferior brainpower and help them write
  their plans and ensure that they measure only irrelevant properties of
  the system being developed so they are kept busy without messing up
  for anyone who have do to deal work.  In this regard, we should go
  talk to the social science people, because they educate a lot of the
  bureaucrats who prevent politicians from doing more damage than they
  absolutely have to with their misguided micro-controlled plans.

-- 
Erik Naggum | Oslo, Norway                                      2004-031

Act from reason, and failure makes you rethink and study harder.
Act from faith, and failure makes you blame someone and push harder.