Subject: Re: Why I can't use Lisp. From: Erik Naggum <firstname.lastname@example.org> Date: 03 Aug 2002 04:11:51 +0000 Newsgroups: comp.lang.lisp Message-ID: <email@example.com> The most important, if not the only, difference between the Common Lisp world and that of most other languages is that it is /perceived/ to be impossible to get anything done with the free Common Lisp systems; you cannot even /learn/ Common LIsp using the free Common Lisp systems; not even the Trial Edition of Allegro CL or the Personal Edition of LispWorks will satisfy the "newbie" who wants to use Common Lisp. I have no idea where this /perception/ comes from or how it originated, but it is apparently deeply ingrained in people who know nothing about Common Lisp and refuse to learn or listen. The amazingly stupid people who remain newbies for their entires lives seem to be upset about something related to commercial systems, but I actually wonder what it might be. I think it /might/ be envy. Do you /need/ the power of the expensive commercial systems to use Common Lisp? No. What is it that people /miss/ when they claim that they cannot use Common Lisp for their applications while they /could/ use just about any other language? I actually do not know, nor do I know where to start to understand this issue. Allow me to illustrate why I fail to understand this issue with a past project of mine. I developed a system for real-time distribution of financial news for a wire service using Common Lisp that depended on multiprocessing support in the Common Lisp environment, in particular the ability to listen for activity on more than one socket. We chose a commercial Common Lisp. The reasons why we made this choice should be illumiating. The real-time nature of the application means that it was not CPU-intensive, and thus did not need preemptive multitasking between the processes. What made it so convenient to use multi-threading in the Common Lisp environment was the shared memory. It would have been possible to use the file system to share information between processes and to use a control connection to each process and then to implement a broker for messages between the processes. It would be a different design that might have required more time to implement, but it would most certainly be doable. The application would need to be implemented as an event dispatcher in each process, and it would have to be taught how to listen for activity on several sockets (as in the select system call) if the Common Lisp system did not already know how to do that. I estimate that this complexity would have meant an additional month of development time, but it would also have meant that we would have needed more powerful hardware, since the memory requirements would be much higher and the processing power to deal with the interprocess communication (IPC) would be significantly higher. However, my system was to replace an overly complex system that basically did a tremendous amount of IPC and which had serious problems and shortcomings because of the constraints of the IPC methodology used, such as starting and stopping parts of the system independently. (You would think not, but it was written by an idiot.) It would have been a hard sell if I had argued that I would replace the existing mess with another mess of approximately the same magnitude. So what I sold the project on was in fact that I could make all this a lot simpler in Common Lisp and thus remove the manual monitoring required by the frequently failing system. In this regard, it was indeed the benefits of the commercial Common Lisp environment that made the project. This was a highly commercial project, with high risk and high benefits. It had to succeed in a hostile environment,I would have to bring all the clients of the news wire on board with a new communication protocol to solve major problems in the previous design, and I needed all the confidence that money could buy -- in this case, it was for sale in the form of a commercial Common Lisp system. The commercial Common Lisp development environment was dwarfed the other costs involved. (We were much less happy with the "runtime" licensing, but the time and money I spent on that was still worth the cost compared to reimplementing it with a lesser environment, again mostly because of the confidence required in the multi-threading support.) /However/, had this /not/ been a project to salvage a wire service that would in all likelihood have faced survival issues if it had continued to crash and involve lots of manpower to keep it running, and I had been hired to do the original project, instead, I could easily have taken a chance on a "free" environment first, and in case of problems or costs or failure, have purchased "safety" in a supported commercial environment. I included this fairly long exposition of this project to show that I chose a commercial Common Lisp environment with commercial values in mind. I would have to implement some of the things I would have gotten with that commercial environment if I used a free environment, and I would have to make myself an expert in the performance of IPC and socket communication and streams in Common Lisp, and many other internal details. I have become that expert over time, but I was not ready to do it at that time. So another commercial value is development and testing time before deployment. I wager that these are /not/ issues for the free environments, /nor/ of their usability. If you /can/ afford to develop this for yourself, you should do it. If you can /neither/ afford to develop it yourself /nor/ purchase it from someone who has, you had better realize that you should not take on the project. What /is/ missing in the perception of the immature and inexperienced Common Lisp programmers is that /even/ if you had to do the internal stuff you would have to do in any other programming language and /even/ if this non-portable and implementation-dependent, you are /less/ system-dependent and /less/ forced to work at the internal /level/. This is hard to empathize enough to those who have made up their mind that they should get everything for free before they can even /start/ to think about using Common Lisp, but Common Lisp offers so much more convenience in the higher-level logic of the application and so much better ways to think about the system-dependency issues and the native operating system support that it is not worth the effort for anyone to sit down and "standardize" the missing features. It is not the same level of pain and suffering it is in other languages. Too many programmers would want a different take on the too many operating systems that might offer them that it would not make sense to "standardize" a huge number of features. It would, in fact, be detrimental to many development efforts to have to rely on many software packages from independent sources. The reason many of the other programming languages do all these things is that they are single-vendor and single-implementation and can very easily merge independent projects into the common fold and people are willing to hand over copyrights and ownership to this single vendor. This is utterly infeasible in the multi-vendor, multi- implementation world of Common Lisp where some people actually make money from supplying just these kinds of tools. In fact, many programmers are annoyed that a vendor has chosen to join the "open source" community to undermine their own customers' ability to make money on similar efforts, and because the supposed "open source" is so heavily dependent on internal coding practices. It may be difficult for outsiders and newcomers to understand this, but the commercial /success/ of Common Lisp has made it less appealing for people to give stuff away. It is /because/ Common Lisp is so eminently able to make money for those who use it that programmers and managers are reluctant to give away the genius and hard work behind the kind of quality and solid design that we are used to in the Common Lisp world. It is because of the high level of pain and suffering to do relatively simple things that people share the burden in other languages -- the comparable implementation in Common Lisp is far simpler and more elegant. On the other hand, Common Lisp programmers are finicky lot and woudl most probably want a different take on the same features from the operating system or other libraries -- satisfying them all would be an additional cost that would be hard to recover. Common Lisp is on a level /between/ the "free" languages that are quite painful to use and the commercial success of a mass-market language where you can make money on third-party libraries. Those who are new to Common Lisp seldom realize that it is such a different language from the rest of the crop -- it is almost a different crop. This may sound arrogant to people who have not used Common Lisp for real projects. I may sound hopelessly out of touch with other languages and their advantages to those who come from environments where the cost of learning everything you need is much higher than the cost of implementing it in Common Lisp. Those who succeed with Common Lisp are usually very bright people with experience in solving hard problems by making them simpler in surprising ways. To many of them, the boneheaded brute force of many other languages are /distasteful/, not features. Simplicity of implementation is generally not a value, because the elegance of use is what matters to users. Common Lisp programmers are far more /practical/ than the average lot, who are often more self-indulgent and childishly delighted that anyone uses their code or rewards them for their hacks or compact form of expression. Common Lisp has, in essence, a whole different value system. I have said that Common Lisp is the language you graduate into. Common Lisp is what you get to use when you have been a good programmer for a while and grew tired of all the cruft. Common Lisp programmers usually know dozens of languages and are not afraid to use them when practical, but because the cost of using a specialized language can sometimes be very high, such as when you have to keep track of which particular version of "language" and libraries and system libraries and kernels and whatnot in order to keep your system running. Knowing lots of languages also means that you are able to integrate them with ease and use particular libraries from particular languages more easily because you have a deeper understanding of the issues involved and a much broader knowledge of your actual needs. Instead of being blinded by the glitz and the hype factors, Common Lisp programmers are generally more /practical/ and solution-oriented than the random youngsters who want attention and try to get it by giving away their time to strangers. However, if you are the perennial newbie, you will never understand this. All you see is that your perennial newbiehood is rejected and your opinions are not validated by group hugs and cheers. Neither are you able to write code that anyone will applaud, so being a newbie in Common Lisp is not the kind of "fun" it is to be a newbie in Python or Perl or whatever, where everybody is no more experienced than you are and you can impress just about anyone if you are above average. Common Lisp requires more of you, but when you have grown up to meet those requirements, it is also extremely rewarding. If you remain on the unmet side of those requirements, your task is to meet them, not to sulk over the rejection or wax apologetical about your lacking prowess: Your task should simply be to realize that it takes more than you got, then either decide to return later when you have what it takes, or decide to take on the task of learning what it takes. If you hang around feeling inferior, you will do nobody any good, especially not yourself. Despite the nervousness of some Germans, I consider it proper to make it clear to people that they should make this choice, but for some reason, some people prefer to sulk and whine than sit down and figure out what it takes to become a member in good standing. Common Lisp is not a "free" language in that it takes no effort to learn. Common Lisp is not a "free" language in that you can as easily discard it. Common Lisp is not a "free" language in that you will be unchanged by it. Common Lisp is not an investment-free language, but neither it is profit-free. -- Erik Naggum, Oslo, Norway Act from reason, and failure makes you rethink and study harder. Act from faith, and failure makes you blame someone and push harder.