The thing with Lisp

So, Lisp. Awesome language. It's not popular by any stretch of the imagination, and that's okay. Popularity for popularity's sake is a waste of time. But what is this lack of popularity telling the Lisp community? Lisp has great potential, but I don't think it's being met. Maybe there are problems with Lisp in this current day and age.

Syntax

Lisp looks different from other languages, but many newbies that write their Lisp critique (as all aspiring Lisp programmers inevitably do) think that "looking different" is the problem. It isn't. Syntax can be learned.

So what is the problem?

Lisp doesn't have a syntax. This makes it easy to manipulate (with Lisp macros), so why doesn't every language do this?

Syntax provides visual cues. In a city where all the buildings look the same and the roads are laid out in a grid, it's easy to get lost; there are no landmarks to go by.

S-expressions, the de-facto standard for representing expressions in Lisp, have been compared to XML on more than a few occasions. But consider this:

(task
(name "Do things")
(desc "It's important. " (em "Really important."))

Spot the mistake? Here it is again in XML:

<task>
<name>Do things</name>
<desc>It's important. <em>Really important</em>
</task>

The second sample makes it easy to see the mistake. The missing </desc> stands out.

This doesn't mean that XML is necessarily better than S-expressions. The second makes it easier for humans to see the mistake.

Which brings me to my point: lack of editor support. Emacs and vim come with support for parenthesis matching. Outside of those editors from the 1980s, support is sporadic at best, so the whole missing "closing paren/tag" thing becomes a big issue.

Nobody has succeeded in adding syntax to Lisp yet, though Paul Graham's arc has it in small bits.

Lisp's lack of syntax is one of those strengths that's also a weakness. All I can hope for in the future is better editor support. And unfortunately, that requires a largish community. Guess we won't be seeing that for a while. Until then, I'm happy with Emacs.

Here's the minimum standard for code editors today:

• syntax highlighting
• automatic indentation

Once "parenthesis matching" joins those, problems with using S-expressions should vanish.

Lack of brevity

I recently completed a simple Pong game in Common Lisp (I'll post about it later.) The code wasn't brief by any stretch of the imagination. To be fair, CL isn't known for brevity. I took out all the duplication that was immediately obvious, but I felt I could have done the same thing in about the same number of lines in C, with less characters per line. I'd have chosen Scheme, but libraries tend for that dialect of Lisp tend to be highly implementation-dependent. Do not want.

Writing maths was verbose. Explicitly defining a package seemed like a needless hassle. I had to consult Zach Beane's article, even though I'd done it before in the past, because I couldn't remember if the :use clause accepted a list or took a variable number of symbols (for the curious, it's the latter), plus the syntax for an ASDF system definition file. Samples for simple string processing seemed needlessly verbose, since common tasks like splitting strings seemed missing from core CL. In a language where everything can be treated like first-class citizens, they all feel like second-class citizens.

Perl is an interesting case study in syntax. I don't like Perl, so I don't know it too well, but I can appreciate certain aspects of it, and one of those aspects are string processing. Perl does for strings what ALGOL-style languages do for maths. Strings are first-class citizens in Perl, and you can tell. Perl-compatible regular expressions are one of the most important things to come out of the language.

Perl makes string processing brief, just like any language that supports infix maths makes numeric processing brief.

No matter how I tried, the maths in my Pong game looked ugly. Maybe it's my own ineptitude, but I really felt I could have done without the parens. Yeah, I know there's a CL library for infix math, and I know there's one for PCREs too. There's no doubt about it: syntax makes things shorter. (Well, the little stuff anyway. Lisp's powerful abstractions make growing bigger things shorter.)

One of arc's aims is to make things brief, which I like. My question is, how brief can you get with S-expressions until you hit a wall? I hope that arc's direction will let us find out.

Again, I'd have chosen Scheme for my Pong experience, but libraries, plus eschewing state variables and iteration constructs are big turn-offs.

Stuff like CL's loop macro is awesome. Similar macros for working with numbers and strings would totally solve this.

Plurality

This is a community issue, and it really weakens Lisp as a development environment. There are just too many choices. Common Lisp or Scheme? If Common Lisp, the CLiki page lists 23 choices, and none obviously stand over any of the others. If Scheme, Wikipedia lists 21 alternatives. If you're new to Lisp, how on Earth are you supposed to make an informed decision?

Making the first choices are just an entry barrier though: it becomes a non-issue once you're in. So it's not a problem. Or is it?

One of the things that Common Lisp seems to have over Scheme is that libraries are developed with support for other implementations of Common Lisp. Chicken Scheme has its own "eggs" system for libraries, PLT Scheme and its flavours have PLaneT, and it doesn't seem like they're interoperable.

That's just a specific instance of a more general issue: if you write a library, it only works for a small subset of the users. It's a big disincentive compared to say, in Python, where if you make a module for distribution, it's available to generally everybody that can use Python. With the same amount of effort, you can reach out to a small subset of Lisp users, or the vast majority of Python users. Effort is divided, communities are divided, and it all leads to a lot of energy being poured out for little return.

The Lisp community is filled with smart, talented hackers. If Lisp were the one, single language, it should be some super language with enough libraries to run circles around even the most LOL ENTERPRISE READY languages. And yet it's not. Maybe there are enough Lisp libraries to run circles around everything. Maybe we're not seeing that because of the sheer amount of duplication going on from all this plurality.

This is one of those things that can really be solved: if plurality is there, it'll always be there. You do see some exceptions. Linux is divided to all hell, but the domination of Ubuntu has visibly strengthened Linux as a whole.

My CL implementation of choice is Steel Bank Common Lisp, but it doesn't obviously stand over any of the other implementations: it's just open-source and damn fast. Anyway, you can't ask all but one CL implementation to just die off. It'd be equally dumb to tell CL or Scheme to kill themselves for the sake of the other. It'd take a miracle for one implementation to rise head-and-shoulders above all the others because they're pretty much all mature and have reached their full potentials.

So how can this be solved? A new Lisp. Yeah, I know there are a billion of those already, but it's the only way to draw away from the image of plurality, the confusion and the duplicated effort. That's not all there is to it, otherwise one of those new Lisps would have dominated, but only a Lisp that isn't Common Lisp or Scheme can hope to escape the black hole of plurality.

I'd go as far as saying that a new Lisp should not call itself a Lisp. It could be included as a footnote on its website, but it shouldn't be generally advertised as such. This point is purely a PR note for the express purpose of community building.

Again, I hold out hope for arc. It's still advertised as a Lisp dialect, but at least its name doesn't contain "Lisp". As I said before, one of its aims is to aid hackability, but I hope it will have another effect too: users of arc will be united under the one umbrella. Under one name, working towards writing stuff that all other arc users can use.

One single, canonical language with a single, canonical implementation would do universes of good.

Setup

This is how it should be: I type "lisp" into Google/a package manager. It pops up a single official website, or shows the official package as (one of) the first results. I download and install the archive/package. I launch "lisp" from a menu/terminal, and a REPL pops up. That much maps to the real world, sans the single, canonical implementation. Now I should be able to use my favourite text editor, write blah.lisp, and type lisp blah.lisp, and it'll run.

Here's where things get hairy. Some Common Lisp implementations can do this. There are probably Scheme implementations that also allow this. But if you're using any libraries, good luck. My Pong game can't be launched like this, since it uses ASDF to load SDL and SDL-ttf. I could put the library loading in the main Lisp file, but that's ugly duplication right there.

What if you really want to develop in Lisp the way it was meant to be used? Now it gets really hairy. No conventional text editor comes with the level of inter-process communication that Lisp needs for it to reach its true potential. The choices are Emacs (with SLIME), Eclipse *shudder* (with Cusp), or vim (with an experimental plugin named Limp). There's a fair amount of setup involved with all of those, with Cusp being the simplest, just involving copying files to the right places. SLIME requires adding some lines to your .emacs file, and getting the right values involves some digging around with your CL implementation of choice. Limp involves tweaking bits of the scripts themselves, far from optimal. (edit: Actually, the Limp defaults should Just Work. Open source moves fast.) With choices like those, the true Lisp development model is as substantial as a mirage on the horizon to most people.

So let's review: command-line invocation of Lisp is limited, and the true Lisp development model requires esoteric setup. There have been efforts to solve this, like Peter Seibel's Lispbox, but the lack of official backing from the groups behind Common Lisp means that it's just "an option" rather than the option, and so it remains fairly obscure. It's still not ideal: it really should be just an editor that hooks into the one CL implementation that's already there, which would be trivial if there were a single canonical CL implementation. It's a step in the right direction though.

Some setup to have the full Lisp experience is inevitable. There's not too much that can be done which isn't already being done, and I praise the efforts of the people behind SLIME, Limp and Cusp.

What about for the other case, running Lisp from the command line? That could definitely see improvement, and it leads to the next section.

Over-abstraction

Abstraction is not free. You're always paying some cost for using a "function" (in the programming sense of the word) in procedural abstraction, for instance. Abstracting the lower levels has an inevitable hit to application performance. In exchange, we have solutions that are easier to understand.

But can it go on forever? Does abstraction, applied continuously, make things ever easier to understand? Is it possible to be too abstract?

Yes, and in fact, it occurs more often than you'd think. In primary school, you're taught about numbers. That numbers have to be taught say something about numbers themselves. Numbers are abstractions for the expression of quantities: how much "stuff" do we have? Numbers are an abstraction, and they have to be taught, so at some point in all of our lives, we didn't know what numbers were. Once we poured in time and effort (or were forced into it), we understood them and built on top of them.

There's a point in mathematics where it just gets too hard for mere mortals. If I were given a non-trivial integral calculus problem today, I'd probably stare blankly at it. You can learn and learn and learn, but eventually the benefits outweigh the costs.

What has this got to do with Lisp? Well, Lisp is old, and it shows.

One example is the pathname abstraction in Common Lisp. What. The. Hell. Making a path in Common Lisp involves a lot of parameters that are now totally obsolete. As Peter Seibel explains in Practical Common Lisp, the pathname abstraction comes from an age where the Unix-style directory tree wasn't the dominant data storage structure. That it still exists is purely a legacy detail. Today, the simplest path abstraction is the string, with components separated by either forward or back slashes. A step above that is the URI, with the protocol prefixes like http://, ftp://, file://, etc. come in. That's about as complicated as it should be: a string. CL-FAD, a CL library for making file and directory access easier for the modern day, shouldn't really exist at all: it should be part of the language, provided as a module.

Another weird thing Common Lisp is the whole separation of packages and ASDF systems. For the uninitiated, packages are part of CL itself, used for bunching together symbols (which in turn allow access to functions and variables), whereas ASDF is the de facto system for loading libraries and your Lisp files in the right order, and for allowing your own software to be loaded as a library, like make.

For both of these things, you need to completely and totally qualify these things. I mentioned that I had to consult Zach Beane's article to figure out how to correctly write package and system definitions, and they look like this (shamelessly ripped):

(defpackage #:stumpgrinder
(:use #:cl))

... and this:

(asdf:defsystem #:stumpgrinder
:depends-on (#:cl-ppcre)
:components ((:file "package")
             (:file "string"
                    :depends-on ("package"))
             (:file "stumpgrinder"
                    :depends-on ("package"
                                 "string"))))

Some of you might be asking what the difference is between loading a Lisp file that's part of your project, and one in a library. And rightly so. Why does this abstraction exist? Python has shown that both of these concepts don't have to be more than an import statement in each of the relevant files.

Python has an unfair advantage. It was born in the age where the tree structure for files and directories was dominant, and it fully capitalised on those. The Lisp way of these things are too abstract for this day and age. Things have crystallised since the old days, and Lisp should have changed with it. I've seen some Scheme code samples that show this improvement, but with the plurality, the improvement is limited to whatever Scheme implementation that code was meant for.

Again, arc, as a new language, has an opportunity to re-introduce this simplicity to both pathnames and modules. I hope that these at least are looked at while arc grows.

Lack of change

This point is mostly from the perspective of Common Lisp, since Scheme is better at changing than CL.

Common Lisp was formed as a unification of a bunch of Lisps that were floating around back in the day. Lots of groups were involved in making this happen, so they all have a say in what direction the language should go in.

Fast forward to today. Recently, in comp.lang.lisp, somebody wanted to make the Common Lisp standard open for change. There was a lot of talk about this, but it ultimately led to nothing. The groups holding the copyrights had no intentions of pouring effort into something that would give them no benefit and burn lots of time.

Common Lisp hasn't changed for a long time. There's plenty of activity in the lower levels where people are making libraries for the community, but without the language itself changing, the whole thing is stagnating. Library-makers tend to stick with "safe" problems: the ones where people have a itch that needs scratching, like sockets or regular expressions, interfacing with relational database systems, and parsing XML and JSON. There's little to no action on people really changing the fundamental primitives of Common Lisp, the way that people fundamentally code with it. Kenny Tilton has Cells, and Rob something-or-other has his lexicons, but they're still outside changing the language itself.

Languages that don't change will eventually die. I don't believe that Lisp will die, but I do believe that Common Lisp will die, simply because it isn't changing. The community that's interested in it doesn't have full ownership of the language, and likely never will. It won't die without conferring its lessons to other, newer languages.

A new Lisp could escape this entirely. This is the age of new open-source languages, and once again, arc is one such language. arc itself has a community-maintained hotbed called anarki, which contains a bunch of experiments with extending the language in various ways. If such changes are good enough, Paul Graham may be inspired to incorporate similar features into the official arc implementation. This is a very good thing, and another reason that I think arc will succeed.

Common Lisp is beyond saving. Scheme will likely survive, but it won't make any progress while it's fragmented as much as it is at the moment.

Do it yourself

The Lisp community is filled with bright people. The entry barrier into using the language ensures this. They're familiar with hard and simple problems alike. But how do you decide if a problem is hard or simple? The answer is that it varies from person to person. It also varies from community to community. A community of smart people are far more likely to see a problem as simple than a not-so-bright one. What's easy on average to a Lisp programmer may be non-trivial to the average C++ programmer.

So when somebody asks the Lisp community how to do such-and-such, the Lisp community is likely to go tell them to do it themselves. They're not being mean. Often they even provide a few code samples to get the person started. It's just these problems seem simple to them. Lisp makes it easy enough. To the Lisp community, easy enough is a fairly high bar. To mere mortals it may even be frighteningly high.

What does that mean in the greater scheme of things? It comes back to the libraries, in a couple of ways. In Common Lisp, CFFI can be used to load up and use C libraries, for instance. Hardly anybody makes bindings for C libraries because, hey, the tools you need to bind and load them are already there. Thus, it seems that library support is lacking, but as a matter of fact, it's seen as such a trivial exercise that it's hard to justify writing a library just to load things via an FFI. A non-obvious task is obvious to experts.

Such little things make a greater difference to the language. I used SDL bindings for my pong game in Common Lisp. Those bindings had already been written for me, which is a good example to go on. Common Lisp needs more of that sort of thing.

Another point is for lower level primitives for using Common Lisp in general. A simple roll-it-yourself solution can save a few seconds for a seasoned Lisp programmer. The same thing may take hours for a new Lisp programmer to learn, after either finding it hidden in documentation online, or consulting with an online group. The seasoned programmer doesn't have to save the utility: they'll either have their own locally-developed personal bank of such utilities, or they'll just say "screw that" and just write it when they need it because it's faster.

The fact that people are making these utilities should say something about Common Lisp: if the seasoned programmer is using them, maybe other people could benefit too. I know of at least one Common Lisp system dedicated to the utilities of a particular programmer (I can't remember his name). It's mostly used as a dependency for a bunch of other libraries he provides. The fact that the system was named after him implies that there isn't much focus, so it'd be more useful if the utilities were bunched off into useful little libraries of their own.

Making utilities widely available saves an experienced programmer some time, and learning programmers lots of time. Everybody wins.

It's a social issue here: things seen as trivial may actually be more beneficial than they seem at first glance. Telling people to solve their own problems, even if you provide help, doesn't improve the language. Sharing and distributing code does.

From the inherent nature of Lisp communities, I don't think there's a simple solution to this. The best that can be done is to encourage lots of rapid change to the language itself. There was a lot of buzz with arc when it was released earlier this year. Things have slowed down a bit, but the community-maintained anarki is still very active, which is good to see.

Lisp communities need to lose the DIY attitude. They should still be open to solving problems for people, but rather than just forgetting, a review process for changing and improving the language would recognise the problems that repeatedly show up so that they could be included in future iterations of said language. Python, once again, presents itself as a good example, with PEPs providing an official channel for improving the language.

arc, for a Lisp community, seems much more open to changing the language for the better than Common Lisp. Of course, it's hard for Common Lisp programmers to change the language as a whole because of the lack of change, and the plurality. The arc forum uses a self-moderating system, so there's incentive to be open and receptive to others. This means that more ideas come in. Couple this with arc's open-source nature, and it could well be onto a winning formula here.

Lisp for the new age

A language is more than it syntax. It's more than a standard, and it's more than its implementation. A programming language is a unique harmony of the language semantics, implementation, community, benevolent dictator for life, and philosophy.

So what would the ideal Lisp be? It would have to contain many of the following:

1. Have no allergy to syntax.
2. Live in a world where parenthesis matching is standard for text editors.
3. Be brief.
4. Have a single, canonical implementation.
5. Allow for easy command-line invocation.
6. Have plugins for editors outside of Emacs and vim that configure themselves automatically to the best of their ability, or an official editor that's more than just a tack-on text box.
7. Not abstract beyond the point of simplicity.
8. Be open to change.
9. Accept that problems that seem easy may be hard for others, no matter how easy the language makes solving the problem.

A new Lisp may not even call itself a Lisp. There's a lot that Lisp could learn from today's scripting languages, just like many of today's scripting languages have learned so much from Lisp. Or maybe the lessons have already been learned. All we need now is an opportunity to demonstrate them.

There's a few other things that I haven't covered above which would benefit a new Lisp:

1. Be ideal for scripting, i.e. it should be a scripting language.
2. Have a small core.
3. Foster a good collection of libraries.
4. Have "Batteries included": standard library collection should cover common problems.

arc has a lot of potential. It's not perfect, but it's young and open to change. There are only a few things that irk me about it at the moment: its dependence on MzScheme, inadequate error reporting, lack of modules/packages/whatever, and its current singular focus on web development.

But arc shows promise for the Lisp world, and it may just be the thing to bring Lisp into the 21st century. And if Paul Graham has anything to say about it, the 22nd too.

Edit: Mikael Janssen says that Limp, the Lisp plugin for vim, should actually work out the box.