Skip to content

Lisp: The Programmable Programming Language

Lisp is a fascinating programming language with a long and storied history. It was first invented in 1958, making it the second-oldest high-level programming language still in common use today, after only Fortran. But far from being a historical relic, Lisp and its family of dialects remain powerful, flexible tools for software development sixty-five years later.

A Brief History of Lisp

The story of Lisp begins with John McCarthy, a computer scientist who coined the term "artificial intelligence." As a young professor at MIT in the late 1950s, McCarthy was interested in developing better ways to reason about computation mathematically. He drew inspiration from Alonzo Church‘s lambda calculus to define a language based on recursive functions and symbolic expressions.

McCarthy‘s initial description of Lisp was remarkably simple. It consisted of just a handful of primitive operations and a notation based on nested parenthesized lists. But this humble core had immense expressive power. Lisp made it easy to define complex abstractions, work with structured data, and manipulate code as data using macros. Many now-ubiquitous programming concepts, from garbage collection to lexical scoping to REPL-driven development, had their origins in the early implementations of Lisp.

Over the decades, Lisp evolved into an entire family tree of languages. Different communities took the core ideas of Lisp down divergent paths, creating offshoots specialized for systems programming, education, AI research, application scripting, and more. Here are some of the most notable branches:

Dialect Developed Description
MacLisp 1966 The canonical Lisp of the early hacker culture at MIT, with pioneering features like macros and garbage collection.
Interlisp 1967 An influential Lisp system developed at BBN and Xerox PARC, known for its IDE and graphical tools.
Scheme 1975 A minimalist dialect with a focus on functional programming, lexical scoping, and small standard library. Popular in computer science education.
Common Lisp 1984 An attempt to standardize and consolidate the fragmented Lisp ecosystem. A large language and ecosystem suitable for production use.
Clojure 2007 A modern Lisp dialect that compiles to Java bytecode and embraces functional programming idioms. Known for its concurrency primitives.
Racket 2010 An ambitious project to create a Lisp platform for language-oriented programming, with a focus on clean design and pedagogy.

Overall, the Lisp family has shown remarkable diversity and longevity. Each dialect has its own personality and use cases, but all share the fundamental Lisp philosophy of building up complex ideas by composing together simple parts.

The Lisp Way of Thinking

More than just a technology, Lisp embodies a distinct philosophy of programming. The core tenets of this Lisp way of thinking include:

  1. Minimalism. Lisp gets by with a very small set of core primitives and syntactic rules. Everything else is built up from these foundations as an extensible tower of abstractions.

  2. Homoiconicity. Lisp represents code and data using the same basic structures (S-expressions), allowing seamless manipulation of programs by other programs.

  3. Interactive development. Lisp systems excel at providing live, interactive programming environments (REPLs) to enable fluid experimentation and debugging.

  4. Functional programming. While not strictly a functional language, Lisp encourages a functional style of programming with immutable data, higher-order functions, and recursion.

  5. Extensibility. Through macros and other metaprogramming techniques, Lisps allow the language itself to be extended and adapted to different problem domains.

  6. Exploration. Lisp has always attracted programmers who enjoy pushing the boundaries of what‘s possible with code. It rewards curiosity and creative experimentation.

These ideas have been deeply influential far beyond the Lisp community itself. Traces of Lisp DNA can be seen in the design of Python, Ruby, Haskell, Scala, Julia, and many other languages. Concepts like garbage collection, just-in-time compilation, and domain-specific languages all have roots in the Lisp world.

Lisp and Artificial Intelligence

Lisp has played a particularly outsized role in the history of artificial intelligence research. From the very beginning, Lisp was the language of choice for AI pioneers like McCarthy, Marvin Minsky, Nathaniel Rochester, and others. They appreciated how the language supported key AI programming patterns:

  • Symbolic expressions for knowledge representation
  • Recursion and conditional logic for heuristic search
  • Automatic memory management to handle complex data
  • Interactive environments for rapid prototyping
  • Macros and DSLs for specialized notation

Some notable milestones in AI history implemented in Lisp dialects:

  • The General Problem Solver (1957) – One of the earliest AI programs that searched for solutions to formal problems
  • SHRDLU (1968) – A program that carried on a natural language conversation to manipulate block structures
  • MYCIN (1972) – An early expert system that diagnosed bacterial infections based on a knowledge base of rules
  • Cyc (1984) – A long-running project to assemble a comprehensive knowledge base and reasoning engine
  • ACL2 (1989) – An automated theorem prover used for hardware and software verification
  • DeepBlue (1997) – The chess system that defeated world champion Garry Kasparov, with some components written in Lisp

Today, of course, languages like Python and specialized frameworks like TensorFlow have eclipsed Lisp for the most common types of AI development. But Lisp maintains a loyal following for certain applications, especially those involving symbolic AI, computational creativity, logic programming, and architecture discovery. Lisp‘s flexibility and expressiveness are well-suited to domains that require pushing the boundaries of what‘s possible with code.

Lisp by the Numbers

Lisp has never been a language for the mainstream masses in the way that Java or JavaScript have. But it maintains a small yet vigorous community of enthusiasts. Let‘s look at some numbers:

  • According to the TIOBE Index, Lisp (including Scheme and Clojure) is the 28th most popular programming language as of February 2023, with 0.41% market share.
  • The State of the Common Lisp survey in 2020 gathered responses from 865 Common Lisp programmers, 85% of whom use the language professionally.
  • There are over 1,500 libraries available in the Quicklisp package manager for Common Lisp.
  • The Scheme language family has over 25 actively maintained implementations.
  • Clojure was used by over 66% of respondents in a 2020 survey of functional programming languages.
  • Over 3,800 companies have declared usage of Lisp variants on StackShare.

These numbers may seem small compared to mainstream languages. But they represent a significant and stable community of developers who appreciate Lisp‘s power and expressiveness.

The Future of Lisp

As Lisp approaches its 65th anniversary, it‘s natural to wonder about its future prospects. Will this strange, parenthetical language still be relevant in a rapidly changing technological landscape?

There are reasons to be optimistic. Lisp‘s core ideas around abstraction, metaprogramming, and interactive development are arguably more relevant than ever. As software grows more complex and specialized, the ability to craft embedded languages and state-of-the-art tooling becomes increasingly crucial. Lisp dialects like Clojure and Racket are attracting new generations of programmers who care about expressiveness and maintainability.

At the same time, Lisp faces significant headwinds. It has a reputation (somewhat unfairly) as an esoteric, academic language that‘s difficult to learn. Other languages like Python and JavaScript offer some of Lisp‘s benefits with a gentler learning curve and broader ecosystem. And the rise of languages like Rust shows the appeal of powerful static type systems, an area where most Lisps have not historically focused.

On balance, I don‘t expect Lisp to ever become a dominant language on the level of JavaScript or Java. Its strengths and style are too specific. But I‘m confident that Lisp will continue to thrive in certain domains and attract programmers looking for something different. As Paul Graham wrote, "The reason Lisp is not already obsolete is that there are some things Lisp lets you do easily that can‘t be done easily in other languages."

Learning Lisp Today

What‘s the case for learning Lisp today? Not everyone needs to, of course – your choice of language should be driven by your goals and context. But I think every programmer can potentially benefit from exposure to Lisp, even if they don‘t use it as their daily workhorse.

Lisps make some of the best tools for thinking about the process and structure of computation. The minimalist elegance of Scheme, the metaprogramming power of Clojure macros, the language-hacking spirit of Racket – diving into these Lisp worlds will make you a better programmer, full stop. You‘ll come out with new abstractions and paradigms in your head.

Lisp is also a great proving ground for ideas. You can mold it into all kinds of crazy and powerful shapes without being weighed down by ceremony or bureaucracy. For folks on the cutting-edge of programming techniques – in AI, programming languages, security, software tooling, and elsewhere – the unique combination of features in Lisp remains hard to beat.

If you‘re convinced, there‘s no shortage of resources out there for getting started with Lisp. The classic textbook Structure and Interpretation of Computer Programs uses MIT Scheme to teach the fundamentals of abstraction and computation. How to Design Programs is a more modern take, using Racket. Practical Common Lisp and Land of Lisp are accessible and fun introductions to the Common Lisp ecosystem. The tutorial track at Clojure.org is great for Clojure.

Online, the Racket How To guides are excellent. TryClojure.org has an interactive tutorial. For more advanced topics, papers from the history of Lisp are often enjoyable. Check out the back archives of venues like the International Lisp Conference, European Lisp Symposium, and Scheme Workshop for ideas. Commercial training is available from providers like Cognitect and LispCast.

Lisp for the Ages

Lisp is a true pioneer among programming languages. With a history stretching back to the earliest days of computer science and a technical legacy that spans garbage collection, JIT compilation, functional programming, and more, Lisp has earned its status many times over.

More than just a grab bag of features, the Lisp family shares a coherent design philosophy. Lisp shows the power of starting from a small core and extending the language toward the needs of a given problem domain. It puts the programmer in full control, erasing the barriers between data, code, and metalinguistic abstraction. Lisp‘s interactive style invites us to carry on a conversation and experiment with computational ideas.

Far from a dead relic, Lisp continues to thrive and evolve in the 21st century. Dialects like Clojure, Racket, and Hy bring Lisp onto modern platforms like the JVM and Python ecosystem. The language remains at the forefront of AI research, language design, software tooling, and more. Not bad for a sexagenarian.

In the end, Lisp is far more than just a language. It‘s an onramp to the joy of computational thinking and a signpost to what‘s possible at the outer limits of programming. Every programmer should spend a little time with Lisp, if only to peek through the looking glass and come back changed. Love it or hate it, Lisp is here to stay.