I’ve found out about this feature through and old work colleague and was fascinated.

A Lisp Core Image is basically a binary that contains all of the context of a CL program at a specific point in time, and it can be easily created and loaded back into a Common Lisp REPL.

Theoretically, you could do this on errors and have some form of “image based development”, where you would be able to play with the execution at that specific point in time.

For me, it’s amazing that such an old language has such an advanced and interesting feature, while other newer languages don’t.

Software development usually relies on log and code debugging to find errors and unwanted behaviours. Although useful and, in some sense, industry standard, it would be nice to have more “in depth” approaches.

Debugging is usually only possible locally, because it’s the kind of thing that stops a software execution in the middle of it; which is undesired on production code.

Common Lisp has a way to create a runnable binary from a single command, on any point of the code. This is an attempt to use save-lisp-and-die to imagine and, if possible, create a workflow that could rely on executable binaries to debug problematic code.

Work on Common Lisp started in 1981 after an initiative by ARPA manager Bob Engelmore to develop a single community standard Lisp dialect.[7] Much of the initial language design was done via electronic mail.[8][9] In 1982, Guy L. Steele Jr. gave the first overview of Common Lisp at the 1982 ACM Symposium on LISP and functional programming.[10]

Wikipedia

• As the name suggests, Common Lisp is a Lisp

(print "Hello world!")

(= 1 1)

(if (= 0 1)
    "True"
    "Not True")

• Let’s create a bad function

  (defun divide-by-zero (n)
    (/ n 0))
      

• Let’s try to call it and see what happens

  (divide-by-zero 1)
      

• Lisp also can’t divide by zero. This is a clear error. We don’t exit the REPL, but it’s undesired behaviour. Let’s handle the error a bit better.

   (defun divide-by-zero (n)
     (handler-case 
      (/ n 0)
       (error (c)
  	(print "You should not divide by zero")))) 
      

• What we did up to now doesn’t make any sense. Let’s write something that makes sense.

  (defun main ()
    (let ((args sb-ext:*posix-argv*))
      (print (eval (rest (read-from-string (format nil "~&(~{~A~^ ~})~%" args)))))))
  
  (sb-ext:save-lisp-and-die "calc"
    :executable t
    :toplevel 'main)
      

  This is a little quick app that allows us to access Common Lisp super powers from the command line. Quickly:

  • main

    The main functions reads the parameters passed to the program through command line using (args sb-ext:*posix-argv)*. Specifically, we bind all of the command line arguments to a scoped args symbol.

  • let body

    Remaining code is basically receiving the parameters from the command line, transforming them into an s-expression and evaluating it into Common Lisp.

  • save-lisp-and-die

    From the current lisp REPL, we save everything we have and create an executable. This could also be a core image, which we will talk about soon.

Loading this into SBCL, we define a main function and then create an executable, whose entry point will be the main function.

If we run it, we can execute Common Lisp forms from outside the application, like additions and subtractions.

We can do the same thing we did last time!

./calc / 1 0

• This is a more “real life” example. When we get an exception that is not caught, we end the application execution. Usually, we are able to check what’s going on by reading the log, which is possible here.

  However, Common Lisp gives us another option! We can grab the execution and redo the error!

  First, let’s try to catch this exception.

  (defun main ()
    (let* ((args sb-ext:*posix-argv*)
  	   (form (rest (read-from-string (format nil "~&(~{~A~^ ~})~%" args)))))
      (print (handler-case (eval form)
  	       (error ()
  		 (format t "Error! ~&"))))))
  
  (sb-ext:save-lisp-and-die "calc"
    :executable t
    :toplevel 'main)
  
      

  Cool. Now, if we perform any invalid operation, handler-case will swallow it and we will spit an error.

  ./calc / 1 0
  > Error! 
      

• Not very compelling. Also, it really doesn’t really help us, in fact, it’s worse. However, as said before, we can use save-lisp-and-die to do some cool tricks.

  We kind of know what’s going on here, We know that we are going to evaluate a form of Common Lisp and want to see some results. What if we saved the form that it’s going to be evaluated?

  (defvar *ERRORS*)
  
  (defun main ()
    (let* ((args sb-ext:*posix-argv*)
  	   (form (rest (read-from-string (format nil "~&(~{~A~^ ~})~%" args)))))
      (print (handler-case (eval form)
  	       (error ()
  		 (format t "Error! ~&")
  		 (setf *ERRORS* form)
  		 (sb-ext:save-lisp-and-die "calc-error"))))))
  
  (sb-ext:save-lisp-and-die "calc"
    :executable t
    :toplevel 'main)
      

  Creating our executable and running it will not yield a different result:

  ./calc / 1 0
  > Error! 
      

  However, we can see that the code is a bit different:

  • let* is actually just a way to bind the symbols one after the other.
  • defvar creates a variable that we called ERRORS.
  • (setf *ERRORS form)* sets the value of our to-be evaluated form to the ERRORS variable.
  • And, if we get an error, we create a core file named “calc-error”. We can load this file and look at the variable.

    sbcl --core calc-error
    > *ERRORS*
    > (/ 1 0)
            

    With this, it’s easy to see what was the error.

Now, let’s try this approach on a modern application. This could mean a lot of things, but for the sake of simplicity, let’s call a modern application our simple HTTP server that is able to receive requests from the web.

(load "~/quicklisp/setup.lisp")

(ql:quickload :hunchentoot)

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (format nil "Hey~@[ ~A~]!" name))

(hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4243))

This gives us a little HTTP Server, where you can call like:

http://localhost:4243/yo?name=Gustavo

And get a little hi.

Now, what if we get an error?

In this case, I’ll just force it.

(load "~/quicklisp/setup.lisp")

(ql:quickload :hunchentoot)

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (error "Could not finish request.") ;; Throwing error
  (format nil "Hey~@[ ~A~]!" name))

(hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4243))

This get’s us into the interactive debugger of Common Lisp, from which we have access to the REPL and can even retry the last execution. Of course, we still get an error from that.

Let’s extract the function and execute it while catching the error.

(load "~/quicklisp/setup.lisp")

(ql:quickload :hunchentoot)

(defun print-and-format (name)
  (error "Could not finish request.");; Throwing error
  (format nil "Hey~@[ ~A~]!" name))

(hunchentoot:define-easy-handler (say-yo :uri "/yo") (name)
  (setf (hunchentoot:content-type*) "text/plain")
  (handler-case     (print-and-format name)
    (error ()
      (sb-ext:save-lisp-and-die "calc-error"))))  ;; create image on error

(hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4243))

When accessing http://localhost:4243/yo?name=Gustavo, we get this:

[2024-04-19 13:31:25 [ERROR]] Cannot save core with multiple threads running.

                              Interactive thread (of current session):
                                #<THREAD "main thread" RUNNING {7005530453}>

                              Other threads:
                                #<THREAD "hunchentoot-listener-*:4243" RUNNING
                                   {70077582B3}>,
                                #<THREAD "hunchentoot-worker-127.0.0.1:54357" RUNNING
                                   {7007AA80C3}>
See also:
  The SBCL Manual, Node "Saving a Core Image"
Backtrace for: #<SB-THREAD:THREAD "hunchentoot-worker-127.0.0.1:54357" RUNNING {7007AA80C3}>
0: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #<SB-IMPL::STRING-OUTPUT-STREAM {102D70F63}>)
1: (HUNCHENTOOT::GET-BACKTRACE)
2: ((LAMBDA (COND) :IN HUNCHENTOOT:HANDLE-REQUEST) #<SB-IMPL::SAVE-WITH-MULTIPLE-THREADS-ERROR {700885EC13}>)
3: (SB-KERNEL::%SIGNAL #<SB-IMPL::SAVE-WITH-MULTIPLE-THREADS-ERROR {700885EC13}>)
4: (ERROR #<SB-IMPL::SAVE-WITH-MULTIPLE-THREADS-ERROR {700885EC13}>)
5: (SB-IMPL::DEINIT)
6: (SAVE-LISP-AND-DIE "calc-error" :TOPLEVEL NIL :EXECUTABLE NIL :SAVE-RUNTIME-OPTIONS NIL :CALLABLE-EXPORTS NIL :PURIFY T :ROOT-STRUCTURES #<unused argument> :ENVIRONMENT-NAME #<unused argument> :COMPRESSION NIL)
7: ((:METHOD HUNCHENTOOT:HANDLE-REQUEST (HUNCHENTOOT:ACCEPTOR HUNCHENTOOT:REQUEST)) #<HUNCHENTOOT:EASY-ACCEPTOR (host *, port 4243)> #<HUNCHENTOOT:REQUEST {70085D9383}>) [fast-method]
8: ((:METHOD HUNCHENTOOT:PROCESS-REQUEST (T)) #<HUNCHENTOOT:REQUEST {70085D9383}>) [fast-method]
9: (HUNCHENTOOT::DO-WITH-ACCEPTOR-REQUEST-COUNT-INCREMENTED #<HUNCHENTOOT:EASY-ACCEPTOR (host *, port 4243)> #<FUNCTION (LAMBDA NIL :IN HUNCHENTOOT:PROCESS-CONNECTION) {7008380E9B}>)
10: ((:METHOD HUNCHENTOOT:PROCESS-CONNECTION (HUNCHENTOOT:ACCEPTOR T)) #<HUNCHENTOOT:EASY-ACCEPTOR (host *, port 4243)> #<USOCKET:STREAM-USOCKET {7007AA5DD3}>) [fast-method]
11: ((:METHOD HUNCHENTOOT:PROCESS-CONNECTION :AROUND (HUNCHENTOOT:ACCEPTOR T)) #<HUNCHENTOOT:EASY-ACCEPTOR (host *, port 4243)> #<USOCKET:STREAM-USOCKET {7007AA5DD3}>) [fast-method]                             
12: ((:METHOD HUNCHENTOOT::HANDLE-INCOMING-CONNECTION% (HUNCHENTOOT:ONE-THREAD-PER-CONNECTION-TASKMASTER T)) #<HUNCHENTOOT:ONE-THREAD-PER-CONNECTION-TASKMASTER {700713EA73}> #<USOCKET:STREAM-USOCKET {7007AA5DD3}>) [fast-method]
13: ((LABELS BORDEAUX-THREADS::%BINDING-DEFAULT-SPECIALS-WRAPPER :IN BORDEAUX-THREADS::BINDING-DEFAULT-SPECIALS))
14: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
15: ((FLET "WITHOUT-INTERRUPTS-BODY-167" :IN SB-THREAD::RUN))
16: ((FLET SB-UNIX::BODY :IN SB-THREAD::RUN))
17: ((FLET "WITHOUT-INTERRUPTS-BODY-160" :IN SB-THREAD::RUN))
18: (SB-THREAD::RUN)

Well, that’s scary!

Let’s focus on the upper part, which is the beginning of our problem. Common Lisp tells us we can’t save a core with multiple threads, which is probably always the case of modern web based applications.

This is somewhat of a limitation on using this feature for production debugging.

I couldn’t figure out another way to do it yet, but we still could use a global variable for saving errors and maybe killing all the threads but the main one, but this gives us an error for SBCL, and it felt like I was twisting something that should not be twisted in such a way.

This is a more sophisticated approach to debugging. In this case, we’re talking about something specific and known, but this could be applied in any other situation: We can save values and inputs from functions in a global variable, dump a program on errors and inspect those values directly!

Unfortunately, save-lisp-and-die is not prepared to deal with multithread programs. For example, if we start an http server and try to create a core image, we will receive an error, because it starts other threads, so this is currently unable to be directly used at modern web applications.