CLISP runs tests in reverse order about lisp-koans HOT 4 CLOSED

Root issue is that the framework bundles the tests into a hash and then iterates over that hash to get test names:

Common Lisp hyperspec clairifies that the order of evaluation in common lisp for looping over hashes is undefined (http://www.lispworks.com/documentation/HyperSpec/Body/06_abaf.htm).

It appears that (at least on my machine configuration / common lisp version), replacing the run-koans method in lisp-unit.lsp with the following ameliorates the issue in practice, but this solution is itself implementation-dependent, and a real solution would involve reworking how the lisp-unit.lsp package manages the list of tests to treat it as a list, not a hashtable.

(defun run-koans (package)
  "Run the list of test thunks in the package. Stopping                                                                                       
   at a failure or incomplete, with more helpful messaging"
  (let ((tests (loop
                  with results = nil
                  for test-name being each hash-key in (package-table package)
                  using (hash-value unit-test)
                  do (push `(,test-name . ,unit-test) results)
                    finally (return results))))
    (loop

    with koan-results = nil
    for (test-name . unit-test) in tests
    if unit-test do
      (push (list test-name (koan-result (code unit-test))) koan-results)
    else do
      (push (list test-name :missing) koan-results)
    until (and (not *proceed-after-failure*) (any-non-pass-p koan-results))
    finally (return koan-results))))

from lisp-koans.

Yes

from lisp-koans.

This happens on SBCL too, however not for all koans. It does happen in assert.lsp though.

from lisp-koans.

I'm hitting this as well with CLISP. It's frustrating writing solutions to each exercise in top-down order and then validating them in bottom-up order.

from lisp-koans.