Handle printing of clojure.spec.test.alpha/check results about expound HOT 17 CLOSED

clj is great for debugging these sorts of problems! Running with the clj deps and the above code, I do get the expected result. Perhaps Boot is not using the correct Expound jar for some reason. Gotta love SNAPSHOTs... 🙃

from expound.

@kennyjwilli Hmm, this is very strange. I fear that I've done something wrong here, or there is some context that I'm missing. I appreciate your help on this!

What happens if you do the following?

clj -Srepro -Sdeps '{:deps {expound {:mvn/version "0.5.1-20180423.185123-3"} org.clojure/test.check {:mvn/version "0.9.0"} org.clojure/clojure {:mvn/version "1.9.0"}}}'

And then, in the REPL, you do:

(require '[expound.alpha :as expound] 
              '[clojure.spec.test.alpha :as st] 
              '[clojure.spec.alpha :as s]
              '[clojure.test.check])
(set! s/*explain-out* (expound/custom-printer {:theme :figwheel-theme}))
(defn ithrow
  [x]
  (throw (ex-info "ithrow" {})))

(s/fdef ithrow
        :args (s/cat :x number?)
        :ret string?)

(expound/explain-results (st/check `ithrow))

Do you see the same behavior as before?

Thanks for your continued help!

from expound.

@kennyjwilli Excellent, glad to hear it worked. Thanks for all your help!

from expound.

@kennyjwilli @aviflax This feature is in expound 0.6.0. Thanks for all your help!

from expound.

For example, take this simple function:

(defn fabulous
  [x y]
  nil)

(s/fdef fabulous
        :args (s/cat :x number? :y number?)
        :ret string?)

Running (clojure.spec.test.alpha/check fabulous) results in this mountain of text: https://pastebin.com/bxcaiYzn (put in pastebin because of its length).

We can do a little better by passing some data from the returned list to expound:

(expound/printer
  (ex-data (get-in (first (st/check `fabulous)) [:clojure.spec.test.check/ret :result-data :clojure.test.check.properties/error])))
-- Function spec failed -----------

  nil

returned an invalid value

  nil

should satisfy

  string?



-------------------------
Detected 1 error
=> nil

It looks like expound.alpha/spec-name is not correctly resolving the spec name, thus the first nil. The map passed to expound.alpha/printer has :clojure.spec.alpha/failure set as :check-failed which causes spec-name to return nil.

from expound.

@kennyjwilli Thanks for reporting this! I haven't used clojure.spec.test.alpha/check myself, so I hadn't considered this use case, but it's a good idea. I'll look into it.

from expound.

OMG yes please!

from expound.

@kennyjwilli @aviflax I've added a few functions: explain-results, explain-result, explain-results-str and explain-result-str. Here's an example:

(require '[clojure.spec.alpha :as s])
(require '[clojure.spec.test.alpha :as st])
(require '[expound.alpha :as expound])
(defn fabulous
  [x y]
  nil)

(s/fdef fabulous
        :args (s/cat :x number? :y number?)
        :ret string?)

(set! s/*explain-out* expound/printer)

(expound/explain-results (st/check `fabulous))
;;== Checked expound.alpha/fabulous ===========
;;
;;-- Function spec failed -----------
;;
;;  nil
;;
;;returned an invalid value
;;
;;  nil
;;
;;should satisfy
;;
;;  string?
;;
;;
;;
;;-------------------------
;;Detected 1 error

As you can see, you can pass the results from check to this function to have it print the results with expound. Does this work for your use case? I think the error messages themselves need some work (the above message isn't very good), but I'm wondering about the general approach. What do you think?

If you're so inclined, you can try it by installing "0.5.1-SNAPSHOT"

from expound.

The approach seems good - that's is exactly how I'd use it.

I know you said the error messages need some work, but I thought I'd still let you know of a couple cases that aren't reported correctly.

This example fails with an ClassCastException but Expound thinks it was successful.

(defn my-add2
  [x y]
  (+ (str x) y))

(s/fdef my-add2
        :args (s/cat :x number? :y number?)
        :ret string?)

(expound/explain-results (st/check `my-add2))
== Checked user/my-add2 =

Success!
=> nil

Sometimes test.check may fail to generate. In this case we get an exception that says "Unable to construct gen at: [:f] for: fn?". This exception is thrown from text.check and is stored in the path [:clojure.spec.test.check/ret :result].

(defn takes-fn
  [f]
  (f 1))

(s/fdef takes-fn
        :args (s/cat :f fn?)
        :ret any?)

(st/check `takes-fn)
=>
({:spec #object[clojure.spec.alpha$fspec_impl$reify__2451
                0x2c417d47
                "clojure.spec.alpha$fspec_impl$reify__2451@2c417d47"],
  :clojure.spec.test.check/ret {:result #error{:cause "Unable to construct gen at: [:f] for: fn?",
                                               :data #:clojure.spec.alpha{:path [:f],
                                                                          :form clojure.core/fn?,
                                                                          :failure :no-gen}}},
  :sym user/takes-fn,
  :failure #error{:cause "Unable to construct gen at: [:f] for: fn?",
                  :data #:clojure.spec.alpha{:path [:f], :form clojure.core/fn?, :failure :no-gen}}})

(expound/explain-results (st/check `takes-fn))
== Checked user/takes-fn 

=> nil

from expound.

@kennyjwilli This is extremely helpful feedback. Thanks!

from expound.

@kennyjwilli @aviflax Thanks for the detailed help on how failure modes of check. I've released another version of 0.5.1-SNAPSHOT that covers the cases you've mentioned above. Can you give it another try? Thanks!

from expound.

Looks good. Here's another example that isn't working correctly.

(defn my-broken-add
  [x y]
  (+ (str x) y))

(s/fdef my-broken-add
        :args (s/cat :x number? :y number?)
        :ret string?)

(expound/explain-results (st/check `my-broken-add))
== Checked user/my-broken-add 

Success!
=> nil

Running (st/check my-broken-add) does not pass. This applies for any function that throws an exception while executing:

(defn ithrow
  [x]
  (throw (ex-info "ithrow" {})))

(s/fdef ithrow
        :args (s/cat :x number?)
        :ret string?)

(expound/explain-results (st/check `ithrow))
== Checked user/ithrow ==

Success!
=> nil

from expound.

@kennyjwilli That's weird, I see different behavior locally. When I try, I get

expound(expound/explain-results (st/check `ithrow))
== Checked user/ithrow ======================

  (user/ithrow -1)

 threw error

#error {
 :cause "ithrow"
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "ithrow"
   :data {}
   :at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
 :trace

Is it possible that your local version of 0.5.1-SNAPSHOT is out of date or cached? I just updated it a few days ago.

from expound.

Perhaps try rm -rf ~/.m2/repository/expound/expound/0.5.1-SNAPSHOT and try again to force a re-download?

from expound.

Running with [expound "0.5.1-20180423.185123-3"] yields the same results I stated earlier. I also tried deleting the expound folder and am still getting the same results.

from expound.

@kennyjwilli What's weird is that I thought I saw the behavior you reported at one point, but now I cannot reproduce. I wonder if there is something weird with check behaving in non-deterministic ways, or expound missing some specific error case. I'll continue to look.

from expound.

@bhb Thanks for your responsiveness and great work. Looking forward to using this feature!

from expound.