phmarek / yason Goto Github PK

The macro WITH-OUTPUT-TO-STRING* binds the json-stream to YASON::S and doesn't take an argument to provide another symbol. Is this intended behavior, or a mistake in the macro?

https://github.com/hanshuebner/yason/blob/1779dbe8cd1e09060e6571a79b011a5b7eb759f2/encode.lisp#L209-L215

returns 1.5798061E9
should return 1.57980604E9
I think the solution is that parse should bind
read-default-float-format to 'double-float.

In parse.lisp there is an (error 'expected-colon) but expected-colon hasn't been defined as a condition anywhere. This results in an error "EXPECTED-COLON doesn't designate a condition class" under SBCL when that error is reached.

Consider:

CL-USER> (yason:with-output (*standard-output*)
           (yason:with-object ()
             (yason:encode-object-elements
              "name" "Ingmar Bergman"
              "movies" (yason:with-array ()
                         (yason:encode-array-elements
                          "The Seventh Seal" "Persona" "The Silence")))))
{["The Seventh Seal","Persona","The Silence"],"name":"Ingmar Bergman","movies":null}

Here, the array is written to the stream out of place, presumably because the call to with-array is being evaluated before encode-object-elements has time to do its thing.

What should I do here to encode this kind of objects, other than turn them into an alist and call encode on that?

It seems to print to *standard-output* by default and return an empty string. From the documentation:

with-output-to-string* (&key indent stream-symbol) &body body => result*
Set up a JSON streaming encoder context on stream-symbol (by default a gensym), then evaluate body. Return a string with the generated JSON output. See WITH-OUTPUT for the description of the indent keyword argument

but...

(defparameter plt `(("$schema" . "https://vega.github.io/schema/vega-lite/v5.json")
 ("data"
  ("values"
   . #((("SOURCE" . "Other") ("COUNT" . 19))
       (("SOURCE" . "Wikipedia") ("COUNT" . 52))
       (("SOURCE" . "Library") ("COUNT" . 75))
       (("SOURCE" . "Google") ("COUNT" . 406)))))
 ("mark" . "bar")
 ("encoding"
  ("x" ("field" . "SOURCE") ("type" . "nominal") ("axis" ("labelAngle" . 0)))
  ("y" ("field" . "COUNT") ("type" . "quantitative")))))

encoding this:

(let ((yason:*list-encoder* 'yason:encode-alist))
  (yason:with-output-to-string* ()
    (yason:encode plt)))
{"$schema":"https://vega.github.io/schema/vega-lite/v5.json","data":{"values":[{"SOURCE":"Other","COUNT":19},{"SOURCE":"Wikipedia","COUNT":52},{"SOURCE":"Library","COUNT":75},{"SOURCE":"Google","COUNT":406}]},"mark":"bar","encoding":{"x":{"field":"SOURCE","type":"nominal","axis":{"labelAngle":0}},"y":{"field":"COUNT","type":"quantitative"}}}
""

Note the empty string as the return value on the last line.

Am I missing something? I need the value as a string to pass to other functions. Actually, I need it as an escaped JSON string for this use case.

Here's a snippet of code I've been using to check whether something is true or not, regardless of whether null and the booleans are parsed as symbols or not:

(defun true-p (yason-bool)
  (case yason-bool
    ('yason:false nil)
    (:null nil)
    (otherwise yason-bool)))

It's especially helpful when writing libraries using YASON where you don't want to make assumptions about what the user might have bound *parse-json-booleans-as-symbols* and *parse-json-null-as-keyword* but still need to reliably check truth values.

The question, before I go ahead and make a patch out of it, is how strictly it should adhere to what is true and false in JavaScript. In JavaScript the number 0 and the empty string is also false, which is just two more clauses in the case expression, but empty array in JavaScript is true, so is empty array in Lisp, but that conflicts with parsing arrays as lists, or bools as t/nil.

I would prefer to keep it as it stands, lispers are already used to using zerop and checking sequence lengths, and the latter is probably too much of a fringe case to be worth the trouble.

The function encode-object-elements, which is supposed to encode a plist into JSON, improperly encodes lists.

For example,

CL-USER> (json:with-output (*standard-output*)
       (json:with-object ()
         (json:encode-object-elements '("a" 1 "b" 2 "c" 3))))
{["a",1,"b",2,"c",3]:null}
NIL

encode-object-elements has (&rest elements) as its lambda list; changing this to simply (elements) solves the problem:

CL-USER> (json:with-output (*standard-output*)
       (json:with-object ()
         (json:encode-object-elements '("a" 1 "b" 2 "c" 3))))
{"a":1,"b":2,"c":3}
NIL

NaNs are not supported in JSON by default (though in Perl parser apparently). Some JSON files contain them, though.

I found this to be a useful change in parse.lisp

;; export *allow-nan* and 'nan symbols in package.lisp
(defvar *allow-nan* t) 
		    
(defun parse-constant (input)
  (let ((buffer (make-adjustable-string)))
    (loop while (alpha-char-p (peek-char nil input))
          do (vector-push-extend (read-char input) buffer))
    (cond ((string= buffer "true")
	   (if *parse-json-booleans-as-symbols* 'true t))
	  ((string= buffer "false")
	   (if *parse-json-booleans-as-symbols* 'false nil))
	  ((string= buffer "null")
	   (if *parse-json-booleans-as-symbols* 'null  nil))
	  ((and *allow-nan* (string-equal buffer "nan")) ;; allow NaN, nan, etc
	   'nan)
	  (t
	   (error "invalid constant '~A'" buffer)))))

In a followup, I appended updated version that does NaN,nan,inf,infinity,+infinity,-infinity (case insensitive). In parse-number it switches to parse-symbol if it encounters a +/- followed by 'i' or 'I' (for infinity)

Also, the original yason didn't allow a number to have a leading plus sign like {x: +99} because parse% checked only for a leading (#- #\0 #\1 #\2 #\3 #\4 #\5 #\6 #\7 #\8 #\9). This is correct according to JSON standard, but the new version loosens this restriction to allow parsing of slightly non-compliant JSON.

parse.lisp.txt

I'm trying to use the new yason:*symbol-key-encoder* to encode a hash table with symbols as keys, but it doesn't seem to work.

Example:

(ql:quickload :yason)

(defparameter *example-alist* '((:foo . 1)))
(defparameter *example-hash-table* (make-hash-table))
(defparameter *example-hash-table-with-string* (make-hash-table))

(setf (gethash :foo *example-hash-table*) 1)
(setf (gethash "foo" *example-hash-table-with-string*) 1)

Set the symbol key encoder:

(setf yason:*symbol-key-encoder* #'yason:encode-symbol-as-lowercase)

Encoding alists with symbol keys now works as expected:

(yason:encode-alist *example-alist*)
; => {"foo":1}

Hash tables with string keys work fine, as always:

(yason:encode *example-hash-table-with-string*)
; => {"foo":1}

But hash tables with symbol keys still don't work:

(yason:encode *example-hash-table*)
; => {
; => debugger invoked on a SB-PCL::NO-APPLICABLE-METHOD-ERROR in thread
; => #<THREAD "main thread" RUNNING {10005204C3}>:
; =>   There is no applicable method for the generic function
; =>     #<STANDARD-GENERIC-FUNCTION YASON:ENCODE (12)>
; =>   when called with arguments
; =>     (:FOO #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {1000006373}>).
; => See also:
; =>   The ANSI Standard, Section 7.6.6
; =>
; => restarts (invokable by number or by possibly-abbreviated name):
; =>   0: [RETRY] Retry calling the generic function.
; =>   1: [ABORT] Exit debugger, returning to top level.
; =>
; => ((:METHOD NO-APPLICABLE-METHOD (T)) #<STANDARD-GENERIC-FUNCTION YASON:ENCODE (12)> :FOO #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {1000006373}>) [fast-method]

A json parsed to alist by yason, doesn't encode back to json, but gets an error:

Example from repl:

"{"testdb\/complex":{"nested":{"#":"testdb\/complex\/nested"},"_":{">":{"nested":1624963327298},"#":"testdb\/complex"}}}"

(yason:parse *)
(("testdb/complex"
("_" ("#" . "testdb/complex") (">" ("nested" . 1624963327298)))
("nested" ("#" . "testdb/complex/nested"))))

(yason:encode-alist *)
{"testdb/complex":[["_",["#"
; Debugger entered on #<TYPE-ERROR expected-type: LIST datum: "testdb/complex">

I would instead expect to get back a string with the equivalent json of what was parsed.

I am not seeing symbol-encoder in the version distributed by quicklisp. Did something get out of sync moving from hans github to here?

I wonder if it would make sense to add a default method like this to YASON:ENCODE:

(defmethod yason:encode (o &optional s)
  (yason:with-output (s)
    (yason:encode-object o)))

This way, ENCODE will automatically work with any objects that define YASON:ENCODE-SLOTS.

(setf yason:*parse-object-as* :alist)
(yason:parse "{\"foo\":0,\"bar\":1}")

returns (("bar" . 1) ("foo" . 0))

Ordering of original list is not preserved.

This is in yason 0.7.5 under CCL

? (format t "~@{~A~%~}"
        (asdf:component-version (asdf:find-system :yason))
        (lisp-implementation-type)
        (lisp-implementation-version)
        (software-type)
        (software-version)
        (machine-type)
        (machine-version))
0.7.5
Clozure Common Lisp
Version 1.10-r16196  (LinuxX8664)
Linux
3.16.0-50-generic
x86_64
Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz
NIL

The following works fine:

? (yason:encode-alist '((:symbol . 0) ("string" . 0)))
{"SYMBOL":0,"string":0}
((:SYMBOL . 0) ("string" . 0))

However, if we reorder the two items in the a-list, it signals an error:

? (yason:encode-alist '(("string" . 0) (:symbol . 0)))
[["string"
> Error: The value 0 is not of the expected type LIST.
> While executing: YASON::ENCODE-LIST%, in process listener(1).

Reproduction steps:

(yason:with-output (*standard-output*)
           (yason:with-object ()
             (yason:with-object-element ("foo")
               (yason:encode "bar"))))

returns "{"foo""bar"}" when it should return "{"foo":"bar"}"

There is missing method to encode symbol as string (or if somebody has better solution, I'm open for discussion)

(defmethod encode ((object symbol) &optional (stream *standard-output*))
  (encode (string object) stream))

that will allow encoding hash-table with keys that are symbols (in my case keywords actually)

(defparameter *config* (make-hash-table :test #'equal))
(defparameter *config-filename* "config.json")

(defun intern-keyword (astring)
   (intern (string-upcase astring) "KEYWORD"))

(defun load-config (&optional (filename *config-filename*))
	(let ((yason:*parse-object-key-fn* #'config:intern-keyword))
		(with-open-file (stream filename
			:direction :input :if-does-not-exist :error)
			(setf *config* (yason:parse (config:file-at-once stream))))))

the load-config will parse json and convert all keys to :keywords. The subsequent call to yason:encode will produce

? (yason:encode *config*)
{
> Error: There is no applicable method for the generic function:
>          #<STANDARD-GENERIC-FUNCTION ENCODE #x3020008B5FBF>
>        when called with arguments:
>          (:ORACLE_USER #<SYNONYM-STREAM to *TERMINAL-IO* #x302000596F2D>)
> While executing: #<CCL::STANDARD-KERNEL-METHOD NO-APPLICABLE-METHOD (T)>, in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Try calling it again
> Type :? for other options.

After adding method listed on the top everything works as expected.

? (in-package yason)
#<Package "YASON">
(defmethod encode ((object symbol) &optional (stream *standard-output*))
  (encode (string object) stream))
#<STANDARD-METHOD ENCODE (SYMBOL)>
? (in-package cl-user)
#<Package "COMMON-LISP-USER">
? (yason:encode *config*)
{"ORACLE_USER":"SCOTT","ORACLE_PWD":"something123","LOG_FORMAT":"%&%<%I%;<;;>;-5p [%D] %g{}{}{:downcase}%:; ;F (%C{}{ }{:downcase})%2.2N - %:_%m%>%n","ORACLE_SERVER":"1.2.3.4:1525/DCADB","REST_SERVER":"someserver.somwhere.com"}
#<HASH-TABLE :TEST EQUAL size 5/60 #x30200098C12D>
?

can somebody modify the source, or can author allow me to contribute into this project?

I might be missing something, but I couldn't see a way to parse a JSON object as a CLOS class.

If there isn't then perhaps it could be generated when defining the encoder using macros.

The JSON spec requires that quotation marks, reverse-solidus and the control characters U+0000 - U+001F be escaped inside strings. Yason currently outputs the control-character as-is, which can result in emitting JSON that won't be accepted by parsers.

The fix for this is straightforward, except for some questions of design: should the escapes be added to *char-replacements*, or should they be done in the encode method for strings? Unlike \t et al, these are easy to transform programmatically.

*char-replacements* isn't exported, but it feels like the sort of thing a user might want to rebind to control the optional escaping of other characters. If that were the case, the mandatory-escaping characters probably ought to be handled in the encode method, and it raises the further question of whether the replacements from *char-replacements* should be assumed correct, or whether they would need checking for characters that need to be escaped.

This is also somewhat related to an issue handling non-BMP unicode characters. Non-BMP characters must be encoded as an escaped UTF-16 surrogate pair; this is also a fairly easy transform, except there are so many non-BMP characters that storing them in a replacements table probably doesn't make sense.

Any thoughts, opinions, or preferences on how this gets implemented?

Commit 76c1f59 remove json nickname but it still used in encode.lisp, test.lisp, doc.xml and doc.html

After 7daea00, yason:encode always requires a stream for the second argument. Did the API change accidentally, or intended?

(yason:encode (make-hash-table))

Unhandled UNBOUND-VARIABLE in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                        ***10014EC073***>:
  The variable YASON::*JSON-OUTPUT* is unbound.

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING ***10014EC073***>
0: ((:METHOD YASON:ENCODE (HASH-TABLE)) #<HASH-TABLE :TEST EQL :COUNT 0 ***100447DB63***>) [fast-method,optional]

In the documentation, it says Stream defaults to *STANDARD-OUTPUT*..

[Generic function]
encode object &optional stream => object

    Encode object in JSON format and write to stream. May be specialized by applications to perform specific rendering. Stream defaults to *STANDARD-OUTPUT*. 

Hello,

Thank you very much for maintaining YASON. I am installing it today (0.8.3), but there seems to be some problems with the tests (see below).

I would be grateful if you could help out with this issue.

Best regards,
Lyle
........................
Lyle J. Graham
Neurophysiology of Visual Computation
Centre Giovanni Borelli - CNRS UMR 9010
Université Paris Cité
45 rue des Saint-Pères, 75006 Paris, FRANCE
Tel: [33]6.01.36.66.74
http://lyle.neurophysics.eu/
..................................

SH> (unit-test::run-all-tests :unit :yason)
Running "parser.basic" ...
=> 2 passed.
Running "parser.basic-with-whitespace" ...
=> 2 passed.
Running "dom-encoder.basic" ...
=> 2 passed.
Running "dom-encoder.w-o-t-s*" ...
Test "dom-encoder.w-o-t-s*" issued error No matching method for the generic
function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments
(#<YASON::JSON-OUTPUT-STREAM
{60F8DAF5}>).
error: (Test "dom-encoder.w-o-t-s*") CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {60F8DAF5}>).
=> 1 crashed.
Running "dom-encoder.w-o" ...
Test "dom-encoder.w-o" issued error No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments
(#<YASON::JSON-OUTPUT-STREAM {6115294D}>).
error: (Test "dom-encoder.w-o") CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {6115294D}>).
=> 1 crashed.
Running "dom-encoder.indentation" ...
error: (WITH-OUTPUT-TO-STRING (S) ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {611B0745}>).
error: (REMOVE-IF #'WHITESPACE-CHAR-P ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {61257D7D}>).
error: (REMOVE-IF #'WHITESPACE-CHAR-P ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {612A8ABD}>).
error: (REMOVE-IF #'WHITESPACE-CHAR-P ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {612A92B5}>).
=> 1 passed. 4 crashed.
Running "dom-encoder.object-indentation" ...
error: (WITH-OUTPUT-TO-STRING (S) ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {612B25E5}>).
=> 1 crashed.
Running "dom-encoder.empty-array-and-object-indentation" ...
=> 1 passed.
Running "stream-encoder.basic-array" ...
=> 1 passed.
Running "stream-encoder.basic-object" ...
=> 1 passed.
Running "stream-encode.unicode-string" ...
error: (WITH-OUTPUT-TO-STRING (S) ..) CRASHES.
Error was: Type-error in KERNEL::OBJECT-NOT-TYPE-ERROR-HANDLER:
119070 is not of type (UNSIGNED-BYTE 16)
=> 1 crashed.
Running "stream-encoder.application-struct" ...
=> 1 passed.
Running "recursive-alist-encode" ...
=> 1 passed.
Running "symbols-as-keys" ...
=> 2 passed.
Running "symbols-as-ht-keys" ...
error: (LET* ((SYMBOL-KEY-ENCODER #'ENCODE-SYMBOL-AS-LOWERCASE)) ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {6033DD2D}>).
=> 1 crashed.
Running "ENCODE-as-obj-value" ...
error: (WITH-OUTPUT-TO-STRING (STANDARD-OUTPUT) ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {603A70BD}>).
=> 1 crashed.
Running "ENCODE-as-obj-value-2" ...
error: (WITH-OUTPUT-TO-STRING (STANDARD-OUTPUT) ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {603F6CFD}>).
=> 1 crashed.
Running "object-in-array" ...
error: (WITH-OUTPUT-TO-STRING (STANDARD-OUTPUT) ..) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {6049DE25}>).
=> 1 crashed.
Running "output-to-string-with-indentation" ...
error: (WITH-OUTPUT-TO-STRING* (INDENT 2 STREAM-SYMBOL S) (ENCODE #(1 2 3))) CRASHES.
Error was: No matching method for the generic function
#<STANDARD-GENERIC-FUNCTION EXTENSIONS:STREAM-LINE-COLUMN (3)
{28F06519}>,
when called with arguments (#<YASON::JSON-OUTPUT-STREAM {604F640D}>).
=> 1 crashed.
Running "parse-double-float" ...
=> 1 passed.
Running "parse-ordering-hash" ...
=> 3 passed.
Running "parse-ordering-alist" ...
=> 1 passed.
Running "parse-ordering-plist" ...
=> 1 passed.
Running "duplicate-key" ...
=> 2 passed.

====================================
35 tests total.
22 passed.
13 crashed.
NIL
SH>

Running "dom-encoder.basic" ...
equal: (WITH-OUTPUT-TO-STRING (S) (ENCODE *BASIC-TEST-JSON-DOM* S))             FAILS.
        Should have been "[{\"foo\":1,\"bar\":[7,8,9]},2,3,4,[5,6,7],true,null]", was "[{\"bar\":[7,8,9],\"foo\":1},2,3,4,[5,6,7],true,null]".
=> 1 failed.

This is on ABCL, yason 0.4.0. Full log: https://cl-test-grid.appspot.com/blob?key=274297

This happens not always, here is ABCL log from previous test run: https://cl-test-grid.appspot.com/blob?key=269153
no failures.

quicklisp is still uses yason-0.4.0

According to this ticket, quicklisp/quicklisp-projects#401,
it's because the yason tarball is not updated with new releases.

By adding a new variable *parse-json-null-as-keyword* and redefining yason:parse and yason::parse-constant we should be able to parse "null" as the keyword :NULL.

I've forked a current version of yason github repo and provided these changes along with some documentation on my local master branch (develop appears to be well behind master). I should be issuing a pull request shortly.

Reproduction steps:

(yason:with-output (*standard-output*)
  (yason:with-array ()
    (yason:encode-array-element
      (yason:with-object ()
        (yason:encode-object-element "foo" "bar")))))

returns [{"foo":"bar"},"bar"] when it should return [{"foo":"bar"}].

Hi!

Likely I am not able to understand how to encode object to JSON using this library.

I have a function like this:

(defgeneric  encode-to-json (object stream))

(defmethod encode-to-json ((object CLASS-NAME) stream)
 ; encode an instance of CLASS-NAME to stream
)

now let's say i would want to encode a list of the aforementioned instances in a JSON array, so i wrote:

(defmethod encode-to-json ((object list) stream)
   (yason:with-output (stream)
    (yason:with-array ()
      (loop for item in object do
        ;; **problem here**
          (encode-to-json item stream)))))

I tryed to use (yason:encode-array-element (encode-to-json item stream)) but the fact is that is responsibility of the encode-json-function to encode, what i need is something like that:

(defmethod encode-to-json ((object list) stream)
   (yason:with-output (stream)
    (yason:with-array ()
      (loop for item in object do
        (yason:with-array-element   ; this macro does not exists yet
          (encode-to-json item stream))))) => "[encoded-object-1, encoded-object-2..., encoded-object-n]"

This seems to me similar to what yason:with-object does.

I have checked the code and seems encode-array-element could be used to get yason:with-array-element, i could even try to send a patch if this is a good idea.

Otherwise i am just doing it wrong, in this case an hit to solve this issue would be very appreciated. :))

Thanks in advance and happy new year!
C.

Yason does not encode a lisp pure cons (a pure cons is a cons whose cdr is not a cons) into json correctly. For example, an attempt to encode (1 . 2) returns the error.

* (ql:quickload :yason)
* (yason:encode (cons 1 2) *standard-output*)
#<THREAD "main thread" RUNNING {7008680273}>:
  The value
    2
  is not of type
    LIST

This is because (defmethod encode ((object list) ..) treats "pure cons" as lists too,
which seems to be a "feature" of common lisp.

* (defmethod test ((object list)) (pprint object))
#<STANDARD-METHOD COMMON-LISP-USER::TEST (LIST) {7005484C83}>
* (test (cons 1 2))
(1 . 2)

Solutions:

1. Make it more explicit that "pure cons" are not supported.
2. Support encoding pure cons.

The problem is that some streams are arriving in real time, and just
cause nothing is available to read now doesn't mean that the next
thing to arrive won't be whitespace.
I believe that all callers to this function continue by reading more
characters, which they expect not to be whitespace. If so, there's
nothing lost if waiting for the next character happens here rather
than in the caller.
I see there's a newer version than mine, but it looks like the issue is still there.
In my old version my proposed fix is
defun skip-whitespace (input)
(loop
while (and ;; (listen input) ;; this should be removed!
(whitespace-p (peek-char nil input)))
do (read-char input)))

I use Yason extensively in my personal codebases and 7daea breaks all sorts of code that has been stable for years: before, yason would default to *standard-output* for encoding and now it defaults to the stream referenced by the unbound symbol *json-output*. This makes one of my primary use-cases (quickly testing things out at the repl) much more burdensome, because I have to either (setf yason::*json-output* *standard-output*) or add a bunch of boilerplate to make a json output stream.

The change itself makes sense because it makes :indent and friends less surprising but, IMO, it should be done in a non-breaking way.

This is with yason 0.7.6 under CCL:

? (format t "~@{~A~%~}"
        (asdf:component-version (asdf:find-system :yason))
        (lisp-implementation-type)
        (lisp-implementation-version)
        (software-type)
        (software-version)
        (machine-type)
        (machine-version))
0.7.6
Clozure Common Lisp
Version 1.10-r16196  (LinuxX8664)
Linux
3.16.0-50-generic
x86_64
Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz
NIL

When representing a JSON object as an a-list there is ambiguity if JSON arrays are lists, or nil is used for JSON false or null. For example, both [["a", "b"]] and {"a":["b"]} parse as the same S-expression (("a" "b")). However, while there is not a unique encoding of such an S-expression, I wouldn't expect it to simply give up this way, particularly since it has emitted a partial encoding:

? (load (merge-pathnames "test.lisp" (asdf:system-source-directory :yason)))
#P"/home/dfm/common-lisp/yason/test.lisp"

? yason-test::*basic-test-json-string* 
"[{\"foo\":1,\"bar\":[7,8,9]},2,3,4,[5,6,7],true,null]"

? (yason:parse yason-test::*basic-test-json-string* :object-as :alist)
((("bar" 7 8 9) ("foo" . 1)) 2 3 4 (5 6 7) T NIL)

? (yason:encode-alist *)
{
> Error: The value ("bar" 7 8 9) is not of the expected type (OR STRING SYMBOL CHARACTER).
> While executing: STRING, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
1 > :pop

Even worse, though, is that if we parse JSON arrays as Lisp vectors, and use symbols/keywords for JSON true, false and null we should have a clean, bijective mapping between JSON and Lisp, apart from floating point number issues. And yet even in this case yason is signalling an error, again after emitting a partial encoding:

? (yason:parse yason-test::*basic-test-json-string*
             :object-as :alist
             :json-arrays-as-vectors t
             :json-booleans-as-symbols t
             :json-nulls-as-keyword t)
#((("bar" . #(7 8 9)) ("foo" . 1)) 2 3 4 #(5 6 7) YASON:TRUE :NULL)

? (yason:encode-alist *)
{
> Error: The value #<VECTOR 20 fill-pointer 7, adjustable> is not of the expected type LIST.
> While executing: YASON:ENCODE-ALIST, in process listener(1).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
1 > 

I have encountered a situation where a small JSON string cannot be round-tripped via an alist. For example, using the simple bar chart specification example on the Vega-Lite website:

(defparameter *sbc*
"{
  \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.json\",
  \"description\": \"A simple bar chart with embedded data.\",
  \"data\": {
    \"values\": [
      {\"a\": \"A\", \"b\": 28}, {\"a\": \"B\", \"b\": 55}, {\"a\": \"C\", \"b\": 43},
      {\"a\": \"D\", \"b\": 91}, {\"a\": \"E\", \"b\": 81}, {\"a\": \"F\", \"b\": 53},
      {\"a\": \"G\", \"b\": 19}, {\"a\": \"H\", \"b\": 87}, {\"a\": \"I\", \"b\": 52}
    ]
  },
  \"mark\": \"bar\",
  \"encoding\": {
    \"x\": {\"field\": \"a\", \"type\": \"nominal\", \"axis\": {\"labelAngle\": 0}},
    \"y\": {\"field\": \"b\", \"type\": \"quantitative\"}
  }
}")

decode it as an alist:

(yason:parse *sbc* :object-as :alist)

(("encoding" ("y" ("type" . "quantitative") ("field" . "b"))
  ("x" ("axis" ("labelAngle" . 0)) ("type" . "nominal") ("field" . "a")))
 ("mark" . "bar")
 ("data"
  ("values" (("b" . 28) ("a" . "A")) (("b" . 55) ("a" . "B"))
   (("b" . 43) ("a" . "C")) (("b" . 91) ("a" . "D")) (("b" . 81) ("a" . "E"))
   (("b" . 53) ("a" . "F")) (("b" . 19) ("a" . "G")) (("b" . 87) ("a" . "H"))
   (("b" . 52) ("a" . "I"))))
 ("description" . "A simple bar chart with embedded data.")
 ("$schema" . "https://vega.github.io/schema/vega-lite/v5.json"))

and now trying to encode it again with (yason:encode-alist *) lands us in the debugger:

The value
  "quantitative"
is not of type
  LIST
   [Condition of type TYPE-ERROR]

A round trip does work with a hash-table format though.

I am using yason to encode alists for specifying Vega-Lite plots. This works, but is verbose, so I'm switching over to plists. This works well, with the exception I'm asking about here. For example:

  (yason:with-output (*standard-output* :indent t)
    (yason:encode `("$schema" "https://vega.github.io/schema/vega-lite/v5.json"
			     :description "A simple bar chart with embedded data."
			      :mark bar
			      :data (:values #((:x ,mtcars:mpg
						:y ,mtcars:hp)))
			      :encoding (:x (:field a :type nominal :axis ("labelAngle" 0))
					 :y (:field b :type quantitative)))))

The problem is in the :data encoding. Vega-Lite is a bit clumsy with data specifications, having several variations; making the user construct a plist for the data is unnecessarily cumbersome and I want to simplify it.

My first thought was to create a method for encode specialising on the :data object:

(defmethod yason:encode ((object (eql :data)) &optional (stream yason::*json-output*))
  (write-string "found :data" stream)
  object)

But that doesn't work because encode-plist's object is key/value pairs, and everything up that call chain is a function, e.g. encode-key/value, etc.

I'm currently thinking of creating a data function that will return a properly constructed alist and pass it via backquote/comma (like the two vectors are in the first example: ,mtcars:mpg, which are symbol macros), :data ,(data mtcars:mpg .... However turning the data into an alist isn't quite as straightforward as it appears because, depending on the data types and format, I need to write another key/value pair to the JSON specification to transform the data. This makes me think I'll need to write to the output stream to handle this properly.

Does anyone here have any ideas on how to handle this use case? At worst I can create a version of encode-plist and look for :data in the loop, but this seems like a blunt instrument, and being able to override processing of a single key in the key/value pair a more generally useful thing to be able to do.

I use the following macro in my projects to implement ENCODE-SLOTS. If you deem it useful in general, I would write a patch with documentation.

(defmacro encode-object-slots (object slots)
  `(with-slots ,slots ,object
     (yason:encode-object-elements
      ,@(loop :for slot :in slots
           :append `(',slot ,slot)))))

Would you consider re-encoding yason.asd as UTF-8 or ASCII? The "Hübner" makes some Lisps choke.

Since CLISP does not have a double-float class, I get this error:

*** - FIND-CLASS: DOUBLE-FLOAT does not name a class

A patch like this seems to work:

diff --git a/encode.lisp b/encode.lisp
index 80a23f5..b7a06d8 100644
--- a/encode.lisp
+++ b/encode.lisp
@@ -51,12 +51,8 @@ PLIST. Hash table is initialized using the HASH-TABLE-INITARGS."
   (encode (float object) stream)
   object)

-(defmethod encode ((object double-float) &optional (stream *standard-output*))
-  (encode (coerce object 'single-float) stream)
-  object)
-
 (defmethod encode ((object float) &optional (stream *standard-output*))
-  (princ object stream)
+  (princ (coerce object 'single-float) stream)
   object)

 (defmethod encode ((object integer) &optional (stream *standard-output*))

Hi, I'm seeing something I don't quite understand when trying to stream encode. Construction of the outer container appears fine, and I loop through a list creating internal elements. When I'm done with each iteration, it appears that the last thing I encoded is written to the stream again. I've moved the order of elements, and this shows that it is always the very last item encoded that is repeated.

I've shuffled the code around several ways, and have now moved the substructure into its own method but the behavior remains consistent.

I'm wondering if it is not possible to use a loop inside of a YASON construct? I'm including a code snippet below to show what I'm talking about. This may very well be user error, but if so still might provide an opportunity to clarify in the documentation.

(defun encode-site (site)
  (let ((name (getf site :name))
	(loc (getf site :loc)))
    (yason:encode-array-element
     (yason:with-object ()
       (yason:encode-object-element "type" "Feature")
       (yason:with-object-element ("properties")
	 (yason:with-object ()
	   (yason:encode-object-element "name" name)))
       (yason:with-object-element ("geometry")
	 (yason:with-object ()
	   (yason:encode-object-element "type" "Point")
	   (yason:encode-object-element "coordinates" loc)))))))

(defun generate-geojson ()
  (let* ((sites '((:name "A" :loc (40.12345 -75.12345))
		  (:name "B" :loc (40.13452 -75.01234))
		  (:name "C" :loc (40.10352 -74.89024)))))
    (format t "~a~%"
	    (yason:with-output-to-string* (:indent t)
	      (yason:with-object ()
		(yason:with-object-element ("features")
		  (yason:with-array ()
		    (loop for site in sites
		       do (encode-site site))))
		(yason:encode-object-element "type" "FeatureCollection"))))))

I see the following output:

{
  "features":
  [
    {
      "type":"Feature",
      "properties":
      {
        "name":"A"
      },
      "geometry":
      {
        "type":"Point",
        "coordinates":[40.123451232910156,-75.12345123291016]
      }
    },
    [40.123451232910156,-75.12345123291016],
    {
      "type":"Feature",
      "properties":
      {
        "name":"B"
      },
      "geometry":
      {
        "type":"Point",
        "coordinates":[40.134521484375,-75.01233673095703]
      }
    },
    [40.134521484375,-75.01233673095703],
    {
      "type":"Feature",
      "properties":
      {
        "name":"C"
      },
      "geometry":
      {
        "type":"Point",
        "coordinates":[40.103519439697266,-74.89024353027344]
      }
    },
    [40.103519439697266,-74.89024353027344]
  ],
  "type":"FeatureCollection"
}

Thanks!

The README still links to http://hanshuebner.github.io/yason, it should be updated to http://phmarek.github.io/yason/

I am writing a plist DSL for possibly non-technical users that is translated via yason to a JSON specification. Simple example:

'(:title "foo")
 (:command my-command)
...)

where my-command is a symbol whose property list I need to encode as part of the transformation.

My first thought was to subclass symbol and then write a specialised encode method for it, but this implementation (SBCL) doesn't allow base classes as super classes.

The documentation for *symbol-encoder*:

symbol-encoder
Function to call to translate a CL symbol into a JSON string. The default is to error out, to provide backwards-compatible behaviour.

A useful function that can be bound to this variable is YASON:ENCODE-SYMBOL-AS-LOWERCASE.

seems to suggest that I could write a streaming method and then bind it to *symbol-encoder*, however in the case of yason:encode ((object symbol)) ..., *symbol-encoder* simply returns a string for further processing and not JSON (and there's no available stream either).

What I've managed to get working is to bind to *symbol-encoder* a function that extracts the properties and writes them out as a plist, or string for symbols that don't have interesting properties for this application. However, this requires me to modify yason's encoder for symbols to relax the assertion for a string value. It doesn't seem like there's an easy way for me to override the processing for symbols.

I'd rather not ask users to download a custom version of yason, and would like to know if I missed other options, or if there's a more idiomatic way to do this. If there's no better way to do this, could we make the assertion accept a/p-lists as well as strings? (or perhaps anything that could be further encoded by yason)

The link to https://common-lisp.net/project/yason/ take you to a 403 Forbidden page.

Encoding alists and plists both works with symbolic keys. Passing a hash table to ENCODE which uses symbols as keys, though, fails.

A naive patch might be:
(defmethod encode ((obj symbol) &optional (stream standard-output))
(encode (symbol-name obj) stream))

Hey all,

on first loading yason, I receive the following warnings regarding undefined variables:

; file: C:/Users/Zulu/quicklisp/dists/quicklisp/software/yason-20230214-git/encode.lisp
; in: DEFMETHOD YASON:ENCODE (SYMBOL)
; (EQ YASON::OBJECT YASON:FALSE)
;
; caught WARNING:
; undefined variable: YASON:FALSE

; (EQ YASON::OBJECT YASON:TRUE)
;
; caught WARNING:
; undefined variable: YASON:TRUE
;
; compilation unit finished
; Undefined variables:
; YASON:FALSE YASON:TRUE
; caught 2 WARNING conditions
; printed 1 note
I'm assuming it's an issue with the load order of the files in the project, as those variables do exist

When I am try to enter this https://gist.github.com/4147727 then I get an error: https://gist.github.com/4147728

(yason:parse nil) gives the error:

*
debugger invoked on a SB-PCL::NO-APPLICABLE-METHOD-ERROR in thread
#<THREAD "main thread" RUNNING {1005760183}>:
  There is no applicable method for the generic function
    #<STANDARD-GENERIC-FUNCTION YASON::PARSE% (3)>
  when called with arguments
    (NIL).
See also:
  The ANSI Standard, Section 7.6.6

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY] Retry calling the generic function.
  1: [ABORT] Exit debugger, returning to top level.

((:METHOD NO-APPLICABLE-METHOD (T)) #<STANDARD-GENERIC-FUNCTION YASON::PARSE% (3)> NIL) [fast-method]