Git Product home page Git Product logo

xqjson's Introduction

With XQuery 3.1's support of JSON parsing and serialization, this package is no longer being maintained. You are strongly encouraged to migrate your code to XQuery 3.1. (For eXist users, an implementation of the fn:json-to-xml and fn:xml-to-json functions, see https://gist.github.com/joewiz/d986da715facaad633db.)

Parsing JSON into XQuery

An XQuery module for parsing and serializing JSON, originally written by John Snelson, with minor bug fixes applied, and packaged in the EXPath Package format for convenient installation in any XQuery implementation that supports it.

Documentation

Snelson's original article is the official documentation. The information below focuses on how to install this module and get up and running. A table from Snelson's article about how each aspect of JSON is captured as XML is reproduced below.

Requirements and Compatibility

The original module was designed for use with XQilla, but since it is written in pure XQuery 3.0, it is compatible with other XQuery 3.0 processors. It has been tested with eXist 2.0+.

You can download the core module from the src/content/ directory and import it in your own XQuery. For many systems, it is more convenient to install the module as an EXPath Package (.xar file). A pre-built package is available on the Releases page. To build the source into a package, you will need Apache Ant. To install the package, you need an implementation of XQuery that supports the EXPath Package system.

Installation for eXist-db

To install in eXist-db, clone this repository and run ant, which will construct an EXPath Archive (.xar) file in the project's build folder. Then install the package via the eXist-db Package Manager, or place it in eXist-db's 'autodeploy' folder.

Usage

Import the module

import module namespace xqjson="http://xqilla.sourceforge.net/lib/xqjson";

Note that the original module used "xqilla" as the module's namespace prefix, but this module uses "xqjson" instead, and the original module used "http://xqilla.sourceforge.net/Functions" as the module's namespace, but this module has adopted the more specific "http://xqilla.sourceforge.net/lib/xqjson".

xqjson:parse-json($json as xs:string?) as element()?

This function translates a valid JSON string into an XML representation.

Note: This function assumes that the JSON string supplied is valid JSON. If you encounter an error with this function, please check to make sure your JSON is valid using a free, online validator like jsonlint.com.

xqjson:serialize-json($json-xml as element()?) as xs:string?

This function reverses the above process.

Note: The resulting JSON is not pretty-printed, and no effort is made to preserve whitespace when roundtripping from JSON to parsed XML back to serialized JSON.

Examples

This example shows how the parse-json() function translates and captures JSON objects, arrays, strings, numbers, booleans, and nulls. (The JSON string was taken from wikipedia.)

let $json := 
    '{
        "firstName": "John",
        "lastName": "Smith",
        "isAlive": true,
        "age": 25,
        "height_cm": 167.6,
        "address": {
            "streetAddress": "21 2nd Street",
            "city": "New York",
            "state": "NY",
            "postalCode": "10021-3100"
        },
        "phoneNumbers": [
            {
                "type": "home",
                "number": "212 555-1234"
            },
            {
                "type": "office",
                "number": "646 555-4567"
            }
        ],
        "children": [],
        "spouse": null
    }')
return
    xqjson:parse-json($json)

This will return the following result:

<json type="object">
    <pair name="firstName" type="string">John</pair>
    <pair name="lastName" type="string">Smith</pair>
    <pair name="isAlive" type="boolean">true</pair>
    <pair name="age" type="number">25</pair>
    <pair name="height_cm" type="number">167.6</pair>
    <pair name="address" type="object">
        <pair name="streetAddress" type="string">21 2nd Street</pair>
        <pair name="city" type="string">New York</pair>
        <pair name="state" type="string">NY</pair>
        <pair name="postalCode" type="string">10021-3100</pair>
    </pair>
    <pair name="phoneNumbers" type="array">
        <item type="object">
            <pair name="type" type="string">home</pair>
            <pair name="number" type="string">212 555-1234</pair>
        </item>
        <item type="object">
            <pair name="type" type="string">office</pair>
            <pair name="number" type="string">646 555-4567</pair>
        </item>
    </pair>
    <pair name="children" type="array"/>
    <pair name="spouse" type="null"/>
</json>

Using xqjson:serialize-json() on this <json> element will return the original JSON, sans pretty printing:

{"firstName":"John","lastName":"Smith","isAlive":true,"age":25,"height_cm":167.6,"address":{"streetAddress":"21 2nd Street","city":"New York","state":"NY","postalCode":"10021-3100"},"phoneNumbers":[{"type":"home","number":"212 555-1234"},{"type":"office","number":"646 555-4567"}],"children":[],"spouse":null}

JSON object with a single pair, illustrating string type

{
    "firstName": "John"
}
<json type="object">
    <pair name="firstName" type="string">John</pair>
</json>

JSON object with multiple pairs, illustrating string, number, and boolean types

{
    "firstName": "John",
    "lastName": "Smith",
    "age": 25,
    "isAlive": true
}
<json type="object">
    <pair name="firstName" type="string">John</pair>
    <pair name="lastName" type="string">Smith</pair>
    <pair name="age" type="number">25</pair>
    <pair name="isAlive" type="boolean">true</pair>
</json>

JSON array containing objects, which in turn contain pairs and arrays

[
    {
        "label": "node1",
        "children": [
            "child1",
            "child2"
        ]
    },
    {
        "label": "node2",
        "children": ["child3"]
    }
]
<json type="array">
    <item type="object">
        <pair name="label" type="string">node1</pair>
        <pair name="children" type="array">
            <item type="string">child1</item>
            <item type="string">child2</item>
        </pair>
    </item>
    <item type="object">
        <pair name="label" type="string">node2</pair>
        <pair name="children" type="array">
            <item type="string">child3</item>
        </pair>
    </item>
</json>

JSON object with a pair whose value is another object

{
    "address": {
        "streetAddress": "21 2nd Street",
        "city": "New York",
        "state": "NY",
        "postalCode": "10021-3100"
    }
}
<json type="object">
    <pair name="address" type="object">
        <pair name="streetAddress" type="string">21 2nd Street</pair>
        <pair name="city" type="string">New York</pair>
        <pair name="state" type="string">NY</pair>
        <pair name="postalCode" type="string">10021-3100</pair>
    </pair>
</json>

JSON object with a pair whose value is an array of objects

{
    "phoneNumbers": [
        {
            "type": "home",
            "number": "212 555-1234"
        },
        {
            "type": "office",
            "number": "646 555-4567"
        }
    ]
}
<json type="object">
    <pair name="phoneNumbers" type="array">
        <item type="object">
            <pair name="type" type="string">home</pair>
            <pair name="number" type="string">212 555-1234</pair>
        </item>
        <item type="object">
            <pair name="type" type="string">office</pair>
            <pair name="number" type="string">646 555-4567</pair>
        </item>
    </pair>
</json>

JSON Object with two pairs showing an empty array and a null value

{
    "children": [],
    "spouse": null
}
<json type="object">
    <pair name="children" type="array"/>
    <pair name="spouse" type="null"/>
</json>

JSON-XML Mapping

JSON type(JSON) toXML(JSON)
JSON N/A <json type="type(JSON)">toXML(JSON)</json>
{ "key1": value1, "key2": value2 } object <pair name="key1" type="type(value1)">toXML(value1)</pair> <pair name="key2" type="type(value2)">toXML(value2)</pair>
[ value1, value2 ] array <item type="type(value1)">toXML(value1)</item> <item type="type(value2)">toXML(value2)</item>
"value" string value
number number number
true / false boolean true / false
null null empty

Running the test suite

A test suite, written using the XQSuite framework for eXist, can be run with the following command, assuming Apache Ant is installed (some properties in build.xml may need to be adapted to your system):

ant test

The result should show something like:

<testsuites>
    <testsuite package="http://exist-db.org/xquery/test/xqjson"
        timestamp="2014-12-16T01:39:11.326-05:00" failures="0" pending="0" tests="38" time="PT0.191S">
        <testcase name="array-parse" class="xj:array-parse"/>
        <testcase name="array-serialize" class="xj:array-serialize"/>
        <!--more testcases...-->
    </testsuite>
</testsuites>

If all is well, the @failures attribute should read 0.

xqjson's People

Contributors

adamretter avatar albicocca avatar joewiz avatar ljo avatar peterstadler avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

xqjson's Issues

error when unescaping json string containing U+00B7

For JSON strings that contain \u00b7 (middle dot), xqjson unescapes this as hex string 23, but codepoints-to-string(23) returns "Codepoint 23 is not a valid character" in eXist and "F Invalid XML Character [x 17] err:FOCH0001, codepoint not valid" in Saxon (within oXygen).

Failing test:

let $json := '{"status": "\u00b7"}'
return 
    xqjson:parse-json($json)

Test illustrating where this problem occurs:

let $escaped-json := '\u00b7'
return 
    xqjson:unescape-json-string($escaped-json)

Add tests

To prevent regressions when code is submitted, or fear of them; and thus to speed acceptance of pull requests.

Bug in eXist 2.0 position() function causes invalid JSON serialization

Due to a problem with the position() function in eXist 2.0, XQJSON does not serialize JSON properly anymore. This is not an "issue" per se with XQJSON, I just wanted to post this so others were aware of this problem along with my workaround. I rewrote the two functions that used position() so that they now use string-join() instead. They are given below. This will work until the eXist team has a fix in place. I have notified them and they are working on the problem.

Cheers,
Mike

declare %private function xqjson:serializeJSONObject($e as element())
  as xs:string*
{
    "{",
    string-join(
        $e/*/(
            string-join(('"', xqjson:escape-json-string(@name), '":', xqjson:serializeJSONElement(.)), "")
        ),
        ","
    ),
    "}"
};

declare %private function xqjson:serializeJSONArray($e as element())
  as xs:string*
{
  "[",
  string-join(
      $e/*/(
        string-join(xqjson:serializeJSONElement(.), "")
      ),
      ","
  ),
  "]"
};

Regression in 0.2.0?

Using xqjson v0.2.0, the JSON at https://web.archive.org/web/20130501233406/http://joewiz.posterous.com/responses/recent?post_id=34174218 causes a stackoverflow error in eXist. Downgrading to v0.1.6, the library completes the task without error. @peterstadler If you have the chance, would you please see if saxon has a similar error when processing this JSON using xqjson v0.2.0? Thanks in advance!

The code to reproduce the error in eXist with xqjson v0.2.0:

xquery version "3.0";

import module namespace xqjson="http://xqilla.sourceforge.net/lib/xqjson";

let $url := "https://web.archive.org/web/20130501233406/http://joewiz.posterous.com/responses/recent?post_id=34174218"
let $request := <http:request method="GET" href="{$url}"/>
let $response := http:send-request($request)
let $json := util:binary-to-string($response[2])
let $xml := xqjson:parse-json($json)
return
  $xml

The error from exist.log (trimmed where an infinite loop becomes apparent):

2014-12-26 22:53:40,371 [eXistThread-37] ERROR (XQueryServlet.java [process]:566) -  
java.lang.StackOverflowError
    at java.util.regex.Pattern$7.isSatisfiedBy(Pattern.java:5171)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Slice.match(Pattern.java:3870)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$Branch.match(Pattern.java:4502)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556)
    at java.util.regex.Pattern$Loop.match(Pattern.java:4683)
    at java.util.regex.Pattern$GroupTail.match(Pattern.java:4615)
    at java.util.regex.Pattern$BranchConn.match(Pattern.java:4466)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
    at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694)
</snip>

serialize-json() bug - extra commas

The serialize-json() function appears to be inserting extra commas in the result. Here's a reproducible test, using eXide in the latest revision from the eXist-db develop branch:

xquery version "3.0";

import module namespace xqjson="http://xqilla.sourceforge.net/lib/xqjson";

let $json := '[1,[2,3]]'
let $parsed := xqjson:parse-json($json)
let $re-serialized := xqjson:serialize-json($parsed)
return
    $re-serialized

This returns '[1,[,2,3]]' instead of the expected original input. Notice the extra comma preceding 2.

The parse-json() function doesn't seem to be the cause of the problem, since $parsed is correct here:

<json type="array">
  <item type="number">1</item>
  <item type="array">
    <item type="number">2</item>
    <item type="number">3</item>
  </item>
</json>

So the problem as I see it is with the serialize-json() function.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.