whitlockjc / json-refs Goto Github PK

Imagine your JSON document has i18n tokens in its JSON Reference URIs. It would be nice to be able to either pre-process a JSON Reference prior to generating its metadata via #getRefDetails/#findRefs and/or post-process the JSON Reference metadata after the fact to alter the metadata. While i18n is one example, any situation where you have dynamic URIs could benefit from this.

/cc @brettz9

The depths in options is ignorred

JsonRefs.resolveRefs(json,{depth:1}, function (err, resolved,meta) {

Here an example

https://cdn.rawgit.com/robotnic/json-refs/master/browser/tester.html

Expected behaviour

Mechanism to reduce $ref resolve after some steps.

Hi, I'm using swagger-node and it, out of the blue, stopped showing exceptions at some point. Then I started debugging and after a while I noticed that, using bluebird promises instead native-promise-only on your module, the exception content started shown properly.

I know that the "real" bug is in native-promise-only but I thought it would be nice to tell you.

If you use #findRefsAt or #resolveRefsAt and the location start starts with .., resolution fails.

We need a browser distribution.

Hi,

In my code I have the next part:

"/customers/{uuid}/payout-detail": {
      "$ref": "./paths/customers/{uuid}/payout-detail.yaml"
    },

Which leads to this in the result json:

However, if I change the path to

"/customers/{uuid}/payout-detail": {
      "$ref": "./paths/customers/uuid/payout-detail.yaml"
    },

It would expand it properly

I think it's valid to use { in paths.

Regards,

Hi,

I am a newbie to the world of json-schemas so forgive me if I ask anything stupid.

I have default values coming back for each of the field properties in the schema, but after calling resolveRefs the resolved schema no longer contains the default values originally supplied.

So in original schema say one of the properties looks like so:

Forename: {
$ref: "#/definitions/ASCIIString39"
default: "0"
maxLength: 39
type: "string"
}

And the $ref value maps to:

ASCIIString39: {
maxLength: 39
pattern: "^[ -~]*$"
type: "string"
}

Then after resolveRefs is called, the property becomes:

Forename: {
maxLength: 39
pattern: "^[ -~]*$"
type: "string"
}

Is there some option I need to be passing into the resolveRefs call so I can keep the default property or does this go beyond what this library was originally intended for?

Thanks

i'm not sure if this is per spec or not but it is causing issues with swagger when having two API endpoints which are the same but have different methods

Example

/path:
$ref: ./path.def.get.yaml
/path:
$ref: ./path.def.post.yaml

Looking at the content coming into processContent one of the keys never gets loaded

var swagger = {
  "swagger": "2.0",
  "info": {
    "version": "0.9",
    "title": "test API"
  },
  "definitions": {
    "ContactList": {
      "properties": {
        "customfields": {
          "$ref": "ContactCustomFieldSchema"
        }
      }
    },
    "ContactListUpdate": {
      "properties": {
        "customfields": {
          "$ref": "ContactCustomFieldSchema"
        }
      }
    }
  },
  "paths": {}
};

var JsonRefs = require('json-refs');
JsonRefs.resolveRefs(swagger, {}, function (err, resolved, metadata) {
  console.log(JSON.stringify(resolved, null, 2));
});

This code example produce following result:

{
  "swagger": "2.0",
  "info": {
    "version": "0.9",
    "title": "test API"
  },
  "definitions": {
    "ContactList": {
      "properties": {
        "customfields": {
          "$ref": "ContactCustomFieldSchema"
        }
      }
    },
    "ContactListUpdate": {
      "properties": {
        "customfields": {
          "swagger": "2.0",
          "info": {
            "version": "0.9",
            "title": "test API"
          },
          "definitions": {
            "ContactList": {
              "properties": {
                "customfields": {
                  "$ref": "ContactCustomFieldSchema"
                }
              }
            },
            "ContactListUpdate": {
              "properties": {
                "customfields": {}
              }
            }
          },
          "paths": {}
        }
      }
    }
  },
  "paths": {}
}

Trying to execute the example at https://www.npmjs.com/package/json-refs, getting failure: "Uncaught ReferenceError: refs is not defined" at line 21 of source file contents below. (Please advise if there is a better place to submit such a report. Thanks.)

Trying to run schema de-referencing.

While looking at OAI/OpenAPI-Specification/issues/545, I realized that while json-refs can identify invalid JSON Pointers/JSON References, it can also generate invalid JSON Pointers because it does not URI encode the path segments prior to turning them into a JSON Pointer.

You still have a number of references to "Superagent" across files. My PR updates one of them, but I presume you need another PR to change to "path-loader"?

Two related issues:

1. Regarding a proper JSON Reference, the JSON Reference spec states:

 If a URI contains a fragment identifier, then the fragment should be
   resolved per the fragment resolution mechansim of the referrant
   document.  If the representation of the referrant document is JSON,
   then the fragment identifier SHOULD be interpreted as a
   [JSON-Pointer].

The JSON Pointer spec, in turn, says about JSON Pointers:

   Since the characters '~' (%x7E) and '/' (%x2F) have a special meaning
   in JSON Pointer, they need to be encoded as '~0' and '~1'
   respectively, when appearing in a reference token.

I would think therefore, that when executing isPtr upon encountering any tildes not followed by 0 or 1, false should be given. Similarly when executing isRef, especially if we detect that the target document is indeed JSON. This will also have the consequence of potentially throwing for this reason in pathFromPtr.

The spec seems to make no mention of any requirement to do this, however:

Evaluation of each reference token begins by decoding any escaped
   character sequence; this is performed by first transforming any
   occurrence of the sequence '~1' to '/', then transforming any
   occurrence of the sequence '~0' to '~'.

...but I think that when one asks whether something is a pointer or reference, it is good to know whether it has had proper escaping.

2 .I also think that as the JsonRefs tool, you should make the segment encoding and decoding functions public.

json-refs doesn't resolve this code:

    "swagger": "2.0",
    "info": {
        "version": "0.0.0",
        "title": "Simple API"
    },
    "paths": {
        "/": {
            "get": {
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "definitions": {
        "ScoredTopic": {
            "properties": {
                "topics": {
                    "type": "array",
                    "items": {
                        "$ref": "#/definitions/ScoredTopic"
                    }
                }
            }
        }
    }

What is the best filter option to resolve this?

Hello!

I'm trying to use this library in my app that uses requirejs to load modules. However, the file that ends up being published to bower looks for the traverse and path-loader dependencies on the global object. Since both of these libraries are correctly set up to support AMD-style loading, neither of them actually puts anything on the global object. It looks like the "-standalone" version would work as long as there was a shim config to define the dependencies, but those files are excluded from the bower package for some reason.

I'd like to submit a PR with either the main bower file set to the -standalone version, or at least stop that file from being excluded, but I just wanted to check in here and make sure I wasn't missing anything first.

Thanks!

We need a contributing guide.

To avoid continuously hitting the same server for the same resource for documents reusing the same remote reference, we should cache these things.

If I want to use JsonRefs.findRefs(obj, {filter: 'local', includeInvalid: true}), it always returns {}.

Given the following scenario, should project be undefined or whitlockjc/json-refs? The order in which we resolve references dictates this. Do we resolve local references after all remote references have been resolved or do we resolve references as we run into them?

file.json

{
  remote: {
    $ref: './project.json'
  },
  project: {
    $ref: '#/remote/name'
  }
}

project.json

{
  name: 'whitlockjc/json-refs
}

Per requests here and here users want to resolve a multi-file JSON file into one file without resolving local (fragment URIs) refs. If we can specify which of these types of refs to resolve it would be perfect:

• Fragment URIs
• Relative URIs
• Absolute URIs

Hey, this is a great library, thanks for all the work on it.

My use case: I'm working on a JSON Schema documentation library and am using $ref in my JSON Schema to point to other documents. The easiest way when creating the JSON Schema $refs is to use relative paths. We have schema for an endpoint that returns a list of objects, so our schema describes and array with items $ref-ed to a different file, for example.

I'd love to see improvements on the "isRemotePointer" method. It seems to just look for https at the beginning but from what I can tell, if the ref doesn't start with a '#', it is remote. I may need to look over the JSON Pointer spec again however (https://tools.ietf.org/html/rfc6901) I looked to implement this myself a bit, here is my simple strategy:

1. Check for '#', if not continue
2. Using a provided "basepath" resolve this path with the basepath to create a URI for the schema.

Ideally that would return true from "isRemotePointer" so I can use resolveRefs method to flatten my remote JSON schema into my schema.

Thoughts?

Right now all JSON Pointers only work if prefixed with a # but it is a valid JSON Pointer if it starts with / as well. So #/definitions/Pet and /definitions/Pet are valid, and the same.

If you write a test that uses http://petstore.swagger.io/v2/swagger.json#/definitions/Pet as a reference, you realize that the local references in #/definitions/Pet no longer resolve. _(This is a result of us resolving remote references prior and _then* resolving relative references.)*

Right now, the metadata returned from #resolveRefs is only for the original document. Whenever we resolve remote references, we need to include their resolved references and so on, to the returned metadata so we have the full picture of the resolution table.

4 tests are currently failing, all apparently related I think to relativeBase and file path joining being different on Windows.

 1) json-refs #findRefsAt should handle a valid location:
     Error: ENOENT: no such file or directory, open 'c:\wamp\www\json-refs\%5Cwamp%5Cwww%5Cjson-refs%5Ctest%5Cbrowser%5Cdocuments\test-document.yaml'
      at Error (native)

  2) json-refs #resolveRefs should return the expected value:
     Error: ENOENT: no such file or directory, open 'c:\wamp\www\json-refs\%5Cwamp%5Cwww%5Cjson-refs%5Ctest%5Cbrowser%5Cdocuments\nested\test-nested.yaml'
      at Error (native)

  3) json-refs #resolveRefs should support options.subDocPath:
     TypeError: obj must be an Array or an Object
      at findRefs (C:\wamp\www\json-refs\index.js:9:26900)
      at C:\wamp\www\json-refs\index.js:9:30571
      at run (C:\wamp\www\json-refs\node_modules\karma\node_modules\core-js\modules\es6.promise.js:107:47)
      at C:\wamp\www\json-refs\node_modules\karma\node_modules\core-js\modules\es6.promise.js:118:28
      at flush (C:\wamp\www\json-refs\node_modules\karma\node_modules\core-js\modules\_microtask.js:19:5)

  4) json-refs #resolveRefsAt should return the expected value:
     TypeError: obj must be an Array or an Object
      at C:\wamp\www\json-refs\index.js:9:31385
      at run (C:\wamp\www\json-refs\node_modules\karma\node_modules\core-js\modules\es6.promise.js:107:47)
      at C:\wamp\www\json-refs\node_modules\karma\node_modules\core-js\modules\es6.promise.js:118:28
      at flush (C:\wamp\www\json-refs\node_modules\karma\node_modules\core-js\modules\_microtask.js:19:5)

I think it's time to plan for json-refs v2.0.0. Long story short, json-refs started life really as a utility for swagger-tools. As people have began to use it, we've tacked on new features, fixed bugs and a number of other things...things that were not originally planned. The project has evolved and now we must take a step back and create a release with these features planned in from the beginning. My plan is to start working on this now and update you with the changes that happen. Feel free to request new features/ideas/options/... in this thread.

Initial plans:

• Collapse APIs for checking reference types into one API that returns the reference type
• Generate reference metadata when finding references (This is an efficiency thing)
• Make json-refs more efficient
• Refactor options so that the ones passed downstream to path-loader are obvious
• Support all types of references (Right now, we support the hash-based references but the spec actually says you can omit the # and that needs to be handled to.)
• Support id-based references (Issue #3)

I'm copying all who have filed bug/feature requests and PRs as I think your opinion is invaluable.

@brettz9
@HyperBrain
@ippeiukai
@IvanGoncharov
@mjkaye
@mohsen1
@NadavK
@parky128
@Pharylon
@robotnic
@sacarro
@schrag

Whenever a $ref is encountered, an error/warning should be thrown/emitted as those properties will be removed whenever references are resolved.

If you have a reference at the root of a document will generate an invalid resolved document. Here is an example:

{
  "$ref": "https://cdn.rawgit.com/apigee-127/swagger-tools/master/test/browser/people.json#/paths/~1people~1{id}"
}

results in:

{
  "$ref": "https://cdn.rawgit.com/apigee-127/swagger-tools/master/test/browser/people.json#/paths/~1people~1{id}",
  "undefined": CONTENTS_OF_REFERENCE
}

I also don't see prepareRequest anywhere in the code except the README and test file. Are you missing a file commit? I do see processContent in the code but not prepareRequest except a comment with "prepareRequestCallback".

It is possible to create a circular reference that json-refs does not catch but it only happens for remote references. Let's say you serve a document at http://yourhostname/swagger.json and within that document you have references that look like http://yourhostname/swagger.json#/definitions/User, when json-refs resolves the remote reference it will find another remote reference pointing to itself and infinitely loop.

swagger: '2.0'
info:
  version: "0.0.0"
  title: Example
paths:
  /test:
    get:
      responses:
        200:
          description: OK
definitions:
  Two:
    allOf:
      - $ref: '#/definitions/One' 
      - format: something
  One:
    type: object 

I'm not sure what's going on but a document like that throws ptr is not defined error.

When resolving remote references, there can be situations where you need to alter the remote request for it to succeed. For example, if the JSON document is behind Basic Authentication, you would need to update the remote request for such requirements.

I am running "resolve" on a YAML file with tabs instead of spaces, and receive the following error:

"stack": "while scanning for the next token\nfound character \t that cannot start any token\n  on line 3, column 1\n    at ScannerError.YAMLError

If you alter the obj argument for options.refPreProcessor it modifies the provided document.

whitlockjc/path-loader#4

With a file such as where there is a JSON remote reference at root (but with a JSON pointer hash ending), like if we modfiy your README example to look like this:

JsonRefs.resolveRefs(
    {$ref: 'https://api.github.com/repos/whitlockjc/json-refs#/owner'},
    function (err, rJson, metadata) {
        alert('done:'+JSON.stringify(rJson)); // Still at root with an owner property here
    }
);

...the rJson variable is the root of "https://api.github.com/repos/whitlockjc/json-refs"; it does not resolve into the specific owner property as it should.

Looking at the code at https://github.com/whitlockjc/json-refs/blob/master/index.js#L331-L339 ...

...the parentPathwill be empty (as a result of being at root), so the first if block will be followed in this case, but this also means that there is no chance to utilize ref for resolving the specific subpath, i.e., #/owner, and thus the wrong JSON is returned (i.e., the parent of owner instead of owner).

The following replacement code (which defines missing and value the same regardless of parentPath) seems to fix it, at least in this case, but I haven't considered/studied your code carefully enough to know if this will work in other situations:

    missing = !from.has(pathFromPointer(ref));
    value = from.get(pathFromPointer(ref));
    if (parentPath.length === 0) {
      to.value = value;
    } else {
      to.set(parentPath, value);
    }

I actually suspect it will not work for all cases, and thus I am not submitting a PR, but wanted to help pinpoint the source of the problem. Thanks!

FYI, I've only tested in the browser, but it doesn't appear to me that it should make any difference here.

I'm wondering if I'm doing something wrong. The following throws the error Uncaught (in promise) Error: json must be an object(…):

var json = [
{
  $id: 1,
  name: 'Alice',
  Sibling: {
    $id: 2,
    name: "Bob",
    Sibling: {
      $ref: 1
    }
  },
},
  {
    $ref: 2
  }
];

JsonRefs.resolveRefs(json)
  .then(function (resolved) {
    //do something with resolved
  });

json-refs will always output JSON by default but there is a --yaml flag for outputting YAML. These are the rules regardless of the document being resolved was JSON or YAML. It would be nice if the input document was YAML, the output document was YAML and the same for JSON. Of course, there would be --json and --yaml flags to force it either way.

Thanks for the great library. I use json-refs in conjunction with angular-schema-form as suggested here: json-schema-form/angular-schema-form#118 (comment)

I'd like to suggest a new feature to enhance support of JSON Reference as it is used in JSON Schema.
JSON Schema extends JSON Reference with alteration of URI resolution scope by reserving "id" keyword for the purpose:
http://json-schema.org/latest/json-schema-core.html#anchor25

I would love to see this implemented as an option.
e.g. JsonRefs.withUriResolutionScopeAlteration({"idField":"id"}).resolveRefs(json, done)

I found URI Resolution Scope Alteration would also be useful for my own purpose when embedding nested resources.

{
    "a" : [{"$ref" : "#"}, "x"]
}

Originally reported in #5.

What do you think of having a utility for retrieving the original JSON object as well as optionally resolving references with it?

Since loading code is already required by your library anyways, and since JSON references are a general purpose requirement in working with JSON, I think it makes sense for json-refs to be usable as a general purpose JSON loader. (FWIW, I have a simple promise-based JSON loader (without JSON Reference handling) allowing loading of multiple JSON files at https://github.com/brettz9/getJSON )

If you are amenable to such a utility, I would also be interested to see storage of any header-attached JSON schema URL as well as ideally, optional auto-loading this discovered schema URL, optionally applying the user's own JSON reference resolution options for the schema loading.

json-refs removes circular references, at a configurable depth as of a few commits ago, but it loses some information in the process. Sure, you could look at the original document and figure this stuff out yourself but we already store reference resolution metadata so we could probably store information in there about the circular references as well.

in resolveRefs API, add an option to resolve circular $refs to more than 2 levels

Imagine I have this object:

{
  "definitions": {
    "Pet": {
      "properties": {
        "name": {
          "$ref": "#/definitions/Name"
        }
      }
    },
    "Name": {
      "properties": {
        "firstName": {
          "type": "string"
        },
        "lastName": {
          "type": "string"
        },
        "pet": {
          "$ref": "#/definitions/Pet"
        }
      }
    }
  }
}

After resolving $refs I get this:

{
  "definitions": {
    "Pet": {
      "properties": {
        "name": {
          "properties": {
            "firstName": {
              "type": "string"
            },
            "lastName": {
              "type": "string"
            },
            "pet": {
              "properties": {
                "name": {
                  "properties": {
                    "firstName": {
                      "type": "string"
                    },
                    "lastName": {
                      "type": "string"
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "Name": {
      "properties": {
        "firstName": {
          "type": "string"
        },
        "lastName": {
          "type": "string"
        },
        "pet": {
          "properties": {
            "name": {
              "properties": {
                "firstName": {
                  "type": "string"
                },
                "lastName": {
                  "type": "string"
                },
                "pet": {
                  "properties": {}
                }
              }
            }
          }
        }
      }
    }
  }
}

Can you add an option to resolveRefs to go deeper than two levels?

I think browser/json-refs.js needs to be rebuilt with the browserify task, as it is currently looking for a global pathloader, and it should instead be requiring one, I believe...

I guess you'll also need to release a new tag so that bower can pick it up...

Regarding our discussion on documenting promises--or wherever errors are thrown, I was thinking we should also document those errors which can be produced in the promise as a result of other dependent functions throwing errors. For example, findValue can throw an error if the pointer does not lead to a valid location, and findValue is also called by setValue, within findRefs, etc. Would you like a PR with a more exhaustive list of errors that might be encountered?

Originally reported here: apigee-127/sway#24

Hi,

The approach I envision for a pull parser would be:

1. Detect a ref
2. Evaluate some condition
3. If the condition is true
   1. Resolve the ref
   2. Optionally repeat steps

The use cases I have in mind:

1. For https://github.com/s3u/JSONPath/ , I'm interested in being able to recursively load JSON References, but only loading (and then caching) as required. If the user specifies a JSONPath expression such as $.aProperty.anotherProperty, there may be no need for loading all of the JSON References in the document (e.g., within a $.aThirdProperty branch), so as JSONPath is traversing the JSON object to find the path, it will only resolve references (and any descendant references) if they are encountered along the way of searching for the user-specified path.
2. For the likes of https://github.com/jdorn/json-editor/ which uses JSON Schemas to build an HTML form for editing JSON, if the schema is built with JSON references, in some cases, there is no need to load the entire schema at once. For example, a JSON-referenced sub-schema file may only be required for reflection if the user indicates through the UI that they wish to create a certain kind of subobject. Lazy loading of JSON data could also come in handy in JSONEditor for the same reason, as well as for the likes of working with this use case.

Ideally an iterator (finding or resolving) could follow any of these strategies:

1. Follow iteration order
2. Follow local references before remote ones (to possibly avoid unnecessary HTTP requests).

Given the increase of availability of streams, and the potential for applications such as chat, relying on them, I'd also like to see Streams API support go along with this request, at least being anticipated if browser adoption is too low to justify support now (unless an existing streams API matches the planned one).

This was originally discussed in #9 and the idea is that people want to use relative references for files and json-refs should support that. Now to support this we need to extend the resolveRefs API in the following ways:

1. The first argument should be allowed to be a string that will represent the file/URL path to load
2. There should be an optional second argument that will allow the consumer to provide options for json-refs to consume (This work will actually be done elsewhere but since that work is not done yet, I'm mentioning it here)
3. There should be options for the following:
   1. The base path/url to use for the document current provided to argument 1 (If argument 1 is a string path/url, this should not be required)
   2. The function used to take a relative reference and return a full path/url for json-refs to consume (This should not be required if you provide the first option but to allow full control, we should support this.)

This change should not break backward compatibility and it should allow the proper integration points to handle the known usage patterns discussed in #9. Also, this is a high level design and is subject to change upon implementation based on need.

/cc @four43, @sacarro

As was being discussed at #46 (comment) , when I run the gulp routine, quotes are getting added in the docs to non-strings regardless, apparently due to gulp-jsdoc-to-markdown or jsdoc-to-markdown.

(The reason it appeared to you that the gulp routine had fixed them was because I had also manually fixed them in the built docs before I knew about that routine.)

Hi,

I'm finding it useful to use a lot of JSON refs in a file, but sometimes, I'm only trying to resolve some relative path within the document and wish to avoid other references in the file getting resolved.

Would you be open to adding such a method?