pb33f / libopenapi Goto Github PK

Hi it doesn't seem like I can access the reference string of a parameter or check its from a reference.

For example:

openapi: 3.0.3
info:
  title: Test
  version: 0.1.0
paths:
  /test:
    get:
      operationId: getTest
      parameters:
        - $ref: '#/components/parameters/test'
      responses:
        "200":
          description: OK
components:
  parameters:
    test:
      name: test
      in: query
      required: true
      schema:
        type: string
        enum:
          - test
          - test2

I can't figure out how to get #/components/parameters/test and determine this parameter is a reference.

I would expect to be able to get the reference for at least each component type listed here https://swagger.io/docs/specification/components/#structure

I love the DigitalOcean OpenAPI spec (raw | github) for its ability to break tools: loads of relative references and even the text fields like description use $refs to separate YAML files containing Markdown prose. It seems like libopenapi can't load it when loading from a URL due to the references:

$ restish digitalocean --help
panic: errors [path item build failed: cannot find reference: resources/1-clicks/oneClicks_install_kubernetes.yml at line 539, col 13]

That makes sense, because when creating a new libopenapi.Document you don't pass in a base URI, just the loaded bytes of the YAML. The code to determine whether some reference should be loaded via HTTP is run much later down the chain:

func DetermineReferenceResolveType(ref string) int {
	if ref != "" && ref[0] == '#' {
		return LocalResolve
	}
	if ref != "" && len(ref) >= 5 && (ref[:5] == "https" || ref[:5] == "http:") {
		return HttpResolve
	}
	if strings.Contains(ref, ".json") ||
		strings.Contains(ref, ".yaml") ||
		strings.Contains(ref, ".yml") {
		return FileResolve
	}
	return -1
}

From the OpenAPI 3.1.0 spec:

Unless specified otherwise, all properties that are URIs MAY be relative references as defined by [RFC3986].

Relative references, including those in Reference Objects, PathItem Object $ref fields, Link Object operationRef fields and Example Object externalValue fields, are resolved using the referring document as the Base URI according to [RFC3986].

This doesn't seem like a quick fix, so I wanted to open this issue to discuss how to approach it.

My suggestion is to have the document take a base URI which could be a file path or URL, and any reference handling code should respect that and resolve the reference against the base unless it's a fully qualified URI itself (e.g. beings with https://...). This is somewhat similar to how kin-openapi approaches it by taking bytes and a location (file path or URL).

To anyone who stumbles across this issue, there are two workarounds currently:

1. Update relative references to fully qualified URLs.
2. Download all files locally and load from disk.

If a OpenAPI document defines a security scheme name starting with X- it is not showing up in model.Components.SecuritySchemes

for example:

securitySchemes:
    OAuth:
      flows:
        authorizationCode:
          authorizationUrl: "https://example.com/v1/oauth/authorize"
          refreshUrl: "https://example.com/v1/oauth/token"
          scopes:
            account.manage: This scope grants permissions to perform read/edit/delete actions on Account data
            account.view: This scope grants permissions to perform read only actions on Account data
            openid: This scope grants permissions that enable SSO by granting an id token JWT that stores account data. Not used in v1/account endpoints
          tokenUrl: "https://example.com/v1/oauth/token"
      type: oauth2
    X-API-Key:
      in: header
      name: X-API-Key
      type: apiKey

X-API-Key can't be found

Would be great if a v3.PathItem allowed iterating through each of the operations it contains. Currently I will need to have an awkward set of 'if` clauses to evaluate each of the valid operations in the document for a particular path.

Hi,

I create my small validator based on libopenapi.

Unfortunately, any attempt to access an OperationId that does not appear to exist in the source document will result in the following runtime error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x102ec4748]

Here is a simplified function to show how I played with it:

func validateDoc(model *libopenapi.DocumentModel[v3high.Document]) {
	for _, item := range model.Model.Paths.PathItems {

		// Here is my playground

	}
}

Ways I've tried:

• item.Options.OperationId
• item.Options.GoLow().OperationId
• ...

I tried to get up with reflection and other black magic, but unfortunately there was no success.

Could you advise me the most ideomatic way to check the presence of fields? Below I provide a file to reproduce the error.

openapi: 3.0.3
info:
  version: 1.0.0
  title: Example
servers:
  - url: 'http://127.0.0.1:8080'
paths:
  /v1/foo:
    get:
      summary: Some summary
      responses:
        '204':
          description: No content

My environment:

% go version
go version go1.19.4 darwin/arm64

% uname -a
Darwin Mercury.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103 arm64

The project's go.mod file I'm playing with looks like this (the significant part):

// omitted for brevity

require (
	github.com/pb33f/libopenapi v0.4.0
	github.com/spf13/afero v1.9.3
	github.com/spf13/cobra v1.6.1
)

require (
	github.com/dprotaso/go-yit v0.0.0-20220510233725-9ba8df137936 // indirect
	github.com/inconshreveable/mousetrap v1.0.1 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/stretchr/testify v1.8.1 // indirect
	github.com/vmware-labs/yaml-jsonpath v0.3.2 // indirect
	golang.org/x/text v0.3.7 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
)

Similar to #28, it looks like not should be a single item instead of an array.

• https://json-schema.org/understanding-json-schema/reference/combining.html#not
• https://json-schema.org/draft/2020-12/json-schema-core.html#section-10.2.1.4
• https://github.com/pb33f/libopenapi/blob/main/datamodel/high/base/schema.go#L59

It also looks like maybe conditional subschemas via if, then, else are missing as well, but 😱 💀...

At the moment there is a GetAllSchemas method on an index but this is a little misleading as this is only returning schemas that live within components/schemas.

In the below reproducible I am getting

panic: runtime error: index out of range [3] with length 3

goroutine 1 [running]:
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0xc001920900, 0xc00104a8c0, 0xc001043e00, {0xc0031066c0?, 0x7ff0eae23e00?, 0x400?}, 0x7ff0eae9dfff?, 0x40?, {0x0, 0x0})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:638 +0x2e7b
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0xc001920900, 0xc001043e00, 0xc000f74500, {0xc0031066c0?, 0x7ff0eadb2000?, 0x71c780?}, 0x7ff0eae9dfff?, 0x40?, {0x0, 0x0})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:634 +0x2af
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0xc001920900, 0xc000f74500, 0xc000f743c0, {0xc000bc6c00?, 0x7ff0d02f8080?, 0x0?}, 0x12c?, 0x40?, {0x0, 0x0})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:634 +0x2af
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0xc001920900, 0xc000f743c0, 0xc00018c1e0, {0xc001e2bd40?, 0x15?, 0x0?}, 0xc002a397b0?, 0xc8?, {0x0, 0x0})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:634 +0x2af
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractRefs(0xc001920900, 0xc00018c1e0, 0xc00018c000, {0xc002a39998?, 0x890000000040eb47?, 0x2?}, 0xc0004c04e0?, 0x80?, {0x0, 0x0})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:634 +0x2af
github.com/pb33f/libopenapi/index.NewSpecIndex(0xc00018c000)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:254 +0x773
github.com/pb33f/libopenapi/datamodel/low/v3.CreateDocument(0xc00018c0a0)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/datamodel/low/v3/create_document.go:25 +0x17d
github.com/pb33f/libopenapi.(*document).BuildV3Model(0xc000756000?)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/document.go:153 +0x4e
main.main()
        /home/trist/workspace/scratch/main.go:32 +0x37
exit status 2

package main

import (
	"fmt"
	"io"
	"net/http"

	"github.com/pb33f/libopenapi"
)

func read() []byte {
	res, err := http.Get("https://raw.githubusercontent.com/speakeasy-api/openapi-directory/main/APIs/atlassian.com/jira/1001.0.0-SNAPSHOT/openapi.yaml")
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	data, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	return data
}

func main() {
	doc, err := libopenapi.NewDocument(read())
	if err != nil {
		panic(err)
	}

	m, errs := doc.BuildV3Model()
	if errs != nil {
		panic(errs)
	}

	fmt.Println(m)
}

I have an extension x-cli-config that enables client auto-configuration given an Open API 3 document (think stuff like security set up, additional headers to always send, and an ability to prompt users for information). This configuration object is somewhat complex with a number of nested fields/objects.

It looks like libopenapi loads extensions as interface{} while kin-openapi loaded them as json.RawMessage. You can unmarshal the raw message into your struct fairly easily, but there is no stdlib way to unmarshal from already existing Go maps. For now I'm trying to use mapstructure to do something similar going from interface{} -> my struct, like so:

var config *autoConfig

if err := mapstructure.Decode(model.Extensions["x-cli-config"], &config); err != nil {
  fmt.Fprintf(os.Stderr, "Unable to unmarshal x-cli-config: %v", err)
  return
}

Is this the right way to do it? Is there some other way you expect users of the library to load their extensions? Some kind of way to register an extension with a type before loading might be nice, and then during loading it could instantiate that type, though maybe that's overkill for this library.

Part of danielgtaylor/restish#115.

From my understanding of the OpenAPI 3 spec, the security field of a document or operation is an array of maps ie []map[string][]string but the Security field of an v3.Document is *base.SecurityRequirement which contains a field Requirements map[string][]string so is not capturing that fact its an array of security requirements.

For reference https://swagger.io/docs/specification/authentication/

We are using extensions in the server object which I can't access through libopenapi.

The spec states This object MAY be extended with [Specification Extensions](https://spec.openapis.org/oas/v3.0.3#specificationExtensions). https://spec.openapis.org/oas/v3.0.3#server-object.

We use a number of extensions in various places, I haven't yet got far enough along in our integration to identify where else access to extensions may be missing, but the Swagger documentation is actually incorrect in regards to the various places extensions are supported for example https://swagger.io/docs/specification/openapi-extensions/ doesn't show that extensions can be supported by the server object but the official documentation does

I am actively investigating this on my end trying to see if I am misusing the library somewhere but after switching from another library to libopenapi we have notice quite a large memory leak.

We use the lib both on a server and in a long running CLI command that iterates over thousands of OpenAPI documents, and have experienced OOM sigkills.

Here is a top10 from pprof of our CLI towards the end of its run:

3394.52MB 47.98% 47.98%  3395.02MB 47.99%  gopkg.in/yaml%2ev3.(*parser).node
 1185.63MB 16.76% 64.74%  1185.63MB 16.76%  reflect.mapassign_faststr
  670.35MB  9.47% 74.21%  3334.26MB 47.13%  gopkg.in/yaml%2ev3.(*parser).scalar
  421.93MB  5.96% 80.18%   421.93MB  5.96%  os.ReadFile
     359MB  5.07% 85.25%      359MB  5.07%  encoding/json.Marshal
  331.97MB  4.69% 89.94%   368.50MB  5.21%  github.com/pb33f/libopenapi/datamodel/low/base.(*SchemaProxy).Schema
  186.04MB  2.63% 92.57%  4326.40MB 61.15%  gopkg.in/yaml%2ev3.(*parser).parseChild
  163.51MB  2.31% 94.88%   163.51MB  2.31%  reflect.makemap
      88MB  1.24% 96.13%       89MB  1.26%  gopkg.in/yaml%2ev3.(*decoder).scalar
      74MB  1.05% 97.17%       74MB  1.05%  gopkg.in/yaml%2ev3.read_line

let me know if there is anything I can provide to help track down the issue

JSON Schema specification states that value in the enum array can be any type. Field Enum in high-level Schema should have type []any in order to contain any possible value from a schema definition.

When building a schema via Schema.Schema() it would be great if this could return the error that occurs during building so it can be handle using normal go patterns

	// load an OpenAPI 3 specification from bytes
	springdocDefault, _ := ioutil.ReadFile("test_specs/springdocDefault.json")

	// create a new document from specification bytes
	document, err := libopenapi.NewDocument(springdocDefault)

	// if anything went wrong, an error is thrown
	if err != nil {
		panic(fmt.Sprintf("cannot create new document: %e", err))
	}

	// because we know this is a v3 spec, we can build a ready to go model from it.
	v3Model, errors := document.BuildV3Model()

	// if anything went wrong when building the v3 model, a slice of errors will be returned
	if len(errors) > 0 {
		for i := range errors {
			fmt.Printf("error: %e\n", errors[i])
		}
		panic(fmt.Sprintf("cannot create v3 model from document: %d errors reported",
			len(errors)))
	}

	for key, _ := range v3Model.Model.Components.Schemas {
		fmt.Printf("Adding schema node: %s\n", key)
	}

is resulting in a SIGSEGV

ReadOnly/WriteOnly are *bool but could be simplified to bool as they default to false always. It is always a bit awkward in Golang to work with pointers and if possible its great to avoid them especially when the absence of a value in the yaml/json doc defaults to false

While RFC 6648 deprecated the X- prefix we still have an API that includes the header (and some common proxies still include stuff like X-Forwarded-For). Interestingly, these seem to be completely ignored by libopenapi. I haven't dug into it yet but suspect this might clash with code handling the OpenAPI extensions which start with x- as well. Notably the spec does not list headers as extensible via this method even though the response object itself does support it, so these should just be parsed as normal keys.

$ go run ./bug
Headers [Last-Modified]

package main

import (
	"fmt"

	"github.com/pb33f/libopenapi"
	"golang.org/x/exp/maps"
)

var data = `
openapi: "3.1"
paths:
  /fails:
    get:
      responses:
        "200":
          headers:
            'Last-Modified':
              description: desc
              schema:
                type: string
            X-fails:
              description: desc
              schema:
                type: string
`

func main() {
	doc, err := libopenapi.NewDocument([]byte(data))
	if err != nil {
		panic(err)
	}

	result, errs := doc.BuildV3Model()
	if len(errs) > 0 {
		panic(errs)
	}

	m := result.Model

	fmt.Println("Headers", maps.Keys(m.Paths.PathItems["/fails"].Get.Responses.Codes["200"].Headers))
}

Part of danielgtaylor/restish#115

openapi: 3.1.0
info:
  title: Test
  version: 0.0.1
servers:
  - url: http://localhost:35123
paths:
  /example/{action}:
    get:
      operationId: example
      parameters:
        - name: action
          in: path
          required: true
          schema:
            type: string
        - name: action
          in: query
          required: true
          schema:
            type: string
      responses:
        "200":
          description: OK

For the above document the index retrieved by GetAllParametersFromOperations contains only one of the above parameters even though each is unique as they have different values for the in attribute with the above being a perfectly valid document according the the OpenAPI specification and a perfectly valid API

I can see the indexing is creating a operationParamErrors for this and only considering uniqueness on name

Trying to iterate over callbacks for an operation but the map is always empty.

Here is a reproducible

package main

import (
	"fmt"

	"github.com/pb33f/libopenapi"
)

func main() {
	doc, err := libopenapi.NewDocument([]byte(`openapi: 3.1.0
info:
  title: Test
  version: 0.0.4
  description: A test API
servers:
  - url: https://example.com
paths:
  /test:
    get:
      operationId: test
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    example: Hello World
      callbacks:
        newPet:
          '{$request.body#/callbackUrl}':
            post:
              requestBody:
                description: Information about a new pet in the system
                content:
                  application/json:
                    schema:
                      $ref: '#/components/schemas/Pet'
              responses:
                '200':
                  description: Return a 200 status to indicate that the data was received successfully
components:
  schemas:
    Pet:
      type: object
      properties:
        name:
          type: string
        tag:
          type: string
`))
	if err != nil {
		panic(err)
	}

	m, errs := doc.BuildV3Model()
	if errs != nil {
		panic(errs)
	}

	fmt.Println(m.Model.Paths.PathItems["/test"].Get.Callbacks)
}

Hi I noticed while looking at the way of dealing with documents that have circular references that the docs show an example using an index and resolver directly but then looking through the code here https://github.com/pb33f/libopenapi/blob/main/datamodel/low/v3/create_document.go#L30 I see CheckForCircularReferences is called but the errors returned are ignored.

I am wondering why the errors here aren't returned? It would be a nice improvement to usage if I didn't have to manually check for circular references first (and going through having to unmarshal, create index, create resolvers etc) and it was just returned as an error upon loading the document.

Or at the very least a method is exposed on the Document to allow a CheckForCircularReferences to be called without needing to go through all the setup required to do it.

This doc: https://raw.githubusercontent.com/speakeasy-api/openapi-directory/main/APIs/opensuse.org/obs/2.10.50/openapi.yaml

Produces the below error:

go run .
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6938d4]

goroutine 1 [running]:
github.com/vmware-labs/yaml-jsonpath/pkg/yamlpath.(*Path).find(0xc00088b140?, 0x5b?, 0x78a586?)
        /home/trist/go/pkg/mod/github.com/vmware-labs/[email protected]/pkg/yamlpath/path.go:29 +0x14
github.com/vmware-labs/yaml-jsonpath/pkg/yamlpath.(*Path).Find(...)
        /home/trist/go/pkg/mod/github.com/vmware-labs/[email protected]/pkg/yamlpath/path.go:25
github.com/pb33f/libopenapi/index.(*SpecIndex).FindComponentInRoot(0xc000101800, {0xc000182180, 0x5a})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:1836 +0x99
github.com/pb33f/libopenapi/index.(*SpecIndex).FindComponent(0xc000101800, {0xc000182180, 0x5a}, 0x29?)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:1572 +0x2f2
github.com/pb33f/libopenapi/index.(*SpecIndex).ExtractComponentsFromRefs(0xc000101800, {0xc000769680, 0x24, 0xc0005a60a0?})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:1532 +0x2b0
github.com/pb33f/libopenapi/index.NewSpecIndex(0xc0000d37c0)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:245 +0x786
github.com/pb33f/libopenapi/datamodel/low/v3.CreateDocument(0xc0000d3860)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/datamodel/low/v3/create_document.go:25 +0x17d
github.com/pb33f/libopenapi.(*document).BuildV3Model(0xc00062a000?)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/document.go:150 +0x4e
main.main()
        /home/trist/workspace/scratch/main.go:33 +0x32

with this code:

package main

import (
	"io"
	"net/http"

	"github.com/pb33f/libopenapi"
)

func read() []byte {
	res, err := http.Get("https://raw.githubusercontent.com/speakeasy-api/openapi-directory/main/APIs/opensuse.org/obs/2.10.50/openapi.yaml")
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	data, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	return data
}

func main() {
	data := read()

	doc, err := libopenapi.NewDocument(data)
	if err != nil {
		panic(err)
	}

	_, errs := doc.BuildV3Model()
	if len(errs) > 0 {
		panic(errs)
	}
}

While writing some tests I ran across this strange issue where a scalar default value is never loaded properly into the low or high level models. I've traced it to SetField(...) where the default case of doing nothing is triggered so the field remains nil. I haven't gone too deep on understanding that code for a fix and figure you probably have a ton more context than me. Note that if I switch this to an object and make the default e.g. {"hello": "world"} that does load properly. Example:

package main

import (
	"fmt"

	"github.com/pb33f/libopenapi"
)

var data = `
openapi: "3.1"
paths:
  /fails:
    get:
      responses:
        "200":
          content:
            application/json:
              schema:
                type: number
                default: 5
`

func main() {
	doc, err := libopenapi.NewDocument([]byte(data))
	if err != nil {
		panic(err)
	}

	result, errs := doc.BuildV3Model()
	if len(errs) > 0 {
		panic(errs)
	}

	m := result.Model

	fmt.Println("Fails", m.Paths.PathItems["/fails"].Get.Responses.Codes["200"].Content["application/json"].Schema.Schema().Default)
}

Result:

$ go run ./bug
Fails <nil>

Part of danielgtaylor/restish#115

While considering moving to this, from kin-openapi, one big usecase that'd be handy to have support for is being able to perform request/response validation according to the spec.

Is this something you'd see being officially as part of this package, or a separate package that builds on top of this one? I'd be interested in helping get this over the line, but thought I'd check first where this would best fit.

libopenapi is currently optimized for reading. Writing was not an MVP priority. However, as we move forward, this use case is becoming more requested in various threads in chat.

So, let's make models fully editable, allowing easy high-level editing/adding/removing of objects and then being able to 're-render' the model (so all the line numbers and modifications all shift accordingly).

For this OpenAPI Document:

openapi: 3.0.0
servers:
  - url: http://localhost:8080
paths:
  /test:
    get:
      parameters:
        - name: action
          in: query
          schema:
            type: string
        - name: action
          in: query
          schema:
            type: string

I would expect the index function GetAllParametersFromOperations() to return both parameters in the array of parameters it returns at the bottom of the tree.

Currently it seems to only have one of the parameters so our validation can't pick up on this and alert the user

The new UnpackExtensions function introduced in v0.3.2 seems to have trouble when the casing between the field of a Golang struct and its intended field in the extension of the API spec differs

For example, foobar in

x-custom-extension:
   foobar: baz

be matched and written to the field Foobar in

type myStruct struct {
    Foobar string
}

but FooBar, fooBar, or foo_bar will not match the struct field FooBar

x-custom-extension:
   FooBar: baz
   fooBar: baz
   foo_bar: baz

type myStruct struct {
    FooBar string
}

Hi and thanks for the great library. We were using your lib in various read/analyse scenarios and it is doing great job. We are mostly using it during execution of build pipelines. We have a need to remove some PathItems and serialize altered document for the need of javascript sdk generation based on altered version of document.
Unfortunately I have stuck on deleting the whole PathItem from document. As far as Mutating specific fields works great but deleteing PathItem is not affecting final serialized document.
I'm suspecting it has something to do with GoLow() misusage but I would appreciate some feedback on that.
Let's say we want to delete all Paths:

type OpenApiCleaner struct {
	Document libopenapi.Document
	ModelV3  *libopenapi.DocumentModel[v3.Document]
}

func (o *OpenApiCleaner) RemoveItemsWithSecurityName(item string) {
	for path, _ := range o.ModelV3.Model.Paths.GoLow().PathItems {
		delete(o.ModelV3.Model.GoLow().Paths.Value.PathItems, path)
	}
	d, err := o.Document.Serialize()
	if err != nil {
		fmt.Print(err)
	}
	fmt.Print(string(d))
}

Given the two documents:

{
    "openapi": "3.1.0",
    "paths": [
        {"/": {
            "get": {}
        }}
    ]
}

{
    "openapi": "3.1.0",
    "paths": [
        "/": {
            "get": {}
        }
    ]
}

import (
	"github.com/pb33f/libopenapi"  
	oahigh "github.com/pb33f/libopenapi/datamodel/high/v3"
	oalow "github.com/pb33f/libopenapi/datamodel/low/v3"
)

oas, err := libopenapi.NewDocument(spec)
if err != nil {
	return err
}

if doc, errs := oalow.CreateDocument(oas.GetSpecInfo()); errs == nil {
	oad := oahigh.NewDocument(doc)
}

The call to oalow.CreateDocument causes the following panic: panic: runtime error: index out of range [1] with length 1

Stacktrace:

goroutine 26 [running]:
github.com/pb33f/libopenapi/index.(*SpecIndex).GetOperationsParameterCount(0x140000a0c00)
	/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:1342 +0x142c
github.com/pb33f/libopenapi/index.runIndexFunction.func1(0x0?, 0x0?)
	/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:185 +0x28
created by github.com/pb33f/libopenapi/index.runIndexFunction
	/go/pkg/mod/github.com/pb33f/[email protected]/index/spec_index.go:184 +0x30

Related to this previous issue: #26

The new optional Explode attribute is being set to false even when not present in the document, so its not possible to treat it as optional and treat as true if unset.

Here is some example code that reproduces the issues

package main

import (
	"fmt"

	"github.com/pb33f/libopenapi"
)

func main() {
	testDoc := `openapi: 3.0.3
info:
  title: Test
  version: 0.0.1
paths:
  /test:
    get:
      parameters:
        - in: query
          name: test
          required: false
          schema:
            type: string
      responses:
        "200":
          description: OK
`

	doc, err := libopenapi.NewDocument([]byte(testDoc))
	if err != nil {
		panic(err)
	}

	model, errs := doc.BuildV3Model()
	if len(errs) > 0 {
		panic(errs)
	}

	if model.Model.Paths.PathItems["/test"].Get.Parameters[0].Explode == nil {
		fmt.Println("nil")
	} else {
		fmt.Println(*model.Model.Paths.PathItems["/test"].Get.Parameters[0].Explode)
	}
}

The above code prints false instead of nil

I get the below panic when parsing some OpenAPI documents:

panic: runtime error: index out of range [2] with length 2

goroutine 1 [running]:
github.com/pb33f/libopenapi/utils.FindKeyNodeTop({0xc0004191a0, 0x9}, {0xc0002cf430, 0x2, 0x84ccf0?})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/utils/utils.go:209 +0xe7
github.com/pb33f/libopenapi/datamodel/low.BuildModel(0xc000309c20, {0x7350c0?, 0xc0006eed80?})
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/datamodel/low/model_builder.go:43 +0x4b0
github.com/pb33f/libopenapi/datamodel/low/base.(*Schema).Build(0xc000462800, 0x75f500?, 0xc000100c00)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/datamodel/low/base/schema.go:629 +0x17a5
github.com/pb33f/libopenapi/datamodel/low/base.(*SchemaProxy).Schema(0xc000636ff0)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/datamodel/low/base/schema_proxy.go:84 +0x77
github.com/pb33f/libopenapi/datamodel/high/base.(*SchemaProxy).Schema(0xc000287f80)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/datamodel/high/base/schema_proxy.go:58 +0x2b
github.com/pb33f/libopenapi/datamodel/high/base.(*SchemaProxy).BuildSchema(0xc000287f80)
        /home/trist/go/pkg/mod/github.com/pb33f/[email protected]/datamodel/high/base/schema_proxy.go:70 +0x1e
main.main()
        /home/trist/workspace/scratch/main.go:48 +0x2c8
exit status 2

here is some example code that reproduces it:

package main

import (
	"io"
	"net/http"

	"github.com/pb33f/libopenapi"
)

func main() {
	res, err := http.Get("https://raw.githubusercontent.com/speakeasy-api/openapi-directory/main/APIs/amazonaws.com/sdb/2009-04-15/openapi.yaml")
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	data, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}

	doc, err := libopenapi.NewDocument(data)
	if err != nil {
		panic(err)
	}

	model, errs := doc.BuildV3Model()
	if len(errs) > 0 {
		panic(errs)
	}

	for _, item := range model.Model.Paths.PathItems {
		for _, op := range item.GetOperations() {
			for _, param := range op.Parameters {
				schema, err := param.Schema.BuildSchema()
				if err != nil {
					panic(err)
				}

				if schema.Items != nil && schema.Items.IsA() {
					itemSchema, err := schema.Items.A.BuildSchema()
					if err != nil {
						panic(err)
					}

					if itemSchema.AllOf != nil {
						for _, s := range itemSchema.AllOf {
							_, err := s.BuildSchema() // PANICS HERE
							if err != nil {
								panic(err)
							}
						}
					}
				}
			}
		}
	}
}

So we get the below errors:

ERROR   validation error: [line 3989] failed to build model: cannot resolve reference `#/paths/~1companies~1%7BcompanyId%7D~1data~1bills/get/responses/200/content/application~1json/schema/allOf/0/properties/results/items/properties/lineItems/items/properties/tracking/properties/isBilledTo`, it's missing: $.paths['/companies/{companyId}/data/bills'].get.responses['200'].content['application/json'].schema.allOf[0].properties.results.items.properties.lineItems.items.properties.tracking.properties.isBilledTo [3989:21]
ERROR   validation error: [line 3989] failed to build model: cannot resolve reference `#/paths/~1companies~1%7BcompanyId%7D~1data~1bills/get/responses/200/content/application~1json/schema/allOf/0/properties/results/items/properties/lineItems/items/properties/tracking/properties/isBilledTo`, it's missing: $.paths['/companies/{companyId}/data/bills'].get.responses['200'].content['application/json'].schema.allOf[0].properties.results.items.properties.lineItems.items.properties.tracking.properties.isBilledTo [3989:21]

when trying to iterate through this OpenAPI document https://github.com/codatio/oas/blob/main/yaml/Codat-Accounting.yaml

This is a document built by Stoplight (which can resolve these references) and while the referencing here is completely wild it is totally valid according to spec as far as I can tell.

The reference references a response object in another response that is itself a reference to a component but the path is expecting that reference to be resolved inline.

For an array type, there are two properties that specify the schemas of array items:

• items is a JSON Schema (either object or boolean).
• prefixItems is an array of JSON Schema correspond to elements in an array instance.

High-level Schema has Items field with type []*SchemaProxy, but it does not store any of the values of prefixItems. Low-level Schema is similar.

I think confusion would arise if prefixItems are stored in Items and items schema stored as the last element of Items. There would be no way to disambiguate between items being present or absent. So, one recommendation is to define field Items of type *SchemaProxy and field PrefixItems of type []*SchemaProxy.

I believe currently libopenapi is searching for external references ie things like $ref: 'external.yaml#/components/schemas/Item' within the current working directory.

This unfortunately limits the usability of loading external references for us as we provide the ability to specify a path to a document that could live outside the current working directory of our application and in this case we get errors like: schema build failed: reference 'external.yaml#/components/schemas/Item' cannot be found at line 15, col 23 (btw would be useful if the error could contain more info like where it is trying to find the file).

What I propose is providing a way to specify search directories (or even just a single working directory) to a libopenapi.Document

ie something like:

doc, err := libopenapi.NewDocument(mainYaml)
if err != nil {
	panic(err)
}

doc.SetSearchDirectories([]string{".", "./specs"})

m, errs := doc.BuildV3Model()
if errs != nil {
	panic(errs)
}

AllOf is expressed as sequence in yaml. On parsing it to high schema object order is not preserved, and even more, on each parse it is random. Also Low.AllOf and high.AllOf order are different.

For composition we rely on certain field order in the array therefore its crucial to maintain the order in high and low objects same as in yaml.Node. Of course one may argue that original order can be extracted from yaml.Node which is true. But then I think purpose of the library diminishes as we can just work with yaml.Node directly all the time.
We use Open API to generate our API to different languages and platforms where order matters and it would be nice to rely on that on lib level objects too.

In this document blob:https://docs.attentive.com/b6583e28-b7cf-40d0-a044-b6eef49eff3a

I get a circular reference error:

Circular reference detected: OrderProductModifier: Order -> OrderProduct -> OrderProductModifier -> OrderProductModifierCustomField [11250:31]

Looking into the document the actual circular reference seems to be Order -> OrderProduct -> OrderProductModifier -> OrderProductModifier through the modifiers field on line 11287

But it is listing OrderProductModifierCustomField as being where the circular reference is occurring but that is a simple object with no references in it.

Also from my understanding this is a "valid" circular reference as the modifiers field isn't a required field so the circular chain will break at some point (when that field is unset in the next level down from what we understand). So it won't infinitely be populated.

My understanding of this relationship between fields that cause circular references and required fields comes from this comment OAI/OpenAPI-Specification#822 (comment)

For example for this OpenAPI document: https://github.com/speakeasy-api/openapi-directory/blob/main/APIs/nexmo.com/conversation/2.0.1/openapi.yaml

At line 982 it has a ref #/components/schemas/channel/properties/from/oneOf/0 that exists in the doc at line 883

The error I get is reference cannot be found: #/components/schemas/channel/properties/from/oneOf/0, line 982, col 21

The official Open API spec shows an example of a relative schema document reference without an explicit #:

{
  "$ref": "Pet.json"
}

When I try this with libopenapi it fails to build the schema:

$ go run ./bug
panic: schema build failed: reference 'https://api.rest.sh/schemas/ErrorModel.json' cannot be found at line 11, col 23

Given this code linking directly to a JSON Schema file as the reference:

package main

import (
	"github.com/pb33f/libopenapi"
)

var data = `
openapi: "3.1"
paths:
  /fails:
    get:
      responses:
        "200":
          content:
            application/json:
              schema:
                $ref: https://api.rest.sh/schemas/ErrorModel.json
`

func main() {
	doc, err := libopenapi.NewDocument([]byte(data))
	if err != nil {
		panic(err)
	}

	_, errs := doc.BuildV3Model()
	if len(errs) > 0 {
		panic(errs[0].Error())
	}
}

As soon as I add a trailing # it loads fine. I've traced it down to FindComponent's HttpResolve branch which assumes refs will always have a # to split on:

case HttpResolve:
	uri := strings.Split(componentId, "#")
	if len(uri) == 2 {
		return index.performExternalLookup(uri, componentId, remoteLookup, parent)
	}

This is an issue instead of a PR because I'm not sure of what the correct behavior is. Should we just append a # if it is missing and leave the rest of the code as-is or make some deeper changes? It is assumed to be present in a few different places.

I also guess my use-case here is atypical but as Open API 3.1 and JSON Schema have converged it now makes much more sense to link directly to published JSON Schemas, and with the excellent VS Code support for docs/linting as you type when a $schema is present I've been publishing my schemas separately for a while now to make live-editing API resources easier.

Looking into the code AdditionalProperties is an any type which seems to be populated directly from the yaml node.

But it can be either a bool or a schema according to spec.

If its a schema the code currently returns this as a map[string]any I believe, how do I resolve this to the schema type?

Hi 👋 I'm the author of Restish and am working on adding OpenAPI 3.1 support. First off - awesome project and thanks for taking the time to write this library! Migrating from kin-openapi to libopenapi has been mostly straightforward so far.

I ran into an interesting bug where JSON schemas with objects defined that have a property named default seem to get set as the *base.Schema.Default field when it should be either nil or the actual default. Here's a simple example that when run outputs the following:

$ go run ./bug
Works1 <nil>
Works2 map[foo:hello]
Fails map[type:string]      <--- this should be `nil`!

Here is the code. I would link to the Go playground but it times out, so you'll have to run locally.

package main

import (
	"fmt"

	"github.com/pb33f/libopenapi"
)

var data = `
openapi: "3.1"
paths:
  /works1:
    get:
      responses:
        "200":
          content:
            application/json:
              schema:
                type: object
                properties:
                  foo:
                    type: string
  /works2:
    get:
      responses:
        "200":
          content:
            application/json:
              schema:
                type: object
                default:
                  foo: hello
                properties:
                  foo:
                    type: string
  /fails:
    get:
      responses:
        "200":
          content:
            application/json:
              schema:
                type: object
                properties:
                  default:
                    type: string
                  foo:
                    type: string
`

func main() {
	doc, err := libopenapi.NewDocument([]byte(data))
	if err != nil {
		panic(err)
	}

	result, errs := doc.BuildV3Model()
	if len(errs) > 0 {
		panic(errs)
	}

	m := result.Model

	fmt.Println("Works1", m.Paths.PathItems["/works1"].Get.Responses.Codes["200"].Content["application/json"].Schema.Schema().Default)

	fmt.Println("Works2", m.Paths.PathItems["/works2"].Get.Responses.Codes["200"].Content["application/json"].Schema.Schema().Default)

	fmt.Println("Fails", m.Paths.PathItems["/fails"].Get.Responses.Codes["200"].Content["application/json"].Schema.Schema().Default)
}

As discussed in this issue #25

It would be great to be able to easily determine if high level models (via their GoLow() low level model) where resolved from references.

Some of the types we need to determine are references are mentioned in this swagger documentation https://swagger.io/docs/specification/components/

At the moment it seems the v3.Parameter's explode field is just a boolean. Meaning if its not set in the OpenAPI document it will default to the boolean's zero value of false. But in the case of query parameters for example, the default value should be true, and because the type is boolean (as opposed to *boolean) I can't tell if it is unset in the document and I should default handling of explode to true.

As discussed in #83, This feature would introduce https://pkg.go.dev/io/fs as an interface to allow references to be stored in abstracted storage like databases.

Here is an example of a schema that I believe isn't handled correctly:

schema:
  title: destination-amazon-sqs
  $ref: '#/components/schemas/destination-amazon-sqs'

For OpenAPI 3.0 this is being handle such that it is just treated as a reference which I believe is correct.

But for 3.1 which is using JSON Schema Specification Draft 2020-12 this

"allows keywords alongside $ref as a shorthand to putting two schemas inside an allOf"

which is a quote from a comment in this stack overflow question https://stackoverflow.com/questions/56749522/using-a-ref-and-other-properties-within-a-json-schema and while it is not explicitly called out here https://json-schema.org/understanding-json-schema/structuring.html#ref it does list the 3.0 behaviour versus 3.1

In Draft 4-7, $ref behaves a little differently. When an object contains a $ref property, the object is considered a reference, not a schema. Therefore, any other properties you put in that object will not be treated as JSON Schema keywords and will be ignored by the validator. $ref can only be used where a schema is expected.

So my thoughts are that this should potentially return a allOf array with the two schemas split out. Thoughts?

It would be great to have a way after loading the document to have each schema prebuilt instead of needing to call "Schema.Schema()" while iterating through the document and needing to deal with the build errors.

Basically I would want to be able to load & validate the document immediately and then have the confidence to iterate through the document knowing it is a valid document and I don't need to deal with any errors caused by a invalid document later

When comparing the two documents with what-changed, the call to CompareDocuments panics. It happens when TotalChanges is called.

The ExampleChanges in MediaTypeChanges contains key test but the value is nil.

func (m *MediaTypeChanges) TotalChanges() int {
    c := m.PropertyChanges.TotalChanges()
    for k := range m.ExampleChanges {
        c += m.ExampleChanges[k].TotalChanges()   <-- panics
    }
    ...
}

openapi: "3.0.0"
paths:
  /:
    post:
      requestBody:
        content:
          application/json:
            examples:
              test:
                value:
                  date: 2020-01-01  # not quoted

openapi: "3.0.0"
paths:
  /:
    post:
      requestBody:
        content:
          application/json:
            examples:
              test:
                value:
                  date: "2020-01-01"  # quoted

Greetings!

I suppose there is something wrong with v2 model parameters. If I'm wrong and this behavior is correct, please explain me why.

The problem is when single pathItem have a few operations (2 in example). The parameters of first operation somehow fall into pathItem-level operations.

The example spec:

swagger: "2.0"
info:
  version: 1.0.2
  title: Example OpenAPI Spec

host: api.example.com
basePath: /v1
schemes:
  - http


paths:
  /post:
    get:
      parameters:
        - name: param1
          in: query
          type: string
        - name: param2
          in: query
          type: string
      responses:
        200:
          description: OK
    post:
      parameters:
        - name: param3
          in: query
          type: string
      responses:
        200:
          description: OK

POC to reproduce:

package main

import (
	"fmt"
	"os"

	"github.com/pb33f/libopenapi"
)

func main() {
	file, err := os.ReadFile(os.Args[1])
	if err != nil {
		panic(err)
	}
	document, err := libopenapi.NewDocument(file)
	if err != nil {
		panic(err)
	}
	v2Model, errs := document.BuildV2Model()
	if len(errs) > 0 {
		panic(errs)
	}
	for _, pi := range(v2Model.Model.Paths.PathItems) {
		fmt.Println("Top Level Params")
		for _, p := range(pi.Parameters) {
			fmt.Printf("%+v\n", p)

		}
		fmt.Println("POST Params")
		for _, p := range(pi.Post.Parameters) {
			fmt.Printf("%+v\n", p)
		}
	}
}

The output:

Top Level Params
&{Name:param1 In:query Type:string Format: Description: Required:false AllowEmptyValue:false Schema:<nil> Items:<nil> CollectionFormat: Default:<nil> Maximum:0 ExclusiveMaximum:false Minimum:0 ExclusiveMinimum:false MaxLength:0 MinLength:0 Pattern: MaxItems:0 MinItems:0 UniqueItems:false Enum:[] MultipleOf:0 Extensions:map[] low:0xc000224000}
&{Name:param2 In:query Type:string Format: Description: Required:false AllowEmptyValue:false Schema:<nil> Items:<nil> CollectionFormat: Default:<nil> Maximum:0 ExclusiveMaximum:false Minimum:0 ExclusiveMinimum:false MaxLength:0 MinLength:0 Pattern: MaxItems:0 MinItems:0 UniqueItems:false Enum:[] MultipleOf:0 Extensions:map[] low:0xc000224280}
POST Params
&{Name:param3 In:query Type:string Format: Description: Required:false AllowEmptyValue:false Schema:<nil> Items:<nil> CollectionFormat: Default:<nil> Maximum:0 ExclusiveMaximum:false Minimum:0 ExclusiveMinimum:false MaxLength:0 MinLength:0 Pattern: MaxItems:0 MinItems:0 UniqueItems:false Enum:[] MultipleOf:0 Extensions:map[] low:0xc000224500}

So, the parameters from the GET operation are represented as the pathItem-level parameters.

If we swap operations,

swagger: "2.0"
info:
  version: 1.0.2
  title: Example OpenAPI Spec

host: api.example.com
basePath: /v1
schemes:
  - http


paths:
  /post:
    post:
      parameters:
        - name: param3
          in: query
          type: string
      responses:
        200:
          description: OK

    get:
      parameters:
        - name: param1
          in: query
          type: string
        - name: param2
          in: query
          type: string
      responses:
        200:
          description: OK

and change the POC a bit for _, p := range(pi.Post.Parameters) -> for _, p := range(pi.Get.Parameters), then we get

Top Level Params
&{Name:param3 In:query Type:string Format: Description: Required:false AllowEmptyValue:false Schema:<nil> Items:<nil> CollectionFormat: Default:<nil> Maximum:0 ExclusiveMaximum:false Minimum:0 ExclusiveMinimum:false MaxLength:0 MinLength:0 Pattern: MaxItems:0 MinItems:0 UniqueItems:false Enum:[] MultipleOf:0 Extensions:map[] low:0xc0002a8000}
POST Params
&{Name:param1 In:query Type:string Format: Description: Required:false AllowEmptyValue:false Schema:<nil> Items:<nil> CollectionFormat: Default:<nil> Maximum:0 ExclusiveMaximum:false Minimum:0 ExclusiveMinimum:false MaxLength:0 MinLength:0 Pattern: MaxItems:0 MinItems:0 UniqueItems:false Enum:[] MultipleOf:0 Extensions:map[] low:0xc0002a8280}
&{Name:param2 In:query Type:string Format: Description: Required:false AllowEmptyValue:false Schema:<nil> Items:<nil> CollectionFormat: Default:<nil> Maximum:0 ExclusiveMaximum:false Minimum:0 ExclusiveMinimum:false MaxLength:0 MinLength:0 Pattern: MaxItems:0 MinItems:0 UniqueItems:false Enum:[] MultipleOf:0 Extensions:map[] low:0xc0002a8500}

Thank you!

I've run into a minor bug where an OpenAPI 3 MediaType example is being incorrectly loaded from the low-level model into the high-level model. It is being represented as a low-level node ref. The same example field inside the schema does work correctly.

As a temporary workaround I have done this:

for mt, item := range op.RequestBody.Content {
	// HACK: example is just a low-level node ref which may in fact just
	// be a node representing `null` or the real example.
	if n, ok := item.Example.(low.NodeReference[any]); ok {
		item.Example = n.Value
	}

Here's a simple test to show the bug in action:

$ go run ./bug
Obj {{"foo": "hello"} 0x1400011da40 0x1400011d9a0}             <---- whoops
Schema {"bar": "world"}

Code:

package main

import (
	"fmt"

	"github.com/pb33f/libopenapi"
)

var data = `
openapi: "3.0"
paths:
  /test:
    get:
      responses:
        "200":
          content:
            application/json:
              example: "{\"foo\": \"hello\"}"
              schema:
                type: object
                example: "{\"bar\": \"world\"}"
                properties:
                  foo:
                    type: string
                  bar:
                    type: string
`

func main() {
	doc, err := libopenapi.NewDocument([]byte(data))
	if err != nil {
		panic(err)
	}

	result, errs := doc.BuildV3Model()
	if len(errs) > 0 {
		panic(errs)
	}

	m := result.Model

	fmt.Println("Obj", m.Paths.PathItems["/test"].Get.Responses.Codes["200"].Content["application/json"].Example)

	fmt.Println("Schema", m.Paths.PathItems["/test"].Get.Responses.Codes["200"].Content["application/json"].Schema.Schema().Example)
}

Part of danielgtaylor/restish#115.

https://json-schema.org/understanding-json-schema/reference/array.html

Items can either be a single schema (for v2, v3.0 and v3.1) or a boolean (v3.1).

And along with PrefixItems there should be a Contains field as well which is a single schema along with minContains / maxContains fields.

When following the example code in the README, using v0.3.4 of the library, and the following OpenAPI spec:

openapi: "3.0.0"
info:
  version: 1.0.0
  title: Custom Client Type Example
paths:
  /client:
    get:
      operationId: getClient
      responses:
        200:
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Client"
components:
  schemas:
    Client:
      type: object
      required:
        - name
      properties:
        name:
          type: string

We receive:

Using:

panic: cannot create new document: &{%!e(string=unable to parse specification: yaml: control characters are not allowed)}

Any idea why? I can't see any control characters 🤔