./swagger/specification/eventhub/resource-manager/Microsoft.EventHub/preview/2018-01-01-preview generates:

	internal class ListByEventHubOptions
	{
		[QueryStringName("$skip")]
		[Optional]
		public integer Skip { get; set; }
	
	{
		[QueryStringName("$top")]
		[Optional]
		public integer Top { get; set; }
	
	}

which should be:

	internal class ListByEventHubOptions
	{
		[QueryStringName("$skip")]
		[Optional]
		public int Skip { get; set; }
	
		[QueryStringName("$top")]
		[Optional]
		public int Top { get; set; }
	}

Also:

public override object? OptionsObject()
		{
			return new "ListByNamespace"Options();
		}

Haven't seen this used often, but: https://github.com/Azure/azure-rest-api-specs/blob/c00cf4ad1e2c44ccddb08b98a472cf9822b4d687/specification/mediaservices/resource-manager/Microsoft.Media/stable/2021-05-01/Accounts.json#L81

API's can require values to be sent in Headers - at this point in time the 'Options' functionality (Go Importer & API only supports QueryString values rather than Header values - we should support both

Ultimately this'd need changes to:

• Importer: Parser & this Model
• Data Generator:

  pandora/tools/importer-rest-api-specs/generator/generator.go

  Lines 279 to 281 in b24a506

  	if optionDetails.QueryStringName != "" {
  	optionsCode = append(optionsCode, fmt.Sprintf("\t\t[QueryStringName(%q)]", optionDetails.QueryStringName))
  	}

• API
  • Data Model:

    pandora/data/Pandora.Data/Models/OptionDefinition.cs

    Line 16 in b24a506

    public string QueryStringName { get; set; }

  • Transform Model:

    pandora/data/Pandora.Data/Transformers/Options.cs

    Line 28 in b24a506

    QueryStringName = property.QueryStringName()

  • Presumably a new Attribute so this can be exposed in the Data Models
  • API - Schema: Mapping and Model

Example: MediaServicesCreate (within MediaServices) or NamespaceGet (within Namespace) can become Create and Get respectively

We could probably work around this by making the Request/Response objects return a type rather than an instance of the object, which probably makes more sense anyway, I guess?

Response objects may not always include a reference model for value:

E.g

        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "$ref": "#/definitions/ResourceCollection"
            }
          },

...

    "ResourceCollection": {
      "description": "Collection of resources.",
      "required": [
        "value"
      ],
      "type": "object",
      "properties": {
        "value": {
          "description": "Collection of resources.",
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "nextLink": {
          "description": "Link to next page of resources.",
          "type": "string",
          "readOnly": true
        }
      }
    }

Resulting in:

error: parsing Swagger files: parsing definition: finding resources for tag "AppServicePlans": finding operations: parsing operation "ListWebAppsByHybridConnection": determining response operation for "ListWebAppsByHybridConnection" (method "GET" / uri "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/serverFarms/{name}/hybridConnectionNamespaces/{namespaceName}/relays/{relayName}/sites"): parsing model "ResourceCollection" for list operation to find real model: missing model reference for field 'value'

We've got a selection of common top-level Resource ID's:

• Resource Groups
• Subscriptions
• Management Groups

These should probably go into resourcemanager/resourceids within hashicorp/go-azure-helpers and be referenced - rather then redefining once per package.

Once those are available the references would need to be switched out based, presumably, on exact matches of the ID's?

Turns out these can exist within URI parameters e.g. header/querystring values, which warrants investigating

For example, in specification/timeseriesinsights/resource-manager/Microsoft.TimeSeriesInsights/stable/2020-05-15/timeseriesinsights.json
(line 1484)

    "EnvironmentResource": {
      "type": "object",
      "discriminator": "kind",
      "properties": {
        "sku": {
          "$ref": "#/definitions/Sku",
          "description": "The sku determines the type of environment, either Gen1 (S1 or S2) or Gen2 (L1). For Gen1 environments the sku determines the capacity of the environment, the ingress rate, and the billing rate."
        },
        "kind": {
          "type": "string",
          "description": "The kind of the environment.",
          "enum": [
            "Gen1",
            "Gen2"
          ]
        }
      },
      "required": [
        "kind",
        "sku"
      ],
      "allOf": [
        {
          "$ref": "#/definitions/TrackedResource"
        }
      ],
      "description": "An environment is a set of time-series data available for query, and is the top level Azure Time Series Insights resource."
    },

Right now the Service Definitions are hard-coded, which works for prototyping but won't work when these are generated - instead we'll need to replace the fixed data we have today (Service example / Version example) with a dynamic lookup

This would allow us to add services dynamically by generating them into the namespace and compiling - rather than having to hook these up manually as today.

Right now these aren't great - but they're fairly easy to identify with a debugger, the context is there (namespace, class, field) but needs to be output so these can make sense

We should also add a unit test to catch these

Example:
./specification/web/resource-manager/Microsoft.Web/stable/2020-12-01/WebApps.json

    "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/sites/{name}/functions/admin/token": {
      "get": {
        "tags": [
          "WebApps"
        ],
        "summary": "Fetch a short lived token that can be exchanged for a master key.",
        "description": "Description for Fetch a short lived token that can be exchanged for a master key.",
        "operationId": "WebApps_GetFunctionsAdminToken",
        "parameters": [
          {
            "$ref": "#/parameters/resourceGroupNameParameter"
          },
          {
            "name": "name",
            "in": "path",
            "description": "Name of web app.",
            "required": true,
            "type": "string"
          },
          {
            "$ref": "#/parameters/subscriptionIdParameter"
          },
          {
            "$ref": "#/parameters/apiVersionParameter"
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "schema": {
              "type": "string"
            }
          },
          "default": {
            "description": "App Service error response.",
            "schema": {
              "$ref": "./CommonDefinitions.json#/definitions/DefaultErrorResponse"
            }
          }
        }
      }
    },

This results in a missing implementation for the response model:

The List operations are output with a Predicate method and struct (example) - however at this point that's purely a placeholder and needs implementing - which'll be useful in data sources

This'd also need to be added to the Data Layer / Go SDK Generator too - but these are used in a handful of places (primarily Data Plane) that we should investigate

We'll need to normalise these either in the Generator or the Importer

Looks like this is actually a separate issue where multiple tags exist in the Swagger with the same key, NetWorkRuleSets and NetworkRuleSets:

Originally posted by @tombuildsstuff in #11 (comment)

https://github.com/Azure/azure-rest-api-specs/blob/b39d3318a9cba7203da3d56e4cdc69a7e14d3912/specification/postgresqlhsc/resource-manager/Microsoft.DBforPostgreSQL/preview/2020-10-05-privatepreview/postgresqlhsc.json#L1263-L1267

At this point this is output as boolean and not bool:

namespace Pandora.Definitions.ResourceManager.MediaServices.v2020_05_01.LiveEvents
{
	internal class Create : PutOperation
	{
		public override bool LongRunning()
		{
			return true;
		}

		public override object? RequestObject()
		{
			return new LiveEvent();
		}

		public override ResourceID? ResourceId()
		{
			return new LiveEventId();
		}

		public override object? ResponseObject()
		{
			return new LiveEvent();
		}

		public override object? OptionsObject()
		{
			return new CreateOptions();
		}
	}
	internal class CreateOptions
	{
		[QueryStringName("autoStart")]
		[Optional]
		public boolean AutoStart { get; set; }
	}
}

Turns out Enums can be from non-string types too, so far we've seen Integers and Floats - but presumably this could be Booleans too?

Example Swagger, from Analysis Services (any API version):

"managedMode": {
          "type": "integer",
          "format": "int32",
          "enum": [
            0,
            1
          ],
          "x-ms-enum": {
            "modelAsString": false,
            "name": "ManagedMode"
          },
          "default": 1,
          "description": "The managed mode of the server (0 = not managed, 1 = managed)."
        }

The AutoRest Test Server also shows 2 examples of this, although in this case those are transformed as strings: https://github.com/Azure/autorest.testserver/blob/e1b9f0cee785a27a0a14ff185d7e7989ac40dddf/swagger/non-string-enum.json#L116-L138

Needs to be fixed in:

• Importer (#26)
• Data Generator (#30)
• API (#30)
• Go SDK Generator (#30)

Very much not urgent, but we'll want a schedule trigger daily to update the submodule for the Swagger repository to the latest at some point

This appears to be a bug in the upstream library, where when parsing a recursive/nested model we can get:

parsing file “networkSecurityGroup.json”: flattening swagger file “./specification/network/resource-manager/Microsoft.Network/stable/2020-11-01/networkSecurityGroup.json”: at #/definitions/networkInterfacePropertiesFormat/properties/networkSecurityGroup/properties/properties/properties/securityRules/items, object has no key “networkSecurityGroupOAIGen”

Whilst today we support Discriminators in the Data/API layer and the Go SDK generator - we're not outputting that metadata, which should look like this:

    public abstract class Animal
    {
        [JsonPropertyName("type")]
        [ProvidesTypeHint]
        public string Type { get; set; }
    }

    [ValueForType("cat")]
    public class Cat : Animal
    {
        [JsonPropertyName("isFluffy")]
        public bool IsFluffy { get; set; }
    }

    [ValueForType("dog")]
    public class Dog : Animal
    {
        [JsonPropertyName("barks")]
        public bool Barks { get; set; }
    }

Where Animal is the base/parent class and Cat/Dog are the implementation/child classes here

It appears this is controlled using x-ms-code-generation-settings -> useDateTimeOffset (from ./specification/synapse/data-plane/Microsoft.Synapse/preview/2019-06-01-preview/artifacts.json):

{
  "swagger": "2.0",
  "info": {
    "version": "2019-06-01-preview",
    "title": "ArtifactsClient",
    "x-ms-code-generation-settings": {
      "useDateTimeOffset": true
    }
  },

Whilst the API layer supports outputting other date formats - at this point in time the Data Generator assumes we're outputting RFC3339 dates (and not RFC3339 nano, which it supports - or other formats, which it/the API/Go SDK generator don't at present)

OpenAPI/Swagger definitions for AzureAD are available here: https://github.com/microsoftgraph/msgraph-metadata

Presumably this'll need to be a distinct importer from the Rest API Specs parser, however the Data Generator should be reusable (and can be pulled out into a common library) - but details TBD.

The imports for Constants and Models can vary, for example:

import (
	"context"
	"fmt" // <- not necessary in some circumstances
	"net/http"

	"github.com/Azure/go-autorest/autorest"
	"github.com/Azure/go-autorest/autorest/azure"
	"github.com/hashicorp/go-azure-helpers/polling" // <- not necessary in some circumstances
)

As such we should make these dynamic so they're consistent

The Data/API supports validation (albeit limited to a couple of types right now) - which ultimately allows generated models to expose a .Validate() method which can be used in downstream projects.

We should look to add support for this, first to the Go SDK Generator (to output the existing validation methods we have), for which a stub currently exists - and then the Importer (adding new validation types to the Data Layer and Generator as needed).

Having these validation requirements defined, allows these to be reused in both the Go SDK and Terraform in the future.

Error:

Error: creating Authorization Rule: (Name "acctest-210524135735632414" / Eventhub Name "acctesteventhub-210524135735632414" / Namespace Name "acctesteventhubnamespace-210524135735632414" / Resource Group "acctestRG-210524135735632414"): authorizationruleseventhubs.AuthorizationRulesEventHubsClient#EventHubsCreateOrUpdateAuthorizationRule: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="NoRegisteredProviderFound" Message="No registered resource provider found for location 'westeurope' and API version '2018-01-01-preview' for type 'namespaces/eventhubs/authorizationrules'. The supported api-versions are '2014-09-01, 2015-08-01, 2017-04-01'. The supported locations are ''."

We can query the ARM API's to ensure an API version is available - to ensure we don't output an API which isn't available - we should look to do that.

Whilst we generate Resource ID's, at this point in time we're not generating Validation functions for these as we do in the AzureRM Provider, but we should look into doing so. Alternatively a generic Resource ID validation function which takes a Resource ID instance (including a Parser) could work to the same end.

That said, when generating these we should ensure that these validation functions don't directly rely on the Terraform Plugin SDK as we do today - since this allows us to wrap the Plugin SDK as we do today.

For Constants which are integers or floats we update the keys from numbers to words (e.g. 1 -> One or 1.2 to OnePointTwo) - however there's some string constants which are actually integers/floats, for example:

https://github.com/Azure/azure-rest-api-specs/blob/c00cf4ad1e2c44ccddb08b98a472cf9822b4d687/specification/web/resource-manager/Microsoft.Web/stable/2020-12-01/CommonDefinitions.json#L2678-L2690

Which generates:

As such we should apply the same string replacements to the keys for string constants as we do for integer/float constants

We already do this on a per-package basis, but this likely needs normalising when the Packages are generated

or we update the API to output the Resource ID's used within the Package rather than the API Version, which'd solve this the other way (since these are unique there)

When a response model (Animals) contains a nested model (Animal), which contains an inline model (Dog) defined using additionalProperties - the model Dog isn't pulled out and the field for Dog is an Object rather than a Model reference

In Resource Manager this is an issue for User Assigned Identities for Eventhub Namespaces in 2020-01-01-preview:

https://github.com/Azure/azure-rest-api-specs/blob/a252633188c5e02c4071cc1c3b5cead249db8a54/specification/eventhub/resource-manager/Microsoft.EventHub/preview/2021-01-01-preview/namespaces-preview.json#L757-L777

Totally missed this wasn't wired through - but this needs supporting in:

• Rest API Specs Importer (#27)
• API Data Generator
• API Data / Transforms
• Go SDK Generator

In a small number of cases a List may nest other lists, which may have simple types or other objects as their entries.

Examples:
https://github.com/Azure/azure-rest-api-specs/blob/9875f8eebd8c1392a83c052bc50a79c8251b1174/specification/web/resource-manager/Microsoft.Web/stable/2020-12-01/Diagnostics.json#L1582

https://github.com/Azure/azure-rest-api-specs/blob/9875f8eebd8c1392a83c052bc50a79c8251b1174/specification/web/resource-manager/Microsoft.Web/stable/2020-12-01/CommonDefinitions.json#L781

Models nested within other models (which reference other models which may do the same) aren't always being pulled out - so that'll need fixing

Constants shouldn't be output as Optional fields in the data (e.g. ConstantName?) but instead should be the concrete type (ConstantName) - the presence of the [Required] tag or not allows these to be defined without relying on the C# type information

Although separately we should re-evaluate that, since we should be able to use that information (and update the translation/reflection layer) now that models and mappings are separate

https://github.com/Azure/azure-rest-api-specs/blob/9875f8eebd8c1392a83c052bc50a79c8251b1174/specification/appconfiguration/data-plane/Microsoft.AppConfiguration/stable/1.0/appconfiguration.json#L206-L210

We should pull this out too: https://github.com/Azure/azure-rest-api-specs/blob/b39d3318a9cba7203da3d56e4cdc69a7e14d3912/specification/postgresqlhsc/resource-manager/Microsoft.DBforPostgreSQL/preview/2020-10-05-privatepreview/postgresqlhsc.json#L1304-L1313

As a part of generating the data we should generate a Service Definition:

using System.Collections.Generic;
using Pandora.Definitions.Interfaces;

namespace Pandora.Definitions.ResourceManager.EventHub
{
    public class Service : ServiceDefinition
    {
        public string Name => "EventHub";
        public bool Generate => true;
        public string? ResourceProvider => "Microsoft.EventHub";
    }
}

.. and an API Version Definition:

using System.Collections.Generic;
using Pandora.Definitions.Interfaces;

namespace Pandora.Definitions.ResourceManager.EventHub.v2018_01_01_preview
{
    public class Definition : ApiVersionDefinition
    {
        public string ApiVersion => "2018-01-01-preview";
        public bool Generate => true;
        public bool Preview => true;
        
        public IEnumerable<ApiDefinition> Apis => new List<ApiDefinition>
        {
            new AuthorizationRulesDisasterRecoveryConfigs.Definition(),
            new AuthorizationRulesEventHubs.Definition(),
            new AuthorizationRulesNamespaces.Definition(),
            new CheckNameAvailabilityDisasterRecoveryConfigs.Definition(),
            new ConsumerGroups.Definition(),
            new DisasterRecoveryConfigs.Definition(),
            new EventHubs.Definition(),
            new EventHubsClusters.Definition(),
            new EventHubsClustersConfiguration.Definition(),
            new EventHubsClustersNamespace.Definition(),
            new IpFilterRules.Definition(),
            new Namespaces.Definition(),
            new NamespacesPrivateEndpointConnections.Definition(),
            new NamespacesPrivateLinkResources.Definition(),
            new NetworkRuleSets.Definition(),
            new VirtualNetworkRules.Definition(),
        };
    }
}

.. and the API Definitions:

using System.Collections.Generic;
using Pandora.Definitions.Interfaces;

namespace Pandora.Definitions.ResourceManager.EventHub.v2018_01_01_preview.EventHubs
{
    internal class Definition : ApiDefinition
    {
        public string ApiVersion => "2018-01-01-preview";
        public string Name => "EventHubs";
        public IEnumerable<ApiOperation> Operations => new List<ApiOperation>
        {
            new CreateOrUpdate(),
            new Delete(),
            new DeleteAuthorizationRule(),
            new Get(),
            new GetAuthorizationRule(),
            new ListByNamespace(),
        };
    }
}

Longer-term we'll want to use partial classes to toggle the generation flags as needed, but that's a future problem. that should be addressed at this time

Example:

namespace Pandora.Definitions.ResourceManager.Network.v2020_08_01.NetworkInterfaces
{
	[ConstantType(ConstantTypeAttribute.ConstantType.String)]
	internal enum SecurityRuleProtocol
	{
		[Description("*")]
		*,

		[Description("Ah")]
		Ah,

		[Description("Esp")]
		Esp,

		[Description("Icmp")]
		Icmp,

		[Description("Tcp")]
		Tcp,

		[Description("Udp")]
		Udp,
	}
}

Special case substitute for Any ?

Should continue here.

Resources containing a user assigned identity output an object in the format Dictionary<string, SomeObject> which we need to support across different objects

Ultimately it'd be nice to use the aliases when outputting these, if possible: https://github.com/terraform-providers/terraform-provider-azurerm/tree/master/azurerm/internal/identity

Some enum description strings contain invalid chars, e.g

These need to be replaced, either with _ or, perhaps if the cases are few in number, + -> Plus ?

This one didn't:

using Pandora.Definitions.Operations;
using System.Collections.Generic;
using System.Net;
using Pandora.Definitions.Interfaces;

namespace Pandora.Definitions.ResourceManager.EventHub.v2017_04_01.AuthorizationRulesDisasterRecoveryConfigs
{
	internal class DisasterRecoveryConfigsListAuthorizationRules : GetOperation
	{
		public override string? FieldContainingPaginationDetails()
		{
			return "nextLink";
		}

		public override ResourceID? ResourceId()
		{
			return new DisasterRecoveryConfigId();
		}

		public override object? ResponseObject()
		{
			return new AuthorizationRuleListResult();
		}

		public override string? UriSuffix()
		{
			return "/authorizationRules";
		}
	}
}

All of the known []required values should be appended together to give a canonical type for a list - in Postgres ServerPropertiesForCreate has createMode as required but it's not for ServerPropertiesForDefaultCreate

Since these are now located within the SDK packages, we should prefix the ID parsing methods with Parse so these are more easily discoverable via autocomplete

It's possible for multiple operations within a package to have Options classes (and/or for those to be defined constants/models) - as such these can conflict when generating, for example:

using Pandora.Definitions.Attributes;
using Pandora.Definitions.Interfaces;
using Pandora.Definitions.Operations;
using System.Collections.Generic;
using System.Net;

namespace Pandora.Definitions.ResourceManager.Compute.v2021_03_01.VirtualMachines
{
	internal class Delete : DeleteOperation
	{
		public override IEnumerable<HttpStatusCode> ExpectedStatusCodes()
		{
			return new List<HttpStatusCode>
			{
				HttpStatusCode.Accepted,
				HttpStatusCode.NoContent,
				HttpStatusCode.OK,
			};
		}

		public override bool LongRunning()
		{
			return true;
		}

		public override ResourceID? ResourceId()
		{
			return new VirtualMachineId();
		}

		public override object? OptionsObject()
		{
			return new DeleteOptions();
		}
	}
	internal class DeleteOptions
	{
		[QueryStringName("forceDeletion")]
		[Optional]
		public bool ForceDeletion { get; set; }
	}
}

can become:

using Pandora.Definitions.Attributes;
using Pandora.Definitions.Interfaces;
using Pandora.Definitions.Operations;
using System.Collections.Generic;
using System.Net;

namespace Pandora.Definitions.ResourceManager.Compute.v2021_03_01.VirtualMachines
{
	internal class Delete : DeleteOperation
	{
		public override IEnumerable<HttpStatusCode> ExpectedStatusCodes()
		{
			return new List<HttpStatusCode>
			{
				HttpStatusCode.Accepted,
				HttpStatusCode.NoContent,
				HttpStatusCode.OK,
			};
		}

		public override bool LongRunning()
		{
			return true;
		}

		public override ResourceID? ResourceId()
		{
			return new VirtualMachineId();
		}

		public override object? OptionsObject()
		{
			return new DeleteOptions();
		}

		internal class DeleteOptions
		{
			[QueryStringName("forceDeletion")]
			[Optional]
			public bool ForceDeletion { get; set; }
		}
	}
}

Which should work around this - Compute (2021-03-01) has several examples of this:

 $ cd data/Pandora.Definitions.ResourceManager/Compute/v2021_03_01/VirtualMachines/
 $ grep -R "DeleteOptions" .
./Model-VirtualMachineNetworkInterfaceConfigurationProperties.cs:		public DeleteOptions? DeleteOption { get; set; }
./Model-NetworkInterfaceReferenceProperties.cs:		public DeleteOptions? DeleteOption { get; set; }
./Model-VirtualMachinePublicIPAddressConfigurationProperties.cs:		public DeleteOptions? DeleteOption { get; set; }
./Constant-DeleteOptions.cs:	internal enum DeleteOptions
./Operation-Delete.cs:			return new DeleteOptions();
./Operation-Delete.cs:	internal class DeleteOptions

Since Resource ID's can be present in multiple packages within an API version - we should consider generating these into an ids package and aliasing them into each package, so they can be referenced across them - rather than being re-defined (and thus different types)