Git Product home page Git Product logo

3scale_toolbox's Introduction

3scale toolbox

CircleCI License GitHub release codecov

Description

3scale toolbox is a set of tools to help you manage your 3scale product. Using the 3scale API Ruby Client.

Table of contents

Requirements

Supported Ruby interpreters

  • MRI 3.0

Installation

Install the toolbox:

$ gem install 3scale_toolbox

The 3scale toolbox packaging repo provides packages and installation/deployment steps for the following platforms:

  • CentOS/Fedora
  • Ubuntu/Debian
  • Mac OS X
  • Windows
  • Docker
  • Kubernetes / Openshift

Usage

$ 3scale help
NAME
    3scale - 3scale toolbox

USAGE
    3scale <sub-command> [options]

DESCRIPTION
    3scale toolbox to manage your API from the terminal.

COMMANDS
    account              account super command
    activedocs           activedocs super command
    application          application super command
    application-plan     application-plan super command
    backend              backend super command
    copy                 copy super command
    help                 show help
    import               import super command
    method               method super command
    metric               metric super command
    policies             policies super command
    policy-registry      policy-registry super command
    product              product super command
    proxy                proxy super command
    proxy-config         proxy-config super command
    remote               remotes super command
    service              services super command

OPTIONS
    -c --config-file=<value>      3scale toolbox configuration file (default:
                                  $HOME/.3scalerc.yaml)
       --disable-keep-alive       Disable keep alive HTTP connection mode
    -h --help                     show help for this command
    -k --insecure                 Proceed and operate even for server
                                  connections otherwise considered insecure
    -v --version                  Prints the version of this command
       --verbose                  Verbose mode

Development

After checking out the repo, run bin/setup to install dependencies. You can also run bin/console for an interactive prompt that will allow you to experiment. Run bundle exec 3scale to use the gem in this directory, ignoring other installed copies of this gem.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Testing

To run all tests run rake.

There are two kinds of tests:

rake spec:unit
rake spec:integration

Integration tests can be run locally or against a real 3scale account. When details of the account are set via environment variables, integration tests are run against the given account. Otherwise, tests are run locally with mocked 3scale clients.

The easiest way to set everything up is it to have a .env file in the root of the project with the following environment variables (set your own values):

ENDPOINT=https://your-domain-admin.3scaledomain
PROVIDER_KEY=abc123
VERIFY_SSL=true (by default true)

Develop Core Command

Very simple core command to list existing services. Helps to illustrate basic command code structure and helper methods to deal with remotes.

$ cat lib/3scale_toolbox/commands/service_list_command.rb
module ThreeScaleToolbox
  module Commands
    class ServiceListCommand < Cri::CommandRunner
      include ThreeScaleToolbox::Command

      def self.command
        Cri::Command.define do
          name        'service_list'
          usage       'service_list <3scale_remote>'
          summary     'service list'
          description 'list available services'
          param       :remote
          runner ServiceListCommand
        end
      end

      def run
        puts threescale_client(arguments[:remote]).list_services
      end
    end
  end
end

A few things worth highlighting:

  • Your module must include the ThreeScaleToolbox::Command module. It allows your command to be added to the toobox command tree.
  • You must implement the command module function and return an instance of Cri::Command from cri
  • threescale_client helper method returns 3scale API client instance. All the process remote parsing, fetching from the remote list and client instantiation is done out of the box.

Then register the core command in lib/3scale_toolbox/commands.rb

--- a/lib/3scale_toolbox/commands.rb
+++ b/lib/3scale_toolbox/commands.rb
@@ -4,6 +4,7 @@ require '3scale_toolbox/commands/copy_command'
 require '3scale_toolbox/commands/import_command'
 require '3scale_toolbox/commands/remote_command'
+require '3scale_toolbox/commands/service_list_command'

 module ThreeScaleToolbox
   module Commands
@@ -12,7 +13,8 @@ module ThreeScaleToolbox
       ThreeScaleToolbox::Commands::CopyCommand,
       ThreeScaleToolbox::Commands::ImportCommand,
-      ThreeScaleToolbox::Commands::RemoteCommand::RemoteCommand
+      ThreeScaleToolbox::Commands::RemoteCommand::RemoteCommand,
+      ThreeScaleToolbox::Commands::ServiceListCommand
     ].freeze
   end
 end

Running the new core command:

$ 3scale service_list my-3scale-instance
{ ... }

Licenses

It is a requirement that we include a file describing all the licenses used in the product, so that users can examine it.

Run rake license_finder:check to check licenses when dependencies change.

Run rake license_finder:report > licenses.xml to update licenses file.

Plugins

As of 3scale Toolbox 0.5.0, 3scale Toolbox will load plugins installed in gems or $LOAD_PATH. Plugins are discovered via Gem::find_files then loaded. Install, uninstall and update plugins using tools like RubyGems and/or Bundler.

Make your own plugin

Troubleshooting

  • SSL errors: If you run into SSL issues with the toolbox, you can take actions to resolve them.

Contributing

If you are interested in contributing to 3scale Toolbox, please refer to instructions available here

3scale_toolbox's People

Contributors

abarrak avatar andrewdavidmackenzie avatar dependabot[bot] avatar eguzki avatar jillisterhove-rubix avatar jjkiely avatar kevprice83 avatar leathersole avatar macejmic avatar mayorova avatar miguelsorianod avatar mikz avatar nmasse-itix avatar orimarti avatar poojachandak avatar velemas avatar y-tabata avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

3scale_toolbox's Issues

Docker Image

Create a docker image for easy utilization of this component.

copying to the same Target returns error

One of the typical usage of this tool is to make an exact copy of a service inside the same instance to a new service (example: sandbox and production API).
At the moment this is not working:
command launched:
3scale copy service 2555417745587 --source=https://[email protected] --destination=https://[email protected]

error returned:

/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': /usr/local/share/gems/gems/3scale-api-0.1.4/lib/3scale/api.rb:8: syntax error, unexpected ',' (SyntaxError)
    def self.new(endpoint:, provider_key:)
                           ^
/usr/local/share/gems/gems/3scale-api-0.1.4/lib/3scale/api.rb:14: syntax error, unexpected keyword_end, expecting end-of-input
        from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/share/gems/gems/3scale_toolbox-0.2.2/exe/3scale-copy:73:in `<main>'

RFE: Set the integration settings when importing a service from an OpenAPI Specification

Current Situation

Currently, when importing a new service from an OpenAPI with:

3scale import openapi -t echo_api -d https://$ACCESS_TOKEN@$TENANT-admin.3scale.net examples/echo-api.yaml

The service is created but the integration settings are left empty and the service is not ready to use.

Proposed change

Re-use the host and schemes fields of the OpenAPI Specification to set the private base URL.
Re-use the standard securityScheme and securityRequirements fields to set the correct credentials location (query string or headers + header / parameter name).

Also set the OpenID Connect issuer endpoint either from the OpenAPI Specification or from the command line.

Ideally, those values should be taken from the OpenAPI Specification as default value that can be overridden from the command line.

Option: accept a command line option to strip or replace the basePath field with custom value (hence setting the url_rewritting policy in the policy chain).

Business justification

If a service is left half-provisioned, the service is not usable and someone has to come finished the configuration manually or write a higher-level script.

RFE: API semantic versioning

Version numbering allows the clear identification of built artefacts and their compatibility to their respective consumers. Semantic versioning’ which is based on 3 numbers and an alphanumeric string:
[major] . [minor] . [patch]

  • The major number is bumped up when an incompatible (ie. breaking) change has been made to the api from the previous version. So any developer planning to use this library can easily see that a breaking change has been included.
  • The minor version is bumped up when a change has been made to the functionality within the artefact that is still backwards compatible. ie. the api has not been changed however the functionality is now different to previous.
    I would focus on managing Major and Minor releases, that implies:
  1. Major releases (2.0.0) : if toolbox detects a previous major release in 3scale, for instance already exists the release 1.x then -> Create a new services and import service specification.
  2. Minor releases (1.0.0) : if toolbox detects this is minor upgrade, then rename older service in 3scale and reimport the Active Docs.

Authentication mode is not copied

Not sure if this is an issue or an enhancement request, but when copying services the authentication mode is not taken into account, i.e it always stays as user_key even if the service being copied has app_id/app_key auth mode

Command to create new tenant

Would be great to have a CLI to create a new Tenant and get admin Access Token and URL.

Will be needed for APIcast testing.

/cc @eguzki
/cc @3scale/qe

3scale update service errors

When running the 3scale update service command I receive the following error:

3scale update service 2555417742467 2555417759842 -s https://[email protected] -d https://[email protected]
updating service settings for service id 2555417759842...
Traceback (most recent call last):
        2: from /home/rhuser/.gem/ruby/gems/3scale_toolbox-0.4.0/exe/3scale-update:259:in `<main>'
        1: from /home/rhuser/.gem/ruby/gems/3scale_toolbox-0.4.0/exe/3scale-update:235:in `update_service'
/home/rhuser/.gem/ruby/gems/3scale_toolbox-0.4.0/exe/3scale-update:106:in `copy_service_settings': undefined method `update_service' for #<ThreeScale::API::Client:0x000056014c987b70> (NoMethodError)
Did you mean?  update_proxy

If I pass the --rules-only flag the update seems to work ok.

Framework to make it easy adding new commands

Avoid argument duplication in new commands, leverage general commands and methods to set tokens, endpoints.

  • Fully POSIX-compliant flags
  • Nested subcommands
  • Easy generation of commands
  • Modular system based on plugins

copy service "-t" argument is confusing

Documentation for "copy service" makes it appear as though the "-t" or "--target-system-name" argument is optional, but it appears to be required. The purpose of this argument is also not very clear from the documentation.

It would be best if this argument was made optional and the tool re-used the system name from the source service by default. The argument could then be used to override that behavior.

At the very least, please update the documentation to reflect the current behavior of the tool.

verbose mode

It will be great to have verbose mode so user can watch command progress or/and operations/api calls done during command execution. I think this feature is essential for help user by another person in case of any troubles.

error messages

There should be error message with explanation if command failed. Also return code should be non-zero in case of failure or error.

Command to import service using OpenAPI / Swagger spec or RAML format

Using an API definition format like OpenAPI / Swagger or RAML you can automate some setups and save a lot of time in configuration.

When you import an API defintion, the following actions will be performed in the background:

  1. Create a new service (unless you specify one)
  2. Create methods in the 'Definition' section
  3. Attach newly created methods to the 'Hits' metric
  4. Create mapping rules and show them under API > Integration

cannot duplicate a service

when trying to make a copy of the service on the same instance I receive the following error

/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- json (LoadError)
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /home/luferrar/.gem/ruby/2.4.0/gems/3scale-api-0.1.5/lib/3scale/api/http_client.rb:1:in `<top (required)>'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
	from /home/luferrar/.gem/ruby/2.4.0/gems/3scale-api-0.1.5/lib/3scale/api.rb:9:in `new'
	from /home/luferrar/.gem/ruby/2.4.0/gems/3scale_toolbox-0.3.0/exe/3scale-copy:77:in `<main>'

the command I ran was
3scale copy service 2555417752749 --source=https://[provider-key]@psd2-env-admin.3scale.net --destination=https://[provider-key]@psd2-env-admin.3scale.net

ruby version is 2.4.2p198

complete duplication of a service

the following settings are not copied when copying a service:
the base path, integration mode and API authentication are not copied to the new service.

Ideally the tool would be used to have a full duplicate and then just modify the staging and production endpoints.

RFE: Create an ActiveDocs with updated environmental fields when importing a service from OpenAPI

Current Situation

When importing a service from an OpenAPI Specification, using:

3scale import openapi -t echo_api -d https://$ACCESS_TOKEN@$TENANT-admin.3scale.net examples/echo-api.yaml

The service is created but no ActiveDocs is created.

Proposed Changed

The OpenAPI Specification used to create the service is pushed as an ActiveDocs file, using the same system_name as the service.

This OpenAPI Specification is patched to take into account the new environment:

  • the host field is updated with the hostname component of the public production base URL
  • the schemes field is updated with the scheme component of the public production base URL
  • the basePath field is updated if rewriting rules have been set
  • the securityDefinitions/*/authorizationUrl and securityDefinitions/*/tokenUrl fields are updated with the token and authorize endpoints listed in the OpenID Connect wellknown endpoint (when OIDC is in use)

Business need

Currently, the missing ActiveDocs implies for the customer to have a tedious manual step to update the OpenAPI Spec and upload it. This is needed to achieve a "ready-to-use" API on the Developer Portal.

This is something I'm currently doing in the Ansible playbooks.

Login command to manage remotes

Command to

  • Add remote (currently provider key and provider URI)
  • List remotes
  • Show remote info
  • Remove remote
  • Rename remote

print usage when missing subcommand

Current situation

When you use a command and forget a required subcommand, say:

$ 3scale import

You get a "dry" response that doesn't help much.

import: no command given

So, you need to ask for help:

$ 3scale help import
NAME
    import - 3scale import command

USAGE
    3scale import <command> [options]

DESCRIPTION
    3scale import command.

SUBCOMMANDS
    csv     Import csv file

OPTIONS FOR 3SCALE
    -h --help          show help for this command
    -k --insecure      Proceed and operate even for server connections
                       otherwise considered insecure
    -v --version       Prints the version of this command

This helps a lot more and now I know that import requires a sub-command, such as csv

Proposed Change

When a command requires a subcommand and it is not supplied, print out the help for that command directly.

to_h method undefined when copy a service

Hello team,

I've encountered the following error while trying to run the copy command on a service.

"to_h" method is undefined.

It worked after removing "to_h" from "end.to_h" @ Line 167 in 3scale-copy file (3scale_toolbox-0.2.2/exe)

Cannot import OpenAPI that has vendor extensions

Current Situation

Currently, when importing a new service from an OpenAPI with:

3scale import openapi -t echo_api -d https://$ACCESS_TOKEN@$TENANT-admin.3scale.net examples/echo-api.yaml

If that OpenAPI file has vendor extensions in it, the 3scale import openapi command fails with:

Error: OpenAPI schema validation failed: Cannot coerce property "info" from Hash to Swagger::V2::Info: The property 'x-threescale-system-name' is not defined for Swagger::V2::Info.

The ability to use vendor extensions is allowed in most places of the OpenAPI Specifications (See Vendor Extensions and for this specific example, the infoObject).

See this OpenAPI file for example: echo-api.yaml.

Proposed Changes

Any vendor extension in the OpenAPI file should be accepted by the 3scale toolbox since it is part of the OpenAPI Specification.

Copying services: ability to set the system name of the target service

Problem

When copying a service, the system name is read only and if you do it within the same account, you have two services with the same system name.

Suggested solution

Add a command line switch --target-system-name=. The other fields can be changed manually as they're not read only.

Notes

Right now when services are copied to a different account, the system_name is the same as the original (without "_copy").

RFE: Have a standard way to report errors

The 3scale toolbox is being improved to become the recommended way to configure and deploy services in 3scale. This fits in a broader approach of Continuous Delivery that has been started by several customers / communities (I personally maintain the threescale-cicd ansible role. Those approaches are using different means to drive the 3scale admin portal, some are using the 3scale-cli other (like the threescale-cicd role) are using the 3scale REST APIs.

If we want to make the 3scale_toolbox the de facto standard for interacting with 3scale, we need it to be usable in a CI/CD approach.

At a glance, here are the features I expect from the 3scale toolbox :

  • #81: Is idempotent
  • #82: Works with system_name and not service_id
  • #83: Can output human readable information as well as json or yaml (akin to oc cmd -o yaml)
  • #80: Have a standard way to report errors

A standard way to report errors

Since the 3scale toolbox will be used from scripts / pipelines / etc, any error must be reported without any ambiguity.

Proposition

The plugins have a standard way to report errors, the unix return code is set appropriately and the output can be parsed easily by a script (JSON / YAML for instance).

Examples

If the requested operation cannot be completed, the Unix return code is > 1 and if the operation has been completed successfully the Unix return code is 0.
If the requested operation cannot be completed, the output would look like :

{
  "error_code": "E_CANNOT_READ_OPENAPI",
  "error_message": "Cannot parse OpenAPI Specs: missing field bla bla bla",
  "kind": "local",
  "stacktrace": "..."
}

This feature needs to be in the toolbox's core so that every plugin can leverage this feature.

RFE: Command to export/import Application Plan limits, pricing rules and features

Current Situation

Currently, when importing a new service from an OpenAPI with:

3scale import openapi -t echo_api -d https://$ACCESS_TOKEN@$TENANT-admin.3scale.net examples/echo-api-oidc.yaml

The metrics / methods and mapping rules are created but the limits / pricing rules / features that compose an application plan are not created.

Proposed changes

A new command is introduced to import application plan artefacts from a YAML file.

3scale import application-plan -p my_app_plan -t echo_api -f /path/to/app-plan.yaml -d https://$ACCESS_TOKEN@$TENANT-admin.3scale.net examples/echo-api-oidc.yaml

A new command is introduced to export application plan artefacts from a YAML file.

3scale export application-plan -p my_app_plan -t echo_api -f /path/to/app-plan.yaml -d https://$ACCESS_TOKEN@$TENANT-admin.3scale.net examples/echo-api-oidc.yaml

Usage in a CI/CD pipeline

Upon commit in the GIT repository, the CI/CD pipeline would :

  1. call the 3scale import openapi command to create / update the service
  2. call the 3scale import application-plan command to create / update the application plans from the artefact file (if provided in the GIT repo)

If the limits / pricing rules are not in the hand of the developers (maybe something more in the hands of the product owner), the CI/CD pipeline would be configured to stop after the first environment if there is no artefact file, if it is not up-to-date or if the 3scale service has no up-to-date application plan.

The product owner can connect to the 3scale tenant and fill-in the limits, pricing rules and features using the Web Interface.

He then can re-launch the CI/CD pipeline and this time the CI/CD pipeline would export the application plan artefacts from the first env into a file using the 3scale export application-plan, commit the file to the repo, and deploy the API in all the subsequent environments using:

  1. call the 3scale import openapi command to create / update the service
  2. call the 3scale import application-plan command to create / update the application plans from the artefact file

Business justification

Currently, application plans are on the critical path to use an API (without them, the API is not usable). So the Application Plan deployment should be automated.

RFE: Set the correct security scheme when importing an OpenAPI Specification

Current Situation

Currently, when importing a new service from an OpenAPI with:

3scale import openapi -t echo_api -d https://$ACCESS_TOKEN@$TENANT-admin.3scale.net examples/echo-api-oidc.yaml

The service is created with API Key security even if the OpenAPI Specification mandates OpenID Connect and I found no specific way to override this from the command line.

See this sample OpenAPI Specification: echo-api-oidc.yaml.

Proposed Change

The 3scale toolbox looks for the securityRequirements and securitySchemes fields to determine the kind of security to apply and sets it accordingly when creating or updating the service.

The current logic that I'm using in the Ansible playbooks are:

  • I'm looking for the global security requirements and I require to have one and only one requirement
  • I'm searching for the matching security scheme
  • I'm determining whether the scheme is API Key or OIDC (the two we are supporting in 3scale)

If there is no security requirement, we can consider it as an "Open API" and add the default_credentials policy to the policy chain.

Business Justification

One of our customer is using exclusively OpenID Connect APIs and thus cannot use the 3scale toolbox to provision its APIs.

RFE: Provide a parseable output (JSON / YAML)

The 3scale toolbox is being improved to become the recommended way to configure and deploy services in 3scale. This fits in a broader approach of Continuous Delivery that has been started by several customers / communities (I personally maintain the threescale-cicd ansible role. Those approaches are using different means to drive the 3scale admin portal, some are using the 3scale-cli other (like the threescale-cicd role) are using the 3scale REST APIs.

If we want to make the 3scale_toolbox the de facto standard for interacting with 3scale, we need it to be usable in a CI/CD approach.

At a glance, here are the features I expect from the 3scale toolbox :

  • #81: Is idempotent
  • #82: Works with system_name and not service_id
  • #83: Can output human readable information as well as json or yaml (akin to oc cmd -o yaml)
  • #80: Have a standard way to report errors

Outputting JSON or YAML

Since the 3scale toolbox will be used from scripts / pipelines / etc, its output must be easily parseable by a program without any ambiguity. 

Proposition

Akin to the "oc" or "kubectl" command, 3scale toolbox output can be changed by specifying a flag in the command line. 

Exemple

By default, the 3scale toolbox output human readable information

$ 3scale-toolbox services
my-service1 (My Service 1)
my-service2 (My Service 2)
...

But a YAML output can be used instead

$ 3scale-toolbox services -o yaml
services:
- system_name: my-service1
  name: My Service 1
- system_name: my-service2
  name: My Service 2

This feature needs to be in the toolbox's core so that every plugin can leverage this feature.

Ideas for future commands to add (for discussion)

Not intended to be implemented, just interested in hearing what commands people would find useful in the toolbox. Then some of us could volunteer to try implementing some of them, or prototype them at least....contributing to grow our own toolbox...

The idea is that these would be useful for us and also for providers.

"3scale login" (like heroku toolbelt). It takes provider_key/token and domain, and stores (in ENV? in CWD?) who you are as a provider, and you don't need to send provider key and domain with each command?

"3scale account" Dumps information about the account identified via key/domain. When created, admin and member users, etc.

"3scale whois" Takes an identified (any of user_key?, email? name?) and uses the API to tell you who that is...User info, on what account, a provider user or developer etc... May involve searching using multiple methods, depending if it can distringuish what type of key was provided.

"3scale invoices" Return a set of invoices for a specified time period, in cvs/json/xml format.

"3scale stats" Return data on defined metrics for a given service & time period in csv/json/xml format.

RFE: Working with system_name and not id

The 3scale toolbox is being improved to become the recommended way to configure and deploy services in 3scale. This fits in a broader approach of Continuous Delivery that has been started by several customers / communities (I personally maintain the threescale-cicd ansible role. Those approaches are using different means to drive the 3scale admin portal, some are using the 3scale-cli other (like the threescale-cicd role) are using the 3scale REST APIs.

If we want to make the 3scale_toolbox the de facto standard for interacting with 3scale, we need it to be usable in a CI/CD approach.

At a glance, here are the features I expect from the 3scale toolbox :

  • #81: Is idempotent
  • #82: Works with system_name and not service_id
  • #83: Can output human readable information as well as json or yaml (akin to oc cmd -o yaml)
  • #80: Have a standard way to report errors

Working with system_name and not id

Currently, the 3scale_toolbox and the underlying 3scale APIs are using numerical IDs to reference the object to work on. At the same time, a system_name is present on each object with a human readable name and a unique constraint. As a user, I know that I want to update the service "my-service" but using the numerical ID is harder. 

In an infrastructure as code approach, the state of the system is determined by a formal description. The system can be completely determined by this description. When using numerical id, it is not anymore possible since the description would depends on the state of the system.

Last but not least, customers that have a separate installation of 3scale for their DRP site have different id's for the same service on both site. This means the CI/CD pipeline that deploys the service on the main site and on the DRP site needs to either do the discovery by itself or have a mapping table.

Proposition

The 3scale_toolbox uses the system_name instead of the numerical id to reference the object to work on. Then it can translate the system_name to the numerical id to finally call the underlying 3scale API.

Example

3scale copy service -t my-prod-api -s dev -d prod my-dev-api

This feature needs to be in the toolbox's core so that every plugin can leverage this feature.

Cannot import mapping rules to services

Followed the master dev instructions https://github.com/3scale/3scale_toolbox/blob/master/docs/import.md

After I ran the command on my test account, it created a service with methods, but not mapping rules

Command:
bundle exec exe/3scale import csv --destination=https://<provider_key>@foobar-admin.3scale.net --file=import_endpoints.csv

Log:
Method v1_Taxing_RecalculateTax has been created.
Method v1_Lead_Leads has been created.
Method v1_Products_GetProducts has been created.
Method v1_Products_GetCompanyList has been created.
Method SupplierInfo_Supplier has been created.
Method v2_Product_GetProductEligibility has been created.
....
....
....
....

1 services in CSV file
1 services have been created
0 metrics have been created
25 methods have beeen created

Link to download the import_endpoints.csv

Expected result:
Create mapping rules and map that to http method name (Verb) and application plan method name (Metric or Method) in Integrations tab.

Dry run option on commands

Implementation of dry-run mode on commands with heavy write operations.

Running potentially huge import with no option to roll back could have a devastating impact on someone's portal.

The dry mode could be "read only mode". Just read what is the current state and print out the plan (... to be deleted, ... to be changed from .. to ...).

Redesign of import command

After extensive brainstorming we determined that a lower level command is more useful and simpler in 80-90% of the use cases.

3scale import-(methods|metrics)
--destination https://<provide_key>@<admin_domain>-admin.3scale.net
--file <path_to_CSV>
--service (service-id|service-name)
[--force-delete] without this flag metric deletions will be prompted for confirmation

Example CSV file (commas not shown):

url_path http_method [system-name] [friendly-name]
/movies/biography GET movies_biography Movies (Biography)
/movies/drama GET movies_drama Movies (Drama)

http_method <GET | POST | PUT | DELETE | PATCH> in future add support for wildcard "*" and combos with "+"
[system-name] default will be to use url_path
[friendly-name] default will be to use url_path

Desired behavior if data already exists in the destination service in the 3scale account:

  • Metrics/methods:
    • New metrics/methods: always append
    • Update existing metric: ignore but notify as "Duplicate metric <system_name>, original metric definition unchanged"
    • Delete existing metric (i.e. previously the metric existed and in the new file it does not exist): If ´--force-delete´ option is not specified, prompt "The following metrics will be deleted: Confirm to delete? (Y/N)"
  • Mapping rules
    • If any mapping rules exist and ´--force-delete´ is not specified then prompt "There are existing mapping rules . Confirm to delete? (Y/N)"
    • ...then ALL existing mapping rules would be deleted and replaced with the new mappings in the CSV

The existing 3scale import command can be renamed to ´3scale import-extended´ with the same params and the file format changed to be more consistent with the format described here.

export command

I think will be great to have export commands for all available import commands so user can export from one 3scale instance and import to another one.

SSL Error while copying the service

I am trying to copy the service from one environment to another environment running the similar command :
3scale copy service NUMBER --source=https://[email protected] --destination=https://[email protected]

I am getting the following error :
<-----
5.2/bin/ in PATH, mode 040777
Traceback (most recent call last):
9: from /usr/local/lib/ruby/gems/2.5.0/gems/3scale_toolbox-0.4.0/exe/3scale-copy:87:in <main>' 8: from /usr/local/lib/ruby/gems/2.5.0/gems/3scale-api-0.1.5/lib/3scale/api/client.rb:16:in show_service'
7: from /usr/local/lib/ruby/gems/2.5.0/gems/3scale-api-0.1.5/lib/3scale/api/http_client.rb:34:in get' 6: from /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/net/http.rb:1213:in get'
5: from /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/net/http.rb:1455:in request' 4: from /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/net/http.rb:909:in start'
3: from /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/net/http.rb:920:in do_start' 2: from /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/net/http.rb:981:in connect'
1: from /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/net/protocol.rb:44:in ssl_socket_connect' /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/net/protocol.rb:44:in connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain) (OpenSSL::SSL::SSLError)

----->

I am using clusterup OCP environment (self-signed certificates). Any suggestions on how to fix this?

RFE: Add ability to update certain parts of a service

Right now there is a --rules-only flag for 3scale update service which normally works great, but if a method exists in source but not in the destination it will throw an error:

the source service has 2 mapping rules
the target has 0 mapping rules
missing 2 mapping rules
Traceback (most recent call last):
        4: from /home/rhuser/.gem/ruby/gems/3scale_toolbox-0.4.0/exe/3scale-update:257:in `<main>'
        3: from /home/rhuser/.gem/ruby/gems/3scale_toolbox-0.4.0/exe/3scale-update:226:in `copy_mapping_rules'
        2: from /home/rhuser/.gem/ruby/gems/3scale_toolbox-0.4.0/exe/3scale-update:226:in `each'
        1: from /home/rhuser/.gem/ruby/gems/3scale_toolbox-0.4.0/exe/3scale-update:228:in `block in copy_mapping_rules'
/home/rhuser/.gem/ruby/gems/3scale_toolbox-0.4.0/exe/3scale-update:228:in `fetch': key not found: 2555418096690 (KeyError)
Did you mean?  2555418015767

This makes some sense but there is no way to copy the rules AND methods over to get around this. You have to manually create new methods on the destination first.

The use case here is to create a staging service and a production service that are connected to different backends. And then have a simple way to keep the mapping rules/methods in sync while not wiping out the Private Base URL or other settings.

Provide some utility methods to command writers

Heh..... looking at existing commands and playing with writing a new one..... I suspect there will be some code that will be repeated across many commands?

Examples:

  • Command/option parsing and help printing
  • API url manipulation

On that second one, I see these two methods:

def provider_key_from_url(url)
URI(url).user
end

def endpoint_from_url(url)
uri = URI(url)
uri.user = nil

uri.to_s
end

These could be provided by a shared class? (cli?)

Maybe even parsing of a "standard" url used for many commands ("source") could be done in a common file too? Get a tuple of key,url back, or even a Client object directly - all ready to talk to that source API....?

Copy services across accounts

We would like to copy a service to a different account.
That would require:

  • entering second endpoint
  • entering second provider key

It might be cool to introduce a way to bundle endpoint with the provider key into one value. Like https://[email protected].

RFE: Make the toolbox idempotent

The 3scale toolbox is being improved to become the recommended way to configure and deploy services in 3scale. This fits in a broader approach of Continuous Delivery that has been started by several customers / communities (I personally maintain the threescale-cicd ansible role. Those approaches are using different means to drive the 3scale admin portal, some are using the 3scale-cli other (like the threescale-cicd role) are using the 3scale REST APIs.

If we want to make the 3scale_toolbox the de facto standard for interacting with 3scale, we need it to be usable in a CI/CD approach.

At a glance, here are the features I expect from the 3scale toolbox :

  • #81: Is idempotent
  • #82: Works with system_name and not service_id
  • #83: Can output human readable information as well as json or yaml (akin to oc cmd -o yaml)
  • #80: Have a standard way to report errors

Idempotency

When continuous delivering an API, the delivery pipeline needs to create or update the 3scale service definition in addition to each linked object.

Forcing the user to specify if the service needs to be created or updated leads to a lots of code in the pipeline (list existing services, create is missing, update if existing).

Proposition

The 3scale_toolbox does not enforce the user to choose between create or update operations. Instead, if the object exists, it updates it, otherwise it creates it.

Example

3scale copy service -t my-prod-api -s dev -d prod my-dev-api

=> If the service exists, it is updated
=> Otherwise it is created

This feature needs to be in the toolbox's core so that every plugin can leverage this feature.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.