Git Product home page Git Product logo

python-ambariclient's Introduction

Python bindings to the Apache Ambari API

All syntax examples will assume you are using the client against the test host created by following the Ambari Quick Start Guide. I've set up a 5 node cluster which has hostnames c6401-c6405.ambari.apache.org. The examples will translate to other clusters, just the hostnames and number of hosts might change, obviously.

Basic API bindings design

The starting point for users is the Ambari class in ambariclient.client:

>>> from ambariclient.client import Ambari
>>> client = Ambari('c6401.ambari.apache.org', port=8080, username='admin', password='admin')

Ambari Shell

The Ambari Shell will automatically set up the ambari client for you and let you play around with the API to learn the syntax. To load it run the ambari-shell executable. It should have been put in a system path for you automatically at install time, but if you're just playing with the source code, you can call bin/ambari-shell directly. It is based on the IPython shell, so for more comprehensive documentation of all the capabilities provided by that, refer to the IPython documentation. Note that the shell functionality requires that you install IPython:

$ pip install IPython

Here is an

$ ambari-shell

Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Logging level set to 50
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
Type "copyright", "credits" or "license" for more information.

IPython 3.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.


Ambari client available as 'ambari'
 - Ambari Server is http://c6401.ambari.apache.org:8080
 - Ambari Version is 2.0.0

 - log(new_level) will reset the logger level
 - ambari_ref() will show you all available client method chains

In [1]:

You can then just reference the ambari variable as the client. The client is configurable via command-line options and/or a config file. The defaults are those for the Ambari Quick Start setup. Username and password are defaulted to admin, and the host is defaulted to http://c6401.ambari.apache.org:8080

You can override them by creating a JSON configuration file in $HOME/.ambari:

{
    "host": "my.ambari.server.com",
    "port": 80,
    "username": "my-username",
    "password": "my-password"
}

You can also pass in any of those options on the CLI:

$ ambari-shell --host http://my.ambari.server.com --username my-username --password my-password

Or you can specify some in the config file and then override some using the CLI switches.

You can also override the logger level by passing it in or putting it in the config file:

$ ambari-shell --logger INFO

To get more information about the switches, run ambari-shell --help

Basic Examples

All of the various resources are broken down into what are known as collections. Each collection contains one or more models, which represent resources in the Ambari API. For example, to get the usernames of all users in the system:

>>> for user in ambari.users:
...     user.user_name
...
u'admin'

You can get a specific model from a collection if you have the primary identifier for that model. So, for a user, the user_name is the primary identifier. To get the 'admin' user:

>>> user = ambari.users('admin')
>>> user.user_name
u'admin'

Each model can then, in turn, contain collections of models that are subordinate to it. In the case of users, there is a relationship to what are called privileges:

>>> for privilege in ambari.users('admin').privileges:
...     privilege.permission_name
...
u'AMBARI.ADMIN'

The API tries to mimic a promises-style API (also called futures). It will only load data when it is required to proceed. So, for example, in the previous example, we had to issue a GET request when we called privilege.permission_name because permission_name was not known until that point. However, if we simply needed to get the permission_id instead, that information is returned by the GET request on the user object. I'm including the relevant debug log statements to show the difference:

>>> for privilege in ambari.users('admin').privileges:
...     privilege.permission_name
...
DEBUG:requests.packages.urllib3.connectionpool:"GET /api/v1/users/admin HTTP/1.1" 200 416
DEBUG:ambariclient.client:Response: {
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/users/admin",
  "Users" : {
    "active" : true,
    "admin" : true,
    "groups" : [ ],
    "ldap_user" : false,
    "user_name" : "admin"
  },
  "privileges" : [
    {
      "href" : "http://c6401.ambari.apache.org:8080/api/v1/users/admin/privileges/1",
      "PrivilegeInfo" : {
        "privilege_id" : 1,
        "user_name" : "admin"
      }
    }
  ]
}
DEBUG:requests.packages.urllib3.connectionpool:"GET /api/v1/users/admin/privileges/1 HTTP/1.1" 200 287
DEBUG:ambariclient.client:Response: {
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/users/admin/privileges/1",
  "PrivilegeInfo" : {
    "permission_name" : "AMBARI.ADMIN",
    "principal_name" : "admin",
    "principal_type" : "USER",
    "privilege_id" : 1,
    "type" : "AMBARI",
    "user_name" : "admin"
  }
}
u'AMBARI.ADMIN'

Notice that two GET requests were sent to load the appropriate data. The first is sent as soon as you access the privileges attribute on ambari.users('admin'). Most people would expect it to be sent on ambari.users('admin') but at that point you aren't attempting to access any data unknown to the object, so no API call is sent. Accessing a relationship requires the object to be populated as the relationship data is often returned in the original request, saving a separate API call later. Now, back to the example:

>>> for privilege in ambari.users('admin').privileges:
...     privilege.privilege_id
...
DEBUG:requests.packages.urllib3.connectionpool:"GET /api/v1/users/admin HTTP/1.1" 200 416
DEBUG:ambariclient.client:Response: {
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/users/admin",
  "Users" : {
    "active" : true,
    "admin" : true,
    "groups" : [ ],
    "ldap_user" : false,
    "user_name" : "admin"
  },
  "privileges" : [
    {
      "href" : "http://c6401.ambari.apache.org:8080/api/v1/users/admin/privileges/1",
      "PrivilegeInfo" : {
        "privilege_id" : 1,
        "user_name" : "admin"
      }
    }
  ]
}
1

In this case, only the GET request for the user object was required, since it provided the needed privilege_id information.

The last concept you need to be aware of is the wait() method that exists on nearly all objects in the system, both collections and models. Calling wait() will force a sync operation to occur. What this means in practical terms is:

  • If the object is backed by a URL, it will be loaded with fresh data from the server
  • If the object had a method called on it that generated a Request object on the server (i.e. a long-running asynchronous process), it will poll that request until it is completed

The basic idea is to wait until the object is in a 'ready' state, so that it can be further acted upon. The method is designed for method-chaining, so you can do fun things like:

>>> host.components.install().wait().start().wait()

It's most-commonly useful after a create() call:

>>> cluster = ambari.clusters.create(name, blueprint=bp_name,
                                     host_groups=host_groups,
                                     default_password=pwd
                                    ).wait(timeout=1800, interval=30)

You can override the default timeout and interval settings if you have a good idea of how long you expect the process to take. Both values are in seconds. The default is to poll every 15s for an hour, which is often excessive.

Host bootstrapping

For testing things out, the bootstrap API is very useful. To start a bootstrap process using this library, it's pretty simple:

>>> hosts = ["c6401.ambari.apache.org", "c6402.ambari.apache.org", "c6403.ambari.apache.org", "c6404.ambari.apache.org"]
>>> ssh_key = ''
>>> with open("/path/to/insecure_private_key") as f:
...    ssh_key = f.read()
...
>>> bootstrap = ambari.bootstrap.create(hosts=hosts, sshKey=ssh_key, user='vagrant')
>>> bootstrap.wait()

There is no way to retrieve a list of bootstrap operations, so don't lose track of the one you started. This is a server-side API restriction:

>>> ambari.bootstrap.wait()
ambariclient.exceptions.MethodNotAllowed: HTTP request failed for GET http://c6401.ambari.apache.org:8080/api/v1/bootstrap: Method Not Allowed 405: {
  "status": 405,
  "message": "Method Not Allowed"
}

API Hierarchy

For reference, this is the currently-supported hierarchy of collections and models available for you to use. This list can be regenerated by calling reference() in the ambari-shell. We'll try to keep it up-to-date:

  • ambari.actions
  • ambari.actions(action_name)
  • ambari.alert_targets
  • ambari.alert_targets(id)
  • ambari.blueprints
  • ambari.blueprints(blueprint_name)
  • ambari.blueprints(blueprint_name).host_groups
  • ambari.blueprints(blueprint_name).host_groups(name)
  • ambari.bootstrap
  • ambari.bootstrap(requestId)
  • ambari.clusters
  • ambari.clusters(cluster_name)
  • ambari.clusters(cluster_name).alert_definitions
  • ambari.clusters(cluster_name).alert_definitions(id)
  • ambari.clusters(cluster_name).alert_groups
  • ambari.clusters(cluster_name).alert_groups(id)
  • ambari.clusters(cluster_name).alert_history
  • ambari.clusters(cluster_name).alert_history(id)
  • ambari.clusters(cluster_name).alert_notices
  • ambari.clusters(cluster_name).alert_notices(id)
  • ambari.clusters(cluster_name).alerts
  • ambari.clusters(cluster_name).alerts(id)
  • ambari.clusters(cluster_name).configurations
  • ambari.clusters(cluster_name).configurations(type)
  • ambari.clusters(cluster_name).host_components
  • ambari.clusters(cluster_name).host_components(component_name)
  • ambari.clusters(cluster_name).hosts
  • ambari.clusters(cluster_name).hosts(host_name)
  • ambari.clusters(cluster_name).hosts(host_name).alert_history
  • ambari.clusters(cluster_name).hosts(host_name).alert_history(id)
  • ambari.clusters(cluster_name).hosts(host_name).alerts
  • ambari.clusters(cluster_name).hosts(host_name).alerts(id)
  • ambari.clusters(cluster_name).hosts(host_name).components
  • ambari.clusters(cluster_name).hosts(host_name).components(component_name)
  • ambari.clusters(cluster_name).privileges
  • ambari.clusters(cluster_name).privileges(privilege_id)
  • ambari.clusters(cluster_name).requests
  • ambari.clusters(cluster_name).requests(id)
  • ambari.clusters(cluster_name).requests(id).tasks
  • ambari.clusters(cluster_name).requests(id).tasks(id)
  • ambari.clusters(cluster_name).services
  • ambari.clusters(cluster_name).services(service_name)
  • ambari.clusters(cluster_name).services(service_name).alert_history
  • ambari.clusters(cluster_name).services(service_name).alert_history(id)
  • ambari.clusters(cluster_name).services(service_name).alerts
  • ambari.clusters(cluster_name).services(service_name).alerts(id)
  • ambari.clusters(cluster_name).services(service_name).components
  • ambari.clusters(cluster_name).services(service_name).components(component_name)
  • ambari.clusters(cluster_name).services(service_name).components(component_name).host_components
  • ambari.clusters(cluster_name).services(service_name).components(component_name).host_components(component_name)
  • ambari.clusters(cluster_name).services(service_name).components(component_name).metrics
  • ambari.clusters(cluster_name).services(service_name).components(component_name).metrics(name)
  • ambari.clusters(cluster_name).upgrades
  • ambari.clusters(cluster_name).upgrades(request_id)
  • ambari.clusters(cluster_name).upgrades(request_id).groups
  • ambari.clusters(cluster_name).upgrades(request_id).groups(group_id)
  • ambari.clusters(cluster_name).upgrades(request_id).groups(group_id).items
  • ambari.clusters(cluster_name).upgrades(request_id).groups(group_id).items(stage_id)
  • ambari.clusters(cluster_name).upgrades(request_id).groups(group_id).items(stage_id).tasks
  • ambari.clusters(cluster_name).upgrades(request_id).groups(group_id).items(stage_id).tasks(id)
  • ambari.groups
  • ambari.groups(group_name)
  • ambari.groups(group_name).members
  • ambari.groups(group_name).members(user_name)
  • ambari.hosts
  • ambari.hosts(host_name)
  • ambari.services
  • ambari.services(service_name)
  • ambari.services(service_name).components
  • ambari.services(service_name).components(component_name)
  • ambari.services(service_name).components(component_name).host_components
  • ambari.stacks
  • ambari.stacks(stack_name)
  • ambari.stacks(stack_name).versions
  • ambari.stacks(stack_name).versions(stack_version)
  • ambari.stacks(stack_name).versions(stack_version).operating_systems
  • ambari.stacks(stack_name).versions(stack_version).operating_systems(os_type)
  • ambari.stacks(stack_name).versions(stack_version).operating_systems(os_type).repositories
  • ambari.stacks(stack_name).versions(stack_version).operating_systems(os_type).repositories(repo_id)
  • ambari.stacks(stack_name).versions(stack_version).repository_versions
  • ambari.stacks(stack_name).versions(stack_version).repository_versions(id)
  • ambari.stacks(stack_name).versions(stack_version).services
  • ambari.stacks(stack_name).versions(stack_version).services(service_name)
  • ambari.stacks(stack_name).versions(stack_version).services(service_name).components
  • ambari.stacks(stack_name).versions(stack_version).services(service_name).components(component_name)
  • ambari.stacks(stack_name).versions(stack_version).services(service_name).configurations
  • ambari.users
  • ambari.users(user_name)
  • ambari.users(user_name).privileges
  • ambari.users(user_name).privileges(privilege_id)
  • ambari.views
  • ambari.views(view_name)
  • ambari.views(view_name).versions
  • ambari.views(view_name).versions(version)
  • ambari.views(view_name).versions(version).instances
  • ambari.views(view_name).versions(version).instances(instance_name)
  • ambari.views(view_name).versions(version).permissions
  • ambari.views(view_name).versions(version).permissions(permission_id)

Testing

Since doing good unit tests for this library would require mocking pretty much the entire Ambari API, we didn't see a lot of benefit in that. There are some basic unit tests that will be expanded upon over time, but most testing is currently done manually.

$ tox -e py27
$ tox -e py33
$ tox -e

Ambari Versions

The goal of the client is to work with multiple versions of Ambari by smoothing out the differences for the user. The client can automatically detect the version of Ambari running on the server:

>>> ambari.version
('1', '7', '0')

This client has only been tested against Ambari 1.7+. If you need support for older versions, you might need to submit patches for anything that was changed after that time. Supporting anything prior to 1.6 is problematic as the basic method of creating clusters changed to require blueprints in that release, but if you have a good idea on how to make it seamless, then we can try to integrate the solution.

Python Versions

The goal is to support Python 2.7+ and 3.3+. I'm not opposed to adding 2.6 support, but someone else will need to do the work.

Command-Line Interface

A CLI was originally planned, but never implemented. Some methods that require a large JSON body are not amenable to a CLI, but we could easily build one for other methods if people want one. We replaced the idea with the Ambari Shell for now, but are open to revisiting this if demand is present.

python-ambariclient's People

Contributors

alan-rackspace avatar dimaspivak avatar jimbobhickville avatar kozmagabor avatar rnirmal avatar scott-rackspace avatar senjaliya avatar srids avatar thesquelched avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-ambariclient's Issues

Command to start/stop all services?

This project is beautiful and really makes working with Ambari incredibly straightforward. The only thing I haven't been able to find out from my examination of the docs and the code is how to invoke the "Start all services" endpoint that Ambari has for a given cluster. The corresponding code looks straightforward enough, so it is just a question of the functionality not being present or am I overlooking something obvious?

Allow user to provide context while performing an operation

install, start & stop have default "Stop component_name" context, a generic one.
having a way to provide context while performing operation is very useful because this context appear on ambari operation list, so it would be easy to identify among lot of other operations or similar ones from other user or service.

Getting content from request

Hi @jimbobhickville,

I have one more question for an enhancement I'd like to make to your project (if it's alright with you). I have a use case that requires me to get a tarstream of the client configs for a given service from Ambari. This is straightforward enough to implement with a new ClusterServiceComponentCollection class with a get_client_config_tar method, but I'm having trouble deciding how to make it fit into your existing semantics. In short, the self.loads(self.client.get(...)) implementation pattern used throughout fails for me because the self.client.get part ends up returning a response.json() which can't be parsed with the JSON encoder (i.e. because the response we're getting isn't a JSON but a tarstream).

A working example I implemented drops abstractions to the wayside and simply uses the underlying requests machinery, but I wanted to see if you had a thought on if there's a cleaner way to architect this:

class ClusterServiceComponentCollection(base.QueryableModelCollection):
    def get_client_config_tar(self):
        url = self.url + "?format=client_config_tar"
        http_client = self.client.client
        return http_client.session.get(url, **http_client.request_params)

beef up ambari-shell

  • Add tab-completion or autocompletion
  • Add persistent history
  • Add CLI config options
  • Add way to adjust log-level on the fly
  • Add help

Service restart failes

Hi,

I'm facing a problem when I try to restart my HDFS service:
cluster.services('HDFS').restart()
I get this stack trace:

Traceback (most recent call last):
  File "C:\AmPy\restart.py", line 22, in <module>
    cluster.services('HDFS').restart()
  File "C:\AmPy\ambariclient\models.py", line 739, in restart
    hosts = [hc.host_name for hc in component.host_components]
  File "C:\AmPy\ambariclient\base.py", line 129, in __iter__
    self.inflate()
  File "C:\AmPy\ambariclient\base.py", line 248, in inflate
    self.load(self.client.get(self.url, params=self._filter))
  File "C:\AmPy\ambariclient\client.py", line 151, in request
    handle_response(response)
  File "C:\AmPy\ambariclient\exceptions.py", line 199, in handle_response
    raise cls(**kwargs)
ambariclient.exceptions.NotFound: HTTP request failed for GET https://test-cluster:9443/api/v1/clusters/Cluster1/services/HDFS/components/JOURNALNODE/host_components: Not found 404:

I think it is caused by missing host_components in the component JOURNALNODE.
The result of the call to the JOURNAL host_components is:

DEBUG:ambariclient.client:Response: {
  "href" : "https://test-cluster:9443/api/v1/clusters/Cluster1/services/HDFS/components/JOURNALNODE",
  "ServiceComponentInfo" : {
    "category" : "SLAVE",
    "cluster_name" : "Cluster1",
    "component_name" : "JOURNALNODE",
    "display_name" : "JournalNode",
    "init_count" : 0,
    "install_failed_count" : 0,
    "installed_count" : 0,
    "recovery_enabled" : "false",
    "service_name" : "HDFS",
    "started_count" : 0,
    "state" : "STARTED",
    "total_count" : 0,
    "unknown_count" : 0
  },
  "metrics" : {
    "cpu" : {
      "cpu_idle" : 97.44444444444446,
...
    },
    "memory" : {
      "mem_cached" : 5.8982518666666664E7,
...
      "swap_total._sum" : 1.25829E7
    },
    "network" : {
      "bytes_in" : 22481.626135457936,
...
      "pkts_out._sum" : 235.80516374046846
    },
    "process" : {
      "proc_run" : 0.6666666666666666,
...
      "proc_total._sum" : 1103.0
    }
  },
  "host_components" : [ ]
}

If more information is needed, I'm happy do give more details.

Add tests

  • add basic unit tests for util methods
  • add unit tests for pub/sub framework
  • add unit tests for base classes, as much as possible without a huge mocking framework
  • add integration tests for testing against actual cluster
  • wrap everything with tox configurations to make it easy

Use Sphinx for docs

We should update all the docs to use sphinx documentation so it's pretty.

Throw exceptions on .wait() when requests fail

PollableMixin should add an is_error abstract method that subclasses can implement to detect an error in the polled process and abort with an exception. This should be implemented by the Request, Bootstrap, and ClusterHost model classes to account for potential errors.

HostComponent stop doesn't work

Trying to stop a specific component for a host doesn't work currently. The request is made, but the operation_level is not set correctly, so nothing happens in ambari.

I don't do this much, so not quite sure how to push this fix from my fork. But in models.py, HostComponent stop should be changed to:

    def stop(self, context=None):
        if not context:
            context = "Stop %s" % normalize_underscore_case(self.component_name)

        """Starts this component on its host, if already installed and started."""
        self.load(self.client.put(self.url, data={
            "RequestInfo": {
                "context": context,
                "operation_level": {
                        "level": "HOST_COMPONENT",
                        "cluster_name": self.cluster_name,
                        "host_name": self.host_name,
                    }
            },
            "HostRoles": {
                "state": "INSTALLED",
            },
        }))
        return self

I will try to push the fix upstream at some point when I get some time (unless anyone is monitoring these issues and wants to do it before then).

Support for getting and modifying configurations by type

While trying to retrieving and modifying 'core-site.xml', it seems ambariclient does not support configurations completely. For e.g., for my hdp named 'cluster' with Ambari host/port as node-1.cluster:8080, issuing this command: ambari.clusters('cluster').configurations('core-site').to_dict() results to a HTTP 404 error as:

>>> ambari.clusters('cluster').configurations('core-site').to_dict()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/ambariclient/base.py", line 494, in to_dict
    self.inflate()
  File "/usr/local/lib/python3.6/site-packages/ambariclient/base.py", line 611, in inflate
    self.load(self.client.get(self.url))
  File "/usr/local/lib/python3.6/site-packages/ambariclient/client.py", line 151, in request
    handle_response(response)
  File "/usr/local/lib/python3.6/site-packages/ambariclient/exceptions.py", line 199, in handle_response
    raise cls(**kwargs)
ambariclient.exceptions.NotFound: HTTP request failed for GET http://node-1.cluster:8080/api/v1/clusters/cluster/configurations/core-site: Not found 404:

http://node-1.cluster:8080/api/v1/clusters/cluster/configurations/core-site is not a right call to Ambari, it should resolve to something of "http://node-1.cluster:8080/api/v1/clusters/cluster/configurations?type=core-site" and later to query as "http://node-1.cluster:8080/api/v1/clusters/cluster/configurations?type=core-site&tag=TOPOLOGY_RESOLVED"

With this issue, am hoping that there will be support for retrieving and modifying configurations by type (where type resolves to one of http://node-1.cluster:8080/api/v1/clusters/cluster?fields=Clusters/desired_configs)

conda recipe available

Thx for the project. Just to let you know that I created a conda package out of it (see here). Just added a conda.recipe folder for that. Wondering if this cannot be nice to add for those of us who want to create conda packages. Just let me know and I'll try to come up with some pull request.

License file for python-ambariclient

Hi Greg,

Thank you so much for putting this library publicly available on github. I found it very useful to control ambari programatically from my python scripts. However, I can't find any license file for this project. So I can't use this beyond personal projects.

Could you please put up a license for this project? I bet it'll be very useful for many developers.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.