juju / python-libjuju Goto Github PK

View Code? Open in Web Editor NEW

55.0 25.0 93.0 21.69 MB

Python library for the Juju API

License: Apache License 2.0

Python 99.91% Makefile 0.03% Shell 0.07%

python-libjuju's Introduction

A Python library for Juju

Source code: https://github.com/juju/python-libjuju

Bug reports: https://github.com/juju/python-libjuju/issues

Documentation: https://pythonlibjuju.readthedocs.io/en/latest/

Requirements

Python 3.9/3.10

Design Notes

Asynchronous - Uses asyncio and async/await features of Python
Websocket-level bindings are programmatically generated (indirectly) from the Juju golang code, ensuring full api coverage
Provides an OO layer which encapsulates much of the websocket api and provides familiar nouns and verbs (e.g. Model.deploy(), Application.add_unit(), etc.)

Installation

pip3 install juju

Quickstart

Here's a simple example that shows basic usage of the library. The example connects to the currently active Juju model, deploys a single unit of the ubuntu charm, then exits:

Note : Pylibjuju requires an already bootstrapped Juju controller to connect to.

#!/usr/bin/python3

import logging
import sys

from juju import jasyncio
from juju.model import Model


async def deploy():
    # Create a Model instance. We need to connect our Model to a Juju api
    # server before we can use it.
    model = Model()

    # Connect to the currently active Juju model
    await model.connect()

    try:
        # Deploy a single unit of the ubuntu charm, using the latest revision
        # from the stable channel of the Charm Store.
        ubuntu_app = await model.deploy(
          'ubuntu',
          application_name='my-ubuntu',
        )

        if '--wait' in sys.argv:
            # optionally block until the application is ready
            await model.wait_for_idle(status = 'active')

    finally:
        # Disconnect from the api server and cleanup.
        await model.disconnect()


def main():
    logging.basicConfig(level=logging.INFO)

    # If you want to see everything sent over the wire, set this to DEBUG.
    ws_logger = logging.getLogger('websockets.protocol')
    ws_logger.setLevel(logging.INFO)

    # Run the deploy coroutine in an asyncio event loop, using a helper
    # that abstracts loop creation and teardown.
    jasyncio.run(deploy())


if __name__ == '__main__':
    main()

More examples can be found in the docs, as well as in the examples/ directory of the source tree which can be run using tox. For example, to run examples/connect_current_model.py, use:

tox -e example -- examples/connect_current_model.py

REPL

To experiment with the library in a REPL, launch python in asyncio mode

$ python3 -m asyncio

and then, to connect to the current model and fetch status:

>>> from juju.model import Model
>>> model = Model()
>>> await model.connect_current()
>>> status = await model.get_status()

Versioning

The current Pylibjuju release policy tracks the Juju release cadence. In particular, whenever Juju makes a latest/stable release, pylibjuju pushes out a release with the same version in the following week. Newly generated schemas will be updated per Juju releases.

python-libjuju's People

Contributors

Stargazers

Watchers

Forkers

bac pengale howbazaar simonklb sfeole seman mitechie tengu-team gabriel-samfira radiator-software adamisrael sebastienpattyn93 rogpeppe-contrib gnuoy canonical fnordahl n-pochet chrismacnaughton ryan-beisner mattcurrycom jcsackett alainivars aieri karp773 peppepetra ycliuhw wallyworld simonrichardson gerardo-garcia xiwenc variabledeclared xiaoruiguo hpidcock matuskosut nammn tlm basanneh gnomechompsky kapilt laashub-soa tai271828 eric-seekas afreiberger souvikghosh05 thedac macduff23 jackge007 xavpaice matheushent domfleischmann mhilton juanmanuel-tirado cderici jack-w-shaw wolsen dparv addyess oescal simskij isabella232 ittner pietropasotti charmed-kubernetes jdkandersson nsklikas mert-kirpici hemanthnakkina stonepreston arturo-seijas gabrielcocenza shayancanonical tbaumann beliaev-maksim irishgordo mehdi-bendriss dnplas markbeierl neoaggelos danielarndt maxcode123 marceloneppel pcompassion sed-i tonyandrewmeyer freyes gboutry aflynn50 mthaddon jneo8 thanhphan1147 yanksyoon luissimas

python-libjuju's Issues

jujubundlelib 0.5.2 has requirement PyYAML==3.11, but you have PyYAML 3.12

$ pip check
No broken requirements found.
$ pip install juju
Collecting juju
Collecting pyyaml (from juju)
Collecting python-dateutil (from juju)
  Using cached python_dateutil-2.6.0-py2.py3-none-any.whl
Collecting theblues (from juju)
Collecting websockets (from juju)
  Using cached websockets-3.2-py33.py34.py35-none-any.whl
Collecting six>=1.5 (from python-dateutil->juju)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting jujubundlelib>=0.4.1 (from theblues->juju)
Collecting requests>=2.1.1 (from theblues->juju)
  Using cached requests-2.12.4-py2.py3-none-any.whl
Installing collected packages: pyyaml, six, python-dateutil, jujubundlelib, requests, theblues, websockets, juju
Successfully installed juju-0.1.2 jujubundlelib-0.5.2 python-dateutil-2.6.0 pyyaml-3.12 requests-2.12.4 six-1.10.0 theblues-0.3.8 websockets-3.2
$ pip check
jujubundlelib 0.5.2 has requirement PyYAML==3.11, but you have PyYAML 3.12.

Add publicly exposed testing helpers.

Most notably it would be great to have a blessed (and maintained) testing double for the Juju API. Landscape already has something along those lines: https://launchpad.net/fake-juju.

Landscape also has some other testing helpers: https://bazaar.launchpad.net/~landscape/landscape/trunk/view/head:/canonical/juju/testing.py

connect_current fails ugly if you have a non-admin account with password

I'm connecting to MAAS with a non-admin controller account. The account is the result of an add-user and register and has a custom password.

This should catch and output something a bit nicer if possible, or sanity check and fail in a clear way.

I added a password to .local/share/juju/accounts.yaml and it worked out.

It'd also be nice, since this is a common multi-user scenario, to allow connect_current to accept some flags so that I don't have to fill in all the info but I can supply a password/etc. I guess I'd like some way to setup a cli script to allow the user to specify some info via shortcut (what controller/model via cli args) and then supply stuff like passwords in some clean way (env var?)

Failure:

Connecting to model
Traceback (most recent call last):
  File "stressjuju/stressjuju.py", line 39, in <module>
    loop.run(main())
  File "/home/rharding/src/jujutest/python-libjuju/juju/loop.py", line 31, in run
    raise task.exception()
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "stressjuju/stressjuju.py", line 16, in main
    await model.connect_current()
  File "/home/rharding/src/jujutest/python-libjuju/juju/model.py", line 395, in connect_current
    self.connection = await connection.Connection.connect_current()
  File "/home/rharding/src/jujutest/python-libjuju/juju/client/connection.py", line 222, in connect_current
    '{}:{}'.format(controller_name, model_name))
  File "/home/rharding/src/jujutest/python-libjuju/juju/client/connection.py", line 274, in connect_model
    endpoint, model_uuid, username, password, cacert, macaroons)
  File "/home/rharding/src/jujutest/python-libjuju/juju/client/connection.py", line 186, in connect
    await client.login(username, password, macaroons)
  File "/home/rharding/src/jujutest/python-libjuju/juju/client/connection.py", line 297, in login
    "macaroons": macaroons or []
  File "/home/rharding/src/jujutest/python-libjuju/juju/client/connection.py", line 95, in rpc
    raise JujuAPIError(result)
juju.errors.JujuAPIError: no credentials provided
Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.run() running at /home/rharding/src/jujutest/.venv/lib/python3.5/site-packages/websockets/protocol.py:413> wait_for=<Future pending cb=[Task._wakeup()]>>
Fatal write error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fad895c0eb8>
transport: <_SelectorSocketTransport fd=6>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 700, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fad895c0eb8>
transport: <_SelectorSocketTransport closing fd=6>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 700, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/sslproto.py", line 632, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 704, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 589, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 601, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

'pip install juju' keeps failing

Everytime I want to install the JUJU pip package(pip install juju) it fails and I get the following error:

Collecting juju
  Downloading juju-0.1.2.tar.gz (127kB)
    100% |████████████████████████████████| 133kB 935kB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-N2anhR/juju/setup.py", line 24, in <module>
        version=version.read_text().strip(),
    AttributeError: 'PosixPath' object has no attribute 'read_text'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-N2anhR/juju/

I already updated all my other packages but I still got the error. Is there any package still missing or do I need to downgrade a certain package?

python-libjuju is broken against the latest alpha

If you call the FullStatus facade, python-libjuu breaks due to a missing model-status key. It looks like we have a breaking change in the latest version of the websocket api.

Pushing a test that reproduces this shortly.

@tvansteenburgh @johnsca

Model.connection should be private

Model.connection should be private, since you need to call a method after adding it. Especially since that method is private. Also, unsetting an existing connection without first calling Model.disconnect is wrong. Model.connection just does not behave like a plain variable, so it should be private.

Possibly a read-only property should be supplied.

KeyError when deploying bundles with blank strings as default config values

I filed this against core here, https://bugs.launchpad.net/juju/+bug/1651260, but I'm working on a workaround.

... it appears that we need to pass our config as "config-yaml" rather than "config"

controller.destroy-models should accept names of the models

I've created a script that adds a new model but then I went to clean it up and found the destroy call takes the uuid. I think it's worthwhile to allow the user to supply the name and leave it to the function call to deal with uuids as internal implementation details. To the user the model names is what they use/live off of via juju models calls and such.

ReturnMapping is incorrectly mapping ErrorResults to methods with no errors

A set of classes in _client.py is decorated with the @ReturnMapping decorator, called with the argument "ErrorResults". Reading through the code, it looks like the intent is to cast the class to ErrorResults if there is an error in the result that comes back from the API.

It does not appear that this code is working properly, however, as successful messages are being cast ErrorResults, too.

To reproduce:

pick a decorated method, and pick a place in the code where you know that method is being called successfully.
Add a check like the following:

    ret = await <the thing that is running successfully>
    if type(ret) is client.ErrorResults:
        messages = [res.error.message for res in ret.results]
        raise JujuError(messages)

Run the integration tests (tox -r -e integration)
Note that a JujuError gets raised.

Note also that "messages" will be an empty list. I believe that the problem is that all results come back with an error field -- we need to check to make sure that it is truthy before casting the class to ErrorResults

We don't interpret all errors from the api as errors

I am currently trying to troubleshoot any issue deploying the haproxy charm via python-libjuju. While troubleshooting, I discovered that the call to .Deploy in the Application facade doesn't always throw an error when the websocket interface returns an error. I suspect that this problem exists in the other facades as well.

Here's an example of an error that python-libjuju does catch (where by "catch", I mean that it throws a JujuAPIError):

client << Frame(fin=True, opcode=1, data=b'{"request-id":6,"error":"json: cannot unmarshal object into Go value of type string
","response":{}}')

Here's an example of an error that produces no JujuAPIError:

 client << Frame(fin=True, opcode=1, data=b'{"request-id":6,"response":{"results":[{"error":{"message":"cannot parse settings d
ata: yaml: unmarshal errors:\\n  line 1: cannot unmarshal !!str `queue 6...` into charm.Settings\\n  line 2: cannot unmarshal !!str `True` into charm.Settings\\n 
 line 3: cannot unmarshal !!str `0.0.0.0/0` into charm.Settings\\n  line 4: cannot unmarshal !!str `haproxy` into charm.Settings\\n  line 5: cannot unmarshal !!st
r `` into charm.Settings\\n  line 6: cannot unmarshal !!str `backports` into charm.Settings\\n  line 7: cannot unmarshal !!str `SELFSIGNED` into charm.Settings","
code":""}}]}}')

Note that, with the second one, we get a response key rather than an error key, but we have an error in the embedded results dict.

This is an inconsistency on the websocket api's part, but we should probably work around it here.

Implement Model.add_machines()

We have BundleHandler.addMachines() but that's not a public api. Maybe the logic can be relocated to Model.add_machines() and then BundleHandler.addMachines() can call that.

Provide method to get a Controller instance from a Model instance

e.g. Model.get_controller()

Handle websocket connection resets

Running against a shared controller in aws, I'm observing the websocket connections being forcefully reset by the server end. It appears to happen consistently when waiting for a machine to be provisioned, which can take a while. Not sure if this is a bug in Juju or expected behavior, but either way we should probably add some auto-reconnect stuff to the client.

Use juju commands instead of yaml files?

When using connection shortcuts like connect_current() and connect_model(), the information necessary to make the connection is obtained from yaml files in ~/.local/share/juju/.

I just saw a case where a model was present (showed in the output of juju list-models), yet did not exist in ~/.local/share/juju/models.yaml yet.

To avoid this we should use the output of juju commands instead of using the yaml files directly. Either way assumes the juju cli is installed. To connect without the juju cli installed, the plain connect() method can be used.

Implement Controller.get_model()

Should return a connected Model instance.

Figure out how to handle version changes

See https://lists.ubuntu.com/archives/juju/2017-January/008479.html for context.

Loading the correct facade version dynamically is the way to go. The schema json file will need to be appended to instead of overwritten so that old facade versions are preserved. API classes will need to be namespaced so that two versions of the same facade (with the same name) don't collide.

That's the easy part. The tricky part is the coupling between the OO and api layers. For example, Application.add_units() calls ApplicationFacade.AddUnits() with parameters. What happens when the next version of that facade changes the parameters? The OO layer needs some way to deal with that...

To_Json()

To_json call() on a result object fails when there are more objects in the result (as in every result).
CODE:

import asyncio
import logging

from juju.controller import Controller

logging.basicConfig(level=logging.DEBUG)

async def run():
    controller = Controller()

    controller_endpoint = 'xxx.xxx.xxx.xxx:17070'
    username = 'admin'
    password = 'xxx'

    # Left out for brevity, but if you have a cert string you should pass it in.
    # You can copy the cert from the output of The `juju show-controller`
    # command.
    cacert = None

    await controller.connect(
        controller_endpoint,
        username,
        password,
        cacert,
    )
    result = await controller.get_models()
    print(result.to_json())
    await controller.disconnect()


logging.basicConfig(level=logging.DEBUG)
ws_logger = logging.getLogger('websockets.protocol')
ws_logger.setLevel(logging.DEBUG)
loop = asyncio.get_event_loop()
loop.set_debug(False)
loop.run_until_complete(run())

DEBUG:

DEBUG:asyncio:Using selector: EpollSelector
INFO:websocket:Driver connected to juju wss://xxx.xxx.xxx.xxx:17070/api
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 3,\n  "params": {},\n  "request": "RedirectInfo",\n  "type": "Admin",\n  "request-id": 1\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":1,"error":"not redirected","response":{}}')
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 3,\n  "request-id": 2,\n  "request": "Login",\n  "type": "Admin",\n  "params": {\n    "auth-tag": "user-admin",\n    "nonce": "J~v#8EK-GCs(",\n    "credentials": "xxx",\n    "macaroons": []\n  }\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":2,"response":{"servers":[[{"value":"104.154.205.122","type":"ipv4","scope":"public","port":17070},{"value":"10.128.0.3","type":"ipv4","scope":"local-cloud","port":17070},{"value":"127.0.0.1","type":"ipv4","scope":"local-machine","port":17070},{"value":"::1","type":"ipv6","scope":"local-machine","port":17070}]],"controller-tag":"controller-2acad8f2-fc47-43db-8ebb-81bcbdb91e5c","user-info":{"display-name":"","identity":"user-admin","last-connection":"2017-03-16T15:08:42Z","controller-access":"superuser","model-access":""},"facades":[{"name":"AllModelWatcher","versions":[2]},{"name":"Bundle","versions":[1]},{"name":"Cloud","versions":[1]},{"name":"Controller","versions":[3]},{"name":"HighAvailability","versions":[2]},{"name":"MigrationTarget","versions":[1]},{"name":"ModelManager","versions":[2]},{"name":"Pinger","versions":[1]},{"name":"UserManager","versions":[1]}],"server-version":"2.0.0"}}')
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 3,\n  "request-id": 3,\n  "request": "AllModels",\n  "type": "Controller",\n  "params": {}\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":3,"response":{"user-models":[{"model":{"name":"controller","uuid":"58d66c4c-c541-4785-8504-b52e278b7b95","owner-tag":"user-admin"},"last-connection":"2017-03-16T15:11:43Z"},{"model":{"name":"default","uuid":"68770514-8f40-4c9b-8a59-c89ae572e903","owner-tag":"user-admin"},"last-connection":"2017-03-16T14:11:40Z"}]}}')
Traceback (most recent call last):
  File "test.py", line 36, in <module>
    loop.run_until_complete(run())
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 27, in run
    print(result.to_json())
  File "/usr/local/lib/python3.5/dist-packages/juju/client/facade.py", line 459, in to_json
    return json.dumps(self.serialize())
  File "/usr/lib/python3.5/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <juju.client._client.UserModel object at 0x7fdd2840d0b8> is not JSON serializable
ERROR:asyncio:Fatal write error on socket transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fdd278b39b0>
transport: <_SelectorSocketTransport fd=6>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 700, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fdd278b39b0>
transport: <_SelectorSocketTransport closing fd=6>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 700, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/sslproto.py", line 632, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 704, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 589, in _fatal_error
    self._force_close(exc)
  File "/usr/lib/python3.5/asyncio/selector_events.py", line 601, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
    handle = self._call_soon(callback, args)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
    self._check_closed()
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<WebSocketCommonProtocol.run() running at /usr/local/lib/python3.5/dist-packages/websockets/protocol.py:413> wait_for=<Future pending cb=[Task._wakeup()]>>

VERSION not included in pypi package

$ pip install juju
Collecting juju
  Downloading juju-0.1.0.tar.gz (115kB)
    100% |████████████████████████████████| 122kB 3.5MB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-g2u8_xcm/juju/setup.py", line 24, in <module>
        version=version.read_text().strip(),
      File "/usr/lib/python3.5/pathlib.py", line 1164, in read_text
        with self.open(mode='r', encoding=encoding, errors=errors) as f:
      File "/usr/lib/python3.5/pathlib.py", line 1151, in open
        opener=self._opener)
      File "/usr/lib/python3.5/pathlib.py", line 1005, in _opener
        return self._accessor.open(self, flags, mode)
      File "/usr/lib/python3.5/pathlib.py", line 371, in wrapped
        return strfunc(str(pathobj), *args)
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-build-g2u8_xcm/juju/VERSION'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-g2u8_xcm/juju/

Websocket responses are not matched by request-id

Websocket responses are asynchronous and have a request-id to match responses with the correct request. However, Connection.rpc just assumes that the first response it receives is the correct one. If multiple requests are issued, this can lead to incorrect behavior, like the log below which shows an attempt to deploy a charm before the request to add the charm has returned.

2017-03-06 19:36:18 TRACE juju.apiserver request_notifier.go:113 <- [3C5] user-admin {"request-id":4,"type":"Client","version":1,"request":"AddCharm","params":{"url":"cs:haproxy-40","channel":""}}
2017-03-06 19:36:18 TRACE juju.apiserver request_notifier.go:113 <- [3C5] user-admin {"request-id":5,"type":"Client","version":1,"request":"AddCharm","params":{"url":"cs:ghost-18","channel":""}}
2017-03-06 19:36:18 TRACE juju.apiserver request_notifier.go:113 <- [3C5] user-admin {"request-id":6,"type":"Client","version":1,"request":"AddCharm","params":{"url":"cs:mysql-56","channel":""}}
2017-03-06 19:36:19 TRACE juju.apiserver request_notifier.go:129 -> [3C5] user-admin {"request-id":4,"response":{}}
2017-03-06 19:36:19 TRACE juju.apiserver request_notifier.go:113 <- [3C5] user-admin {"request-id":7,"type":"Application","version":3,"request":"Deploy","params":{"applications":[{"application":"haproxy","series":"xenial","charm-url":"cs:haproxy-40","channel":"","num-units":1,"config-yaml":"haproxy: {}\n","constraints":{}}]}}
2017-03-06 19:36:19 TRACE juju.apiserver request_notifier.go:129 -> [3C5] user-admin {"request-id":6,"response":{}}
2017-03-06 19:36:19 TRACE juju.apiserver request_notifier.go:113 <- [3C5] user-admin {"request-id":8,"type":"Application","version":3,"request":"Deploy","params":{"applications":[{"application":"ghost","series":"xenial","charm-url":"cs:ghost-18","channel":"","num-units":1,"config-yaml":"ghost: {}\n","constraints":{}}]}}
2017-03-06 19:36:19 TRACE juju.apiserver.common errors.go:198 server RPC error [{github.com/juju/juju/apiserver/application/application.go:199: } {github.com/juju/juju/state/charm.go:600: charm "cs:ghost-18" not found}]
2017-03-06 19:36:19 TRACE juju.apiserver request_notifier.go:129 -> [3C5] user-admin {"request-id":8,"response":{"results":[{"error":{"message":"charm \"cs:ghost-18\" not found","code":"not found"}}]}}
2017-03-06 19:36:19 TRACE juju.apiserver request_notifier.go:113 <- [3C5] user-admin {"request-id":9,"type":"Application","version":3,"request":"Deploy","params":{"applications":[{"application":"mysql","series":"xenial","charm-url":"cs:mysql-56","channel":"","num-units":1,"config-yaml":"mysql:\n  dataset-size: 80%\n  max-connections: '20000'\n","constraints":{}}]}}
2017-03-06 19:36:19 TRACE juju.apiserver request_notifier.go:129 -> [3C5] user-admin {"request-id":5,"response":{}}

controller.destroy_model should block

controller.destroy_model should block until the model is destroyed and cleaned up in core. That way, we can avoid situations where we attempt to re-create a model with the same name, and get weird behavior.

To reproduce some weird behavior, do the following:

in examples/add_model.py, replace model_name = "addmodeltest-{}".format(uuid.uuid4()) with model_name='foo'
Run the script twice.
Note that it fails its trick with the ssh key on the second try.

Implement ssh

For purposes like Matrix's reboot and kill_juju_agent actions, "ssh" is probably less likely to error than "run".

Best way to retrieve model uuid?

I wrote a few functions to do get the model uuid. Take that string and pass it into controller.destroymodels()

in juju.client.connection()

@classmethod
async def get_model_uuid(cls):
    """Connect to the currently active model.

    """
    jujudata = JujuData()
    controller_name = jujudata.current_controller()
    models = jujudata.models()[controller_name]
    model_name = models['current-model']
    model_uuid = models['models'][model_name]['uuid']

    return model_uuid

in juju.models:

async def get_uuid(self):
    """Get the UUID of the current model.

    """
    uuid = await connection.Connection.get_model_uuid()
    return uuid

And I use the following to delete the model once I'm done with testing:

async def run():
model = Model()
controller = Controller()
await controller.connect_current()
await model.connect_current()
await controller.destroy_models(await model.get_uuid())

Would this be helpful in a PR? Is there a better way of obtaining the uuid?

Document process to generate new API bindings from juju golang source

Looking for the correct syntax to pass into "entities" to DestroyModels()

In the ModelManagerFacade Class, there is a method to DestroyModels

@ReturnMapping(ErrorResults)
  async def DestroyModels(self, entities):
      '''
      entities : typing.Sequence<+T_co>[~Entity]<~Entity>
      Returns -> typing.Sequence<+T_co>[~ErrorResult]<~ErrorResult>
      '''
      # map input types to rpc msg
      _params = dict()
      msg = dict(type='ModelManager', request='DestroyModels', version=2, params=_params)
      _params['entities'] = entities
      reply = await self.rpc(msg)
      return reply

I wrote the glue in juju/controllers.py to interact with this.. Example Snippet

    async def destroy_model(self, entities):
        """Add a model to this controller to Destroy.

        :param str entities: Name of the model

        """
        model_facade = client.ModelManagerFacade()
        model_facade.connect(self.connection)

        log.debug('Creating model %s', entities)

        model_info = await model_facade.DestroyModels(entities)

When writing my script to interact with libjuju , I'm not sure what to pass into destroy_model()
What format should I be using? Is there a list of prefixes that I should be using, for example:

await controller.destroy_models(
"model-" + string ? )

The error I see is: juju.errors.JujuAPIError: json: cannot unmarshal string into Go value of type []params.Entity

Expose in bundle deploy throws KeyError

Traceback (most recent call last):
  File "/home/johnsca/juju/matrix/matrix/model.py", line 163, in execute
    result = await self.execute_plugin(context, cmd, rule)
  File "/home/johnsca/juju/matrix/matrix/model.py", line 180, in execute_plugin
    result = await cmd(context, rule, self, event)
  File "/home/johnsca/juju/matrix/matrix/tasks/deploy.py", line 5, in deploy
    context.apps.extend(await context.juju_model.deploy(task.args['entity']))
  File "/home/johnsca/juju/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 832, in deploy
    await handler.execute_plan()
  File "/home/johnsca/juju/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 1232, in execute_plan
    result = await method(*step.args)
  File "/home/johnsca/juju/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 1393, in expose
    return await self.model.applications[application].expose()
KeyError: 'kibana'

Looks like the expose might be happening before the application is available in the model.

Travis CI integration

For unit tests at least, but...

I want to try running functional tests too. Should be possible to have the travis build install juju and lxd, bootstrap a lxd controller/model, and run real api tests on it. Travis doesn't offer xenial yet so there will be the extra effort of getting lxd installed on trusty, but doable I think. Worth a try anyway.

API Docs

Get some sphinx autodocs going. These will be horribly ugly at first but will at least give us a starting point to work from.

Handle local charms

Local charms currently don't work either for single charm deploy nor in a bundle. We'll need to pull in this code from pythonclient.

For the bundle support, I think we'll also need to munge the bundle to change the charm paths to a format accepted by the API (currently gives an error like charm path in application "topbeat" does not exist: /tmp/builds/topbeat).

asyncio & libjuju

I've been seeing this exception occurring in many of my libjuju scripts I have, Just to make sure it was not an issue with my code, I checked the libjuju/examples/*.py files and also see this error while running most of the example scripts.

I've been looking through asyncio bugs and have somewhat a good understanding of why this is happening. So I'm wondering if any other folks here have been hitting this error below?

<.SNIP.>

DEBUG:juju.model:Stopping watcher task
DEBUG:juju.model:Closing watcher connection
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=8, data=b'\x03\xe8')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":5,"error":"watcher was stopped","response":{}}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=8, data=b'\x03\xe8')
DEBUG:juju.model:Closing model connection
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=8, data=b'\x03\xe8')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=8, data=b'\x03\xe8')
DEBUG:juju.controller:Closing controller connection
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=8, data=b'\x03\xe8')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=8, data=b'\x03\xe8')
Exception ignored in: <bound method BaseEventLoop.del of <_UnixSelectorEventLoop running=False closed=True debug=False>>
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/base_events.py", line 431, in del
File "/usr/lib/python3.5/asyncio/unix_events.py", line 58, in close
File "/usr/lib/python3.5/asyncio/unix_events.py", line 139, in remove_signal_handler
File "/usr/lib/python3.5/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

model.get_status() returns a None value

controller_endpoint = 'x.x.x.x:17070'
user = '<user>'
password = '<pass>'
model_uuid = '<uuid>'


def execute_task(command):
    loop = asyncio.get_event_loop()
    loop.set_debug(False)
    loop.run_until_complete(command())

async def run(): 
    model = Model()
    await model.connect(controller_endpoint, model_uuid, user, password)
    print(model.get_status())

execute_task(run)

When I try to get my model status I see that a None is printed. My endpoint, user, password and model_uuid are right because when I replace model.get_status() by model.state.__dict__ I get the json with my model information

Add docstrings in juju/client/connection.py

Specifically in the Connection class.

Improve docstrings for auto-generated API bindings

Deploy needs to include resources

See https://github.com/conjure-up/conjure-up/blob/master/conjureup/juju.py#L458

No Method to Create a Model

Looking at the Class Model()

It appears that we have the ability to connect to an existing Model. Deploy a Unit and Run an Action.

It would be nice to have the ability to Create a New Model , Deploy a Unit and Run Actions in there. I see that the module juju/controller.py has this method stubbed out.

Problem with ModelModifyAccess

I'm writing the grant-wrapper in model.py, however, using the following code,

async def grant(self, username, acl='read'):  #line number 1117
        """Grant a user access to this model.

        :param str username: Username
        :param str acl: Access control ('read' or 'write')

        """
        model_facade = client.ModelManagerFacade()
        model_facade.connect(self.connection)
        user = tag.user(username)
        model = tag.model(self.info.name)
        action = 'revoke'
        changes = client.ModifyModelAccess(acl, action, user, model)
        await model_facade.ModifyModelAccess([changes])
        action = 'grant'
        changes = client.ModifyModelAccess(acl, action, user, model)
        return await model_facade.ModifyModelAccess([changes])

I get the following error:

DEBUG:asyncio:Using selector: EpollSelector
INFO:websocket:Driver connected to juju wss://xxx.xxx.xxx.xxx/model/0c6d354d-fbb0-41cc-856e-6942be7968a8/api
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 3,\n  "type": "Admin",\n  "request-id": 1,\n  "params": {},\n  "request": "RedirectInfo"\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":1,"error":"not redirected","response":{}}')
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 3,\n  "type": "Admin",\n  "request-id": 2,\n  "params": {\n    "credentials": "xxx",\n    "nonce": "DmMy(n\\r35*0&",\n    "macaroons": [],\n    "auth-tag": "user-admin"\n  },\n  "request": "Login"\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":2,"response":{"servers":[[{"value":"xxx.xxx.xxx.xxx","type":"ipv4","scope":"public","port":17070},{"value":"10.140.0.8","type":"ipv4","scope":"local-cloud","port":17070},{"value":"127.0.0.1","type":"ipv4","scope":"local-machine","port":17070},{"value":"::1","type":"ipv6","scope":"local-machine","port":17070}]],"model-tag":"model-0c6d354d-fbb0-41cc-856e-6942be7968a8","controller-tag":"controller-02e4ed54-b887-4a34-8ad5-4d5196837ec8","user-info":{"display-name":"","identity":"user-admin","last-connection":"2017-03-20T13:54:26Z","controller-access":"superuser","model-access":"admin"},"facades":[{"name":"Action","versions":[2]},{"name":"Agent","versions":[2]},{"name":"AgentTools","versions":[1]},{"name":"AllWatcher","versions":[1]},{"name":"Annotations","versions":[2]},{"name":"Application","versions":[1,2,3]},{"name":"ApplicationRelationsWatcher","versions":[1]},{"name":"ApplicationScaler","versions":[1]},{"name":"Backups","versions":[1]},{"name":"Block","versions":[2]},{"name":"Bundle","versions":[1]},{"name":"CharmRevisionUpdater","versions":[2]},{"name":"Charms","versions":[2]},{"name":"Cleaner","versions":[2]},{"name":"Client","versions":[1]},{"name":"Deployer","versions":[1]},{"name":"DiscoverSpaces","versions":[2]},{"name":"DiskManager","versions":[2]},{"name":"EntityWatcher","versions":[2]},{"name":"FilesystemAttachmentsWatcher","versions":[2]},{"name":"Firewaller","versions":[3]},{"name":"HighAvailability","versions":[2]},{"name":"HostKeyReporter","versions":[1]},{"name":"ImageManager","versions":[2]},{"name":"ImageMetadata","versions":[2]},{"name":"InstancePoller","versions":[3]},{"name":"KeyManager","versions":[1]},{"name":"KeyUpdater","versions":[1]},{"name":"LeadershipService","versions":[2]},{"name":"LifeFlag","versions":[1]},{"name":"LogForwarding","versions":[1]},{"name":"Logger","versions":[1]},{"name":"MachineActions","versions":[1]},{"name":"MachineManager","versions":[2]},{"name":"MachineUndertaker","versions":[1]},{"name":"Machiner","versions":[1]},{"name":"MeterStatus","versions":[1]},{"name":"MetricsAdder","versions":[2]},{"name":"MetricsDebug","versions":[2]},{"name":"MetricsManager","versions":[1]},{"name":"MigrationFlag","versions":[1]},{"name":"MigrationMaster","versions":[1]},{"name":"MigrationMinion","versions":[1]},{"name":"MigrationStatusWatcher","versions":[1]},{"name":"ModelConfig","versions":[1]},{"name":"NotifyWatcher","versions":[1]},{"name":"Payloads","versions":[1]},{"name":"PayloadsHookContext","versions":[1]},{"name":"Pinger","versions":[1]},{"name":"Provisioner","versions":[3]},{"name":"ProxyUpdater","versions":[1]},{"name":"Reboot","versions":[2]},{"name":"RelationUnitsWatcher","versions":[1]},{"name":"Resources","versions":[1]},{"name":"ResourcesHookContext","versions":[1]},{"name":"Resumer","versions":[2]},{"name":"RetryStrategy","versions":[1]},{"name":"SSHClient","versions":[1,2]},{"name":"Singular","versions":[1]},{"name":"Spaces","versions":[2]},{"name":"StatusHistory","versions":[2]},{"name":"Storage","versions":[3]},{"name":"StorageProvisioner","versions":[3]},{"name":"StringsWatcher","versions":[1]},{"name":"Subnets","versions":[2]},{"name":"Undertaker","versions":[1]},{"name":"UnitAssigner","versions":[1]},{"name":"Uniter","versions":[4]},{"name":"Upgrader","versions":[1]},{"name":"VolumeAttachmentsWatcher","versions":[2]}],"server-version":"2.0.3"}}')
DEBUG:juju.model:Starting watcher task
INFO:websocket:Driver connected to juju wss://xxx.xxx.xxx.xxx:17070/model/0c6d354d-fbb0-41cc-856e-6942be7968a8/api
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 3,\n  "type": "Admin",\n  "request-id": 1,\n  "params": {},\n  "request": "RedirectInfo"\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":1,"error":"not redirected","response":{}}')
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 3,\n  "type": "Admin",\n  "request-id": 2,\n  "params": {\n    "credentials": "xxx",\n    "nonce": "hm5$<ZD\\u000b \'E\\"",\n    "macaroons": [],\n    "auth-tag": "user-admin"\n  },\n  "request": "Login"\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":2,"response":{"servers":[[{"value":"35.185.152.227","type":"ipv4","scope":"public","port":17070},{"value":"10.140.0.8","type":"ipv4","scope":"local-cloud","port":17070},{"value":"127.0.0.1","type":"ipv4","scope":"local-machine","port":17070},{"value":"::1","type":"ipv6","scope":"local-machine","port":17070}]],"model-tag":"model-0c6d354d-fbb0-41cc-856e-6942be7968a8","controller-tag":"controller-02e4ed54-b887-4a34-8ad5-4d5196837ec8","user-info":{"display-name":"","identity":"user-admin","last-connection":"2017-03-20T13:54:42Z","controller-access":"superuser","model-access":"admin"},"facades":[{"name":"Action","versions":[2]},{"name":"Agent","versions":[2]},{"name":"AgentTools","versions":[1]},{"name":"AllWatcher","versions":[1]},{"name":"Annotations","versions":[2]},{"name":"Application","versions":[1,2,3]},{"name":"ApplicationRelationsWatcher","versions":[1]},{"name":"ApplicationScaler","versions":[1]},{"name":"Backups","versions":[1]},{"name":"Block","versions":[2]},{"name":"Bundle","versions":[1]},{"name":"CharmRevisionUpdater","versions":[2]},{"name":"Charms","versions":[2]},{"name":"Cleaner","versions":[2]},{"name":"Client","versions":[1]},{"name":"Deployer","versions":[1]},{"name":"DiscoverSpaces","versions":[2]},{"name":"DiskManager","versions":[2]},{"name":"EntityWatcher","versions":[2]},{"name":"FilesystemAttachmentsWatcher","versions":[2]},{"name":"Firewaller","versions":[3]},{"name":"HighAvailability","versions":[2]},{"name":"HostKeyReporter","versions":[1]},{"name":"ImageManager","versions":[2]},{"name":"ImageMetadata","versions":[2]},{"name":"InstancePoller","versions":[3]},{"name":"KeyManager","versions":[1]},{"name":"KeyUpdater","versions":[1]},{"name":"LeadershipService","versions":[2]},{"name":"LifeFlag","versions":[1]},{"name":"LogForwarding","versions":[1]},{"name":"Logger","versions":[1]},{"name":"MachineActions","versions":[1]},{"name":"MachineManager","versions":[2]},{"name":"MachineUndertaker","versions":[1]},{"name":"Machiner","versions":[1]},{"name":"MeterStatus","versions":[1]},{"name":"MetricsAdder","versions":[2]},{"name":"MetricsDebug","versions":[2]},{"name":"MetricsManager","versions":[1]},{"name":"MigrationFlag","versions":[1]},{"name":"MigrationMaster","versions":[1]},{"name":"MigrationMinion","versions":[1]},{"name":"MigrationStatusWatcher","versions":[1]},{"name":"ModelConfig","versions":[1]},{"name":"NotifyWatcher","versions":[1]},{"name":"Payloads","versions":[1]},{"name":"PayloadsHookContext","versions":[1]},{"name":"Pinger","versions":[1]},{"name":"Provisioner","versions":[3]},{"name":"ProxyUpdater","versions":[1]},{"name":"Reboot","versions":[2]},{"name":"RelationUnitsWatcher","versions":[1]},{"name":"Resources","versions":[1]},{"name":"ResourcesHookContext","versions":[1]},{"name":"Resumer","versions":[2]},{"name":"RetryStrategy","versions":[1]},{"name":"SSHClient","versions":[1,2]},{"name":"Singular","versions":[1]},{"name":"Spaces","versions":[2]},{"name":"StatusHistory","versions":[2]},{"name":"Storage","versions":[3]},{"name":"StorageProvisioner","versions":[3]},{"name":"StringsWatcher","versions":[1]},{"name":"Subnets","versions":[2]},{"name":"Undertaker","versions":[1]},{"name":"UnitAssigner","versions":[1]},{"name":"Uniter","versions":[4]},{"name":"Upgrader","versions":[1]},{"name":"VolumeAttachmentsWatcher","versions":[2]}],"server-version":"2.0.3"}}')
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 1,\n  "type": "Client",\n  "request-id": 3,\n  "params": {},\n  "request": "WatchAll"\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":3,"response":{"watcher-id":"1"}}')
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "type": "AllWatcher",\n  "params": {},\n  "request": "Next",\n  "Id": "1",\n  "version": 1,\n  "request-id": 4\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":4,"response":{"deltas":[]}}')
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "type": "AllWatcher",\n  "params": {},\n  "request": "Next",\n  "Id": "1",\n  "version": 1,\n  "request-id": 5\n}')
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 1,\n  "type": "Client",\n  "request-id": 3,\n  "params": {},\n  "request": "ModelInfo"\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":3,"response":{"name":"default","uuid":"0c6d354d-fbb0-41cc-856e-6942be7968a8","controller-uuid":"","provider-type":"gce","default-series":"xenial","cloud-tag":"cloud-google","cloud-region":"asia-east1","cloud-credential-tag":"cloudcred-google_admin_qrama","owner-tag":"user-admin","life":"alive","status":{"status":"","info":"","since":null},"users":null,"machines":null}}')
DEBUG:juju.model:Got ModelInfo: {'life': 'alive', 'provider_type': 'gce', 'owner_tag': 'user-admin', 'name': 'default', 'users': [], 'machines': [], 'cloud_credential_tag': 'cloudcred-google_admin_qrama', 'default_series': 'xenial', 'cloud_tag': 'cloud-google', 'status': <juju.client._client.EntityStatus object at 0x7f55a0786b70>, 'cloud_region': 'asia-east1', 'controller_uuid': '', 'uuid': '0c6d354d-fbb0-41cc-856e-6942be7968a8'}
DEBUG:websockets.protocol:client >> Frame(fin=True, opcode=1, data=b'{\n  "version": 2,\n  "type": "ModelManager",\n  "request-id": 4,\n  "params": {\n    "changes": [\n      {\n        "model-tag": "user-test",\n        "action": "revoke",\n        "access": "write",\n        "user-tag": "model-default"\n      }\n    ]\n  },\n  "request": "ModifyModelAccess"\n}')
DEBUG:websockets.protocol:client << Frame(fin=True, opcode=1, data=b'{"request-id":4,"error":"facade \\"ModelManager\\" not supported for model API connection","error-code":"not supported","response":{}}')
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run() done, defined at test.py:9> exception=JujuAPIError('facade "ModelManager" not supported for model API connection',)>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 25, in run
    res = await model.grant('test', 'write')
  File "/usr/local/lib/python3.5/dist-packages/juju/model.py", line 1130, in grant
    await model_facade.ModifyModelAccess([changes])
  File "/usr/local/lib/python3.5/dist-packages/juju/client/facade.py", line 317, in wrapper
    reply = await f(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/juju/client/_client.py", line 15279, in ModifyModelAccess
    reply = await self.rpc(msg)
  File "/usr/local/lib/python3.5/dist-packages/juju/client/facade.py", line 436, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 95, in rpc
    raise JujuAPIError(result)
juju.errors.JujuAPIError: facade "ModelManager" not supported for model API connection

Testscript

import asyncio
import logging

from juju.controller import Controller
from juju.model import Model

logging.basicConfig(level=logging.DEBUG)

async def run():
    model = Model()

    controller_endpoint = 'xxx.xxx.xxx.xxx:17070'
    username = 'admin'
    password = 'xxx'
    muuid = '0c6d354d-fbb0-41cc-856e-6942be7968a8'

    cacert = None

    await model.connect(
        controller_endpoint,
        muuid,
        username,
        password,
    )
    res = await model.grant('test', 'write')
    print(res.serialize()['results'][0].serialize())
    await controller.disconnect()


logging.basicConfig(level=logging.DEBUG)
ws_logger = logging.getLogger('websockets.protocol')
ws_logger.setLevel(logging.DEBUG)
loop = asyncio.get_event_loop()
loop.set_debug(False)
loop.create_task(run())
loop.run_forever()

Cannot ssh into units in model created by libjuju

Originally reported in juju-solutions/matrix#59

If a model is created with libjuju and an application deployed into that model, juju ssh to that unit fails.

It doesn't matter if the application is deployed with libjuju or the CLI, juju ssh will fail if the model was created with libjuju and succeed if the model was created by the CLI.

The exact error is:

$ juju ssh ubuntu/0
Received disconnect from 54.166.31.6 port 22:2: Too many authentication failures
Connection to 54.166.31.6 closed by remote host.
Connection to 54.166.31.6 closed.

Package name conflicts with existing package on pypi

https://pypi.python.org/pypi/juju

That appears to be a very old version of the Python implementation of juju. We can probably reclaim the name from Kapil.

Replace "pass" with NotImplementedError in method stubs

Destroy_controller

The destroy controller function does not destroy the controller. It removes all the models, but does not remove the controller model. This model remains in status terminating, so the controller never get's removed.

No Model.connect method that accepts a Connection

There's no straightforward way to associate an existing Connection with a Model. Simply assigning to Model.connection won't trigger Model._after_connect.

Bundles with resources fail to deploy

To reproduce:

Attempt to deploy a bundle containing resources.
Get this traceback:

(from matrix, trying to deploy landscape)

Traceback for <Task finished coro=<RuleEngine.rule_runner() done, defined at /home/petevg/Code/matrix/matrix/rules.py:175> exception=JSONDecodeError('Expecting value: line 1 column 1 (char 0)',)> (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 292, in _step
    self = None  # Needed to break cycles when an exception occurs.
  File "/home/petevg/Code/matrix/matrix/rules.py", line 263, in rule_runner
    break
  File "/home/petevg/Code/matrix/matrix/model.py", line 344, in execute
    result = await self.task.execute(context, self)
  File "/home/petevg/Code/matrix/matrix/model.py", line 171, in execute
    raise
  File "/home/petevg/Code/matrix/matrix/model.py", line 181, in execute_plugin
    result = await cmd(context, rule, self, event)
  File "/home/petevg/Code/matrix/matrix/tasks/deploy.py", line 5, in deploy
    await context.juju_model.deploy(str(context.config.path))
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 880, in deploy
    await handler.execute_plan()
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 1314, in execute_plan
    result = await method(*step.args)
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 1442, in deploy
    resources=resources,
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/client/_client.py", line 620, in __init__
    self.constraints = Value.from_json(constraints) if constraints else None
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/client/facade.py", line 442, in from_json
    data = json.loads(data)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This is related to translating stuff from the Planner yaml into Value objects that can be packaged up in JSON for the core API. Apparently, we need to translate Resources in a way similar to the way that we translate constraints. (see constraints.py)

Error: no credentials provided

When running one of the provided examples (for instance, the one given in the Quickstart in the docs), I get a no credentials provided JuJuAPIError. But there are valid macaroons given in the login function.
Running Python 3.5.2 on Ubuntu 16.4, with the latest version of python-libjuju installed.

DEBUG:asyncio:Using selector: EpollSelector
INFO:websocket:Driver connected to juju wss://xxx.xxx.xxx.xxx:17070/model/85d19189-e553-4885-8939-81a49a582420/api
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run() done, defined at test.py:9> exception=JujuAPIError('no credentials provided',)>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 15, in run
    await model.connect_current()
  File "/usr/local/lib/python3.5/dist-packages/juju/model.py", line 396, in connect_current
    self.connection = await connection.Connection.connect_current()
  File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 222, in connect_current
    '{}:{}'.format(controller_name, model_name))
  File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 274, in connect_model
    endpoint, model_uuid, username, password, cacert, macaroons)
  File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 186, in connect
    await client.login(username, password, macaroons)
  File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 297, in login
    "macaroons": macaroons or []
  File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 95, in rpc
    raise JujuAPIError(result)
juju.errors.JujuAPIError: no credentials provided

Add support for fetching and discharging new macaroons

The library currently has no support for doing this. You can use macaroon auth, but you must already possess valid macaroons for the controller or model you are connecting to. In practice this usually means you have connected with the juju cli, which gives you valid macaroons in ~/.go-cookies.

This was a good quick-and-dirty first start for enabling libjuju to connect to shared controllers, but it's not a long term solution and the pain of this approach is already being felt. It's important that we implement proper macaroon support asap.

Wrong params in auto-generated code

See ClientFacade.AddMachines() for one example. It accepts arg 'params' of type AddMachineParams, but it should actually accept 'params' of type AddMachines.

Not sure what the proper fix is for this yet. Either the expected type should change, or the auto-generated method body (of AddMachines(), in this example) would need to wrap the passed 'params' in a call to AddMachines.

Missing calls to parse_constraints in model.py

Looks like we missed some places where we need to parse constraints that we get back from the planner in in model.py.

To reproduce:

Deploy the landscape bundle
Note the following traceback (actual traceback from matrix, but the problem is in python-libjuju):

matrix:170:execute: Error in Rule(task=Task(command='matrix.tasks.deploy', args={}), conditions=[])'s task Task(command='matrix.tasks.deploy', args={})
Traceback (most recent call last):
  File "/home/petevg/Code/matrix/matrix/model.py", line 164, in execute
    result = await self.execute_plugin(context, cmd, rule)
  File "/home/petevg/Code/matrix/matrix/model.py", line 181, in execute_plugin
    result = await cmd(context, rule, self, event)
  File "/home/petevg/Code/matrix/matrix/tasks/deploy.py", line 5, in deploy
    await context.juju_model.deploy(str(context.config.path))
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 880, in deploy
    await handler.execute_plan()
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 1314, in execute_plan
    result = await method(*step.args)
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 1442, in deploy
    resources=resources,
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/client/_client.py", line 620, in __init__
    self.constraints = Value.from_json(constraints) if constraints else None
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/client/facade.py", line 442, in from_json
    data = json.loads(data)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
matrix:313:run_once: Pending tasks remain, aborting due to failure
matrix:322:run_once: Exception processing test: deployment
Traceback for <Task finished coro=<RuleEngine.rule_runner() done, defined at /home/petevg/Code/matrix/matrix/rules.py:175> exception=JSONDecodeError('Expecting value: line 1 column 1 (char 0)',)> (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 292, in _step
    self = None  # Needed to break cycles when an exception occurs.
  File "/home/petevg/Code/matrix/matrix/rules.py", line 263, in rule_runner
    break
  File "/home/petevg/Code/matrix/matrix/model.py", line 344, in execute
    result = await self.task.execute(context, self)
  File "/home/petevg/Code/matrix/matrix/model.py", line 171, in execute
    raise
  File "/home/petevg/Code/matrix/matrix/model.py", line 181, in execute_plugin
    result = await cmd(context, rule, self, event)
  File "/home/petevg/Code/matrix/matrix/tasks/deploy.py", line 5, in deploy
    await context.juju_model.deploy(str(context.config.path))
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 880, in deploy
    await handler.execute_plan()
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 1314, in execute_plan
    result = await method(*step.args)
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/model.py", line 1442, in deploy
    resources=resources,
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/client/_client.py", line 620, in __init__
    self.constraints = Value.from_json(constraints) if constraints else None
  File "/home/petevg/Code/matrix/.tox/py35/lib/python3.5/site-packages/juju/client/facade.py", line 442, in from_json
    data = json.loads(data)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Model.add_machine is an attractive nuisance

add_machine is documented as if it does something, but it does nothing. The fact that it exists and is documented is an attractive nuisance, because it leads people to try to use it.

Would-be users will eventually discover that its implementation is "pass", but it should be easier to know that it's not implemented.

Some alternatives:

it could be deleted until it is implemented.
it could raise NotImplementedError

Integration tests

I'd very much like it if python-libjuju had integration tests with the following properties:

They don't run by default, but can be run with "tox integration" or similar. (This is because they may take a long time to run.)
They all inherit from a class that creates a new, uniquely named model, runs tests, then tears down that model (this way, the tests won't accidentally clobber your production code.)
They sanity check deploys for wiki-simple and a bundle with placement and machine directives. (This would save us from my laziness, where I tested a bundle with placement and machine directives, but didn't test the simpler case, and managed to break addUnits.)

I'll try to set aside and write something like the above -- I wanted to think aloud here in case other people had feedback, or somebody has the spare time right now to write them, though :-)

Error: unkown version (3) of interface "Application"

When trying to deploy an application to a model, an APIError is thrown: unknown version (3) of interface "Application"

DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.jujucharms.com
DEBUG:requests.packages.urllib3.connectionpool:https://api.jujucharms.com:443 "GET /charmstore/v5/ubuntu-10/meta/any HTTP/1.1" 200 21
DEBUG:juju.model:Deploying cs:ubuntu-10
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run() done, defined at test.py:7> exception=JujuAPIError('unknown version (3) of interface "Application"',)>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 33, in run
    channel='stable',
  File "/usr/local/lib/python3.5/dist-packages/juju/model.py", line 1045, in deploy
    result = await app_facade.Deploy([app])
  File "/usr/local/lib/python3.5/dist-packages/juju/client/facade.py", line 317, in wrapper
    reply = await f(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/juju/client/_client.py", line 7618, in Deploy
    reply = await self.rpc(msg)
  File "/usr/local/lib/python3.5/dist-packages/juju/client/facade.py", line 436, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/usr/local/lib/python3.5/dist-packages/juju/client/connection.py", line 95, in rpc
    raise JujuAPIError(result)
juju.errors.JujuAPIError: unknown version (3) of interface "Application"

can't deploy hadoop-processing with python-libjuju

I recently ran cwr-ci on https://jujucharms.com/hadoop-processing/. CWR calls bundletester which calls matrix, which does libjuju stuff.

I think it adds apps, machines, and then adds units -- it feels like the add-unit piece isn't working. A matrix model was created, apps were added, machines were added, but no units ever showed up:

$ juju status -m matrix-close-dingo                                                                                                                                                                                   Thu Mar  2 15:26:05 2017

Model               Controller  Cloud/Region         Version
matrix-close-dingo  lxd         localhost/localhost  2.0.3

App                   Version  Status       Scale  Charm                   Store       Rev  OS      Notes
client                         blocked          0  hadoop-client           jujucharms    3  ubuntu
ganglia                        waiting          0  ganglia                 jujucharms    5  ubuntu
ganglia-node                   waiting        0  ganglia-node            jujucharms    6  ubuntu
namenode                       maintenance      0  hadoop-namenode         jujucharms    8  ubuntu
plugin                         maintenance      0  hadoop-plugin           jujucharms    8  ubuntu
resourcemanager                maintenance      0  hadoop-resourcemanager  jujucharms    8  ubuntu
rsyslog                        waiting          0  rsyslog                 jujucharms    7  ubuntu
rsyslog-forwarder-ha           waiting        0  rsyslog-forwarder-ha    jujucharms    7  ubuntu
slave                          maintenance      0  hadoop-slave            jujucharms    8  ubuntu

Unit                       Workload     Agent       Machine  Public address  Ports  Message
### HEY!  THERE'S NOTHING HERE

Machine  State    DNS           Inst id        Series  AZ
0        started  10.38.19.117  juju-d91d66-0  xenial
1        started  10.38.19.56   juju-d91d66-1  xenial
2        started  10.38.19.225  juju-d91d66-2  xenial
3        started  10.38.19.79   juju-d91d66-3  xenial
4        started  10.38.19.119  juju-d91d66-4  xenial
5        started  10.38.19.166  juju-d91d66-5  xenial

Relation         Provides         Consumes              Type
juju-info        client           ganglia-node          subordinate
hadoop-plugin    client           plugin                subordinate
juju-info        client           rsyslog-forwarder-ha  subordinate
node             ganglia          ganglia-node          regular
...

I was able to manually go in and add units like this:

juju add-unit -m matrix-close-dingo client --to 0

But I'd like to not have to do that. @petevg is on the case!

JujuData.models() does not reflect current running models , when created via juju.controller.addmodel

After I create a new model with the following:

    controller = Controller()
    await controller.connect_current()

    model = await controller.add_model(
        'libjujutest',
        'cloud-localhost',
        'cloudcred-localhost_admin_default',
    )

I can easily deploy a charm as outlined in examples/controller.py

doing.

    await model.deploy(
        'ubuntu-0',
        service_name='ubuntu',
        series='xenial',
        channel='stable',
    )

However If I try to use methods from the juju.model library It appears as if the newly created model does not exist at least according to the api.

For Example: If I try to connect to the newly created model using this example below. (Which utilizes juju.model.ModelObserver() to run a series of tests) I see an error.

async def run():
    controller = Controller()
    await controller.connect_current()

    await controller.add_model(
        'libjujutest',
        'cloud-localhost',
        'cloudcred-localhost_admin_default',
    )

    model = Model()
    await model.connect_model(
        'localhost-lxd:admin/libjujutest'
         )
    model.add_observer(MyModelObserver())
    await model.deploy(
        'ubuntu-0',
        service_name='ubuntu',
        series='xenial',
        channel='stable',
    )

DEBUG:asyncio:Using selector: EpollSelector
INFO:websocket:Driver connected to juju wss://10.0.8.157:17070/api
DEBUG:LibJuju.juju.controller:Creating model libjujutest
INFO:websocket:Driver connected to juju wss://10.0.8.157:17070/model/97153d75-564f-400d-88dd-db85e6009695/api
DEBUG:LibJuju.juju.model:Starting watcher task
INFO:websocket:Driver connected to juju wss://10.0.8.157:17070/model/97153d75-564f-400d-88dd-db85e6009695/api
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run() done, defined at ./controller.py:39> exception=KeyError('admin/libjujutest',)>
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "./controller.py", line 52, in run
'localhost-lxd:admin/libjujutest'
File "/home/sfeole/git/hyperscale/lib/LibJuju/juju/model.py", line 375, in connect_model
self.connection = await connection.Connection.connect_model(arg)
File "/home/sfeole/git/hyperscale/lib/LibJuju/juju/client/connection.py", line 220, in connect_model
model_uuid = models['models'][model_name]['uuid']
KeyError: 'admin/libjujutest'

If the model is created via the command line, "juju add-model FOO" I usually don't see the problem when trying to run model.connect_controller(). And also found that juju.client.connections JuJuData.models() sees the model I created via the CLI. But that defeats the purpose of using the api.

I've looked in juju.cliient.client and juju.client.connections when I see this error. It appears that Jujudata.models() knows nothing about the newly created model by juju.controller.add_model()
If I query JuJuData.Models() during the lifetime of the script, I never see the model, "libtest-juju"


In [10]: JUJU = connection.JujuData()

In [11]: JUJU.models()
Out[11]: 
{'localhost-lxd': {'models': {'admin/controller': {'uuid': 'cb782591-ebfe-44a4-8371-abe3943448db'},
   'admin/seantest': {'uuid': '2450573f-27ae-4dd3-81b6-37b293347ad2'},
   'admin/default': {'uuid': '4c5fd23c-d25a-4d84-8c93-fccd2a3089a8'}}}}

If I run juju list-models via the CLI,

Model        Cloud/Region         Status     Machines  Cores  Access  Last connection
controller   localhost/localhost  available         1      -  admin   just now
default      localhost/localhost  available         0      -  admin   4 hours ago
seantest*    localhost/localhost  available         0      -  admin   never connected
libjujutest  localhost/localhost  available         1      -  admin   5 minutes ago

There is the model, in fact alive and well and has machines attached and units "Ubuntu/0" which we deployed.


sfeole@sfmadmax:~$ juju switch libjujutest
localhost-lxd:admin/seantest -> localhost-lxd:admin/libjujutest
sfeole@sfmadmax:~$ juju status
Model        Controller     Cloud/Region         Version
libjujutest  localhost-lxd  localhost/localhost  2.0.2.1

App     Version  Status   Scale  Charm   Store       Rev  OS      Notes
ubuntu           unknown      1  ubuntu  jujucharms    0  ubuntu  

Unit       Workload  Agent  Machine  Public address  Ports  Message
ubuntu/0*  unknown   idle   0        10.0.8.28              

Machine  State    DNS        Inst id        Series  AZ
0        started  10.0.8.28  juju-38a03a-0  xenial

By running juju switch I bring the model in focus, after doing so. I can now see the model in connection.JuJuData() and run my script after commenting out the "add_model()" method.

Does the following code in juju.controllers.add_model() automatically bring the model "in focus" or "connect" to it after creation?

        model = Model()
        await model.connect(
            self.connection.endpoint,
            model_info.uuid,
            self.connection.username,
            self.connection.password,
            self.connection.cacert,
            self.connection.macaroons,
        )

        return model

juju / python-libjuju Goto Github PK

python-libjuju's Introduction

A Python library for Juju

Requirements

Design Notes

Installation

Quickstart

REPL

Versioning

python-libjuju's People

Contributors

Stargazers

Watchers

Forkers

python-libjuju's Issues

Recommend Projects

Recommend Topics

Recommend Org