There are still a lot of places where a connection reset by peer will bug swiftly.client.Client._request out. I fixed it at the getresponse point where it was happening quite often to me, but I've got to get the other places fixed. I know they're there and know how to fix them, I've just got to get it done.

Even though it'd have to be Rackspace specific (at least for now), it'd be useful if Swiftly could output the CDN URLs for various items.

It would be awesome if get's support the -n/d flag to only retrieve newer/changed files.

Per discussion with @ostackbrian and others:

Swiftly should have a feature for downloading segmented objects where each segment is downloaded in parallel with etag checking. It could download the segments to different slices of the same on disk file to conserve space, time, and disk fragmentation.

Related: Normal object downloads could also check etags.

With no swiftly configuration set, doing swiftly trans emits an error about the auth url not being set up even though it isn't going to make any service calls.

Hi,

I have installed swiftly using pip:

$ swiftly --version
2.04

$ swiftly get
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/swiftly/cli/cli.py", line 448, in _perform_command
self.commandscommand
File "/Library/Python/2.7/site-packages/swiftly/cli/get.py", line 513, in call
return cli_get(context, path)
File "/Library/Python/2.7/site-packages/swiftly/cli/get.py", line 282, in cli_get
return cli_get_account_listing(context)
File "/Library/Python/2.7/site-packages/swiftly/cli/get.py", line 110, in cli_get_account_listing
with context.client_manager.with_client() as client:
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "/Library/Python/2.7/site-packages/swiftly/client/manager.py", line 75, in with_client
client = self.get_client()
File "/Library/Python/2.7/site-packages/swiftly/client/manager.py", line 56, in get_client
client = self.client_class(_self.args, *_kwargs)
File "/Library/Python/2.7/site-packages/swiftly/client/standardclient.py", line 140, in init
self.sleep = eventlet.sleep
AttributeError: 'module' object has no attribute 'sleep'

The eventlet version is
$ pip list | grep eventlet
eventlet (0.14.0)

Can you please help me with this? Am I doing something wrong?

Thanks

With verbose mode turned on, swiftly does not return transaction IDs. This may be especially important for large object PUTs, where one swiftly command may result in many transactions on Swift.

It'd be nice when using swiftly put -i dir/ container to be ablet o have a progress bar, optionally, for how far along it is.

swiftly for ... do works great if you specify all the auth information via the SWIFTLY_* environment variables, but breaks if you use the -A -U -K command line options style. I'm sure I just need to make sure the context gets copied from the original CLI context, but making this issue so I don't forget to get it done.

These are great for put and should be added to get.

Update to support Swift's Swob or WebOb. Swob is new in the 1.7.5 release of Swift and means that WebOb isn't required by Swift anymore. So, Swiftly really shouldn't require WebOb either.

Some systems, such as Windows 8.1, use subsecond mtimes. This causes unnecessary uploads using put -d or put -n.

I love to have a 'swiftly clone' command that could send one cluster's container worth of objects to another cluster's container. Maybe something like: swiftly clone localcontainer -A <remote_auth_url> -U <remote_user> -K <remote_key> remotecontainer

There's a bug where if you do swiftly get container/object -o somedir/ the object is sent to stdout instead of landing in the output directory. swiftly get container/object -o somefile works fine.

Reported by word of mouth:

Make objects bob and bob/sue and then try to download with --all-objects

To reproduce: create a static large object then delete a segment (not the first, though, there's a bug in swift where that will result in a 500 on the swift side).

download output:
VERBOSE 0.00 1 Attempting auth v2 RAX-KSKEY:apiKeyCredentials with https://identity.api.rackspacecloud.com/v2.0
VERBOSE 0.00 1 Establishing HTTPS connection to identity.api.rackspacecloud.com
VERBOSE 0.00 1 > POST /v2.0/tokens
VERBOSE 0.00 1 > {"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "xxxxx", "apiKey": "000000"}}}
VERBOSE 0.46 1 < 200 OK
VERBOSE 0.46 1 Establishing HTTPS connection to snet-storage101.iad3.clouddrive.com
VERBOSE 0.48 1 > GET /v1/MossoCloudFS_xxxxx/largeobjtest/BioCoder_Winter2014.epub User-Agent: Swiftly v2.02 X-Auth-Token: 00000
VERBOSE 0.70 1 < 200 OK

note that the error code returned is 0

$ echo $?
0
$ ls -l test.download.epub
-rw-rw-r-- 1 brianr brianr 2097152 Apr 18 16:14 test.download.epub

When you do a swiftly head, you can see what the content length should be:
Content-Length: 3554828
Content-Type: application/octet-stream
Etag: "ea2ab35ca8df0b430602ff758136a8b5"
Last-Modified: Sun, 02 Mar 2014 05:02:59 GMT
X-Static-Large-Object: True
X-Timestamp: 1393736578.95527
X-Trans-Id: txd1a21961c062419dbd386-005351532eiad3

Here's how the python-swiftclient handles this:
swift --os-auth-token $API_AUTH_TOKEN
--os-storage-url $SWIFT_ENDPOINT
download
-o /dev/null
${CONTAINER} ${OBJECT}

output:

BioCoder_Winter2014.epub [headers 0.737s, total 7.889s, 0.266s MB/s]
BioCoder_Winter2014.epub: md5sum != etag, 6e48448aa823613256386664e115d457 != "ea2ab35ca8df0b430602ff758136a8b5"
BioCoder_Winter2014.epub: read_length != content_length, 2097152 != 3554828
$ echo $?
1

Delay update until the next Cloud Files release when POST bulk deletes will actually work.

Fix bug where specifying an invalid region will just silently use the default region.

Downloading all objects in account resulted in objects in ALL THE RANDOM PLACES!

Reported by @pandemicsyn

Enhance ping reports to report the count of requests past a given threshold and the percentage of requests past a given threshold.

swiftly -A http://auth.stg.swift.racklabs.com/auth/v1.0 -U lunrbackupsclab:lunrclab -K

Traceback (most recent call last):
File "/usr/local/bin/swiftly", line 7, in
execfile(file) │
File "/home/letterj/repos-pub/swiftly/bin/swiftly", line 4, in
exit(CLI().main()) │
File "/home/letterj/repos-pub/swiftly/swiftly/cli.py", line 482, in main
self._main_options, args = self._main_parser.parse_args(self.args)
File "/usr/lib/python2.7/optparse.py", line 1399, in parse_args
stop = self._process_args(largs, rargs, values)
File "/usr/lib/python2.7/optparse.py", line 1443, in _process_args
self._process_short_opts(rargs, values)
File "/usr/lib/python2.7/optparse.py", line 1550, in _process_short_opts
option.process(opt, value, values, self)
UnboundLocalError: local variable 'value' referenced before assignment

Just default CDN end points to non-snet or publicURL values. The bandwidth that would be saved with requests to CDN end points is tiny anyway.

from swiftly import Client

should be

from swiftly.client import Client

Just a reminder to myself that I haven't updated the documentation with respect to all the conf file changes.

Specifically x-delete-at/x-delete-after but probably just all of them.

After researching this, it seems the high CPU usage is only when using Eventlet. I did a bit of digging into Eventlet to try to figure out why, but I got lost pretty quick with what's doing what. Appears to be at the socket level with all the trampolining while calling send.

Easiest solution for a tool like the swiftly cli is to not bother with Eventlet by default, but I'll make it into an option so users can decide. The swiftly library itself certainly has many use cases where Eventlet would be preferred, and I'll keep it using Eventlet by default, but allow not using it even if it is installed.

Maybe:
swiftly delete container/object --segments

Not sure on this, just an idea. A 'swiftly sync' would be cool. One that uploads a local directory structure's information, but also uploads deletion markers and such. Then another directory structure could be kept in sync. Just the basic idea right now.

fhines@OMGHOSTS1:/var/log/swift$ swiftly auth
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/swiftly/cli/cli.py", line 331, in call
self.commandscommand
File "/usr/local/lib/python2.6/dist-packages/swiftly/cli/auth.py", line 135, in call
return cli_auth(context)
File "/usr/local/lib/python2.6/dist-packages/swiftly/cli/auth.py", line 78, in cli_auth
fp.flush()
File "/usr/lib/python2.6/contextlib.py", line 34, in exit
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.6/contextlib.py", line 113, in nested
yield vars
File "/usr/local/lib/python2.6/dist-packages/swiftly/cli/auth.py", line 75, in cli_auth
fmt = '%%-%ds %%s\n' % (max(len(t) for t, v in info) + 1)
ValueError: max() arg is an empty sequence

When you get a chance can you bump the version on pypi, 1.8 is the current version and is missing: 3441061

We currently have:
swiftly delete container --recursive
So I guess something like:
swiftly delete container --recursive-prefix myprefix/