We get an error when trying to call save() on test_storage_simple_save.

The Python 3 changes in the develop branch seem to work fine on python 2.7 and 3.4, can a new release be added to PyPi?

Really enjoying my use of django-queued-storage but have come across a few little hiccups over time which could be addressed in the same way.

When a Task is run to push the files from one File backend to another, it'd be really useful [for me, at least :)] to write back to the parent Model with the backend that's in use. This provides two benefits:

1. Less reliance on memcache - if your memcache instance(s) go down, your first application requests aren't trying to figure out if files are stored locally or remotely. memcache can still be used for speed reasons, however at the moment memcache is storing data that should probably be less transient.
2. If you are using template output caching based on the model's "last change date", that caching is unable to be aware of the file location. In my example this means many many more database requests than would otherwise be necessary.

A possibility::

class MyModel(models.Model):
    image = models.ImageField('Photo', upload_to=item_image_path, blank=True, null=True, storage=queued_s3storage)
    image_storage_backend = models.CharField(editable=False)

When the Task changes the storage for the ImageField, the _storage_backend is updated automatically with the name of the new storage. This is a bit of a hack, so an alternative would be to publish a Signal when the task is complete, which would allow us to write our own code to update the model in any way we please. This addresses the second item above for me.

What do you think? I am happy to put together a patch but would like a bit of feedback first so that whatever I build is useful for others, too...

right now it can be a bit ambiguous what happens when delete is called on a storage, as it might leave files dangling. we should be able to configure what to do in this case

from queued_storage.backends import QueuedS3BotoStorage
from queued_storage.fields import QueuedFileField


class Video(models.Model):
    video = QueuedFileField(storage=QueuedS3BotoStorage(delayed=True))
    thumbnail = models.FileField()
    date_taken = models.DateTimeField(auto_now_add=True)

I got this in the doc for the Storage class

generate_filename(filename)[source]¶
New in Django 1.10.
Validates the filename by calling get_valid_name() and returns a filename to be passed to the save() method.

I am using this package , and i found that most of the methods which are written in year around 2017 are still not updated in the package present at pypi. The last update of package is done on Dec 15, 2015. @jezdez @ashwoods please look into it.

Any chance of supporting Python 3 now that Celery supports it? I'm catching errors while trying to install the package:

TypeError: can't use a string pattern on a bytes-like object

please, correct me if I'm wrong, but there are no option to specify that for the local file access, file should be accessed locally?

IMHO it will be a great performance improvement (and sometimes can save money) when files that are uploaded to the remote storage wil be accessed locally for the local operations (e.g. generating thumbnails etc.)

please let me know if such an option exists or even if it is planned!

Thanks for this extension!
Janusz

The test_storage_methods doesn't pass. This is one of the issues that I am considering a "magic" refactor, trading a bit of explicitness for resilience to change, but not entirely sure where the potential "leaks" with this abstraction are so I'd like to discuss this before implementing it.

@jezdez, love this app, but I'm having trouble with it on a production server. After uploading a large video (>500MB), it doesn't look like the celery task is ever run.

I confirmed that the transfer() method in the storage backend is called. I have not be able to confirm that the actual task methods are being called. The Celery log doesn't show any problems or any indication that the task has been queued.

This is only happening on the production server. It works locally using the Django dev server, and it also works for smaller uploads, i.e. 4MB, on both my local pc and production. I'm stumped, and was wondering if you had any ideas about what could be causing the problem. Appreciate any help you could provide.

I just followed the example on this page, http://django-queued-storage.readthedocs.org/en/latest/fields.html
<===============================================
from queued_storage.backends import QueuedS3BotoStorage
from queued_storage.fields import QueuedFileField

class MyModel(models.Model):
image = QueuedFileField(storage=QueuedS3BotoStorage(delayed=True))

my_obj = MyModel(image=File(open('image.png')))

my_obj.save()

my_obj.image.transfer()
===================================================================>

At first, everything gose well, but when I tried the local save
<==============================================
my_obj.save()
===============================================>

It threw out a exception:
<================================================
S3ResponseError: S3ResponseError: 403 Forbidden
=================================================>

After some research, I realized that the boto library just try to use the name of image field (my_obj.image.name, in this case) as a key-name to lookup the corresponding key in s3 server. Because the local file have not been uploaded to s3, off course, you can not find this key in s3. Then s3 return a 403 forbidden response.

This is a very strange behavior, and I am not sure which library caused this behavior.
Any suggestion?

$ python manager makemigrations

...
  File "/opt/project/lib/python3.5/site-packages/django/db/migrations/serializer.py", line 332, in serializer_factory
    "topics/migrations/#migration-serializing" % (value, get_docs_version())
ValueError: Cannot serialize: <queued_storage.backends.QueuedStorage object at 0x7f06cf150e10>
There are some values Django cannot serialize into migration files.
For more, see https://docs.djangoproject.com/en/2.2/topics/migrations/#migration-serializing

$python --version
Python 3.7.2

When the transfer task fails, i get the following error:

[2012-03-07 11:03:53,846: ERROR/MainProcess] Task queued_storage.tasks.Transfer[7f928629-5494-4bbf-858b-5c5180a95ffb] raised exception: TypeError("delay_task() got multiple values for keyword argument 'task_name'",)
Traceback (most recent call last):
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/celery/execute/trace.py", line 47, in trace
    return cls(states.SUCCESS, retval=fun(*args, **kwargs))
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/celery/app/task/__init__.py", line 247, in __call__
    return self.run(*args, **kwargs)
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/queued_storage/tasks.py", line 86, in run
    local_options, remote_options, **kwargs)
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/celery/app/task/__init__.py", line 542, in retry
    self.apply_async(args=args, kwargs=kwargs, **options)
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/celery/app/task/__init__.py", line 456, in apply_async
    **options)
TypeError: delay_task() got multiple values for keyword argument 'task_name'

I'm using celery 2.4.6 and django-queued-storages 0.4. I also tried celery 2.5.1 and i got:

Traceback (most recent call last):
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/celery/execute/trace.py", line 153, in trace_task
    R = retval = task(*args, **kwargs)
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/queued_storage/tasks.py", line 86, in run
    local_options, remote_options, **kwargs)
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/celery/app/task/__init__.py", line 566, in retry
    self.apply_async(args=args, kwargs=kwargs, **options)
  File "/Users/vad/Source/Envs/moka/lib/python2.7/site-packages/celery/app/task/__init__.py", line 477, in apply_async
    **options)
TypeError: delay_task() got multiple values for keyword argument 'task_name'

I'd like to use the DQS package in my app. However, it seems like that the task to transfer a file from local to remote is not called. I am using Django 1.8.4.

Is there any way to test this ?

I have also created a question on SO: http://stackoverflow.com/questions/38918751/django-queuedstorage-task-not-being-executed-when-file-is-uploaded

i have try queued-storage with celery 3.0.4, and i have this message in celery.log (issue subject)
is it due to incompatible celery version? (celery <3.0 is required)
if yes, what i need to do, do be comaptible with celery >3.0? could you advice me?

I have an issue with invalidating my cache after I transfer and delete the local image.

What would be the best way to invalidate the cache after a successful transfer?

I don't see any information about the object on the actually transfer task

django-celery is not compatible with the latest Celery version, which already comes with Django integration. Is it possible to remove this requirement, or make a conditional requirement such as django-celery>=3.1 or celery>=4.0?

Is there something like this for flask?

Using Django 1.4, queued_storage 0.5

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/core/management/commands/validate.py", line 9, in handle_noargs
    self.validate(display_num_errors=True)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/core/management/base.py", line 266, in validate
    num_errors = get_validation_errors(s, app)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/core/management/validation.py", line 30, in get_validation_errors
    for (app_name, error) in get_app_errors().items():
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/db/models/loading.py", line 158, in get_app_errors
    self._populate()
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/db/models/loading.py", line 64, in _populate
    self.load_app(app_name, True)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/db/models/loading.py", line 88, in load_app
    models = import_module('.models', app_name)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/issackelly/Projects/python/replayswithfriends/replayswithfriends/sc2match/models.py", line 16, in <module>
    'storages.backends.s3boto.S3BotoStorage'
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/queued_storage/backends.py", line 84, in __init__
    handler=import_attribute)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/queued_storage/backends.py", line 100, in _load_backend
    return handler(backend, options)
  File "/Users/issackelly/.virtualenvs/replayswithfriends/lib/python2.7/site-packages/queued_storage/utils.py", line 18, in import_attribute
    (module, e))
django.core.exceptions.ImproperlyConfigured: Error importing module queued_storage.tasks: "Import by filename is not supported."

TypeError Traceback (most recent call last)

/home/kuno/utopia/sogoke-django/lib/python2.7/site-packages/queued_storage/backends.pyc in transfer(self, name, cache_key)
182 print(type(self.local))
183 print(type(self.remote))
--> 184 return self.task.delay(name, self.local, self.remote, cache_key)
185
186 def get_valid_name(self, name):

It seems that queued_storage passed the local and remote lazy backnend to celery.
As far as I understand that the most suitable parameters for celery task is simple data structure such as string or number.

So, this caused this error?

The celery docs advise against using tasks_always_eager (and is currently failing anyways in my current env)

http://docs.celeryproject.org/en/latest/userguide/testing.html

it recommends using mock to test celery tasks in unit tests.