This for Django 1.7:

# Django >= 1.7
./manage.py migrate
./manage.py createsuperuser

is missing ./manage.py makemigrations todo (see: https://docs.djangoproject.com/en/1.7/intro/tutorial01/#activating-models).

Also there is no mention in the tutorial about getting server.py. It just says to run the script.
See either this or #17.

Tutorial should also mention that you need to launch redis-server.

Hi

I'm interested in developing an Ember addon for swamp dragon but I'm struggling on where to start with the client code. I'm looking at the repo 'swampdragon-bower' and also this repo in the folder 'swampdragon/swampdragon/static/swampdragon/js'..

What code should I start with to develop something that relies on the swampdragon frontend code?

Additionally - I think calling connect() and accessing window variables in the connection.js file is going to cause problems for me and I'm wondering what your openness is to changing that?
It doesn't seem like there is any way to 'initialize' swampdragon on the client side - it appears to all rely on global variables and functions which get executed as soon as the script is run on the client - it doesn't look like there is any way to stop this?

As we added the SelfPublishModel to more of our existing models we started noticing a massive increase in the number of queries to our database server. It appears that when SelfPublishModel saves the state of a ForeignKey field it triggers the query to retrieve the related model from the db. It seems like it should just need to save the primary key.

For now we've had to back off on how many models we add SelfPublishModel to until we can figure out how to get these excess queries under control.

Thanks!

I was looking at the Network Frames sent and received using the Chrome Developer Tools. I noticed frames like this:
a["{"action": "deleted", "data": {"id": 374}}"]

They seemed odd because nothing was deleted, and even if something was deleted that doesn't look like enough info to know what route/channel something was deleted from.

The deleted messages are sent if I have a subscribe with a filter ie:
swampdragon.subscribe("user_group_members", "1", {group_id: 1}, null, null);

Then change a model that is not included in that filter, ie group_id=2.

It looks like the last few lines of pubsub_providers\model_publisher.py is publishing the deleted action in this case.

Swampdragon is a little too quiet and error 3001 connection aborted isn't very helpful for debugging.

I can't find in the docs about this:

where is that file suppose to be and what should contain in it?

I'm experimenting with using swamp dragon with django rest framework(drf). I want all writes to go through our drf api. The main reason is to pass validation and permissions already setup with drf. Now I want to use swampdragon to notify clients when models have been updated. I got that working using the SelfPublishModel mixin, and the ModelRouter. But it looks like by default that also allows the model to be created, updated and deleted. What is the best way to disable all writes? For now I'm thinking of subclassing ModelRouter and override create, update, and delete functions to do nothing. This is a bit scary though, it would be nice to have an official way to say this route is read only and be certain there is no way the model can be changed via websocket/swampdragon.

Thanks!

Hi @jonashagstedt !

I noticed in the documentation of uWSGI that it supports the tornado loop engine (https://uwsgi-docs.readthedocs.org/en/latest/Tornado.html). Also, I see in the docs that you recommend uWSGI - http://swampdragon.net/blog/deploying-swampdragon/ for the main django application.

The question is if you have any recommendations/caveats of running SD through uWSGI.

SwampDragon needs a logo.

If anyone fancy doing one please post it here (or a link to it)

I am not very familiar with AngularJS. Whats the proper way to include SwampDragonServices?

var ChatControllers = angular.module('chatApp', []).config(function($interpolateProvider){
    $interpolateProvider.startSymbol('{[').endSymbol(']}');
}).service('$dragon', function () { /* ... */ });


ChatControllers.controller('chatCtrl', ['$scope', '$dragon', function($scope, $dragon) {
    $scope.channel = 'chat';
    $scope.messages = [];

    /// Subscribe to the chat router
    $dragon.onReady(function() {
        $dragon.subscribe('chat-route', $scope.channel).then(function(response) {
        });
    });

    $dragon.onChannelMessage(function(channels, message) {
        if (indexOf.call(channels, $scope.channel) > -1) {
            $scope.$apply(function() {
                $scope.messages.unshift(message);
            });
        }
    });

    $scope.sendMessage = function() {
        $scope.errors = null;
        $dragon.create('chat-route', {message: this.message, name: this.name})
            .then(function(response) {
                $scope.message = '';
            })
            ["catch"](function(response) {
                $scope.errors = response.errors;
            });
    }
}]);

<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js"></script>
<script type="text/javascript" src="{% static 'swampdragon/js/angular/services.js' %}"></script>
<script type="text/javascript" src="{% static 'swampdragon/js/dist/swampdragon.min.js' %}"></script>
<script type="text/javascript" src="{% static 'chat/js/chat-controllers.js' %}"></script>

Getting TypeError: $dragon.onReady is not a function

Hi, Jonas. Thank you so much for doing this. It has been really useful and simple to use!

I was wondering if you can explain me how can I use this to send notifications via signals. For example:

I have a todo list. I want to asign a taks to an user and then I want the system notify that user "User.name has assigned you Task.name"

It's there a way to do this real-time?

Thanks you again and forgive my English! (I'm from Argentina)

I'm trying to create a Message model using Javascript, and I've been left scratching my head.

When I use create on my router and pass in a related user object, I receive an error "Duplicate entry 'test' for key 'username'"; The serializer is trying to create a new user model rather than hook into the old one.

Have I missed something obvious, or is there no way to create a model re-using an existing foreign key?

Create code

var data = {
  user: { id: 1, username: 'test', password: 'test' },
  body: 'testing'
};
swampdragon.create('message', data, function (context, data) {
  console.log('MessageStore: onCreate success.', context, data);
}, function (context, data) {
  console.log('MessageStore: onCreate failed.', context, data);
});

Model

class Message(SelfPublishModel, models.Model):
    serializer_class = MessageSerializer

    user = models.ForeignKey(User, related_name='messages')
    body = models.TextField(blank=True)

Serializers

class UserSerializer(ModelSerializer):

    class Meta:
        model = User
        publish_fields = ('id', 'username')
        update_fields = ('username', 'password',) # Had to add these to prevent validation fail on the model, but don't really want these to be updateable

class MessageSerializer(ModelSerializer):
    user = UserSerializer

    class Meta:
        model = 'chat.Message'
        publish_fields = ('body', 'user', )
        update_fields = ('body', 'user', )

Currently using sys.argv to check if 'test' is present.

This is a bad solution

I have been working on something using a twitter stream based on server monitor tutorial on the site. I am calling publish_data on every iteration of a for loop. Should I be making use of the metgod in a different way. The code is something like this:
def get_tweets():
iterator = twitter_stream.statuses.filter(track='bae')
for tweet in iterator:
publish_data('tweet',{
'text': tweet,
})

With get_tweets being called in the route

If I define a custom field serializer, the field is always included in the output from serialize regardless of publish_fields:

class UserSerializer(ModelSerializer):
    def serialize_avatar(self, obj):
        return obj.get_avatar()

    class Meta:
        model = User
        publish_fields = ('username', )

serializer = UserSerializer(instance=testUser)
serializer.serialize()
# {"username": "test", "avatar": "/images/test.png"}

Allow for channels to be prefixed by a setting, to differentiate between environments, as this could be a potential issue if a Redis server is shared between different websites.

Related Models

Hello,

I'm trying to figure out, how I could update the Foreign Model when I'm changing data at the base-model.

Currently I have two Models:

• Media
• Thumbnail

Models

The Thumbnail has a ForeignKey field to the Media-Model. The code looks like this:

class Media(SelfPublishModel, models.Model):
    serializer_class = MediaSerializer

    file = models.FileField(...)


class Thumbnail(SelfPublishModel, models.Model):
    serializer_class = ThumbnailSerializer

    media = models.ForeignKey(MediaFile)
    image = models.ImageField()

Controller

Now, at the Webinterface, I would like to subscribe to all Changes according to the Media-Models (using Angular) like so:

var MediaControllers = angular.module('MediaControllers', []);

MediaControllers.controller('MediaCtrl',
    ['$scope', '$dragon', function ($scope, $dragon) {

    $scope.mediaItems = [];
    $scope.router = 'media-router';
    $scope.channel = 'media';

    $dragon.onReady(function() {

        // Subscribe media-file
        $dragon.subscribe($scope.router, $scope.channel, {})
            .then(function(response) {
                $scope.dataMapper = new DataMapper(response.data);
            }
        );

        // Request all media_files when script is loaded
        $dragon.getList($scope.router, {}).then(function(response) {
            $scope.mediaItems = response.data;
        });

    });

    $dragon.onChannelMessage(function(channels, message) {
        // Channel-Message for media (channel)
        if (indexOf.call(channels, $scope.channel) > -1) {
            $scope.$apply(function() {
                $scope.dataMapper.mapData($scope.mediaItems, message);
            });
        }
    });
}]);

Serializers

The Serializers look like:

class MediaSerializer(ModelSerializer):
    thumbnail_set = ThumbnailSerializer

    class Meta:
        model = 'media.Media'
        publish_fields = [
            'file',
            'thumbnail_set',
        ]

class ThumbnailSerializer(ModelSerializer):
    class Meta:
        model = 'thumbnails.Thumbnail'
        publish_fields = [
            'image',
        ]

Routers

And finally the Routers:

class MediaRouter(ModelRouter):
    route_name = 'media-router'
    serializer_class = MediaSerializer
    model = MediaFile

    def get_object(self, **kwargs):
        return self.model.objects.get(pk=kwargs['id'])

    def get_query_set(self, **kwargs):
        return self.model.objects.all()


class ThumbnailRouter(ModelRouter):
    route_name = 'thumbnail-router'
    serializer_class = ThumbnailSerializer
    model = Thumbnail

    def get_object(self, **kwargs):
        return self.model.objects.get(pk=kwargs['id'])

    def get_query_set(self, **kwargs):
        return self.model.objects.all()

Celery Task

Because it's not complex enough, I'm creating the Thumbnails in a celery-Task in order to avoid blocking the Media creation with the Thumbnail creation process.

The Task looks like:

@app.task
def make_thumbnail(thumbnail_model):
    media_file = thumbnail_model.media.file

    if path.isfile(media_file.path):
        name, _ = path.splitext(path.basename(media_file.path))

        if not path.exists(thumbnail_model.filesystem_path):
            os.makedirs(thumbnail_model.filesystem_path)

        thumbnail_name = "{name}_{size}x{size}_thumbnail.{extension}".format(
            name=name,
            size=thumbnail_model.size,
            extension='jpeg',
        )

        tn_url = "/".join((thumbnail_model.url_path, thumbnail_name))
        tn_file = os.path.join(thumbnail_model.filesystem_path, thumbnail_name)

        # Generate and save Thumbnail image
        img = Image.open(mediafile_file.path)
        img.thumbnail((thumbnail_model.size, thumbnail_model.size))
        img.save(tn_file, 'JPEG')
        img.close()

        # Save thumbnail model
        thumbnail_model.image = tn_file
        thumbnail_model.save()

The Problem

Now - finally - the problem. If I'm creating and changing Media-Models, all works like charm. But if I'm creating new Thumbnail-Models, the Webinterface does not receive any update messages.

I've forced this by calling in the celery-task. Now the Webinterface got an Update, every time a thumbnail was created.

thumbnail_model.media_file.save()

Gotten Javascript Object

But this time, the Media-Object in the Webinterface has only the ID's of the thumbnail in the 'thumbnail_set' - Array. The Media-Object at the frontend now looks like this:

{
    "_type":"media",
    "id":2349,
    "thumbnail_set":[5425,5424],
    "file": (...),
}

Excpeted Javascript Object

If I'm reloading the browser, the right Media-Object appears. I guess this happens because of the getList() function in the onReady() function.

The Media-Object now looks like this:

{
    "_type":"media",
    "id":2349,
    "thumbnail_set":[
        {"id":5424,"image":"thumbnail_name","_type":"thumbnail"},
        {"id":5425,"image":"thumbnail_name","_type":"thumbnail"},
    ],
    "file": (...),
}

Thanks!

I hope that I could describe my problem clearly enough and would be very happy about your feedback.

Regards,
winkelchri

PS:
Thank you for the great project!

>>manage makemigrations

File "/virtualenvs/project/lib/python3.4/site-packages/swampdragon/serializers/serializer_tools.py", line 5, in
from django.db.models.related import RelatedObject
ImportError: No module named 'django.db.models.related'

>>python --version
Python 3.4.2

>>pip freeze
certifi==14.05.14
django-debug-toolbar==1.3.0
django-redis-sessions==0.4.0
Django==1.8
pip-tools==0.3.5
pycrypto==2.6.1
python-dateutil==2.4.1
redis==2.10.3
six==1.9.0
sockjs-tornado==1.0.1
sqlparse==0.1.14
SwampDragon==0.4.1.2
tornado-redis==2.4.18
tornado==4.1

Is there anyway to use a similar approach to https://github.com/jrief/django-websocket-redis which:

• Runs a seperate Django main loop in a cooperative concurrency model using gevent, thus only one thread/process is required to control all open websockets simultaneously.
• No dependency to any other asynchronous event driven framework, such as Tornado, Twisted or Node.js.

That way the user can just start their server and not worry about also running the Tornado server.py.

Is there a way to get request in ModelRouter?

I've been bitten by this:

I thought that using different values in SWAMP_DRAGON_REDIS_DB would isolate my dev django context from my test context, but it is not the case.

cf http://redis.io/topics/pubsub, section Database & Scoping

It happened for me between test and local environments, which is not a big problem, but it can happen between multiple websites if they use the same redis instance.

Side note:

I am in an unexpected case where I am using the redis publisher in tests, instead of the mock publisher : at https://github.com/jonashagstedt/swampdragon/blob/9adb5579c507b76e34b6be2e617c42e7b6bcaa19/swampdragon/pubsub_providers/publisher_factory.py#L10, the code looks for test in sys.argv but I use python manage.py jenkins to run tests.

Additionally, this is a rather strange way to switch backends, it would seem more natural and explicit to use something similar to what is done for emails : https://docs.djangoproject.com/en/1.7/topics/email/#dummy-backend

functions passed to the sampdragon.ready function end up being bound the the connection's 'open' signal. If the connection is lost and then restablished, all those functions are still bound to the 'open' signal and are called again. For subscribe methods, it makes sense to call the subscribe again to reestablish the link.

However all callRouter calls are called again (even non-subscribe) and they shouldn't be because they were probably a call once sort of method.

Would it make sense to have two different flavors of a ready function (one that should be called again if the connection is lost and regained) and another that deregisters the callback once it's been called?

Given this description:

Model serializers
Because SwampDragon will be communicating with the frontend via JavaScript, and JavaScript don't understand Django models, we need to tell SwampDragon how to serialize these models into JSON (short for JavaScript Object Notation).

why not use Django REST Framework (DRF)'s serializers rather than re-inventing the wheel? Code here and description here and here. Perhaps use the renderers? They seem to solve the exact goal that you outlined.

I know you mentioned that you did not use them here: https://news.ycombinator.com/item?id=8676967:

About the serializers: You do need separate serializers from DRF (they are not the same package after all).

but you don't address the reason why.

This looks very interesting. Most django projects have serializers setup in some form or other. Is it possible to use other serializers like django-rest-framework with this?

..or is it possible to just send push events to the browser without integrating the models and other things?

SWAMP_DRAGON_HOST and SWAMP_DRAGON_PORT are missing in the settings doc section.

Btw: I think it would be better to have a repo for the docs as asked for in #16. I think that allowing to make pull requests and all this stuff would be easier than opening an issue for every typo/thing we find =).

In a single page app, I need to clear the swampdragon connection:

• I can unsubscribe from all the channels
• but I can't empty what had been added to callbackQueue
• if I resubscribe to a channel, I don't want the old functions previously added in onChannelMessage to be called again.

Any ideas ?

Hello,
I would like to help to update the docs. Some entrys have no syntax highlighting and in some examples, there missing imports which may help to better understand the relationships between the modules.

I was starting to fork the project and edit the docs section as I realized, that the docs seems to be moved to your own server. Is the documentation on your server still based on the github data?

Regards
winkelchri

Shouldn't mapData change the value of datesource and return the new value?

mapData calls mapCreated/mapUpdate/mapDelete and they can possibly assign another object to the datesource reference and return the value such as in [1] [2].

Couldn't that become a problem?

swampdragon uses the PUBSUB command which was introduced in Redis 2.8

I am new to SD.How to get the amount of new messages according to different users.any example?

Hi,
Is it possible to call send multiple times from one router call?
I want to from the client send a query and as the results come back from the query push them to the client from the server (1 to 1 message).
Also I am struggling to get the chat_example running (CSRF errors) - anything I need to know? Django 1.7

Thanks for a great library.

Hi! I very appreciate the work you've done, especially your explanations, related to different issues that people submits. A lot of useful information can be found there.

It seems to me that I've found the out-of-date docs. Please check it:
https://github.com/jonashagstedt/swampdragon/blame/master/docs/routers/base-model-publisher-router.md#L18
So, there is no 'ModelPublisherRouter' inside the package as of now. Maybe you renamed those class to 'ModelPubRouter' (https://github.com/jonashagstedt/swampdragon/blob/master/swampdragon/route_handler.py#L311)

Provide license for SwampDragon.

There's a few typos, etc in the documentation. I was going to send you a PR to fix them but it looks like the doco has moved out of this repo.

The problem is clients can receive the updated action message for a model before it is actually committed to the database. It goes something like this:

client A does a put to our django rest api to update a model
client B receives notification of the change, and retrieves the model with a regular http get. But it receives the model without any changes that client A just wrote. If we put a second or two delay in the client it does receive the correct changes.

We're re-requesting the data from rest instead of just using the data from the swamp dragon message because we already have complex serializers set up with django rest framework and don't want to recreate them with swamp dragon serializers just yet. So for now we're just using swamp dragon to trigger the client to refresh from http rest.

We're using Mysql innodb tables with transaction support turned on with "ATOMIC_REQUESTS": True set in settings.DATABASES

Another reason for delaying until after the transaction commits is to avoid sending out incorrect data if the transaction fails.

On this page DataMapper isn't injected in controller and uses globals which isn't good
http://swampdragon.net/tutorial/swampdragon-tutorial-part-2/

TodoControllers.controller('TodoListCtrl', ['$scope', '$dragon', function ($scope, $dragon) {
    $scope.todoList = {};
    $scope.todoItems = [];
    $scope.channel = 'todos';

    $dragon.onReady(function() {
        $dragon.subscribe('todo-item', $scope.channel, {todo_list__id: 1}).then(function(response) {
            $scope.dataMapper = new DataMapper(response.data);
        });

It's not angular way. Maybe write DataMapperService to inject?

Is there any manual on how to connect to swampdragon websocket so I can receive notifications from a python client?

Thanks!

EDIT: I'm trying to connect to the server with the websocket python client to 127.0.0.1:9999/data but it returns a 200, I've also tried /data/chat, /data/chat-route, /chat and more combinations but all return a 404. Where do I have to connect? (I'm running the chat example)

Uncaught ReferenceError: channel is not defined swampdragon.js:157

line responsible for this error:
'''
eventHandler.emit('heartbeat', [channel, e.data]);
'''
and there are no 'channel' variable, even more there are no 'channel' property in event's data

I read in the docs that something like this would bring the relation:

class UserSerializer(ModelSerializer):
    class Meta:
        model = 'app.User'


class ChatSerializer(ModelSerializer):
    username = 'chat.UserSerializer'

    class Meta:
        model = 'chat.Chat'
        publish_fields = ('message', 'user', 'username', 'date', )

But username is null. Any clue?

After some time the tornado server runs into the following error for each WebSocket connection. Any idea why this might happen while the MySQL server is still there? I guess it might be a problem related to connection timeouts:

ERROR:tornado.application:Uncaught exception in /data/782/e73lchdl/websocket
Traceback (most recent call last):
  File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/tornado/websocket.py", line 369, in _run_callback
    callback(*args, **kwargs)
  File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/sockjs/tornado/transports/websocket.py", line 70, in on_message
    self.abort_connection()
  File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/sockjs/tornado/websocket.py", line 28, in abort_connection
    self.ws_connection._abort()
AttributeError: 'NoneType' object has no attribute '_abort'
ERROR:tornado.general:WebSocket
Traceback (most recent call last):
  File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/sockjs/tornado/transports/websocket.py", line 60, in on_message
    self.session.on_messages(msg)
  File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/sockjs/tornado/session.py", line 423, in on_messages
    self.conn.on_message(msg)
  File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/swampdragon/connections/sockjs_connection.py", line 39, in on_message
    raise e
OperationalError: (2006, 'MySQL server has gone away')

Current file upload solution needs improving.
Allow drag and drop files as well as progress hook

Hi,

when updating to the newest version of Swampdragon I have the problem that the Angular service is not working anymore.

I load most of the javascript files using require.js. The problem occours when the Angular service is loaded:

ReferenceError: swampdragon is not defined
    at Object.<anonymous> (services.js:23)

The problem is that https://github.com/jonashagstedt/swampdragon/blob/master/swampdragon/static/swampdragon/js/angular/services.js tries to access the global variable swampdragon which is not defined.
I think this has something to do with the recently introduced AMD support. When swampdragon.js is loaded via require.js it doesn't expose a global variable and the Angular service runs into this error.

I am following the the tutorial. I was able to add a TodoList just fine, but when trying to add a TodoItem, I get this error:

TodoItem has no todo_list

Hi, sorry if it's a dumb question but I can't get the settings.js file.

I didn't found anything in the documentation that mention where we can get this file. The url "http://my-domain.com:9999/settings.js" returns nothing.

I'm running my Django Project on an Nginx Server with uWSGI.

Thanks for the answer.

Hi,

I need to serve my application over HTTPS and I had some trouble getting the correct protocol into the window.swampdragon_host variable which is returned by the swampdragon javascript settings file.

I am using nginx as the webserver and configured it as described in the official blog post about deployment. Additionally I configured it to use SSL:

upstream swampdragon {
    server 127.0.0.1:9999;
}

server {
    listen 8080 ssl;
    ssl_certificate     ...;
    ssl_certificate_key ...;

    server_name my.site.com;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_set_header Host $http_host;
        proxy_pass http://swampdragon;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

With this configuration the settings file is correctly served over HTTPS but the window.swampdragon_host variable still delivers an HTTP url.

My question is if this can be fixed by just adding some nginx configuration because the only solution I came up with was to set the X-Forwarded-Proto header to HTTPS and modify the settings_provider.py file in swampdragon to get the protocol from this header.

I'm trying to set up the angularjs tutorial, using react instead. With react, they provide some specific methods for updating the state of data stores, which then triggers the view re-rendering. It seems that the DataMapper function is a bit misaligned with how you would use it with reactjs, since it is directly mutating the array of todo items. I think it might be ok if I create a clone of the array of items, before passing it in, then use that in react's setState method.

My primary issue at the moment is that I cannot get the subscription to return back a meaningful response. I always get an empty array back. I did take the todo example and basically replace that with servers, as I was going to make an example that manages server, computers and users. As of right now, it is simply the same as the todo example, with the words changed.

Here is a screenshot of where I'm at. You can see in console that I've logged out the response from the highlighted line of javascript.

Other things I had to change was:

• Moved all javascript loading into a require configuration
• Moved the dragon component (created by angularjs factory) into a regular requirejs component
• Added in some modules that are required

I've uploaded the django project at my swampdragon_react repository, which should directly work after initializing the admin user/data.