hagsteel / swampdragon Goto Github PK
View Code? Open in Web Editor NEWswampdragon
License: Other
swampdragon
License: Other
I'm experimenting with using swamp dragon with django rest framework(drf). I want all writes to go through our drf api. The main reason is to pass validation and permissions already setup with drf. Now I want to use swampdragon to notify clients when models have been updated. I got that working using the SelfPublishModel mixin, and the ModelRouter. But it looks like by default that also allows the model to be created, updated and deleted. What is the best way to disable all writes? For now I'm thinking of subclassing ModelRouter and override create, update, and delete functions to do nothing. This is a bit scary though, it would be nice to have an official way to say this route is read only and be certain there is no way the model can be changed via websocket/swampdragon.
Thanks!
On this page DataMapper isn't injected in controller and uses globals which isn't good
http://swampdragon.net/tutorial/swampdragon-tutorial-part-2/
TodoControllers.controller('TodoListCtrl', ['$scope', '$dragon', function ($scope, $dragon) {
$scope.todoList = {};
$scope.todoItems = [];
$scope.channel = 'todos';
$dragon.onReady(function() {
$dragon.subscribe('todo-item', $scope.channel, {todo_list__id: 1}).then(function(response) {
$scope.dataMapper = new DataMapper(response.data);
});
It's not angular way. Maybe write DataMapperService
to inject?
SwampDragon needs a logo.
If anyone fancy doing one please post it here (or a link to it)
This looks very interesting. Most django projects have serializers setup in some form or other. Is it possible to use other serializers like django-rest-framework with this?
..or is it possible to just send push events to the browser without integrating the models and other things?
Hi! I very appreciate the work you've done, especially your explanations, related to different issues that people submits. A lot of useful information can be found there.
It seems to me that I've found the out-of-date docs. Please check it:
https://github.com/jonashagstedt/swampdragon/blame/master/docs/routers/base-model-publisher-router.md#L18
So, there is no 'ModelPublisherRouter' inside the package as of now. Maybe you renamed those class to 'ModelPubRouter' (https://github.com/jonashagstedt/swampdragon/blob/master/swampdragon/route_handler.py#L311)
Hello,
I would like to help to update the docs. Some entrys have no syntax highlighting and in some examples, there missing imports which may help to better understand the relationships between the modules.
I was starting to fork the project and edit the docs section as I realized, that the docs seems to be moved to your own server. Is the documentation on your server still based on the github data?
Regards
winkelchri
Hi, sorry if it's a dumb question but I can't get the settings.js file.
I didn't found anything in the documentation that mention where we can get this file. The url "http://my-domain.com:9999/settings.js" returns nothing.
I'm running my Django Project on an Nginx Server with uWSGI.
Thanks for the answer.
Hi, Jonas. Thank you so much for doing this. It has been really useful and simple to use!
I was wondering if you can explain me how can I use this to send notifications via signals. For example:
I have a todo list. I want to asign a taks to an user and then I want the system notify that user "User.name has assigned you Task.name"
It's there a way to do this real-time?
Thanks you again and forgive my English! (I'm from Argentina)
There's a few typos, etc in the documentation. I was going to send you a PR to fix them but it looks like the doco has moved out of this repo.
Allow for channels to be prefixed by a setting, to differentiate between environments, as this could be a potential issue if a Redis server is shared between different websites.
I am new to SD.How to get the amount of new messages according to different users.any example?
swampdragon uses the PUBSUB command which was introduced in Redis 2.8
I'm trying to set up the angularjs tutorial, using react instead. With react, they provide some specific methods for updating the state of data stores, which then triggers the view re-rendering. It seems that the DataMapper function is a bit misaligned with how you would use it with reactjs, since it is directly mutating the array of todo items. I think it might be ok if I create a clone of the array of items, before passing it in, then use that in react's setState method.
My primary issue at the moment is that I cannot get the subscription to return back a meaningful response. I always get an empty array back. I did take the todo example and basically replace that with servers, as I was going to make an example that manages server, computers and users. As of right now, it is simply the same as the todo example, with the words changed.
Here is a screenshot of where I'm at. You can see in console that I've logged out the response from the highlighted line of javascript.
Other things I had to change was:
I've uploaded the django project at my swampdragon_react repository, which should directly work after initializing the admin user/data.
Hi
I'm interested in developing an Ember addon for swamp dragon but I'm struggling on where to start with the client code. I'm looking at the repo 'swampdragon-bower' and also this repo in the folder 'swampdragon/swampdragon/static/swampdragon/js'..
What code should I start with to develop something that relies on the swampdragon frontend code?
Additionally - I think calling connect() and accessing window variables in the connection.js file is going to cause problems for me and I'm wondering what your openness is to changing that?
It doesn't seem like there is any way to 'initialize' swampdragon on the client side - it appears to all rely on global variables and functions which get executed as soon as the script is run on the client - it doesn't look like there is any way to stop this?
I am following the the tutorial. I was able to add a TodoList just fine, but when trying to add a TodoItem, I get this error:
TodoItem has no todo_list
As we added the SelfPublishModel to more of our existing models we started noticing a massive increase in the number of queries to our database server. It appears that when SelfPublishModel saves the state of a ForeignKey field it triggers the query to retrieve the related model from the db. It seems like it should just need to save the primary key.
For now we've had to back off on how many models we add SelfPublishModel to until we can figure out how to get these excess queries under control.
Thanks!
functions passed to the sampdragon.ready function end up being bound the the connection's 'open' signal. If the connection is lost and then restablished, all those functions are still bound to the 'open' signal and are called again. For subscribe methods, it makes sense to call the subscribe again to reestablish the link.
However all callRouter calls are called again (even non-subscribe) and they shouldn't be because they were probably a call once sort of method.
Would it make sense to have two different flavors of a ready function (one that should be called again if the connection is lost and regained) and another that deregisters the callback once it's been called?
Provide license for SwampDragon.
Currently using sys.argv
to check if 'test' is present.
This is a bad solution
Hi @jonashagstedt !
I noticed in the documentation of uWSGI that it supports the tornado loop engine (https://uwsgi-docs.readthedocs.org/en/latest/Tornado.html). Also, I see in the docs that you recommend uWSGI - http://swampdragon.net/blog/deploying-swampdragon/ for the main django application.
The question is if you have any recommendations/caveats of running SD through uWSGI.
I can't find in the docs about this:
where is that file suppose to be and what should contain in it?
I was looking at the Network Frames sent and received using the Chrome Developer Tools. I noticed frames like this:
a["{"action": "deleted", "data": {"id": 374}}"]
They seemed odd because nothing was deleted, and even if something was deleted that doesn't look like enough info to know what route/channel something was deleted from.
The deleted messages are sent if I have a subscribe with a filter ie:
swampdragon.subscribe("user_group_members", "1", {group_id: 1}, null, null);
Then change a model that is not included in that filter, ie group_id=2.
It looks like the last few lines of pubsub_providers\model_publisher.py is publishing the deleted action in this case.
The problem is clients can receive the updated action message for a model before it is actually committed to the database. It goes something like this:
client A does a put to our django rest api to update a model
client B receives notification of the change, and retrieves the model with a regular http get. But it receives the model without any changes that client A just wrote. If we put a second or two delay in the client it does receive the correct changes.
We're re-requesting the data from rest instead of just using the data from the swamp dragon message because we already have complex serializers set up with django rest framework and don't want to recreate them with swamp dragon serializers just yet. So for now we're just using swamp dragon to trigger the client to refresh from http rest.
We're using Mysql innodb tables with transaction support turned on with "ATOMIC_REQUESTS": True set in settings.DATABASES
Another reason for delaying until after the transaction commits is to avoid sending out incorrect data if the transaction fails.
Current file upload solution needs improving.
Allow drag and drop files as well as progress hook
I am not very familiar with AngularJS. Whats the proper way to include SwampDragonServices?
var ChatControllers = angular.module('chatApp', []).config(function($interpolateProvider){
$interpolateProvider.startSymbol('{[').endSymbol(']}');
}).service('$dragon', function () { /* ... */ });
ChatControllers.controller('chatCtrl', ['$scope', '$dragon', function($scope, $dragon) {
$scope.channel = 'chat';
$scope.messages = [];
/// Subscribe to the chat router
$dragon.onReady(function() {
$dragon.subscribe('chat-route', $scope.channel).then(function(response) {
});
});
$dragon.onChannelMessage(function(channels, message) {
if (indexOf.call(channels, $scope.channel) > -1) {
$scope.$apply(function() {
$scope.messages.unshift(message);
});
}
});
$scope.sendMessage = function() {
$scope.errors = null;
$dragon.create('chat-route', {message: this.message, name: this.name})
.then(function(response) {
$scope.message = '';
})
["catch"](function(response) {
$scope.errors = response.errors;
});
}
}]);
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js"></script>
<script type="text/javascript" src="{% static 'swampdragon/js/angular/services.js' %}"></script>
<script type="text/javascript" src="{% static 'swampdragon/js/dist/swampdragon.min.js' %}"></script>
<script type="text/javascript" src="{% static 'chat/js/chat-controllers.js' %}"></script>
Getting TypeError: $dragon.onReady is not a function
Given this description:
Model serializers
Because SwampDragon will be communicating with the frontend via JavaScript, and JavaScript don't understand Django models, we need to tell SwampDragon how to serialize these models into JSON (short for JavaScript Object Notation).
why not use Django REST Framework (DRF)'s serializers rather than re-inventing the wheel? Code here and description here and here. Perhaps use the renderers? They seem to solve the exact goal that you outlined.
I know you mentioned that you did not use them here: https://news.ycombinator.com/item?id=8676967:
About the serializers: You do need separate serializers from DRF (they are not the same package after all).
but you don't address the reason why.
This for Django 1.7:
# Django >= 1.7
./manage.py migrate
./manage.py createsuperuser
is missing ./manage.py makemigrations todo
(see: https://docs.djangoproject.com/en/1.7/intro/tutorial01/#activating-models).
Also there is no mention in the tutorial about getting server.py
. It just says to run the script.
See either this or #17.
Tutorial should also mention that you need to launch redis-server
.
I read in the docs that something like this would bring the relation:
class UserSerializer(ModelSerializer):
class Meta:
model = 'app.User'
class ChatSerializer(ModelSerializer):
username = 'chat.UserSerializer'
class Meta:
model = 'chat.Chat'
publish_fields = ('message', 'user', 'username', 'date', )
But username is null. Any clue?
SWAMP_DRAGON_HOST and SWAMP_DRAGON_PORT are missing in the settings doc section.
Btw: I think it would be better to have a repo for the docs as asked for in #16. I think that allowing to make pull requests and all this stuff would be easier than opening an issue for every typo/thing we find =).
Is there a way to get request
in ModelRouter?
I've been bitten by this:
I thought that using different values in SWAMP_DRAGON_REDIS_DB
would isolate my dev
django context from my test
context, but it is not the case.
cf http://redis.io/topics/pubsub, section Database & Scoping
It happened for me between test and local environments, which is not a big problem, but it can happen between multiple websites if they use the same redis instance.
Side note:
I am in an unexpected case where I am using the redis publisher in tests, instead of the mock publisher : at https://github.com/jonashagstedt/swampdragon/blob/9adb5579c507b76e34b6be2e617c42e7b6bcaa19/swampdragon/pubsub_providers/publisher_factory.py#L10, the code looks for test
in sys.argv
but I use python manage.py jenkins
to run tests.
Additionally, this is a rather strange way to switch backends, it would seem more natural and explicit to use something similar to what is done for emails : https://docs.djangoproject.com/en/1.7/topics/email/#dummy-backend
Swampdragon is a little too quiet and error 3001 connection aborted isn't very helpful for debugging.
Hello,
I'm trying to figure out, how I could update the Foreign Model when I'm changing data at the base-model.
Currently I have two Models:
The Thumbnail has a ForeignKey field to the Media-Model. The code looks like this:
class Media(SelfPublishModel, models.Model):
serializer_class = MediaSerializer
file = models.FileField(...)
class Thumbnail(SelfPublishModel, models.Model):
serializer_class = ThumbnailSerializer
media = models.ForeignKey(MediaFile)
image = models.ImageField()
Now, at the Webinterface, I would like to subscribe to all Changes according to the Media-Models (using Angular) like so:
var MediaControllers = angular.module('MediaControllers', []);
MediaControllers.controller('MediaCtrl',
['$scope', '$dragon', function ($scope, $dragon) {
$scope.mediaItems = [];
$scope.router = 'media-router';
$scope.channel = 'media';
$dragon.onReady(function() {
// Subscribe media-file
$dragon.subscribe($scope.router, $scope.channel, {})
.then(function(response) {
$scope.dataMapper = new DataMapper(response.data);
}
);
// Request all media_files when script is loaded
$dragon.getList($scope.router, {}).then(function(response) {
$scope.mediaItems = response.data;
});
});
$dragon.onChannelMessage(function(channels, message) {
// Channel-Message for media (channel)
if (indexOf.call(channels, $scope.channel) > -1) {
$scope.$apply(function() {
$scope.dataMapper.mapData($scope.mediaItems, message);
});
}
});
}]);
The Serializers look like:
class MediaSerializer(ModelSerializer):
thumbnail_set = ThumbnailSerializer
class Meta:
model = 'media.Media'
publish_fields = [
'file',
'thumbnail_set',
]
class ThumbnailSerializer(ModelSerializer):
class Meta:
model = 'thumbnails.Thumbnail'
publish_fields = [
'image',
]
And finally the Routers:
class MediaRouter(ModelRouter):
route_name = 'media-router'
serializer_class = MediaSerializer
model = MediaFile
def get_object(self, **kwargs):
return self.model.objects.get(pk=kwargs['id'])
def get_query_set(self, **kwargs):
return self.model.objects.all()
class ThumbnailRouter(ModelRouter):
route_name = 'thumbnail-router'
serializer_class = ThumbnailSerializer
model = Thumbnail
def get_object(self, **kwargs):
return self.model.objects.get(pk=kwargs['id'])
def get_query_set(self, **kwargs):
return self.model.objects.all()
Because it's not complex enough, I'm creating the Thumbnails in a celery-Task in order to avoid blocking the Media creation with the Thumbnail creation process.
The Task looks like:
@app.task
def make_thumbnail(thumbnail_model):
media_file = thumbnail_model.media.file
if path.isfile(media_file.path):
name, _ = path.splitext(path.basename(media_file.path))
if not path.exists(thumbnail_model.filesystem_path):
os.makedirs(thumbnail_model.filesystem_path)
thumbnail_name = "{name}_{size}x{size}_thumbnail.{extension}".format(
name=name,
size=thumbnail_model.size,
extension='jpeg',
)
tn_url = "/".join((thumbnail_model.url_path, thumbnail_name))
tn_file = os.path.join(thumbnail_model.filesystem_path, thumbnail_name)
# Generate and save Thumbnail image
img = Image.open(mediafile_file.path)
img.thumbnail((thumbnail_model.size, thumbnail_model.size))
img.save(tn_file, 'JPEG')
img.close()
# Save thumbnail model
thumbnail_model.image = tn_file
thumbnail_model.save()
Now - finally - the problem. If I'm creating and changing Media-Models, all works like charm. But if I'm creating new Thumbnail-Models, the Webinterface does not receive any update messages.
I've forced this by calling in the celery-task. Now the Webinterface got an Update, every time a thumbnail was created.
thumbnail_model.media_file.save()
But this time, the Media-Object in the Webinterface has only the ID's of the thumbnail in the 'thumbnail_set' - Array. The Media-Object at the frontend now looks like this:
{
"_type":"media",
"id":2349,
"thumbnail_set":[5425,5424],
"file": (...),
}
If I'm reloading the browser, the right Media-Object appears. I guess this happens because of the getList()
function in the onReady()
function.
The Media-Object now looks like this:
{
"_type":"media",
"id":2349,
"thumbnail_set":[
{"id":5424,"image":"thumbnail_name","_type":"thumbnail"},
{"id":5425,"image":"thumbnail_name","_type":"thumbnail"},
],
"file": (...),
}
I hope that I could describe my problem clearly enough and would be very happy about your feedback.
Regards,
winkelchri
PS:
Thank you for the great project!
Hi,
Is it possible to call send multiple times from one router call?
I want to from the client send a query and as the results come back from the query push them to the client from the server (1 to 1 message).
Also I am struggling to get the chat_example running (CSRF errors) - anything I need to know? Django 1.7
Thanks for a great library.
Hi,
I need to serve my application over HTTPS and I had some trouble getting the correct protocol into the window.swampdragon_host
variable which is returned by the swampdragon javascript settings file.
I am using nginx as the webserver and configured it as described in the official blog post about deployment. Additionally I configured it to use SSL:
upstream swampdragon {
server 127.0.0.1:9999;
}
server {
listen 8080 ssl;
ssl_certificate ...;
ssl_certificate_key ...;
server_name my.site.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_set_header Host $http_host;
proxy_pass http://swampdragon;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
With this configuration the settings file is correctly served over HTTPS but the window.swampdragon_host
variable still delivers an HTTP url.
My question is if this can be fixed by just adding some nginx configuration because the only solution I came up with was to set the X-Forwarded-Proto header to HTTPS and modify the settings_provider.py file in swampdragon to get the protocol from this header.
After some time the tornado server runs into the following error for each WebSocket connection. Any idea why this might happen while the MySQL server is still there? I guess it might be a problem related to connection timeouts:
ERROR:tornado.application:Uncaught exception in /data/782/e73lchdl/websocket
Traceback (most recent call last):
File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/tornado/websocket.py", line 369, in _run_callback
callback(*args, **kwargs)
File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/sockjs/tornado/transports/websocket.py", line 70, in on_message
self.abort_connection()
File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/sockjs/tornado/websocket.py", line 28, in abort_connection
self.ws_connection._abort()
AttributeError: 'NoneType' object has no attribute '_abort'
ERROR:tornado.general:WebSocket
Traceback (most recent call last):
File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/sockjs/tornado/transports/websocket.py", line 60, in on_message
self.session.on_messages(msg)
File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/sockjs/tornado/session.py", line 423, in on_messages
self.conn.on_message(msg)
File "/var/local/pyweb/virtualenv/webgcal/local/lib/python2.7/site-packages/swampdragon/connections/sockjs_connection.py", line 39, in on_message
raise e
OperationalError: (2006, 'MySQL server has gone away')
Is there any manual on how to connect to swampdragon websocket so I can receive notifications from a python client?
Thanks!
EDIT: I'm trying to connect to the server with the websocket python client to 127.0.0.1:9999/data but it returns a 200, I've also tried /data/chat, /data/chat-route, /chat and more combinations but all return a 404. Where do I have to connect? (I'm running the chat example)
Uncaught ReferenceError: channel is not defined swampdragon.js:157
line responsible for this error:
'''
eventHandler.emit('heartbeat', [channel, e.data]);
'''
and there are no 'channel' variable, even more there are no 'channel' property in event's data
>>manage makemigrations
File "/virtualenvs/project/lib/python3.4/site-packages/swampdragon/serializers/serializer_tools.py", line 5, in
from django.db.models.related import RelatedObject
ImportError: No module named 'django.db.models.related'
>>python --version
Python 3.4.2
>>pip freeze
certifi==14.05.14
django-debug-toolbar==1.3.0
django-redis-sessions==0.4.0
Django==1.8
pip-tools==0.3.5
pycrypto==2.6.1
python-dateutil==2.4.1
redis==2.10.3
six==1.9.0
sockjs-tornado==1.0.1
sqlparse==0.1.14
SwampDragon==0.4.1.2
tornado-redis==2.4.18
tornado==4.1
In a single page app, I need to clear the swampdragon connection:
Any ideas ?
I have been working on something using a twitter stream based on server monitor tutorial on the site. I am calling publish_data on every iteration of a for loop. Should I be making use of the metgod in a different way. The code is something like this:
def get_tweets():
iterator = twitter_stream.statuses.filter(track='bae')
for tweet in iterator:
publish_data('tweet',{
'text': tweet,
})
With get_tweets being called in the route
Hi,
when updating to the newest version of Swampdragon I have the problem that the Angular service is not working anymore.
I load most of the javascript files using require.js. The problem occours when the Angular service is loaded:
ReferenceError: swampdragon is not defined
at Object.<anonymous> (services.js:23)
The problem is that https://github.com/jonashagstedt/swampdragon/blob/master/swampdragon/static/swampdragon/js/angular/services.js tries to access the global variable swampdragon
which is not defined.
I think this has something to do with the recently introduced AMD support. When swampdragon.js is loaded via require.js it doesn't expose a global variable and the Angular service runs into this error.
If I define a custom field serializer, the field is always included in the output from serialize
regardless of publish_fields
:
class UserSerializer(ModelSerializer):
def serialize_avatar(self, obj):
return obj.get_avatar()
class Meta:
model = User
publish_fields = ('username', )
serializer = UserSerializer(instance=testUser)
serializer.serialize()
# {"username": "test", "avatar": "/images/test.png"}
I'm trying to create a Message
model using Javascript, and I've been left scratching my head.
When I use create
on my router and pass in a related user object, I receive an error "Duplicate entry 'test' for key 'username'"
; The serializer is trying to create a new user model rather than hook into the old one.
Have I missed something obvious, or is there no way to create a model re-using an existing foreign key?
var data = {
user: { id: 1, username: 'test', password: 'test' },
body: 'testing'
};
swampdragon.create('message', data, function (context, data) {
console.log('MessageStore: onCreate success.', context, data);
}, function (context, data) {
console.log('MessageStore: onCreate failed.', context, data);
});
class Message(SelfPublishModel, models.Model):
serializer_class = MessageSerializer
user = models.ForeignKey(User, related_name='messages')
body = models.TextField(blank=True)
class UserSerializer(ModelSerializer):
class Meta:
model = User
publish_fields = ('id', 'username')
update_fields = ('username', 'password',) # Had to add these to prevent validation fail on the model, but don't really want these to be updateable
class MessageSerializer(ModelSerializer):
user = UserSerializer
class Meta:
model = 'chat.Message'
publish_fields = ('body', 'user', )
update_fields = ('body', 'user', )
Is there anyway to use a similar approach to https://github.com/jrief/django-websocket-redis which:
- Runs a seperate Django main loop in a cooperative concurrency model using gevent, thus only one thread/process is required to control all open websockets simultaneously.
- No dependency to any other asynchronous event driven framework, such as Tornado, Twisted or Node.js.
That way the user can just start their server and not worry about also running the Tornado server.py
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.