publish_data functionality about swampdragon HOT 38 CLOSED

You could just publish the entire list of tweets in one go:

tweets = twitter_stream.statuses.filter(track='bae')  # tweets needs to be a list
publish_data('tweets', {'tweets': tweets})

from swampdragon.

So just to be clear the publish_data method is not called on each iteration?

from swampdragon.

Yes.
You can do that if you want to of course, but you might as well publish all the tweets.

from swampdragon.

The reason why I need to loop is that the tweets are coming in as a "constant" stream and twitter_stream.statuses.filter(track='bar') returns an iterator object. I'm just not being able to send anything down the channel. I'll keep banging my head against it.

from swampdragon.

Okay is the issue that your tweets aren't showing up when viewing the page?

from swampdragon.

I am not able to receive anything in the browsers javscript. I assumed it was my approach of calling publish_data in the for loop. It must be something else I'm doing wrong.

from swampdragon.

Can you post your routers.py, and the javascript (most importantly when you create the connection to SD, your on_channel_message function and your subscribe function

from swampdragon.

route.py
`from swampdragon import route_handler
from swampdragon.route_handler import BaseRouter
from .twitterstream import get_tweets

class TweetsRouter(BaseRouter):
route_name = 'tweets'

def get_subscription_channels(self, **kwargs):
    get_tweets()
    return ['tweet']

route_handler.register(TweetsRouter)`

tweets.js

`var dragon = new VanillaDragon({onopen: onOpen, onchannelmessage: onChannelMessage});

function onChannelMessage(channels, message) {
console.log(message);
}

function onOpen(){
dragon.subscribe('tweets', 'tweet', null)
}`

from swampdragon.

does get_tweets() start sending tweets?

To make sure you successfully subscribe, change the onOpen code to look like this:

function onOpen(){
    dragon.subscribe('tweets', 'tweet', null, function(response) {
        console.log(response);
    }, function(response) {
        console.log("Failed to subscribe");
    })
}

Let me know if this works (if you get a response or if you get "Failed to subscribe")

from swampdragon.

I an managing to connect the console returns Object {client_callback_name: "cb_0", state: "success", verb: "subscribe"}

get_tweets is the method in which publish_data is called. I can print each tweet on each iteration but I cannot publish the data.

from swampdragon.

If you print the tweets, does it keep printing them after you have subscribed? (want to be sure the tweets are coming in after the user subscribes, or there won't be any tweets to publish)

from swampdragon.

Yes it does. Just as expected.

from swampdragon.

You have redis running? and it's 2.8 or higher?
The code looks right (unless there is a typo we aren't seeing)

from swampdragon.

I have redis 2.8.19 running. Should I be able to see connections in redis?

from swampdragon.

You would be able to see the pubsub channels but since subscribe is successful it's probably not there.

Let's do this:
Create a management command that simply publish a message to the tweet channel.

mkdir management
cd management
mkdir commands
cd commands
vim foo.py (or whatever editor you use)

Add the following code (don't forget the imports)

class Command(BaseCommand):
    def handle(self, *args, **options):
        publish_data(channel='tweet', data={'foo': 'bar'})

then run python manage.py foo
and see if anything is published in the console of the browser

from swampdragon.

That works. I can log the data to the console..

from swampdragon.

okay so we've isolated the issue to be in get_tweets()

comment out the publish_data command in get_tweets and add the one from the management command that we now know works.
See if that outputs data. If not, we know where the problem is

from swampdragon.

I have the following: def get_tweets(): call_command('foo') The connection is established but nothing is published.

from swampdragon.

Could you share the code of get_tweets (just remove any API keys or such, if there are any).

The problem seems to be there

from swampdragon.

I actually managed to publish with the same get_tweets function calling the foo command. How ever if I put call_command('foo') in a while loop the command is executed nothing published and the browser does not log a connection or the result of the management command publishing.

def get_tweets():
    while True:
         call_command('foo')
         time.sleep(5)

from swampdragon.

aah, don't use time.sleep.

Try this

from tornado.ioloop import PeriodicCallback


pcb = None


def get_tweets():
    global pcb

    if pcb is None:
        pcb = PeriodicCallback(get_tweets, 500)
        pcb.start()

     call_command('foo')

let me know if that is working (should publish every half a second)

from swampdragon.

That works fine. Edit for anyone who looks at the above code. the first global pcb should be pcb = None

from swampdragon.

Updated.

Okay you can now tweak your tweet stream code.
So take all the incoming tweets and put them in a list and publish the list every Nth millisecond.

I assume the tweet stream is a constant incoming stream (based on your previous code).

If you wanted to take it one step further you could write it to a list in redis and truncate it every time it reached X number of entries, and then for new entries and publish them if there are any, if that makes sense

from swampdragon.

I think I'll have look at doing something like that. So we can conclude that it is not possible to call publish_data in the iteration of the for loop?

from swampdragon.

You should be able to call publish_data in a normal for-loop (try it in the management command: for x in range(10): publish_data(channel='tweet', data={'foo': 'bar'}), however it does seem like maybe something is happening in that particular for-loop that is preventing it from working

from swampdragon.

Firefox has been a little more informative I'm getting a Cross-orign blocked error:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://127.0.0.1:9999/data/info. This can be fixed by moving the resource to the same domain or enabling CORS.

from swampdragon.

What version of SD are you running?

This shouldn't be an issue though since you are actually receive the data if you run the management command, so I think that might be a red herring.

from swampdragon.

SwampDragon==0.3.9

from swampdragon.

Just to be clear this code: for x in range(10): publish_data(channel='tweet', data={'foo': 'bar'})works as expected.

from swampdragon.

Okay wanted to be sure you didn't use an older version.

So to conclude, everything works except when you put it inside the for-loop of the tweet stream?

It does seem like something in your for-loop is blocking SD.
Is there any way I can replicate this?

from swampdragon.

I have created this repo but it will require you to have twitter credentials https://github.com/ernestjumbe/meetupdemo/tree/master

from swampdragon.

I will give it a go in a bit (have to finish up at work first).
There are a couple of things:

• How frequent are the tweets?
• Have you tried printing a tweet before publishing it (so first print it and then publish)? This is just to confirm that the tweet is actually not published

Give this a go:

for tweet in iterator:
    text = tweet.get('text')
    if not text:
       continue
    print(text)
    publish_data(channel='tweet', {message: text})

from swampdragon.

First of all thanks for so much help. I've learnt some new approaches to debugging. The code above prints the text but does not publish and a connection isn't established.

from swampdragon.

I'll finish my current work in 30 minutes.

In the meantime, have a look at ipdb. If you aren't already using it.
If you are using ipdb ignore the following message:

1. Install ipdb
   pip install ipdb
2. Anywhere you want to look at the contents of variable etc. when you are running your code you can add: import ipdb; ipdb.set_trace()

Take the router as an example:

def get_subscription_channels(self, **kwargs):
    foo = 'hello'
    import ipdb; ipdb.set_trace()
    get_tweets()
    return ['tweet']

If you then run your application, when the client gets the subscription channel you will drop into a shell where you can inspect the variables.
As we have set foo to 'hello' you can type:

print(foo) and see the output.

You can then use s for stepping and c to continue the execution.

from swampdragon.

Ah I found the problem!

So this is what happens:
When it calls get_tweets it gets stuck there. Since the loop never exists it will never continue to the line return ['tweet']

def get_subscription_channels(self, **kwargs):
    get_tweets()  # <-- It's stuck here
    return ['tweet']

from swampdragon.

Okay here is the fix:

Remove get_tweets from the router.

Open the management command foo.py and change it to this:

class Command(BaseCommand):
    def handle(self, *args, **options):
        get_tweets()

start your servers

In a new terminal run python manage.py foo

Presto!

The reason it didn't work was because get_tweets was blocking.
So when a client tried to subscribe it just got stuck on get_tweets forever.

Let me know if you have any problems or if it's working.

from swampdragon.

Perfect that works. Any advise on a production situation and running such a script?

from swampdragon.

In terms of deploying SD:
http://swampdragon.net/blog/deploying-swampdragon/

If you look in the supervisord section you can add another entry for running foo.py.
That would look something like this:

[program:tweetsstream]
command=/path/to/virtualenv/bin/python /path/to/project/manage.py foo
user=myuser
autostart=true
autorestart=true
stderr_logfile=/path/to/logfiles/foo.log
stdout_logfile=/path/to/logfiles/foo.log
stopsignal=INT

Since it's a one-way street you only need one process for it.

If you are unfamiliar with supervisor or deployment in general I would recommend:

• supervisord for managing your processes (starting django, starting swampdragon, celery if you use that etc.)
• NGINX for serving up Django (works well with SwampDragon)
• uWSGI for Django

For deployment I would strongly recommend Fabric.

from swampdragon.