Only return 100 users about python-twitter HOT 28 CLOSED

Interesting idea, though beware that I'm very reluctant to accept patches that
automatic trigger multiple queries to the Twitter API.

I suspect most users do not follow more than 200 users, so that would only 
trigger 2
queries.  We could enforce a maximum return say 1000 users, which still would 
only
amount to 10 queries.  Maybe the method could take a parameter which states the
number of users to return.

Hi, I'm attaching a simple fix to this issue. The patched functions (listed 
below)
would now carry a new keyword argument 'page' and this will allow one to 
retrieve all
the sets of friends, direct messages, followers, etc.

Here are the functions that have been changed in this patch that I've attached
(twitter_new.py):

* GetFriends(self, user=None)
* GetFollowers(self)
* GetDirectMessage(self, since=None)

With new signatures being:

* GetFriends(self, user=None, page=1)
* GetFollowers(self, user=None, page=1)
* GetDirectMessage(self, since=None, page=1)

As seen, GetFriends now has page support and GetFollowers has both user and 
page support.

To get all Friends, for instance, we could now do the following outside 
twitter.py:

    friends, counter = [], 1
    while not len(friends) % 100:
        friends += api_instance.GetFriends('someuser', page=counter)
        counter += 1

You could then check the length of the friends list to verify if all users have 
been got.

P.s. Also attaching the example retrieving code (sample.py), in case it does not
display well here.

Forgot to add, even replies are enhanced.

The following:

* GetReplies(self)

Becomes:

* GetReplies(self, page=1)

Just another vote for the OPTIONAL parameter page=x. I consider myself an 
average
Twitter user and have about two pages of 100 friends and followers.

PS, thanks for the patch, it helped me out with my problem :-)

Can anybody tell me how to apply patch in windows? And it seems that 
qwertymaniac
forgot to attach twitter_new.py

hi dclinton - are you going to incorporate the "page" option as outlined here? 
It
would be great if you did!

Another option would be to use a yield statement for returning users, which 
would
only trigger only when the caller requested users beyond the initial 100.

Assuming we already know the number of followers (which Twitter gives us), we 
can do
something like this:

page = 0
while (page * 100) < follower_count:
    url = 'http://twitter.com/statuses/followers.json?page=%d' % (page + 1)
    json = self._FetchUrl(url)
    data = simplejson.loads(json)
    self._CheckForTwitterError(data)
    page += 1
    for x in data:
        yield User.NewFromJsonDict(x)

(Note! this is untested code, but I've used similar functions in home-brew 
scripts
and they seem to work just fine.)

Similar methods could also be written for any Twitter API method that could
potentially use the page parameter.

Something like this seems necessary.  One way or another, the python API needs 
to be able to get the full 
complement of users (or other information which is "doled out" by the Twitter 
API)

I agree that this is needed. I just started using python-twitter and this was 
the
very first thing I bumped into. I have 500+ followers ... I guess I will try 
the patch. 

RE: Comment 3. Patch is against older version of twitter-python. I just tried 
it on
Rev 136 with following results. 
boblq@boblq-desktop:~/Projects/python-twitter/python-twitter$ patch -p5 
<twitter.patch 
patching file twitter.py
Hunk #1 FAILED at 1053.
Hunk #2 FAILED at 1061.
Hunk #3 FAILED at 1083.
Hunk #4 succeeded at 1580 with fuzz 1 (offset 480 lines).
Hunk #5 FAILED at 1617.
Hunk #6 FAILED at 1636.
5 out of 6 hunks FAILED -- saving rejects to file twitter.py.rej
boblq@boblq-desktop:~/Projects/python-twitter/python-twitter$ ls -l twitter.py*
-rwxr-xr-x 1 boblq boblq 67502 2009-03-11 12:02 twitter.py
-rwxr-xr-x 1 boblq boblq 67375 2009-03-11 12:01 twitter.py.orig
-rw-r--r-- 1 boblq boblq  4035 2009-03-11 12:02 twitter.py.rej
boblq@boblq-desktop:~/Projects/python-twitter/python-twitter$ 

Do we want to integrate this into the main trunk? 

Absolutely, this is a necessary addition.  If you don't have over 100
followers/followees you're not even a serious Twitterer.  Well, let me explain 
...

There are two kinds of people who use Twitter: People who only follow close 
friends
whose every tweet they intend on reading, and social networking people who 
"build
their web" by following LOTS of people.  The former type won't likely run into 
the
100 user limit, while the latter type will run into it within a couple days of
account creation.

Personally, I have two Twitter accounts, one that just follows close friends 
and one
that does the whole social networking thing (which is also the account I wrote 
the
auto-follow script for), and Python-Twitter is of limited utility for the second
account at the moment.  We already have the patch here to fix it, so let's do 
it.

OK. I made a pass at updating the patch originally submitted by querymaniac.
(Comments 3 & 4)

This patch breaks four of the unit tests. I looked at twitter_test.py, tried to 
fix
the tests, but did not understand the code. My test mojo is broken :( 

I did some minimal testing against my own twitter account and it seems to work
correctly. 

Maybe this will be of some use, but it should be tested more thoroughly and the 
unit
tests need to be fixed. (Also docs :) 

Still it is a start ... publish early, publish often. 

BobLQ

Re Comment 15: The patch is against svn 136. 

Okay, I'm looking at the patch now and I see those four failed tests.  I should 
be
able to fix the tests pretty simply and have a patch for the tests as well.

The only problem is this patch has some (very minor, trivial to resolve) 
conflicts
with my recently submitted patch for support for non-Twitter sites.  The longer 
we
leave all these patches hanging around without committing them, the uglier the
divergences get.  That's why quick committal is important, and why I'm angling 
for
commit access.

Just to be sure I understand, this change is simply to enable the 'page' 
parameter,
not to automatically make requests for multiple pages, right?

Okay, here's a fix for the tests.  The issue patch #20 now passes all tests.  I 
had
to append the new page parameter to the end of each URL that the testing 
program was
expecting.  I assert that this is correct because this is the same way that the
testGetPublicTimeline works.

And yes dclinton, this patch simply allows the page parameter; it does not
automatically request multiple pages of results.

Doing so will be an option for a future patch (not a default option, mind you, 
but
it's probably a good idea to handle summation of paged results once in the 
library
instead of repeatedly across a whole number of different programs using the 
library).

Whoops, I accidentally included the page patch for twitter.py in my diff as 
well as
the fixes to the test.  Ah well, I guess that just makes it more convenient to 
apply
and commit, eh?  :-D

Question about test_pagination.patch 

Why does the 
 def testGetReplies(self): 
not change to 
 def testGetReplies(self, page):

Or some such?

Do you have a url to a good tutorial on the python unit test framework (Else I 
Google)

Robert, I don't have a good URL, but let me try to explain.

Each function that starts with test... and then has a name of the function from 
the
PyTwitter library is a complete, self-contained, separate test that is called
directly by the unit testing framework.  As such, it can have no parameters, as 
the
test framework wouldn't know what to do with them.  We ensure that the page 
parameter
is handled properly within the unit test method.

I checked in a variant of the 'page' parameter patch in r137.  Please verify:

  http://code.google.com/p/python-twitter/source/detail?r=137

Will apply the test patch and mark this resolved.

A modified test patch applied in r138:

  http://code.google.com/p/python-twitter/source/detail?r=138

Marking fixed.  Please verify.

And thanks for the patches!

Looks correct to me.  Thanks dclinton.

RE Comment 23. Ok, I got it. Pretty obvious when you look at the changed code
http://code.google.com/p/python-twitter/source/detail?r=138 Thanks Cyde.

Nice to get this one fixed.

Thanks!