praw-dev / prawtools Goto Github PK

They don't generate link karma on reddit and are often used for mod announcements. Recalculating without them will show purely user stats.

Traceback (most recent call last): File "/home/jonas/local/reddit/bin/subreddit_stats", line 9, in <module> load_entry_point('prawtools==0.3', 'console_scripts', 'subreddit_stats')() File "/home/jonas/local/reddit/local/lib/python2.7/site-packages/prawtools/stats.py", line 450, in main options.commenters, 5, 5, options.top, options.debug) File "/home/jonas/local/reddit/local/lib/python2.7/site-packages/prawtools/stats.py", line 338, in publish_results t_submitters = self.top_submitters(submitters, num_submissions) File "/home/jonas/local/reddit/local/lib/python2.7/site-packages/prawtools/stats.py", line 250, in top_submitters retval += ' 0. [{0}]({1})'.format(title, sub.url) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 31: ordinal not in range(128)

This happens in /r/eagles because of this thread which includes unicode in the title: http://redd.it/uar6r

Trying to get list of flairs for a subreddit, getting the following error:

AttributeError: 'Subreddit' object has no attribute 'get_flair_list'

Is the tool actually updated to use praw4 completely, or is it unsafe to use? I don't want to destroy my subs' current properties :)

https://github.com/praw-dev/prawtools/blob/master/prawtools/stats.py#L31 on every post and in the subreddit sidebar surely violate reddit's self-promotion rule. If you want to include that in the code, you might want to disable it by default so others don't inadvertently advertise shady "give me cash" links.

Relevant comments:

• http://www.reddit.com/r/canada/comments/vugzs/subreddit_stats_canada_top_submissions_from/c57qmua
• http://www.reddit.com/r/canada/comments/vugzs/subreddit_stats_canada_top_submissions_from/c57t4js

/u/dequeued wrote:

Excellent!
I'd love to see a table or two with statistics on frequent good commenters/submitters rather than the emphasis on very highly upvoted comments and submissions. Maybe something like:

• Count of upvoted comments (2 or more upvotes, just pure count)
• Count of submissions with 25+ upvotes
• Ratio of upvoted comments to downvoted comments for people with 100+ comments

Thanks for bringing this to Reddit.

Reference: https://www.reddit.com/r/modclub/comments/4vnd2x/submitter_and_commenter_statistics_for_your/d60qp42

Relevant comment

I receive this message when running subreddit_stats:

DepreiationWarning: Password-based authentication will stop working on 2015-08-03 and as a result will be removed from Praw4.

Are there plans to update PRAWtools to OAuth?

Spreadsheets should be private to anyone with the link, and a link should be added to the submission with the statistics.

The None user should be represented as '[deleted]'.

Example of bug:
http://www.reddit.com/r/mylittlefortress/comments/t6iht/subreddit_stats_mylittlefortress_submissions_from/

Due to not being able to fetch upvote and downvote totals.

I'm on windows 8.1. I have a multi-threaded reddit bot that runs three scans, one is using helpers.comment_stream, another that runs helpers.submission_stream. The third thread scans my inbox looking for messages every 60 secs to tell the bot to update configuration from a wiki.

Most of the time all three run without a problem. But intermittently my praw-multiprocess gives the following error twice (I'm assuming one for each stream thread):

Traceback (most recent call last):
File "C:\Python34\lib\threading.py", line 921, in _bootstrap_inner
self.run()
File "C:\Python34\lib\threading.py", line 869, in run
self._target(_self._args, *_self._kwargs)
File "C:\Python34\lib\socketserver.py", line 616, in process_request_thread
self.handle_error(request, client_address)
File "C:\Python34\lib\socketserver.py", line 613, in process_request_thread
self.finish_request(request, client_address)
File "C:\Python34\lib\socketserver.py", line 344, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python34\lib\socketserver.py", line 669, in init
self.handle()
File "C:\Python34\lib\site-packages\praw\multiprocess.py", line 80, in handle
cPickle.dump(retval, self.wfile, # pylint: disable-msg=E1101
UnboundLocalError: local variable 'retval' referenced before assignment

Afterwards, I stop seeing requests for either of my stream threads. The mail requests still show up every 60 seconds. I have to restart the bot to recover.

This error is becoming more common. This log for /r/Texas. The same error happens with I run a report for /r/Eagles (year), and /r/ultrahardcore (all).

Thanks for any help you can offer.

subreddit_stats -R subreddit_stats -u USERNAME -p PASSWORD -vvv -t month
Enter subreddit name: texas
You chose to analyze this subreddit: texas
Logging in
DEBUG: Fetching submissions
DEBUG: Found 446 submissions
DEBUG: Processing Submitters
DEBUG: Processing Commenters on 446 submissions
Traceback (most recent call last):
  File "/usr/local/bin/subreddit_stats", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/2.7/site-packages/prawtools/stats.py", line 572, in main
    srs.process_commenters()
  File "/Library/Python/2.7/site-packages/prawtools/stats.py", line 241, in process_commenters
    skipped = submission.replace_more_comments()
  File "/Library/Python/2.7/site-packages/praw/objects.py", line 1122, in replace_more_comments
    new_comments = item.comments(update=False)
  File "/Library/Python/2.7/site-packages/praw/objects.py", line 700, in comments
    return self._continue_comments(update)
  File "/Library/Python/2.7/site-packages/praw/objects.py", line 683, in _continue_comments
    assert len(self.children) == 1 and self.id == self.children[0]
AssertionError

Moderators make a lot of comments/posts as they go about their duties.

For example, I'm listed as the top commenter in the /r/Austin report that I recently ran even though a good portion of my comments are janitorial work.

I'd like an option to exclude these distinguished posts/comments from the statistics report, for a more accurate view of the submitters.

This should be an average of the a submissions / comments per hour. Subsequent subreddit stat updates will need to be made to see a change in submission rate.

I use subreddit_stats monthly on several subreddits I moderate. I'd like to be able to specify to which subreddit the stat output will be posted, rather than having all of them posted to /r/subreddit_stats. As it stands, I must go to the post in /r/subreddit_stats, copy the text, and submit it again to the appropriate subreddit. Another problem is that reddit forces me to wait ten minutes between subreddit_stats runs, which makes running a batch script impossible.

Re-adding the ability to specify the target subreddit as a command line switch would take care of both of these problems at once.

Flair is becoming more and more common throughout reddit. For example, in /r/Austin we use it to categorize question posts with the "Ask Austin" flair.

Can you add an option to run statistics for one of a subreddit's flair? In /r/Austin, that might be statistics for just the "Ask Austin" flair.

I don't know if this is problem with praw or subreddit_stats, but I get this error when I run it on PowerShell with Python 2.7.9:

Traceback (most recent call last):
  File "C:\Python27\Scripts\subreddit_stats-script.py", line 9, in <module>
    load_entry_point('prawtools==0.19', 'console_scripts', 'subreddit_stats')()
  File "build\bdist.win32\egg\prawtools\stats.py", line 554, in main
  File "build\bdist.win32\egg\prawtools\stats.py", line 107, in login
  File "C:\Python27\lib\site-packages\praw\decorators.py", line 88, in wrapped
    return function(self, *args, **kwargs)
  File "C:\Python27\lib\site-packages\praw\__init__.py", line 1316, in login
    pswd = getpass.getpass('Password for %s: ' % user)
  File "C:\Python27\lib\getpass.py", line 95, in win_getpass
    msvcrt.putch(c)
TypeError: must be char, not unicode

I will try to see what can be done within subreddit_stats to get it working cross-platform on Python 2.7...

When I leave reddit_alert running, it keeps eventually crashing with this traceback:

Traceback (most recent call last):
File "/usr/local/bin/reddit_alert", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/prawtools/alert.py", line 77, in main
for comment in session.subreddit(subreddit).stream.comments():
File "/usr/local/lib/python2.7/site-packages/praw/models/util.py", line 40, in stream_generator
limit=limit, params={'before': before_fullname}))):
File "/usr/local/lib/python2.7/site-packages/praw/models/listing/generator.py", line 70, in next
return self.next()
File "/usr/local/lib/python2.7/site-packages/praw/models/listing/generator.py", line 43, in next
self._next_batch()
File "/usr/local/lib/python2.7/site-packages/praw/models/listing/generator.py", line 53, in _next_batch
self._listing = self._reddit.get(self.url, params=self.params)
File "/usr/local/lib/python2.7/site-packages/praw/reddit.py", line 207, in get
data = self.request('GET', path, params=params)
File "/usr/local/lib/python2.7/site-packages/praw/reddit.py", line 258, in request
return self._core.request(method, path, params=params, data=data)
File "/usr/local/lib/python2.7/site-packages/prawcore/sessions.py", line 119, in request
params)
File "/usr/local/lib/python2.7/site-packages/prawcore/sessions.py", line 73, in _request_with_retries
raise self.STATUS_EXCEPTIONSresponse.status_code
prawcore.exceptions.Forbidden: received 403 HTTP response

Any ideas?

Stats bot, which continuously runs subreddit_stats, was consuming over 4.6GB of memory. That's too much. Rather than saving all the submissions and comments for later aggregation, they should be processed and discarded immediately.

This is my first time using this on a Mac. Sorry if this problem is user error.

This is the command I ran: subreddit_stats -t month -vvv -s 10 -c 10 -R MYPRIVATESUBREDDIT -u MYUSERNAME -p MYPASSWORD

It errors out with this:

Traceback (most recent call last):
  File "/usr/local/bin/subreddit_stats", line 11, in <module>
    sys.exit(main())
  File "/private/tmp/pip_build_root/prawtools/prawtools/stats.py", line 576, in main
  File "/private/tmp/pip_build_root/prawtools/prawtools/stats.py", line 401, in publish_results
  File "/private/tmp/pip_build_root/prawtools/prawtools/stats.py", line 334, in top_commenters
  File "/private/tmp/pip_build_root/prawtools/prawtools/stats.py", line 60, in _pts
ValueError: zero length field name in format

Everytime I try to run subreddit_stats on /r/bitcoin it gives me the following error.

"No submissions were found."

I'm using the following command
subreddit_stats -d30 -vv -R phloatingman -u phloating_man bitcoin

I successfully ran it on /r/videography and /r/photography, but it doesn't work on /r/bitcoin. I'm using Ubuntu 12.10 32-bit.

Great work by the way.

Hi!

I'm trying to get results up to 9/1, but it is only returning results up until 8/30 20:08 PDT.

I don't believe I'm hitting the 1000 post limit, though I'm no doubt doing something wrong:

vagrant@precise32:~$ subreddit_stats -d23 -vv -R r_javascript_stats -u r_javascript_stats -p hunter2 javascript                                                                                                                                  
You chose to analyze this subreddit: javascript
Logging in
DEBUG: Fetching submissions
DEBUG: Found 401 submissions
DEBUG: Processing Submitters
DEBUG: Processing Commenters on 401 submissions
Ignored 0 comments (1 MoreComment objects)
Ignored 0 comments (1 MoreComment objects)
You are about to submit to subreddit 'r_javascript_stats' as 'r_javascript_stats'.

Here's my environment:

vagrant@precise32:~$ subreddit_stats --version
BBoe's PRAWtools 0.18
vagrant@precise32:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04 LTS
Release:        12.04
Codename:       precise

Please let me know if you need additional information!

Pi31415926 wrote:

Hiya, two more ideas for you.. :)

• top domains - which were the most frequently submitted, and/or, which domains earned the most karma?
• period length - and/or start/end dates of period

• A character is needed for the title when linking to comments: https://www.reddit.com/comments/b4cxpd//ej6ut44?context=1 should be https://www.reddit.com/comments/b4cxpd/_/ej6ut44?context=1
• List numbering doesn't quite work out like it used to in the new layout.

I am using python 3.3, which I believe is supported from what I can tell in setup.py. I get the following error when running subreddit_stats:

Username: user
Traceback (most recent call last):
  File "c:\python33\lib\runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python33\lib\runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "C:\Python33\Scripts\subreddit_stats.exe\__main__.py", line 9, in <module
>
  File "c:\python33\lib\site-packages\prawtools\stats.py", line 554, in main
    srs.login(options.user, options.pswd)
  File "c:\python33\lib\site-packages\prawtools\stats.py", line 107, in login
    self.reddit.login(user, pswd)
  File "<decorator-gen-51>", line 2, in login
  File "c:\python33\lib\site-packages\praw\decorators.py", line 78, in wrap
    return function(*args, **kwargs)
  File "c:\python33\lib\site-packages\praw\__init__.py", line 1450, in login
    .encode('ascii', 'ignore'))
  File "c:\python33\lib\getpass.py", line 104, in win_getpass
    msvcrt.putwch(c)
TypeError: must be a unicode character, not int

I am trying to run prawtools, but I keep running into this error:

traceback (most recent call last):
  File "/usr/local/bin/subreddit_stats", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/2.7/site-packages/prawtools/stats.py", line 554, in main
    srs.login(options.user, options.pswd)
  File "/Library/Python/2.7/site-packages/prawtools/stats.py", line 107, in login
    self.reddit.login(user, pswd)
  File "/Library/Python/2.7/site-packages/praw/__init__.py", line 1266, in login
    self.user = self.get_redditor(user)
  File "/Library/Python/2.7/site-packages/praw/__init__.py", line 890, in get_redditor
    return objects.Redditor(self, user_name, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/praw/objects.py", line 663, in __init__
    fetch, info_url)
  File "/Library/Python/2.7/site-packages/praw/objects.py", line 72, in __init__
    self.has_fetched = self._populate(json_dict, fetch)
  File "/Library/Python/2.7/site-packages/praw/objects.py", line 127, in _populate
    json_dict = self._get_json_dict() if fetch else {}
  File "/Library/Python/2.7/site-packages/praw/objects.py", line 120, in _get_json_dict
    as_objects=False)
  File "/Library/Python/2.7/site-packages/praw/decorators.py", line 161, in wrapped
    return_value = function(reddit_session, *args, **kwargs)
  File "/Library/Python/2.7/site-packages/praw/__init__.py", line 526, in request_json
    data = json.loads(response, object_hook=hook)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Would it be possible to get subreddit stats from a date range.

For example, just get just the stats for the month of January 2017?

Top month submissions should include the month name rather than the date range.

Top year submissions should include the year name rather than the date range.

The range should be shown when more than 1% of the submissions are outside of the bucketed range (11 out of 1000 submissions occurred in 2013, the remainder in 2012).