kharrigian / retriever Goto Github PK

There's a bug in retrieve_submission_comments. The general except Exception hid it. Don't do that. The PRAW fallback block didn't turn the list into a DataFrame, which threw an error when it got to df.sort_values, and the program thought it was a request error instead of a code error so it keeps making requests and failing.

Here's the modified code:

                ## Fall Back to PRAW
                if hasattr(self, "_init_praw") and self._init_praw and len(df) == 0:
                    temp_dfs = []
                    for s in submissions_clean:
                        temp_dfs.append(self._retrieve_submission_comments_praw(submission_id=s))
                    df = pd.concat(temp_dfs)
                ## Sort
                if len(df) > 0:
                    df = df.sort_values("created_utc", ascending=True)
                    df = df.reset_index(drop=True)
                return df
            except Exception as e:
                self.logger.warning(f"{e}")
                sleep(backoff)
                backoff = 2 ** backoff

When retrieving Reddit data via retrieve_subreddit_submissions (and possibly others) there is a log line for every 100-post request submitted via the PushShift API. Would be nice to have a toggle on how much logging happens. Example image below:

PSAW is no longer maintained and has a number of bugs that lead to sub-optimal query performance. Either find an alternative package or fork and update.

PSAW + Reddit APIs appear to have changed, making much of the functionality in this package obsolete. A complete re-factor is probably the best approach for addressing these issues and making things functional again.