otosky / medium_stats Goto Github PK
View Code? Open in Web Editor NEWCommand Line and Python tool for Scraping Your Medium Stats
License: GNU General Public License v3.0
Command Line and Python tool for Scraping Your Medium Stats
License: GNU General Public License v3.0
This is a fantastic library, thanks for building it!
I've got a publication with more than 50 articles, and would like to fetch stats for all of them. However, it seems like it's not possible to fetch more than 50 at the moment due to Medium's pagination - see you comment here:
medium_stats/medium_stats/scraper.py
Line 297 in ef96a20
I am happy to test this using the publication I manage, if this helps.
Until a week ago scraping publication stats worked.
Suddenly, last week, it stopped working.
Command:
medium-stats scrape_publication -u <username> -s <pubname> --output_dir . --sid "<mySID>" --uid "<myUID>" --all
Traceback (most recent call last):
File "/Users/dave/data-projects/marketing-pipeline/venv/bin/medium-stats", line 8, in <module>
sys.exit(main())
File "/Users/dave/data-projects/marketing-pipeline/venv/lib/python3.10/site-packages/medium_stats/__main__.py", line 220, in main
data = sg.get_all_story_overview()
File "/Users/dave/data-projects/marketing-pipeline/venv/lib/python3.10/site-packages/medium_stats/scraper.py", line 294, in get_all_story_overview
data = self._decode_json(response)
File "/Users/dave/data-projects/marketing-pipeline/venv/lib/python3.10/site-packages/medium_stats/scraper.py", line 146, in _decode_json
return json.loads(cleaned)["payload"]
File "/Users/dave/.pyenv/versions/3.10.9/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/Users/dave/.pyenv/versions/3.10.9/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Users/dave/.pyenv/versions/3.10.9/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
JSON is expected, but not returned.
medium_stats would output the stats to ./stats_export/<publication>
Anyone else running into issues, got a workaround? Or is Medium updating its stats pages?
Hi,
Here
medium_stats/medium_stats/scraper.py
Line 277 in dbe4164
As a consequence, when you get a new follower and you can set up a start date prior to this date anymore because of
medium_stats/medium_stats/scraper.py
Line 281 in dbe4164
It would be awesome to get the story topics for a specific story as it makes it easer to see which story topics were getting the most traction during a given time. It would then help any user to focus on content around the topics which are getting the most hits.
Story topics can be fetched from : https://medium.com/p/<POST_ID>/settings
I would be glad to contribute towards this issue. Would really appreciate if you can push me in the right direction.
I think it's common for people to be part of multiple publications. Right now, you have to make a separate section for each and copy the same sid/uid values, which would all have to be updated when they change.
It would be great to have a field like aliases
to list the names of publications the values in a section also apply to.
I'm sure I'm just missing something. I'm fairly new to python.
I'm running pip install medium_stats
(and also medium-stats
) but when I run the sample script included in the readme.md (with my info) it errors with:
File "<directory path>/medium_stats.py", line 3, in <module> from medium_stats.scraper import StatGrabberUser
ModuleNotFoundError: No module named 'medium_stats.scraper'; 'medium_stats' is not a package
This is happening on 2 different systems where other python scripts are running fine.
When I'm trying to scrape a publication under a custom URL, that works fine. But a regular publication under medium.com seems to require the full URL (at least including the domain).
So this works:
scrape_publication -u medium.com/name-of-publication
while this doesn't:
scrape_publication -u name-of-publication
The error is socket.gaierror: [Errno 8] nodename nor servname provided, or not known
, which to me suggests that the script doesn't prepend the medium.com/ bit.
When scraping a publication, it seems that the credits file is matched against the URL the user specifies, which might include the medium.com/ part or not. There should be a canonical name for a publication (IMHO just the part of the URL after https://medium.com/) that is used everywhere. That way, even if I specified the full URL, it would still match the creds based on just the canonical name.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.