Comments (18)
from mastodon-archive.
Line 1124 in Mastodon.py is a simple sleep
instruction. This remains a mystery. The only place where a General API problem is raised is when a request returns with a code 500 ("Internal Server Error"). That's why I suspect it's a problem on the server side.
from mastodon-archive.
If the problem doesn't occur anymore, I'll just close this issue. What do you think?
from mastodon-archive.
I just tried running your updated 'archive' package instructions on mammouth.cafe.
The package installed fine.
I opted to skip favourites. Btw, I think you have a mistake in your command there. It should be...
mastodon-archive archive --no-favourites [email protected]
It was still a little slow, but eventually it kicked this out, which looks similar to dexamphetamine's error:
Get statuses (this may take a while)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/bin/mastodon-archive", line 11, in <module>
load_entry_point('mastodon-archive==0.0.3', 'console_scripts', 'mastodon-archive')()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon_archive/__init__.py", line 65, in main
args.command(args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon_archive/archive.py", line 110, in archive
first_page = statuses)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon/Mastodon.py", line 907, in fetch_remaining
current_page = self.fetch_next(current_page)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon/Mastodon.py", line 866, in fetch_next
return self.__api_request(method, endpoint, params)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mastodon/Mastodon.py", line 1124, in __api_request
raise MastodonAPIError('General API problem.')
mastodon.Mastodon.MastodonAPIError: General API problem.
I will try it again on the mastodon.social account to see if there's a difference.
Btw, mammouth.cafe is running Mastodon 1.8.x. At least one version behind.
from mastodon-archive.
Just tested on mastodon.social (again using the 'no favourites' option) and it backed up quickly and without errors.
Maybe there is something related with the version of Mastodon?
Nope, maybe not. social.wxcafe.net appears to be running 2.0.0 so that theory is blown. And come to think of it, DredMorbius backed up his account on mammouth.cafe successfully.
I guess I'm just destined to lose those toots. No way around it.
from mastodon-archive.
from mastodon-archive.
Sadly, I never posted too much on mammouth.cafe:
Get user info
Get statuses (this may take a while)
Get favourites (this may take a while)
Saving 64 statuses and 65 favourites
from mastodon-archive.
On line 77 is the setup for the Mastodon API in case you have successfully authorized the app before. This is what it looks like:
mastodon = Mastodon(
client_id = client_secret,
access_token = user_secret,
api_base_url = url)
Based on the documentation this is the equivalent of:
mastodon = Mastodon(
client_id = client_secret,
access_token = user_secret,
api_base_url = url,
ratelimit_method='wait',
ratelimit_pacefactor=1.1,
request_timeout=300)
Thus, these are the three parameters we can play with. Let's assume the server has not changed the 300s time window for requests. Perhaps we can simply avoid running into timeouts? The documentation has some suggestions. How about pacing our requests and have them be slightly slower than expected. We should never run into a timeout:
mastodon = Mastodon(
client_id = client_secret,
access_token = user_secret,
api_base_url = url,
ratelimit_method='pace',
ratelimit_pacefactor=0.9,
request_timeout=300)
Give it a try and let us know?
If you have mastodon-archive 0.0.3 installed, you need to download the latest release, unzip it into a directory, make the changes, and install those changes using pip install -e .
(run from the directory where the setup.py
file is).
Verify:
$ ls /usr/local/lib/python3.6/site-packages/mastodon-archive*
/usr/local/lib/python3.6/site-packages/mastodon-archive.egg-link
$ cat /usr/local/lib/python3.6/site-packages/mastodon-archive.egg-link
/Users/alex/Source/mastodon-backup
As you can see, the system wide installation is but a link that points to my working directory.
To undo your changes, run pip uninstall mastodon-archive
followed by pip install mastodon-archive
and you should get back the default.
from mastodon-archive.
If you have mastodon-archive 0.0.3 installed, you need to download the latest release...
Do you mean if I don't have it installed? Because I have the latest version (0.0.3) installed, but I installed it before your notes above. Should I re-install it and then make the noted changes?
Anyway, I'll try it with the install I have, and then again with new if it doesn't work. ;)
from mastodon-archive.
install those changes using
pip install -e .
(run from the directory where thesetup.py
file is)
Is the command going to be pip
or pip3
? (I've been using the latter up to this point since installing Python 3.6.)
Also, where is setup.py
? I can find all kinds of .py
files, but not that one.
At this point I've modified archive.py
, but I don't quite understand the install changes part your talking about.
from mastodon-archive.
Hm, perhaps I was writing this from a developer perspective where I always have two copies of the thing:
/Users/alex/src/mastodon-backup
is where I work and make my changes and /Library/Frameworks/Python.framework/Versions/3.6/bin/mastodon-archive
is where it was installed. So if I make changes in in the first directory, and run the app, it will run the unchanged files from the second directory.
If you only have the second directory, and you make your changes there, then running the app should run the changed files and you're fine. Sorry for the confusion!
The setup.py
is only available if you checked out the sources or downloaded a zip file from the GitHub releases, which you didn't since you used pip to install it.
As for pip
vs. pip3
: I think the way it was meant to be was the tool is called pip
. But then people started switching to Python 3 and others decided to keep Python 2 around, and they both came with a slightly different pip. What to do? Installers these days will simply rename python
to python3
and pip
to pip3
if they're installing Python 3 on a system where Python 2 already exists.
from mastodon-archive.
If you only have the second directory, and you make your changes there, then running the app should run the changed files and you're fine.
Okay, I went ahead and tried that anyway. ;) It's still sitting at:
Get statuses (this may take a while)
And that was at least 20 minutes ago.
Anyway, if it doesn't work this time, I'm not going to worry about it anymore for mammouth.cafe. Of all the content there, I would probably only re-publish a fraction of it. I was more interested in some of the ideas I shared because they were topics to expand on as blog articles. That's the only bummer. But I can live it. Sometimes it's better to just let shit go ;)
from mastodon-archive.
Before giving up altogether, you might want to give the alternatives a try. Mastotool
downloads the public stuff that you can see on https://mammouth.cafe/@wion
i.e. none of the replies but all of your "top level" posts. Mastotool Atom
downloads the toots including your replies but Mastodon has a bug such that the feed ends if it ever gets to a "gap" in your toots: if you ever deleted a toot, that's where your Atom feed ends. I haven't tried MastoUserScrape.py
.
from mastodon-archive.
Switching to pace and reducing the factor like you said worked perfectly, no need to skip favorites. Thanks!
from mastodon-archive.
from mastodon-archive.
Happy to report it worked for me too! It took a while (I lost track how long) but I came back to the computer later and 2011 statuses were successfully backed up.
from mastodon-archive.
Nice work on this tool. And thanks for your patience with the likes of me.
from mastodon-archive.
Excellent. 1a876e5 adds the --pace
option, so the next release will have this.
from mastodon-archive.
Related Issues (20)
- HTML export to use cropped, centered images HOT 4
- HTML Export posts videos with a <img> tag HOT 6
- HTML Export pages should lazy load videos and images to be usable as static sites HOT 14
- Add pagination numbers in html exports
- TypeError: 'NoneType' object is not subscriptable HOT 8
- Backup other account's public posts HOT 1
- Don't burst the API rate limiting HOT 1
- Archiving context HOT 3
- Not an issue, just saying thanks :) HOT 1
- Install "the Debian way" HOT 8
- getting mastodon-backup packaged (for Linux) HOT 19
- Generate tarball compatible to Mastodon's export HOT 5
- Specifying bookmarks or statuses with text search resets other specified collections HOT 2
- Feature: Backup / Export Filters HOT 8
- incremental runs of 'mastodon-archive' giving 404s on deleted mentions/replies on STDERR HOT 7
- Compress JSON HOT 8
- Consider making --with-followers and --with-following on by default? HOT 3
- Improve followers feature : consider more interactions HOT 4
- Report fails if no bookmark in archive (old archive) HOT 4
- Version check fails on Pleroma instance HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mastodon-archive.