Git Product home page Git Product logo

debirdify's Introduction

Debirdify

Find out which of your Twitter followers have a Mastodon account

This is a simple (and experimental) web app that uses the Twitter API to find out which people you follow and searches their names and bios for things that look like Mastodon IDs, and, if that fails, checks if there are any seemingly Mastodon-related keywords in their bios.

A CSV export for import into Mastodon is also provided.

Canonical instance

An experimental canonical instance of this is hosted on https://pruvisto.org/debirdify.

Implementation

This project uses Django and Python. Configuration of things like the callback URL and API keys and secrets is done with WSGI environment variables.

The Twitter authentication information is stored in a cookie for 7 days so that users do not have to re-authenticate every time they revisit the website.

The logical core of the implementation is in main/extract_mastodon_ids.py. Using this, you can also create a standalone application to find Mastodon IDs in account bios/names.

Installation

Install postgresql, libpq-dev, python3 (version >=3.9), python3-pip, and python3-venv. Then run

python3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt

Database

Create a new database user debirdify, a database debirdify while also granting all privileges to the user debirdify.

TODO: initialisation of database

Configuration

Environment variables being used are:

  • DEBIRDIFY_ALLOWED_HOSTS: a list of hosts the server will accept in the format ['host1', 'host2']
  • DEBIRDIFY_CONSUMER_CREDENTIALS: Twitter consumer key and secret in the format key:secret
  • DEBIRDIFY_DJANGO_SECRET: the Django secret (not sure whether we actually need this)
  • DEBIRDIFY_CALLBACK_URL: the callback URL to be used by the Twitter auth
  • DEBIRDIFY_DEBUG (0 or 1): whether Django should run in debug mode (absolutely switch this off for production use)

In Apache, you can, for example, set them using SetVar in your webserver configuration. For Nginx, you can, for example, set them by using uWSGI, adding the environment variables in an uWSGI init file.

Running

Set up a webserver and WSGI, pointing to debirdify/wsgi.

Caveats

The current implementation only looks at the first 1000 followed accounts returned by Twitter. This could easily be extended to more, although Twitter does apply some fairly harsh rate limiting.

The API requests are redone every time the page is refreshed. The rate limiting also means that if users refresh the site too often or make more than a few requests, they will get an error message about rate limiting. I don't think this can be fixed without caching results on the server, which is something I actively decided not to do for privacy reasons.

Also note that I am not very well-versed in Python and wrote this code in just a few hours without ever having written any Django before (or any other web applications in the past 10 years), so I am sure there are some aspects that are not great and not according to established best practices.

Note that if you use this software you will have to get a Twitter developer account, create your own app, get your own credentials etc. and are responsible for checking that the use of this software complies with Twitter's guidelines.

Contact

If you have any questions or suggestions, contact @[email protected]. Pull requests are welcome, but I may be too busy to put too much work into this project. Forks/other instances of this are also very welcome.

debirdify's People

Contributors

j-f1 avatar kappelmann avatar patrickhoefler avatar pruvisto avatar vaurora avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

debirdify's Issues

Add "Follow on Mastodon" buttons

It should be possible to circumvent the laborious CSV export/import business by simply adding a button next to each found account that causes you to follow them on Mastodon (or some other Fediverse account, although I am not sure if all of them expose an API like that).

Doesn't handle redirects

I get this error:

We found the following Fediverse IDs in your Twitter profile:

[[email protected]](https://d.sb/@dan) โŒ
This account does not seem to exist.

https://d.sb/.well-known/webfinger?resource=acct:[email protected] works, but it redirects to https://toot.d.sb/.well-known/webfinger?resource=acct:[email protected]. This is a valid configuration - see the WEB_DOMAIN section of https://docs.joinmastodon.org/admin/config/

Your code needs to follow the redirect.

Support webfinger / alias domains

Currently debirdify seems to build links to profiles by just taking @A@B as https://B/@A. This doesn't work when the tag domain is different from the web interface domain, which is the case for my @[email protected] account.

The correct way to handle this seems to be webfinger, making a query like this:

https://treehouse.systems/.well-known/webfinger?resource=acct:[email protected]

This will return JSON with the correct profile URI, which can then be used to return the correct link.

Export all Twitter accounts you follow , regardless of mastodon links found

To preserve data for the future it would be great if you could export all accounts you follow with username & bio as well as links found in it as well.
Not everyone thinks about linking a mastodon account in their bio, some just leave.
Therefore a list with all accounts you follow would be great.

FYI: Twitter is modifying Mastodon urls in profiles

This is an FYI, something I thought you'd want to know. Feel free to just close it after you see it.

Every time I edit my profile on Twitter, it gets saved, but also modified (not what I submitted).

A random character is added to the end of my Mastodon url in my profile. So
Mstdn.ca/@Deborahh
becomes
Mstdn.ca/@deborahhw or @deborahhu

I assume it is so apps like debirdify will find invalid IDs. ๐Ÿค” now, how could that happen?

PS: I moved my url to the Location field, and it was not modified. Solved, for me.

"This account does not exist", but my account works on Fedifinder and Mastodon

On https://pruvisto.org/debirdify/, I get:

We found the following Fediverse IDs in your Twitter profile:

[email protected] โŒ
This account does not exist. Do you perhaps have a typo?

But it in fact works & is fine.

Twitter user: @DLitz
Mastodon account: @[email protected]

Note that the mastodon server associated with this ID is https://mastodon.dlitz.net, but I believe I have the correct aliases set up, and both Fedifinder and Mastodon itself seem to resolve my account just fine.

Feature request: also check lists

Would it be possible to fetch additional accounts from a user's saved lists? I use Tweetdeck, and read a large number of accounts by adding them to lists and displaying them in separate columns, without actually following them (because I don't want to clutter up my main feed). It would be helpful if these accounts could be added to the search.

Flow layout for lists

If a user has a large number of lists, the current layout becomes very bulky. A responsive layout that arranges them in a table would be preferable.

Allow to upload lists

It's great that we can search within lists #2 but sometimes I have a list of handles I'm interested in for various reasons (someone else's Twitter list I follow, other random lists, Wikidata items or other). It's tedious to move these to my own lists, so it would be great to be able to upload a CSV file or simple txt with a list of handles to check.

sends malformed WebFinger requests

I see in my log:

2a0a:4cc0:0:d8b:4803:cbff:fe28:a5a3 - - [01/Dec/2022:00:36:07 +0100] "GET /.well-known/host-meta HTTP/1.1" 200 5339 "-" "python-requests/2.28.1"
2a0a:4cc0:0:d8b:4803:cbff:fe28:a5a3 - - [01/Dec/2022:00:36:07 +0100] "HEAD /.well-known/[email protected] HTTP/1.1" 404 5069 "-" "python-requests/2.28.1"
2a0a:4cc0:0:d8b:4803:cbff:fe28:a5a3 - - [01/Dec/2022:00:36:11 +0100] "GET /.well-known/host-meta HTTP/1.1" 200 5339 "-" "python-requests/2.28.1"
2a0a:4cc0:0:d8b:4803:cbff:fe28:a5a3 - - [01/Dec/2022:00:36:11 +0100] "HEAD /.well-known/[email protected] HTTP/1.1" 404 5069 "-" "python-requests/2.28.1"

But this is wrong: the resource query parameter must have an URI as its argument, in this case resource=acct:[email protected].

This is specified by RFC 7033 section 4.5.

Make different sections for different instances collapsible

So, the people I follow are spread out between a bunch of different instances:
Screen Shot 2022-11-12 at 20 34 17
This makes scrolling through all the different lists of instances take a really long time. What I'm looking for is like a little triangle that I could click in each of the instance section headers to collapse that list so that I can skip scrolling past it once I'm done with it. By "instance section headers" I mean the parts like this:
Screen Shot 2022-11-12 at 20 34 43

Import following list from account archive

After twitter's recent change in ownership, I, like many people, downloaded my account archive and deactivated the account.

Many more people have joined or switched since, but it has now been more than 30 days, so my twitter account has been purged.

Would it be possible to allow for import of follower and following lists from the follower.js and following.js json blobs provided in the account archive?

Add ability to set host instance

Following accounts on a separate instance from yours is a multi step process and can be slow.

However, it's easy when you open that account on your own instance, e.g. if logged in to mastodon.social, https://mastodon.social/@[email protected] is easier than opening hachyderm.io/@eKuber and following from there.

There is a bulk import feature, however when using Debirdify I found myself wanting to check individual accounts to see how active they were. Ideally, those links are on my own instance, so i can quickly follow if i need. Could be a simple text box i set at the top to change all the links

Thanks for making this, it was very useful! And I appreciate making it easy to log out and revoke permissions.

Error authorizing Twitter app - no request token

I get this error when trying to authorize the app for my Twitter account:

There is no request token for this page. That's the special key we need from applications asking to use your Twitter account. Please go back to the site or application that sent you here and try again; it was probably just a mistake.

Hide/mark accounts that you already follow on Mastodon

When querying the people you follow on Twitter, that follow you on Twitter, that you blocked on Twitter etc. it would be good to have an option to mark or even hide those that you already follow on Mastodon and exclude them from the generated CSV.

This necessitates interaction with the Mastodon API.

Mastodon account server migration is not detected

Issue:
When a Mastodon account is migrated to a new server, the old (original) account is detected and listed in the CSV.

Expected:
The new account is listed instead of the old account (or both?).
That way, when you follow someone, you are automatically following the active account instead of the old account.

For example [email protected] has moved to @[email protected]

Kudos:
This tool is awesome. A great help for everyone migrating to Mastodon. ๐Ÿ‘

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.