Export all chats about keybase-export HOT 11 CLOSED

Hi! The strings inside the chats array are in the format that Keybase uses in its API. For example, a chat between users alice and bob would be alice,bob.

To get all your conversations, execute:

keybase chat api -p -m '{"method": "list"}'

The string that you should enter into the config to export a chat is located in the field name of the channel objects.

If you need to export a channel of a group (to export a group, list all channels of the group), then topic_name also needs to be used. For example, with name as my_team and topic_name as general you should write my_team#general in the keybase-export config.

There is no option in keybase-export to specify that you just need to export all chats, but perhaps you can write all of them manually.

We can conveniently get all the needed information using jq:

keybase chat api -p -m '{"method": "list"}' | jq '.result.conversations[].channel | {name,topic_name}'

Then, to get all the names in the needed format, this seems to work (then we replace newlines with , to use it in the JSON arrays):

keybase chat api -p -m '{"method": "list"}' | jq '.result.conversations[].channel | {name,topic_name} | if (.topic_name != null) then .name + "#" + .topic_name else .name end' | tr '\n' ','

This should output something like this:

"alice,bob","team#general","team#offtopic","alice","alice,william",

And you use it in the config as "chats": ["alice,bob","team#general","team#offtopic","alice","alice,william"],.

keybase-export will scrape the chats sequentially. However, note that, it will still store all of them in the same file. It shouldn't be too difficult to split it it in a chat per file if you want.

For the other parts of the config, simply copy config.example.json, replace yourusername and enter your paper key with the real values. The result will be in the export.jsonl file.

from keybase-export.

Hi -

A few questions, please:

What platform is your app written in?

Can the app export attachments (e.g., images, files, etc.)?

Can other events (e.g., non-chats) such as, users added, reactions, etc., be exported as well?

Thank you,
Kevin

from keybase-export.

What platform is your app written in?

Node.js. Is that what you mean?

Can the app export attachments (e.g., images, files, etc.)?

It cannot, there's a checklist in https://github.com/Bannerets/keybase-export#features, "Attachment downloading" is not yet implemented

Can other events (e.g., non-chats) such as, users added, reactions, etc., be exported as well?

Those events are not exported. It's not difficult to add, but I wasn't adding reactions to not clutter the export file

from keybase-export.

Thank you, Bannerets, for the prompt response.

Would you be willing to add the capability to include "reactions," "users added," and other non user chat-type information etc., to the export? If so, how long do you think it would take to implement into code? Thanks.

from keybase-export.

I can add reactions when I will have time. Though I'm not sure if there's a "users added" message at all. I don't remember seeing such a message in group chats, for example.

from keybase-export.

Here is an example from my client's Keybase for "adding users" to a team and for changing the retention policy (very important in litigation) of messages. If you cannot see the image, I can send it to you via email. Thanks!

from keybase-export.

Yes, I see. I can add those as well. I think I can work on this tomorrow (European time). Though I wonder if the format in which messages are exported will be helpful for you. It is generally machine-readable. The exported file looks like this:

{"text": "hello","id":1,"sent_at":1647139702,"sent_at_ms":1647139702720,"sender_username":"alice","sender_uid":"59d326cd5ab2a825c6e91602b4f29354","device_id":"f36b81fdd122164ce5bf5bb31a4d5bfe","device_name":"macbook","channel_name":"alice,bob"}
{"text": "hi!","id":2,"sent_at":1647139712,"sent_at_ms":1647139712415,"sender_username":"bob","sender_uid":"7d5e4b0d6ba8cd20bd0adbcfd051d490","device_id":"bca556d87c182078fbd8c00179e4b080","device_name":"iphone","channel_name":"alice,bob"}
.... other messages ...

from keybase-export.

BTW - I followed your instructions (above) and was able to export the chats from all Teams from my client's Keybase instance. Thank you, very much, for your assistance.

from keybase-export.

Super!!!

My Analytics team said they are able to use the current format of the JSON export file.

Attachment downloading: I wouldn't worry about adding that feature unless you think you can do it fairly easily and quickly. I am not seeing that many attachments in my client's Keybase instance and can just download them manually.

from keybase-export.

I added support of reactions and system messages to keybase-export :).

To update, if you already cloned the repository, just run git pull and npm run build.

I made changes in the config, it now looks as follow (though all messageTypes should be true by default even if they are not specified in the config):

{
  "chats": ["alice,bob", "alice", "alice,emma", "family#general"],
  "init": {
    "type": "init",
    "username": "yourusername",
    "paperkey": "enter your paper key"
  },
  "watcher": {
    "enabled": false,
    "timeout": 20
  },
  "attachments": {
    "addStub": true
  },
  "messageTypes": {
    "reactions": true,
    "reactionMessages": true,
    "systemMessages": true,
    "headline": true
  },
  "jsonl": {
    "enabled": true,
    "file": "export.jsonl"
  },
  "elasticsearch": {
    "enabled": false,
    "indexPattern": "keybase_$channelname$",
    "config": {
      "host": "localhost:9200",
      "log": "info"
    }
  },
  "eol": "\n"
}

from keybase-export.

Well done! I re-exported all of the Teams and the Reactions and System Messages were also included. Much appreciated.

from keybase-export.