evoluhq / evolu Goto Github PK

https://blog.testproject.io/2021/02/01/using-github-actions-to-run-automated-tests/

So it's easier to reuse.

A web for Evolu development. Docs, showcases, etc. Use some Tailwind starters like https://github.com/timlrx/tailwind-nextjs-starter-blog or similar.

Because of the license and because OPFS and LocalStorage are good enough.

Vitest with Playwright for all Evolu use-cases. Test network traffic.

The current config silences SQLite because it's verbose.

self.sqlite3ApiConfig = {
  debug: constVoid,
  log: constVoid,
  warn: constVoid,
  error: constVoid,
};

e.g.

import { getError, subscribeError } from "evolu"

useEffect(() => {
  const notifyOnError = () => {
    console.error(getError())
  }

  return subscribeError(notifyOnError)
}, [])

What is missing in the readme? I think I should write more about CRDT, links are not enough. Should I?

• a helper for full text search

Run prepublishOnly on PRs

Switch to upcoming fp-ts 3 with better ESM support for Node.js.

The API for rows history.

• more tables and joins
• an example of how to filter nullables on UI
• custom string and number branded types
• form validation
• leverage patched Evolu state to prevent rerendering

• https://github.com/changesets/changesets
• https://pnpm.io/using-changesets
• https://github.com/vercel/turborepo/tree/main/examples/design-system
• https://www.robinwieruch.de/javascript-monorepos/

Waiting for the next https://github.com/evanw/esbuild major version with breaking changes for Web Workers.

The app has to be able to import data created from a newer version because forcing users to update is bad UX. Evolu should detect a missing column error, alter the table, and then repeat the import of the affected row.

Really interested in evolu so was trying out the example app on Safari on Mac and iPhone. Works fine on Safari on Mac but crashes on iOS safari (iPhone 8 - iOS 15.1)

Here is a screenshot of the crash - probably mobile Safari does not support one of the browser APIs used by evolu.

The underlying question I was trying to figure out is whether evolu can synchronise data when one of the peers is offline. Does the server store a complete copy of the data or does it delete the data once it has been relayed to the other device?

Nostr has a similar architecture. We should investigate whether we can reuse something.

And generate docs from that. Consider https://github.com/gcanti/docs-ts

• https://pnpm.io/using-changesets
• https://github.com/changesets/action (enable Github release)
• https://github.com/apps/changeset-bot

It was a workaround for the non-ESM build.

Blocked by SQLite, but hopefully, I will convince them.

For Automerge and the other binary-based libs within Evolu.

https://github.com/nostr-protocol/nips/blob/master/07.md

• Use Trezor to store mnemonic
• Super secret feature (data visible only on Trezor device)

The sync worker should be created from the DB worker. It's finally supported in Safari 160 preview. Now we have to wait several months, at least, maybe even more, until it will be generally available.

https://wpt.fyi/results/workers/nested_worker.worker.html?label=master&label=experimental&aligned&q=nested%20workers&view=subtest

stackblitz or codesandbox?

Something like

• https://github.com/beenotung/fix-esm-import-path](https://github.com/beenotung/fix-esm-import-path
• https://github.com/koskimas/kysely/blob/master/scripts/check-esm-imports.js

The idea is similar to https://remotestorage.io.

Use the same data across different apps. Create a to-do list in one app, and track the time on your tasks in another one. Say goodbye to app-specific data silos.

Evolu app is any app using Evolu lib. To protect Evolu Mnemonic, #118 must be implemented.

Because any Evolu app using a mnemonic has access to all owner's data, Evolu apps must be audited for fetch, etc. usage.

Let's discuss this.

https://github.com/acdlite/rfcs/blob/first-class-promises/text/0000-first-class-support-for-promises.md

Replace reset/show/recover mnemonic with "Connect with FIDO2".

Via hmac-secret. It should be what Evolu needs.

has is a very useful helper and should be integrated into the existing Evolu API.

All row columns except for id are nullable by default. A column can be null for two reasons. It's not loaded yet, or it's obsolete. That's why the has helper is so useful, and that's why it belongs to the application itself.

I'm not sure whether it can be refactored into the createHooks, but I'm pretty sure it can be refactored into useQuery. And with memorization, of course.

Some nullable columns are nullable for a reason, while others are just not yet loaded. The best place is createHooks, I believe.

Rows are actually not deleted. They are only marked as deleted. But the current API is not ideal because practically all queries have to be filtered. A proposal:

• replace isDeleted column with internal/invisible __isDeleted
• store data in invisible tables with a suffix [tableName_all]
• create views for useQuery [tableName] and [tableName_deleted]
• add 'remove' function to useMutation

With such a change, developers wouldn't have to filter rows everywhere.

Optimize for row records. There is an interesting PR chbrown/rfc6902#88. Another idea is to leverage https://github.com/google/diff-match-patch. It's for text, but we can map an array of rows to their hashes written in the line.

Sqlite for React Native and https://github.com/craftzdog/react-native-aes-gcm-crypto or similar for symmetric encryption.

We are using a fork of https://github.com/bitcoinjs/bip39, because the original library is not tree-shakeable.

bitcoinjs/bip39#169 (comment)

We have to port validateMnemonic implementation as well.

https://medium.com/@JasonWyatt/squeezing-performance-from-sqlite-indexes-indexes-c4e175f3c346

https://www.sqlitetutorial.net/sqlite-index

All row columns except for id are nullable by default. A column can be null for two reasons. It's not loaded yet, or it's obsolete. That's why the has helper is so useful, and that's why it belongs to the application itself.

TODO: Add an example to README.md.

const nodes = pipe(
  useQuery((db) =>
    db
      .selectFrom("node")
      .select(["id", "md"])
      .where("isDeleted", "is not", model.cast(true))
      .where("id", "in", ids)
  ),
  (a) => ({ ...a, rows: a.rows.filter(has(["md"])) })
);

It solves ESM issue with required .js. effect-ts is using it.

There is no reason for the Evolu server to be part of the evolu library.

• remove mutation code from kysely somehow
• replace some dependencies with own code tailored for Evolu

• Merkle Tree, something like https://github.com/hoytech/quadrable
• Remove mutation code from kysely somehow, pnpm patch?
• Garbage collection for both RowsCache stores

Another idea is to hold the clock, Merkle tree, and table definitions in tab memory to skip reads, but it's unfortunately impossible because inactive tabs would have to be updated in the background, and such an async operation can't be reliable, AFAIK. We can already do that because we have one DB instance per many tabs.

For m:n relations, consider https://github.com/ccorcos/tuple-database#Motivation as an additional abstraction. SQL has to be always supported because that's what people use and know. The core idea of a tuple database is to build indexes by queries; I think Evolu can do that, too, because it does not expose mutation SQLite API.

• https://maximeblanc.fr/blog/sq-lite-json-and-denormalization/
• ccorcos/tuple-database#11 (comment)

IMHO unnecessary

• ULID (https://github.com/asg017/sqlite-ulid, https://github.com/ulid/spec)

• Append
• Prepend
• Edit
• RemoveAt

These operations can still be detected quickly. Remember, the perfect diff is impossible to compute quickly.

https://github.com/aermin/web-locks

Related #106