cycraft / magnetar Goto Github PK

maybe it's already compatible out of the box? should add some tests to make sure.

this ticket is a reminder to revisit this. still gotta

• think of the exact goals we want to achieve with an "pagination syntax"
• think of the limitations that come from Firestore

--

the fetchMore concept explained here: #91
is great for endless scroll pages

Is there a method to force all of the batched updates out to Firestore?

In my use case, the user will slowly be updating the data of a document that, for the most part, only needs to be pushed out to Firestore periodically. I found syncDebounceMs that appears to let me delay the batch sync for an arbitrary amount of time. I could set this to a suitably large value (likely measured at a minute or more) to be sure that several updates get merged together while still slowly syncing to Firestore.

However, there are a few cases where I need to be certain that the data has been pushed out to Firestore. For example, when an operation will be triggered on the data by cloud functions, etc.

I know when this will be done, so it would be easy to make a call to a sync function. I could then wait for it to complete before triggering the remote operation.

This would allow me to keep the number of writes to Firestore down while the user is active while still achieving consistency when required.

Thanks.

I create a Magnetar instance with a syncDebounceMs of 10000 specified for the Firestore plugin.

Then
docRef.merge({ data: 'a' })

Wait 5s and then
docRef.merge({ data: 'b' }, { syncDebounceMs: 0 })

Wait 6s more and then
docRef.merge({ data: 'c' }, { syncDebounceMs: 0 })

expected: If syncDebounceMs of 0 was to result in any pending changes being flushed out, then I would expect the first two merges to be applied together at about 5s from the first call to merge. The third merge should then occur at about 11s from the first merge call.

actual: Instead, what I am seeing is all three merges being applied together at about 21s after the first call to merge. It would appear that the debounce timer is simply being reset to the syncDebounceMs value of the first merge instead of picking up the new debounce value.

@wifisher

if you have some bandwidth to try and write an automated test based on your findings above, it would greatly help me to come in and fix the issue!

The automated test can be added in this file, and you can simply copy paste parts of other tests and then tweak:
https://github.com/cycraft/magnetar/blob/production/packages/plugin-firestore/test/internal/multipleWrites.ts

if not let's keep this issue open for a while until I find time to do it.

anyone using magnetar with firebase plugin will probably use firebase auth as well, and how to best implement redirection to a login page in a good way seems to be something that a lot of new devs struggle with.

some example on this in the FAQ would be nice!

Why does the local store need a random id? Does it have to be a Firestore doc id? If not I wonder if it would be better for Magnetar to generate an ID itself.

I would like to see the license information updated for this software.

I can't truly consider using it without first knowing the licensing details.

Thanks.

open questions:

• Should/can the example be interactive and live in an example section?
• Should the example be a different kind of mini file browser where you see file name list at the right and the file content on the left?
• Should the example include user auth (because in eg. a todo list this would be a requirement)
• Should I link to repo links to Vue 2 / Vue 3 / Vue3+Firestore examples?
• Should I add example repo with Firestore+Vue2?

I think at this stage with the library you’d be best to target Vue and make the docs examples Vue specific. Or at least keep the plain js docs and add Vue specific docs where it’s useful. You’ll save yourself a lot of questions and issues down the road if you spell out how to setup and tear down with Vue 2 and 3

We should have documentation on each official plugin, with:

1. how to do basic setup
2. what all the config & options are
3. a link to an example repo with magnetar and that plugin being used
4. some extra explanation on if this plugin provides extras or not
5. some extra explanation on what this plugin does under the hood

should think of a short hand for this

dbAuthUser().data.clear()
dbAuthUser()
  .openStreams.values()
  .forEach((close) => close())

I wondered where to call stream(), and I also wondered if we need to close the stream in unmounted?

• limit max width of page content to 900ish pixels
• #14
• #16
• CyCraft/planetar#42
• CyCraft/planetar#41
• CyCraft/planetar#40
• CyCraft/planetar#39

Hi @mesqueeb,

First of all, thanks a lot for the great work you have done here ! It is much appreciated 🙇‍♂️

I am trying to configure magnetar in a project I recently started with Vue 3. However, I am having trouble to make it work with TypeScript. To simplify my description, I am going to use the example provided in your documentation.

When I try to transpile this snippet :

export type Pokemon = { name: string, nickName?: string, id: string, level: number }

export function pokemonDefaults(payload: Partial<Pokemon>): Pokemon {
  const defaults: Pokemon = { name: '', nickName: '', id: '', level: 0 }

  return { ...defaults, ...payload }
}

// here is how you inject the Type: collection<Pokemon>
export const pokedexModule = magnetar.collection<Pokemon>('pokedex', {
  modifyPayloadOn: { insert: pokemonDefaults },
  modifyReadResponseOn: { added: pokemonDefaults },
})

I get this error:

ERROR in src/services/pokemon.ts:18:27
TS2322: Type '(payload: Partial<Pokemon>) => Pokemon' is not assignable to type 'OnAddedFn'.
  Types of parameters 'payload' and 'docData' are incompatible.
    Type 'Record<string, any> | undefined' is not assignable to type 'Partial<Pokemon>'.
      Type 'undefined' is not assignable to type 'Partial<Pokemon>'.
    16 | export const pokedexModule = magnetar.collection<Pokemon>('pokedex', {
    17 |   modifyPayloadOn: { insert: pokemonDefaults },
  > 18 |   modifyReadResponseOn: { added: pokemonDefaults },
       |                           ^^^^^
    19 | })
    20 |

Do you have an idea what could have gone wrong?

Here is my config

node 16.13.1
magnetar 0.5.4
typescript 4.6.2

Hi very congratulations for all your excellent work... im starting to use data in all this modern js world in a nice moment... and i was looking into vuex then vuex orm, then i found you, then i found magnetar... what you suggest to a production thing, magnetar as it is today, or stick with easy firestore at moment?

ah yes the topic.... i seen your SO post about the pricing, but i´ve also seen another one

https://www.javaer101.com/fr/article/35020648.html

im really confuse...

THANKS FOR EVERYTHING!!!!

required setup:

const profileCollection = magnetar.collection('profiles').where('show', '==', true).where('archived', '==', false)

const query = profileCollection
  .orderBy('lastActiveOrSomething')
  .limit(50)

const lastProfile  = computed(() => {
  return [...profileCollection.orderBy('lastActiveOrSomething').data.values()].pop()
})

then either:

const profiles = ref([])

const fetchMore = async () => {
  const newProfiles = await query.startAfter(lastProfile.value).fetch()
  // push to profiles
  profiles.value.push(...newProfiles)
}

or

const fetchMore = async () => {
  await query.startAfter(lastProfile.value).fetch()
}

and

<template>
  {{ [...profiles.data.values()] }}
</template>

I have a documentation that is currently Quasar but I wanna rebuild it at Vitepress 1.0 documentation website.

• Vitepress docs: https://vitepress.vuejs.org/

• My current Quasar docs: https://magnetar.cycraft.co/docs

• set up vitepress project from scratch, following the Vitepress documentation on how to create a docs website with that framework

• Just port over the content which is only text! I don't care about keeping the "looks" that I had on the quasar website.

Splitting this up into smaller steps would look like:
2a) try to set up a vitepress docs website and get comfortable with how it works in general
2b) try to set one up inside the magnetar mono-repo
2c) copy paste all the docs that I wrote so far from the source code without caring about the "looks", it's almost 100% just markdown content anyway
2d) then we can look at the result together and decide if we wanna tweak the "looks" of the vitepress documentation

somehow when I do npm run build inside packages/test-utils it will nest the built types inside the types folder.

It didn't do this before, and I tried everything and cannot find how to prevent it.

https://github.com/cycraft/magnetar/blob/a881cd818e893a244e586499cc8502437fefd450/packages/utils/src/logger.ts#L1-L4

Hi,

I noticed an odd behavior from magnetar in a edge case. It might not be a bug but then I think it could be highlighted in the documentation 🙂

Use case

I have a method to stream a document that looks like this :

const gamesModule = magnetar.collection<Game>(GAMES_COLLECTION, {
  modifyPayloadOn: { insert: gamesDefaults },
  modifyReadResponseOn: { added: gamesDefaults },
})
export const getGame = (id: string) => {
  const gameModule = gamesModule.doc(id);
  gameModule.stream().catch((error) => {
    console.error(`Game ${id} stream was closed due to an error`, error);
  });
  return gameModule.data;
}

Expected result

Magnetar streams the document with the id id. If id is undefined, nothing happens.

Actual result

If, for some reasons, id is undefined (e.g. when the id is a reactive variable and is not yet set), magnetar streams the whole collection. This behavior led me to massive amount of document streams/fetches on my firestore DB (I reached the 50k free quota in 2h).

Workaround

My current solution was to add a check on the value of the id parameter.

export const getGame = (id: string) => {
  if(!id) return undefined;
  const gameModule = gamesModule.doc(id);
  gameModule.stream().catch((error) => {
    console.error(`Game ${id} stream was closed due to an error`, error);
  })
  return gameModule.data;
}

Suggestion

I think it would be nice if magnetar could handle undefined document ids natively. Else, it would be good to document this behavior.

Again, I am very grateful for the work you have done here! Thanks to magnetar, I was able to write a soon-to-be-published opensource full-stack pwa in ~3 weeks. This app is going to help my non-profit organisation to manage the games and the players of a real-life game that gathers +1200 boyscouts in Belgium next weekend 😉

Hi there Luca, thank you for your Magnetar, it is a wonderful state management for my firestore project.

Recently notice some problem/bug (not sure about it)
If I would like to fetch some collection with time/date query,
-> magnetar.collection(doc).where('date', '>', new Date('1 May 2021')).fetch()

it will return nothing even the document date is in range of the query.

Thank you

I'm getting "FirebaseError: Expected first argument to collection() to be a CollectionReference, a DocumentReference or FirebaseFirestore" when I try to initiate a stream or fetch.

I'm using Firebase 9, with the fancy imports. Basically copy and pasted the mangetar setup from the docs. FIrestore is working fine, I can access it and pull docs down.

My setup is

// ---------------------------------------------
// 0. Initialise firebase
// ---------------------------------------------
import Vue from 'vue';
import { initializeApp } from 'firebase/app';
import { getFirestore, collection, doc } from 'firebase/firestore';
// const firebaseApp = initializeApp({ /* pass your config... */ })

const config = {
//
};

const firebaseApp = initializeApp(config);
const db = getFirestore();
/**
 * A helper function that generates a random Firestore ID
 */
function generateRandomId() {
  return doc(collection(db, 'random')).id;
}

// ---------------------------------------------
// 1. the plugin firestore for remote data store
// ---------------------------------------------
import { CreatePlugin as PluginFirestore } from '@magnetarjs/plugin-firestore';

// create the remote store plugin instance & pass your `db`:
const remote = PluginFirestore({ db });

// ---------------------------------------
// 2. the plugin vue2 for local data store
// ---------------------------------------
import { CreatePlugin as PluginVue } from '@magnetarjs/plugin-vue2';

// create the local store plugin instance & pass your `generateRandomId`:
const local = PluginVue({ vueInstance: Vue, generateRandomId });

// -----------------------------------------------------
// 3. instantiate the Magnetar instance with the store plugins
// -----------------------------------------------------
import { Magnetar } from 'magnetar';
import { logger } from '@magnetarjs/utils';

export const magnetar = Magnetar({
  stores: { local, remote },
  localStoreName: 'local',
  executionOrder: {
    read: ['local', 'remote'],
    write: ['local', 'remote'],
    delete: ['local', 'remote']
  },
  on: { success: logger } // disable this on production builds
});

1. fetch 2 docs by ID at the same time, one that the user has no permission to, one that he does have permission to

• expected: we should still receive 1 doc in our state, even if the other fails
• currently: we do not get any docs

docs: add section on closing all streams and clearing data

currently local data is all saved together as long as the collection path is the same.

in some cases this is bad when a doc leaves your local storage because it doesn't adhere the where filters anymore but if you still want to keep it because it adheres other where filters you also use.

in this case we need to differentiate the local data not only on the collection path but also on the query (the where filters.)

I have to think of some flag or something for this. and add a section on it in the documentation as well. (obviously)

current architecture:

• local data is saved together for same collection path no matter the where filters
• each time .data is accessed the proxy checks the where filters and returns filtered local data for you

Similar to Vuex-Persist, it's super easy to do your own offline data caching via localStorage.
in this case, a super simple plugin, just add it to your magnetar setup, will take care of everything!

it will then work no matter which other local or remote stores you use.

not yet sure if I want this
wait for community requests

• code Vuex plugin for Vue 3 (Vuex 4.+)
• add dev-vuex-for-vue3 folder for manual testing
• See if Vuex 3 & 4 can be the same plugin?
  • Eg. the user passes vueVersion: 2 or 3 and then Vuex is only provided by peer dependency
  • Can a peer dependency be either Vuex 3 or Vuex 4?

import { Magnetar } from '@magnetarjs/core'
import { logger } from '@magnetarjs/utils'

export const magnetar = Magnetar({
  localStoreName: 'local',
  stores: { local, remote },
  executionOrder: {
    read: ['local', 'remote'],
    write: ['local', 'remote'],
    delete: ['remote', 'local'],
  },
  on: process.env.DEV ? { success: logger } : {},
})

Main Setup/Example: on: { success: logger }, // disable this on production builds. I think you need to add an example or some help on how to do this. (A check on process.env for example)

write a test that tests this part:

https://github.com/cycraft/magnetar/blob/9d7eb1efb2fb350e9cfa899adf7be8f4ff6ed3fd/packages/core/src/moduleActions/handleStreamPerStore.ts#L112-L118

steps:

1. open stream for a doc
2. edit a doc locally (core goes into 5 sec writeLock)
3. that doc is deleted from the server side during these 5 seconds
4. the doc gets and stays deleted from local module data after the 5 seconds

potential bug without the highlighted lines (hypothesis):

• it will delete the doc locally during the 5 seconds and is then brought back to life after the 5 seconds write lock

basically I need to write explanation for what's happening here:

test('insert: can mutate payload (config in module - action from doc)', async (t) => {
  function addSeen(payload: any) {
    if (!('seen' in payload)) return { ...payload, seen: true }
  }
  // get resolves once all stores have given a response with data
  const { magnetar } = createMagnetarInstance()
  const pokedexModule = magnetar.collection('pokedex', {
    modifyPayloadOn: { write: addSeen },
  })
  try {
    const resultWithoutSeen = await magnetar.collection('pokedex').doc('10').insert(pokedex(10))
    const resultWithSeen = await pokedexModule.doc('7').insert(pokedex(7))
    // the remote result not from pokedexModule SHOULD NOT HAVE the applied defaults
    t.deepEqual(resultWithoutSeen.data, { ...pokedex(10) })
    t.falsy('seen' in resultWithoutSeen)
    // the remote result SHOULD HAVE the applied defaults
    t.deepEqual(resultWithSeen.data, { ...pokedex(7), seen: true })
  } catch (error) {
    t.fail(error)
  }
  // the local store result not from pokedexModule SHOULD NOT HAVE the applied defaults
  t.deepEqual(pokedexModule.data.get('7'), { ...pokedex(7), seen: true })
  // the local store SHOULD HAVE the applied defaults
  t.deepEqual(pokedexModule.data.get('10'), { ...pokedex(10) })
})

when the developer sets:

await firestore.enablePersistence({
    synchronizeTabs: true
 })

all chrome tabs sync the local data via the indexedDB local data, so preventing updating the local store on "local updates" completely, means the other tabs will not receive those changes.

I need an automated test that confirms this.......... o_O

perhaps Cypress and a test site will be needed, but only if Cypress can do multiple tabs. (2 Cypress instances would not be the correct tab)

if not possible with Cypress I'm at a loss how to create an automated test for this.

• code Vuex plugin for Vue 2 (Vuex 3.+)
• add dev-vuex folder for manual testing
• write data backup and restore system
• write the possibility for the users to add the actions that link to the library (in a way that's explicit)
• add documentation on how to setup & use plugin-vuex

in order to add the actions, the dev could do something like this. And they could then choose between modern and vuex-easy-firestore 1.0 syntax by importing one action set or the other!

need to create a method we can use on the magnetar root

eg.
this collection for companies:

• adds default fields on inserts
• adds default fields for updates & removes undefined props

export const dbCompanies = magnetar.collection('companies', {
  modifyPayloadOn: {
    insert: (payload) => addCreatedAtById(defaultsCompany(payload)),
    write: (payload) => removeProp(addUpdatedAtBy(payload), undefined),
  },
  modifyReadResponseOn: { added: defaultsCompany },
})

but in case a doc instance is used to insert or update content, these fns won't be run...

dbCompanies.doc('some-id').merge({ newField: undefined })
// or
dbCompanies.doc('some-id').insert({ name: 'howdy Co.,Ltd.' })

but there's no easy way with the current syntax to use the same modifyPayloadOn fns from the collection.

• The question is: should we by default always use a collection's modifyPayloadOn functions if the parent collection has them?