cozemble / monorepo Goto Github PK

Options:

• Simple state what the underlying json type is: string, number, boolean, object, array
• Make the event to create a new property instance automatic
• Make the configuring of properties (i.e. min value, max value, regex) easier. Automatic for primitive values should be possible. Adding a complex regex also should be possible. Find a way so that the property developer is enabled.
• Provide a fixture to run the property configurer, editor and view in

Description:

Use of special paths like $lib/ is throwing errors with eslint-plugin-import in Svelte projects, both in *.svelte and *.ts files.

Related:

• https://github.com/sveltejs/eslint-plugin-svelte3/blob/master/OTHER_PLUGINS.md
• sveltejs/kit#1560

Tried:

• Using eslint-import-resolver-custom-alias:
  Requires setting up ESLint in every Svelte project directory. Not ideal.

• Overriding *.ts files:
  Affects every file in the monorepo

• Overriding with pattern:
  There is no simple way of selecting all the Svelte project files.

Suggested:

Simply removing eslint-plugin-import seems to be the easiest solution. It will cost securing the imports but I don't think it's a bigger problem than the current issue.

Seems postgres does not do this by default, an can be a performance issue at scale

Rather than adding number and date next, maybe do some of the more interesting cell types. Number and date are very similar to string, so we will learn more (possibly) by doing the ones that are most different. Some examples:

• Image - upload an image, have it stored, see a thumbnail in the cell, click to see the full thing.
• OCR of image, given an image cell, do OCR of it and present the text
• String template - given other (text) cells, run a given template and present the resulting text
• ChatGPT - given a text cell, issue its contents to ChatGPT, and present the response

If these cell types were available, it would be possible to quickly assemble an expense app:

• Drop image of receipt into Image call
• OCR of the receipt happens in the next cell
• ChapGPT prompt to find the price from the receipt text gets created in the next cell, using a string template
• ChatGPT executes the prompt and presents the result
• After that, who knows

It's unlikely that this will be 100% accurate due to the imprecision of OCR and the hallucinations ChapGPT sometimes has, but it would be a great prove point for the value of cozemble as an "app fabric".

God forgive me, but I am rolling my own Auth here and guaranteeing future disaster
The need I have that I think is not covered by supabase is that tenants will require sub user pools
for their customers etc
Raising a ticket to loop at proper IDP like Auth0 or Keycloak before really going to prod.

The paginated data editor right now is a UI only component, it does not persist changes to the database.

Current best idea of how to approach this is similar to the model editor - wrap a DataRecord in an EventSourcedDataRecord, event source the changes in the UI, apply them to the database on save.

Attempt to keep the saving method abstracted away, because while the core of cozemble is saving to the associated postgres, we need to keep the door open to other saving strategics, like Airtable, Google Sheets, Rest APIs and who knows what else.

It will be tricky, possibly, to neatly map the mutation events to coherent GQL or SQL, so lets see.

If I have a Booking that references-one Customer, and I am creating a new Booking or editing an existing Booking, when it comes to associating it (the Booking) to a customer, I want to be able to pick from existing customers, of course, but this feature is about being able to create a new Customer right there.

I guess the UI can slide over to a new Customer view, and on save, the Customer gets saved, the Booking view slides back into view, and the Customer relationship is filled in

Support deletion of a record and cascade the deletion along all its has-* relationships

What to do with any references-* relationships it might be in?

If I have a Booking referencing a Customer, and I delete the Customer, what should happen to the Booking?

If we support soft deletion, the reference remains intact. If we do hard deletes, we need to loop around all referenced data and null, it probably.

Soft deletion "feels" better, but it does bring up GDPR issues around the right to be forgotten etc. There are ways to manage that like tagging PII fields and masking them on deletion, so still feels like the best approach.

The desire is a model_record sequence number. In other words, while the record table might contain customers for one tenant and invoices for another tenant, the first tenant will want their customer records numbered from 1, and the second tenant will want their invoices numbered from 1. So a sequence on the record table will not work.

So this task is about finding a way to make that happen.

One sequence for each aggregate root model is possible I suppose. But in docsndata maintaining a sequence for the record table allowed me to , for each model_id, find the latest record and get its model_record_id, and then assign that + 1 to the record being created.

If #50 is done, this will be model_record_environment_count

• Input that takes only integers (including during pasting)
• div that takes only characters matching regex (including pasted)
• Theme selector in frontend/main-app
• Publishing of selected workspaces to npm
• dynamic loading of plugins
• secrets management UI
• turborepo cache compatible running of cypress tests
• Fix the vite 4.2 issue
• Get rid of DataRecordsTable in favour of DataRecordsTableInContext, also get rid of RecordsContext

• new repo in cozemble organisation
• svelte library contents
• github actions to push to npm.org
• given jsonschemas for customer (first name , last name, email, phone number) and booking (link to a customer, booking date)
• data entry of new customer
• data entry for new booking
• listing of customers
• listing of bookings
• data editing of an existing customer and/or booking
• delete of existing customer and/or booking

Assumptions:

• we will be able to convert cozemble models to jsonschema (maybe with extentions)
• The data editor that works from json schema (#6 ) will mesh with this approach.

Lets try!

Given an invoice with three line items, indices are 0, 1 and 2
If I delete the first one, indices will be 0 and 1
And add another, indices will be 0, 1 and 2 again
But any index based reference to 0, could have meant the deleted record or the former second record, and now first record

A safer way to address records in an array is by id

Instead of this in DataRecord:

values: { [key: string]: any }

Make the value a proper DataRecordValue type. This will open the door to extension via facets or similar. An example being colour.

Another example is from the docsndata codebase. Formulas could be disabled so a user could override the value. Whether the value was formula derived or user provided was modelled as metadata, but in a weird way because there was no clean way for a value to have metadata.

The idea of supporting N backend types is nice, but If we doubled down on supabase, that might increase velocity and focus.

Auth, storage, database, graphql, permissions, rest etc

And it's a good bet for the long run.

Spike it out

• fire up supabase free
• migrate the schema into existence
• test the graphql api
• wire in the data editor to issue mutations back to supabase

Currently events have a hard coded value for test user in there - record create events and record deleted events etc

Once authentication is wired in, we will need to revisit these insertion points and view in the real user id

• Anyone can sign up and create a tenant, of which they become the owner. So this is an insert into tenant and user and tenant_member
• Only tenant members can CRUD tenant models and model events
• Only tenant members can CRUD tenant records

Implement as RLS

Given that a user has supplied their supabase Postgres credentials as secrets, re-work the existing migration code to migrate that database.

Is gql api tracking automatic in supabase? If so, then we're done. If on the other hand there is some api call to make to register tables in the api, then this work includes that task.

The acceptance criteria are:

• define a model in the model editor
• apply to the database
• load up the gql console in supabase
• the api reflected there should map to the model in cozemble

Will need to cope with has-* relationships (which exist at the time of writing) and references-* relationships (which are in ticket #3 )

We have paginated editor that shows a list of records and enables edit of one record at a time.

But we need the view the opens when a use clicks the Open button for a record.

This will show the edit view, along with references-* relationships

Events were settled on to enable sql schema migrations, but they have been binned.

Events are still nice for history and undo etc.

But can we keep the best of both worlds by either:

1. Providing the property as a Writable to the Configurer, and subscribing to changes
2. Using something like valtio to achieve the same

Given that aurora is set up
And the user has made some models (lets say Customer and Address)
And those models are being stored local storage in the browser

When a customer clicks "Apply to database" for the first time:

• create a schema for that user in aurora
• create the tables matching the models using knex (i.e something like Customer and Address)

Now the data editor has to use that database instead of local storage. What API will it use? Because we only have a Postgres database

Option 1

Given that all data changes in the data editors are made using events, send the events to https://api.cozemble.com/:teamId/:modelId/:recordId

That endpoint can load all the models for the team, and the record, and "re-play" the events into the record, save it, and return the new record

Option 2

Make our our GraphQL endpoint that reflects the "shape" of the model. i.e if we have a Customer with a has-one Address, we can make this GraphQL api:

const schema = buildSchema(`
  type Address {
    id: ID!
    street: String!
    city: String!
    state: String!
    zip: String!
    customerId: ID!
  }

  type Customer {
    id: ID!
    name: String!
    email: String!
    address: Address
  }

  input AddressInput {
    street: String!
    city: String!
    state: String!
    zip: String!
  }

  input CustomerInput {
    name: String!
    email: String!
    address: AddressInput!
  }

  type Query {
    customer(id: ID!): Customer
  }

  type Mutation {
    createCustomer(input: CustomerInput!): Customer
  }
`);

And this would be hosted at https://api.cozemble.com/:teamId/graphql/v1.

But this would be dynamic, based on the models, not based on code generation. So if I have two teams, teamA and teamB, and teamA has models for Customer and Address, and teamB has models for Invoice and Order, when I hit this URL:

https://api.cozemble.com/teamA/graphql/v1, that GraphQL api will dynamically be about Customer and Address

And if I hit the url

https://api.cozemble.com/teamB/graphql/v1, that api will be about Invoice and Order

To implement this:

• Dynamically create a GraphQL schema from cozemble models
• Write one generic resolver that maps from mutations and queries, using models, to the correct INSERT, UPDATE statements
• Write a node server to expose this as an endpoint

Add the "Image" cell type, so people can upload photos from their phone, tablet or laptop.

Permit multiple images in the same cell.

Show a thumbnail for each, and a means to seeing the image at full size.

Support deletion of one or more of the images in the cell.

When I view a customer record in detail, I want to see tabs or similar to related records, like bookings and payments etc.

When I click on one of those tabs, I see the related bookings or payments.

And there is a button to add a booking or payment

Doing so will not require me to enter the customer for the booking or payment, because it is implied by the fact that this is happening in the view of this particular customer

Currently the gql generated by the data editor is intended for the Hasura gql dialect, that of insert_ and update_. The supabase one might be different, so this task is about supporting the supabase dialect of gql, and testing that the data editor works correctly.

The model editor UI is unstyled now. Get it into shape against the agreed design.

Currently we only have String, by choice. This task is to bring Number into existence.

issues to consider:

• sql migrations
• changing of type from string to number, and back again
• how to do that if data exists already, and can't be converted to Number from String
• number formatting
• number validation

Once the date type is added (or maybe a natural part of adding it?), I want to be able to filter a list of records by date

An example being: I have a list of Bookings for my Bike Shop, I want to be able to see what's coming in next Saturday.

To support synthetic testing, and testing of the software as a "dev" environment, add logic environments to the stack. This will mean at last the following:

• database tables (all of them)
• database functions
• endpoints
• application state
• signing in - which environment of tenant and users table

Data editor right now can create a new record and gql it into a database, but it has no load capability, so when it starts it fetches a page of records of the model it is viewing.

Once the 60 mins expire, we need to refresh the access token. get this done

It is currently completely unstyled, get it into shape relative to the design we agreed

https://stackoverflow.com/questions/75843346/why-is-sveltekit-vite-loading-commonjs-code-when-es6-is-available-and-type-mo?noredirect=1#comment133781366_75843346

To reproduce:

change vite dependency in all package.json files to 4.2.0
pnpm i
build all
run the backend
cd frontend/main-app
npm run dev
go to http://localhost:5173
login
when you are redirected to the main home page, you will see this error in the sveltekit log for the app (in the terminal):

08:24:50 [vite] Error when evaluating SSR module /src/lib/MainPanel.svelte: failed to import "@cozemble/model-assembled"

08:24:50 [vite] Error when evaluating SSR module /src/routes/tenants/[tenantId]/+page.svelte:

Internal server error: require() of ES Module /Users/mikehogan/repos/personal/cozemble/monorepo/model/string/ui/package/index.js from /Users/mikehogan/repos/personal/cozemble/monorepo/model/assembled/dist/cjs/index.js not supported.
Instead change the require of /Users/mikehogan/repos/personal/cozemble/monorepo/model/string/ui/package/index.js in /Users/mikehogan/repos/personal/cozemble/monorepo/model/assembled/dist/cjs/index.js to a dynamic import() which is available in all CommonJS modules.
at Object. (/Users/mikehogan/repos/personal/cozemble/monorepo/model/assembled/dist/cjs/index.js:4:27)
at async Promise.all (index 0)
at async nodeImport (file:///Users/mikehogan/repos/personal/cozemble/monorepo/node_modules/.pnpm/[email protected]/node_modules/vite/dist/node/chunks/dep-c167897e.js:54024:21)
at async eval (/src/lib/MainPanel.svelte:10:31)
at async instantiateModule (file:///Users/mikehogan/repos/personal/cozemble/monorepo/node_modules/.pnpm/[email protected]/node_modules/vite/dist/node/chunks/dep-c167897e.js:53981:9)

Instead of just app_public, we need to app_test and app_live. Also two knex migration tracking tables in the cozemble schema, one for test and the other for live.

Applying migrations happens in app_test, and promoting to live applies them to the app_live schema.

Keep this abstract and data driven. Some contexts might be best served with only live, others by test and live and others still maybe by dev, test and live.

Core idea is a chain of logic environments, possibly of length one, the last one is production

We have has-one and has-many relationships, which express containment relationships between parent and child models (think Invoice has-many Line Items, or Customer has-one address).

For the has-* relationships, if the parent is deleted, the children get deleted too. CASCADE DELETE in database terms.

What we need is similar, but without these delete semantics. For example, an Invoice would reference-one Customer. The Customer has a lifecycle independent of the Invoice. If the Invoice is deleted, the Customer remains. It is interesting to ask what should happen to an Invoice it its Customer get deleted.

A Mailing List might reference-many Customers. If the Mailing List is deleted the Customers remain.

These two relationship types unlock a super important part of modelling the graph of models to represent a domain.

Make date and time types available as a plugin

Issues to consider:

• how to migrate back and forth between data and the other data types - especially if this "plugin" should not be aware of the other data types!

If I have a customer and bookings database, when I am viewing bookings, I want to be able to filter by customer.

I want to be able to select the customer I am filtering easily, probably via a typeahead

She got everything done. We had a customer and bookings database at the end, about 30 minutes in actually. But it confused her in places, bugged in other places, and did not wow her in any way.

• Did not notice the Save button on the record row. Clicked away and lost her data. I had to explain to her that there was a small save button on the right hand side of the row
• Added Customer and Booking table as first action.
• Added "Bookings" as a "Link to other record" field to Customer. While Cozemble is meant to link from Customer to Booking, this is what she is used to doing in Airtable, and it (I guess?) permits viewing and editing data from either end of the relationship. Need to look into this.
• It became impossible to rename this field - the Customer.Bookings field. It kept staying as "Field 5".
• Opened the table edit dialog (by clicking on the caret beside the Customer table name) then clicked the Booking table name, and the table edit dialog stayed on Customer
• Added a field (Bike Make) to Booking, did not click Save, clicked Add Field again, and things got messed up
• When it came to adding the Notes field to the Booking, she did not see the "Multiple lines of text" checkbox under text, she was expecting to see Field types of "Text", "Long Text" and "Rich Text", which is Airtable lingo.
• She typed dates into text fields, rather than use the date picker. I think she did not see the date picker icon, and/or was expecting one to pop immediately. Additionally, when typing the date, as soon as she entered the first character of the year, the input submitted the value and tabbed to the next field.
• She wanted tab and keyboard arrow navigation when navigating the data cells i.e. not in edit model. Because she wanted to copy and paste a value from one cell to another
• She wanted to add a second booking for the same customer with a different bike. Her instinct was to create another row.
• When filtering on dates, she wanted abbreviations for common date options, like today, tomorrow, next Wednesday
• She also expected to be able to free text search on dates. If we're going to do this, identifying which date format is in effect (in the user's mind, not in system config) will be a challenge.
• When she added a new customer when adding a new booking record, the new customer did not get put into the Booking.Customer cell. This may be due to the fact that Customer was a bit messed up with the Customer.Bookings field, but need to check.
• She did not get "Sub sections" at all. When I asked her about it, we were looking at Customer data. She said maybe it is a sub account, or an address book. So she, like all others, thought this was a data entry thing, not a model building thing.
• She expressed a need for an Autonumber field, a RecordID field, and a formula field so she could make ID fields of her own format. The RecordID field was so she could make urls to external services with ?recordId=${RecordId} in it. The custom format ID fields is so she can make friendly IDs, like YYDDDNNN
• She expressed a need for a drop down field, single select for something like "Employee.Level" and multi select for tagging.
• She asked what does Unique. She was trying to add Level to an Employee record that she added as a sub record of Customer, and she was reaching for the ability to make level a drop down list. But it did make me wonder about the understandability of Unique in a nested context. Logically its no different to Customer.Email being unique at the top level, but it feel weirder when nested.

When I have a model, let's say Customer, I want to be able to say that the Email field and the Phone Number field are unique.

These things need to happen:

• Configure a Property as being unique
• Have that manifest in the database schema as a unique column
• Have uniqueness constraint failures on save reported in the data editor, at the offending cell, not just "on the page"

Currently we have the model editor and data editor as separate components.

We need to pull these together into a coherent single UI.

Tabs or menu bar to flick between them

Let's say we have a Booking record type, and it has a notes field, that notes field needs to support multiline text. Right now pressing Enter in a text field submits the data.

Change the Record Editor to be driven off of jsonschema definition instead of Models

This will require mapping Models to JSON Schema definitions, and mapping DataRecords back and forth between plain json.

JSON Schema supports regex validation statements, some number validations, etc, and supports extension. So there might be mileage in this, but it might be best regardless for errors to be passed down thru the component tree, and validation be done about the root UI element, so custom validation is possible.

Question is: why am I think about this? If the component uses something standard, it might be more generally useful, and folks might adopt it more, use it more, test it more, augment it more. It just “feels” better to play with open standards where possible.

If a jsonschema driven editor, with extensions like document generation etc, can deliver the UX we’re aiming for, then it makes sense to do that.

The door will always be open to many Rich text editors, based on lexical or prosemirror or editorjs etc etc.

But this task is to pick one of the above and create a custom cell type using it, so a user can add a rich text editor to a model, and edit rich text data in their records

Just picking and implementing one example will act as a path finder for folks looking to integrate other rich text editors

Right now you can have only one root model, extend so its possible to have many

When I am viewing a list of records, let's say a list of Customers, I want to be able to filter the list by free text search.

So if I type in "mike" into the search bar, all Customers containing the string "mike" anywhere in their schema are shown

To orchestrate a user's supabase in the way we want to, we will need to store their:

• supabase_personal_token for the management api
• postgres database url
• postgres database password

These will need to be stored securely as secrets

If a customer references many bookings, when I view a customer or a list of customers, I want to be able to have a cell that shows:

• this customer has three bookings
• this customer has no bookings
• this customer has five bookings

And when I click on link showing this information, I will go to the bookings view, with the list pre-filtered by the customer

This will require #20

Define booking, add a booking record
Add nested "Bike" model to booking with Make, Model and Picture
Edit the booking record.
App breaks

Actually this happens regardless - nested model editing is broken

it really isn't that great that I have to make a booking record to discover than I can make a sub record of Bike underneath it. In general, it's not great that the developer has to enter data to unfurl all of the model editing features.

checkout
build all
cd backend/tenanted-api
npm run first-time
cd ../../frontend/main-app
npm run first-time