speckleworks / speckleserver Goto Github PK

As it actually is a geometry hash, not an object hash

On an update from Grasshopper, previously stored info from viewer gets overwritten by update. To check.

Read the following as a spec sheet for the future!

• Comments can be assigned to streams, objects or another comment (allow nested discussions).
• Comments may have an author; ie anonymous comments are allowed but disabled by default
• Comments can be edited by their authors
• The comment's accessibility is determined by its root object's accessibility.
• Comments may have a camera view associated with them

An anonymous user can post/read comments only on resources that have anonymousComments===true

A logged in user can post/read comments on any public resource or a resource that is shared with him with "Read" access minimum.

To be refined once implementation takes shape.

Step 0:

• I've read the contribution guidelines!

Actual Behaviour

The speckle image fails to run as the server is looking for a .env file
https://twitter.com/Jos_ols/status/1009066622194208769

Proposed Solution (if any)

The docker compose file should set the defaults usually set in the .env-base file, namely:

• MONGODB_URI="mongodb://mongo:27017/speckle"
• REDIS_URL="redis://redis:6379"
• SESSION_SECRET="helloworld"

Step 0:

• I've read the contribution guidelines!

Expected Behaviour

Server should refuse streamId changes.

Actual Behaviour

Right now you can programatically change streamIds, which is really really bad.

For example, a preliminary schema:

role: { type: String }, // enum: Receiver, ParametricSender (ie has sliders), Sender
documentGuid: { type: String }, // document unique id. / name
documentType: { type: String }, // Rhino, Grasshopper, Node, Browser, etc.
owner: { type: mongoose.Schema.Types.ObjectId, ref: 'User' },
streamId: { type: String } // stream that the client interacts with
online: { type: Boolean, default: true }

Radu [10:04 PM]
hmm, is this expected behaviour @Dimitrie? when registering a new user, the user password is dumped in cleartext in the logs
Radu [10:04 PM]
uploaded this image: image.png

Step 0:

• I've read the contribution guidelines!

Proposed Solution (if any)

Deploy and admin module that comes with every speckle server and rarely gets updated. This should be accessible only to "god" users and force her/his registration on initial server deployment.

Should allow for:

• upgrading the server
• upgrading installed plugins
• installing new plugins
• CRUD on the database
  • delete/ban users
  • crud streams, projects, etc.
  • ...

For the sake of simplicity, things like

• Viewer
• Parametric Viewer
• Admin
• User Dashboard

Will be included with the server. This will allow an integrated deployment, which simplifies stuff.

For all models:

• streams
• users
• objects
• clients
• comments

will be useful & needed for various frontends atomic updates. see ref for how to implement

Due to the fact that for permission checking, we enforce several fields using by adding them to fields=x,y,z, resulting in a 404 failure (omit and fields are incompatible).

Solution:
check if query is asking for fields or omitting them:

• if omit, make sure the permission fields are not there
• if select (fields), make sure the permission fields are there

• POST /objects
• POST /objects/bulk

Check all routes that apply:

• GetStreamObjects
• GetLayerObjects
• ...

Will sender clients push update notifications, or will they be handled by the server?
Clients push update:

• PRO: easier server side, more granular control
• CON: more difficult to write clients

Server pushes update:

• PRO: Less things to worry about client side
• CON: Messier room management & authentication issues

Context

I'm thinking about proposing we abstract the DB layer at some point in the future, so SpeckleServer would work with a number of databases. This would address some of the limitations we've encountered with MongoDB's max sizes, etc.

This would obviously be quite a bit of work and a major change, so this issue is a sort of feasibility study.

What's required

• list where SpeckleServer is coupled to MongoDB
• list secondary interfaces/dependencies to MongoDB

I'd kindly like to ask for some help & input from those more familiar with the codebase ( 🤗 @didimitrie)
Also anyone feel free to tell me off if you think it's not a worthwhile exercise 🤓

Docker-compose should use a specific tag with image that it pulls. I updated the compose from dev -> latest as it was pulling dev image which was built 2 months ago and did not contain latest API updates. speckle/speckleserver:v1 would be a good suggested tag to use here. Todo: Build and push to dockerhub.

So that if you don't have any layers, existing clients (gh am looking at you) won't freak out and not output anything.

So: if there's a layer structure, cools - if not, just output everything into one layer (output gh/layer rhino)

what a surprise, it was doing this before

Step 0:

• I've read the contribution guidelines!

Expected Behaviour

You should not be able to trawl for emails when searching for users.

Actual Behaviour

You can get everyone's email out if you're patient enough.

Proposed Solution (if any)

Do not return email address on user search route.

Right now there's a generic history-update announcement and a live-update announcement.

We need a specific announcement for every historyId. Clients should listen no more for live-updates but for the historyId they're currently set to receive.

Phase out the live-update, as it would be gobbled up by the above.

Step 0:

• I've read the contribution guidelines!

Right now plugins need to be registered manually in server.js. We should have a system whereby on startup we scan all top level folders of the node_modules folder and look for anything that contains a speckle plugin manifest file, eg. speckle-plugin-manifest.json

This manifest file should contain basic info about the respective plugin, ie:

{
  "name": "Dashboarder",
  "description": "Basic speckle project dashboard creator and viewer",
  "serveLocation": "/dash",
  "index": "dashboard.html"
}

This way third parties can develop their own frontends, publish to npm or the package manager of their choice, and then simply do npm install speckle-dashboards@latest to have it running with a live speckle server.

Step 0:

• I've read the contribution guidelines!

Expected Behaviour

API should allow for queries on *GetAll routes.

Actual Behaviour

All queries are ignored.

ie:

• PUT streams/{streamId}/layers
• PUT streams/{streamId}/name
• PUT streams/{streamId}/objects
• PUT streams/{streamId}/permissions

Should be an env value.

Step 0:

• I've read the contribution guidelines!

Expected Behaviour

• When adding a user to a project (canRead, canWrite) propagate that to all the streams in the project.
• Likewise when adding a stream to a project, add to its permissions all canRead/canWrite users.

Proposed Solution (if any)

For now, the end clients can update this, but it's not elegant :)

Step 0:

• I've read the contribution guidelines!

This is needed for passing smarter info back to users:

• when was the sender last seen online
• when i push an update, how many receivers actually receive it
• how many receivers wait for one stream
• how many of them are online, how many offline

Proposed Solution (if any)

Create GET /clients/bystream/{streamId} or better GET /streams/{streamId}/clients.

Otherwise we have login problems between [email protected] and [email protected]

It should handle gracefully other non-dangerous stream properties too, ie:

• parent
• children
• globalProperties (where theoretically the viewer will store info)
• etc.

Step 0:

• I've read the contribution guidelines!

Right now there's a bit of a chaotic moment in the speckle server config stuff. I would suggest we move to having .env (not committed) and .env-base files which centralise all config options.

These can be subsequently read back in via dotenv (which does not overwrite already set vars, so docker will be fine).

Problem: on some computers and with certain node versions (...) compression fails, and zlib throws an incorrect headers error. This manifests itself as clients not being able to send data to the server.

Solution: disabling compression middleware in express; comment out app.use( compression() ) AND remove all compression from the grasshopper components.

Compression should be enabled back in nginx or whatever your server's running and not in the nodejs app.

This will be fixed in the new-api branch, but not backwards.

IE, ?format=speckle or ?format=rhino, plus scaffolding for adding future typologies. Important for speedy online clients and non massive retrieval of useless data.

This needs to happen inside AssembleObjects and be mirrored in the GeometryGet route (which thus should probs become an outsourced controller).

@PaulPoinet's discovery.

What happens: because a stream can contain duplicate objects, when assembling the response from routes of the like of /stream/:streamId/objects/ and /stream/:streamId/layers/:layerId/objects, there's a reduce pass to make sure the response contains them all

So now we need to check for null insertions and be done, and flag up a warning that when querying stuff you don't have the guarantee of getting back an ordered list.

☮️

Open question: should the viewer be a submodule (as it is now), or essentially a step in the Dockerfile image (RUN git clone etc)? Bearing in mind this would/should expand to other front-end apps, etc.

any help/info welcome. bear in mind, main point is to get the install of a speckle server as simple as possible...

SpeckleObjects should have their own unique hashes.

When a user tries to insert a huge doc, ie a mesh with >100k vertices or some sort, mongo goes crazy, topology gets destroyed (sounds like a wizard spell) and a full server restart is needed (mongod + app).

How to protect from this/work around this?

• clients should not send this kind of requests in the first place
• different storage engine for mongo/big objects (wired tiger vs. gridfs)
• ...

e.g. https://s003.speckle.works/api/streams/HkpvHF8pZ

From GeometryObject.find( { 'hash': { $in: geometriesToRequest.map( xx => xx.hash ) } } ).lean() to potentially something like:
Promise.All( geometries.ToRequest.map( gO => GeometryObject.**findOne**( { 'hash': gO.hash } ) ) )

Step 0:

• I've read the contribution guidelines!

Expected Behaviour

You should not be able to trawl for emails.

Actual Behaviour

You can get everyone's emails.

Affected Projects

Affects speckleworks/SpeckleSpecs and speckleworks/SpeckleAdmin, at least.

Reproduction Steps

To get emails quickly:

1. Search for users by email (this doesn't return email, but does return id)
2. Get user by the id returned above

Alternatively, you can abuse regex/substring matching:

1. Repeatedly search for users with queries like "john.smi[a-m].*", narrowing down options until you find full email addresses

Proposed Solution

• Don't return emails from the UserGetProfileById endpoint.
• Only search by substrings of name and company?
• Consider other methods of permissions (e.g. EXPOSE_EMAILS parameter in .env, per-user publicity flag?)
• Check how these issues manifest in the v0 api, and resolve them if applicable.
• Update specs
• Update admin app

Step 0:

• I've read the contribution guidelines!

Expected Behaviour

Go to your server url and actually see something informative!

Actual Behaviour

You get an error: Cannot GET / C'mon, user friendliness!

Proposed Solution (if any)

Implement a static no thrills no frills landing page for the server. Should contain basic information and potentially some links to plugins registered via #77.

so we can deprecate get layers, get name, get random field

Step 0:

• I've read the contribution guidelines!

Websocket clientIds are not checked on connection for uniqueness. This can be a potential hazard down the line, as client identities can get confused.

This should be checked on connection.