unavi-xyz / unavi Goto Github PK

cargo2nix is a different way to build rust packages within nix. Currently we use the built-in buildRustPackage, but is seems cargo2nix would have better caching, and im not sure whether the current setup actually works correctly.

We currently use rapier for physics, which is a powerful and well tested physics engine, but it may be nice to use bevy_xpbd instead as it is built with the bevy ecs and should have less overhead. We use the rapier character controller currently, so would need a new solution for that, maybe bevy-tnua (which works with either rapier or xpbd).

Not 100% certain I want to do this, need to try it out and see.

Set up infrastructure for deploying the main branch to app.unavi.xyz. This could be as simple as an ssh using a github action. Could also set up docker containers for the server and build + deploy those, which might be what we want long term?

current handles suck, DIDs are good, lets just do it

going to use did:web for now, but more can be added later, such as did:ethr for ethereum addresses (removing the need for a "home" server, making it much easier to self host / run your own client locally).

in the future can also add a better handle system using domain names, like @alice.unavi.xyz that resolves to the underlying DID, but that will be a separate issue, not sure how (or if you even can) handle subdomains within nextjs.

Probably just text for now, could go further and add profile pictures though which would be nice 🤔

Add the spawn to the scene tree so you can move it around

The scripting ECS, defined in wired-protocol, is partially working right now but I ran into issues when returning resources. This might be a limitation of our runtime as opposed to wasm components themselves (or not, im not sure) but either way we can redesign the WIT to use u32s instead of resources.

Ability to add glTF models into the world from build mode. Will want drag and drop support eventually but for now just file select a .glb is fine.

How do we send this over the wire? (well, over "The Wired" xD)

Option 1

• typical lattice gltf import -> traverse node and create commands for each item
  • aka split the import into many small atomic events (createNode, editNode, createMesh, etc etc)

Option 2

• a single new "importGltf" event
• how to send gltf model? upload to S3, then send url to it in event?

this also raises the question of storing the model independently on S3, vs merging into the scene like option 1. independent would make more sense if we were using glxf, and in the future maybe we will? It seems to make more sense than a single huge gltf file, but for now we can just merge, seems like the simpler solution

Got basic VRM loading working in https://github.com/unavi-xyz/bevy_vrm. Need to import that and add VRM avatars to the player controls. This will involve supporting the first person toggles, as well as loading some animations (probably from mixamo) and applying them to the model.

The explore feed is on-chain data so we can only filter in the client-side temporary for now?
I think the better way is run a contract event listener and update to the backend. maybe we can build it later.

Most the networking code for this is already there, we had the feature in the past, just lost it when upgrading to the new engine.

I had thought we were blocked by lattice not having audio, but I don't think we even need to use lattice for this, we just need to move around some AudioPanners. Should be fairly simple to add this back in?

Need to figure out the networking layer.

Original plan was WebSocket, then WebRTC for UDP + voice chat (using mediasoup).
Am a little unsure whether mediasoup will work client side in native and WASM web build, so would need to test.

Also was looking at WebTransport which seems cool, so could potentially do everything over it instead of WebSocket + WebRTC. Seems most browsers support it except for Safari (which has it in a technology preview), good enough for me. The spec is evolving so might be less stable.

Also also, should see how things would work with our existing axum server.

Ultimately need to see what kind of crates exist already for us to use.

Hello,how to change from localhost to hostname likes unavi.tech in order to set correctly the social login redirect uri .The server start Always in localhost:3000. I made change in .env file without result.
Also wich Is the http link and the port for start the Main Page in pnpm docker:prod? Thanks in advances

No way to set it from the UI currently. Can keep it simple for now and just use the node's current mesh, rather than the ability to select any mesh shape from the scene for the collider.

When prototyping things out I used a lot of .unwrap and .expect, rather than properly handling errors. A great purge is on the horizon, probably with the aid of thiserror, where we will need to remove these and do things proper.

Ability to add portals to your world from within the editor. Each portal will have an ID and can be used to enter the world (from other portals, or via URL, ie /play?id=1234&portal=5678.

If two portals link up (from the same world, or between worlds), they will effectively form a hard link between the worlds for seamless navigation.

Migrate all components / systems to thyseus v14. Waiting on kayhhh/houseki#118

Right now the editor uses names to identify items, which is not great as names must be unique or there are conflict, which is not enforced by gltf. If a model doesn't have names, we must generate names for them, which is currently done by the client only, assuming other clients import the model with the same deterministic algo for generating names.

A better approach is to use IDs. When the editor is opened, it will generate a unique ID for each item (node, mesh, etc), then network these IDs to other players, using the gltf indices to identify items.

These IDs could be saved into gltf extras on save, but I don't think we need to save them.

In the editor, changes to the scene are sent as one time events to other players. When the owner saves, a new glb is exported and set as the world model. When new players join a world as it is being edited, they wont see any changes that happened before they joined. We need to somehow get them up to speed.

Event history

One way to do this would be to store a history of all edit events. When a new player joins, send them all the events, in order, and they will end up in the same scene state as you.

It gets a little more complex however when we take into account world saving, as players that join after a save will have a new base glb model with the changes already made.

Maybe on save, the event history is cleared, and the world is reloaded with the exported model? It seems annoying to have to reload the world, instead of being able to save whenever with no friction.

Export current model

Another option is to export the current scene as a new glb, and have any joiners load it instead of the normal world model. This would solve the problems around saving, as the owner's edited version is always the model people load. However, there are some problems it brings. Especially for larger models, it could seriously lag the owner when people join, as they have to export the scene to glb, then upload it to the other person (or maybe to a temporary S3 bucket? not sure if large files could, or should, be chunked over websocket). They wouldnt have to optimize it like a full production export so it might not be too terrible, but is still not great.

We need some format for storing scenes / worlds. I see two options:

glXF

• A very unsupported format, would likely need to write our own reader / writer for it
• Works great with glTF

glTF Export

Is there a way to export a bevy scene as glTF? I don't see one, which would mean we also have to write this, which is not trivial.

BSN

• Bevy's native scene format, easy to use
• Not generalized, is specific to our implementation of the engine

I think glXF is ideal option. Would be better for long term compatibility, and would mean we lean in to the glTF format, doing everything within defined extensions which I like the idea of. However, we would have to write our own glXF import, exporter, and a glTF exporter... which is a lot.

Need to figure out how the editor will work and create some basic functionality. The idea is for you to be able to edit worlds live, without having to exit to a separate app, and have everything sync in the multiplayer environment.

Would be awesome to find some sort of generalized networking solution for syncing between players that will work outside of the editor, like picking up objects in the world and throwing them around.

How do you enter edit mode?

• Or is there even an "edit mode"? Maybe you just pull out a tool and start using it?

How do we propagate updates to other players in the world?

• Do we need some sort of CRDT scene representation?
  • Seems annoying to maintain + keep in sync with ECS
• Bevy scene serialization?
  • Existing libraries could automatically handle most of it
  • Wouldn't be very generalized, might only work between players on the same client version

How do you save changes?

• Is there a "save" button?
• Or does your client auto save?

always a good idea to version your api. all this really takes is moving a bunch of files around, and making sure the few places in the code that use the api path are updated.

Create new worlds and connect to them.

take a screenshot of your current view to use as the world image

I've been playing around with different identity options - handles, DIDs - but I think just using eth addresses is the simplest and best option. There is a benefit from being able to sign in with google or other socials, but its too much work for what its worth, especially at this stage of things. Will be good to narrow focus.

Plan is to remove the handle system, remove sign in with google, and resolve ENS names / profiles when available. We can still keep the profile system (profile picture, name, bio, etc). This will be used more as a fallback for people without ENS, as it is stored in the database, not accessible to other clients.

I laid out some ideas in the wired repo, but we need something concrete to start with and work off of.
We need a way to let users write scripts, save them to the scene, then load + run them in a sandbox at runtime.

What language?

• Original plan was isolated WASM modules (with a Rust API)
• I did see a few bevy scripting libraries for using scripting languages like Rhai or Lua (https://github.com/makspll/bevy_mod_scripting)
  • Might work better for hot-reloading
  • Scripting languages would be easier for users to write than Rust
  • Unsure about the sandboxxing capabilities

But I think I'll still go with WASM. Rust is what I enjoy writing, and WASM works better for the vision I put in the wired repo. I eventually want some sort of node system built on this lower level Rust API, so people can use that instead of writing code for simpler behaviors.

How will you write scripts?

• create a project folder they can open and edit with their IDE and work out of.
  • there is a web api for linking to the file system so web support is not out of the question, just not super supported by browsers right now (and I have no idea whether wasm-bindgen supports it)
• can we support hot reloading scripts with WASM?

Networked dynamic physics objects.
I see two approaches.

1. P2P Object Ownership

This is the approach that I believe VRChat uses. When someone grabs an object they become the owner of the object, acting as a source of truth for it's location. Can be easily spoofed - letting them put the object anywhere - but should work fine enough.

2. Central Authority

We could have the host server act as the source of truth for object locations. This is more expensive for the server. How much the server validates player input could vary - from a hands-off full trust approach, to running a physics instance of the world, checking the movement of objects, player movements, and more to determine whether they are "cheating".

Thoughts

I like that the first approach imposes no cost on the server. I was interested in the central authority approach, but upon further reflection I'm not sure of any immediate use cases that would require it. Main thing I think of is some competitive game involving physics, whether objects or player movement, but it would take such an advanced setup to work effectively that I don't think its worth planning for now. Other competitive games, for example battleship, could still potentially run logic on the server (not sure yet, still need to figure out scripting), without the overhead of a full physics simulation.

Most editor actions (spawning objects, unlocking tree items, etc) are broken right now, just doing weird things. I think its from memory issues, hoping #795 fixes this.

We should create a library of UI components designed for VR (but also works in non-VR).

We could try building with some existing UI library, but from my knowledge these are all 2D UI components, which would mean in VR we render that on a flat plane and interact with it using hand lasers. This is the de-facto approach most VR apps take, but I want to experiment with more expressive 3D UI elements to capitalize on the VR medium.

Additionally, I got to thinking about how we should implement this, or more where we should implement this. Rather than build this as bevy components and systems, what if we created the UI library within the WASM scripting framework? This would allow us to provide a well-tested UI library that anyone could use within their worlds. Dogfooding our scripting APIs and tooling would be a great way to ensure they are both high quality and powerful.

This would probably mean we write all menus using these UI components within the WASM as well - settings menu, world browser, friends list, etc. To use the settings menu as an example, we would need some set of "system" scripts connecting the WASM environment to the corresponding settings in bevy. Not sure exactly how this works currently but it should be doable. We may need to figure out script permissions to do this, so only our trusted settings menu script is allowed to change settings. We don't want some random user script changing the settings of everyone in the world.

Saving is broken right now, things like physics colliders can disappear. I think its from memory issues, hoping #795 fixes this.

Confusing / annoying conflicts with unavi-app

Check the chain and display any NFT avatars you own, and let you equip them

Add a mirror to the demo world. Still not sure what the long term plan is for mirrors, but for now can throw something together using https://github.com/Selene-Amanita/bevy_basic_portals

white dev texture ground
light gray walls
white bg or skybox idk

Ability to drop a temporary portal at playtime within worlds, which will take you to the spawn of the target world.

The server is designed to be modular, allowing for flexibility in what features you support, and with how you deploy it (working on a single machine, or split up across many). I have been using cargo features but its a bit confusing and annoying to work with, and will only get worse.

Will have a main unavi-server crate, then separate crates for each service (unavi-server-world, unavi-server-identity, etc)

The landing page (route /) is not being cached correctly, and can take several seconds to load.

The landing page fetches worlds in the explore section, these fetches should be stale cached - ie I go to / and immediately get the page, if the cache is stale, it should fetch the latest worlds and update the page, the next time someone requests / they will get the fresh page.

Things should be set up to do this using the app dir export const revalidate = ... settings but for some reason its not working.

There should be UI somewhere in the editor to export the world model, currently you have to open networking devtools to grab it.

Hello , is there the possibility of signin with social and not wallet ? If yes of i can do ? Thanks in advances

When in third person mode, sometimes when rotating around your character you will experience a jitter, and be teleported back a little bit. It seems to happen at a consistent location around the character. The bug is likely found somewhere in the rotation math.

Add the ability to enter VR and move around