m3-org / characterstudio Goto Github PK

Per 3D model (Mesh parented to armature) export a manifest.json with the following info:

In the future this can be NFT metadata (every trait could be a composable NFT wearable)

Currently the process of adding custom animations is complex, need to figure out a way to optimize exported animations and make them work with any vrm file (probably a unity plugin)

The current configuration serves files in the public directory at the root path, which leads to incorrect asset pathing when trying to import them from JavaScript. Below are the encountered errors:

files in the public directory are served at the root path.
Instead of /public/style.css, use /style.css.
Assets in public cannot be imported from JavaScript.
Instead of /public/sound/background/cc_bgm_balanced.wav, put the file in the src directory, and use /src/sound/background/cc_bgm_balanced.wav instead.
Assets in public cannot be imported from JavaScript.
Instead of /public/sound/sound-files.json, put the file in the src directory, and use /src/sound/sound-files.json instead.
Assets in public cannot be imported from JavaScript.
Instead of /public/sound/sounds.mp3, put the file in the src directory, and use /src/sound/sounds.mp3 instead.
Assets in public cannot be imported from JavaScript.
Instead of /public/ui/weba.png, put the file in the src directory, and use /src/ui/weba.png instead.
Assets in public cannot be imported from JavaScript.
Instead of /public/ui/traits/shuffle.svg, put the file in the src directory, and use /src/ui/traits/shuffle.svg instead.
Assets in public cannot be imported from JavaScript.
Instead of /public/ui/selector/cancel.png, put the file in the src directory, and use /src/ui/selector/cancel.png instead.
Assets in public cannot be imported from JavaScript.
Instead of /public/ui/selector/tick.svg, put the file in the src directory, and use /src/ui/selector/tick.svg instead.
files in the public directory are served at the root path.
Instead of /public/sound/sounds.mp3, use /sound/sounds.mp3.
files in the public directory are served at the root path.
Instead of /public/ui/weba.png, use /ui/weba.png.

Upon exporting with a lower resolution than 4k, each atlas slot gets scaled wrong

Currently, the manifest decides how the mesh below will be culled by top meshes, but sometimes its hard to get a perfect culling. We should be able to shoot hidden geometry to trun it on again, as a final polish before exporting.

We should be able to not only to upload custom textures, but also transparent textures that adds on top of existing texture.

This will help adding things like tattoos, cloth branding or underwear without mesh

Let the users drag and drop a custom vrm while they are in a trait, then apply culling with base manifest trait information

Loading any character just displays the character without any effect, we need to find what is causing the shader not to trigger:

We should be able to drag and drop a texture, and display it on current selected trait.

Camera distance can be set for each trait, but there is a maximum camera distance that will reset back to this max distance.
Let set the max allowed distance as a base parameter in the manifest of each character.

Currently vrm download size is huge when combining a mesh.

This has to do with morph targets saving all the data from the skinned meshes for each morh attribute.

This can be fixed by using sparse accessors, and will drastically remove size, as it will only save morph target vertex deformations

Currently, creating a new character requires user to manually create a manifest and write all options, this can be easily done with a custom unity script to create this visually

Part of the code is filtering bones, but it seems there is a case were some bones are getting removed were they shouldnt, this happens in spring bones, in the image is shown how the pony tail no longer has spring bones after being exported

Make a beautiful readme with links to community, screenshots, how-to information, etc. Search for best practices on generating good README files.

Be able to tweak the culling distance very carefully (+ input custom number if need be) with a slider, preview effects in real time

  "cullingDistance": [
    0.3,
    0.01
  ],
  
  https://github.com/M3-org/character-assets/blob/main/anata/manifest.json

Following the install instructions, I am getting a 404 in console after running npm run dev

brave browser, arch linux, using --legacy-peer-deps

Mouse follow is working different on vrm0 traits (its inversed) so when body is vrm0 and hair is vrm1, they move in oposite directions

most probably don't realize it's 2 repos to build the project, one for the main program and the other for the assets being loaded in. Mention in README + documentation

could be a button to flick between wireframe / render mode

right now it's exporting each armature + mesh as vrm 0.x

Currently, if an icon from the manifest for a trait is not found, we get an error for each thumbnail that was not found

lets add a default thumbnail in case the icon is not found

There are some issues when trying to export vrm1 from blender with some models (Mainly hair)

Exported some hair models in vrm0 format and vrm1 format from blender and seems vrm0 has no issues, this is more notorious in hair, which include spring bones, exported vrm1 from blender is not including this information

Also, for some reason, exported hips for vrm1 format are rotated 18 degrees in t-pose, which is making the character look bald

Should be 0 like vrm0 exported model

So the best approach is to also support vrm0 as import model. I still have to check how complex this is, if its too complex, to finish the objective in time, we can export vrm0 and reexport from unity as vrm1 as unity exports the models correctly:

(image shows hair working correctly, by exporting from blender as vrm0 then exporting from unity as vrm1)

Currently the culling faces is not working when combining VRM1 traits with VRM0, we need to calculate bvh colliders with vrm0 rotated 180 degrees

Current texture atlas generation creates same section size for each texture as image

But some of them are more important than other (for example, body section should be bigger than small details like litlle plant in the head)

We should make the atlas creation consider texture sizes and trait importance to determine it's size in the atlas

can use emojis for now

Right now exports VRM 0.X, which is ironic given that the program requires VRM 1.0 assets for ingestion. IMO best to give user option to download VRM 0.X or 1.0

Currently to test different animations we have to open any trait option to see animation options and mouse look.

User should be able to select animation without opening trait options, also it should be able to pause and play. Probably a new lower menu can fix this (include randomize, debug, animation selection, play, pause, mouselook)

something like this ( design can be improved )

Users should be able to optimize vrm's by dragging and dropping a VRM with mutliple materials. Character studio should merge it, create an atlas and download it

Before

After

• Group traits with the category
• Inspect details / information on the right side

Scope out what it'd take to make a version for MetaFactory in terms of UI, assets, avatar bases (can use hyperfy bot). Will need to make new wearable meshes

There should be a checkbox/options to turn off texture atlasing, by default it should be set to false

Collider bones are not being exported from the character studio (only spring bones)

But we might also need to define which trait will be used for colliders as we dont want to use every collider from every trait, we can defrine this in the manifest. (example image shows how collider bones should be displayed in unity, current exported model from character studio does not have this info)

Current atlas will grab all materials from every vrm model and combine it to a large atlas, but this atlas is meant to be opaque only.

In case the user has anytransparent material on their model, we need to create a secondary atlas map for all transparent materials it has

We should be able to choose to export ktx2 textures for optimized performance in supporting platforms

Imagine this is character select, or populated once you connect wallet / see the NFTs you own

Next screen, have ability to have a custom background per each persona

mention how the assets repo is separate usually from the program, link to documentation on such there (separate issue on character-assets repo): https://github.com/M3-org/character-assets

Use CC0 assets like m3taloot for sample files

Missing to add full support for spring bones, every asset that gets merged, should also include the information

I notice it just hangs when importing to hyperfy or https://vrm-validator.fern.solutions/, tried all the different options for atlas

for some reason it always looks faded / light, something to do with blending? This only occurs with female anata

Something sassy and expressive movement in order to give us more chances to debug / find clipping issues. Would be cool / useful to select different animations

Support DRACO mesh compression for optimizer and character creation

Add some UI about what item is being equipped or hovered over, this can help us better debug like visual print statements. Would be useful to have overall stats visualizes in addition, like how blender / hyperfy display them.

Sometimes it works, other times it just freezes entirely. It will eventually unfreeze tho. Seems like a regression from a recent patch. It doesn't matter if using arrow or slider.

currently when exporting VRM without atlas for artists to tune up, the scale shrinks to 0.7 of what most are working with in anata project. This looks fine when fully assembled, but if we need to tune something up it would be better to keep consistent with rest of the project files.

we can use an outline of a default VRM armature from blender as the reference point, then a slider / scale (like how we do 0.7 currently).

I don't think this is a big priority right now, because if we drag and drop a VRM trait into character studio it will scale to appropriate size. I just want to post this just in case.

right now its just in optimize character option

would be good to add with character creation flow, here right before downloading it

The way how currently animations work in Character studio is complex and not realiable as sometimes it gets desynchronized making loading new traits look awkward.

Lets find a way to implement vrm animation spec to achieve animations and be able to simplify the process.

I've noticed that the vrm exporter from blender already includes a way to export animations:

We can do some testing with an exported animated file from blender and see if it gets support in vrm-three plugin loader

I converted all the metadata in anata files to match filenames in blender

generate manifest.json based on anata metadata (will upload here later https://github.com/M3-org/anata/tree/main/vrm/female)

previews show up here when wallet signs in, for now don't worry - this next steps can be headless

export VRM for each anata (1000 female)

As suggested from chat, lets use WebXR to support AR in character studio https://immersive-web.github.io/webxr-samples/