qodotplugin / qodot-plugin Goto Github PK

Since the geometry generation code is a brute-force algorithm across potentially large datasets (a.k.a. complex many-faced brushes), porting it to GDNative would be good for performance. This also opens up the possibility for the GDScript-side code to be strictly engine integration, making prospective ports to other engines easier.

This might conflict with the #4? Unsure of the status of multi-threading + GDNative, though I suppose the DLL could handle all of that native-side without involving Godot.

Collision wireframes will only partially render after being added to the scene, but can be refreshed by tabbing to a different scene and back.

Likely some sort of property update signal / order of execution bug.

Need to create one MeshInstance for each brush face, then parent them to a new QodotBrush node for structural consistency.

Collisions are generated using a separate point cloud approach, so that shouldn't be an issue.

This is a prerequisite for #15

This looks like it could be a race condition with assembling the geo tree structure, more research is needed into Godot's renderer threading modes.

Actually this looks like a more general-purpose error condition that may or may not crash.
So far most occurrences have been with the renderer in single-safe mode, but having done some research it doesn't seem like it should be effected.

This is backed up by being able to generate ad_chapters semi-reliably on it, whereas I'd expect more consistent issues if multi-threading alongside a single-threaded renderer was fundamentally unsafe.

Error sample:

ERROR: body_attach_object_instance_id: Condition ' !body ' is true.
   At: modules/bullet/bullet_physics_server.cpp:574
ERROR: get: Condition ' !id_map.has(p_rid.get_data()) ' is true. returned: __null
   At: ./core/rid.h:154
ERROR: body_add_shape: Condition ' !body ' is true.
started thread job 0 / 4
   At: modules/bullet/bullet_physics_server.cpp:504
started thread job 0 / 4
ERROR: get: Condition ' !id_map.has(p_rid.get_data()) ' is true. returned: __null
   At: ./core/rid.h:154
ERROR: body_set_shape_transform: Condition ' !body ' is true.
   At: modules/bullet/bullet_physics_server.cpp:524
ERROR: get: Condition ' !id_map.has(p_rid.get_data()) ' is true. returned: __null
   At: ./core/rid.h:154
ERROR: body_set_shape_disabled: Condition ' !body ' is true.
   At: modules/bullet/bullet_physics_server.cpp:553

From the look of it, the physics server doesn't have a resource ID for whatever body is being attached. Must be an issue with one of the add_child calls during geo generation.

Upload it to Godot AssetLib please.

Each of the methods in the QodotMap inheritance interface needs to be separated out into Qodot*Mapper scripts and exposed as an editor option a-la QodotEntityMapper.

Since the setup / visuals / static collision / area collision / lighting build steps already form natural groups, the build pipelines should either be split appropriately (since QodotMap assembles on-demand anyway) or arranged with a new BuildStepGroup class.

I think avoiding a new class is preferable, maybe pipelines themselves need to be renamed to something befitting a smaller group of steps.

Either way, also need to refactor pipeline assembly with a function for combining groups.

Tree attachment time has increased to the point where ad_sepulcher is no longer feasible to load. My initial suspicion is that NodePath is slowing things down, though that may not be the case given that the aggregated build steps are also affected.

If this is the case, attach paths need to be converted back into indices. There are wrapper functions for generating attach paths now, which should make it less painful.

The atlas shader mip levels are imperfect right now, which causes some slight pixelation artifacting when moving the camera toward an unfiltered texture. Visible with the door across the bridge in the second room of E1M1.

Need to figure out how to manipulate the mip level optimally and minimize or eliminate artifacting.

Similar to the recent visual geo generation refactor, a new collision shape build process needs to be implemented that uses a single CollisionObject with many CollisionShape nodes at its children, rather than one CollisionObject per brush.

Areas will still need to be done per-brush, as they represent triggers and thus should not be shared.

func_illusionary is similar to CLIP and SKIP, but is used for brushes that have a visual component but no collision component.

Reference - https://quakewiki.org/wiki/func_illusionary

Need to implement face properties in the TrenchBroom game config and parser, then output ShaderMaterial instances in the geo generator.

Need to create one TextureArray per PBR channel. Is it preferable to minimize space wasted by non-PBR textures's blank spots in these arrays, or preferable to index them symmetrically for GPU-friendliness?

.map file asset IDs change on every reimport, so the QuakeMapNode needs to be manually refreshed

This is presumably due to the MeshInstance it maintains not being present in the editor tree, and thus not being exposed to the BakedLightmap node.

BakedLightmap appears to have a bake() function that takes a Node to bake from, but I'm not sure if that means it'll be able to see a MeshInstance that doesn't live in the editor tree. Need to experiment.

The first step toward supporting formats- need to save out the test geometry in each different map format, then run text diffs to compare them and determine how the formats are differentiated.

Currently importing a map of modest size (E1M1 from Quake, for example) will lock the editor as soon as the file has been saved.

The hang is down to some hacky code left in the import plugin that calls load() immediately after saving the map to a .tres in an attempt to force an update of scene instances that reference it. I don't think that code does anything useful since the asset IDs change on every reimport at the moment anyway, so will likely be removing it.

The problem may extend further than that though- I had the editor lock up upon selecting the imported E1M1.map last night, which may point to Godot having issues with massive .tres files. Need to do some more research.

Single-threaded generation will cause maps to stall on load, and will only get worse with scale.

It should be possible to multithread on a per-entity and/or per-brush basis. I read about some caveats to using SurfaceTool off the main thread (need to pass null ArrayMesh to commit() and use return value instead).

The autogenerated tangents from SurfaceTool are broken for all Qodot faces, and cause the deep parallax depth mapping functionality of the SpatialMaterial to crash the engine, and in some cases the GPU driver.

Adding custom tangents during the SurfaceTool building process makes it work, though I don't know if my algorithm is valid.

I'm getting some extreme texture warping from certain angles, but enabling deep parallax makes that stop so I'm unsure if it's simply a known aspect of non-parallax depth mapping on heavily-zoomed textures.

Probably worth reporting the hard crash to the Godot repo too, time allowing.

While this isn't an issue for smaller maps, it becomes noticeable for denser and larger ones such as ad_chapters.

This should run on a thread, much like the geometry generation.

QuakeMapNode manually loads a .map asset from a string path instead of an exported Resource to sidestep editor UX issues and hanging references.

Not much to be done outside of fixing the editor for now, but worth keeping in mind for 3.2 and beyond.

Need to allow visual geometry to be created per-entity and per-mesh to support things like culling and give the user fine-grained control over generated geo.

Lighting functionality varies game-by-game, so I don't want to standardize Qodot around a single existing implementation.

However, it would be much less useful to ignore light entities completely, given that there are creators deep into the Quake map ecosystem who've already done a lot of lighting work for maps they might want to use.

Current thoughts:

• Game-specific support can be implented through an EntityMapper subclass
• Having basic support for light params shared across all games would be good - position, intensity, etc
• Being able to spawn the lights outside the tree so they don't get nuked on reload would be a good optional pipeline for doing the broad strokes in TB, then tightening them up in Godot
• Could implement Godot-specific light definitions in the TB game profile that feed directly into the types and parameters available, so the option for full TB is available despite its lack of light rendering

Possible pipelines:

• Don't place any lights in the .map, do it all in Godot
• Use .map lights as a rough prepass, then manually tighten them up in Godot
• Use .map lights exclusively

Should be implemented as a checkbox on QodotMap and cause information to be printed to the log during build.

Possible data:

• Entity / brush / face / vertex counts
• Map / entity / brush / face build times
• Custom node spawn count
• Loaded texture count
• Generated material count

Feel free to drop extra ideas in here if you have any @Calinou

The README.md is already becoming a bit bloated, so any hard documentation that isn't necessary for a glance-read understanding of the project should be moved onto the wiki.

From theratikfire on the TrenchBroom discord:

• SKIP and CLIP are special textures used to remove faces from brushes
• SKIP is called 'nodraw' in source (VMF)
• Faces with SKIP should be skipped completely
• Faces with CLIP should be ignored by the visual geo generator, but still considered for collision

I added the plugin to the 3.2 beta to see if any updates will be required for, and it appears to fail currently:

ERROR: Cannot get class 'QuakeMap'.
   At: core/class_db.cpp:343

Doing convex collision for a whole map is valid since the user might be assembling props as individual .map files. May as well support split control for both convex and concave.

Spawn a top-level CSG combiner node, then populate it with CSGMesh nodes in Union mode using brush vertex data.

This should cull away any unnecessary internal faces, but might be a bit performance-heavy at build time and subject to edge cases.

Ideally the user should be able to export a ready-to-go TrenchBroom config for their game, complete with a custom set of entity and face definitions defined within the editor.

This would require a custom resource for the game definition containing an icon (to be resized to 32x32), the data for GameConfig.cfg, and the data for a game-specific .fgd file.

FGD stands for 'Forge Game Data' and is an extension for Hammer's game definition files. Spec is available here:
https://developer.valvesoftware.com/wiki/FGD

Need to create proper step-by-step guides that break down various Qodot features and use-cases, preferably including screenshots for context.

Areas to cover:

• Setup
• Basic map building
• Automatic material / texturing
• Extending the build system via build step classes and build pipelines

As far as I know the per-face data in the .map file isn't sufficient to infer texture shear since UVs are by-plane instead of by-vertex.

Need to figure out the black magic TrenchBroom uses to encodes this and make sure it's properly supported.

Errors are spammed to the console for every face in the map, causing the editor to lock until they finish.

Ideally the distinction between valve-format UVs and everything else should be automatic, but the practical solution right now is to have the geo generation process cancel itself and print a warning to the user.

Currently brushes locate a texture asset directly, create a new SpatialMaterial, and then assign the texture to its Albedo property. This is fine for basic stuff, but doesn't allow the user to provide a custom material.

Couple of options to fix this:

• Make Qodot speak Material instead of Texture
  • User would have to prepare a material for every texture (nope nope nope)
• Expose a default material otion
  • Would force all textures to share a single material (also nope)
• Use a Dictionary to store texture key -> material maps, and provide a fallback material param for textures without a material specified
  • Allows for full per-texture material control
  • Could make it hierarchical by texture subdirectory, or use regex matching on names
  • Should this be a new resource type, perhaps tying into .WAD support?

Should be part of the Static Lighting build options. Create BakedLightmap volume based on AABBs.

Ideally should be available in single, per-entity and per-brush flavours, dependent on how the 'build from node' functionality in the BakedLightmap GDScript interface behaves.

Check the Godot source for the default from-node used when invoking a bake from the editor UI. Will probably shed some light on its behaviour re. Node parents breaking it out of the hierarchy.

The atlased mesh pipeline has small texture wrapping seams on certain edges, visible in the first room of E1M1.

Need to figure out the cause of the misalignment and account for it.

Certain build steps currently run in single mode, when they could be better-paralellized by running in per-entity or per-brush mode to take advantage of pool threads.

Need to sweep through and rewrite where necessary.

Specifically func_group and func_detail. From therektafire:

since func_groups are technically worldspawn as far as QBSP is concerned (and as far as qodot should be concerned as well), func_groups, func_details etc should be combined into single meshes since they are meant to be static level geometry. On the other hand other types of entities would probably be better suited for being imported on a per brush basis, like func_breakable. So what I would do is either add a cfg file to the plugin or an array entry to qodotmap which lists some defaults for which types of things should be imported in which way, and the user can edit it if they want different behavior
Oh and the same goes for collision as well as the visual model

This mainly seems like it would affect the per-entity node spawning pipeline, since the worldspawn no longer exists after the map file has been parsed into other kinds of geo.

VMF
Official Valve Map Format documentation:
https://developer.valvesoftware.com/wiki/Valve_Map_Format

Doom 1/2 Format

Case in point the TECH04_3 on the roof of the E1M1 starting area, it appears to be scaling down to an extremely small size in the U axis.

This feature is specific to Qodot's atlas implementation, and should generate the tiled images in a build step before passing them to the QodotTextureLayeredMesh for direct application.

Once that's done, the improvements made to QodotTextureLayeredMesh can be migrated back to the base TextureLayeredMesh repo.

When the multithreaded generation process completes, QodotMap recursively traverses its children and sets their owned to the active editor tree to make them appear in the scene tree.

This process is expensive and causes the editor to hang momentarily. Much like .map file parsing, it's fine for smaller maps but becomes noticeable for more complex ones like ad_chapters.

If possible this should be done in a threaded manner, and failing that it should be timesliced.

This is destructive, and should be filtered by node class.

Combine all meshes into one and use a texture atlas, presumably with a customized fragment shader to allow for wrapping textures that are looked up?

Should Qodot have its own system for this, since Godot doesn't have anything built-in as of 3.1?

My initial reaction is that building a solution to operate in more general Godot terms would be a better time investment, since each brush gets converted into a standard MeshInstance using an ArrayMesh to hold vertex data.

However, under the current model culling would have to be AABB / Box-Sphere based based, since you can't toggle visibility per face.

A better solution, and one that would take advantage of the .map format in a similar way Quake does, would be to instead create one mesh per brush face. That way each face could update its visibility separately based on simple point-in-frustum + normal-dot-camera checks.

It would be better if ArrayMesh offered the ability to selectively show/hide its component surfaces, but that doesn't appear to be a thing. Could store the vertices in an array somewhere and filter them into the ArrayMesh based on visibility, but that seems like it would be a lot of data-thrashing vs the multiple mesh solution.

Most (if not all) existing Quake-format maps use WAD files instead of loose textures, so Qodot should support a texture import workflow at the very least, and full import/export if possible.

QodotMap should export a script class property that allows the user to select a set of build steps.

It should also define the initial state of the build context, for things like texture loading.

There are still some vertex winding errors indicating incomplete brushes on these two maps, need to figure out which brushes / entities it applies to, isolate them, and fix it.

I noticed this repository is under the CC BY 4.0 license. Since Creative Commons licenses aren't recommended for software, what do you think about using a license like MIT instead?

All node-creation functionality within QodotMap should eventually be modular, to support a variety of rendering workflows.

Case in point from Arkii:
"I'd like to play with rendering it all as one draw call, without culling it might be fine even on decent modern hardware"

Right now that'd require directly inheriting from QodotMap and copy-pasting a ton of existing code around, which isn't a great way to extend. The geo generation stuff needs a refactor anyway since it's getting tall enough to tip over, so might as well do this at the same time.