carmineos / rage-toolkit Goto Github PK

The data pointers in shader parameters block are not pointing to the embedded textures blocks when the drawable has an embedded txd

ResourceSimpleArray currently inherits from ListBase which has a workaround to cache block length.
This actually isn't required as covariance isn't a problem in this case because ResourceSimpleArray is never used with blocks of type IResourceXXSystemBlock or blocks of variable length (except in ShaderParametersBlock_GTA5_pc which uses is temporary but won't use it anymore once IPackedBlock interface is implemented)

This means that we can be sure that block length is always Count * BlockLength of T
We should also consider refactoring all the base collections blocks to implement common .NET collections interfaces.

Some blocks like EmitterRule and EffectRule have embedded arrays of KeyframeProp followed by an embedded list which points to the same array.
We should wrap such data in another block to ensure that stuff isn't moved around when the GetReferece is invoked on the list?

I load the TXD like so:

        var ytd = new TextureDictionaryFileWrapper_GTA5_pc();
        ytd.Load(file.FullName);
        Console.WriteLine(file.FullName);

Then export again doing nothing in between the last line and this line:

        var output = Path.Join(file.Directory.FullName, "[done]", file.Name);
        Console.WriteLine(output);
        ytd.Save(output);

The input file size is 4,699kb output is 5,201kb.
When attempting to load into game some of the textures dont load and some do?
Digging into it further OpenIV when attempting to import the original dds files it duplicates them?
Not sure if this could be an issue with naming or something else but thought I'd try be in depth since I've been at this for about 5 hours now.
Please let me know if I'm being dumb or this is a bug because I cant seem to figure it out.

We should define a format to export/import data from/to binary resources.

• The format should be platform agnostic (even if we only support PC resources for now)
• The format should be modular (pgBase blocks can either be root of a resource or embedded into another resource)
• The format should use an easy syntax to be easily parsed by third party tools (like a Blender I/O plugin)

openFormats

openFormats seem a valuable format although it has no public specs and everything is strictly coupled to openIV software, it only supports a few rage resource types and furthermore it has some few little inconsistencies:

• how many attributes can a node have?
• why sometimes an array has a count attribute but sometimes it doesn't?

CodeWalker XML

CodeWalker XML on its side supports almost all the rage resource types, but it's very verbose and very inconsistent.

Another format to consider could be glTF (either json or binary) which is easily extended using "extensions" so any software which already handles glTF will only need a plugin to handle extensions for the rage engine specific data, in addition it also supports external binary buffers.
Although has still some limitations which would require too many work on extensions and currently it only supports a Y+ up coordinates system by design which would require to transform bases of all the transformations data in rage formats on export/import.

Seems that blocks SimpleList64<T> always have the Capacity field set to 1 when the array is empty.
ResourceSimpleList32_64<T> instead seems to always have an array of Capacity > Count (not exactly equals), infact BVHNodes array has some nodes left to read although they seem to be all zeroes except for a field set at 1 (defualt init of a value type?), should we read (and write back) them too?

PgDictionary should have its entries sorted by hash

The ResourceDataReader block pool currently will create a duplicated block if you read a derived type after a base type at the same position.

Considering 81e55a0 :
Reading the shader parameters first will add the Texture Data blocks to the pool.
If the ShaderGroup has an embedded texture dictionary it will also read blocks of type TextureDX11 (which derives from Texture ) which are the same blocks and so they are at the same position in the resource.
At this point it will create a duplicated object as the cached one is not of type TextureDX11 but a base Texture. The block pool should replace the block with the most derived one but also somehow update all the blocks which referenced to the old one to now reference the replaced one.

hi, are typed meta structures like in indilos fork planned?

https://github.com/indilo53/gta-toolkit/tree/master/RageLib.GTA5/ResourceWrappers/PC/Meta/Structures

Try to reduce allocations when GetReferences and GetParts are invoked.

Actually blocks like BVHNode, BVHTreeInfo, BoundPrimitive, BoundVertex, BoundMaterial could be used as structs which should improve a lot the read/write speed and as well reduce the required allocations when dealing with #bn files.

The current DataReader/DataWriter implementation from dev/performance branch doesn't support swapping endianess on such structs as they would just swap the whole byte arrays.
An idea could be to add an interface which adds a SwapEndianess method and all such struct blocks will have to implement that (to manually swap each own field) if endianess needs to be swapped in DataReader/DataWriter

The list blocks should be replaced with value types, they are part of the structure data of the blocks which contain them, they are never referenced by another block, the actual references are the arrays the point at.
This will improve invoking GetParts which won't have to iterate on such "parts" and will drastically reduce number of allocations specially in resources like Particles.
They are all 16 bytes structs (pointer to array, count, capacity, padding).

The involved blocks are:

• SimpleList64
• ResourceSimpleList64
• ResourceSimpleList32_64
• ResourcePointerList64

Some blocks seems to store 4x4 matrices in row-major order, some examples seems to be:

• BoundComposite.CurrentMatrices
• BoundComposite.LastMatrices
• FragDrawable.Unknown_B0h
• FragDrawable.Unknown_108h_Data

TODO: check all the matrices and update DataReader/DataWriter to handle both single matrices and arrays

We should consider reworking the whole meta wrappers design.

• Recheck all the known structs for possible errors
• Add all the new know structs from other projects (CodeWalker, alexguirre's dumps)
• Fix unrequired EnumInfo being mistakenly written into the resource
• Consider hardcoding Structure/Enum hashes names and entries names
• Consider hardcoding StructureInfo/EnumInfo structs as they are static data
• Design a system Jenkins dictionary which will always be the first choice for resolving hashes
• Design a user Jenkins dictionary which will be used as fallback and will load/save user's strings
• Move MetaTool core to RageLib/RageLib.GTA5.Services
• Consider keeping the current MetaInformationXml (de)serialization as fallback for users' custom structs

Then of course check PSO/RBF.

Define If a binary file has to be imported compressed and/or encrypted, probably based on the file extension.

The code needs a general refactor, we should consider adopting an .editorconfig file.

Some more things to consider:

• Rename IResourceFile wrapper because it uses the same name of the actual IResourceFile in RageLib.Resources
• Rename RageLib.Resources namespaces to match the folders structure
• Change accessibility of padding/unrequired fields in resource structs to be private
• Change referenced blocks in resource structs from fields to properties
• Refactor all the classes for resource wrappers
• Consider enabling Nullable types

DataReader should get a JenkinsDictionary injected to collect strings everytime the ReadString method is invoked

This is a reminder to delete Geometries block from references of DrawableModel block.
Geometries block is a part block and shouldn't be listed as reference too.

Todo: delete https://github.com/carmineos/gta-toolkit/blob/master/RageLib.GTA5/Resources/PC/Drawables/DrawableModel.cs#L99

I've fixed locally will open a pull request shortly.

DirectXTex is only required to pack/unpack DDS textures, we could consider dropping the C++/CLI project and rely on a third party solution.

Some projects to consider:

• DirectXTexNet (nuget)
• ImageSharp.Textures
• Stride.TextureConverter (nuget)
• TeximpNet

Implement Rebuild method for all the block that require to update some fields to be coherent with parts/references blocks.