warpten / dbclientfiles.net Goto Github PK

Hopefully not a major implementation issue but WDBC and WDB2 don't necessarily have an Index constraint, for example pre-WotLK CharBaseInfo. The client handles this by generating Ids based on the record index on load.

struct CharBaseInfoRec // Alpha 0.5.3.3368
{
  char m_raceID;
  char m_classID;
  int m_proficiency;
  int m_generatedID; // not in file
};

P.S. Great work, I'll be using this for WDBX once write support is in!

Type.GetProperties and associates return an array of PropertyInfo in no particular order. They coincidentally match order in declaration but that might change, as theres, per MSDN:

The GetProperties method does not return properties in a particular order, such as alphabetical or declaration order.

might be a problem in the future

The two-fold approach to arrays leads to a glaring issue (not really a bug).

Given this declaration

[MarshalAs(UnmanagedType.ByValArray, SizeConst = 5)]
public Substructure[] SomeMember {get;set;}

But the corresponding file metadata instead says that the member is not of cardinality 5, the generator will produce this kind of code:

{
    var structure = new ...();
    structure.SomeMember = new Substructure [5];
    for (var i = 0; i < 5; ++i)
        structure.SomeMember[i]= new Substructure();

    // Deserialization happens now
    structure.SomeMember[i] = reader.ReadArray<Substructure>(3);
    // ...

    return structure;
}

User code should never have to hardcode array sizes but instead use SomeMember.Length, which here will yield two extra elements.

This is an even more glaring issue when the declared structure has cardinality lower than what the file declares.

https://github.com/wowdev/WoWDBDefs

Steps needed

• Separate loading logic from parsing logic
• Provide front-facing code that hides away all the type generation code and lets the user specify members using an IEnumerable<some interface>.
• Either expand on the existing code, which means:
  • Runtime type generation: slow, needs to be cached, no user control, not pretty
  • ???

Possible issues:

• Input structurew will need to match file definitions 1:1 to not fail.
• Provide clear errors to the user if something is done wrong.

Surface layer

public interface IMember
{
    tring Name { get; }
    MemberType Type { get; } /* enumeration of int, etc */
    int Cardinality { get; } /* cardinality, default to 1 */
    // ?
}

Example of user code

using (var fileStreram = ...)
{
    var storage = new Storage(fileStream, new DynamicStructure().field("ID", MemberType.Int32).field("Name", MemberType.String).build(), StorageOptions.Default);

    // 0 is the record key (different from ID - it's still the user's task to create a dictionary)
    Console.WriteLine("{0} - {1}", storage[0]["ID"].Integer, storage[0]["Name"].String);

    var allNames = storage["Name"].Strings; // IEnumerable<(int, string)>
}

We currently force get and set to not be null (barring IgnoreAttribute). get is however enough to create a backing field if autogenerated. Maybe allow private protected (or internal protected) setters?

Problems to expect:

• Heuristics for section assignment of each member:
  1. Bitpacking (bit counting if column.max_value < (column.type.max_value >> 8)
  2. Choosing between pallet and common (default value heuristic?)
• Maintain a string pool - whenever an item is added to the container, we need to extract strings from it
• Subsidiary: when is a string pool not needed?

Headers are poorly handled and need a rewrite. stat.

Current implementation:

• A front-facing Header type is provided to the user - it only exposes record count, signature, and uh, that's it I think ? Takes an implementation of IFileHeader or another instance of itself as ctor argument.
• IFileHeader is an interface that acts as a base for all header types, which are defined per-signature. This means that header implementations turn into property hell.

Desirable properties of new headers:

• value types (struct), bonus points for immutability (readonly struct).
• Unsafe.SizeOf<T>() must match the binary size of the header in the file (give or take the 4 bytes for magic identifier).
• Avoid copying as much as possible (ref readonly getter returns).
• Can leverage existing block handling mechanics for serialization.
• Avoid property hell where => throw new NotImplementedException(); is the default "i shouldn't be called" implementation. If you don't have it, don't expose it.

Is this possible?

This is not a pretty solution but it would help avoiding the issue of value type getters returning temporary copies of the underlying types. The problem is that autogenerated getters typically retrieve a field named <prop>k_BackingField, (IsSpecialName returns true) which is compiler-specific. So not exactly portable.

The idea is to not limit the user to reference type substructures:

    public class C3Vector // Runtime exception if value type.
    {
        public float X { get; set; }
        public float Y { get; set; }
        public float Z { get; set; }
    }

    public struct AreaTriggerEntry
    {
        public uint ID { get; set; }
        public uint MapID { get; set; }
        public C3Vector Position { get; set; }
        public float Radius { get; set; }
        public C3Vector BoxSize { get; set; }
        public float BoxOrientation { get; set; }
    }

this is an entirely new can of worms, but something that must be done eventually.

[ ] find heuristics for section assignment
[ ] write more code for emitting serializers

WDC2 and WDC3 allows blizzard to roll out encrypted sections until they push tact keys out. WDC2 isn't known to use this.

There's currently no way to know the type of the index field as declared in the structure, because it only happens once segments are read (this is when field mapping happens).

This leads to issues where StorageDictionary is just unable to check if _keyGetter HAS to be set.

This is currently done in a very ugly way which forces users to decorate the proper field with IndexAttribute, which is pointless and shouldn't be needed since an index is either:

• for wdbc/wdb2: first column, always
• else: column index specified by header

those need to be tested, might be bugged

Possible bug to investigate as i was redacting #2.

if header.flags & 0x04, index is not in record, and thus

        public FileMemberInfo IndexFileMember => FileMembers.FirstOrDefault(m => m.Index == IndexColumn);

is wrong.