As a consumer of this library and writing code in languages other than Nim, I'd like to read and write files through a simple API.

Nim's bidirectional interfacing promises simple interfacing with C like targets, but an API must be defined and a library build must be shipped. The README already documents an API but the types, such as GffRoot, are complex and challenging to get right when crossing languages. A simpler API with basic data types such as cstring would make interoperability easier.

Example API:

proc readJsonFromGff*(gffFilename: cstring): cstring {.cdecl, exportc, dynlib.}
proc writeJsonToGff*(json: cstring, gffFilename: cstring) {.cdecl, exportc, dynlib.}

Example use from C#:

[DllImport("libnwn_gff.so", CallingConvention = CallingConvention.Cdecl)]
private static extern IntPtr readJsonFromGff(
  [MarshalAs(UnmanagedType.LPUTF8Str)] string gffFilename
);

[DllImport("libnwn_gff.so", CallingConvention = CallingConvention.Cdecl)]
private static extern void writeJsonToGff(
  [MarshalAs(UnmanagedType.LPUTF8Str)] string json,
  [MarshalAs(UnmanagedType.LPUTF8Str)] string gffFilename
);

It seems useful to me that the API be defined and shipped with this project, but I may be missing some considerations. I also understand the maintenance implied by defining APIs, so please let me know how you feel about this. If you agree, I've already worked out the implementation and can PR the changes.

I have installed latest nim and included it in my environment variables. Build.cmd runs fine. When I try to run one of the commands I get this error:

could not load: pcre32.dll

This is on Windows 10.

Output of build:

PS C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim> git pull --recurse-submodules --tags
Fetching submodule src/extlib/neverwinter.nim
Already up to date.
PS C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim> .\build.cmd

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>for %v in (src\*.nim) do nim -o:"bin\%~nv.exe" -d:release c "%v"

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_erf.exe" -d:release c "src\nwn_erf.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_gff.exe" -d:release c "src\nwn_gff.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_key_pack.exe" -d:release c "src\nwn_key_pack.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_key_shadows.exe" -d:release c "src\nwn_key_shadows.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_key_transparent.exe" -d:release c "src\nwn_key_transparent.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_key_unpack.exe" -d:release c "src\nwn_key_unpack.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_net.exe" -d:release c "src\nwn_net.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]
lib\pure\asyncnet.nim(150, 7) Warning: Cannot prove that 'result' is initialized. This will become a compile time error in the future. [ProveInit]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_resman_cat.exe" -d:release c "src\nwn_resman_cat.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_resman_diff.exe" -d:release c "src\nwn_resman_diff.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_resman_extract.exe" -d:release c "src\nwn_resman_extract.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_resman_grep.exe" -d:release c "src\nwn_resman_grep.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_resman_pkg.exe" -d:release c "src\nwn_resman_pkg.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_resman_stats.exe" -d:release c "src\nwn_resman_stats.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_tlk.exe" -d:release c "src\nwn_tlk.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\nwn_twoda.exe" -d:release c "src\nwn_twoda.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim>nim -o:"bin\packets.exe" -d:release c "src\packets.nim"
Hint: used config file 'C:\Users\John\.choosenim\toolchains\nim-0.18.0\config\nim.cfg' [Conf]

Output of attempting to run a command:

PS C:\Users\John\Documents\NeverwinterNights_Projects\neverwinter_utils.nim> nwn_resman_pkg -h
could not load: pcre32.dll

Installation loops 'round and 'round...

$ nimble install neverwinter

...
/home/user/.choosenim/toolchains/nim-1.6.6/lib/impure/db_sqlite.nim(184, 1) Warning: Circular dependency detected. codeReordering pragma may not be able to reorder some nodes properly [User]
...

Simplifies (and speeds up) hooking custom build steps such as a preprocessor.

Also would help if ncs can be emitted to stdout for a custom postprocessing step, but I figure that's particularly niche.

Seeing a failure in the latest version that works fine on an older version (1.4.2)

Error: Could not convert C:\Users\xxxx\Documents\Neverwinter
... Nights\modules\Arabel\acb_room_name.dlg: fatal.nim(53)
... sysFatal
... Error: unhandled exception: value out of range: 4294967295 notin
... -2147483648 .. 2147483647 [RangeDefect]

possibly this:

        "Delay": {
          "type": "dword",
          "value": 4294967295
        },

Either that value is out of whack, or maybe that needs to be an unsigned int?

Currently expressions like n = 3+2; and s = "a"+"b"; actually generate two CONST and one ADD instruction always. These can be evaluated at compile time and replaced with a single CONST.

This is probably best done at codegen stage, when emitting an ADD (or similar) instruction, look back and see if the previous two are consts, and if so, just patch them up.

Alternatively, this can be done at a new postprocessing optimization pass, but that will require rewriting all jumps.

Script example of something that should work:

void main()
{
    int nCount;
    for (nCount = 1;GetIsObjectValid(GetNearestObjectByTag("WP_PEASANT_EXIT",OBJECT_SELF,nCount));nCount++);

    // Do more things here...
}

Works with ; replaced by {} or a while loop.

For it used in Bioware's code see nw_pw_peasant9.nss in Chapter 1 and Chapter 4 of the OC.

@mtijanic asked me to log.

void main()
{
  float iter;
  for (
    iter = 0.0f;  // ERROR: NON INTEGER EXPRESSION WHERE INTEGER REQUIRED
    iter < 10.0f;
    iter += 0.5f  // ERROR: NON INTEGER EXPRESSION WHERE INTEGER REQUIRED
  ) { }
}

This works fine on nwnsc.

A WAR is to explicitly cast first and last for param to int, e.g. FloatToInt(iter = 0.0f)

At the moment this information is not clear and it is necessary to look in the source code to understand what encodings are available. I suggest to add the new command line option outputting the list of all supported encodings

It would be nice to have a set of tests for the script compiler to be used to validate all PRs. CI integration as a stretch goal, but just having something to quickly run locally would be super helpful.

Best if they live in the same repo so future additions can add tests with the same PR.

nwn_erf writes out alphabetically to maintain conformed ERF reproduction, but, without resource types factored as additional criteria, different OS's write out different files with matching resrefs in different orders, creating non-identical ERF files from the same sources when built in different environments.

Restype sub-sorting should unify writeout order.

Initially opened as squattingmonk/nasher#78

When filtering json through "jq", integer floats (0.0, 1.0, ...) are converted to integers without decimals (0, 1, ...)

This blocks the packing of the module with an error as Error: Could not convert src/ux/areas.tpl/area_technical.git.json to area_technical.git: {"type":"float","value":0}

This error could be ignorable by being tolerant and treating int as float: 0 as 0.0, etc...

void main()
{
  int a = -1;
  int b = +1; // ERROR: UNKNOWN STATE IN COMPILER
  int c = +a; // ERROR: UNKNOWN STATE IN COMPILER
}

This works fine on nwnsc.

Today I tried to convert all my json-files in my github folder structure by batch and a for loop, but it throws an error. Below I wrote my syntax and following the error message:

for /r E:\data\mod\ %%i in (*.json) do (nwn_gff -i %%i -o %%~ni)

E:\data>(nwn_gff -i E:\data\mod\uts\sample.uts.json -o sample.uts )
parsejson.nim(522)       raiseParseErr
Error: unhandled exception: E:\data\mod\uts\sample.uts.json(1, 0) Error: { expected [JsonParsingError]

Is my attempt the right way to do this mass conversion?

On Manjaro the pre-compiled binaries throw the following error:

Upon manual compilation the compiler works:

After a discussion we had on Discord the suspected culprit seems to be the static linking of lstdc++ (https://github.com/niv/neverwinter.nim/blob/master/neverwinter/nwscript/compiler.nim#L3).

If the twoda has multiple empty lines at the end of the file they will be filled out with **** when read with readTwoDA, these lines are included if you then write it.

When using nwn_gff to convert a GFF to JSON, the output is different on Mac and Windows:

Looks like floats are formatted differently.
Attaching the source .git file and JSON outputs on both platforms.
al_auri_asm2.zip

nwn_gff 1.5.7 and later does not deserialize jsons with tlk string references correctly, turns them into empty strings instead. Last working version is 1.5.6.

Having #ifdef and similar preprocessor commands as part of the standard compiler would be good since it means code can be supplied for engine, toolset or 3rd party compiler use and all 3 work, compared to running your own preprocessor which limits its use elsewhere.

My main use case is #ifdef to block out code which isn't needed unless, say, debug is enabled, speeding up compile and run times since you no longer have to have long strings added. Having #define would also be useful for use solely in #ifdef and similar although presumably in normal code would end up like any usual const value.

Just code cleanup, we have a lot of cases such as:

		// Add the case statement to the list.
		if (m_nSwitchLabelNumber >= m_nSwitchLabelArraySize)
		{
			int32_t *pNewIntArray = new int32_t[m_nSwitchLabelArraySize * 2];
			for (nCount = 0; nCount < m_nSwitchLabelNumber; ++nCount)
			{
				pNewIntArray[nCount] = m_pnSwitchLabelStatements[nCount];
			}
			m_nSwitchLabelArraySize *= 2;
			delete[] m_pnSwitchLabelStatements;
			m_pnSwitchLabelStatements = pNewIntArray;
		}

This should all go away and m_pnSwitchLabelStatements should be a std::vector
Some such cases:

• m_pnSwitchLabelStatements
• m_pchResolvedOutputBuffer
• m_pSymbolLabelList
• m_pSymbolQueryList
• m_pchDebuggerCode
• m_pSRStack

There's probably a good 300 lines of code to just be deleted with this; and makes it less error prone.

I don't think the mod extension is supposed to be there.

More details on discord: https://discordapp.com/channels/382306806866771978/382306806866771980/759917420772589585

When I use nwn_twoda to tranform a 2da file to a csv file the first column which, gives the data line number, is missing. Is that deliberate or a bug? I know the column is not important to the game engine but for humans trying to navigate around in multiple 2da files it is.

BTW, I tried to build-update to the latest version of these tools so could try to play with the code myself but I cannot get the latest version of nim to install with either choosenim or with the latest nim package.

Sample script

void main() {
    if (1) {
        string s = "AAA";
    }
}

gets compiled down to

     0  JSR, 8                                  
     6  RET                                     
     8  CONSTANT, TYPE_INTEGER, 1               
    14  JZ, 41                                  
    20  RUNSTACK_ADD, TYPE_STRING               
    22  CONSTANT, TYPE_STRING, "AAA"            
    29  ASSIGNMENT, TYPE_VOID, -8, 4            
    37  MODIFY_STACK_POINTER, -4                
    43  MODIFY_STACK_POINTER, -4                
    49  JMP, 6                                  
    55  RET         

the two MODIFY_STACK_POINTER instructions could be a single one. There are other such instances.

I think for this to work, we probably need another pass over the generated NCS before writing it out, and then fixing up the jumps.
Alternatively, we can just keep track of the last instruction, and if it is the same as current, we modify it instead.

Am I doing something wrong? I was experimenting with downloading csv files from google sheets. You can upload a 2da file to sheets by just changing the extension to csv but when you later download the edited file and try to change it back to a 2da file it results in an out of memory error and a huge 2da file.

$ nwn_twoda.exe -i baseitems.csv -o "baseitems.2da" -k "2da" --csv-separator ","
out of memory
$ ls -lrt
total 16256
drwxr-xr-x 1 197609 0 Mar 28 10:48 ../
-rw-r--r-- 1 197609 25870 Mar 29 00:11 baseitems.2dax
-rw-r--r-- 1 197609 25518 Mar 29 00:26 temp.csv
-rw-r--r-- 1 197609 25870 Mar 29 00:50 baseitems0.csv
-rw-r--r-- 1 197609 25984 Mar 29 01:12 baseitems.csv
drwxr-xr-x 1 197609 0 Mar 29 01:13 ./
-rw-r--r-- 1 197609 16525676 Mar 29 01:13 baseitems.2da

The baseitems.csv file is here: https://docs.google.com/document/d/1bbNqVIzaROdaxNB_9-hNZK8fGR9jdkSZUPqP5qDZNBs/edit?usp=sharing

PS: maybe this issue will help you find a bug and so be of some use but I just realized I can copy n paste directly from a 2da into a google sheet and back again with no need to run any conversion to or from csv. I'll just need to change tabs to spaces and maybe trim excess spaces from the first few lines.

PPS: probably not of much interest to you but just for the sake of completeness I'll link this related thread.

Unary operator (like ~ or !) defined for all stock types as producing a string. e.g.:

int n = 3;
string s1 = $n; // s == "3"
vector v = Vector(1.0, 2.0, 3.0);
string s2 = $v; // s == "{1.0, 2.0, 3.0}"
string s = "hello";
string s3 = $s; // s3 == "hello"

This would allow you to print a variable as string regardless of its actual input type, which allows for better code reuse, especially if coupled with a macro system of sorts.

As a stretch goal, it could support custom types too, as:

struct MyStruct { int n; };
string $_MyStruct(struct MyStruct x) { return "MyStruct: { n = " + $x.n + "}"; }

Would need VM changes, but the bulk is compiler-side

I use Python, so I did not understand at all how this thing would help me. I got my nim compiler now to the latest (the one on my system was 1.0.6, which is why nothing worked). Now I see, it support JSON, and sqllite. That, is radical. Is there any kind of way to compile nim to C, and use it in python as a module? The problem is PyNWN is so old it is time to upgrade things. Thanks. Nimble says you can compile to C, but I am very new to Nim and have no idea what in the world to do yet.

When converting the attached dlg file from json to gff, the following error appears:

/home/squattingmonk/Code/neverwinter/nwn_gff.nim(56) nwn_gff
/home/squattingmonk/Code/neverwinter/neverwinter/gffjson.nim(147) gffRootFromJson
/home/squattingmonk/Code/neverwinter/neverwinter/gffjson.nim(135) gffStructFromJson
/home/squattingmonk/Code/neverwinter/neverwinter/gffjson.nim(75) gffStructFromJson
/home/squattingmonk/Code/neverwinter/neverwinter/gff.nim(317) []=
/home/squattingmonk/Code/neverwinter/neverwinter/gff.nim(312) putValue
/home/squattingmonk/Code/neverwinter/neverwinter/gff.nim(286) newGffField
/home/squattingmonk/Code/neverwinter/neverwinter/gff.nim(248) assignValue
/home/squattingmonk/.choosenim/toolchains/nim-1.6.14/lib/system/fatal.nim(54) sysFatal
Error: unhandled exception: value out of range: 4294967295 notin -2147483648 .. 2147483647 [RangeDefect]

It is choking on the following section:

"Delay": {
  "type": "dword",
  "value": 4294967295
}

Looks like it fails due to

Expected Behaviour
JSON output always uses LF line endings, or the correct line endings for the current platform.

Actual Result
The JSON output always ends with a single CRLF line ending. Tested on both Windows and Linux
If pretty output is on, the rest of the output uses LF line endings.

Would it be easy to add a json in/out format option to nwn_twoda (and any other filetypes not yet supporting json format if it would be at all useful but especialy nwn_twoda)? Large 2da files, especially ones with many columns, editing them in a basic text editor is a pain, I was thinking of making a VS Code extension that would sit atop these tools. I was poking at the old TlkEdit code but I'd rather be making something new.

class CScriptParseTreeNode
{
        ...
	CExoString *m_psStringData;
        ...
	CExoString *m_psTypeName;
        ...

There's really no need for these to be pointers that are then nullchecked and realloced everywhere. Just embed the strings directly into the class. The string data will be heap-allocated anyway.

Windows releases have the .exe extension being added twice.

Release 1.6.1

• neverwinter.windows.amd64.zip
• neverwinter.windows.i386.zip

Fairly certain it's these lines in the release.yml workflow causing it.
release.yml:72
for f in bin/*; do mv "$f" "$f.exe"; done
release.yml:86
for f in bin/*; do mv "$f" "$f.exe"; done

The nimble builds already output .exe files. No need to rename them.
nimble build -d:release -d:mingw --cpu:amd64
nimble build -d:release -d:mingw --cpu:i386

I tried installing this both on Windows and Linux through nimble and I get the same error about sqlite. Here's a copy of the verbose response on Linux

$ nimble install neverwinter --verbose
     Info:  Nimble data file "/home/deltreey/.nimble/nimbledata2.json" has been loaded.
    Reading official package list
Downloading https://github.com/niv/neverwinter.nim using git
    Cloning latest tagged version: 1.6.3
  Verifying dependencies for [email protected]
    Reading official package list
   Checking for docopt@>= 0.6.8
     Info:  Dependency on docopt@>= 0.6.8 already satisfied
  Verifying dependencies for [email protected]
    Reading official package list
   Checking for regex@>= 0.11.1
     Info:  Dependency on regex@>= 0.11.1 already satisfied
  Verifying dependencies for [email protected]
    Reading official package list
   Checking for unicodedb@>= 0.7.2
     Info:  Dependency on unicodedb@>= 0.7.2 already satisfied
  Verifying dependencies for [email protected]
 Installing [email protected]
   Building neverwinter/nwn_resman_cat using c backend
  Executing /home/deltreey/.nimble/bin/nim c --colors:on --noNimblePath -d:release -d:NimblePkgVersion=1.6.3 --path:/home/deltreey/.nimble/pkgs2/docopt-0.7.1-387f1ae17f3b7620c7814cf44a52d3d91b8906e9 --path:/home/deltreey/.nimble/pkgs2/unicodedb-0.12.0-4452416471e2fe8726eb6070ed6ea7368171cc09 --path:/home/deltreey/.nimble/pkgs2/regex-0.21.0-3e6b482026bcd9113b0e05026d641248e6c51709 -o:/tmp/nimble_57623/githubcom_nivneverwinternim/bin/nwn_resman_cat /tmp/nimble_57623/githubcom_nivneverwinternim/nwn_resman_cat.nim
Hint: used config file '/home/deltreey/.choosenim/toolchains/nim-2.0.0/config/nim.cfg' [Conf]
Hint: used config file '/home/deltreey/.choosenim/toolchains/nim-2.0.0/config/config.nims' [Conf]
Hint: used config file '/tmp/nimble_57623/githubcom_nivneverwinternim/nim.cfg' [Conf]
..................................................................................................................................
/tmp/nimble_57623/githubcom_nivneverwinternim/private/shared.nim(1, 88) Hint: duplicate import of 'strutils'; previous import here: /tmp/nimble_57623/githubcom_nivneverwinternim/private/shared.nim(1, 8) [DuplicateModuleImport]
.....
/tmp/nimble_57623/githubcom_nivneverwinternim/neverwinter/resman.nim(19, 52) Warning: use command `nimble install checksums` and import `checksums/sha1` instead; sha1 is deprecated [Deprecated]
........
/tmp/nimble_57623/githubcom_nivneverwinternim/neverwinter/private/zstd.nim(12, 6) Warning: A custom '=destroy' hook which takes a 'var T' parameter is deprecated; it should take a 'T' parameter [Deprecated]
/tmp/nimble_57623/githubcom_nivneverwinternim/neverwinter/private/zstd.nim(17, 6) Warning: A custom '=destroy' hook which takes a 'var T' parameter is deprecated; it should take a 'T' parameter [Deprecated]
.
/tmp/nimble_57623/githubcom_nivneverwinternim/neverwinter/key.nim(1, 71) Warning: use command `nimble install checksums` and import `checksums/sha1` instead; sha1 is deprecated [Deprecated]
..........
/tmp/nimble_57623/githubcom_nivneverwinternim/neverwinter/erf.nim(1, 65) Warning: use command `nimble install checksums` and import `checksums/sha1` instead; sha1 is deprecated [Deprecated]
........
/tmp/nimble_57623/githubcom_nivneverwinternim/neverwinter/resnwsync.nim(2, 6) Warning: use command `nimble install checksums` and import `checksums/sha1` instead; sha1 is deprecated [Deprecated]
/tmp/nimble_57623/githubcom_nivneverwinternim/neverwinter/resnwsync.nim(2, 16) Error: cannot open file: std/db_sqlite

Is there an unlisted dependency I need?

Under nim 1.4.0 (latest stable), compilation fails with the following error message:

$ nimble build
  Verifying dependencies for [email protected]
   Building neverwinter/nwn_resman_diff using c backend
/home/squattingmonk/.local/src/neverwinter/neverwinter/key.nim(227, 8) Warning: method has lock level <unknown>, but another method has 0 [LockLevel]
/home/squattingmonk/.local/src/neverwinter/neverwinter/key.nim(230, 8) Warning: method has lock level <unknown>, but another method has 0 [LockLevel]
/home/squattingmonk/.local/src/neverwinter/neverwinter/key.nim(245, 8) Warning: method has lock level <unknown>, but another method has 0 [LockLevel]
/home/squattingmonk/.local/src/neverwinter/neverwinter/resfile.nim(6, 16) Warning: imported and not used: 'util' [UnusedImport]
/home/squattingmonk/.local/src/neverwinter/neverwinter/resmemfile.nim(27, 8) Warning: method has lock level 0, but another method has <unknown> [LockLevel]
/home/squattingmonk/.local/src/neverwinter/neverwinter/resmemfile.nim(3, 16) Warning: imported and not used: 'util' [UnusedImport]
/home/squattingmonk/.local/src/neverwinter/neverwinter/resdir.nim(7, 16) Warning: imported and not used: 'util' [UnusedImport]
/home/squattingmonk/.local/src/neverwinter/private/shared.nim(31, 10) Warning: use exitprocs.addExitProc; addQuitProc is deprecated [Deprecated]
/home/squattingmonk/.local/src/neverwinter/private/docopt.nim(177, 28) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
/home/squattingmonk/.local/src/neverwinter/private/docopt.nim(179, 19) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
   Building neverwinter/nwn_gff using c backend
/home/squattingmonk/.local/src/neverwinter/neverwinter/key.nim(227, 8) Warning: method has lock level <unknown>, but another method has 0 [LockLevel]
/home/squattingmonk/.local/src/neverwinter/neverwinter/key.nim(230, 8) Warning: method has lock level <unknown>, but another method has 0 [LockLevel]
/home/squattingmonk/.local/src/neverwinter/neverwinter/key.nim(245, 8) Warning: method has lock level <unknown>, but another method has 0 [LockLevel]
/home/squattingmonk/.local/src/neverwinter/neverwinter/resfile.nim(6, 16) Warning: imported and not used: 'util' [UnusedImport]
/home/squattingmonk/.local/src/neverwinter/neverwinter/resmemfile.nim(27, 8) Warning: method has lock level 0, but another method has <unknown> [LockLevel]
/home/squattingmonk/.local/src/neverwinter/neverwinter/resmemfile.nim(3, 16) Warning: imported and not used: 'util' [UnusedImport]
/home/squattingmonk/.local/src/neverwinter/neverwinter/resdir.nim(7, 16) Warning: imported and not used: 'util' [UnusedImport]
/home/squattingmonk/.local/src/neverwinter/private/shared.nim(31, 10) Warning: use exitprocs.addExitProc; addQuitProc is deprecated [Deprecated]
/home/squattingmonk/.local/src/neverwinter/private/docopt.nim(177, 28) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
/home/squattingmonk/.local/src/neverwinter/private/docopt.nim(179, 19) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
   Building neverwinter/nwn_net using c backend
/home/squattingmonk/.local/src/neverwinter/private/jser.nim(44, 23) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
/home/squattingmonk/.local/src/neverwinter/private/jser.nim(81, 8) Warning: imported and not used: 'sequtils' [UnusedImport]
/home/squattingmonk/.local/src/neverwinter/private/docopt.nim(177, 28) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
/home/squattingmonk/.local/src/neverwinter/private/docopt.nim(179, 19) Warning: inherit from a more precise exception type like ValueError, IOError or OSError. If these don't suit, inherit from CatchableError or Defect. [InheritFromException]
/home/squattingmonk/.local/src/neverwinter/private/asyncnetudp.nim(7, 1) Error: redefinition of 'recvFrom'; previous declaration here: /usr/lib/nim/pure/asyncnet.nim(918, 6)
       Tip: 5 messages have been suppressed, use --verbose to show them.
     Error: Build failed for package: neverwinter
        ... Execution failed with exit code 1
        ... Command: /usr/bin/nim c --colors:on --noNimblePath -d:NimblePkgVersion=1.3.1 --hints:off -o:/home/squattingmonk/.local/src/neverwinter/bin/nwn_net /home/squattingmonk/.local/src/neverwinter/nwn_net.nim

OS: Linux 64-bit

This is related to #4 I think.
I ran across an instance of a module that I can't round trip the module.ifo from GFF -> JSON -> GFF -> JSON
The second conversion to JSON results in failure.

The first module.ifo below came directly from an existing MOD file.

$ nwn_gff -p -i module.ifo -o module_first_json.json
$ nwn_gff -i module_first_json.json -o module_first_ifo.ifo
$ nwn_gff -p -i module_first_ifo.ifo -o module_second_json.json
/projects/neverwinter.nim/neverwinter/gff.nim(398) readStructInto
Error: unhandled exception: wanted to read 1024 but only got 239 while immediate-loading field Mod_ID of type Void on struct -1 [IOError]
$ grep -A3 Mod_ID module_first_json.json
  "Mod_ID": {
    "type": "void",
    "value": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
  },
$ ls -l module_second_json.json
-rw-r--r-- 1 eladner eladner 0 Aug 17 21:29 module_second_json.json

nim was 0.20.0. neverwinter.nim project was up to date (pulled today, no changes, rebuilt tools for good measure). Also tried with nim 0.20.2 (same results).

As done with EmitModifyStackPointer(). There's about 90 such cases left, and some 40 different opcodes. Stuff like add/sub/mul/div/etc can probably all go into the same function though. Whatever ends up being cleanest and easiest to build optimizations later.

Followup to #51 and #68. -N is already handled as a special case but ~N and !N are not folded. ~N would be useful for some bitmasks, and !N would be helpful once dead code elimination is done.

nwn_erf throws an AssertionError when trying to pack a mod dir containing an empty NSS file. Letting it slide would make the module packing behavior more coherent with the toolset.

nwn_erf will raise an unhandled ValueError exception when extracting or reading from certain NWN modules, both 1.69 and EE. The binaries were downloaded from github, commit 8bf9e8.

gracken@XXXX ~/git/nwn-devbase $ /home/gracken/bin/nwn_erf --verbose -x -f /home/gracken/git/nwn-devbase/server/modules/v4_33.mod
D [2018-04-14T11:38:37] NWN file encoding: windows-1252
D [2018-04-14T11:38:37] Other file encoding: UTF-8
Error: unhandled exception: '' is not a valid resref [ValueError]```

gracken@XXXX ~/git/nwn-devbase $ ~/bin/nwn_erf --version
LICENCE
=======

Copyright niv <[email protected]> and contributors. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are
permitted provided that the following conditions are met:

   1. Redistributions of source code must retain the above copyright notice, this list of
      conditions and the following disclaimer.

   2. Redistributions in binary form must reproduce the above copyright notice, this list
      of conditions and the following disclaimer in the documentation and/or other materials
      provided with the distribution.

THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES,
INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE.


Library and utilities made with love to support Neverwinter Nights development.

You can find the full source code at:

* https://github.com/niv/neverwinter.nim
   for the backing library
* https://github.com/niv/neverwinter_utils.nim
   for the toolkit code

Author contact for help and suggestions: [email protected]

Version: master (8bf9e8)

I can provide one such module that will load in the nwn toolset, but causes nwn_erf to throw the exception.

Under nim 0.20.0, I get the following error when trying to convert json back to gff:

nwn_gff -i base.gic.json -o base.gic
/home/squattingmonk/Code/neverwinter/nwn_gff.nim(46) nwn_gff
/home/squattingmonk/Code/neverwinter/neverwinter/gffjson.nim(129) gffRootFromJson
/home/squattingmonk/Code/neverwinter/neverwinter/gffjson.nim(119) gffStructFromJson
/home/squattingmonk/Code/neverwinter/neverwinter/gffjson.nim(91) gffStructFromJson
/home/squattingmonk/Code/neverwinter/neverwinter/gff.nim(318) []=
/home/squattingmonk/Code/neverwinter/neverwinter/gff.nim(313) putValue
/home/squattingmonk/Code/neverwinter/neverwinter/gff.nim(286) newGffField
/home/squattingmonk/.choosenim/toolchains/nim-0.20.0/lib/system/assign.nim(243) FieldDiscriminantCheck
/home/squattingmonk/.choosenim/toolchains/nim-0.20.0/lib/system/fatal.nim(39) sysFatal
Error: unhandled exception: assignment to discriminant changes object branch; compile with -d:nimOldCaseObjects for a transition period [FieldError]

This happens with any file that I've generated with nwn_gff, but here's the file I was testing above:

{
  "__data_type": "GIC ",
  "Creature List": {
    "type": "list",
    "value": [
      {
        "__struct_id": 4,
        "Comment": {
          "type": "cexostring",
          "value": ""
        }
      }
    ]
  },
  "Door List": {
    "type": "list",
    "value": []
  },
  "Encounter List": {
    "type": "list",
    "value": []
  },
  "List": {
    "type": "list",
    "value": []
  },
  "Placeable List": {
    "type": "list",
    "value": [
      {
        "__struct_id": 9,
        "Comment": {
          "type": "cexostring",
          "value": "Floor Lever - 1"
        }
      }
    ]
  },
  "SoundList": {
    "type": "list",
    "value": []
  },
  "StoreList": {
    "type": "list",
    "value": []
  },
  "TriggerList": {
    "type": "list",
    "value": []
  },
  "WaypointList": {
    "type": "list",
    "value": []
  }
}

Compiling with -d:nimOldCaseObjects does seem to fix the issue, but from the error message, I assume it's a temporary fix, yes?

The #line <line_number> <filename> directive is emitted by many preprocessors as a way to tell the next stage compiler which un-preprocessed line a certain source line refers to.
Consider:

1  #if 0
2  // deadcode
3  #endif
4  compile_error_here

After preprocessing with cpp(1), line 4 becomes line 1, and then if that is fed into nwn_script_comp, it'll report an error at an incorrect line.
Most preprocessors have a way to emit code like this

#line 1
compile_error_here

which would make the compiler report correct lines. This makes it easier to hook any preprocessor as a build step prior to calling the compiler.

When a 2da entry is specified with a comma and no whitespace, it is not correctly escaped with quotes when converted to a csv:

           LABEL                                                               REF
1         Spider,Ice                                                         ****

Results in:

LABEL,REF
Spider,Ice,****

Expected:

LABEL,REF
"Spider,Ice",****

Currently, the way to write multiline string literals is basically:

string s = "Hello, \n" +
                "World!\n";

which is both cumbersome to write and incurs a runtime cost as the concatenation happens when the script is loaded (see also #51). Proposing a new syntax for multiline raw literals.

I already have a patch that supports the following syntax:

string s = @"AAAAAA
BBBBB
  CCCCC
DD\nEE\x50FF\\


GGGGGG ""hello"" GGGGG\\


";

This is basically exactly the same as C# verbatim strings: All characters in @"..." block are treated literally, including new lines and backslashes. Only "" has a special meaning in that it doesn't close the literal, but is interpreted as a single ".

I guess consider this issue an RFC for the syntax.

I've run into a bic assertion error and need some help understanding what's wrong. I'm using nwn_gff to go from bic to json. The command completes successfully if I use any of the standard localvault bics, but when I use bics from my servervault the conversion will sometimes fail. Logs:

$ nwn_gff --version
[...]
neverwinter 1.4.5 (/955f9e, nim 1.4.8)

$ nwn_gff -i /tmp/aluviandarkstar.bic -l gff -o test.json -k json --verbose
D [2021-09-28T21:11:14] NWN file encoding: windows-1252
D [2021-09-28T21:11:14] Other file encoding: UTF-8
gff.nim(379)             readStructInto
Error: unhandled exception: Expectation failed: not hasKey(into.fields, lbl) [ValueError]

Failing bic: aluviandarkstar.bic.gz

Any idea what's wrong?
What purpose the lbl expectation serve?

Currently, the for loop syntax is

for (expression_opt; expression_opt; expression_opt) statement

, but it should be

for (statement; expression_opt; expression_opt) statement

Specifically, this should be possible for (int i = 0; i < N; i++) {...}

The scope of the first statement is the scope of the entire block. In other words, the above is equivalent to

{
   int i;
   for (i = 0; i < N; i++) {...}
}

struct Dummy {
  int a;
  int b;
  int c;
};

struct Dummy a;
struct Dummy b;

string TakeStruct(struct Dummy d)
{
  return "?";
}

void main()
{
  int discern = 1;
  TakeStruct(discern ? a : b); // ERROR: INCORRECT VARIABLE STATE LEFT ON STACK
}

This code works correctly if Dummy only has one member.

This code works correctly on nwnsc.

WAR is to extract the ?: operator into a discrete if/else.

Error: unhandled exception: .nimble\pkgs\neverwinter-1.4.3\neverwinter\gff.nim(452, 13) totalSz == loader.io.getPosition - previousIoPosition [AssertionDefect]

GffErrors.zip

Two utis are included in the zip, both don't cause an error by tweaking the LocalizedName's field value/strref

For ar_gravewormskin set the STRREF to -1. I assume this item was an edit copy of a standard item so originally has a STRREF.

gs_item057 is a bit weirder and was fixed by adding an extra character to the value of LocalizedName Language ID 0 box: Soul Shell to Soul Shell2. Using less characters than Soul Shell works too. Deleting language id 0 also works.

Note if you look at it in GffEditor: The value for language id 0 is showed for language id 2, but they are different. The correct value for language id 2 does show if you delete language id 0. (At least I assume it's correct, I don't speak the language)

Edit: I Should mention that if I deleted both LocalizedName and add one with language ID 0 of exactly 10 characters it causes the above error too, so it may be LocalizedName's of exactly 10 characters.

I didn't check if output is correct.

Running self-built neverwinter 1.6.3 (core-master/d1a3a5, nim 2.0.0) on Windows 11.

Attempting to output to stdout fails, as below:

> nwn_gff -i module.ifo.json -k gff
IFO V3.2fatal.nim(53)            sysFatal
Error: unhandled exception: gff.nim(700, 9) `io.getPosition == 8 + ioPosStart`  [AssertionDefect]

(Notice it succeed at outputting the result of lines 698 and 699 before failing)
Trying to direct output to a file results in similar but different failure

fatal.nim(53)            sysFatal
Error: unhandled exception: gff.nim(757, 9) `io.getPosition == ioPosStart + 56 + structs.len * 12 + fields.len * 12`  [AssertionDefect]

Attempting the reverse (nwn_gff -i module.ifo -k json) operates as expected.
Identified trying to accommodate stdin/stdout for @nwnt, which utilities the same write function. I've not tried stdout before to know if this is new.

Occasionally the toolset generates bizarre module id's that don't convert both ways with the tools. Converting from GFF -> JSON works fine. Ths error occurs when converting back to GFF.

Error: unhandled exception: ./ifo/module.ifo.json(2693, 19) Error: { expected [JsonParsingError]

  "Mod_ID": {
    "type": "void",
    "value": "ÿÿÿÿ^A^@^@^@^@^@^@^@ã^A^@^@"
  },

That's /////wEAAAAAAAAA4wEAAA== in base64 or the binary below:

11111111 11111111 11111111 11111111 
00000001 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 
11100011 00000001 00000000 00000000

Attempting to use nwn_gff on Windows 10 produces the following error (command included)

nwn_gff -i .nasher\tmp\module.ifo -o module.ifo.json
gff.nim(534) readGffRoot
Error: unhandled exception: Expectation failed: header.structOffset == 56 [ValueError]`

Also attempted by someone on Linux for whom nwn_gff works as expected on their own files.
File is attached. It is from a 'default mod' - the bare minimum the toolset creates before you can save and close. Also tried the area001.are from the same and it failed with the same error.

module.ifo.zip

void main()
{
  float a = 0.f; // ok
  float b = 0f; // ERROR: PARSING VARIABLE LIST
  float c = 0; // ERROR: MISMATCHED TYPES
}

Case b works fine on nwnsc, haven't tried case c; but I think both should be supported.