koffeinflummi / armake Goto Github PK

Armake fails to binarize arrays that contains two macros, with the first one occuring being named like the second one, but with an additional suffix.

Repro:

#define MACRO   ""
#define MACRO_X ""
arr1[] = {MACRO, MACRO_X}; // works
arr2[] = {MACRO_X, MACRO}; // fails

Installing armake 0.5-1 from the AUR outputs

.
.
.
  CC  src/keygen.c
src/keygen.c: In function ‘generate_keypair’:
src/keygen.c:115:27: error: dereferencing pointer to incomplete type ‘RSA {aka struct rsa_st}’
     custom_bn2lebinpad(rsa->n, buffer, length / 8);
                           ^~
make: *** [Makefile:25: src/keygen.o] Error 1
==> ERROR: A failure occurred in build().
    Aborting...
:: failed to build armake package(s)

Currently only prefix is written to header. Argument for additional header entries would be nice (and required for ACE3 and CBA to move).

Example:

• version = <git hash>

Whenever I try to

armake_x64.exe unpack test.pbo somedir

the armake process crashes with a "The process has stopped working" dialog. The same happens for the 32bit binary as well. Here, test.pbo was created with build, but the crash happens on any pbo created by other tools as well (cpbo / PBO manager / BI tools). Running as administrator has no effect.

According to the library trace, https://pastebin.com/w28LuPV7, it seems to crash during or just after processing arguments, though that's where my windows debugging experience ends.

Is there any library that I could be missing, specifically needed by unpack? Any other ideas?

I'm running Windows 10 Pro (x64), in Czech mutation. Using version 0.5.1.

Thanks.

"" in a config value seems to get derapified to \":

$ cat test.cpp
value = """";
$ armake binarize test.cpp test.bin
$ armake derapify -f test.bin test.cpp
$ cat test.cpp
value = "\"";

Currently only private key and target PBO are valid parameters, armake sign uses private key name for sign name.

ACE3 for example adds hash of last git commit to the sign name, which means we basically have to rename it. Would be nice if armake sign would also be able to take a custom (different than private key) sign name.

Several Arma "escaped" strings do not count as escaped strings. But some of these strings have no other way to be properly escaped and work in Arma. See examples below:

AresDisplays = "\achilles\ui_f\scripts\";

These double qoutes also break armake. The false positive is indicated by the above line (it was included in the next file)

onSetFocus = "[_this,""RscAttributeRank"",'AresDisplays'] call (uinamespace getvariable ""Achilles_fnc_initCuratorAttribute"")";

After further playing around one way to bypass the issue is to use no quotes at all. This works in armake and Arma.

AresDisplays = \achilles\ui_f\scripts\;

Version: 8b88b46

  PBO  @acre2/addons/acre_sys_prc148.pbo
input buffer overflow, can't enlarge buffer because scanner uses REJECT

No header files in that component were changed since I could last build it just fine, seems like a change in armake causes it.

This one gave me a headache. It seems that binarization/rapification sometimes fails if strings have a certain length.

I did some tests and this is the draft result:

#define MACRO_12 "0123456789012"
#define MACRO_13 "01234567890123"
#define MACRO_14 "012345678901234"
#define MACRO_15 "0123456789012345"
#define MACRO_16 "01234567890123456"
#define MACRO_17 "012345678901234567"
#define MACRO_18 "0123456789012345678"
#define MACRO_TEST "012345678"

arr12[] = {MACRO_12,MACRO_TEST,"Test","",""}; // works
arr13[] = {MACRO_13,MACRO_TEST,"Test","",""}; // does not work
arr14[] = {MACRO_14,MACRO_TEST,"Test","",""}; // does not work
arr15[] = {MACRO_15,MACRO_TEST,"Test","",""}; // works
arr16[] = {MACRO_16,MACRO_TEST,"Test","",""}; // works
arr17[] = {MACRO_17,MACRO_TEST,"Test","",""}; // works
arr18[] = {MACRO_18,MACRO_TEST,"Test","",""}; // works

This is true if the length of MACRO_TEST is >8. If it is <=8, everything's fine.

Smaller modifications resolve this issue (sometimes):

arr13[] = {MACRO_13,MACRO_TEST,"Test",""}; // works
arr14[] = {MACRO_14,MACRO_TEST,"Test",""}; // works
arr13[] = {MACRO_13,MACRO_TEST,"Test","","",""}; // works
arr14[] = {MACRO_14,MACRO_TEST,"Test","","",""}; // works

Deleting the third element "Test" helps as well.

I do not have a clue regarding the root cause...yet 😄

Having an option to show the file/folder structure (just as a list) and unpacking a single file without unpacking the whole PBO would be a great feature to have.

I don't know what this is. Maybe some type of obfuscation?

config (2).zip

Mikeros DeRap and BI's CfgConvert both fail. But report that the raP at the start was unexpected which is ofcause bullshit on a binarized file.

Platform: GNU/Linux

Leaks detected by valgrind:

• in preprocess.c, the malloc() calls for constants[2].value and constants[3].value will re-alloc a 1 byte memory slice for each recursive call of preprocess(). The memory allocated by the parent invocation of preprocess() is lost.
• in filesystem.c, the traverse_directory_recursive() calls scandir() which allocates memory in the namelist array. This memory is not entirely freed if the calls to traverse_directory_recursive() or callback() returns a non-zero result .

Not a leak, but a bug: in filesystem.c , the array containing is created like this:

char containing[strlen(target)];

Later on, this operation is made:

strcpy(containing, target);

Problem: the terminating zero byte from the source string will not fit into the destination.
Solution: char containing[strlen(target) + 1];

Hey Koffein, not exactly sure where to post this, but I was able to get the armake tool working in OSX with some slight modifications.

1. I had to convert the soft tabs in the Makefiles to actual tabs (I assume it's just a git auto conversion issue)
2. I had to add -I/usr/local/opt/openssl/include to the CLIBS arguments
3. I had to change the install locations to /usr/local instead of just /usr

    mkdir -p $(DESTDIR)/usr/local/bin
    mkdir -p $(DESTDIR)/usr/local/share/bash-completion/completions
    mkdir -p $(DESTDIR)/etc/bash_completion.d
    install -m 0755 $(BIN)/armake $(DESTDIR)/usr/local/bin
    install -m 0644 completions/armake $(DESTDIR)/usr/local/share/bash-completion/completions
    install -m 0644 completions/armake $(DESTDIR)/etc/bash_completion.d

El Capitan's System Integrity Protection causes a lot of issues with just /usr :(

Here is a list of all shaders https://gist.github.com/dedmen/90691e56c2dd77bc152ce0a48caf47ba

SuperExt's ID is 131 (Eliteness knows them)
If I remember I will send you a full list of Shaders and their according ID on monday

I'm getting these error's randomly for my addons.

https://gist.github.com/JasperRab/98e505387092b735d418bd149e0bf909

In the run above only addon main and russians are finalised the other ones are not done and pbo's aren't build for them. Also it's not always the same addon that has the error, sometimes they work all pretty rare tough, most of the time one gives the error.

Running it on windows with the following batch file.

https://gist.github.com/JasperRab/3e3a010c145e7a7f714f7b4c9f4a1beb

Is this something with windows or something with my shitty batch scripting 😛 or something with armake?

Thanks.

warning: Constant "QEFUNC" is being redefined without an #undef in line 16. [redefinition-wo-undef]
is thrown for ACE3's script_macros.hpp line 16, however that define is undefined on line 14.

This is on another project that includes ACE3's script_macros.hpp, not tested on ACE3 itself, however both #undef and #define happen in ACE3's file.

Similarly like ReSharper does it. Could be useful to mitigate some issues that are only picked up by armake and don't affect actual application performance.

For example:

// ReSharper disable once ReplaceWithStringIsNullOrEmpty

with #79 so that SuperExt can work.
When building https://cdn.discordapp.com/attachments/105781923573456896/449707944771321885/wolf_kh.7z

I cannot open the p3d in Mikeros Eliteness Debug: lod size mismatch.
I'm very happy that the rvmat contains the Emissive Stage.. BI and mikeros tools remove it when packing the p3d. I heard mikeros paid tools work.

But when trying to place the model Arma crashes.

BinarizeFail.zip
Armake vs Mikero. Files are different but they should be the same. And armake one is crashing Arma when trying to render. Debinarizing using CfgConvert both files give back the same correct output

p3dOutputs.zip
P3D outputs mikero vs armake. Armake is smaller but contains atleast a reference to my z\wolf\addons\KH\tex\lod0Tex_StairMat_EM.paa Mikeros is bigger, 0 results when searching for EM. Although it was in the binarized rvmat that mikero should be packing.

Empty LODs crashed the game. I "fixed" this issue for now with this amazing commit: 3d6c94f.

That of course could cause issues down the line (for example with empty land contact lods), but I was out of ideas and wanted to get everything else out of the way first.

One day this needs to be fixed properly (thus the issue).

(darkened more than when using BI tools)

The binarisation parser throws an error on the keyword delete. But in ArmA 3 configurations there is no such keyword delete.

This keyword needs to be removed.

Here is a unbinarized pbo:
wolflogo.zip
The rvmat's emmisive should make the model glow brightly in the night. If I pack unbinarized then it does exactly that.
If I build using armake the model doesn't have any emmisive and is just black in the night.

Looking into the binarized model using Eliteness it returns
Emmisive[]={0,0,0,1};
and Emmisive[]={0,0,0,1}; for the two materials. Although it should be
emmisive[] = {100,100,100,100}; and emmisive[] = {1000,10,10,1000}; respectively.

For some reason when the rvmat is binarized it's emmissive is reset to default 0,0,0,1

For those mods that have used PboPrefix.txt for Mikero tool purposes, would be useful for armake to support this as well as $PBOPREFIX$ files.

Would be nice to have a list of what each exit code means.

... or if the target for building is a folder, or a file without the proper permissions etc.

Proper error messages should be printed instead.

Operating system

Windows

Description

When trying to build mission folders with python script on machine where user is not admin armake shows message:

error: failed to create temp folder

Possible reason?

Currently armake uses C:\Windows\Temp as it's temp folder. On windows every user has it's own temp folder inside AppData (C:\Users\*username*\AppData\Local\Temp) where programs save their temp files.

Occurs when inheriting animations from another CfgModel entry, or inheriting animation attributes from another model's animations.

Unpacking a pbo don't work properly on windows 10.
The only thing that is happening is that a $PBOPREFIX$ file is created and written to all files seams to be mashed in there.

Requirements:

• Windows 10
• armake (v0.6.3)

Step to reproduce:

• armake unpack -f path/to/pbo path/to/unpackFolder
• Check the unpack folder

I use this to document all the misc issues I find. That I don't know the appropriate answer to, to make a fix PR.

https://github.com/KoffeinFlummi/armake/blob/master/src/p3d.c#L1789
Makes no sense. num_lods is unsigned int, it can't be negative.

https://github.com/KoffeinFlummi/armake/blob/master/src/derapify.c#L572
target is never initialized. And only used for string comparison with buffer.

Recently I was signing some .pbo files and I discovered some signing errors. Means:
If I create a signature with the BI tools, the result is different to what armake produces.
I compared file hashes to make sure that the files are different at all. Although, comparing those files in a Hex-Editor is also a valid solution to discover the error.
The problematic Parts are the last 128 + 4 + 128 Bytes of a .bisign file.

The interesting thing is that I only can not sign files which contain .bin files. But this has to work.

a3lib.py produces also the same results - same signature hash.

Armake argument: -i "/mnt/e/Arma 3/ArmaWork/a3"
Include: #include "\a3\3den\ui\macros.inc"
Error: error: Failed to find \a3\3den\ui\macros.inc.

I tried to add a --verbose parameter to binarize but because the preprocessor can only print to a file handle I'd have to duplicate every output to also print to stdout.
Or read the file in again and print that which is a bit too cumbersome.

And to make a preprocess option I'd have to print to a temp file and read it instead instead of being able to do it in memory.
Implementing it like that feels too crusty to me so I rather let that be done by people who know this C stuff.

Hi there,

After a fair share of debugging and troubleshooting while trying to build a .pbo on my Archlinux machine, I've figured out that armake, at least in its build command, does not support XFS filesystems.

Indeed, on a XFS filesystem a given dirent's d_type attribute is always set to 0, thus making the traverse_directory_recursive a noop.

I'm currently working around this issue by doing everything in my /tmp directory which is not using XFS, but this might prove difficult for users of CentOS/RHEL 7 and higher to use armake.

v0.5.1

When building multiple folders with a script like:

#!/bin/sh
for d in */; do
    target=${d%/};
    armake build $d $target.pbo;
done

The warnings aren't clear enough on which folder/pbo is affected.

All of the following errors are from building CBA A3 on the following https://github.com/jonpas/CBA_A3/commit/42f7e8ee03da9e79c51ee745feae3360baf2f744

Excessive Zeroes

#define REQUIRED_VERSION 1.54
requiredVersion = REQUIRED_VERSION;

becomes

requiredVersion = 1.540000;

__EVAL

https://github.com/jonpas/CBA_A3/blob/armake/addons/linux/CfgEventHandlers.hpp
becomes

class Extended_PreStart_EventHandlers {
    class cba_events {
        init = "__EVAL([\"call eventsile preProcessFileLineNumbers '\x\cba\addons\events\XEH_preStart.sqf'\", \"call eventsile preProcessFileLineNumbers '\x\cba\addons\events\XEH_preStart_Linux.sqf'\"] select (productVersion select 2 <= 154))";
    };
};
class Extended_PreInit_EventHandlers {
    class cba_common {
        init = "__EVAL([\"call commonile preProcessFileLineNumbers '\x\cba\addons\common\XEH_preInit.sqf'\", \"call commonile preProcessFileLineNumbers '\x\cba\addons\common\XEH_preInit_Linux.sqf'\"] select (productVersion select 2 <= 154))";
    };
    class cba_events {
        init = "__EVAL([\"call eventsile preProcessFileLineNumbers '\x\cba\addons\events\XEH_preInit.sqf'\", \"call eventsile preProcessFileLineNumbers '\x\cba\addons\events\XEH_preInit_Linux.sqf'\"] select (productVersion select 2 <= 154))";
    };
};

(see commonile and eventsile instead of compile - just an example, all lines with __EVAL are like that)

requiredAddons Failure

This one is weird, it looks fine after derapifying and all, but Arma still reports the following error on startup:

Addons 'cba_help' requires addon 'cba_hashes'

cba_jr.pbo also throws a bunch of UBC warnings, apparently due to an error in requiredAddons not being respected. Derapifying shows them correctly though.

paa2img crashes on windows (paa2img.c:422). I'm assuming it's an out-of-bounds error as it is reported the Width to be >32000 where it should be 512.

Strangely, it doesn't crash with small 8x8 paa files.

When building a pbo file with binarizing, the lexer throws an error of syntax error, unexpected T_LBRACE, expecting T_EQUALS or T_LBRACKET if a given config file contains class references.

class test; //<- throws an error
class tester: test {};

The lexer might interpret that this would be an actual class (new defined).

A config with a string element like
foo = " bar";
(there's a tab in the string in front of "bar")
becomes after rapify and derapify
foo = "\tbar";
i.e. the tab character 0x09 gets derapified as \t within the string.