IDA structs not supporting more nested types about binsync HOT 7 CLOSED

hey mahaloz, nice work. i'm not a user, but you're killing it on this project.

i'm using a lookup table to map the sizes to the flags over on my project over here at https://github.com/arizvisa/ida-minsc/blob/master/base/_interface.py#L113.
it's a little confusing, but it's a double-dictionary, first one takes a native python type, next one takes a size that then returns the flag (you'll only care about the the integermap one). instead of hardcoding the stuff in convert_member_flag (https://github.com/angr/binsync/blob/master/plugins/ida_binsync/ida_binsync/compat.py) you can just use those flags at the source i linked to map them to the flag from the correct size.

from binsync.

Hi @arizvisa, wow! It's amazing to know that other devs for ida plugins actually look at this plugin, haha. Very cool. I read a lot of ida-minsc code when I first started writing some the type backends for the ida plugin. Awesome code man.

i'm using a lookup table to map the sizes to the flags

Thanks for the link, I looked over some of the code and it looks pretty promising. I am however still worried about integer sizes changing between architectures and the like. This code is way better than my dumb map right now, so I will for sure switch some stuff out.

I'm currently looking for some better way to turn some ida type into some know flags. For instance, this crash all happened because of this type: https://github.com/mahaloz/dreamland/blob/8fb74133e2aae04d523c705727ce13c77281440e/structs/dreamer.toml#L21 (struct member with a array type).

I need some way to turn an arbitrary type into a flag. Thanks again for the link :).

from binsync.

Hi @arizvisa, wow! It's amazing to know that other devs for ida plugins actually look at this plugin, haha. Very cool. I read a lot of ida-minsc code when I first started writing some the type backends for the ida plugin. Awesome code man.

Oh wow, awesome man. I'm glad you appreciate it. :-)

i'm using a lookup table to map the sizes to the flags

Thanks for the link, I looked over some of the code and it looks pretty promising. I am however still worried about integer sizes changing between architectures and the like. This code is way better than my dumb map right now, so I will for sure switch some stuff out.

Ah, tru. Yeah, I didn't think you were trying to get the correctly-sized types based on the compiler detected by each of the disassemblers. It's just such a weird problem that I haven't seen others really complain about, which is very interesting. I ended up hacking together a different system outside of all of our reversing/debugging tools to have a declarative syntax that I could still manipulate dynamically in order to deal with weird constraints of different types we encounter within said tools, heh. Type-interop is just like a silent problem that we all solve by seeding from some source, and then finishing up with data-entry for actual usage.

I'm currently looking for some better way to turn some ida type into some know flags. For instance, this crash all happened because of this type: https://github.com/mahaloz/dreamland/blob/8fb74133e2aae04d523c705727ce13c77281440e/structs/dreamer.toml#L21 (struct member with a array type).

Oh yikes. IDA doesn't actually support multidimensional arrays as far as I know except through the idaapi.tinfo_t interface, which isn't even really indexed and requires an explicit flag to calculate the boundaries...You can prove this by going into IDA's structure view, and try setting the typeinfo (via "Y") and watching it consolidate all the dimensions of the array, so then the only way to extract the dimensions is literally via the tinfo_t. Although, I think there's a hook for them that you can probably synchronize with, but you have to use CALC_GDT_LAYOUT or something crazy like that to make sure the sizes are there.

Anywayz, your same goal is actually on my (infinitely-long) todo list, namely mapping those tinfo_t types back into the actual flags... However, I couldn't find a good solution short of mapping the tinfo_t to the one of the sizes in the compiler_info_t (for dealing with type size changes between compiler), and then converting from those into the correct flag via a lookup table. I haven't implemented it, however, because I didn't think that it was a clear path going down that route and I've already written so much code without using tinfo_t that it'd be hella work to refactor for that (and it'd also break backwards compatibility).

However, maybe it'd be better to just ignore ida's structure api entirely, and go the route of using tinfo_t for all your types instead. Unfortunately tinfo_t is hellacious to work with since it's essentially a giant union that doesn't index any of its members or anything. It also has the downside in that unless it's been synchronized to the structure view, you can only see its cross references via hexrays instead of the regular (Ctrl-X) xrefs window. Not only does this requires decompiling everything (so that hexrays has cached it), but I also don't know a way of locating these references programmatically (because they prolly point into ctree).

I need some way to turn an arbitrary type into a flag. Thanks again for the link :).

If you go the compiler_info_t route, or really whatever you end up deciding on, let me know. I might implement the solution you come up with if it's not too tedious, and if you don't mind. ;)

from binsync.

IDA doesn't actually support multidimensional arrays as far as I know except through the idaapi.tinfo_t interface

I don't fully follow, but I can confirm you're right that it works with the tinfo_t interface. I have a function already that works 99% of the time for any non-custom type. When I say non-custom I do mean a type the user did not create. For instance, this in the IDA terminal:

Python>out = convert_type_str_to_ida_type("int[30][30]")
Python>out
int[30][30]
Python>out.get_size()
0xe10

https://github.com/angr/binsync/blob/1f65cae2721a86c5b5146036512fd2da64985056/plugins/ida_binsync/ida_binsync/compat.py#L126

So for the most part, I work only with tinfo_t and my C-like type system and convert between the two, like your map. For me, this has not been a problem yet (though it was hellish to discover all the API I could use to get this done). My only problem now is getting those pesky flags from a given tinfo_t.

For the most part, I usually can just make fully fake flags and then overwrite them using type setting. I do something like this for stack vars which tend to be just like struct members... so there may be some nice hack around this.

https://github.com/angr/binsync/blob/1f65cae2721a86c5b5146036512fd2da64985056/plugins/ida_binsync/ida_binsync/compat.py#L415

Anyway, I have a deadline I want to make of May 28th for some major updates to this software, the biggest of these is a new TypeEngine, which will deal with every type conversion to and from C declarations. We'll see how it goes. Feel free to copy past whatever you like from this project, ofc with a little # from the binsync project and a link on a readme or something ;)...

If you go the compiler_info_t route

I actually never heard of compiler_info_t. I will check it out. Thanks for direction again :).

from binsync.

IDA doesn't actually support multidimensional arrays as far as I know except through the idaapi.tinfo_t interface

I don't fully follow, but I can confirm you're right that it works with the tinfo_t interface. I have a function already that works 99% of the time for any non-custom type. When I say non-custom I do mean a type the user did not create. For instance, this in the IDA terminal:

Python>out = convert_type_str_to_ida_type("int[30][30]")
Python>out
int[30][30]
Python>out.get_size()
0xe10

Ah, I mean multi-dimensional arrays don't exist with regards to the structure member api (ida_struct.member_t) as expressed via member_t.flags. But in regards to what I mean by GDT_CALC_LAYOUT:

Python>ti = internal.declaration.parse("IRP")
Python>ok = ti.get_udt_details(udt, idaapi.GDT_CALC_LAYOUT)
Python>ok
True
Python>udt.size()
0x15
Python>for i in range(udt.size()):
Python> print(hex(udt[i].offset), str(udt[i].type))
Python>
0 CSHORT
10 USHORT
20 PMDL
40 ULONG
60 union _IRP::$CBBBB9F4F0755A16DC8A369061485BEC
80 LIST_ENTRY
c0 IO_STATUS_BLOCK
100 KPROCESSOR_MODE
108 BOOLEAN
110 CHAR
118 CHAR
120 BOOLEAN
128 KIRQL
130 CCHAR
138 UCHAR
140 PIO_STATUS_BLOCK
160 PKEVENT
180 union _IRP::$6B96A96ED958C92F2CB4B83EAB343043
1c0 PDRIVER_CANCEL
1e0 PVOID
200 union _IRP::$66699B8BF83DC91F51A70E4C6E3F33A6
Python>internal.declaration.parse('CSHORT').get_size() * 8
0x10

https://github.com/angr/binsync/blob/1f65cae2721a86c5b5146036512fd2da64985056/plugins/ida_binsync/ida_binsync/compat.py#L126

So for the most part, I work only with tinfo_t and my C-like type system and convert between the two, like your map. For me, this has not been a problem yet (though it was hellish to discover all the API I could use to get this done). My only problem now is getting those pesky flags from a given tinfo_t.

For the most part, I usually can just make fully fake flags and then overwrite them using type setting. I do something like this for stack vars which tend to be just like struct members... so there may be some nice hack around this.

Yeah, awesome. They're definitely like struct members. Also, anything with an identifier (ida_struct.struc_t, ida_struct.member_t) has xrefs to it that you can also get via ida_xref.xrefblk_t.

Anyway, I have a deadline I want to make of May 28th for some major updates to this software, the biggest of these is a new TypeEngine, which will deal with every type conversion to and from C declarations. We'll see how it goes. Feel free to copy past whatever you like from this project, ofc with a little # from the binsync project and a link on a readme or something ;)...

Most definitely. I'll totally let you know as well. It was probably originally gonna be something silly like this (https://github.com/arizvisa/ida-minsc/blob/master/base/database.py#L393), but I'll update AUTHORS.md and include a reference in the "Thanks" section to reference you and other plugin authors that I've reached out to borrow code from.

If you go the compiler_info_t route

I actually never heard of compiler_info_t. I will check it out. Thanks for direction again :).

Cheers, good sir!

from binsync.

@arizvisa If you were wondering how this all played out, it turned out that the bug was actually my negligence to return a flag regardless of the size -- I was returning None which made this crash... just some implementation bug.

My current workflow still seems to be the most solid:

• whenever making any member just pass a fake flag
• always make a ida tinf_t
• overwrite the type with the tinf_t, which will auto-correct the flags for a struct

This works on complex types in structs. For your project though, I would assume you will still need converts.
Also I mentioned your project in the comments of the PR :). Thanks for your help.

from binsync.

@arizvisa If you were wondering how this all played out, it turned out that the bug was actually my negligence to return a flag regardless of the size -- I was returning None which made this crash... just some implementation bug.

Ah, lol. Just saw your commit, freaking python makes certain bugs so non-obvious. Glad it turned out into a less-complex solution. ;)

My current workflow still seems to be the most solid:

* whenever making any member just pass a fake flag

* always make a ida `tinf_t`

* overwrite the type with the `tinf_t`, which will auto-correct the flags for a struct

Awesome. Yeah, that seems sound.

This works on complex types in structs. For your project though, I would assume you will still need converts. Also I mentioned your project in the comments of the PR :). Thanks for your help.

Yeah...unfortunately, I need ways of enumerating cross-references. Especially because tinf_t only has them in hex-rays, which requires decompiling everything..and afaik there's no api for getting them either as hex-rays doesn't expose it in any way other than generating them. :-/

100% appreciate the reference. I'll link yours too in my next release as well which'll probably happen in the next month or two depending on free time. I'm thinking about changing the semantics of some operand structure path logic, and am currently testing the usefulness of wrappers over the default til_t...

...which i haven't done yet, because if i wanted to enable globs or pattern matching for programmatic searches (like full-text even), there's not even hooks that i can use to maintain an index for a large number of types...it's like a dead-end api that only allows 1:1 mapping between the name and ordinal.

Cheers, good sir.

from binsync.