cea-sec / sibyl Goto Github PK

View Code? Open in Web Editor NEW

529.0 529.0 54.0 2.3 MB

A Miasm2 based function divination.

License: Other

Makefile 0.58% C 5.82% Python 89.33% C++ 3.46% Dockerfile 0.44% Java 0.37%

sibyl's People

Contributors

Stargazers

Watchers

sibyl's Issues

How sibyl can identify a function ?

I am interested to work with siby , the most interesting thing is that sibyl can detect the name of the function (strsep , strchr , strncmp ....) , how sibyl can do this ? it has a database of templates and it is just comparing ?

Python>identify_all("x86_32", "ABIStdCall_x86_32")
Launch identification on 3056 function(s)
Current: 65.45% (sub_0x80b6d8c)| Estimated time remaining: 49.80s
0x08057258 : strncmp
0x080576ac : strsep
0x08057b48 : strcmp,strncmp
0x080571cc : strchr
0x08057244 : strlen
0x08057704 : strspn
0x08057b30 : strtok

Code needs to be updated for Python3

I was trying to update this to python 3 , but there are some files that were missing when working with the latest version of Miasm.

Yes, I know to work with Miasm stable 1.1 but even that is python2 and it as everyone knows has been dead for a year now.

I was going through slowly updating it until I ran into this

inside

    from miasm2.expression.modint import mod_size2int
ModuleNotFoundError: No module named 'miasm.expression.modint'

and indeed there is no file called modint in Miasm. Now this despite being an error of not finding something in MIASM is a problem with Sibyl using/referencing old code.

only support IDA7.0 above?

Hi,

It seems that the IDA script find.py only support IDA7.0 above ! Only IDA7.0 above has the module ida_kernwin!

Can not dissamble and find function that is here:(

Hi, I faced the question that i can't solve. Yeah, it is a firmware.
First, that i want to start is why sibyl can not even launch test at some addresses? For example, I launch sibyl for one address and after debugging I guessed that launch_tests was not started:


**sibyl find -p -v -t string task 0x103D0                                                                      
Guessed architecture: arml
Found 1 addresses
Found 3 test cases
1 / 1
Address  | Candidates**

After debugging I figured out that, for addresses, which sibyl can work with, in emulation process(qemu), unicorn returned END_ADDR of emulation ( 0x1337babe),


**> /home/vitalym60/.local/lib/python2.7/site-packages/unicorn/unicorn.py(340)reg_read()->322419390L
-> return reg.value
(Pdb) s
--Return--
> /usr/local/lib/python2.7/dist-packages/sibyl/engine/qemu.py(226)__getattr__()->322419390L
-> return self.mu.reg_read(self.pc_reg_value)
(Pdb) getattr(self.cpu, self.ira.pc.name)
322419390L**

and after it, tests are launched.
But for some functions( example above), unicorn return only started address and emulation stopped and tests for guessing wouldn't be started:

if getattr(self.cpu, self.ira.pc.name) != END_ADDR:
   raise UnexpectedStopException()

Because of that sibyl can't find function that is there, for example:
IDA shows that at ROM:00182412 memcpy AND after decompiling this function in pseudo C code and building in binary - sibyl detect memcpy.
But now:

 **~/Downloads/test  sibyl find -p -v -j python -t string -a arml ware.bin 0x00182412                                    
Found 1 addresses
Found 3 test cases
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
1 / 1
Address  | Candidates  

 ~/Downloads/test  sibyl find -p -v -j qemu -t string -a arml ware.bin 0x00182412                                      
Found 1 addresses
Found 3 test cases
1 / 1
Address  | Candidates**  


 ~/Downloads/test  sibyl find -p -v -j gcc -t string -a arml ware.bin 0x00182412                                   
Found 1 addresses
Found 3 test cases
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
WARNING: cannot disasm at 182412
1 / 1
Address  | Candidates

Exactly for this binary, Sibyl can't find any functions.

What the hell is going on?)

Binary Ninja plugin

Hi,

I wrote a simple Binja plugin for using Sibyl.
As of now I'm running it in a single thread but maybe I'll go the way of your plugin and use subprocess.

Anyway, if you can add a link to it in your repo it would be very nice :)

https://github.com/kenoph/binja_sibyl

(The reason why I did not add it as an ext is that binja requires a plugin.json file in the root of the repo, along with __init__.py... I don't want to pollute your repo)

EDIT: Oh yeah, forgot to thank you for the tool. When I read what it does I had an "aha" moment.

Release 0.3

Here is the TODO list for the v0.3 (non-definitive):

Be pip compliant
Integrate others function mining sources (maybe angr)
Add ARM/Thumb information from IDA
Add a debug/explain action to obtain more information on a given recognition attempt
Add regression tests for configuration (dump, configuration file, info gathering)
Fix TestIsCharset, which returns to many false positive
Support Windows (via Unicorn?)
Add Python3 support (now that Miasm supports it)

ImportError: No module named core.utils

Hi,
Thank you for your work, this looks amazing :-)
Upon installing Sibyl, I encountered some install errors. The first one it that the python cli tool imports miasm2, which appear to have be renamed to "miasm".

I replaced every occurrence to miasm2 accordingly in Sibyl. And then I have a weird bug:

python2 run_all_tests.py   
Traceback (most recent call last):
  File "run_all_tests.py", line 6, in <module>
    from learn import test_learn
  File "/home/vladimir/tools/Sibyl/test/learn/__init__.py", line 1, in <module>
    from .run_tests import test_learn
  File "/home/vladimir/tools/Sibyl/test/learn/run_tests.py", line 11, in <module>
    from sibyl.testlauncher import TestLauncher
  File "/usr/lib/python2.7/site-packages/sibyl/testlauncher.py", line 26, in <module>
    from sibyl.engine import QEMUEngine, MiasmEngine
  File "/usr/lib/python2.7/site-packages/sibyl/engine/__init__.py", line 3, in <module>
    from sibyl.engine.qemu import QEMUEngine
  File "/usr/lib/python2.7/site-packages/sibyl/engine/qemu.py", line 1, in <module>
    from miasm.core.utils import pck32, pck64
ImportError: No module named core.utils

However:

python2                                           :(
Python 2.7.16 (default, Mar 11 2019, 18:59:25) 
[GCC 8.2.1 20181127] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from miasm.core.utils import pck32, pck64
>>>

So, what am I doing wrong? Because I have no idea what the heck is going on.

No update?

Unable to import find.py in IDA

How do you use IDA7.0 under Linux and successfully import find.py, I will get the error of not finding the file.

Unable to recognize ABI for void functions in IDA

Hello,

It seems Sibyl (the IDA stub) is unable to recognize the ABI when it finds void functions in IDA.

I believe this issue could be resolved in modifying the gtype_matcher regex as the latter tries to match the keyword before the function arguments.

# int __cdecl(int, int) -> __cdecl
gtype_matcher = re.compile(".+ ([^\(]+)\([^\)]*\)")

Maybe this issue could occur with another keywords. As a reminder, the IDA functions types are defined here: https://www.hex-rays.com/products/ida/support/idadoc/1361.shtml

Using Sibyl under windows: multiprocessing forking errors

from the sibyl installation folder on windows I run:
C:\Python27\python.exe find.py d:\path\to\binary -j python -a arml ABI_ARM 0x1768

This gets me:

File "find.py", line 148, in <module>
    p.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "C:\Python27\lib\pickle.py", line 636, in _batch_appends
    save(tmp[0])
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thre
ad.lock

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 380, in main
    prepare(preparation_data)
  File "C:\Python27\lib\multiprocessing\forking.py", line 495, in prepare
    '__parents_main__', file, path_name, etc
  File "D:\Sibyl\find.py", line 148, in <module>
    p.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 258, in __init__
    cmd = get_command_line() + [rhandle]
  File "C:\Python27\lib\multiprocessing\forking.py", line 358, in get_command_l
ine
    is not going to be frozen to produce a Windows executable.''')
RuntimeError: 
            Attempt to start a new process before the current process
            has finished its bootstrapping phase.

            This probably means that you are on Windows and you have
            forgotten to use the proper idiom in the main module:

                if __name__ == '__main__':
                    freeze_support()
                    ...

            The "freeze_support()" line can be omitted if the program
            is not going to be frozen to produce a Windows executable.

Note that there are two errors (?).

miasm2 link in readme gives 404

error while using sibyl

root@ubuntu:/usr/local/bin# sibyl
Usage: /usr/local/bin/sibyl [action]

Actions:
Traceback (most recent call last):
File "/usr/local/bin/sibyl", line 33, in
key=lambda action: action.name)]
File "/usr/local/bin/sibyl", line 33, in
key=lambda action: action.name)]
AttributeError: 'ActionDesc' object has no attribute 'name'

On latest ubuntu 17.04 x86 and latest miasm

ERROR

i tried to work with sibyle , i run this command but it generates this error .

python2 find.py /home/younes/Bureau/a.out ABIStdCall_x86_32 0x400410

Guessed architecture: x86_64
Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "find.py", line 66, in do_test
    possible_funcs = tl.run(address, timeout_seconds=timeout)
  File "/home/younes/Bureau/Sibyl/sibyl/testlauncher.py", line 181, in run
    self.launch_tests(test, address, *args, **kwargs)
  File "/home/younes/Bureau/Sibyl/sibyl/testlauncher.py", line 169, in launch_tests
    status = test.tests.execute(launch_test)
  File "/home/younes/Bureau/Sibyl/sibyl/test/test.py", line 236, in execute
    if not callback(init, check):
  File "/home/younes/Bureau/Sibyl/sibyl/testlauncher.py", line 144, in launch_test
    self.abi.prepare_call(ret_addr=0x1337beef)
  File "/home/younes/Bureau/Sibyl/sibyl/abi/abi.py", line 83, in prepare_call
    self.vm_push(element)
  File "/home/younes/Bureau/Sibyl/sibyl/abi/x86.py", line 33, in vm_push
    self.jitter.push_uint32_t(element)
AttributeError: jitter_x86_64 instance has no attribute 'push_uint32_t'

Name conflicts in stubs

Hi :)

Sibyl by default includes two stubs: win_api_x86_32.py and linux_stdlib.py. Both stubs define different functions with the same name: get_fmt_args. But this breaks some of the stub functions in win_api_x86_32.py that call get_fmt_args, e.g. msvcrt_sprintf with the following error:

TypeError: get_fmt_args() takes exactly 3 arguments (4 given)

To avoid this issue, we have to specify a stub file explicitly.

Don't support UC_MOD_THUMB

Hi, am i right that sibyl can't work with emulation at UC_MOD_THUMB (unicorn thumb mode didn't use) ?

Because in "qemu.py" is using only:
uc_mode = unicorn.UC_MODE_ARM + unicorn.UC_MODE_LITTLE_ENDIAN

z3 translation crashes

ImportError: No module named miasm2.analysis.machine

After installing the latest version of miasm, it seems to break sibyl based off how it tries to import miasm2 when that no longer appears to be a thing (cea-sec/miasm@944806c).

I'm not sure if I'm doing something wrong, perhaps changing all "miasm2" to "miasm" will help?

Release 0.2

Here is the TODO list for the v0.2:

func action failed when using sibyl cli

I use function discovering of sibyl cli but it is error:
./sibyl func ~/Downloads/quarkslab.elf
Traceback (most recent call last):
File "./sibyl", line 46, in
load_action(guessed[0], sys.argv[2:])
File "/usr/local/lib/python2.7/dist-packages/sibyl/actions/init.py", line 33, in load_action
return getattr(mod, actiondesc.classname)(args)
File "/usr/local/lib/python2.7/dist-packages/sibyl/actions/action.py", line 36, in init
self.run()
File "/usr/local/lib/python2.7/dist-packages/sibyl/actions/func.py", line 90, in run
for addr in fh.guess():
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/func.py", line 191, in guess
for address, value in self.votes.iteritems():
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/heuristic.py", line 29, in votes
self.do_votes()
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/func.py", line 178, in do_votes
super(FuncHeuristic, self).do_votes()
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/heuristic.py", line 21, in do_votes
for name, vote in heuristic(self).iteritems():
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/func.py", line 102, in pattern_matching
for find_iter, vaddr_base in _virt_find(data, pattern):
File "/usr/local/lib/python2.7/dist-packages/sibyl/heuristics/func.py", line 72, in _virt_find
for s in virt.parent.ph:
AttributeError: 'ELF' object has no attribute 'parent'

It is elfesteem problem? If it true, can you give me a version of elfesteem to fix it. Thanks you

PYCParser requirement issue

Calling sibyl find produces an error if pycparser is not installed. Do you think that it could be useful to add a requirements,txt that will install it?

Release 0.1

Here is the TODO list for the first release (0.1):

API stubs: do they work with the qemu engine, the same way as in miasm?

Dear Sibyl developers,

I am writing regarding an issue I have with a function I am trying to identify.

This function is from a windows driver and imports ntoskrnl.memset. I have written a stub function to handle it but I could not find a way to use it with the (default) qemu engine.

The problematic block is :

.text:0000000000028ADC                 mov     [rsp+arg_10], rbx
.text:0000000000028AE1                 mov     [rsp+arg_0], rcx
.text:0000000000028AE6                 push    rbp
.text:0000000000028AE7                 push    rsi
.text:0000000000028AE8                 push    rdi
.text:0000000000028AE9                 push    r12
.text:0000000000028AEB                 push    r13
.text:0000000000028AED                 push    r14
.text:0000000000028AEF                 push    r15
.text:0000000000028AF1                 sub     rsp, 100h
.text:0000000000028AF8                 mov     ebx, edx
.text:0000000000028AFA                 xor     edx, edx        ; Val
.text:0000000000028AFC                 mov     rdi, r8
.text:0000000000028AFF                 mov     rsi, rcx
.text:0000000000028B02                 lea     rcx, [rsp+138h+Dst] ; Dst
.text:0000000000028B0A                 lea     r8d, [rdx+40h]  ; Size
.text:0000000000028B0E                 call    memset
.text:0000000000028B13                 mov     eax, 10h
.text:0000000000028B18                 cmp     ebx, eax
.text:0000000000028B1A                 cmovg   ebx, eax
.text:0000000000028B1D                 xor     r9d, r9d
.text:0000000000028B20                 movsxd  rdx, ebx
.text:0000000000028B23                 mov     r8, r9
.text:0000000000028B26                 cmp     rdx, r9
.text:0000000000028B29                 jle     short loc_28B3E

In my sibyl.conf configuration I added a custom_ntoskrnl.py stubs file. It contains a function for memset : a copy/paste from the miasm code for the same function in user-land.

[find]
stubs = /data/miasm/stubs/custom_ntoskrnl.py,/data/miasm/stubs/linux_stdlib.py,/data/ida-scripts/miasm/stubs/win_api_x86_32.py

import logging

from miasm2.jitter.csts import PAGE_READ, PAGE_WRITE, PAGE_EXEC
from miasm2.os_dep.common import heap, windows_to_sbpath
from miasm2.os_dep.common import set_str_unic, set_str_ansi
from miasm2.os_dep.common import get_fmt_args as _get_fmt_args
from miasm2.os_dep.win_api_x86_32_seh import tib_address

def ntoskrnl_memset(jitter):
    ret_ad, args = jitter.func_args_cdecl(['addr', 'c', 'size'])
    jitter.vm.set_mem(args.addr, chr(args.c) * args.size)
    jitter.func_ret_cdecl(ret_ad, args.addr)

With llvm, it works, but not with qemu. From the following extract (generated by using self.jitter.verbose_mode():

>>> Tracing instruction at 0x28adc, instruction size = 5
>>> Tracing instruction at 0x28ae1, instruction size = 5
>>> Tracing instruction at 0x28ae6, instruction size = 1
>>> Tracing instruction at 0x28ae7, instruction size = 1
>>> Tracing instruction at 0x28ae8, instruction size = 1
>>> Tracing instruction at 0x28ae9, instruction size = 2
>>> Tracing instruction at 0x28aeb, instruction size = 2
>>> Tracing instruction at 0x28aed, instruction size = 2
>>> Tracing instruction at 0x28aef, instruction size = 2
>>> Tracing instruction at 0x28af1, instruction size = 7
>>> Tracing instruction at 0x28af8, instruction size = 2
>>> Tracing instruction at 0x28afa, instruction size = 2
>>> Tracing instruction at 0x28afc, instruction size = 3
>>> Tracing instruction at 0x28aff, instruction size = 3
>>> Tracing instruction at 0x28b02, instruction size = 8
>>> Tracing instruction at 0x28b0a, instruction size = 4
>>> Tracing instruction at 0x28b0e, instruction size = 5
>>> Tracing instruction at 0x51262, instruction size = 6     // Entered memset after call here
Invalid memory access at 0x0L                                           // Invalid memory access just after

After trying to solve this, I couldn't succeed.

Is there a way to have stubs in Sibyl with the qemu engine ?
Do I have to switch to another engine (llvm, gcc, etc) ?

Thank you very much !

Update API to Miasm release v0.1.1

As Miasm now uses versioning, it would be fine to have Sibyl following Miasm stable releases, avoiding APIs breaks.

Update to LocationDB
Update learn action

cea-sec / sibyl Goto Github PK

sibyl's People

Contributors

Stargazers

Watchers

Forkers

sibyl's Issues

Recommend Projects

Recommend Topics

Recommend Org