jinmo / idapkg Goto Github PK
View Code? Open in Web Editor NEWPackages for IDA Pro (written in python but supports all)
License: MIT License
Packages for IDA Pro (written in python but supports all)
License: MIT License
Starting from version 0.10.0 LIEF had dropped the support for Python 2.7. Since idapkg will install the latest version of LIEF this will results in installation error if the user is using IDA before version 7.4 ( = no Python 3 support ).
Right now a dirty fix is to bind the version of LIEF to 0.9.0 ( python -m pip install lief==0.9.0
will resolve the issue ). However I think it is necessary to find an alternative package for LIEF ( e.g. katai struct like you mentioned in #10 ). The package should be Python2/3 compatible if possible, since not everyone has the latest version of IDA :P
OS: Windows 10
Python: 3.8.5
IDA version: 7.5
IDAPython: v7.4.0
idapkg : 0.1.4
During the installation, the install command shows the following error message:
idapkg version 0.1.4
Will install virtualenv at 'C:\\Users\\bruce30262\\idapkg\\python' since pip module is not found...
Downloading virtualenv from 'https://files.pythonhosted.org/packages/57/6e/a13442adf18bada682f88f55638cd43cc7a39c3e00fdcf898ca4ceaeb682/virtualenv-20.0.21-py2.py3-none-any.whl' ...
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 90, in prepare_virtualenv
ImportError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\bruce30262\AppData\Local\Programs\Python\Python38\Lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\bruce30262\AppData\Local\Programs\Python\Python38\Lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<string>", line 2, in install
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\main.py", line 87, in init_environment
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 110, in prepare_virtualenv
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 108, in handler
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 108, in <listcomp>
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 106, in <lambda>
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmpn4i1x85y.zip\idapkg-0.1.4\pkg\virtualenv_utils.py", line 67, in _install_virtualenv
File "<frozen zipimport>", line 259, in load_module
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmp68526rtt.zip\virtualenv\__init__.py", line 3, in <module>
File "<frozen zipimport>", line 259, in load_module
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmp68526rtt.zip\virtualenv\run\__init__.py", line 5, in <module>
File "<frozen zipimport>", line 259, in load_module
File "C:\Users\BRUCE3~1\AppData\Local\Temp\tmp68526rtt.zip\virtualenv\run\app_data.py", line 6, in <module>
ModuleNotFoundError: No module named 'appdirs'
The ImportError
message will be shown while importing the following modules:
All error can be resolved by using pip3 install <module>
Side note: idapkg cannot find pip on my machine because this line of code
After that, the install script will emit another error:
Creating environment using virtualenv...
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 90, in prepare_virtualenv
raise ImportError()
ImportError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\bruce30262\AppData\Local\Programs\Python\Python38\Lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Users\bruce30262\AppData\Local\Programs\Python\Python38\Lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "<string>", line 2, in install
File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\main.py", line 87, in init_environment
prepare_virtualenv(wait=True)
File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 110, in prepare_virtualenv
__work(handler) if not wait else handler()
File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 108, in handler
def handler(): return ([task()
File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 108, in <listcomp>
def handler(): return ([task()
File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 106, in <lambda>
tasks.insert(0, lambda: _install_virtualenv(path))
File "C:\Users\bruce30262\idapkg\packages\idapkg\pkg\virtualenv_utils.py", line 71, in _install_virtualenv
virtualenv.create_environment(path, site_packages=True)
AttributeError: module 'virtualenv' has no attribute 'create_environment'
The error seems to have something to do with this issue
Don't get me wrong ! I've been playing this plugin for a while and it really is an amazing project :) ๐
However when I was testing the plugin, I encountered some problem and have to find a way to re-install idapkg completely.
To be brief, I have two version of IDA in my computer (Windows 10), and when I was testing the plugin, it kind of screwed up the variable in config.json
( I think it's idausr_native_bases
), making me unable to open ida ( it kept crashing ). After I installed idapkg, I first open version A's ida.exe, then open version B's ida64.exe ( both of them worked perfectly ). And then I found I was unable to open version A's ida64.exe and version B's ida.exe ( both of them crashed ). It took me quite a while to find the root cause and have to delete config.json
& re-install idapkg to fix the issue.
This make me think that the project should provide a way to uninstall the plugin (and document it in the manual), in case the user have decided not to use the plugin anymore. As far as I understand it has something to do with idapythonrc.py
, and it's located in C:\Users\<username>\AppData\Roaming\Hex-Rays\IDA Pro\idapythonrc.py
(Windows) , which is not an easy place for user to reach/discover. It would be nice if we can use a single command in IDA's python shell to remove the plugin completely.
I'm going to write a plugin combining which internally uses angr for some analysis, because the angr is dropping Python 2 support, so I decided to use a package system for easier installation. However the angr is recommended to be installed in a separate virtualenv. But after going through the doc of idapkg, it seems that the whole idapkg is sharing one single virtualenv.
Can we add options to somewhere like info.json in order to make some packages use their own env?
Currently idapkg adds package root to sys.path, and I'm not sure if it's a good way.
Some path candidates that can be added to sys.path (but not sure what's best):
- idapkg/packages
- idapkg/packages/<package root> (current)
- idapkg/packages/<package root>/python
IDAPython adds <IDA installation path>/python
to sys.path by default, and some plugins use this scheme in deployment. (e.g. Ghidra <-> IDA converter) One can just move python/* to package root, but I'm not sure if this is a good way. (It works, and doesn't conflict commonly, but if some files like setup.py
exists and some package uses import setup
or etc, it does matter.)
Here, I like the approach of idaenv which tries to wrap all packages as a python (ex. IDArling supports idaenv). Using setup.py or other setuptools/distutils module enables reusing python's package management system. (One can pip install ./idarling/
or etc though)
OS: Windows 10
Python: 3.8.5
IDA version: 7.5
IDAPython: v7.4.0
idapkg : 0.1.4
While installing idapkg, the install command shows the following warning message:
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:94: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:94: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:100: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:94: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:94: SyntaxWarning: "is" with a literal. Did you mean "=="?
idapkg-0.1.4\pkg\vendor\semantic_version\base.py:100: SyntaxWarning: "is" with a literal. Did you mean "=="?
It seems to install idapkg successfully, however the warning messages' kind of annoying
I think the SyntaxWarning
message was introduced in Python 3.8
Changing is
to ==
should fix the problem
I'm gonna remove dependencies or embed it to make the installation part completely offline, and make it minimal, while I'm not sure if it's possible to bundle virtualenv.
idapkg's uninstaller.py suffers a traditional encoding issues while concatenating the string:
............<omitted>
File "C:/<omitted>/idapkg-master/uninstaller.py", line 32, in <module>
update_pythonrc()
File "C:/<omitted>/idapkg-master/uninstaller.py", line 12, in update_pythonrc
sep_with_ver = SEP[0] + __version__
TypeError: can't concat str to bytes
Using the code sep_with_ver = SEP[0] + __version__.encode()
will fix the issue ( I'm not sure if this is the best solution though ).
Package schema is a subject that improvements can be made on and on, so I made this thread.
This is inspired by Pipfile.lock.
Required:
+ info.json
Optional for IDA:
+ plugins/
...
+ procs/
...
+ loaders/
...
+ til/ sig/ ids/
...
Optional for package:
+ README.md
{
// package name
"name": "Community Headers",
// version used in package management (required)
"version": "0.0.1",
"description": "",
"homepage": "https://example.com",
"installers": ["installer.py"],
"dependencies": {
"other-plugin-id": ">=1.0"
}
}
I was thinking about installing required pip packages, but installer can do that. Since there are many implementation for pip packaging, I'm not sure if I should expose a pip-installing function from package manager.
Maybe a website for validating the scheme and hosting some packages would be good.
As titled, I found that I cannot login to idapkg.com ( it says "Network error" on the web page ). Also I wasn't able to access the repo page ( e.g. https://idapkg.com/p/PythonEditor ). Is everything OK ?
Currently, idapkg uses idapythonrc.py to bootstrap idapkg. This structure makes moving idapkg itself to idapkg/packages possible. (Though ~/idapkg/config.json directory must be hardcoded for now)
Doing this gives some benefits: upgrading idapkg easier. Then it can make one more improvement: making the installation script inlined. Currently, idapkg.py does this.
Normally, the author can bundle native libraries as dependencies, but it'll occupy space too much when it comes to python plugin. So I was just thinking about virtualenv-based python package management. Here's some problems:
So pip recommends programmers to call pip as separate process. IDA uses python interpreter installed into system, so maybe I can find and use the interpreter by determining python.dll used in the process. (sys.executable is set to ida.exe in IDAPython. This breaks virtualenv calls in IDAPython)
Solved.
I think that PEP440 or semantic versioning would be good.
While I was installing idapkg in IDA 7.2, I ran into some error with the following error messages:
Downloading idapkg...
Generating initial config at C:\<omit>\idapkg\config.json
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python27\Lib\threading.py", line 801, in __bootstrap_inner
self.run()
File "C:\Python27\Lib\threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "<string>", line 2, in b
File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\__init__.py", line 4, in <module>
File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\commands.py", line 4, in <module>
File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\package.py", line 22, in <module>
File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\env.py", line 79, in <module>
File "c:\<omit>\appdata\local\temp\tmpcgndtj.zip\idapkg-0.1.0\pkg\env.py", line 35, in __load_version_from_ida
File "C:\Program Files\IDA 7.2\python\ida_kernwin.py", line 895, in get_kernel_version
return _ida_kernwin.get_kernel_version(*args)
RuntimeError: Function can be called from the main thread only
I think it is similar to this issue
I think it's because IDA 7.2 introduced checks for IDA API functions to make sure they can only be called from the main thread.
IDAPython: all functions not marked as THREAD_SAFE in the C++ SDK, will now check that they are being called from the main thread, avoiding possible corruption or crashes
Can you please add a license to the repository?
The absence of a license makes usage and contribution tricky.
I've uploaded a loader for IDA Pro: https://idapkg.com/p/mclf_loader
In case you want the zip file: mclf_loader.zip
When I copy the command in the web page and execute it, I got the following error messages:
Python>pkg.install('mclf_loader', repo='https://api.idapkg.com')
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\commands.py", line 41, in install
spec = _parse_spec(spec)
File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\commands.py", line 21, in _parse_spec
semantic_version.Spec(version)
File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 505, in __init__
subspecs = [self.parse(spec) for spec in specs_strings]
File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 511, in parse
return tuple(SpecItem(spec_text) for spec_text in spec_texts)
File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 511, in <genexpr>
return tuple(SpecItem(spec_text) for spec_text in spec_texts)
File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 425, in __init__
kind, spec = self.parse(requirement_string)
File "C:\Users\<omit>\idapkg\packages\idapkg\pkg\semantic_version\base.py", line 440, in parse
raise ValueError("Invalid requirement specification: %r" % requirement_string)
ValueError: Invalid requirement specification: u'_loader'
BTW if I place the mclf_loader dir under the idapkg/packages folder, it works perfectly fine.
In commands.py, remote()
has the following usage comment:
Find a remote package from given repos.
................(omitted)...................
:type repo: str or list(str) or None
:returns: None if package is not found, else InstallablePackage instance.
:rtype: InstallablePackage
The type of repo
should be "list(str) or None", as stated in install()
.
Am too lazy to send a PR
I was testing changing IDAUSR variable to add directories for plugins/ procs/ loaders/ but it didn't work since IDA cached the list after calling get_ida_subdirs() or etc. I was annoyed since it's not exposed to user, making it impossible to invalidate cache without directly modifying the memory with ctypes. This completely breaks loading of processor/loader since idapkg is loaded after caching the environments.
Wrapping processors/loaders with native .py/.dll solves all of these issue, but needs extra work. e.g. Symbolic linking would not work since it modifies the path. This breaks relative imports.
So my plan is: Writing wrapper function with global constructor, like this.
__attribute__((constructor))
void init() {
init_processor(); // copies LPH structure from wrapped dll
init_loader();
init_plugin();
}
define_wrapper(processor_t, LPH, processor);
define_wrapper(loader_t, LDSC, loader);
define_wrapper(plugin_t, PLUGIN, plugin);
Python wrapper:
import os
filename = '<wrapper path>'
dirname = os.path.dirname(filename)
orig_cwd = os.getcwd()
try:
os.chdir(dirname)
execfile(filename, {__file__: filename})
except:
pass
finally:
os.chdir(orig_cwd)
Indeed IDAUSR's feature is rich, and I need to implement these:
Not copyable (path-sensitive):
plugins/
procs/
loaders/
Copyable:
ids/
til/
sig/
Personally I don't like copying files, but there's no symlink support without Administrator privileges for Windows, and non-NTFS directories (but possible in NTFS, and w/ admin privilege).
Unless, I can use hardcoded offset for IDA (or use FLIRT for IDA binary?). I'm uploading PoC.
I think it would be neat to install packages directly from github (or other git repos).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.