mahlet-inc / hobbits Goto Github PK

This includes:

• updating published docs on https://mahlet-inc.github.io/
• adding new youtube tutorials
• updating and publishing doxygen documentation in the code

Probably and import/export plugin, but further research is needed

If it's not set, could be nice to have the python interrupter for the Python Runner auto-populate if python can be found on the system path. Perhaps a drop down if multiple versions are available.

When I use the Kaitai Struct analyzer plugin on PNGs or BMPs the program closes (I presume it's crashing) with no error messages.
I'm using the pre-built binary for Windows.

The largest component of Linux distribution is lib/ directory, which contains 3 or 4 files of each shared library under different names, e.g *.so, *.so.5, *.so.5.6, *.so.5.6.1. Those files are identical, but each takes separate space on disk. And since most of them are too big for gzip to detect their identity, they take separate space in the archive. Replacing them with hard or symbolic links will work the same, but make the binary package almost 4 times smaller and easier to download.

https://github.com/Mahlet-Inc/hobbits/releases/tag/v0.46.3

win link - https://github.com/Mahlet-Inc/hobbits/releases/download/v0.46.3/hobbits-0.46.3-mac.txz
mac link - https://github.com/Mahlet-Inc/hobbits/releases/download/v0.46.3/hobbits-0.46.3-win.txz

And FSF position is that it is not allowed to use GPLed plugins for a non-GPL program to circumvent GPL virality.

If the main program dynamically links plug-ins, and they make function calls to each other and share data structures, we believe they form a single combined program, which must be treated as an extension of both the main program and the plug-ins.

It may make sense to replace FFTW with something else, i.e. https://github.com/clMathLibraries/clFFT

If you don't want to depend on OpenCL, you can use https://cr.yp.to/djbfft/install.html

As hobbits expands to support more than just digital data streams, it would be good to have short descriptions for each of the views so people new to the tool or new to using a particular view can quickly and easily know what information that view is attempting to display. These descriptions can take many forms:

• Additions to the external docs
• In-application Help docs
• Static short descriptions as part of the view panel
• Short descriptions as a mouseover/hover over text popup
• Short descriptions as part of a "help key" + click

This stuff seems to work with the changes I made in bcbd804, but it's not as clean and intuitive as I would like it to be. A few things to think about:

• The additional inputs to a template should have a little more flexibility. For example, right now it would behave weirdly if a stage got multiple inputs from the multiple outputs of another stage (multiple inputs are treated as completely separate lineages)
• it might be time to reconsider the parent/child container relationships. maybe containers should just have sources and sinks? Maybe instead of a tree view, the bit container selection view should be some kind of graph? (that could get nasty...)

Would benefit both platforms a lot. The one way is to do it directly as a plugin, another way is to separate "raster" widgets to be reusable in other projects too, like Cutter.

Radare2 is a highly-portable cross-platform reverse engineering framework and a toolkit without dependencies. It has support for analyzing binaries, disassembling code, debugging programs, attaching to remote GDB/LLDB, WinDbg servers, rich plugin system (see r2pm), and integration with various decompilers. For example, ghidra decompiler plugin - r2ghidra-dec. It is actively developed and can be easily integrated in various open source and commercial products. I believe, it will be highly beneficial to support these and provide a package for install from r2pm, see the package repository here: https://github.com/radareorg/radare2-pm

For documentation on writing plugins for radare2 see Scripting and Plugins Radare2 Book chapters.

Cutter is a crossplatform Qt/C++ GUI frontend to radare2:

For documentation on writing plugins for Cutter see the official tutorial and the curated list of various popular plugins.

See also rizinorg/cutter-plugins#3

> /Applications/hobbits.app/Contents/MacOS/hobbits 
dyld: Library not loaded: /usr/local/opt/gettext/lib/libintl.8.dylib
  Referenced from: /Applications/hobbits.app/Contents/Frameworks/hobbits-cpython/lib/libpython3.9.dylib
  Reason: image not found
fish: Job 1, '/Applications/hobbits.app/Conte…' terminated by signal SIGABRT (Abort)

I would suggest using macdeployqt for mac builds. It packages Qt a bit more nicely.

Mac 11.5.2, hobbits-0.51.1

Now that there's some proof-of-concept stuff for embedded python from #23 , it should just be a matter of some build engineering and some API refactoring and expansion to make a hobbits python package. At first, this will mostly be useful for making tools like the runner, the tester, and the web server ( #47 ).

I don't have anything specific at the moment, but some thought should be put into making BitContainer metadata better. That means:

• BitContainer metadata should be easy for plugins to read and write (easy in a sense of both code brevity, and general intuitiveness)
• BitContainer metadata should be efficiently implemented so that things like huge frame lists don't create weird performance bottlenecks or stability issues
• BitContainer metadata should retain its flexibility, and maybe even become more flexible

The hobbits runner should have a server mode that is configured with a bunch of templates and can apply those templates to uploaded data. It could work something like this:

Server is started with a collection of template files. It makes an HTTP REST interface with endpoints for each of the template files. A user can then upload a file to the endpoint and receive a URL where the output(s) can be downloaded and maybe a websocket channel to listen for progress updates.

With those server capabilities, it should be pretty easy to slap on a Web GUI.

sometimes the hobbits runner hangs after completing a batch. this never seems to happen in the ci tests, but I saw it happen with a kaitai -> extract batch

right now, the GUI from the release binaries is pretty ugly and sometimes weirdly different from the "reference" GNOME build on my machine. There should be a consistent style on all platforms.

Based on what I've seen in other GUIs and what I remember doing in the past, this will require a combination of Qt style sheets and some QStyle (?) management within the application. We only need one theme, preferably dark-ish.

Here's an example (but I don't really like how this one looks): https://github.com/ColinDuquesnoy/QDarkStyleSheet

It might also be worth using some open fonts and shipping with them. I noticed that the "Monospace" font has a weird load error on Mac.

Hi,

got some question about plugins in python :

• In the JSON file, when I put "decimal" or "integer" type for my parameter, it doesn't change anything, in the both cases it's an integer, is it normal ? Do i miss something ?

• Can we put negative values in our integer field ? And so, can we precise a maximum and a minimum value in the JSON file for a specific parameter ?

• When I try to add a boolean parameter in my JSON file and execute my plugin in Hobbits, I've always this error :

"Operator Plugin Error: Plugin 'test' reported an error with its processing: Other errors:
Failed to parse arg 4"
(here the arg 4 is of course my boolean, if i put more parameters, it will say 'arg 5' or 'arg 6')

and this is my script in python :

def operate_on_bits(input, output_bits, output_info, perso_var, progress):
  if input.bits.size() > 300 and perso_var:  # the perso_var is my boolean
    output_info.add_highligt("python", "mychunck", 20, 3000)

  output_bits.resize(input.bits.size())
  for i in range(0, input.bits.size()):
    output_bits.set(i, input.bits.at(i))  # this is the only other modification in the default plugin (I removed the 'not')
    if progress.is_cancelled():
      return
    progress.set_progress(i, input.bit.size())

and my JSON file :

{
  "name": "test",
  "description": "plugin test",
  "tags": ["test"],
  "script": "main.py",
  "type": "operator",
  "extra_paths": [],
  "parameters": [
    {
      "name": "perso_var",
      "type": "boolean"
    }
  ]
}

When "Byte Raster" tab is selected, and the image is over 100 pixels wide, the interface responds to user actions much slower. On my 2-core 2GHz Celeron when a 160x160 image is displayed, switching between other visualization tabs takes negligible time, but switching to and from "Byte Raster" takes between 2 and 3 seconds. Changing its zoom takes about 5 seconds. Other user actions are processed about as slowly.

Is there any way to make the interface faster when "Byte Raster" is visible?

Observed in Ubuntu Linux for versions 0.22.0 and 0.22.1.

A file generated with the following command

 echo 2f fe b8 9b 22 00 00 00  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  00 00 a4 0f 08 00 1a 06 08 af d0 bc 82 06 | xxd -r -p > file.bin

opened with hobbits with the following ksy:

meta:
  file-extension: bla
  id: servermod
  endian: le
  imports:
    - /serialization/google_protobuf
seq:
  - id: records
    type: record
    repeat: eos
types:
  main_header:
    seq:
      - id: ip1
        type: u1
      - id: ip2
        type: u1
      - id: ip3
        type: u1
      - id: ip4
        type: u1
      - id: body_len
        type: u4le
  record:
    seq:
      - id: len_record
        type: main_header
        doc: Size of whole record, including all headers, footers and data
      - id: ctype
        type: u1
        doc: ctype
      - id: flow
        type: u1
        doc: flow
      - id: cid
        type: u4le
        doc: cid
      - id: did
        type: u4le
        doc: did
      - id: seq1
        type: u8le
        doc: seq1
      - id: command
        type: u2le
        doc: command
      - id: protobuf
        size: len_record.body_len - 24
        type: google_protobuf

When you open the file with the struct in kaitai IDE works correctly, but opening in hobbits when you select the protobuf part (the last 10 bytes - 08 af d0 bc 82 06 ) it's highligthed the start of the file.

If you concatenate some times and select the protobuf of any of the registers always it's highligthed the starting bytes of the file.

The command to generate with 3 concats is:

echo 2f fe b8 9b 22 00 00 00  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  00 00 a4 0f 08 00 1a 06 08 af d0 bc 82 06 | xxd -r -p > file.bin ; echo 2f fe b8 9b 22 00 00 00  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  00 00 a4 0f 08 00 1a 06 08 af d0 bc 82 06 | xxd -r -p >> file.bin ; echo 2f fe b8 9b 22 00 00 00  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  00 00 a4 0f 08 00 1a 06 08 af d0 bc 82 06 | xxd -r -p >> file.bin

May i miss something but the old way to make a plugin in C was to install wizards like said in the tutorial, but I'm unable to find it.
Is their something i miss ? Did the way to do that improved ?

I made some researches and found this, from Adam. But i can't find the file in the Code part.

Can someone help me on this point ?

Thanks !

Even the simpest spec consuming 1 byte fails: EOFError: requested 1 bytes, but got only 0 bytes.

1. Start Hobbits.
2. Import any binary file. (I used an uncompressed bitmap or a ZIP.)
3. Open "Width Framer" tab and "List" tab.
4. Enter any number for Width.
5. Press "Analyze" several times. With each press the correlation coefficients increase — their order of magnitude doubles, until they reach ~1e+214 ant then "nan".

This is clearly wrong.

Observed for version 0.22.0 in Ubuntu Linux.

https://api.kde.org/frameworks/ktexteditor/html/

It has syntax highlighting for yaml and blocks folding. One needs to set mode (not highlight mode, but just mode) into YAML

For Windows probanly QScintilla can be used, but for KDE, I guess, it may make sense to use KTextEditor

Obstacles:

1. there is no libqt5xml5-dev package in Debian & Ubuntu. How have they built KF* packages without that? Debian bug
2. It seems CMake doesn't properly discover KF packages. The workaround is to provide paths to their dirs, but it is extremily strange.

My method of black-box reverse engineering of file formats almost always involves generation of similar files (i.e. with incrementally added records) and binary diffing them. I don't know any good tools for binary diffing (there are tools for binary patching, but they are different beasts, and there are tools for "binary" diffing, where "binary" means an executable format (PE/EFL/Mach-O) filled with machine code of a known arch, so CFG are matched and diffed).

So I usually do the following

1. hd all the files
2. split the known stuff with whitespaces
3. then diff
4. then look at boundaries and try to identify more stuff to be splitted
5. then figure out patterns and encode them in Kaitai

An example of such a preliminary analysis is https://github.com/kaitai-io/kaitai_struct_formats/pull/532/files (LTO was initially analysed the same way).

Unfortunately hexdump misses diffs because the data is fit into grid and the grid is diffed too and interferes.

So we need a binary diffing tool.

Diffing is just a sequence alignment problem, there exist lot of impls of sequence alignment, i.e. python standard library contains a one, so it is not a problem.

What is needeed is

1. the mechanism to influence sequence alignment by specifying predefined boundaries
2. the mechanism to store this kind of info
3. the gui for it
4. interoperability with Kaitai

Such unneeded and undisableable warnings as violations of the badly-designed style guide.

It would be nice to have hobbits packaged in a way that let plugin developers update the application/headers/library easily using a package manager. We could provide the following packages:

• hobbits-devel (requires everything?)
• hobbits (requires everything except libhobbits-devel and hobbits-qt-creator-wizards)
• hobbits-gui (gui binary, requires libs and plugins)
• hobbits-runner (CLI runner binary, requires libs and plugins)
• hobits-core-plugins (core plugin binaries, requires libs)
• hobbits-qt-creator-wizards (wizard templates for Qt creator)
• libhobbits-devel (includes headers for plugin development, requires libs)
• libhobbits (binary libraries, requires Qt 5.12+)

A few non-trivial things that might need to be addressed:

• where do we host these?
• which parts of hobbits-core are "public"? everything right now?
• how do we make the packages? (seems like CMake can help, see #13 )

@KOLANICH

The core library should be cleaned up to stop requiring the Qt widgets module. The functions that relate to the widgets should be broken out into a separate library (hobbits-qtils? hobbits-widgets?)

This will allow CLI things like the hobbits runner to get rid of the widgets module as well.

Hi,
just an idea to select an area in a view with the mouse ;

first onclick (left) define the start of the area in the view
SHIFT + onclick (left) define the end of the area

After use the 'Extractor' (Operator plugin) to choose 'Extract Section" or other extract operations.
or may be new 'right click' access for extract operations.

would this be technically feasible?

Best regards

The current parsing of the kaitai output does not go through lazily parsed chunks. This is described in kaitai-io/kaitai_struct#880

The kaitai struct plugin should do this lazy parsing in order to get closer to functional parity with other kaitai GUIs

Currently, ASCII view seems to replace symbols below 32 and above 126 with dots. But if Hobbits' ASCII tab behaves more like Hiew, more information would be available to the user. For example, it could display symbols from an 8-bit code page for 128-255 and glyphs similar to IBM control symbols for 0-31 and 127. Here are symbol set examples for code pages 437 and 866: https://en.wikipedia.org/wiki/File:Codepage-437.png https://en.wikipedia.org/wiki/File:Octets_in_conformant_CP866_ordered_by_nibbles.png

The view portion of the GUI should support a high level of flexibility and usability. Some ideas:

• arbitrary horizontal and vertical splits
• save last viewed tab of views between application runs
• enable or disable visibility of various view plugins in tabs (maybe independently in split views?)
• maybe add the ability to have free floating view windows

Hi,

I detect a problem when the "split view" is applied after 'import' file or data -> the application crashes ;
the "split view" is possible only before 'import', add other view crashes application after 'import'.

i m using pre-built binaries v0.46.7 (Hobbits (64-bit Windows 2016 msvc2017), Same issue with v0.46.4.
Tested under 2 x Windows 10 pro 64-bit (20H2 & 2004).

Have you already observed this ?

Thanks
Best regards

Kaitai is a declarative language for parsing binary formats. We need plugins that will:

• compile and run the kaitai parser on a bitcontainer (using python?)
• display the parsed information kind of like kaitai Web IDE
• easily allow blobs to be broken off into their own bitcontainers for further isolated analysis

https://kaitai.io/

@KOLANICH

@gdot709 proposed using uppercase letters for bytes, e.g. t16 == T2, s128i80 == S16I10, and t3s32 == t3S4

I think this is easy and good.

The current display plugin interface is pretty simple and flexible, but it needs to be updated to support some new things:

• rendering a display without a GUI/Window/Desktop environment should be possible. Right now the display printer exporter renders the QWidget, but the displays should just be producing images (QImage?)
• a display should be able to use the same ParameterDelegate + AbstractParameterEditor setup for its configuration that the other plugins use (the current "control widget" is too flexible, and tedious to reinvent for every plugin)
• display rendering should be threaded by default so as not to interfere with the responsiveness of the GUI.

A lot of the plugins seem to misbehave in the Mac release in 0.29.0 and the operators crash the program. When built from source on a Mac, I could not replicate this problem. There is no requirement to support Mac, so this will not be prioritized (but it would be nice to not be publishing garbage).

When I have parsed a bit-container with a KS spec, and noticed a suspicious pattern within some blob, it'd be nice to view which field is that blob in a tooltip by hovering it in the view and go to that field in Extractor by ctrl+click.

the min and max input container thing seems like it might make more sense in a ParameterDelegate::validate

I'm not sure, but it's worth revisiting before 1.0 is finalized

So I was wonder if there is any way to just set the width of bit / bytes rasters / etc without performing any framing work.
The width framer messes it up and so I have to attach a sync marker and do it that way...

With Width Framer

Using added sync markers

When attempting to launch hobbits.app v0.44.1 downloaded from the release page on a Mac, it crashes with the following lines in the crash report:

Dyld Error Message:
  dyld: Using shared cache: E2B1A630-68DD-30AC-A036-6AC018B154DA
Library not loaded: /Users/*/libpython3.9.dylib
  Referenced from: /Applications/hobbits.app/Contents/MacOS/hobbits
  Reason: image not found

Inspecting the runtime library search paths in the hobbits executable using otool -L hobbits shows the following:

/Applications/hobbits.app/Contents/MacOS/hobbits:
	@rpath/libhobbits-core.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libhobbits-widgets.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libhobbits-python.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/runner/work/1/a/lib/libpython3.9.dylib (compatibility version 3.9.0, current version 3.9.0)
	@executable_path/../Frameworks/libfftw3.3.dylib (compatibility version 10.0.0, current version 10.9.0)
	@executable_path/../Frameworks/libfftw3_threads.3.dylib (compatibility version 10.0.0, current version 10.9.0)
	@rpath/QtWidgets.framework/Versions/5/QtWidgets (compatibility version 5.14.0, current version 5.14.1)
	@rpath/QtGui.framework/Versions/5/QtGui (compatibility version 5.14.0, current version 5.14.1)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1875.0.0)
	/System/Library/Frameworks/Metal.framework/Versions/A/Metal (compatibility version 1.0.0, current version 211.51.0)
	@rpath/QtCore.framework/Versions/5/QtCore (compatibility version 5.14.0, current version 5.14.1)
	/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.6.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

Using otool -l hobbits and searching for LC_RPATH shows the following two rpaths:

         path @executable_path/../Frameworks (offset 12)
         path @executable_path/../Frameworks/python/lib (offset 12)

However, the library path for libpython3.9.dylib is hardcoded in the executable.

While this can be fixed using install_name_tool as follows:

install_name_tool -change "/Users/runner/work/1/a/lib/libpython3.9.dylib" "@rpath/libpython3.9.dylib" hobbits

This needs to be applied to hobbits-runner and libhobbits-python.1.dylib too.

Likewise, libpython3.9.dylib references libintl.8.dylib installed by Homebrew instead of the one bundled with the software. Fixing all of these with install_name_tool allows hobbits.app to function.

I will look at the build scripts and try to see what's going wrong.

While I am testing on an M1 Mac running macOS 11.1, this problem is not limited to Big Sur or Apple Silicon.

Having an interpreter inside the Python runner plugin can speed up python processing and might help with other things.

@KOLANICH

BitContainers need to be able to hold larger bit arrays. I think it will go something like this:

• BitArray will need a file descriptor with a temporary read/write copy of the loaded file
• Reading or writing from the BitArray at a location will cause the referenced bit and its surroundings
  (a few MB?) to be cached (if they weren't already).
• When a cache is replaced, it will write its changes to the file before before going away
• there will need to be some new utility methods for the situations where the entire QByteArray from BitArray was being extracted
• there will need to be safeguards for gigantic BitContainer metadata (e.g. where every frame is 1 bit) (EDIT: punted to #6)

Compilation fails with:

peakselector.cpp: In member function ‘virtual void PeakSelector::paintEvent(QPaintEvent*)’:
peakselector.cpp:71:18: error: aggregate ‘QPainterPath path’ has incomplete type and cannot be defined
   71 |     QPainterPath path;
      |                  ^~~~

Including QPainterPath with every QPainter fixes it.
Thanks.

The tests have been useful for catching regressions when things are refactored. Having tests for all features of all core plugins would be great.

It also might be nice to have a separate fast test command/argument that only does the small file tests (the gigantic file tests take a while to run and will usually only catch problems in the BitArray caching system.)

Sometime I just want to peek or know I just need a small sample of data from an http call.
It would be nice to be able to specify how much data you want and allow response to be interrupted leaving just the data that had been returned thus far.