Linux: Kmsg Unsupported kernel implementation - 3.2 about volatility3 HOT 16 CLOSED

That sounds like it could be a really nice way to handle it. If i can help at all let me know.

Thanks, let me give it a crack first and then you can tweak it if you want improvements to it... 5:)

from volatility3.

Hi, just passing by on the issue :)

I found that printk_log is included in the kernel starting from 3.11-rc4 : https://elixir.bootlin.com/linux/v3.11-rc4/A/ident/printk_log. Doing so, it is missing from 3.2 : https://elixir.bootlin.com/linux/v3.2/A/ident/printk_log.

Here is the definition diff :

• https://elixir.bootlin.com/linux/v3.2/source/kernel/printk.c#L147 # before
• https://elixir.bootlin.com/linux/v3.11-rc4/source/kernel/printk/printk.c#L248 # after

Regarding the 3.2 definition, isn't __log_buf simply a char array ?

edit: there is also an intermediate implementation here : https://elixir.bootlin.com/linux/v3.10.108/source/kernel/printk.c#L251 (#define LOG_ALIGN __alignof__(struct log)). It seems that log was renamed printk_log a few releases after.

from volatility3.

That makes sense @Abyss-W4tcher - thanks for digging into it. I guess we need to add an extra option into to pull out the entries in this case.

from volatility3.

@eve-mem the PR #1075 will make this work.
Two new ring buffer implementations for older kernels were added:

• 3.5 < kernels
• 3.5 <= kernels < 3.11

@ikelos once #1075 is merged, this ticket can be closed.

from volatility3.

"producer": {
      "name": "dwarf2json",
      "version": "0.4.1"
    },

It's actually the samples from the vol foundation page linked from in the readme. You can get the ISF here: https://downloads.volatilityfoundation.org/volatility3/symbols/linux.zip
I like to use that sample as I think it's used in the auto tests when commiting - it might be worth updating them? That's a separate issue completely though.

@eve-mem I'm not sure why those ISFs have wrong symbol types. I will try to find out.

from volatility3.

Maybe it's possible to add a requirement for the ISF producer for a plugin, so that it would only work on version made by dwarf2json higher than x version? Would limit them to working only on ISFs made by dwarf2json though. (which I'm sure is 99.999% of all ISFs, apart from the odd one crafted by hand when you can't get debug symbols - but those users will probably be alright getting past a check like that.)

What do you think?

Absolutely, we recently discussed this with @ikelos in one of the PRs and we agreed that's necessary. Not sure if @ikelos has any update with this. I haven't had the chance to take a look yet.

from volatility3.

Could even do something a little hacky like this to point users in the right direction? It would only fix it for this particular issue rather than doing it generically though.

Hmm, in my opinion, this seems to be an issue with the generated ISF. It needs to have the correct type; otherwise, we'll end up double-checking the types every time we use object_from_symbol(), accompanied by a warning message, which doesn't seem practical to me. Let's see what @ikelos thinks about it.

from volatility3.

I like to use that sample as I think it's used in the auto tests when commiting

Cool, anyway even if those samples are used in the "auto tests", they must be using different ISFs as the PR passed all the tests.

from volatility3.

Ok, so we've got a few options:

• We can ban all dwarf2json files that are too old (we'll need to regenerate all the ones we provide in the pack, so a recently downloaded pack won't trip the issue). This saves a lot of complexity because the symbol loader just fails to load it, but it then invalidates that symbol table for all plugins (whereas clearly some still work just fine).
• We can try and add a requirement, but that gets tricky. Do you want all symbol tables blocked, anywhere in the config tree, what happens if that requirement appears twice but with two different values? Difficult to set a global consequence for an option designed to be used by a plugin? Strictly the plugin should check the value of the config option and then act upon it, so instead probably:
• Don't bother with the requirement, and just provide a framework method that would "check all symbol tables, or check just this symbol table, to see if it's a bad dwarf2jason version". It could be parameterized with generator and version, it could have version values (so a maximum and a minimum) It would be very configurable, but would require plugins that need it to call it (which, is kinda the same as using a dependency).

So I think I'm leaning to the generic symbol table version checker. You can hand it a list of tables, or just get it to check them all, it allows the plugin to decide how to handle a failure, but it doesn't automatically protect all plugins, so you'd need to know which plugins are affected and add this too them. Still seems like the most straightforward without tinkering with the entire framework though? Then the question becomes where does this generic code live, would it be useful for windows? Actually, yeah, probably, so probably want it to live somewhere generic. I'm thinking either a method of the concrete SymbolSpace (then do we add it to the interface or not, then do we bump the framework minor version, and require all plugins that need it depend on that minor version, probably yes) or do we just put it in the framework/symbols/__init__.py but as its won separate function? I think I'd prefer it were linked with the space, so probably there. Ok, I can mock something up at the weekend, and then we can give it a spin? All the plugins that need it will need to call it properly and bump their minimum framework version. No other plugins should require changing.

Does that sound good to everyone?

from volatility3.

That sounds like it could be a really nice way to handle it. If i can help at all let me know.

from volatility3.

Hello @gcmoreira and @ikelos - thanks so much for looking into this.

After this fix I still can't run the plugin. It looks like log_buf, log_buf_len, and log_end are Void types in my ISF which then breaks the pointer_to_string call.

$ python vol.py -f linux-sample-1.dmp linux.kmsg
Volatility 3 Framework 2.5.2
Progress:  100.00               Stacking attempts finished
facility        level   timestamp       caller  line
Traceback (most recent call last):
  File "C:\Users\eve\volatility3\vol.py", line 10, in <module>
    volatility3.cli.main()
  File "C:\Users\eve\volatility3\volatility3\cli\__init__.py", line 790, in main
    CommandLine().run()
  File "C:\Users\eve\volatility3\volatility3\cli\__init__.py", line 447, in run
    renderers[args.renderer]().render(constructed.run())
  File "C:\Users\eve\volatility3\volatility3\cli\text_renderer.py", line 193, in render
    grid.populate(visitor, outfd)
  File "C:\Users\eve\volatility3\volatility3\framework\renderers\__init__.py", line 245, in populate
    for level, item in self._generator:
  File "C:\Users\eve\volatility3\volatility3\framework\plugins\linux\kmsg.py", line 513, in _generator
    for values in ABCKmsg.run_all(context=self.context, config=self.config):
  File "C:\Users\eve\volatility3\volatility3\framework\plugins\linux\kmsg.py", line 104, in run_all
    yield from kmsg_inst.run()
  File "C:\Users\eve\volatility3\volatility3\framework\plugins\linux\kmsg.py", line 217, in run
    log_buf = utility.pointer_to_string(log_buf_ptr, count=log_buf_len)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\eve\volatility3\volatility3\framework\objects\utility.py", line 26, in pointer_to_string
    raise TypeError("pointer_to_string takes a Pointer")
TypeError: pointer_to_string takes a Pointer

Taking a look in volshell it all looks like it should work - it's just the type that is causing issues:

$ python volshell.py -f linux-sample-1.dmp -l
Volshell (Volatility 3 Framework) 2.5.2
Progress:  100.00               Stacking attempts finished
    Call help() to see available functions

    Volshell mode        : Linux
    Current Layer        : layer_name
    Current Symbol Table : symbol_table_name1
    Current Kernel Name  : kernel

(layer_name) >>> 
(layer_name) >>> from volatility3.framework.objects import utility
(layer_name) >>> self.vmlinux = self.context.modules[self.config["kernel"]] #self. to match plugin
(layer_name) >>> log_buf_ptr = self.vmlinux.object_from_symbol(symbol_name="log_buf")
(layer_name) >>> log_buf_len = self.vmlinux.object_from_symbol(symbol_name="log_buf_len")
(layer_name) >>> log_buf = utility.pointer_to_string(log_buf_ptr, count=log_buf_len)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Users\eve\volatility3\volatility3\framework\objects\utility.py", line 26, in pointer_to_string
    raise TypeError("pointer_to_string takes a Pointer")
TypeError: pointer_to_string takes a Pointer
(layer_name) >>> dt(log_buf_ptr)
symbol_table_name1!void (0 bytes)
(layer_name) >>> dq(log_buf_ptr.vol.offset, 8) 
0xffff816177d8    ffffffff81768fb0                     .....v..
(layer_name) >>> db(0xffffffff81768fb0)
0xffffffff81768fb0    3c 36 3e 5b 20 20 20 20 30 2e 30 30 30 30 30 30    <6>[....0.000000
0xffffffff81768fc0    5d 20 49 6e 69 74 69 61 6c 69 7a 69 6e 67 20 63    ].Initializing.c
0xffffffff81768fd0    67 72 6f 75 70 20 73 75 62 73 79 73 20 63 70 75    group.subsys.cpu
0xffffffff81768fe0    73 65 74 0a 3c 36 3e 5b 20 20 20 20 30 2e 30 30    set.<6>[....0.00
0xffffffff81768ff0    30 30 30 30 5d 20 49 6e 69 74 69 61 6c 69 7a 69    0000].Initializi
0xffffffff81769000    6e 67 20 63 67 72 6f 75 70 20 73 75 62 73 79 73    ng.cgroup.subsys
0xffffffff81769010    20 63 70 75 0a 3c 35 3e 5b 20 20 20 20 30 2e 30    .cpu.<5>[....0.0
0xffffffff81769020    30 30 30 30 30 5d 20 4c 69 6e 75 78 20 76 65 72    00000].Linux.ver

I'll see if I can make a little patch to work around this issue. I imagine its not like this in every ISF out there.

from volatility3.

@eve-mem thanks. That's interesting. Not sure why in your ISF those symbols are void, but it should be something wrong there as none of them are void in 3.2 or any other kernel version.
https://elixir.bootlin.com/linux/v3.2.102/source/kernel/printk.c#L148

Have a look at my ISF:

    "log_buf": {
      "type": {
        "kind": "pointer",
        "subtype": {
          "kind": "base",
          "name": "char"
        }
      },
      "address": 18446744071591553768
    },
    "log_buf_len": {
      "type": {
        "kind": "base",
        "name": "int"
      },
      "address": 18446744071591553776
    },
    "log_end": {
      "type": {
        "kind": "base",
        "name": "unsigned int"
      },
      "address": 18446744071593651540
    },

and

>>> self.vmlinux.object_from_symbol(symbol_name="log_buf").vol.type_name
'symbol_table_name1!pointer'

Have you generated the ISF with the latest dwarf2json version?
Could you share your ISF?

from volatility3.

Great question.

A touch old!

"producer": {
      "name": "dwarf2json",
      "version": "0.4.1"
    },

It's actually the samples from the vol foundation page linked from in the readme. You can get the ISF here: https://downloads.volatilityfoundation.org/volatility3/symbols/linux.zip

I like to use that sample as I think it's used in the auto tests when commiting - it might be worth updating them? That's a separate issue completely though.

Based on what you've showed in in printk I have no doubt if I remade the ISF with a later version it would work. I'll have a go at that when I can.

Do you think it's worth a warning/debug message when using old ISF files in general or do we want to fight for backwards compatibility with older ISFs in ? I know there are other issue that were fixed with dwarf2json updates, so ISFs prior to that would never have worked anyway.

Maybe it's possible to add a requirement for the ISF producer for a plugin, so that it would only work on version made by dwarf2json higher than x version? Would limit them to working only on ISFs made by dwarf2json though. (which I'm sure is 99.999% of all ISFs, apart from the odd one crafted by hand when you can't get debug symbols - but those users will probably be alright getting past a check like that.)

What do you think?

Could even do something a little hacky like this to point users in the right direction? It would only fix it for this particular issue rather than doing it generically though.

diff --git a/volatility3/framework/plugins/linux/kmsg.py b/volatility3/framework/plugins/linux/kmsg.py
index d1f17bf9..d6a346bd 100644
--- a/volatility3/framework/plugins/linux/kmsg.py
+++ b/volatility3/framework/plugins/linux/kmsg.py
@@ -14,7 +14,7 @@ from volatility3.framework import (
     renderers,
 )
 from volatility3.framework.configuration import requirements
-from volatility3.framework.objects import utility
+from volatility3.framework.objects import utility, Pointer

 vollog = logging.getLogger(__name__)

@@ -213,6 +213,10 @@ class Kmsg_pre_3_5(ABCKmsg):

     def run(self) -> Iterator[Tuple[str, str, str, str, str]]:
         log_buf_ptr = self.vmlinux.object_from_symbol(symbol_name="log_buf")
+        if not isinstance(log_buf_ptr, Pointer):
+            raise TypeError(
+                f"The type of log_buf is {log_buf_ptr.vol.type_name} when a pointer is expected. This may 
be due to a currupt symbols file or a symbols file created with an old version of dwarf2json."
+            )
         log_buf_len = self.vmlinux.object_from_symbol(symbol_name="log_buf_len")
         log_buf = utility.pointer_to_string(log_buf_ptr, count=log_buf_len)
         log_end = self.vmlinux.object_from_symbol(symbol_name="log_end")

from volatility3.

Yeah - I think doing the hacking thing probably isn't a good idea but felt better than just plain crashing. Wouldn't want it to become the way to do it everywhere as you point out it would just mean checking every single time.

These are the auto tests I mean, the github actions - https://github.com/volatilityfoundation/volatility3/actions/runs/7389566835/job/20102666191. There is probably a proper name for those tests - regression testing?

The link is to the run where your PR get tested, I think it only runs a few of the linux plugins - and I don't think kmsg is one of them. I think it uses those symbols I linked to, you can see the wget in the logs too.

# getting the symbols
cd ./volatility3/symbols
  curl -sLO https://downloads.volatilityfoundation.org/volatility[3](https://github.com/volatilityfoundation/volatility3/actions/runs/7389566835/job/20102666191#step:7:3)/symbols/linux.zip
  unzip linux.zip
  cd -
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.7.17/x6[4](https://github.com/volatilityfoundation/volatility3/actions/runs/7389566835/job/20102666191#step:7:4)
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.7.17/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.7.17/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.7.17/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.7.17/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.7.17/x64/lib
Archive:  linux.zip
   creating: linux/
  inflating: linux/linux-image-4.9.0-3-amd64-dbg_4.9.30-2+deb9u2_amd64.json.xz  
  inflating: linux/linux-image-2.6.32-[5](https://github.com/volatilityfoundation/volatility3/actions/runs/7389566835/job/20102666191#step:7:5)-amd[6](https://github.com/volatilityfoundation/volatility3/actions/runs/7389566835/job/20102666191#step:7:6)4-dbg_2.6.32-48squeeze6_amd64.json.xz  
  inflating: linux/linux-image-3.2.0-4-amd64-dbg_3.2.5[7](https://github.com/volatilityfoundation/volatility3/actions/runs/7389566835/job/20102666191#step:7:7)-3+deb7u2_amd64.json.xz

# the linux tests 
test/test_volatility.py::test_linux_pslist[linux-sample-1.bin] PASSED    [ 14%]
test/test_volatility.py::test_linux_check_idt[linux-sample-1.bin] PASSED [ 28%]
test/test_volatility.py::test_linux_check_syscall[linux-sample-1.bin] PASSED [ [42](https://github.com/volatilityfoundation/volatility3/actions/runs/7389566835/job/20102666191#step:8:43)%]
test/test_volatility.py::test_linux_lsmod[linux-sample-1.bin] PASSED     [ 57%]
test/test_volatility.py::test_linux_lsof[linux-sample-1.bin] PASSED      [ 71%]
test/test_volatility.py::test_linux_proc_maps[linux-sample-1.bin] PASSED [ 85%]
test/test_volatility.py::test_linux_tty_check[linux-sample-1.bin] PASSED [100%]

I think we've proven that the kmsg plugin now should work if we've got the right ISF - so on that front I think it's all sorted. Thanks again for sorting it out so quickly.

from volatility3.

I remembered about @Abyss-W4tcher great collection of ISF which includes the exact one needed for this sample - https://github.com/Abyss-W4tcher/volatility3-symbols/blob/master/Debian/amd64/3.2.0/4/Debian_3.2.0-4-amd64_3.2.57-3%2Bdeb7u2_amd64.json.xz

It's made with a more modern version of dwarf2json:

{"producer":{"name":"dwarf2json","version":"0.6.0"},"format":"6.2.0"}

And as expected your plugin works perfectly @gcmoreira:

python vol.py -f linux-sample-1.dmp linux.kmsg
Volatility 3 Framework 2.5.2

facility        level   timestamp       caller  line

kern    info    0.000000        -       Initializing cgroup subsys cpuset
kern    info    0.000000        -       Initializing cgroup subsys cpu
kern    notice  0.000000        -       Linux version 3.2.0-4-amd64 ([email protected]) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.57-3+deb7u2
kern    info    0.000000        -       Command line: BOOT_IMAGE=/boot/vmlinuz-3.2.0-4-amd64 root=UUID=8f2bb477-848b-4bcb-9173-fd2f982db24d ro quiet
kern    info    0.000000        -       Disabled fast string operations
kern    info    0.000000        -       BIOS-provided physical RAM map:
kern    info    0.000000        -        BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
kern    info    0.000000        -        BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
kern    info    0.000000        -        BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
kern    info    0.000000        -        BIOS-e820: 0000000000100000 - 000000001fee0000 (usable)
kern    info    0.000000        -        BIOS-e820: 000000001fee0000 - 000000001feff000 (ACPI data)        
kern    info    0.000000        -        BIOS-e820: 000000001feff000 - 000000001ff00000 (ACPI NVS)
kern    info    0.000000        -        BIOS-e820: 000000001ff00000 - 0000000020000000 (usable)
kern    info    0.000000        -        BIOS-e820: 00000000f0000000 - 00000000f8000000 (reserved)
kern    info    0.000000        -        BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
kern    info    0.000000        -        BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
<snip>

from volatility3.

Sounds amazing. We can add the requirement checks as we encounter issues. Thanks @ikelos

from volatility3.