elfmaster / ecfs Goto Github PK

The ploblem phenomenon

root@debian-pc:/home/vans-pc/ecfs# make
gcc -fPIC -pie -D_GNU_SOURCE -m64 main/ecfs.c bin/prod/64/ecfs.a libecfs/bin/prod/64/libecfsreader64.a -o bin/prod/64/ecfs -ldwarf -lelf -lm
/usr/bin/ld: bin/prod/64/ecfs.a(personality.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(personality.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(core_notes.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(core_notes.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(util.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(util.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(dlopen.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(dlopen.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(core_text.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(core_text.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(core_headers.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(core_headers.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(symresolve.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(symresolve.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(ptrace.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(ptrace.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(core2ecfs.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(core2ecfs.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(proc.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(proc.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(core_accessors.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(core_accessors.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(so_resolve.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(so_resolve.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(list.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(list.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
/usr/bin/ld: bin/prod/64/ecfs.a(eh_frame.o):(.bss+0x0): multiple definition of global_hacks'; /tmp/ccRPWecy.o:(.bss+0x0): first defined here /usr/bin/ld: bin/prod/64/ecfs.a(eh_frame.o):(.bss+0xa0): multiple definition of opts'; /tmp/ccRPWecy.o:(.bss+0xa0): first defined here
collect2: error: ld returned 1 exit status
make: *** [Makefile:61：bin/prod/64/ecfs] Error:

My OS info

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
gcc version 10.2.1 20210110 (Debian 10.2.1-6)
root@debian-pc:/home/vans-pc/ecfs# dpkg -l | grep libelf
ii libelf-dev:amd64 0.187-1bpo11+1 amd64 libelf1 development libraries and header files
ii libelf1:amd64 0.187-1bpo11+1 amd64 library to read and write ELF files
root@debian-pc:/home/vans-pc/ecfs# dpkg -l | grep dwarf
ii libdwarf-dev:amd64 20201201-1 amd64 library to consume and produce DWARF debug information
ii libdwarf1:amd64 20201201-1 amd64 library to consume and produce DWARF debug information (runtime)

ecfs repo: https://github.com/elfmaster/ecfs

ecfs SEGV's when the dump has more than 4 gigs of memory mapped in, and in the case of a large heap of OOM condition ecfs fails to collect the process. Maybe add the physical max mem to be the max size of the ramdisk to enable collection of any possible process.

I was testing a few different possible crashes and I found a very interesting edge case:

#include <stdio.h>

int main(void) {
    asm ("call 0x41414141");
    return 0;
}

compile:

clang -O0 calladdr.c -o calladdr

Then running the binary with the ECFS x64 collector enabled I get the ecfs file, after flipping the core type to CORE and opening it in gdb:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000041414141 in ?? ()
(gdb) bt
#0  0x0000000041414141 in ?? ()
#1  0x0000000000000000 in ?? ()
(gdb) i frame
Stack level 0, frame at 0x7ffcabbc6d40:
 rip = 0x41414141; saved rip = 0x0
 called by frame at 0x7ffcabbc6d48
 Arglist at 0x7ffcabbc6d30, args: 
 Locals at 0x7ffcabbc6d30, Previous frame's sp is 0x7ffcabbc6d40
 Saved registers:
  rip at 0x7ffcabbc6d38
(gdb) x/16x $rsp
0x7ffcabbc6d38: 0x00000000  0x00000000  0x00000000  0x00000000
0x7ffcabbc6d48: 0x00000000  0x00000000  0x00000000  0x00000000
0x7ffcabbc6d58: 0x00000000  0x00000000  0x00000000  0x00000000
0x7ffcabbc6d68: 0x00000000  0x00000000  0x00000000  0x00000000
(gdb) x/16x $rbp
0x7ffcabbc6d40: 0x00000000  0x00000000  0x00000000  0x00000000
0x7ffcabbc6d50: 0x00000000  0x00000000  0x00000000  0x00000000
0x7ffcabbc6d60: 0x00000000  0x00000000  0x00000000  0x00000000
0x7ffcabbc6d70: 0x00000000  0x00000000  0x00000000  0x00000000
(gdb) i threads 
  Id   Target Id         Frame 
* 1    LWP 12927         0x0000000041414141 in ?? ()

At first I thought this was just a very messed up crash. But then I enabled the regular core pattern:

echo "/tmp/core.%p" > /proc/sys/kernel/core_pattern

then opened the core file (weirdly this standard core file was marked NONE and had to use et_filp):

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000041414141 in ?? ()
(gdb) bt
#0  0x0000000041414141 in ?? ()
#1  0x00000000004004b0 in ?? ()
#2  0x00000000004004c0 in ?? ()
#3  0x00007f6f383f0610 in ?? ()
#4  0x00007ffffea22ad8 in ?? ()
#5  0x00007ffffea22ad8 in ?? ()
#6  0x0000000100000000 in ?? ()
#7  0x00000000004004a0 in ?? ()
#8  0x0000000000000000 in ?? ()
(gdb) i frame
Stack level 0, frame at 0x7ffffea229f0:
 rip = 0x41414141; saved rip = 0x4004b0
 called by frame at 0x7ffffea229f8
 Arglist at 0x7ffffea229e0, args: 
 Locals at 0x7ffffea229e0, Previous frame's sp is 0x7ffffea229f0
 Saved registers:
  rip at 0x7ffffea229e8
(gdb) x/16x $rsp
0x7ffffea229e8: 0x004004b0  0x00000000  0x004004c0  0x00000000
0x7ffffea229f8: 0x383f0610  0x00007f6f  0xfea22ad8  0x00007fff
0x7ffffea22a08: 0xfea22ad8  0x00007fff  0x00000000  0x00000001
0x7ffffea22a18: 0x004004a0  0x00000000  0x00000000  0x00000000
(gdb) x/16x $rbp
0x7ffffea229f0: 0x004004c0  0x00000000  0x383f0610  0x00007f6f
0x7ffffea22a00: 0xfea22ad8  0x00007fff  0xfea22ad8  0x00007fff
0x7ffffea22a10: 0x00000000  0x00000001  0x004004a0  0x00000000
0x7ffffea22a20: 0x00000000  0x00000000  0x366da9f0  0xf9a9fb79
(gdb) i threads 
  Id   Target Id         Frame 
* 1    LWP 17854         0x0000000041414141 in ?? ()

So it appears that the registers contain address that point to null and this means I am unable to unwind the stack in this case. Thanks.

The ramdisk path is define as:

#define ECFS_RAMDISK_DIR "/mnt/ecfs_ramdisk"

Which could clobber a existing mount, maybe use a tempdir as a mount point?

user@host:~/Desktop/test$ ./test2 &
[1] 18690
user@host:~/Desktop/test$ ecfs_snapshot -p $(ps aux | grep test2 | grep -v grep | awk '{print $2}') -o test2
0x00007ffff7ac89a4 in __GI___nanosleep (requested_time=requested_time@entry=0x7fffffffdd40, remaining=remaining@entry=0x7fffffffdd40) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
28	../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory.
Saved corefile core.test2.18690
doing mmap
Copying from mem to copy 220928 bytes from 0x7ffff7f4f000 to 0x7ffff7f18000
Writing 220928 bytes
renaming tmp.core.test2.18690 to core.test2.18690
load_core_file failed

BTW, this is an awesome project. And I wonder whether it is still maintained because I really want to have the "snapshot execution" capability in my current project.

A buffer overflow if there is a large enough symbol string table. In the loop simply add a check to make sure that symstroff does not exceed 8192 * 32, if so, then perhaps a realloc to a larger buf size.

        char *strtab = heapAlloc(8192 * 32);
...  ...

 for (i = 0; i < fncount; i++) {
                symtab[i].st_value = fdp[i].addr;
                symtab[i].st_size = fdp[i].size;
                symtab[i].st_info = (((STB_GLOBAL) << 4) + ((STT_FUNC) & 0xf));
                symtab[i].st_other = 0;
                symtab[i].st_shndx = text_shdr_index;
                symtab[i].st_name = symstroff;
                sname = xfmtstrdup("sub_%lx", fdp[i].addr);
                strcpy(&strtab[symstroff], sname);
                symstroff += strlen(sname) + 1;
                free(sname);
        }

I do not know if the current code is really free software or not. Could you add a license please? GPL3 Seems perfect for me (of course MIT sounds better for me lol).

The -t (opts.text_all) will make sure the complete text of every segment is written out. This results in occasional segments not being completely written out for instance:

LOAD 0x00000000001e1000 0x00007f17296ba000 0x0000000000000000
0x0000000000000000 0x00000000001ff000 1000
LOAD 0x00000000001e1000 0x00007f17298b9000 0x0000000000000000
0x0000000000004000 0x0000000000004000 R 1000

Notice the file size (The 2nd line after the first LOAD line) is 0x000000000000000

And therefore the offset of the following segment starts at the same offset as the previous. This is a bug that will result in missing code.

Is this can be used in arm

I would like to point out that identifiers like “_ECFS_HEURISTICS_H” and “_LIBECFS_UTIL_H” do eventually not fit to the expected naming convention of the C language standard.
Would you like to adjust your selection for unique names?

ecfs was designed to be able to reconstruct a process image completely even if the existing executable has been deleted. There are very few things we look at /proc/self/exe for, and those things are not absolutely necessary and we can endure without them. However, since the birth of ecfs_handler.c (Which allows for cross portability between binary architectures on 64bit) we have to check if a process is 32bit or 64bit. Currently we are checking /proc/self/exe to do this... what if that file has been deleted? Infact, is it possible an executable is deleted with the process still running or will the system stop this? Or if it can be deleted is it still held in an open file handle? Lets check on this

A 64bit version of ecfs should be able to handle both 32bit and 64bit processes. This requires taking into account many things, including the ELF headers. e.g. certain structs etc.

The ability to parse ecfs files from python.

What is the best way to either access (or create accessors) for symbols in the modules for a given ECFS file? I know I can use get_local_symbols() to get the symbols for the primary binary but how should we make something that can resolve symbols over all modules within the binary?

When using -h option certain processes crash because of bug in heuristics.c in checking for dlopen'd libs.

Output from /var/log/syslog

Mar 5 13:06:15 elfmaster ecfs: no calls to dlopen found in /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2 [line: 409]
Mar 5 13:06:15 elfmaster ecfs: no calls to dlopen found in /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1 [line: 409]
Mar 5 13:06:15 elfmaster ecfs: no calls to dlopen found in /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3 [line: 409]
Mar 5 13:06:15 elfmaster kernel: [32130.875632] ecfs[26549]: segfault at f ip 00007f70538a6a03 sp 00007fff4f9c43e0 error 4 in libc-2.19.so[7f705385b000
+1bb000]