brho / akaros Goto Github PK

VMX has a lot of issues that we aren't dealing with. Do a git log on arch/x86/kvm/vmx.c in Linux, and you'll find a list of things to fix.

For instance, the hardware resets the GDT limit when we do a vmexit. Linux handled that here: https://www.spinics.net/lists/kvm/msg38973.html. We probably used to have it, but lost it during the big rewrite - pretty much because there was no comment about why the GDT was being reloaded. We probably need to handle that when we 'finalize' the context.

Similarly, there's some issue with the task register: https://patchwork.kernel.org/patch/9583281/ (x86/kvm/vmx: Defer TR reload after VM exit).

This comment has been identified as either hateful, hurtful or discriminatory as part of a project to #CleanUpGitHub. This pull request was made by a human. For more info, please check EthicalCode’s website at www.ethicalco.de/cleanupgh

http://comments.gmane.org/gmane.os.plan9.general/76188

We have this problem too. The hash tables might need a little more work too. We have multiple NHASH defines which we might be able to consolidate.

For the most part, we intercept every MSR write. We are explicit about saving/restoring the MSRs that aren't intercepted. However the intercepted MSRs run through an emulator, and some of those actually just let the MSR operation happen! In short, any use of emsr_ok is probably wrong. Some of them are MSRs the host isn't using, but if you have multiple VMs, they could be mucking with each others settings.

Similarly, we don't do anything for the IA32_PAT MSR. We might be telling the guest to not even try and use it (via .config or cpuid), since it's intercepted (by default) and not handled by the MSR vmexit codes. This means that the guest is probably using our host's PAT settings. That's OK if the settings are the same, and might save a little overhead on the vmexit/vmenter (assuming the vmcs load/save of IA32_PAT is just a write_msr under the hood, just like autoload).

The best thing for PAT is probably to turn off cpuid 0x01 edx[16], or whichever one tells the guest that the PAT MSR exists, and then set up our host PAT in the sane way: where it acts the way you'd expect if PTE_PCD or PTE_PWT are set. (I think I already did that). Also, we should check and see if our Linux guest is attempting to check IA32_PAT. If not, why not? If so, what is actually happening?

All of the info for doing the fill-kfs stuff is in the Makefile anyway. There's no reason to keep it around as an external script anymore.

Commit 24f29c1 ("Fix virtio net handling of the header.") breaks my ability to ssh into a Linux guest.

From the guest, I see:

tc@box:~$ Sending discover...
Sending discover...
Sending discover...
Sending discover...
Sending discover...
Sending discover...

where the 'sending discover' is new.

This is on my vmm branch. If I revert 24f29c1, things work as usual.

One thing I noticed about that commit is that previously we'd always strip the VIRTIO_HEADER_SIZE from every iov, not just the first one.

Which behavior is correct? Is the header a set of bytes at the beginning of the area mapped by the IOVs? Or is it something prepended to every IOV? If not the latter (old behavior), then why does the guest's networking fail in such a way that stripping 12 bytes off every IOV makes it work?

I have been looking at why netperf throughput is so low (<1Mbps) on my test machines. It was netperf TCP_STREAM from Akaros to Linux. Both machines have mlx4 NIC and I disabled offloads on Linux.

After some debugging with tcpdump and wireshark, it turns out to be

1. Akaros sending packets with incorrect TCP checksum after some number (hundreds, exact number seems to be random) of packets; but not all the checksum are wrong, only some of them; the first packet with incorrect checksum is 1460 bytes long (tcp len).
2. Linux responds with duplicate ACKs.
3. Akaros starts a fast retransmission, but still with the incorrect checksum so it's dropped.
4. Akaros starts retransmissions, but all of them have the incorrect checksum (the same packet is just retransmitted, and since the checksum is wrongly computed, retransmission of course will not solve the problem).
5. Retransmission takes time, hence the very low throughput.

With block extra data disabled, I can get decent throughput.

Now I'm looking at the code...

If you try to run snoopy without the "-p" flag (which turns off promiscuous mode) on an mlx4 NIC, you'll trigger the panic in ether_promiscuous() (kern/drivers/net/mlx4/main.c

If you have panics on multiple cores in parallel, their input gets interleaved. Additionally, we probably want a backtrace (safely!), since 99% of the time, that's the first thing I want to see.

Dan mentioned that multiboot's tables can be anywhere in memory - let alone an initrd. Right now, we don't use an initrd, and we copy-out the command line. But if we ever want to bother with an initrd, we'll need to account for that when bootstrapping the memory allocator.

Additionally, our current process of iterating over the MBI free regions is technically unsafe, since a dastardly bootloader could have put the tables right after our kernel image. You can't write to any RAM outside of your kernel image (to include the BSS) and still read the MBI. Multiboot's structures sit in memory that it marks FREE. Thanks.

The simplest fix is to copy-in the fixed-size multitboot info, into the BSS. I'd consider making an array of memory-map info structs too, and even making fake RESERVED ones for things like the initrd.

This won't be a big deal until / if we use an initrd, since in practice the tables themselves are out of harms way.

brian ran into a rare deadlock with futexes where there is a circular wait between the futex spinlock and the alarm spinlock. we can probably fix by not holding the alarm lock while running the handler, and using something similar to what the kernel's alarm code does for RKM alarms. (delicate issue is handling unset_alarm()).

might as well fix up both the kernel and parlib alarm code in this way. might be able to run the user-level alarms in a uthread, though that's a separate issue. (getting a thread from the 2LS, esp the thread0 sched).

efence intercepts malloc et al. When a uthread exits, its TLS is cleaned up. Ultimately, this calls into glibc, which calls free() on the TLS region in _dl_deallocate_tls. That region was allocated with __libc_memalign() in _dl_allocate_tls_storage. efence can intercept the free (I assume), but not the _libc function.

Whoops. One option would be to change glibc to not call free and call something direct. Or perhaps to call memalign() instead of __libc_memalign. This is in dl-tls.c, and I'm inclined to not muck with things.

Likely, I'll just prevent all code from linking with efence until we can get a better fix. Possibly some sort of 'efence disable' call, though that might be a pain - esp with concurrent ops.

Here's a BT for those curious. Though it is a little weird that it thinks that __pthread_free_stack() called __uthread_free_tls(). Might be BT noise.

#01 Addr 0x000000000040c854 is in signal at offset 0x000000000000c854 fatal_backtrace /usr/local/google/home/brho/akaros/ros-kernel/user/parlib/panic.c:25 #02 Addr 0x0000000000405edb is in signal at offset 0x0000000000005edb EF_Abort /usr/local/google/home/brho/akaros/ros-kernel/user/electric-fence/print.c:119 #03 Addr 0x0000000000405255 is in signal at offset 0x0000000000005255 free /usr/local/google/home/brho/akaros/ros-kernel/user/electric-fence/efence.c:675 #04 Addr 0x000000000040a6f4 is in signal at offset 0x000000000000a6f4 __uthread_free_tls /usr/local/google/home/brho/akaros/ros-kernel/user/parlib/uthread.c:1236 #05 Addr 0x0000000000403f33 is in signal at offset 0x0000000000003f33 __pthread_free_stack /usr/local/google/home/brho/akaros/ros-kernel/user/pthread/pthread.c:445 #06 Addr 0x00000000004026c6 is in signal at offset 0x00000000000026c6 __uthread_yield /usr/local/google/home/brho/akaros/ros-kernel/user/parlib/uthread.c:416 #07 Addr 0x000000000040b3b5 is in signal at offset 0x000000000000b3b5 uthread_yield /usr/local/google/home/brho/akaros/ros-kernel/user/parlib/uthread.c:496 #08 Addr 0x000000000040b815 is in signal at offset 0x000000000000b815 uth_2ls_thread_exit /usr/local/google/home/brho/akaros/ros-kernel/user/parlib/uthread.c:1336 #09 Addr 0x0000000000404a5e is in signal at offset 0x0000000000004a5e pthread_exit_no_cleanup /usr/local/google/home/brho/akaros/ros-kernel/user/pthread/pthread.c:726 #10 Addr 0x0000000000404a80 is in signal at offset 0x0000000000004a80 pthread_exit /usr/local/google/home/brho/akaros/ros-kernel/user/pthread/pthread.c:730

I have a simple pipe test sitting on origin/failing-pipetest that periodically dies with a page fault. It should always run to completion.

I tried to build cross compiler but it returns some error
+cc[vmm]...
fatal error: libelf.h: no such file or directory
....
So that I install elfutils-libelf-devel. However, I have installed this package and the same mistake is still there. I have checked that the libelf.h have already existed in /usr/include. I don't know how to fix it and could anyone give me some idea?

Many thanks

If you're looking to get your hands dirty, here's a good list of projects to get you started.

https://github.com/brho/akaros/wiki

I was trying to install akaros to try to implement another scheduling algorithm. But i cant seem install it. One error after the other.
I built the cross compiler. x86_64
But i am not able to build the kernel.

This is what i did.
opened gedit .bashrc
added this to the end of it-->export PATH="<pathofbin:$PATH"
saved it. opened a new terminal.
went to the akaros-master directory.
then-->sudo make
but the following error was reported
"
Could not find a x86_64-ucb-akaros- version of GCC/binutils. Be sure to build the cross-compiler and update your PATH
make: *** [cc-exists] Error 1
"

then tried-->
$sudo -s
$PATH="/home/matthew/install/x86_64-ucb-akaros-gcc/bin:$PATH"
$make

then i got this error.
"
CC kern/src/error.o
kern/src/error.c:3:1: error: expected expression at end of input
const char errno_strings[] = {
^
make[1]: ** [kern/src/error.o] Error 1
make: *** [kern/src] Error 2
"

Can someone please help me??

Dan triggered a bug in v4delroute() a while ago. I forget the details, other than it was a bad kref release. Dan might be able to recreate it.

#include <pthread.h>
int main()
{
	pthread_mutex_t mu;
	pthread_mutex_init(&mu, 0);
	return 0;
}

crashes with NULL deref on first instruction of pthread_mutex_init which tries to dereference the attr.

POSIX explicitly allows NULL attr:
http://pubs.opengroup.org/onlinepubs/7908799/xsh/pthread_mutex_init.html

akaros is on 6344ed0.

You can't used the build system with emacs, period. This is really frustrating. The way errors messages come out does not allow you to use next-error. I have tried and failed to figure out why it does not work.

We should put it under tools/shells just like we have tools/compilers, and set it up to wget the source, and apply all the stuff we currently do manually from the GETTING_STARTED document.

If you just do a:

bind \#a /prog

on master. Or

bind \#acpi /prog

on my net or efd branches, then depending on your qemu settings, you
could crash the machine and trigger a reboot.

The magic setting seems to be -enable-kvm. Most people use this flag,
I think, and most people don't attach #acpi.

This should error out:

bash-4.3$ echo foo > /net/log

It does from busybox's echo:

/ $ echo foo > /net/log
echo failed: Invalid argument, This space in kern/src/net/netlog.c@213 needs filling in.

or:

bash-4.3$ busybox echo foo > /net/log
echo failed: Invalid argument, This space in kern/src/net/netlog.c@213 needs filling in.

strace shows the write fails.

Syscall 101 ( write):(0x1, 0x7f7fff9fde10, 0x9, 0x0, 0x0, 0x0) ret: 0xffffffffffffffff

Other files that normally accept writes also don't get an error:

bash-4.3$ echo asdas > /prof/mpstat

Files that don't exit or aren't writable do get errors:

bash-4.3$ echo foo > /proc/self/foobar
bash: /proc/self/foobar: Generic Failure (should be ENOENT or something)

bash-4.3$ echo foo > /proc/self/maps
bash: /proc/self/maps: Operation not permitted

I am getting the following crashes. Is it a know issue? If not and you don't see why it happens right away, I can try to create a reproducer. Checkout is on 6344ed0.

bash-4.3$ Unhandled user trap in vcore context from VC 1
HW TRAP frame (partial) at 0xffffffffc82cbd20 on core 1
  rax  0x0000100000011743
  rbx  0x000030000005ced0
  rcx  0x0000000000000001
  rdx  0x0000100000011740
  rbp  0x000030000005ceb0
  rsi  0x0000100000008820
  rdi  0x0000100000008820
  r8   0x0000000000000000
  r9   0x0000000000000000
  r10  0x000030000005ced0
  r11  0x0000000000000200
  r12  0x0000000000000001
  r13  0x0000000000000001
  r14  0x0000000000409720
  r15  0x0000000000000000
  trap 0x0000000d General Protection
  gsbs 0x0000000000000000
  fsbs 0x0000000000000000
  err  0x--------00000000
  rip  0x00000000004005f0
  cs   0x------------0023
  flag 0x0000000000010286
  rsp  0x000030000005ce98
  ss   0x------------001b
err 0x0 (for PFs: User 4, Wr 2, Rd 1), aux 0x0000000000000000
Addr 0x00000000004005f0 is in syz-executor at offset 0x00000000000005f0
VM Regions for proc 540
NR:                                     Range:       Prot,      Flags,               File,                Off
00: (0x0000000000400000 - 0x00000000004b2000): 0x00000005, 0x00000001, 0xffff800101103840, 0x0000000000000000
01: (0x00000000004b2000 - 0x00000000004b3000): 0x00000005, 0x00000002, 0xffff800101103840, 0x00000000000b2000
02: (0x00000000006b3000 - 0x00000000006b6000): 0x00000003, 0x00000002, 0xffff800101103840, 0x00000000000b3000
03: (0x00000000006b6000 - 0x0000000000925000): 0x00000003, 0x00000002, 0x0000000000000000, 0x0000000000000000
04: (0x0000100000000000 - 0x0000100000024000): 0x00000007, 0x00000022, 0x0000000000000000, 0x0000000000000000
05: (0x0000300000000000 - 0x0000300000001000): 0x00000003, 0x00000002, 0xffff800101103840, 0x0000000000000000
06: (0x0000300000001000 - 0x0000300000005000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000
07: (0x0000300000005000 - 0x0000300000007000): 0x00000007, 0x00000022, 0x0000000000000000, 0x0000000000000000
08: (0x0000300000007000 - 0x0000300000031000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000
09: (0x0000300000031000 - 0x000030000005d000): 0x00000007, 0x00000022, 0x0000000000000000, 0x0000000000000000
10: (0x00007f7fff8ff000 - 0x00007f7fff9ff000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000

Backtrace of user context on Core 1:
	Offsets only matter for shared libraries
#01 Addr 0x00000000004005f0 is in syz-executor at offset 0x00000000000005f0
#02 Addr 0x0000000000410394 is in syz-executor at offset 0x0000000000010394
#03 Addr 0x000000006b3a3000 has no VMR
Unhandled user trap in vcore context from VC 0
HW TRAP frame (partial) at 0xffffffffc82cc720 on core 5
  rax  0x0000100000005d03
  rbx  0x00007f7fff9feb80
  rcx  0x0000000000000001
  rdx  0x0000100000005d00
  rbp  0x00007f7fff9feb60
  rsi  0x00001000000046c0
  rdi  0x00001000000046c0
  r8   0x0000000000000000
  r9   0x0000000000000000
  r10  0x00007f7fff9feb80
  r11  0x0000000000000200
  r12  0x0000000000000001
  r13  0x0000000000000000
  r14  0x0000000000409520
  r15  0x0000000000000000
  trap 0x0000000d General Protection
  gsbs 0x0000000000000000
  fsbs 0x0000000000000000
  err  0x--------00000000
  rip  0x00000000004005f0
  cs   0x------------0023
  flag 0x0000000000010206
  rsp  0x00007f7fff9feb48
  ss   0x------------001b
err 0x0 (for PFs: User 4, Wr 2, Rd 1), aux 0x0000000000000000
Addr 0x00000000004005f0 is in syz-executor at offset 0x00000000000005f0
VM Regions for proc 540
NR:                                     Range:       Prot,      Flags,               File,                Off
00: (0x0000000000400000 - 0x00000000004b2000): 0x00000005, 0x00000001, 0xffff800101103840, 0x0000000000000000
01: (0x00000000004b2000 - 0x00000000004b3000): 0x00000005, 0x00000002, 0xffff800101103840, 0x00000000000b2000
02: (0x00000000006b3000 - 0x00000000006b6000): 0x00000003, 0x00000002, 0xffff800101103840, 0x00000000000b3000
03: (0x00000000006b6000 - 0x0000000000925000): 0x00000003, 0x00000002, 0x0000000000000000, 0x0000000000000000
04: (0x0000100000000000 - 0x0000100000024000): 0x00000007, 0x00000022, 0x0000000000000000, 0x0000000000000000
05: (0x0000300000000000 - 0x0000300000001000): 0x00000003, 0x00000002, 0xffff800101103840, 0x0000000000000000
06: (0x0000300000001000 - 0x0000300000005000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000
07: (0x0000300000005000 - 0x0000300000007000): 0x00000007, 0x00000022, 0x0000000000000000, 0x0000000000000000
08: (0x0000300000007000 - 0x0000300000031000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000
09: (0x0000300000031000 - 0x000030000005d000): 0x00000007, 0x00000022, 0x0000000000000000, 0x0000000000000000
10: (0x00007f7fff8ff000 - 0x00007f7fff9ff000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000

Backtrace of user context on Core 5:
	Offsets only matter for shared libraries
#01 Addr 0x00000000004005f0 is in syz-executor at offset 0x00000000000005f0
#02 Addr 0x0000000000410394 is in syz-executor at offset 0x0000000000010394
#03 Addr 0x00009214b0000000 has no VMR
Unhandled user trap in vcore context from VC 0
HW TRAP frame (partial) at 0xffffffffc82cbaa0 on core 0
  rax  0x0000100000005df0
  rbx  0x00007f7fff9feaf0
  rcx  0x00000000004368ee
  rdx  0x0000100000005d00
  rbp  0x00007f7fff9fead0
  rsi  0x00001000000046c0
  rdi  0x00001000000046c0
  r8   0x0000000000000000
  r9   0x0000000000000000
  r10  0x00007f7fff9feaf0
  r11  0x0000000000000200
  r12  0x0000000000000001
  r13  0x0000000000000000
  r14  0x0000000000415400
  r15  0x0000000000000000
  trap 0x0000000d General Protection
  gsbs 0x0000000000000000
  fsbs 0x0000000000000000
  err  0x--------00000000
  rip  0x00000000004005f0
  cs   0x------------0023
  flag 0x0000000000010283
  rsp  0x00007f7fff9feab8
  ss   0x------------001b
err 0x0 (for PFs: User 4, Wr 2, Rd 1), aux 0x0000000000000000
Addr 0x00000000004005f0 is in syz-executor at offset 0x00000000000005f0
VM Regions for proc 506
NR:                                     Range:       Prot,      Flags,               File,                Off
00: (0x0000000000400000 - 0x00000000004b2000): 0x00000005, 0x00000001, 0xffff800101103840, 0x0000000000000000
01: (0x00000000004b2000 - 0x00000000004b3000): 0x00000005, 0x00000002, 0xffff800101103840, 0x00000000000b2000
02: (0x00000000006b3000 - 0x00000000006b6000): 0x00000003, 0x00000002, 0xffff800101103840, 0x00000000000b3000
03: (0x00000000006b6000 - 0x0000000000925000): 0x00000003, 0x00000002, 0x0000000000000000, 0x0000000000000000
04: (0x0000100000000000 - 0x0000100000024000): 0x00000007, 0x00000022, 0x0000000000000000, 0x0000000000000000
05: (0x0000300000000000 - 0x0000300000001000): 0x00000003, 0x00000002, 0xffff800101103840, 0x0000000000000000
06: (0x0000300000001000 - 0x0000300000005000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000
07: (0x0000300000005000 - 0x0000300000007000): 0x00000007, 0x00000022, 0x0000000000000000, 0x0000000000000000
08: (0x0000300000007000 - 0x0000300000019000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000
09: (0x00007f7fff8ff000 - 0x00007f7fff9ff000): 0x00000003, 0x00000022, 0x0000000000000000, 0x0000000000000000

Backtrace of user context on Core 0:
	Offsets only matter for shared libraries
#01 Addr 0x00000000004005f0 is in syz-executor at offset 0x00000000000005f0
#02 Addr 0x0000000000410394 is in syz-executor at offset 0x0000000000010394
#03 Addr 0x000000000c00007f has no VMR
kernel panic at kern/src/pagemap.c:222, from core 0: assertion failed: page && pm_slot_check_refcnt(*page->pg_tree_slot)
Entering Nanwan's Dungeon on Core 0 (Ints on):
Type 'help' for a list of commands.

Got this while running syzkaller on 2b284c4:

bash-4.3$ kernel panic at kern/src/vfs.c:1359, from core 1: assertion failed: buf == buf_end
Entering Nanwan's Dungeon on Core 1 (Ints on):
Type 'help' for a list of commands.
ROS(Core 1)> bash-4.3$ bt
Stack Backtrace on Core 1:
#01 [<0xffffffffc2016074>] in mon_backtrace
#02 [<0xffffffffc2017177>] in monitor
#03 [<0xffffffffc200cbfc>] in _panic
#04 [<0xffffffffc205c51c>] in generic_file_write
#05 [<0xffffffffc2053890>] in sys_write
#06 [<0xffffffffc2056919>] in syscall
#07 [<0xffffffffc2056ad4>] in run_local_syscall
#08 [<0xffffffffc20a28aa>] in sysenter_callwrapper

Unfortunately our ability to localize and produce reproducers is limited due to no Go support.

Program is:

#include <unistd.h>
#include <stdio.h>
#include <errno.h>

void readfd(int fd)
{
	char buf[100];
	int n = read(fd, buf, sizeof(buf) - 1);
	if (n < 0) {
		printf("read(%d) failed: %d\n", fd, errno);
		return;
	}
	buf[n] = 0;
	printf("read(%d): n=%d '%s'\n", fd, n, buf);
}

int main()
{
	if (dup2(0, 100) == -1) {
		printf("dup2 failed: %d\n", errno);
		return 1;
	}
	if (close(0)) {
		printf("close(0) failed: %d\n", errno);
		return 1;
	}
	readfd(100);
	readfd(0);
	readfd(5);
	return 0;
}

Actual output:

$ echo -n Hello | ./a.out
read(100) failed: 9
read(0) failed: 9
read(5): n=5 'Hello'

Expected output:

$ echo -n Hello | ./a.out
read(100): n=5 'Hello'
read(0) failed: 9
read(5) failed: 9

checkout is on 6344ed0

Commit 526df6f ("rcu: Fix quiescent state reporting deadlock") seems to be acting up again.

Brian had a case where the rcu_gp_ktask was neither running nor sleeping, and a kmsg to wake it up was sitting on a core unacknowledged. That core was halted. When it was kicked (via trace coretf -1), the kmsg ran (__launch_kthread), it woke up the GP ktask, and it all unblocked.

Something in that commit probably didn't handle things properly.

Might be that something ran in PRKM and returned to PRKM without going through smp_idle. The recent commit where chan_release finishes in an RKM might be doing this. (that is a rare thing too). A quick looks appears OK (we report_qs in PRKM), and then if report_qs triggers a kmsg, we should be checking it (since we're in a while loop).

maybe we can test this by having a kmsg send itself another kmsg, and see if it runs without getting an IPI (so on an MCP core).

also, next time do a trace verbose first. might be a clue in the BT.

The one from coreutils:

bash-4.3$ rm hello.txt
rm: cannot remove 'hello.txt': Function not implemented

Busybox's still works:

bash-4.3$ busybox rm hello.txt

For anyone stumbling across this project, it might help to create a README file

If you put a cpio in /mnt, say obj/kern/initramfs.cpio, then try to extract it:

ash ifconfig
ash root
cd dir1 (so as to not run into errors where files already exist)
cpio -d -i < /mnt/initramfs.cpio

You get a stat error.

This worked on ddcb5d3, but fails on origin/master.

All pipes are globally allocated with a unique ID:

spin_lock(&(&pipealloc)->lock);
p->path = ++pipealloc.path;
spin_unlock(&(&pipealloc)->lock);

After making 4 billion pipes, we'll start reusing the ID numbers from pipealloc.path.

If this isn't a problem, then we should document why, and possibly just remove the pipe ID.

should be XSDT

(spotted by changyuan)

brian had this once:

Stack Backtrace on Core 0:
#1 [<0xffffffffc200a00c>] in backtrace
#2 [<0xffffffffc20097fd>] in _panic
#3 [<0xffffffffc2048ef6>] in kref_get
#4 [<0xffffffffc200ba71>] in sem_down
#5 [<0xffffffffc200bdbb>] in sem_down_bulk
#6 [<0xffffffffc2050189>] in rcu_barrier
#7 [<0xffffffffc2041ec6>] in tf_dfs_purge
#8 [<0xffffffffc2041f90>] in tf_dfs_purge
#9 [<0xffffffffc2041f90>] in tf_dfs_purge
#10 [<0xffffffffc2041f90>] in tf_dfs_purge
#11 [<0xffffffffc2041f90>] in tf_dfs_purge
#12 [<0xffffffffc2079050>] in gtfs_release
#13 [<0xffffffffc2078deb>] in kref_put
#14 [<0xffffffffc20315d0>] in chan_release
#15 [<0xffffffffc2030cfb>] in kref_put
#16 [<0xffffffffc2048b1a>] in __proc_free
#17 [<0xffffffffc2048a6b>] in kref_put
#18 [<0xffffffffc20a6f21>] in __abandon_core
#19 [<0xffffffffc204c8b9>] in abandon_core
#20 [<0xffffffffc20539cd>] in __smp_idle

While experimenting with akaros I've tried to run a Go program
compiled to linux and it caused kernel panic. I understand that that's
not the way I build programs for akaros, but it still should not cause
kernel panics I guess.

bash-4.3$ ./testgo
kernel warning at kern/src/syscall.c:2802, from core 0: Only one
supported (Debutante calls: 5438792)

HW TRAP frame at 0xfffffff0000a0e60 on core 0
  rax  0x0000000000000000
  rbx  0x0000000000001002
  rcx  0xffff8001015e0e10
  rdx  0xffff8001015e0e10
  rbp  0xfffffff0000a0f78
  rsi  0x0000000000000780
  rdi  0x0000000000001002
  r8   0xffff8000000b8fa0
  r9   0x0000000000000f00
  r10  0xffff8000000b8f00
  r11  0xffff8000000b8ec0
  r12  0x000000000052fd48
  r13  0xffff8001015e0e10
  r14  0x0000000000000000
  r15  0xffff8001001b3700
  trap 0x0000000e Page Fault
  gsbs 0xffffffffc82c8940
  fsbs 0x0000000000000000
  err  0x--------00000000
  rip  0xffffffffc2055308
  cs   0x------------0008
  flag 0x0000000000010246
  rsp  0xfffffff0000a0f28
  ss   0x------------0010

Backtrace of kernel context on Core 0:
#01 [<0xffffffffc2055308>] in run_local_syscall
#02 [<0xffffffffc20a098a>] in sysenter_callwrapper
kernel panic at kern/arch/x86/trap.c:311, from core 0: Proc-ful Page
Fault in the Kernel at 0x0000000000001002!
``

akaros is on 6344ed04e307ba30df879d1d407b10a1b3236784.
go version devel +9735fcfce7 Mon Oct 2 09:57:23 2017 +0000 linux/amd64
Go program is:

```go
package main
import (
  "fmt"
)
func main() {
  fmt.Printf("Hello, Akaros!\n")
}

I built it on host for linux/amd64 and then tried to run on akaros.

git grep TODO kern/ user/

There are about 496 items, based on which branch you're on. Have at it.

I have user/vmm/include which has files and directories init. The directories don't get copied and they should. Make the copy recursive or use rsync, take your pick.

On commit 94fd376
kernel reliably crashes with:

/ $ HW TRAP frame at 0xfffffff000061b90 on core 0
  rax  0xffff8000044a0660
  rbx  0x000000000000000b
  rcx  0x0000000000000000
  rdx  0x000000000000000b
  rbp  0xfffffff000061c58
  rsi  0x0000000000483e33
  rdi  0xffff8000044a0660
  r8   0x0000000000000003
  r9   0x0000000000000000
  r10  0x0000000000000000
  r11  0x0000000000000202
  r12  0xffff800003ab2820
  r13  0x0000000000483e33
  r14  0x0000000000010000
  r15  0xffff8000044a0660
  trap 0x0000000e Page Fault
  gsbs 0xffffffffc82e58c0
  fsbs 0x0000000000000000
  err  0x--------00000000
  rip  0xffffffffc20536a8
  cs   0x------------0008
  flag 0x0000000000010202
  rsp  0xfffffff000061c58
  ss   0x------------0010

Backtrace of kernel context on Core 0:
#01 [<0xffffffffc20536a8>] in memcpy
#02 [<0xffffffffc203a495>] in __qwrite
#03 [<0xffffffffc207ef26>] in pipewrite
#04 [<0xffffffffc203f3c4>] in rwrite
#05 [<0xffffffffc2057ca9>] in syscall
#06 [<0xffffffffc2057e64>] in run_local_syscall
#07 [<0xffffffffc20a957a>] in sysenter_callwrapper
kernel panic at kern/arch/x86/trap.c:311, from core 0: Proc-ful Page Fault in the Kernel at 0x0000000000483e33!
Entering Nanwan's Dungeon on Core 0 (Ints off):
Type 'help' for a list of commands.

Repro instructions:
checkout https://github.com/dvyukov/syzkaller.git dvyukov-akaros-pipe-crash branch into $GOPATH/src/github.com/google/syzkaller
Run:
make execprog
make TARGETOS=akaros SOURCEDIR=/path/to/akaros/toolchain executor
/path/to/akaros/toolchain is the dir containing x86_64-ucb-akaros-gcc
scp bin/akaros_amd64/syz-executor to the akaros VM
create /tmp/simple file with a single line openat(0xffffffffffffff9c, &(0x7f0000000500)='file1\x00', 0x10002, 0x1a0)
finally run:
bin/linux_amd64/syz-execprog -os=akaros -debug -threaded=1 -executor "/usr/bin/ssh -p 5555 -i id_rsa.akaros -o IdentitiesOnly=yes -v root@localhost /syz-executor" /tmp/simple

sys_exec, and probably other functions, use pcpui in such as way as to assume they don't block. that clearly isn't the case. this shows up as a bad kfree, where we try to kfree the kernel string for the etherread4 ktask. what happened is the exec started on core 0 (or some core), blocked, and then resumed on another core. when it went to free_sysc_str, it freed the string of whatever was on its old core.

at least two things:

• make systrace_finish_sysc not take a kth, which cuts down on the number of pcpui uses
• make sys_exec not cache pcpui.
• maybe don't cache the pcpui pointer at all, except in careful places. (i think it is a little faster, but not worth the hassle in less performance-critical syscalls).