Git Product home page Git Product logo

ghost-userspace's Introduction

ghOSt: Fast & Flexible User-Space Delegation of Linux Scheduling

ghOSt is a general-purpose delegation of scheduling policy implemented on top of the Linux kernel. The ghOSt framework provides a rich API that receives scheduling decisions for processes from userspace and actuates them as transactions. Programmers can use any language or tools to develop policies, which can be upgraded without a machine reboot. ghOSt supports policies for a range of scheduling objectives, from µs-scale latency, to throughput, to energy efficiency, and beyond, and incurs low overheads for scheduling actions. Many policies are just a few hundred lines of code. Overall, ghOSt provides a performant framework for delegation of thread scheduling policy to userspace processes that enables policy optimization, non-disruptive upgrades, and fault isolation.

SOSP '21 Paper
SOSP '21 Talk

The ghOSt kernel is here. You must compile and run the userspace component on the ghOSt kernel.

This is not an officially supported Google product.


Compilation

The ghOSt userspace component can be compiled on Ubuntu 20.04 or newer.

1. We use the Google Bazel build system to compile the userspace components of ghOSt. Go to the Bazel Installation Guide for instructions to install Bazel on your operating system.

2. Install ghOSt dependencies:

sudo apt update
sudo apt install libnuma-dev libcap-dev libelf-dev libbfd-dev gcc clang-12 llvm zlib1g-dev python-is-python3

Note that ghOSt requires GCC 9 or newer and Clang 12 or newer.

3. Compile the ghOSt userspace component. Run the following from the root of the repository:

bazel build -c opt ...

-c opt tells Bazel to build the targets with optimizations turned on. ... tells Bazel to build all targets in the BUILD file and all BUILD files in subdirectories, including the core ghOSt library, the eBPF code, the schedulers, the unit tests, the experiments, and the scripts to run the experiments, along with all of the dependencies for those targets. If you prefer to build individual targets rather than all of them to save compile time, replace ... with an individual target name, such as agent_shinjuku.


ghOSt Project Layout

  • bpf/user/
    • ghOSt contains a suite of BPF tools to assist with debugging and performance optimization. The userspace components of these tools are in this directory.
  • experiments/
    • The RocksDB and antagonist Shinjuku experiments (from our SOSP paper) and microbenchmarks. Use the Python scripts in experiments/scripts/ to run the Shinjuku experiments.
  • kernel/
    • Headers that have shared data structures used by both the kernel and userspace.
  • lib/
    • The core ghOSt userspace library.
  • schedulers/
    • ghOSt schedulers. These schedulers include:
      • biff/, Biff (bare-bones FIFO scheduler that schedules everything with BPF code)
      • cfs/ CFS (ghOSt implementation of Linux Completely Fair Scheduler policy)
      • edf/, EDF (Earliest Deadline First)
      • fifo/centralized/, Centralized FIFO
      • fifo/per_cpu/, Per-CPU FIFO
      • shinjuku/, Shinjuku
      • sol/, Speed-of-Light (bare-bones centralized FIFO scheduler that runs as fast as possible)
  • shared/
    • Classes to support shared-memory communication between a scheduler and another application(s). Generally, this communication is useful for the application to send scheduling hints to the scheduler.
  • tests/
    • ghOSt unit tests.
  • third_party/
    • bpf/
      • Contains the kernel BPF code for our suite of BPF tools (mentioned above). This kernel BPF code is licensed under GPLv2, so we must keep it in third_party/.
    • The rest of third_party/ contains code from third-party developers and BUILD files to compile the code.
  • util/
    • Helper utilities for ghOSt. For example, pushtosched can be used to move a batch of kernel threads from the ghOSt scheduling class to CFS (SCHED_OTHER).

Running the ghOSt Tests

We include many different tests to ensure that both the ghOSt userspace code and the ghOSt kernel code are working correctly. Some of these tests are in tests/ while others are in other subdirectories. To view all of the tests, run:

bazel query 'tests(//...)'

To build a test, such as agent_test, run:

bazel build -c opt agent_test

To run a test, launch the test binary directly:

bazel-bin/agent_test

Generally, Bazel encourages the use of bazel test when running tests. However, bazel test sandboxes the tests so that they have read-only access to /sys and are constrained in how long they can run for. However, the tests need write access to /sys/fs/ghost to coordinate with the kernel and may take a long time to complete. Thus, to avoid sandboxing, launch the test binaries directly (e.g., bazel-bin/agent_test).


Running a ghOSt Scheduler

We will run the per-CPU FIFO ghOSt scheduler and use it to schedule Linux pthreads.

  1. Build the per-CPU FIFO scheduler:
bazel build -c opt fifo_per_cpu_agent
  1. Build simple_exp, which launches a series of pthreads that run in ghOSt. simple_exp is a collection of tests.
bazel build -c opt simple_exp
  1. Launch the per-CPU FIFO ghOSt scheduler:
bazel-bin/fifo_per_cpu_agent --ghost_cpus 0-1

The scheduler launches ghOSt agents on CPUs (i.e., logical cores) 0 and 1 and will therefore schedule ghOSt tasks onto CPUs 0 and 1. Adjust the --ghost_cpus command line argument value as necessary. For example, if you have an 8-core machine and you wish to schedule ghOSt tasks on all cores, then pass 0-7 to --ghost_cpus.

  1. Launch simple_exp:
bazel-bin/simple_exp

simple_exp will launch pthreads. These pthreads in turn will move themselves into the ghOSt scheduling class and thus will be scheduled by the ghOSt scheduler. When simple_exp has finished running all tests, it will exit.

  1. Use Ctrl-C to send a SIGINT signal to fifo_per_cpu_agent to get it to stop.

Enclaves, Rebootless Upgrades, and Handling Scheduler Failures

ghOSt uses enclaves to group agents and the threads that they are scheduling. An enclave contains a subset of CPUs (i.e., logical cores) in a machine, the agents that embody those CPUs, and the threads in the ghOSt scheduling class that the enclave agents can schedule onto the enclave CPUs. For example, in the fifo_per_cpu_agent example above, an enclave is created that contains CPUs 0 and 1, though the enclave can be configured to contain any subset of CPUs in the machine, and even all of them. In the fifo_per_cpu_agent example above, two per-CPU FIFO agents enter the enclave along with the simple_exp threads when the simple_exp process is started.

Enclaves provide an easy way to partition the machine to support co-location of policies and tenants, a particularly important feature as machines scale out horizontally to contain hundreds of CPUs and new accelerators. Thus, multiple enclaves can be constructed with disjoint sets of CPUs.

Rebootless Upgrades

ghOSt supports rebootless upgrades of scheduling policies, using an enclave to encapsulate current thread and CPU state for a policy undergoing an upgrade. When you want to upgrade a policy, the agents in the new process that you launch attempt to attach to the existing enclave, waiting for the old agents running in the enclave to exit. Once the old agents exit, the new agents take over the enclave and begin scheduling.

Handling Scheduler Failures

ghOSt also recovers from scheduler failures (e.g., crashes, malfunctions, etc.) without triggering a kernel panic or machine reboot. To recover from a scheduler failure, you should generally destroy the failed scheduler's enclave and then launch the scheduler again. Destroying an enclave will kill the malfunctioning agents if necessary and will move the threads in the ghOSt scheduling class to CFS (Linux Completely Fair Scheduler) so that they can continue to be scheduled until you potentially pull them into ghOSt again.

To see all enclaves that currently exist in ghOSt, use ls to list them via ghostfs:

$ ls /sys/fs/ghost
ctl  enclave_1	version

To kill an enclave, such as enclave_1 above, run the following command, replacing enclave_1 with the name of the enclave:

echo destroy > /sys/fs/ghost/enclave_1/ctl

To kill all enclaves (which is generally useful in development), run the following command:

for i in /sys/fs/ghost/enclave_*/ctl; do echo destroy > $i; done

ghost-userspace's People

Contributors

ashwinchaugule avatar captain5050 avatar derekmauro avatar derkling avatar dohyunkim-dev avatar hannahyp avatar jackhumphries avatar neelnatu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ghost-userspace's Issues

Redefinition of function pointers from bpf_helpers.h

Hi,

It seems that the current version of ghost-userspace fails to build with the current ghost-kernel.

bazel build ... results in :

In file included from third_party/bpf/schedlat.bpf.c:21:
./third_party/bpf/common.bpf.h:32:15: error: redefinition of 'bpf_ghost_wake_agent'
static long (*bpf_ghost_wake_agent)(__u32 cpu) = (void *) 3000;
              ^
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf_helper_defs.h:3743:15: note: previous definition is here
static long (*bpf_ghost_wake_agent)(__u32 cpu) = (void *) 3000;
              ^
In file included from third_party/bpf/schedlat.bpf.c:21:
./third_party/bpf/common.bpf.h:33:15: error: redefinition of 'bpf_ghost_run_gtid'
static long (*bpf_ghost_run_gtid)(__s64 gtid, __u32 task_barrier, __s32 run_flags) = (void *) 3001;
              ^
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf_helper_defs.h:3754:15: note: previous definition is here
static long (*bpf_ghost_run_gtid)(__s64 gtid, __u32 task_barrier, __s32 run_flags) = (void *) 3001;
              ^
In file included from third_party/bpf/schedlat.bpf.c:21:
./third_party/bpf/common.bpf.h:34:15: error: redefinition of 'bpf_ghost_resched_cpu'
static long (*bpf_ghost_resched_cpu)(__u32 cpu, __u64 cpu_seqnum) = (void *) 3002;
              ^
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf_helper_defs.h:3767:15: note: previous definition is here
static long (*bpf_ghost_resched_cpu)(__u32 cpu, __u64 cpu_seqnum) = (void *) 3002;
              ^
3 errors generated.

Commenting these 3 lines in ./third_party/bpf/common.bpf.h fixes the issue and the build succeeds.
I believe the best solution would be to guard this 3 lines with a preprocessor condition.

Questions about running multiple concurrent policies on a single machine using multiple enclaves.

Hi,

Since ghost is able to run multiple concurrent policies on a single machine using multiple enclaves, so I can open two terminals to run two different schedulers concurrently by specifying groups of cpus. But now it involves the following questions:

  1. How to correctly run multiple concurrent policies on a single machine using multiple enclaves?
  2. When a new task arrives, which enclave does it enter, and what rules does it follow?
  3. Is it possible to move one running task from one enclave to another? If possible, how to do it?

Error reproduce experiments

Hello,

I am not sure if I should reproduce experiments by this command, I just want to run some tests in ghost paper, so I tried this:

➜ scripts git:(main) ✗ bazel run centralized_queuing cfs
INFO: Analyzed target //experiments/scripts:centralized_queuing (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //experiments/scripts:centralized_queuing up-to-date:
bazel-bin/experiments/scripts/centralized_queuing
INFO: Elapsed time: 0.588s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Running CFS experiments...
mount: /dev/cgroup/cpu: cgroup already mounted on /sys/fs/cgroup/systemd.
mount: /dev/cgroup/memory: cgroup already mounted on /sys/fs/cgroup/systemd.
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/com_google_ghost/experiments/scripts/centralized_queuing.py", line 94, in
app.run(main)
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/my_deps/pypi__absl_py/absl/app.py", line 312, in run
_run_main(main, args)
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/my_deps/pypi__absl_py/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/com_google_ghost/experiments/scripts/centralized_queuing.py", line 86, in main
RunCfs()
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/com_google_ghost/experiments/scripts/centralized_queuing.py", line 50, in RunCfs
Run(e)
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/com_google_ghost/experiments/scripts/run.py", line 523, in Run
SetUp(experiment.binaries)
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/com_google_ghost/experiments/scripts/setup.py", line 158, in SetUp
CopyBinaries(binaries)
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/com_google_ghost/experiments/scripts/setup.py", line 137, in CopyBinaries
tmp = UnzipPar()
File "/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles/com_google_ghost/experiments/scripts/setup.py", line 122, in UnzipPar
with zipfile.ZipFile(GetPar(), "r") as zf:
File "/usr/lib/python3.8/zipfile.py", line 1251, in init
self.fp = io.open(file, filemode)
IsADirectoryError: [Errno 21] Is a directory: '/root/.cache/bazel/_bazel_root/7433de7b30d6d58a288c26dfb16d43d1/execroot/com_google_ghost/bazel-out/k8-fastbuild/bin/experiments/scripts/centralized_queuing.runfiles'

Could you please tell me how to fix it?

Error when running CFS experiments

When I run the CFS experiment with command sudo bazel-bin/experiments/scripts/centralized_queuing.par cfs ghost, it occurs check failed error like following:
Running CFS experiments... mount: /dev/cgroup/cpu: cgroup already mounted on /dev/cgroup/cpu. mount: /dev/cgroup/memory: cgroup already mounted on /dev/cgroup/cpu. Output Directory: /tmp/ghost_data/2022-08-29 11:28:12 {"throughputs": [10000, 20000, 30000, 40000, 50000, 60000, 70000, 80000, 90000, 100000, 110000, 120000, 130000, 140000, 150000, 160000, 170000, 180000, 190000, 200000, 210000, 220000, 230000, 240000, 250000, 260000, 270000, 280000, 290000, 300000, 310000, 320000, 330000, 340000, 350000, 360000, 370000, 380000, 390000, 400000, 410000, 420000, 430000, 440000, 450000, 451000, 452000, 453000, 454000, 455000, 456000, 457000, 458000, 459000, 460000, 461000, 462000, 463000, 464000, 465000, 466000, 467000, 468000, 469000, 470000, 471000, 472000, 473000, 474000, 475000, 476000, 477000, 478000, 479000, 480000], "output_prefix": "/tmp/ghost_data/2022-08-29 11:28:12", "binaries": {"rocksdb": "/dev/shm/rocksdb", "antagonist": "/dev/shm/antagonist", "ghost": "/dev/shm/agent_shinjuku"}, "rocksdb": {"print_format": "csv", "print_distribution": false, "print_ns": false, "print_get": true, "print_range": true, "rocksdb_db_path": "/dev/shm/orch_db", "throughput": 20000, "range_query_ratio": 0.0, "load_generator_cpu": 0, "cfs_dispatcher_cpu": 1, "num_workers": 6, "worker_cpus": [2, 3, 4, 5, 6, 7], "cfs_wait_type": "spin", "ghost_wait_type": "prio_table", "get_duration": "10us", "range_duration": "5000us", "get_exponential_mean": "1us", "batch": 1, "experiment_duration": "15s", "discard_duration": "2s", "scheduler": "cfs", "ghost_qos": 2}, "antagonist": null, "ghost": null} Running experiment for throughput = 10000 req/s: ['/dev/shm/rocksdb', '--print_format', 'csv', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.0', '--load_generator_cpu', '0', '--cfs_dispatcher_cpu', '1', '--num_workers', '6', '--worker_cpus', '2,3,4,5,6,7', '--cfs_wait_type', 'spin', '--ghost_wait_type', 'prio_table', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '1us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'cfs', '--ghost_qos', '2', '--throughput', '10000'] experiments/rocksdb/orchestrator.cc:97(88745) CHECK FAILED: options_.load_generator_cpu != kBackgroundThreadCpu [0 == 0] PID 88745 Backtrace: [0] 0x56421b36c468 : ghost_test::Orchestrator::Orchestrator() [1] 0x56421b34dc30 : ghost_test::CfsOrchestrator::CfsOrchestrator() [2] 0x56421b32afe2 : main [3] 0x7f8ed2b2dd90 : (unknown)

Fail to run the shinjuku.py for a "IsADirectoryError"

Environment

OS

ghOSt kernel, a VMware Workstation machine

file directory level

/home/xiunian/ghost/ghost-userspace/experiments/scripts/shinjuku.py

My problem

Fail to run shinjuku.py for this reason :

xiunian@xiunian-virtual-machine:~/ghost/ghost-userspace$ sudo python experiments/scripts/shinjuku.py ghost
Running ghOSt experiments...
mount: /dev/cgroup/cpu: cgroup already mounted on /dev/cgroup/cpu.
mount: /dev/cgroup/memory: cgroup already mounted on /dev/cgroup/cpu.
Traceback (most recent call last):
  File "/home/xiunian/ghost/ghost-userspace/experiments/scripts/shinjuku.py", line 84, in <module>
    app.run(main)
  File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/xiunian/ghost/ghost-userspace/experiments/scripts/shinjuku.py", line 80, in main
    RunGhost()
  File "/home/xiunian/ghost/ghost-userspace/experiments/scripts/shinjuku.py", line 58, in RunGhost
    Run(e)
  File "/home/xiunian/ghost/ghost-userspace/experiments/scripts/run.py", line 515, in Run
    SetUp(experiment.binaries)
  File "/home/xiunian/ghost/ghost-userspace/experiments/scripts/setup.py", line 150, in SetUp
    CopyBinaries(binaries)
  File "/home/xiunian/ghost/ghost-userspace/experiments/scripts/setup.py", line 129, in CopyBinaries
    tmp = UnzipPar()
  File "/home/xiunian/ghost/ghost-userspace/experiments/scripts/setup.py", line 114, in UnzipPar
    with zipfile.ZipFile(GetPar(), "r") as zf:
  File "/usr/lib/python3.10/zipfile.py", line 1249, in __init__
    self.fp = io.open(file, filemode)
IsADirectoryError: [Errno 21] Is a directory: '/home/xiunian/ghost'

Could you tell me how to resolve this problem? Thanks.

Error trying to run bazel run.

Hello Ghost Team,

I am trying to run bazel to build/test the fifo_agent, I am getting the error below. Please advise.

Thanks

ubuntu@ubuntu:~/ghost-userspace$ sudo bazel run fifo_agent --sandbox
ERROR: --sandbox :: Unrecognized option: --sandbox
ubuntu@ubuntu:~/ghost-userspace$ sudo bazel run fifo_agent --sandbox_debug
INFO: Analyzed target //:fifo_agent (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: From Compiling lib/agent.cc:
1640223142.796646615: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1640223142.796710627: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1640223142.796910093: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 7308
1640223142.796955246: src/main/tools/linux-sandbox-pid1.cc:447: Pid1Main started
1640223142.797023784: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1640223142.797180864: src/main/tools/linux-sandbox-pid1.cc:189: working dir: /root/.cache/bazel/_bazel_root/a45884f0                                                 d9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/6/execroot/com_google_ghost
1640223142.797196011: src/main/tools/linux-sandbox-pid1.cc:221: writable: /root/.cache/bazel/_bazel_root/a45884f0d9c                                                 9471c8dde34f8ae70b628/sandbox/linux-sandbox/6/execroot/com_google_ghost
1640223142.797202278: src/main/tools/linux-sandbox-pid1.cc:221: writable: /tmp
1640223142.797208014: src/main/tools/linux-sandbox-pid1.cc:221: writable: /dev/shm
1640223142.797274989: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /
1640223142.797284449: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev
1640223142.797290030: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/pts
1640223142.797295847: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1640223142.797300776: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/hugepages
1640223142.797306066: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/mqueue
1640223142.797311393: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run
1640223142.797316731: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/lock
1640223142.797322075: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/snapd/ns
1640223142.797327765: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000
1640223142.797333256: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys
1640223142.797338350: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/security
1640223142.797345475: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup
1640223142.797351899: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/unified
1640223142.797379252: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/systemd
1640223142.797385486: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/freezer
1640223142.797391147: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpuset
1640223142.797396874: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/pids
1640223142.797402622: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpu,cpuacct
1640223142.797408647: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/net_cls,net_prio
1640223142.797414784: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/devices
1640223142.797420367: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/rdma
1640223142.797425911: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/blkio
1640223142.797431355: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/memory
1640223142.797436847: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/hugetlb
1640223142.797442357: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/perf_event
1640223142.797465714: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/pstore
1640223142.797472195: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/bpf
1640223142.797478237: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/debug
1640223142.797484303: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/tracing
1640223142.797490214: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/fuse/connections
1640223142.797497352: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/config
1640223142.797503251: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc
1640223142.797508408: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/2253
1640223142.797521540: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/2128
1640223142.797527631: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core20/1270
1640223142.797533610: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/lxd/21835
1640223142.797539845: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/lxd/21029
1640223142.797544753: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/12704
1640223142.797549777: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/14295
1640223142.797554650: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /boot
1640223142.797564212: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /root/.cache/bazel/_bazel_root/a45884f0d                                                 9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/6/execroot/com_google_ghost
1640223142.797571982: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /root/.cache/bazel/_bazel_root/a45884f0d                                                 9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/6/execroot/com_google_ghost
1640223142.797577099: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /tmp
1640223142.797582722: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1640223142.797638017: src/main/tools/linux-sandbox-pid1.cc:376: calling fork...
1640223142.797742958: src/main/tools/linux-sandbox-pid1.cc:406: child started with PID 2
1640223145.229695822: src/main/tools/linux-sandbox-pid1.cc:423: wait returned pid=2, status=0x00
1640223145.229711681: src/main/tools/linux-sandbox-pid1.cc:441: child exited normally with code 0
1640223145.230027295: src/main/tools/linux-sandbox.cc:233: child exited normally with code 0
INFO: From Compiling schedulers/fifo/fifo_agent.cc:
1640223142.788638180: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1640223142.788676729: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1640223142.788912483: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 7301
1640223142.789340742: src/main/tools/linux-sandbox-pid1.cc:447: Pid1Main started
1640223142.789417087: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1640223142.789560399: src/main/tools/linux-sandbox-pid1.cc:189: working dir: /root/.cache/bazel/_bazel_root/a45884f0                                                 d9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/7/execroot/com_google_ghost
1640223142.789574535: src/main/tools/linux-sandbox-pid1.cc:221: writable: /root/.cache/bazel/_bazel_root/a45884f0d9c                                                 9471c8dde34f8ae70b628/sandbox/linux-sandbox/7/execroot/com_google_ghost
1640223142.789580678: src/main/tools/linux-sandbox-pid1.cc:221: writable: /tmp
1640223142.789586359: src/main/tools/linux-sandbox-pid1.cc:221: writable: /dev/shm
1640223142.789643643: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /
1640223142.789652290: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev
1640223142.789657638: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/pts
1640223142.789663574: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1640223142.789668594: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/hugepages
1640223142.789674169: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/mqueue
1640223142.789679401: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run
1640223142.789684818: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/lock
1640223142.789690234: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/snapd/ns
1640223142.789695785: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000
1640223142.789701195: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys
1640223142.789706265: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/security
1640223142.789713106: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup
1640223142.789719416: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/unified
1640223142.789745984: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/systemd
1640223142.789752159: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/freezer
1640223142.789757854: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpuset
1640223142.789763387: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/pids
1640223142.789769005: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpu,cpuacct
1640223142.789774895: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/net_cls,net_prio
1640223142.789780857: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/devices
1640223142.789786391: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/rdma
1640223142.789791942: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/blkio
1640223142.789797431: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/memory
1640223142.789802948: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/hugetlb
1640223142.789808446: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/perf_event
1640223142.789832201: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/pstore
1640223142.789838328: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/bpf
1640223142.789844048: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/debug
1640223142.789849700: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/tracing
1640223142.789855647: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/fuse/connections
1640223142.789862789: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/config
1640223142.789868542: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc
1640223142.789873524: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/2253
1640223142.789885312: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/2128
1640223142.789890561: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core20/1270
1640223142.789896118: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/lxd/21835
1640223142.789901219: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/lxd/21029
1640223142.789906077: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/12704
1640223142.789911148: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/14295
1640223142.789915743: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /boot
1640223142.789925193: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /root/.cache/bazel/_bazel_root/a45884f0d                                                 9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/7/execroot/com_google_ghost
1640223142.789930978: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /root/.cache/bazel/_bazel_root/a45884f0d                                                 9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/7/execroot/com_google_ghost
1640223142.789935959: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /tmp
1640223142.789940876: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1640223142.789984269: src/main/tools/linux-sandbox-pid1.cc:376: calling fork...
1640223142.790105333: src/main/tools/linux-sandbox-pid1.cc:406: child started with PID 2
1640223145.877179935: src/main/tools/linux-sandbox-pid1.cc:423: wait returned pid=2, status=0x00
1640223145.877194767: src/main/tools/linux-sandbox-pid1.cc:441: child exited normally with code 0
1640223145.877502930: src/main/tools/linux-sandbox.cc:233: child exited normally with code 0
ERROR: /home/ubuntu/ghost-userspace/BUILD:33:11: Compiling lib/enclave.cc failed: (Exit 1): linux-sandbox failed: er                                                 ror executing command
  (cd /root/.cache/bazel/_bazel_root/a45884f0d9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/9/execroot/com_google_gh                                                 ost && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /root/.cache/bazel/_bazel_root/install/64841bf12de13c7518c7ada0994bafe2/linux-sandbox -t 15 -w /root/.cache/bazel/                                                 _bazel_root/a45884f0d9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/9/execroot/com_google_ghost -w /tmp -w /dev/shm -                                                 D -- /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno                                                 -omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/_objs/agent/enclave.pic.d '-frandom-seed=bazel-o                                                 ut/k8-fastbuild/bin/_objs/agent/enclave.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/c                                                 om_google_absl -iquote bazel-out/k8-fastbuild/bin/external/com_google_absl -isystem bazel-out/k8-fastbuild/bin/exter                                                 nal/linux/libbpf/include '-std=c++17' -Wno-sign-compare -DGHOST_LOGGING -fno-canonical-system-headers -Wno-builtin-m                                                 acro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c lib/enclave.cc -o baz                                                 el-out/k8-fastbuild/bin/_objs/agent/enclave.pic.o)
1640223142.812675576: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1640223142.812764790: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1640223142.812968920: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 7317
1640223142.813049493: src/main/tools/linux-sandbox-pid1.cc:447: Pid1Main started
1640223142.813226321: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1640223142.813362884: src/main/tools/linux-sandbox-pid1.cc:189: working dir: /root/.cache/bazel/_bazel_root/a45884f0                                                 d9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/9/execroot/com_google_ghost
1640223142.813378044: src/main/tools/linux-sandbox-pid1.cc:221: writable: /root/.cache/bazel/_bazel_root/a45884f0d9c                                                 9471c8dde34f8ae70b628/sandbox/linux-sandbox/9/execroot/com_google_ghost
1640223142.813384275: src/main/tools/linux-sandbox-pid1.cc:221: writable: /tmp
1640223142.813389998: src/main/tools/linux-sandbox-pid1.cc:221: writable: /dev/shm
1640223142.813446055: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /
1640223142.813454929: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev
1640223142.813460479: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/pts
1640223142.813465874: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1640223142.813470805: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/hugepages
1640223142.813476093: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/mqueue
1640223142.813481369: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run
1640223142.813486649: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/lock
1640223142.813492037: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/snapd/ns
1640223142.813497548: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000
1640223142.813503328: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys
1640223142.813508533: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/security
1640223142.813515171: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup
1640223142.813521439: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/unified
1640223142.813547696: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/systemd
1640223142.813553880: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/freezer
1640223142.813559654: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpuset
1640223142.813565247: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/pids
1640223142.813570896: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpu,cpuacct
1640223142.813576699: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/net_cls,net_prio
1640223142.813582514: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/devices
1640223142.813588047: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/rdma
1640223142.813593590: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/blkio
1640223142.813599187: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/memory
1640223142.813604847: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/hugetlb
1640223142.813610526: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/perf_event
1640223142.813635175: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/pstore
1640223142.813641368: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/bpf
1640223142.813647106: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/debug
1640223142.813652851: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/tracing
1640223142.813658549: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/fuse/connections
1640223142.813666076: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/config
1640223142.813671824: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc
1640223142.813676872: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/2253
1640223142.813713350: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/2128
1640223142.813719234: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core20/1270
1640223142.813724486: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/lxd/21835
1640223142.813729996: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/lxd/21029
1640223142.813734930: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/12704
1640223142.813740010: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/14295
1640223142.813744748: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /boot
1640223142.813755539: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /root/.cache/bazel/_bazel_root/a45884f0d                                                 9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/9/execroot/com_google_ghost
1640223142.813761655: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /root/.cache/bazel/_bazel_root/a45884f0d                                                 9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/9/execroot/com_google_ghost
1640223142.813766581: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /tmp
1640223142.813771530: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1640223142.813819570: src/main/tools/linux-sandbox-pid1.cc:376: calling fork...
1640223142.813931339: src/main/tools/linux-sandbox-pid1.cc:406: child started with PID 2
In file included from ./bpf/user/agent.h:22,
                 from lib/enclave.cc:25:
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf.h:244:6: error: use of enum 'bpf_stats_type' wit                                                 hout previous declaration
  244 | enum bpf_stats_type; /* defined in up-to-date linux/bpf.h */
      |      ^~~~~~~~~~~~~~
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf.h:245:38: error: use of enum 'bpf_stats_type' wi                                                 thout previous declaration
  245 | LIBBPF_API int bpf_enable_stats(enum bpf_stats_type type);
      |                                      ^~~~~~~~~~~~~~
1640223145.909607576: src/main/tools/linux-sandbox-pid1.cc:423: wait returned pid=2, status=0x100
1640223145.909619940: src/main/tools/linux-sandbox-pid1.cc:441: child exited normally with code 1
1640223145.909911880: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1
Target //:fifo_agent failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 4.007s, Critical Path: 3.16s
INFO: 6 processes: 4 internal, 2 linux-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

sched_setscheduler(tid) failed: Bad file descriptor when using pushtosched

Hello,
I have meet an issue when i tried to spawn threads of multithreads app in ghost kernel. Just recently with my previous computer i was able to spawn threads on ghost kernel normally by using pushtosched. Today i now use a new computer, i have compiled and install ghost kernel on Ubuntu 20.04 but after the installation i had met 02 issues.
The first issue was about ghost-userspace compilation where during the compilation I had met an error with the following lines in third_party/bpf/common.bpf.h:

static long (*bpf_ghost_wake_agent)(__u32 cpu) = (void *) 3000;
static long (*bpf_ghost_run_gtid)(__s64 gtid, __u32 task_barrier, __s32 run_flags) = (void *) 3001;
static long (*bpf_ghost_resched_cpu)(__u32 cpu, __u64 cpu_seqnum) = (void *) 3002;

image

To skip this issue I have commented those three lines and the compilation completed successfully.

The second issue was about /sys/fs/ghost/ folder which was not present in my computer, but i had followed #18 (modify the init/kconfig file in ghost-kernel) and that solution solved my problem. I am now able to see /sys/fs/ghost folder.

My problem now is that when I use pushtosched app to spawn thread on ghost kernel. It fail with the following error: sched_setscheduler(tid) failed: Bad file descriptor where tid is the process id of client app.

image

Please can you help me to solve this issue.
thanks.

The ghost is support docker application as well ?

1.we run the docker tensorflowserving in the server pc

image

2.we write the pid to the ghost by the command "echo PID > /sys/fs/ghost/enclave_1/tasks"

image

3.then we run the client to test the tensorflowing continus by the client pc

4.we use the command "mpstat" to see whether if it is really run on the cpu 0 and 1 ,but it is not, as picture show:

image

thanks very much !

Forcibly closing the agent while program running on enclave

I put a program running on the enclave of FIFO and assigned CPU numbers 1-16 to the enclave of fifo.
When the application is executing, the content in /sys/fs/ghost is not deleted after I send ctrl+c SIGINT signal to ghost's interrupt. When I reallocate the corresponding cpus (1-16) as enclave, I get an error below:

Initializing...
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
F0000 00:00:1685453511.454808 1712793 enclave.cc:458] Check failed: retries < kMaxRetries (10 vs. 10)
*** Check failure stack trace: ***
    @     0x5641eab0b504  absl::log_internal::LogMessage::Flush()
    @     0x5641eab0b61d  absl::log_internal::LogMessageFatal::~LogMessageFatal()
    @     0x5641eaaaa222  ghost::LocalEnclave::CreateAndAttachToEnclave()
    @     0x5641eaac1995  ghost::LocalEnclave::LocalEnclave()
    @     0x5641eaab1027  ghost::FullFifoAgent<>::FullFifoAgent()
    @     0x5641eaab21b4  ghost::AgentProcess<>::AgentProcess()
    @     0x5641eaaae50e  main
    @     0x7ff330e16083  __libc_start_main

/sys/fs/ghost:

$ pwd
/sys/fs/ghost
$ ls
ctl  enclave_1  enclave_2  enclave_3  enclave_4  enclave_5  version
$ cat enclave_*/cpulist
1-16





Can I forcibly close the fifo-agent when the application is executing in the enclave? (As I understand it, I can.) But is there a more elegant way to do it, since the information is deleted from /sys/fs/ghost?

Compile agent_shinjuku in raspberry pi 4

Hi,

I want to get ghost-userspace working on the raspberry pi 4. So, I modified the kernel configuration and get the 5.11+ kernel compiled and installed on RPI4. Until this step, everything looks good.

Then, I apt-get install the required package and compiled/installed the bazel 5.2 from scratch.
Issues come out while I'm trying to sudo bazel build -c opt agent_shinjiku:
The top of the error message:

ERROR: /home/ubuntu/ghost-userspace/BUILD:33:11: Compiling lib/enclave.cc failed: (Exit 1): linux-sandbox failed: error executing command 
  (cd /root/.cache/bazel/_bazel_root/a45884f0d9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/103/execroot/com_google_ghost && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /root/.cache/bazel/_bazel_root/install/5d68add8a862e0c465e5b7d6cddc0f77/linux-sandbox -t 15 -w /root/.cache/bazel/_bazel_root/a45884f0d9c9471c8dde34f8ae70b628/sandbox/linux-sandbox/103/execroot/com_google_ghost -w /tmp -w /dev/shm -D -- /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/aarch64-opt/bin/_objs/agent/enclave.d '-frandom-seed=bazel-out/aarch64-opt/bin/_objs/agent/enclave.o' -iquote . -iquote bazel-out/aarch64-opt/bin -iquote external/com_google_absl -iquote bazel-out/aarch64-opt/bin/external/com_google_absl -isystem bazel-out/aarch64-opt/bin/external/linux/libbpf/include '-std=c++17' -Wno-sign-compare -DGHOST_LOGGING -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c lib/enclave.cc -o bazel-out/aarch64-opt/bin/_objs/agent/enclave.o)

The tail of the error message:

/tmp/ccAPf5Pj.s: Assembler messages:
/tmp/ccAPf5Pj.s:2496: Error: unknown mnemonic `pause' -- `pause'
/tmp/ccAPf5Pj.s:12185: Error: unknown mnemonic `pause' -- `pause'
1648844166.528682230: src/main/tools/linux-sandbox-pid1.cc:498: wait returned pid=2, status=0x100
1648844166.528754655: src/main/tools/linux-sandbox-pid1.cc:516: child exited normally with code 1
1648844166.529752103: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1

The intermediate message is in this log.compile.log

I tried to look for the possible issue, but the solutions vary.
Any idea or suggestion will be greatly appreciated!

Errors when reproducing experiments (also when running userspace agents)

Description

Logs

I am trying to reproduce the experiments, and further do something new with ghOst.
But I came across an error below:
(Before I run this command, I finished to compiled the ghost-userspace with bazel build -c opt ...)

sudo ./bazel-bin/experiments/scripts/centralized_queuing.par cfs   # in the root of ghost-userspace. 
# I use "sudo" because it seems the python script ends with "Running CFS experiments... mount: only root can use "--options" option"

It turns out to be

Running CFS experiments...
mount: /dev/cgroup/cpu: cgroup already mounted on /sys/fs/cgroup/systemd.
mount: /dev/cgroup/memory: cgroup already mounted on /sys/fs/cgroup/systemd.
Output Directory: /tmp/ghost_data/2022-04-26 10:22:56                                                                                                                                                                                                                                   
{"throughputs": [10000, 20000, 30000, 40000, 50000, 60000, 70000, 80000, 90000, 100000, 110000, 120000, 130000, 140000, 150000, 160000, 170000, 180000, 190000, 200000, 210000, 220000, 230000, 240000, 250000, 260000, 270000, 280000, 290000, 300000, 310000, 320000, 330000, 340000, 
350000, 360000, 370000, 380000, 390000, 400000, 410000, 420000, 430000, 440000, 450000, 451000, 452000, 453000, 454000, 455000, 456000, 457000, 458000, 459000, 460000, 461000, 462000, 463000, 464000, 465000, 466000, 467000, 468000, 469000, 470000, 471000, 472000, 473000, 474000, 
475000, 476000, 477000, 478000, 479000, 480000], "output_prefix": "/tmp/ghost_data/2022-04-26 10:22:56", "binaries": {"rocksdb": "/dev/shm/rocksdb", "antagonist": "/dev/shm/antagonist", "ghost": "/dev/shm/agent_shinjuku"}, "rocksdb": {"print_format": "csv", "print_distribution": 
false, "print_ns": false, "print_get": true, "print_range": true, "rocksdb_db_path": "/dev/shm/orch_db", "throughput": 20000, "range_query_ratio": 0.0, "load_generator_cpu": 10, "cfs_dispatcher_cpu": 11, "num_workers": 6, "worker_cpus": [12, 13, 14, 15, 16, 17], "cfs_wait_type": 
"spin", "ghost_wait_type": "prio_table", "get_duration": "10us", "range_duration": "5000us", "get_exponential_mean": "1us", "batch": 1, "experiment_duration": "15s", "discard_duration": "2s", "scheduler": "cfs", "ghost_qos": 2}, "antagonist": null, "ghost": null}
Running experiment for throughput = 10000 req/s:
['/dev/shm/rocksdb', '--print_format', 'csv', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.0', '--load_generator_cpu', '10', '--cfs_dispatcher_
cpu', '11', '--num_workers', '6', '--worker_cpus', '12,13,14,15,16,17', '--cfs_wait_type', 'spin', '--ghost_wait_type', 'prio_table', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '1us', '--batch', '1', '--experiment_duration', '15s', '--discar
d_duration', '2s', '--scheduler', 'cfs', '--ghost_qos', '2', '--throughput', '10000']
experiments/rocksdb/cfs_orchestrator.cc:95(23984) CHECK FAILED: ghost::Ghost::SchedSetAffinity( ghost::Gtid::Current(), ghost::MachineTopology()->ToCpuList( std::vector<int>{options().load_generator_cpu})) == 0 [-1 != 0]
errno: 22 [Invalid argument]
PID 23984 Backtrace:
[0] 0x564e0ac5e487 : ghost_test::CfsOrchestrator::LoadGenerator()
[1] 0x564e0ac8561e : ghost_test::ExperimentThreadPool::ThreadMain()
[2] 0x564e0ac8756b : std::_Function_handler<>::_M_invoke()
[3] 0x7fe9aeb77de4 : (unknown)

Furthermore, I also tried this command under root of ghost-userspace

sudo bazel run fifo_agent

and it turns out to

Extracting Bazel installation...                                                                                                                                                                                                                                                        
Starting local Bazel server and connecting to it...                                                                                                                                                                                                                                     
ERROR: Skipping 'fifo_agent': no such target '//:fifo_agent': target 'fifo_agent' not declared in package '' defined by /home/emnets/ghost-userspace/BUILD                                                                                                                              
WARNING: Target pattern parsing failed.                                                                                                                                                                                                                                                 
ERROR: no such target '//:fifo_agent': target 'fifo_agent' not declared in package '' defined by /home/emnets/ghost-userspace/BUILD                                                                                                                                                     
INFO: Elapsed time: 10.222s                                                                                                                                                                                                                                                             
INFO: 0 processes.                                                                                                                                                                                                                                                                      
FAILED: Build did NOT complete successfully (1 packages loaded)                                                                                                                                                                                                                         
FAILED: Build did NOT complete successfully (1 packages loaded)

Env Info

And here is my environment version info:

lsb_release -a 
# LSB Version:	core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch 
# Distributor ID:	Ubuntu 
# Description:	Ubuntu 20.04.2 LTS 
# Release:	20.04 
# Codename:	focal
uname -mrs
# Linux 5.11.0+ x86_64

ghost-kernel hash: 5da05ec77890217e85947ff3573e1480579687d2
ghost-userspace hash: 79ecaeb

P.S. I am using a virtual machine to reproduce ghost. I am wondering if the virtual machine matters.
The virtual machine is using VMware workstation, with 8GB mem and 8 processors (each processor has one core).

Suggestion

My colleagues and I appreciate the paper and this open-source project of ghost.
But I came across many difficulties to conduct the experiments and reproduce the results, because the README do not mentioned this.
I have to refer the the (closed) issues and find the scattered commands to run the experiments.
If you could update the README with a more detailed steps, this will be great. And I could help you out if you need.

use of enum 'bpf_stats_type' without previous declaration

I have installed the ghost-kernel in ubuntu-20, with gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1).
And I replaced build --cxxopt='-std=c++20' with build --cxxopt='-std=c++2a' in the .bazelrc file.

When I Build it with bazel build -c opt agent, I got the following errors :

INFO: Analyzed target //:agent (27 packages loaded, 163867 targets configured).
INFO: Found 1 target...
ERROR: /home/kingdo/ghost-userspace/BUILD:36:11: Compiling lib/enclave.cc failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 29 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from ./bpf/user/agent.h:22,
                 from lib/enclave.cc:26:
bazel-out/k8-opt/bin/external/linux/libbpf/include/libbpf/bpf.h:244:6: error: use of enum 'bpf_stats_type' without previous declaration
  244 | enum bpf_stats_type; /* defined in up-to-date linux/bpf.h */
      |      ^~~~~~~~~~~~~~
bazel-out/k8-opt/bin/external/linux/libbpf/include/libbpf/bpf.h:245:38: error: use of enum 'bpf_stats_type' without previous declaration
  245 | LIBBPF_API int bpf_enable_stats(enum bpf_stats_type type);
      |                                      ^~~~~~~~~~~~~~
Target //:agent failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 420.627s, Critical Path: 313.27s
INFO: 113 processes: 14 internal, 99 linux-sandbox.
FAILED: Build did NOT complete successfully

questions about MSG_TASK_ON_CPU

hi,
Could you please help me understand under what circumstances the message "MSG_TASK_ON_CPU" is generated? I have not seen this message being issued by the kernel in most cases. Is there a particular configuration required to enable the kernel to send this message? Thank you very much for your assistance.

Questions about enclave

Hello, ALL.

It seems that the enclaves in ghOSt is implemented through Linux's cgroup, which is an OS-level resource isolation solution.

I wonder if the enclave in this paper has any connection with Intel’s Software Guard Extensions (SGX) technology? If so, does it mean that ghOSt can only run on Intel processors with SGX support?

Thank You!

How to solve lib/enclave.cc:404(8388) CHECK FAILED: ctl_fd_ >= 0 [-1 > 0] problem ?

Hello,
Please excuse me to reopen this issue but I have meet a similar error when I had try to run the command bazel run agent_shinjuku in a root mode. this is the error that I obtain:
image
I think that it because I don't have a right to write into /sys/fs/ghost/ folder.
Please I don't know how I can change the permission of that folder, since when I try to run the command: sudo chmod -R 777 /sys/fs/ghost/ I just obtain Operation is not permitted as result.
Please I need your help.

Originally posted by @NGUETOUM in #2 (comment)

bazel build error about platforms

Ghost started with qemu

sudo qemu-system-x86_64 \
    -enable-kvm -cpu host -smp 2 -m 2048 -no-reboot -nographic \
    -kernel ghost-kernel/arch/x86/boot/bzImage \
    -drive id=root,media=disk,file=ubuntu.qcow2 \
    -net nic,macaddr=a6:5a:0d:31:01:46 -net tap,ifname=tap0 \
    -append "root=/dev/sda1 console=ttyS0"

When I build ghost-userspace, I get the following error
image

ERROR: /root/.cache/bazel/_bazel_root/a45884f0d9c9471c8dde34f8ae70b628/external/bazel_tools/platforms/BUILD:64:6: in alias rule @bazel_tools//platforms:ios: Constraints from @bazel_tools//platforms have been removed. Please use constraints from @platforms repository embedded in Bazel, or preferably declare dependency on https://github.com/bazelbuild/platforms. See https://github.com/bazelbuild/bazel/issues/8622 for details.
ERROR: /root/.cache/bazel/_bazel_root/a45884f0d9c9471c8dde34f8ae70b628/external/bazel_tools/platforms/BUILD:64:6: Analysis of target '@bazel_tools//platforms:ios' failed
ERROR: /root/.cache/bazel/_bazel_root/a45884f0d9c9471c8dde34f8ae70b628/external/com_google_absl/absl/time/internal/cctz/BUILD.bazel:53:11: errors encountered resolving select() keys for @com_google_absl//absl/time/internal/cctz:time_zone
ERROR: Analysis of target '//:agent_biff' failed; build aborted: 
INFO: Elapsed time: 0.138s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

Is it because I started with qemu or because my bazel has a problem?

An error when moving thread to agent_shinjuku

Sorry to trouble you again. When I run agent_shinjuku and write the user-state thread number to /sys/fs/ghost/enclave_ 1/tasks, the following error was encountered:

$:bazel-bin/agent_shinjuku --ncpus 2
Core map
(  0 )	(  1 )	(  2 )	(  3 )	
Initializing...
Initialization complete, ghOSt active.
PID 2517 Fatal segfault at addr 0x10: 
[0] 0x7fb0313c13c0 : __restore_rt
[1] 0x55ce4c8378a5 : ghost::BasicDispatchScheduler<>::DispatchMessage()
[2] 0x55ce4c8368c9 : ghost::ShinjukuAgent::AgentThread()
[3] 0x55ce4c83ab0a : ghost::LocalAgent::ThreadBody()
[4] 0x7fb031612de4 : (unknown)

When I run the agent again_shinjuku or other programs will get such errors:

lib/enclave.cc:452(2716) CHECK FAILED: retries < kMaxRetries [10 >= 10]
errno: 16 [Device or resource busy]
PID 2716 Backtrace:
[0] 0x55b4422162f6 : ghost::LocalEnclave::CreateAndAttachToEnclave()
[1] 0x55b442216de5 : ghost::LocalEnclave::LocalEnclave()
[2] 0x55b4421cfab1 : ghost::AgentProcess<>::AgentProcess()
[3] 0x55b4421ca9d9 : main
[4] 0x7fd96a91d0b3 : __libc_start_main

How to solve the first error? How can I run another program without reboot? Thank you!

How to check the scheduling time slice of the scheduling algorithm in Ghost?

In the 5.11.0 kernel, sched_ Min_ Granularity_ ns value is 3000000 nanoseconds, and I think this is a scheduling time slice of 3ms for cfs. This means that each ready task will not take more than 3ms to execute on the CPU, and then there will be a scheduling switch, right? So I would like to inquire about where to check the values for cfs and fifo in the ghost system. Thank you very much!

Error occurs during the shinjuku_shenango ghost experiment

I install the ghost-kernel and compiled the ghost-userspace, but when I tried to reproduce the shinjuku_shenango ghost experiment, an error occurred.

Running ghOSt experiments...
mount: /dev/cgroup/cpu: cgroup already mounted on /sys/fs/cgroup/systemd.
mount: /dev/cgroup/memory: cgroup already mounted on /sys/fs/cgroup/systemd.
Output Directory: /tmp/ghost_data/2022-03-09 22:17:06
{"throughputs": [10000, 20000, 30000, 40000, 50000, 60000, 70000, 80000, 90000, 100000, 110000, 120000, 130000, 140000, 141000, 142000, 143000, 144000, 145000, 146000, 147000, 148000, 149000, 150000], "output_prefix": "/tmp/ghost_data/2022-03-09 22:17:06", "binaries": {"rocksdb": "/dev/shm/rocksdb", "antagonist": "/dev/shm/antagonist", "ghost": "/dev/shm/agent_shinjuku"}, "rocksdb": {"print_format": "pretty", "print_distribution": false, "print_ns": false, "print_get": true, "print_range": true, "rocksdb_db_path": "/dev/shm/orch_db", "throughput": 20000, "range_query_ratio": 0.005, "load_generator_cpu": 10, "cfs_dispatcher_cpu": 11, "num_workers": 200, "worker_cpus": [], "cfs_wait_type": "spin", "get_duration": "10us", "range_duration": "5000us", "get_exponential_mean": "0us", "batch": 1, "experiment_duration": "15s", "discard_duration": "2s", "scheduler": "ghost", "ghost_qos": 2}, "antagonist": {"print_format": "pretty", "work_share": 1.0, "num_threads": 7, "cpus": [], "experiment_duration": "15s", "scheduler": "ghost", "ghost_qos": 1}, "ghost": {"firstcpu": 11, "globalcpu": 11, "ncpus": 7, "preemption_time_slice": "30us"}}
Running experiment for throughput = 10000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
libbpf: map 'cpu_data': failed to create: Operation not permitted(-1)
libbpf: failed to load object 'ghost_bpf'
libbpf: failed to load BPF skeleton 'ghost_bpf': -1
lib/enclave.cc:442(25238) CHECK FAILED: agent_bpf_init(tick_on_request) == 0 [-1 != 0]
errno: 1 [Operation not permitted]
PID 25235 Backtrace:
[0] 0x564165c7e53b : ghost::Exit()
[1] 0x564165c47fa9 : ghost::LocalEnclave::LocalEnclave()
[2] 0x564165bda26f : ghost::FullAgent<>::FullAgent()
[3] 0x564165bd90d3 : ghost::FullShinjukuAgent<>::FullShinjukuAgent()
[4] 0x564165bd7a1d : std::make_unique<>()
[5] 0x564165bd68b1 : ghost::AgentProcess<>::AgentProcess()
[6] 0x564165bd2610 : main
[7] 0x7fb7de0220b3 : __libc_start_main

It seems that it failed to create the map for 'cpu_data' because of 'Operation not permitted', but I have the root priviledge and I checked that all the users have rwx rights to the /sys/fs/ghost/enclave_x/cpu_data files. Any suggestions for the error? I tried reboot but it does not fix this issue.

Issues running scheduler

Hello,

This may be a simple question but as I try to run the scheduler (currently fifo but issue arises with agent shinjuku as well), difficulties arise with the command

sudo bazel run fifo_agent

Running this command produces the error
lib/enclave.cc:405(24972) CHECK FAILED: write(cpumask.c_str(), cpumask.length() == cpumask.length() [-1 != 2] errno: 16 [Device or resource busy]

Upon closer inspection, I believe the issue has to do with the fact that I cannot read or write to any of the enclave folders or their subdirectories and files in

sys/fs/ghost/enclave_#/ctl

Since I cannot write to files, write produces -1. This is true regardless of me being root or not.

I was curious if there is anything I am missing in running ghost-user space or if there is a resolution to me not being able to write to the files in the enclave folder.

I appreciate your help, thank you so much!

how to run rocksdb application by two steps and test it ?

sudo bazel-bin/experiments/scripts/centralized_queuing.par cfs ghost can successfully reproduce the experiment

but i am confused with the experiment, so i want to reproduce by two step as README.md:

  1. first Launch the ghOSt scheduler(CFS or shinjuku)
  2. second then Launch rocksdb
  3. then test it ?

can you help me ? thank you very much !

How can i check the application actually run on the ghost?

1.I get the pid of the application
2.run the ghost agent
3.then run the command echo PID > /sys/fs/ghost/enclave_1/tasks
by the three step, i think i run the appication on the ghost, but i want check it whether it actually run on the ghost agent? how can i do?
thanks ,very much!

Which measurement unit is used when we get runtime value from kernel with status_word ?

Hi,
Good morning to you all, I just have a simple question, I want to know which measurement unit is used when we get runtime value from kernel. For example by doing something like task->status_word.runtime(). It is a Millisecond unit or Microsecond unit or Nanosecond unit which is used since the runtime function just return uint64_t value ? Thanks for your help.

par file generation.

Hi,
It's my first time to use 'bazel' and just come across a very naive question....
It looks like some par files are required by the GetPar() function in experiment/scripts/setup.py.
I used the bazel build -c opt ... to compile the code but it didn't generate the par file...
Did I miss anything? Any idea where I should configure it?

Any advises will be appreciated!

Error: failed to open BPF object file: Relocation failed

Good Morning,
I've compiled the kernel and replaced my previous kernel in my ubuntu, then when using the command sudo bazel build -c opt ... to compile ghost-userspace, I got this error:

ERROR: /home/armel/Desktop/ghost/ghost-userspace/BUILD:455:13: Executing genrule //:schedghostidle_bpf_skel failed: (Exit 255): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
libbpf: prog 'sched_ghost_latched': invalid relo against 'nr_latches' in special section 0xfff2; forgot to initialize global var?..
Error: failed to open BPF object file: Relocation failed
INFO: Elapsed time: 192.466s, Critical Path: 3.28s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

The version of ghost-kernel that I am using is 5.11.0+ and i am using Ubuntu 20.04.
Please someone can help me to solve this problem ?

The test results on shinjuku show obvious and stable jitter

Thank you very much for your previous help, I reproduced the test results on RocksDB with shinjuku policy (using the test script from the experiment), but I got strange test results. The test results of Tput and response latency on RocksDB show obvious and stable jitter, just as below:

RocksDB Throughput 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 110000 120000 130000 140000 141000 142000 143000 144000 145000 146000 147000 148000 149000 150000
Tput 9920 19866 29830 37844 49656 33345 45660 31592 42745 27373 40730 23812 38843 20431 34250 21688 36169 19768 33999 21298 34636 22482 35699 19553
min latency 18 18 19 12 12 1203863 872736 2048988 1500120 3354430 2293720 4455261 3010198 5650302 3552800 5336646 3714700 5875088 4004291 5786328 3926504 5344941 3786914 6122442
avg latency 39 45 48 366982 25199 3411579 2794724 4978419 4023087 6215053 5210133 7320167 5987645 8237605 6653896 8200332 6864416 8450028 6978296 8496233 6878794 8228504 6755587 8651108
99% latency 65 73 85 680388 171547 5690785 4476206 7803566 6741342 9326491 8091219 10384306 9042289 11040073 9780570 10953717 9608683 11139904 9882341 11044949 9877358 10966879 9783827 11247820

result

Further, I guess this jitter is related to the number of tests, so I repeat four tests under 90000 req/s( 'e.throughputs = [90000, 90000,90000, 90000]' in ghost-userspace/experiments/scripts/shinjuku.py), I got the below results and it seems to match my guess.

bazel run //experiments/scripts:shinjuku ghost
INFO: Analyzed target //experiments/scripts:shinjuku (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //experiments/scripts:shinjuku up-to-date:
bazel-bin/experiments/scripts/shinjuku
INFO: Elapsed time: 0.728s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Running ghOSt experiments...
mount: /dev/cgroup/cpu: cgroup already mounted on /sys/fs/cgroup/systemd.
mount: /dev/cgroup/memory: cgroup already mounted on /sys/fs/cgroup/systemd.
Output Directory: /tmp/ghost_data/2022-01-26 13:29:23
{"throughputs": [90000, 90000, 90000, 90000], "output_prefix": "/tmp/ghost_data/2022-01-26 13:29:23", "binaries": {"rocksdb": "/dev/shm/rocksdb", "antagonist": "/dev/shm/antagonist", "ghost": "/dev/shm/agent_shinjuku"}, "rocksdb": {"print_format": "pretty", "print_distribution": false, "print_ns": false, "print_get": true, "print_range": true, "rocksdb_db_path": "/dev/shm/orch_db", "throughput": 20000, "range_query_ratio": 0.005, "load_generator_cpu": 10, "cfs_dispatcher_cpu": 11, "num_workers": 200, "worker_cpus": [], "cfs_wait_type": "spin", "get_duration": "10us", "range_duration": "5000us", "get_exponential_mean": "0us", "batch": 1, "experiment_duration": "15s", "discard_duration": "2s", "scheduler": "ghost", "ghost_qos": 2}, "antagonist": null, "ghost": {"firstcpu": 11, "globalcpu": 11, "ncpus": 7, "preemption_time_slice": "30us"}}
Running experiment for throughput = 90000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
['/dev/shm/rocksdb', '--print_format', 'pretty', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.005', '--load_generator_cpu', '10', '--cfs_dispatcher_cpu', '11', '--num_workers', '200', '--worker_cpus', '', '--cfs_wait_type', 'spin', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '0us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'ghost', '--ghost_qos', '2', '--throughput', '90000']
SchedState: 11:I0/2701289 12:J0/2701284 13:K0/2701291 14:L0/2701260 15:M0/2701295 16:N0/2701294 17:none rq_l=194
SchedState: 11:none 12:O0/2701285 13:P0/2701286 14:Q0/2701292 15:L0/2701260 16:R0/2701287 17:M0/2701295 rq_l=194
SchedState: 11:S0/2701280 12:none 13:O0/2701285 14:P0/2701286 15:K0/2701291 16:M0/2701295 17:T0/2701296 rq_l=192
SchedState: 11:none 12:M0/2701295 13:J0/2701284 14:P0/2701286 15:S0/2701280 16:R0/2701287 17:N0/2701294 rq_l=194
SchedState: 11:T0/2701296 12:none 13:U0/2701293 14:O0/2701285 15:J0/2701284 16:K0/2701291 17:N0/2701294 rq_l=194
SchedState: 11:none 12:Q0/2701292 13:V0/2701290 14:K0/2701291 15:M0/2701295 16:L0/2701260 17:P0/2701286 rq_l=194
SchedState: 11:L0/2701260 12:none 13:Q0/2701292 14:T0/2701296 15:M0/2701295 16:S0/2701280 17:K0/2701291 rq_l=193
SchedState: 11:V0/2701290 12:L0/2701260 13:P0/2701286 14:none 15:O0/2701285 16:T0/2701296 17:M0/2701295 rq_l=193
SchedState: 11:none 12:S0/2701280 13:N0/2701294 14:W0/2701288 15:P0/2701286 16:L0/2701260 17:Q0/2701292 rq_l=194
SchedState: 11:S0/2701280 12:none 13:P0/2701286 14:L0/2701260 15:O0/2701285 16:Q0/2701292 17:V0/2701290 rq_l=193
SchedState: 11:none 12:O0/2701285 13:W0/2701288 14:L0/2701260 15:S0/2701280 16:T0/2701296 17:R0/2701287 rq_l=194
SchedState: 11:K0/2701291 12:P0/2701286 13:none 14:Q0/2701292 15:R0/2701287 16:V0/2701290 17:T0/2701296 rq_l=194
SchedState: 11:none 12:S0/2701280 13:Q0/2701292 14:M0/2701295 15:W0/2701288 16:O0/2701285 17:R0/2701287 rq_l=193
SchedState: 11:M0/2701295 12:none 13:T0/2701296 14:V0/2701290 15:N0/2701294 16:O0/2701285 17:K0/2701291 rq_l=193
SchedState: 11:none 12:L0/2701260 13:M0/2701295 14:V0/2701290 15:P0/2701286 16:K0/2701291 17:T0/2701296 rq_l=193
SchedState: 11:Q0/2701292 12:L0/2701260 13:none 14:S0/2701280 15:O0/2701285 16:V0/2701290 17:P0/2701286 rq_l=194
SchedState: 11:none 12:R0/2701287 13:T0/2701296 14:P0/2701286 15:O0/2701285 16:W0/2701288 17:Q0/2701292 rq_l=194
SchedState: 11:R0/2701287 12:none 13:O0/2701285 14:K0/2701291 15:Q0/2701292 16:W0/2701288 17:S0/2701280 rq_l=194
SchedState: 11:none 12:T0/2701296 13:U0/2701293 14:N0/2701294 15:R0/2701287 16:O0/2701285 17:V0/2701290 rq_l=193
SchedState: 11:U0/2701293 12:none 13:V0/2701290 14:K0/2701291 15:M0/2701295 16:N0/2701294 17:Q0/2701292 rq_l=193
SchedState: 11:none 12:W0/2701288 13:L0/2701260 14:U0/2701293 15:V0/2701290 16:R0/2701287 17:Q0/2701292 rq_l=194
SchedState: 11:R0/2701287 12:none 13:T0/2701296 14:Q0/2701292 15:N0/2701294 16:U0/2701293 17:V0/2701290 rq_l=193
SchedState: 11:V0/2701290 12:P0/2701286 13:none 14:S0/2701280 15:M0/2701295 16:L0/2701260 17:Q0/2701292 rq_l=193
SchedState: 11:none 12:M0/2701295 13:V0/2701290 14:W0/2701288 15:K0/2701291 16:O0/2701285 17:L0/2701260 rq_l=194
SchedState: 11:M0/2701295 12:none 13:Q0/2701292 14:S0/2701280 15:N0/2701294 16:V0/2701290 17:L0/2701260 rq_l=194
SchedState: 11:none 12:S0/2701280 13:U0/2701293 14:M0/2701295 15:N0/2701294 16:J0/2701284 17:W0/2701288 rq_l=193
SchedState: 11:L0/2701260 12:none 13:O0/2701285 14:V0/2701290 15:S0/2701280 16:R0/2701287 17:N0/2701294 rq_l=194
SchedState: 11:N0/2701294 12:W0/2701288 13:none 14:O0/2701285 15:S0/2701280 16:R0/2701287 17:J0/2701284 rq_l=194
RocksDB Stats:
Get:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 572656 44050 1465511 4204176 6542181 6580594 6609205 6611629
Repeatable Handle Time 572656 44050 0 0 10 10 11 42
Worker Queue Time 572656 44050 0 1 68 147 742 121093
Worker Handle Time 572656 44050 10 30 115 274 1353 1397035
Total 572656 44050 1465541 4204227 6542240 6580652 6609244 6726298
Range:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 2831 217 1465739 4085682 6550394 6580868 6604253 6610400
Repeatable Handle Time 2831 217 0 0 10 10 11 12
Worker Queue Time 2831 217 0 1 64 115 821 1747
Worker Handle Time 2831 217 28873 36197 156347 183477 1201632 5087580
Total 2831 217 1499557 4127825 6601265 6626808 6678678 8117214
All:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 575487 44268 1465511 4203104 6542215 6580606 6609179 6611629
Repeatable Handle Time 575487 44268 0 0 10 10 11 42
Worker Queue Time 575487 44268 0 1 68 146 744 121093
Worker Handle Time 575487 44268 10 31 269 13562 40787 5087580
Total 575487 44268 1465541 4203424 6542390 6581029 6609531 8117214
Running experiment for throughput = 90000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
['/dev/shm/rocksdb', '--print_format', 'pretty', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.005', '--load_generator_cpu', '10', '--cfs_dispatcher_cpu', '11', '--num_workers', '200', '--worker_cpus', '', '--cfs_wait_type', 'spin', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '0us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'ghost', '--ghost_qos', '2', '--throughput', '90000']
SchedState: 11:I0/2701830 12:none 13:J0/2701848 14:K0/2701840 15:L0/2701811 16:M0/2701843 17:N0/2701837 rq_l=194
SchedState: 11:none 12:O0/2701850 13:P0/2701839 14:M0/2701843 15:N0/2701837 16:Q0/2701833 17:R0/2701849 rq_l=193
SchedState: 11:S0/2701851 12:none 13:K0/2701840 14:Q0/2701833 15:O0/2701850 16:R0/2701849 17:L0/2701811 rq_l=194
SchedState: 11:none 12:S0/2701851 13:K0/2701840 14:M0/2701843 15:J0/2701848 16:T0/2701845 17:P0/2701839 rq_l=194
SchedState: 11:O0/2701850 12:none 13:I0/2701830 14:N0/2701837 15:S0/2701851 16:Q0/2701833 17:T0/2701845 rq_l=193
SchedState: 11:none 12:N0/2701837 13:U0/2701844 14:O0/2701850 15:Q0/2701833 16:J0/2701848 17:T0/2701845 rq_l=194
SchedState: 11:K0/2701840 12:none 13:V0/2701846 14:O0/2701850 15:Q0/2701833 16:I0/2701830 17:J0/2701848 rq_l=194
SchedState: 11:none 12:P0/2701839 13:V0/2701846 14:U0/2701844 15:K0/2701840 16:J0/2701848 17:Q0/2701833 rq_l=194
SchedState: 11:none 12:T0/2701845 13:S0/2701851 14:J0/2701848 15:U0/2701844 16:O0/2701850 17:W0/2701829 rq_l=194
SchedState: 11:Q0/2701833 12:none 13:U0/2701844 14:J0/2701848 15:V0/2701846 16:N0/2701837 17:O0/2701850 rq_l=194
SchedState: 11:none 12:W0/2701829 13:T0/2701845 14:K0/2701840 15:J0/2701848 16:U0/2701844 17:Q0/2701833 rq_l=194
SchedState: 11:P0/2701839 12:none 13:N0/2701837 14:J0/2701848 15:Q0/2701833 16:K0/2701840 17:U0/2701844 rq_l=194
SchedState: 11:none 12:N0/2701837 13:T0/2701845 14:J0/2701848 15:O0/2701850 16:P0/2701839 17:I0/2701830 rq_l=193
SchedState: 11:S0/2701851 12:N0/2701837 13:V0/2701846 14:none 15:T0/2701845 16:W0/2701829 17:U0/2701844 rq_l=194
SchedState: 11:none 12:S0/2701851 13:V0/2701846 14:T0/2701845 15:I0/2701830 16:R0/2701849 17:J0/2701848 rq_l=194
SchedState: 11:W0/2701829 12:none 13:O0/2701850 14:V0/2701846 15:R0/2701849 16:P0/2701839 17:U0/2701844 rq_l=194
SchedState: 11:none 12:J0/2701848 13:R0/2701849 14:T0/2701845 15:U0/2701844 16:O0/2701850 17:V0/2701846 rq_l=194
SchedState: 11:U0/2701844 12:none 13:I0/2701830 14:O0/2701850 15:Q0/2701833 16:R0/2701849 17:J0/2701848 rq_l=193
SchedState: 11:none 12:Q0/2701833 13:S0/2701851 14:I0/2701830 15:V0/2701846 16:J0/2701848 17:K0/2701840 rq_l=194
SchedState: 11:R0/2701849 12:none 13:T0/2701845 14:V0/2701846 15:S0/2701851 16:W0/2701829 17:O0/2701850 rq_l=194
SchedState: 11:none 12:W0/2701829 13:R0/2701849 14:N0/2701837 15:Q0/2701833 16:O0/2701850 17:T0/2701845 rq_l=194
SchedState: 11:none 12:V0/2701846 13:P0/2701839 14:S0/2701851 15:I0/2701830 16:W0/2701829 17:Q0/2701833 rq_l=194
SchedState: 11:N0/2701837 12:none 13:P0/2701839 14:K0/2701840 15:Q0/2701833 16:U0/2701844 17:O0/2701850 rq_l=194
SchedState: 11:W0/2701829 12:U0/2701844 13:none 14:T0/2701845 15:N0/2701837 16:K0/2701840 17:O0/2701850 rq_l=193
SchedState: 11:none 12:S0/2701851 13:P0/2701839 14:U0/2701844 15:J0/2701848 16:O0/2701850 17:I0/2701830 rq_l=194
SchedState: 11:P0/2701839 12:none 13:U0/2701844 14:W0/2701829 15:K0/2701840 16:I0/2701830 17:R0/2701849 rq_l=194
SchedState: 11:none 12:P0/2701839 13:S0/2701851 14:U0/2701844 15:K0/2701840 16:W0/2701829 17:V0/2701846 rq_l=194
SchedState: 11:Q0/2701833 12:none 13:N0/2701837 14:I0/2701830 15:U0/2701844 16:T0/2701845 17:K0/2701840 rq_l=194
RocksDB Stats:
Get:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 343054 26388 3243726 6277485 9107046 9141095 9173059 9173688
Repeatable Handle Time 343054 26388 0 0 10 10 11 19
Worker Queue Time 343054 26388 0 1 65 152 850 170321
Worker Handle Time 343054 26388 10 34 105 298 1863 311271
Total 343054 26388 3243780 6277531 9107184 9141132 9173101 9279417
Range:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 1794 137 3246198 6206710 9101283 9138715 9147969 9172264
Repeatable Handle Time 1794 137 0 0 10 11 11 12
Worker Queue Time 1794 137 0 1 60 85 761 6942
Worker Handle Time 1794 137 41065 51223 218551 365308 474447 560638
Total 1794 137 3302315 6278557 9159617 9188212 9202012 9217862
All:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 344848 26526 3243726 6277479 9106959 9141095 9173058 9173688
Repeatable Handle Time 344848 26526 0 0 10 10 11 19
Worker Queue Time 344848 26526 0 1 65 152 859 170321
Worker Handle Time 344848 26526 10 34 313 43870 59530 560638
Total 344848 26526 3243780 6277531 9110745 9141144 9173114 9279417
Running experiment for throughput = 90000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
['/dev/shm/rocksdb', '--print_format', 'pretty', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.005', '--load_generator_cpu', '10', '--cfs_dispatcher_cpu', '11', '--num_workers', '200', '--worker_cpus', '', '--cfs_wait_type', 'spin', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '0us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'ghost', '--ghost_qos', '2', '--throughput', '90000']
SchedState: 11:I0/2702424 12:none 13:J0/2702442 14:K0/2702440 15:L0/2702429 16:M0/2702437 17:N0/2702422 rq_l=194
SchedState: 11:none 12:O0/2702443 13:K0/2702440 14:L0/2702429 15:P0/2702435 16:Q0/2702419 17:R0/2702444 rq_l=194
SchedState: 11:L0/2702429 12:none 13:S0/2702439 14:K0/2702440 15:T0/2702421 16:I0/2702424 17:J0/2702442 rq_l=193
SchedState: 11:none 12:R0/2702444 13:N0/2702422 14:L0/2702429 15:Q0/2702419 16:K0/2702440 17:P0/2702435 rq_l=194
SchedState: 11:I0/2702424 12:none 13:T0/2702421 14:K0/2702440 15:O0/2702443 16:L0/2702429 17:P0/2702435 rq_l=194
SchedState: 11:none 12:K0/2702440 13:N0/2702422 14:M0/2702437 15:O0/2702443 16:R0/2702444 17:J0/2702442 rq_l=193
SchedState: 11:Q0/2702419 12:none 13:K0/2702440 14:T0/2702421 15:R0/2702444 16:P0/2702435 17:N0/2702422 rq_l=193
SchedState: 11:none 12:T0/2702421 13:Q0/2702419 14:L0/2702429 15:M0/2702437 16:U0/2702446 17:P0/2702435 rq_l=194
SchedState: 11:J0/2702442 12:none 13:P0/2702435 14:Q0/2702419 15:T0/2702421 16:I0/2702424 17:K0/2702440 rq_l=194
SchedState: 11:none 12:S0/2702439 13:N0/2702422 14:M0/2702437 15:R0/2702444 16:T0/2702421 17:K0/2702440 rq_l=194
SchedState: 11:none 12:none 13:L0/2702429 14:U0/2702446 15:N0/2702422 16:T0/2702421 17:M0/2702437 rq_l=195
SchedState: 11:M0/2702437 12:N0/2702422 13:L0/2702429 14:none 15:J0/2702442 16:P0/2702435 17:U0/2702446 rq_l=194
SchedState: 11:none 12:T0/2702421 13:P0/2702435 14:O0/2702443 15:R0/2702444 16:K0/2702440 17:J0/2702442 rq_l=193
SchedState: 11:M0/2702437 12:none 13:I0/2702424 14:Q0/2702419 15:T0/2702421 16:J0/2702442 17:P0/2702435 rq_l=193
SchedState: 11:none 12:O0/2702443 13:I0/2702424 14:U0/2702446 15:P0/2702435 16:R0/2702444 17:Q0/2702419 rq_l=193
SchedState: 11:O0/2702443 12:none 13:Q0/2702419 14:P0/2702435 15:T0/2702421 16:M0/2702437 17:J0/2702442 rq_l=194
SchedState: 11:none 12:S0/2702439 13:M0/2702437 14:L0/2702429 15:O0/2702443 16:P0/2702435 17:J0/2702442 rq_l=192
SchedState: 11:U0/2702446 12:none 13:L0/2702429 14:O0/2702443 15:T0/2702421 16:R0/2702444 17:Q0/2702419 rq_l=193
SchedState: 11:none 12:M0/2702437 13:S0/2702439 14:U0/2702446 15:J0/2702442 16:T0/2702421 17:I0/2702424 rq_l=193
SchedState: 11:K0/2702440 12:none 13:T0/2702421 14:L0/2702429 15:U0/2702446 16:P0/2702435 17:R0/2702444 rq_l=193
SchedState: 11:none 12:L0/2702429 13:O0/2702443 14:K0/2702440 15:I0/2702424 16:U0/2702446 17:M0/2702437 rq_l=194
SchedState: 11:U0/2702446 12:none 13:S0/2702439 14:M0/2702437 15:R0/2702444 16:P0/2702435 17:K0/2702440 rq_l=194
SchedState: 11:T0/2702421 12:M0/2702437 13:none 14:R0/2702444 15:U0/2702446 16:L0/2702429 17:J0/2702442 rq_l=194
SchedState: 11:M0/2702437 12:P0/2702435 13:J0/2702442 14:none 15:I0/2702424 16:T0/2702421 17:N0/2702422 rq_l=194
SchedState: 11:none 12:S0/2702439 13:I0/2702424 14:J0/2702442 15:P0/2702435 16:T0/2702421 17:V0/2702426 rq_l=194
SchedState: 11:O0/2702443 12:none 13:T0/2702421 14:U0/2702446 15:P0/2702435 16:N0/2702422 17:V0/2702426 rq_l=194
RocksDB Stats:
Get:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 563830 43371 1595792 4074412 6648182 6677346 6689267 6696523
Repeatable Handle Time 563830 43371 0 0 10 10 11 69
Worker Queue Time 563830 43371 0 1 66 140 787 99939
Worker Handle Time 563830 43371 10 22 132 320 1549 234801
Total 563830 43371 1595809 4074461 6648240 6677392 6689308 6697744
Range:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 2843 218 1595867 4120598 6648470 6677351 6690197 6696509
Repeatable Handle Time 2843 218 0 0 10 11 11 13
Worker Queue Time 2843 218 0 1 68 164 377 1955
Worker Handle Time 2843 218 27416 35634 170639 228286 411674 2951477
Total 2843 218 1631843 4161835 6694658 6721512 6777858 8264193
All:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 566673 43590 1595792 4074442 6648232 6677349 6689277 6696523
Repeatable Handle Time 566673 43590 0 0 10 10 11 69
Worker Queue Time 566673 43590 0 1 66 140 786 99939
Worker Handle Time 566673 43590 10 22 319 29051 40378 2951477
Total 566673 43590 1595809 4074661 6648736 6677411 6689404 8264193
Running experiment for throughput = 90000 req/s:
['/dev/shm/agent_shinjuku', '--firstcpu', '11', '--globalcpu', '11', '--ncpus', '7', '--preemption_time_slice', '30us']
['/dev/shm/rocksdb', '--print_format', 'pretty', '--noprint_distribution', '', '--noprint_ns', '', '--print_get', '', '--print_range', '', '--rocksdb_db_path', '/dev/shm/orch_db', '--throughput', '20000', '--range_query_ratio', '0.005', '--load_generator_cpu', '10', '--cfs_dispatcher_cpu', '11', '--num_workers', '200', '--worker_cpus', '', '--cfs_wait_type', 'spin', '--get_duration', '10us', '--range_duration', '5000us', '--get_exponential_mean', '0us', '--batch', '1', '--experiment_duration', '15s', '--discard_duration', '2s', '--scheduler', 'ghost', '--ghost_qos', '2', '--throughput', '90000']
SchedState: 11:none 12:I0/2702960 13:J0/2702958 14:K0/2702982 15:L0/2702961 16:M0/2702979 17:N0/2702971 rq_l=193
SchedState: 11:O0/2702941 12:none 13:P0/2702981 14:Q0/2702972 15:R0/2702976 16:S0/2702977 17:M0/2702979 rq_l=194
SchedState: 11:none 12:K0/2702982 13:J0/2702958 14:M0/2702979 15:P0/2702981 16:T0/2702967 17:R0/2702976 rq_l=194
SchedState: 11:S0/2702977 12:none 13:M0/2702979 14:N0/2702971 15:R0/2702976 16:Q0/2702972 17:U0/2702957 rq_l=194
SchedState: 11:none 12:Q0/2702972 13:O0/2702941 14:K0/2702982 15:I0/2702960 16:N0/2702971 17:M0/2702979 rq_l=193
SchedState: 11:I0/2702960 12:none 13:R0/2702976 14:N0/2702971 15:P0/2702981 16:K0/2702982 17:U0/2702957 rq_l=194
SchedState: 11:none 12:R0/2702976 13:J0/2702958 14:N0/2702971 15:I0/2702960 16:Q0/2702972 17:L0/2702961 rq_l=193
SchedState: 11:N0/2702971 12:R0/2702976 13:K0/2702982 14:none 15:T0/2702967 16:Q0/2702972 17:I0/2702960 rq_l=194
SchedState: 11:none 12:J0/2702958 13:S0/2702977 14:P0/2702981 15:Q0/2702972 16:M0/2702979 17:I0/2702960 rq_l=194
SchedState: 11:T0/2702967 12:none 13:J0/2702958 14:Q0/2702972 15:I0/2702960 16:K0/2702982 17:M0/2702979 rq_l=194
SchedState: 11:none 12:J0/2702958 13:Q0/2702972 14:L0/2702961 15:S0/2702977 16:I0/2702960 17:R0/2702976 rq_l=192
SchedState: 11:T0/2702967 12:none 13:O0/2702941 14:N0/2702971 15:L0/2702961 16:Q0/2702972 17:K0/2702982 rq_l=194
SchedState: 11:R0/2702976 12:I0/2702960 13:none 14:L0/2702961 15:U0/2702957 16:O0/2702941 17:S0/2702977 rq_l=193
SchedState: 11:none 12:U0/2702957 13:P0/2702981 14:R0/2702976 15:K0/2702982 16:L0/2702961 17:I0/2702960 rq_l=194
SchedState: 11:none 12:S0/2702977 13:R0/2702976 14:O0/2702941 15:Q0/2702972 16:T0/2702967 17:U0/2702957 rq_l=192
SchedState: 11:J0/2702958 12:none 13:M0/2702979 14:L0/2702961 15:T0/2702967 16:K0/2702982 17:P0/2702981 rq_l=194
SchedState: 11:none 12:I0/2702960 13:R0/2702976 14:T0/2702967 15:L0/2702961 16:N0/2702971 17:O0/2702941 rq_l=193
SchedState: 11:U0/2702957 12:none 13:N0/2702971 14:M0/2702979 15:R0/2702976 16:O0/2702941 17:I0/2702960 rq_l=194
SchedState: 11:M0/2702979 12:none 13:I0/2702960 14:J0/2702958 15:Q0/2702972 16:S0/2702977 17:T0/2702967 rq_l=194
SchedState: 11:none 12:R0/2702976 13:P0/2702981 14:Q0/2702972 15:O0/2702941 16:N0/2702971 17:T0/2702967 rq_l=194
SchedState: 11:Q0/2702972 12:none 13:I0/2702960 14:M0/2702979 15:K0/2702982 16:R0/2702976 17:P0/2702981 rq_l=193
SchedState: 11:none 12:I0/2702960 13:T0/2702967 14:Q0/2702972 15:O0/2702941 16:U0/2702957 17:J0/2702958 rq_l=194
SchedState: 11:none 12:N0/2702971 13:I0/2702960 14:R0/2702976 15:K0/2702982 16:L0/2702961 17:Q0/2702972 rq_l=194
SchedState: 11:M0/2702979 12:none 13:Q0/2702972 14:S0/2702977 15:R0/2702976 16:I0/2702960 17:T0/2702967 rq_l=193
RocksDB Stats:
Get:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 380337 29256 2765284 5802543 8678875 8733697 8743672 8750075
Repeatable Handle Time 380337 29256 0 0 10 10 11 20
Worker Queue Time 380337 29256 0 1 65 150 817 135405
Worker Handle Time 380337 29256 10 34 103 267 1562 230800
Total 380337 29256 2765302 5802627 8678909 8733722 8743707 8750253
Range:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 1885 144 2765459 5766490 8687513 8725394 8745680 8749604
Repeatable Handle Time 1885 144 0 0 10 11 11 11
Worker Queue Time 1885 144 0 1 63 83 672 997
Worker Handle Time 1885 144 39139 51529 214150 291318 454397 624325
Total 1885 144 2814735 5824114 8763182 8782982 8857589 8861919
All:
Stage Total Requests Throughput (req/s) Min (us) 50% (us) 99% (us) 99.5% (us) 99.9% (us) Max (us)
Ingress Queue Time 382222 29401 2765284 5802238 8678890 8733662 8743673 8750075
Repeatable Handle Time 382222 29401 0 0 10 10 11 20
Worker Queue Time 382222 29401 0 1 65 149 813 135405
Worker Handle Time 382222 29401 10 34 261 11885 59095 624325
Total 382222 29401 2765302 5802870 8679397 8733851 8743783 8861919

Could you please help to find the reason?
Thank you very much!

Scheduler is not working

Hi,

My question might be naive but I'm trying to execute some experiments with the provided schedulers (e.g. fifo scheduler, shinjuku scheduler). I've noticed that the agent's executes and initializes the cores successfully, however it appears to me that it does not schedule any process. I have placed a few printfs in the Enqueue method as well as the constructor. The printf of the constructor prints, once the scheduler is initialized, however the prints inside of the enqueue method, do not get called at all. No significant difference in performance observed as well.

I have a web server running on the SCHED_FIFO policy with 99 priority.

The provided tests also pass but the scheduler does not log anything except for core mapping and initialization prints.

I also debugged the kernel to see if the SCHED_GHOST policy of the kernel works. It seems the kernel schedules the agent perfectly, however, the agent does not schedule the real-time (or any) of my processes.

So simply my question is, how to make sure the agent's scheduler is working correctly when the enqueue method seems not to be called at all?

I appreciate your help.

Thanks

Error mounting ghostfs in agent initialization

I'm attempting to run the fifo_centralized_agent (or any agent) after I have gotten the target to build successfully. However, upon running bazel run fifo_centralized_agent, I encounter an error:

INFO: Analyzed target //:fifo_centralized_agent (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:fifo_centralized_agent up-to-date:
  bazel-bin/fifo_centralized_agent
INFO: Elapsed time: 15.846s, Critical Path: 12.69s
INFO: 3 processes: 1 internal, 2 processwrapper-sandbox.
INFO: Build completed successfully, 3 total actions
INFO: Build completed successfully, 3 total actions
lib/ghost.cc:150(19449) CHECK FAILED: (*__errno_location ()) == 16 [2 != 16]
errno: 2 [No such file or directory]
PID 19449 Backtrace:
[0] 0xaaaabd3f521c : ghost::Exit()
[1] 0xaaaabd3eace0 : ghost::Ghost::MountGhostfs()
[2] 0xaaaabd3ead3c : ghost::Ghost::GetSupportedVersions()
[3] 0xaaaabd392db8 : ghost::Ghost::CheckVersion()
[4] 0xaaaabd392674 : __static_initialization_and_destruction_0()
[5] 0xaaaabd3926a4 : _GLOBAL__sub_I_agent.cc
[6] 0xaaaabd4a4108 : __libc_csu_init
[7] 0xffff94507cf8 : __libc_start_main

The ENOENT error indicates a failure to mount ghostfs because the mount point "/sys/fs/ghost" is missing. My output from executing mount -t ghost ghost /sys/fs/ghost is mount: /sys/fs/ghost: mount point does not exist.

Do you have any idea what might be causing this error, or what is wrong with my installation of ghOSt? I should be running the ghOSt kernel version 62, but it is notable that the machine everything was built for and running on is arm64 (which has required some custom tweaks to build but not many) and it's also an embedded board. Thanks in advance for any help!

Compile issue: "invalid relo against 'nr_latches'"

Hi, colleagues
I'm trying to compile the ghost-userspace application, but come across the

libbpf: prog 'sched_ghost_latched': invalid relo against 'nr_latches' in special section 0xfff2; forgot to initialize global var?..
Error: failed to open BPF object file: Relocation failed

I compiled and installed the ghost kernel and commented the redefined variables in the agent.h according to the posts in #3.
My setup is:

$uname -a
Linux ubuntu 5.11.0+ #2 SMP Sat Jan 15 18:10:04 PST 2022 x86_64 x86_64 x86_64 GNU/Linux
$gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

Any idea will be appreciated!

Full debug message is as follows:

ghost@ubuntu:~/ghost-userspace$ bazel build -c opt ... --sandbox_debug
DEBUG: Rule 'linux' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "2993dcd082a5f14d912d7dd60eec7266c3e2560b", shallow_since = "1642200822 -0800" and dropping ["branch"]
DEBUG: Repository linux instantiated at:
  /home/ghost/ghost-userspace/WORKSPACE:9:19: in <toplevel>
Repository rule new_git_repository defined at:
  /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/external/bazel_tools/tools/build_defs/repo/git.bzl:186:37: in <toplevel>
INFO: Analyzed 62 targets (0 packages loaded, 0 targets configured).
INFO: Found 62 targets...
ERROR: /home/ghost/ghost-userspace/BUILD:455:13: Executing genrule //:schedghostidle_bpf_skel failed: (Exit 255): linux-sandbox failed: error executing command 
  (cd /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    TMPDIR=/tmp \
  /home/ghost/.cache/bazel/_bazel_ghost/install/64841bf12de13c7518c7ada0994bafe2/linux-sandbox -t 15 -w /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost -w /tmp -w /dev/shm -D -- /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/k8-opt/bin/external/linux/bpftool/bin/bpftool gen skeleton bazel-out/k8-opt/bin/third_party/bpf/schedghostidle_bpf.o > bazel-out/k8-opt/bin/bpf/user/schedghostidle_bpf.skel.h && sed -i '\''s/#include <bpf/#include <libbpf/g'\'' bazel-out/k8-opt/bin/bpf/user/schedghostidle_bpf.skel.h')
1642306174.978625595: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1642306174.978645467: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1642306174.978909323: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 43757
1642306174.978923332: src/main/tools/linux-sandbox-pid1.cc:447: Pid1Main started
1642306174.978995530: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1642306174.979195929: src/main/tools/linux-sandbox-pid1.cc:189: working dir: /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost
1642306174.979232495: src/main/tools/linux-sandbox-pid1.cc:221: writable: /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost
1642306174.979239022: src/main/tools/linux-sandbox-pid1.cc:221: writable: /tmp
1642306174.979252980: src/main/tools/linux-sandbox-pid1.cc:221: writable: /dev/shm
1642306174.979306772: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /
1642306174.979319554: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev
1642306174.979325888: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/pts
1642306174.979331796: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1642306174.979336313: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/hugepages
1642306174.979340338: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/mqueue
1642306174.979344049: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run
1642306174.979347580: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/lock
1642306174.979351286: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/vmblock-fuse
1642306174.979354975: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000
1642306174.979359065: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000/gvfs
1642306174.979363937: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000/doc
1642306174.979394596: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys
1642306174.979402324: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/security
1642306174.979411365: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup
1642306174.979417473: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/unified
1642306174.979424332: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/systemd
1642306174.979430311: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/rdma
1642306174.979436364: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/freezer
1642306174.979443342: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/net_cls,net_prio
1642306174.979449538: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/hugetlb
1642306174.979455775: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpuset
1642306174.979461877: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/perf_event
1642306174.979468629: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/memory
1642306174.979474932: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/blkio
1642306174.979504823: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpu,cpuacct
1642306174.979513175: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/pids
1642306174.979520327: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/devices
1642306174.979527392: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/pstore
1642306174.979533629: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/bpf
1642306174.979539387: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/debug
1642306174.979546065: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/tracing
1642306174.979552647: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/fuse/connections
1642306174.979559791: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/config
1642306174.979574151: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc
1642306174.979580263: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc/sys/fs/binfmt_misc
1642306174.979586460: src/main/tools/linux-sandbox-pid1.cc:311: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1642306174.979599086: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc/sys/fs/binfmt_misc
1642306174.979605623: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/bare/5
1642306174.979609466: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/1705
1642306174.979624779: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/2284
1642306174.979627969: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core20/1270
1642306174.979630692: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gnome-3-34-1804/24
1642306174.979634116: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gnome-3-38-2004/87
1642306174.979638038: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gnome-3-34-1804/77
1642306174.979695993: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gtk-common-themes/1519
1642306174.979704754: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gtk-common-themes/1506
1642306174.979709121: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snap-store/558
1642306174.979713496: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snap-store/433
1642306174.979717902: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /boot/efi
1642306174.979725429: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/14295
1642306174.979729834: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/7264
1642306174.979734562: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost
1642306174.979744910: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost
1642306174.979750062: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /tmp
1642306174.979754576: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1642306174.979801257: src/main/tools/linux-sandbox-pid1.cc:376: calling fork...
1642306174.979938087: src/main/tools/linux-sandbox-pid1.cc:406: child started with PID 2
libbpf: prog 'sched_ghost_latched': invalid relo against 'nr_latches' in special section 0xfff2; forgot to initialize global var?..
Error: failed to open BPF object file: Relocation failed
1642306174.983345378: src/main/tools/linux-sandbox-pid1.cc:423: wait returned pid=2, status=0xff00
1642306174.983354385: src/main/tools/linux-sandbox-pid1.cc:441: child exited normally with code 255
1642306174.984785169: src/main/tools/linux-sandbox.cc:233: child exited normally with code 255
INFO: Elapsed time: 0.408s, Critical Path: 0.01s
INFO: 8 processes: 8 internal.
FAILED: Build did NOT complete successfully

How to get scheduled threads information with shinjuku scheduler ?

Hello, Please excuse me i have a problem and it is not error during the scheduler execution but i need to have please more explanation about some event that i have observe.
I have reach to make threads migration during the execution of shinjuku scheduler by setting my threads in my userspace application on a PrioTable as @jackhumphries had tell me.
But when the threads has migrated, i just get information about ShinjukuSchedParams not some information about threads state an so on as you can see on this image:

image

I think that this is because the following instruction on shinjuku_scheduler.cc inside the DumpAllTask() method is never execute:

allocator()->ForEachTask([](Gtid gtid, const ShinjukuTask* task) {
absl::FPrintF(stderr, "%-12s%-12s%c\n", gtid.describe(),
ShinjukuTask::RunStateToString(task->run_state),
task->prio_boost ? 'P' : '-');
return true;
});

To reach to execute this method i have just change this instruction global_scheduler_->debug_runqueue_ = false; to global_scheduler_->debug_runqueue_ = true; as you can see in the following image:
image

Please someone can tell me what i am suppose to do before to execute that instruction for getting those information about threads state?

Half of the tests fail after failing to open /sys/fs/ghost/ctl

Hi,

When running # bazel test ... on my computers, half of the tests fail, for instance api_test or agent_test.

I made a script to help reproduce my exact kernel install process, so you can try that if you are not able to reproduce:

Here is what test.log looks like:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:agent_test
-----------------------------------------------------------------------------
Running main() from gmock_main.cc
[==========] Running 13 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 13 tests from AgentTest
[ RUN      ] AgentTest.DestructorCanFree
lib/enclave.cc:420(12) CHECK FAILED: ctl_fd_ >= 0 [-1 > 0]
errno: 30 [Read-only file system]
PID 139761 Backtrace:
[0] 0x7f58edc5a59f : ghost::Exit()
[1] 0x7f58edec3d7a : ghost::LocalEnclave::CreateAndAttachToEnclave()
[2] 0x7f58edec484d : ghost::LocalEnclave::LocalEnclave()
[3] 0x55d62a146d43 : ghost::FullAgent<>::FullAgent()
[4] 0x55d62a1319c2 : ghost::(anonymous namespace)::FullSimpleAgent<>::FullSimpleAgent()
[5] 0x55d62a12e18e : std::make_unique<>()
[6] 0x55d62a12b230 : ghost::AgentProcess<>::AgentProcess()
[7] 0x55d62a1240d9 : ghost::(anonymous namespace)::AgentTest_DestructorCanFree_Test::TestBody()
[8] 0x7f58ed8d72ca : testing::internal::HandleSehExceptionsInMethodIfSupported<>()
[9] 0x7f58ed8d1e09 : testing::internal::HandleExceptionsInMethodIfSupported<>()
[10] 0x7f58ed8b90f6 : testing::Test::Run()
[11] 0x7f58ed8b9acd : testing::TestInfo::Run()
[12] 0x7f58ed8ba389 : testing::TestSuite::Run()
[13] 0x7f58ed8c6ab5 : testing::internal::UnitTestImpl::RunAllTests()
[14] 0x7f58ed8d81ea : testing::internal::HandleSehExceptionsInMethodIfSupported<>()
[15] 0x7f58ed8d2fb1 : testing::internal::HandleExceptionsInMethodIfSupported<>()
[16] 0x7f58ed8c52a7 : testing::UnitTest::Run()
[17] 0x7f58ed921af8 : RUN_ALL_TESTS()
[18] 0x7f58ed921a7a : main
[19] 0x7f58ed235083 : __libc_start_main

The error seems to occur in LocalEnclave::MakeNextEnclave() at lib/enclave.cc:177, when opening /sys/fs/ghost/ctl.

I have no idea as to why the error is Read-only file system, given that the output of ls -l /sys/fs/ghost/ctl is:

-rw-rw---- 1 root root 0 juin  30 15:56 /sys/fs/ghost/ctl

(Note: I am running the tests with root privileges: sudo su then bazel test ...)

Seeing this, the problem might come from the kernel.

I hope you will be able to figure out what is wrong here.
These errors are not a fundamental problem to me as agents are able to run and schedule tasks without problems, but it would be nice to pass tests nonetheless.

Issues compiling

Hi,
I've compiled the kernel and replaced my previous kernel in my ubuntu, then when using the command bazel build -c opt ... , I got this error

                 from lib/enclave.cc:24:
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf.h:244:6: error: use of enum 'bpf_stats_type' without previous declaration
  244 | enum bpf_stats_type; /* defined in up-to-date linux/bpf.h */
      |      ^~~~~~~~~~~~~~
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf.h:245:38: error: use of enum 'bpf_stats_type' without previous declaration
  245 | LIBBPF_API int bpf_enable_stats(enum bpf_stats_type type);
      |                                      ^~~~~~~~~~~~~~
Target //:agent_test failed to build

This may mean we didn't declared the type of enum of C++, but in
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf.h:245:38
there declared the using of extern C

extern "C" {
#endif

And here are the environments:
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04)
bazel 4.2.1
Could you give my some suggestions?
Thank you

How to upgrade scheduling policies dynamically

For example, suppose that policy PA is more appropriate for workload W than PB. W is running, but the current policy is PB.
How to switch from PB to PA without blocking W?
The ghOSt paper mentions the principle, but I want to know the corresponding code and its usage. Thank you!

some question about switchTo and Latched

I am very sorry to disturb you, but I have something really want to know about ghOSt.
I have two questions about the following codes,which is written in schedules/fifo/per_cpu/fifo_schedules.cc :

if (payload->from_switchto) {
    Cpu cpu = topology()->cpu(payload->cpu);
    enclave()->GetAgent(cpu)->Ping();
  }
  • The first question is:what does “from_switchto” means?Under what circumstances will "from_swichto" not be 0?
  • The second question is:I think “enclave()->GetAgent(cpu)->Ping()” can be excuted means the agent is running.So why we do we need to ping the agent?

Besides,I have another two questions:

  • 1.Under what circumstances will we received a MSG_TASK_SWITCHTO?In other words, what does "MSG_TASK_SWITCHTO" mean?
  • 2.Under what circumstances will we received a MSG_TASK_LATCHED?In other words, what does "MSG_TASK_LATCHED" mean?

I’d be greatly grateful if you could answer my questions and I’m very looking forward to your reply!

BPF verifier rejects `biff.bpf.c`

Hi everyone,

I'm testing BIFF scheduler with the latest open-sourced ghost kernel (the kernel supports version 75, and the user space is also 75). It seems there is an issue regarding the following function with R0 unbounded memory access, make sure to bounds check any such access:

static struct biff_bpf_sw_data *gtid_to_swd(u64 gtid)
{
	struct task_sw_info *swi;
	struct __sw_arr *__swa;
	u32 zero = 0;
	u32 idx;

	swi = bpf_map_lookup_elem(&sw_lookup, &gtid);
	if (!swi)
		return NULL;
	idx = swi->index;
	if (idx >= BIFF_MAX_GTIDS)
		return NULL;
	__swa = bpf_map_lookup_elem(&sw_data, &zero);
	if (!__swa)
		return NULL;
	return &__swa->e[idx];
}

I also include the BPF verifiers error here:

libbpf: elf: skipping unrecognized data section(7) .rodata.str1.1
libbpf: Error in bpf_create_map_xattr(global_rq):Invalid argument(-22). Retrying without BTF.
libbpf: load bpf program failed: Permission denied
libbpf: -- BEGIN DUMP LOG ---
libbpf:
; switch (msg->type) {
0: (69) r2 = *(u16 *)(r1 +40)
; switch (msg->type) {
1: (65) if r2 s> 0x45 goto pc+7
 R1=ctx(id=0,off=0,imm=0) R2_w=invP(id=0,umax_value=69,var_off=(0x0; 0x7f)) R10=fp0
2: (65) if r2 s> 0x42 goto pc+13
 R1=ctx(id=0,off=0,imm=0) R2_w=invP(id=0,umax_value=66,var_off=(0x0; 0x7f)) R10=fp0
3: (15) if r2 == 0x40 goto pc+24
 R1=ctx(id=0,off=0,imm=0) R2_w=invP(id=0,umax_value=66,var_off=(0x0; 0x7f)) R10=fp0
4: (15) if r2 == 0x41 goto pc+25

from 4 to 30: R1=ctx(id=0,off=0,imm=0) R2_w=invP65 R10=fp0
; handle_blocked(msg);
30: (85) call pc+244
caller:
 R10=fp0
callee:
 frame1: R1=ctx(id=0,off=0,imm=0) R2_w=invP65 R10=fp0
; static void __attribute__((noinline)) handle_blocked(struct bpf_ghost_msg *msg)
275: (bf) r6 = r1
; u64 gtid = blocked->gtid;
276: (79) r1 = *(u64 *)(r6 +0)
277: (7b) *(u64 *)(r10 -8) = r1
278: (b4) w1 = 0
; u32 zero = 0;
279: (63) *(u32 *)(r10 -12) = r1
280: (bf) r2 = r10
;
281: (07) r2 += -8
; swi = bpf_map_lookup_elem(&sw_lookup, &gtid);
282: (18) r1 = 0xffff9b2e53183800
284: (85) call bpf_map_lookup_elem#1
; if (!swi)
285: (15) if r0 == 0x0 goto pc+9
 frame1: R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R6=ctx(id=0,off=0,imm=0) R10=fp0 fp-8=mmmmmmmm fp-16=0000????
; idx = swi->index;
286: (61) r8 = *(u32 *)(r0 +4)
 frame1: R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R6=ctx(id=0,off=0,imm=0) R10=fp0 fp-8=mmmmmmmm fp-16=0000????
; if (idx >= BIFF_MAX_GTIDS)
287: (25) if r8 > 0xffff goto pc+7
 frame1: R0=map_value(id=0,off=0,ks=8,vs=8,imm=0) R6=ctx(id=0,off=0,imm=0) R8_w=invP(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R10=fp0 fp-8=mmmmmmmm fp-16=0000????
288: (bf) r2 = r10
;
289: (07) r2 += -12
; __swa = bpf_map_lookup_elem(&sw_data, &zero);
290: (18) r1 = 0xffffb6cb8cdf1ef0
292: (85) call bpf_map_lookup_elem#1
293: (bf) r7 = r0
; if (!__swa)
294: (55) if r7 != 0x0 goto pc+1

from 294 to 296: frame1: R0=map_value(id=0,off=0,ks=4,vs=1572864,imm=0) R6=ctx(id=0,off=0,imm=0) R7=map_value(id=0,off=0,ks=4,vs=1572864,imm=0) R8=invP(id=0,umax_value=65535,var_off=(0x0; 0xffff)) R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
; swd->ran_until = bpf_ktime_get_us();
296: (27) r8 *= 24
297: (0f) r7 += r8
; return bpf_ktime_get_ns() / 1000;
298: (85) call bpf_ktime_get_ns#5
; return bpf_ktime_get_ns() / 1000;
299: (37) r0 /= 1000
; swd->ran_until = bpf_ktime_get_us();
300: (7b) *(u64 *)(r7 +8) = r0
 frame1: R0_w=invP(id=0) R6=ctx(id=0,off=0,imm=0) R7_w=map_value(id=0,off=0,ks=4,vs=1572864,umax_value=1572840,var_off=(0x0; 0x1ffff8),s32_max_value=2097144,u32_max_value=2097144) R8_w=invP(id=0,umax_value=1572840,var_off=(0x0; 0x1ffff8)) R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
; task_stopped(blocked->cpu);
301: (61) r6 = *(u32 *)(r6 +24)
302: (b4) w1 = 0
; u32 zero = 0;
303: (63) *(u32 *)(r10 -8) = r1
304: (bf) r2 = r10
;
305: (07) r2 += -8
; __ca = bpf_map_lookup_elem(&cpu_data, &zero);
306: (18) r1 = 0xffffb6cb8b619ef0
308: (85) call bpf_map_lookup_elem#1
; if (!__ca)
309: (15) if r0 == 0x0 goto pc+5
 frame1: R0=map_value(id=0,off=0,ks=4,vs=65536,imm=0) R6=invP(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R7=map_value(id=0,off=0,ks=4,vs=1572864,umax_value=1572840,var_off=(0x0; 0x1ffff8),s32_max_value=2097144,u32_max_value=2097144) R8=invP(id=0,umax_value=1572840,var_off=(0x0; 0x1ffff8)) R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
; if (cpu >= BIFF_MAX_CPUS)
310: (bf) r1 = r6
311: (67) r1 <<= 32
312: (77) r1 >>= 32
313: (b7) r2 = 1024
314: (2d) if r2 > r1 goto pc+1

from 314 to 316: frame1: R0=map_value(id=0,off=0,ks=4,vs=65536,imm=0) R1_w=invP(id=0,umax_value=1023,var_off=(0x0; 0x3ff)) R2_w=invP1024 R6=invP(id=9,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R7=map_value(id=0,off=0,ks=4,vs=1572864,umax_value=1572840,var_off=(0x0; 0x1ffff8),s32_max_value=2097144,u32_max_value=2097144) R8=invP(id=0,umax_value=1572840,var_off=(0x0; 0x1ffff8)) R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
; return &__ca->e[cpu];
316: (bc) w1 = w6
; pcpu->current = 0;
317: (67) r1 <<= 6
318: (0f) r0 += r1
319: (b7) r1 = 0
; pcpu->current = 0;
320: (7b) *(u64 *)(r0 +0) = r1
 frame1: R0_w=map_value(id=0,off=0,ks=4,vs=65536,umax_value=274877906880,var_off=(0x0; 0x3fffffffc0),s32_max_value=2147483584,u32_max_value=-64) R1_w=invP0 R2_w=invP1024 R6=invP(id=9,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R7=map_value(id=0,off=0,ks=4,vs=1572864,umax_value=1572840,var_off=(0x0; 0x1ffff8),s32_max_value=2097144,u32_max_value=2097144) R8=invP(id=0,umax_value=1572840,var_off=(0x0; 0x1ffff8)) R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
R0 unbounded memory access, make sure to bounds check any such access
processed 123 insns (limit 1000000) max_states_per_insn 0 total_states 9 peak_states 9 mark_read 2

Thanks,
Alireza

The origin of the biff_bpf.skel.h file

biff_bpf.skel.h is the header file in ghost-userspace/schedulers/biff/biff_scheduler.h
I would like to know how the biff_bpf.skel.h file is created and where can I find its source file?

The int GhostShmem::OpenGhostShmemFd(const char* suffix, pid_t pid) function still return value lower than 0, How to solve it ?

Hi,
Firstly I want to thanks @jackhumphries for his help. On my previous post I was trying to spawn threads on ghost kernel by modifying multithread application with the code get on experiments folder since when i running shinjuku scheduler and use external app (pushtosched.c script) to spawn threads on ghost kernel I still get Segfault at address 0x10 error. To solve this, @jackhumphries tell me to set all my threads on a PrioTable firstly and then set them as SCHED_ITEM_RUNNABLE. The first approach that I have used is to modify the multithread app by adding a code take on experiments folder. Now I want to spawn threads on ghost kernel by using any multithread application. The aims there is to able to write some pushtosched app which will be able to spawn threads of any multithreads app in ghost kernel without modify the code of that multithread app. To achieve this, I have firstly create a little project with bazel and I have try to externalize (Taking the following files on ghost-userspace and add them in my project) prio_table.cc, prio_table.h, shmem.cc, shmem.h and their dependencies such as base.cc, base.h, ghost_uapi.h, logging.h and Abseil library on my own project. In a main file (The file where I am suppose to spawn threads on ghost kernel) I have try to get pid of all threads of that multithread app given by the following command when I start the program: ls /proc/pid/task | ./launcher 18. the code is look like this:
while (fscanf(stdin, "%d\n", &pid) != EOF) {
threads_pid[numThreads] = pid;
numThreads++;
}

After this, I create a new PrioTable object by using the following code:
ghost::PrioTable table(numThreads, numThreads, ghost::PrioTable::StreamCapacity::kStreamCapacity83);
After I try to attach all threads on a PrioTable by using this code:
for(int j = 0; j <= numThreads; j++){
if(j == 0){
if(table.Attach(threads_pid[j])){
printf("\nThe threads have been attached on PrioTable\n");
}else{
printf("\n The threads cannot be attach on PrioTable \n");
}
}
}

Since the first pid on threads_pid array is the pid of a process during the execution Attach function cannot attach threads on a PrioTable. I have try to track a code and I have see that it is int GhostShmem::OpenGhostShmemFd(const char suffix, pid_t pid)* function in shmem.cc which return value lower than zero and I don't know why.
For my understand (Excuse me if I tell something wrong) I think that the value of p and the value of needle at this line
if (p.rfind(needle, 0) == 0) {
will never be the same since I have try to make some printf on that part of code and this is the way that I obtain:
image
Please can you help me to solve this issue? thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.