Git Product home page Git Product logo

willdurand / arvernos Goto Github PK

View Code? Open in Web Editor NEW
323.0 10.0 32.0 13.34 MB

:floppy_disk: A minimal, experimental and "toy" monolithic kernel to learn about OS development // Work In Progress

Home Page: https://williamdurand.fr/ArvernOS/

License: MIT License

Makefile 3.47% Assembly 5.29% C 90.17% Dockerfile 0.26% Python 0.66% Shell 0.16%
operating-system kernel x86-64 asm 64-bit aarch64 aarch32 32-bit hobby-os learning-by-doing

arvernos's Introduction

ArvernOS

CircleCI

ArvernOS (formerly known as "willOS") is a minimal and experimental monolithic kernel (not really an Operating System because it cannot do a lot of things currently).

Goals

The main goal of this project is to learn about operating systems, kernel development, different architectures and improve my C skills. ArvernOS is a monolithic (and hopefully modular) kernel with a unified homemade libc/libk, and userland programs.

The roadmap isn't clearly defined yet and it mainly depends on what I'd like to work on when I have time to spend on this project.

Architectures, boards and tiers

ArvernOS supports the architectures and boards listed below. Support for different architectures and boards are organized into two tiers, each with a different set of guarantees.

Tier 1

Tier 1 is the reference implementation and very likely the most advanced.

Current Tier 1 archs/boards:

  • x86_64
    • generic

Tier 2

Tier 2 is guaranteed to build but not all features have been implemented yet. Features are defined by the reference implementation and once feature parity is achieved, an architecture and/or board should move to Tier 1. Boards for which we cannot run enough tests in CI (e.g., because QEMU does not support the board) should stay in Tier 2, though.

Current Tier 2 archs/boards:

Hacking on ArvernOS

This section (and its sub-sections) are written for everyone interested in building and working on ArvernOS.

Setting up a development environment

The following dependencies are required to build this project:

  • llvm (version 17 currently)
  • make
  • qemu (version >= 5)

If you want to work on the x86_64 architecture, you'll need the following extra dependencies:

  • nasm
  • grub-mkrescue
  • xorriso

If you want to work on ARM architectures (aarch32 or aarch64), you'll need the following extra dependencies:

  • gcc-arm-none-eabi
  • u-boot-tools

Note: The recommended way to work on this project is to use Docker.

Getting the sources

This project contains git submodules. You have to clone the main project as well as the submodules, either by using this command:

$ git clone --recurse-submodules <url pointing to this repo>

or by using this command if you already have a copy of this git repository:

$ git submodule update --init

Docker (recommended way)

Use Docker with the provided Dockerfile. You can either use the willdurand/arvernos-toolchain image from DockerHub or build your own:

$ docker build -t willdurand/arvernos-toolchain .
[...]

You can then use it with docker run:

$ docker run -it --rm -v $(pwd):/app willdurand/arvernos-toolchain make help
ArvernOS - available commands for arch=x86_64

clean                          remove build artifacts
debug                          build the project in debug mode
docs                           build the docs
fmt                            automatically format the code with clang-format
gdb                            build, run the project in debug mode and enable GDB
help                           show this help message
libc                           build the libc
release                        build the project in release mode
run-debug                      run the project in debug mode
run-release                    run the project in release mode
run-test                       run the project in test mode
test                           run the unit tests
userland                       compile the userland programs (statically linked to libc)
version                        print tool versions

Note: The output of the make help command may contain different commands depending on the architecture and board configured.

MacOS

Install Homebrew, then run the following commands:

$ brew install nasm xorriso qemu llvm u-boot-tools

Linux

The tools/install-linux-deps script is used to install the dependencies. It is currently used by both the Dockerfile and Circle CI.

Building ArvernOS

You first need to install the development dependencies in order to build ArvernOS. The different final files are located in the build/<arch>/dist/ or build/<arch>/<board>/dist/ folder.

Debug mode

To build the image in debug mode, run:

$ make clean ; make debug

To compile the OS in debug mode, build the image, and start qemu with the OS loaded, run:

$ make clean ; make run-debug

Note: Some boards aren't supported in QEMU.

QEMU monitor

When running make run-debug, the QEMU monitor can be accessed over TCP on port 5555. You can use tools like telnet or nc:

$ telnet 127.0.0.1 5555
Logging

In debug mode, logging very likely uses the serial port COM1 to write various debugging information. QEMU is configured to write the output of this serial port to a logfile in ./log/. DEBUG level logs are not necessarily written by default, though, and it is possible to enable DEBUG logs for specific modules like this:

# Enable the debug logs for the "net" and "fs" modules
$ make clean ; make run-debug ENABLE_NET_DEBUG=1 ENABLE_FS_DEBUG=1

The available debug variables are:

  • ENABLE_CONFIG_DEBUG
  • ENABLE_CORE_DEBUG
  • ENABLE_FS_DEBUG
  • ENABLE_MMU_DEBUG
  • ENABLE_NET_DEBUG
  • ENABLE_PROC_DEBUG
  • ENABLE_SYS_DEBUG
  • ENABLE_USERLAND_DEBUG
Stack traces

Log files may contain stack traces without debug symbols if the symbols haven't been loaded:

[...]
DEBUG    | src/kernel/arch/x86_64/kshell/kshell.c:108:run_command(): command='selftest' argc=1
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:30:kernel_dump_stacktrace(): kernel stacktrace:
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace(): 00000000001163B3 - ???+0x0
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace(): 0000000000115941 - ???+0x0
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace(): 0000000000115BE1 - ???+0x0
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace(): 00000000001152BF - ???+0x0
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace(): 000000000010935B - ???+0x0

Use the tools/fix-stacktrace.py script to add missing symbol names to the output:

$ ./tools/fix-stacktrace.py build/x86_64/dist/symbols.txt log/x86_64-debug.log
[...]
DEBUG    | src/kernel/arch/x86_64/kshell/kshell.c:108:run_command(): command='selftest' argc=1
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:30:kernel_dump_stacktrace(): kernel stacktrace:
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace():   00000000001163B3 - selftest+0x63
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace():   0000000000115941 - run_command+0x271
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace():   0000000000115BE1 - kshell_run+0x181
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace():   00000000001152BF - kmain+0x107f
DEBUG    | src/kernel/arch/x86_64/kernel/panic.c:39:kernel_dump_stacktrace():   000000000010935B - long_mode_start+0x13

In addition, you might want to find the corresponding line of code in the source files by using llvm-addr2line or llvm-symbolizer:

$ llvm-symbolizer --obj=build/x86_64/dist/kernel-x86_64.bin 0x01163B3
print_selftest_header
/path/to/ArvernOS/src/kernel/kshell/selftest.c:12:3
selftest
/path/to/ArvernOS/src/kernel/kshell/selftest.c:30:3
$ llvm-addr2line -e build/x86_64/dist/kernel-x86_64.bin 01163B3
/path/to/ArvernOS/src/kernel/kshell/selftest.c:12
Debugging

Use make gdb to run the project in debug mode with QEMU configured to wait for gdb to connect. This has been tested with vim (:Termdebug) and VS Code.

A sensible configuration is automatically generated when this command is executed (see also: make gdbinit). If a file named .gdbinit.local exists in the project's root directory, its content will be appended to the generated .gdbinit file.

Note: make gdb calls make run-debug under the hood so all configuration options are also supported. For example, it is possible to run make gdb KERNEL_CMDLINE="kshell selftest".

Release mode

To compile the OS in release mode, build the image, and start qemu with the OS loaded, run:

$ make clean ; make run-release

config files

config files are used to configure how a build works. The content must be compatible with make. Here is an example:

# LLVM config on MacOS with Homebrew
LLVM_PREFIX = /usr/local/opt/llvm@13/bin/
LLVM_SUFFIX =

# Always enable the Undefined Behavior sanitizer
UBSAN = 1

# Logging
ENABLE_CORE_DEBUG     = 1
ENABLE_PROC_DEBUG     = 1
ENABLE_SYS_DEBUG      = 1
ENABLE_USERLAND_DEBUG = 1

License

ArvernOS is released under the MIT License. See the bundled LICENSE file for details. In addition, some parts of this project have their own licenses attached (either in the source files or in a LICENSE file next to them).

arvernos's People

Contributors

littlecodingfox avatar malbx avatar willdurand avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

arvernos's Issues

Add `unistd.h` header

Similar to #132, we could add a proxy header to ease the process of porting existing code that uses some of the usual unix/posix stuff.

`tar_find()` should not return an inode based on a partial filename

Steps to reproduce

  1. run make debug
  2. in willOS, type cat /bo and press enter

What happened?

The content of the file booo is displayed.

Expected?

It should indicate "file not found" because bo is not a file.

This happens because of an invalid strncmp() call, but fixing it breaks other test cases ๐Ÿ˜ž

Refactor screen driver

The current screen driver is a vga text mode driver. It should probably be renamed as such.

When doing this, we could reuse the existing screen "interface" to have an abstract api for putting stuff on a screen. That would allow us to write a new driver (VESA) in the future.

Create page_t struct

Similar to #84, let's have a page_t type:

typedef struct page {
  bool has_value;
  uint64_t addr;
} page_t;

Implement syscalls

In order to implement userland later, I need syscalls. While 32bits OS use interrupts, it seems 64bits use a sysenter instruction.

When remapping the kernel, some sections overlap

In the debug logs, we can see attempts to map already mapped pages. I didn't pay too much attention to that before but it is probably because we have contiguous sections and we calculate the same "start" and "end" pages.

Enter user mode

"userland" programs run in kernel mode currently (ring 0) but they should run in ring 3 (user mode) instead.

Consider the use of `clang-tidy` as a linter

I gave clang-tidy a try locally and that is interesting. It caught a few minor mistakes and gives good warnings. I see it as a linter/static analysis tool to prevent some mistakes.

We'll adopt clang-format to enforce stylistic conventions. clang-tidy is not about style but errors that can be caught with static analysis so having both would make sense.

That's not high priority, though.

`make fmt` is broken

/Git/willOS/./.clang-format: Invalid argument
YAML:23:38: error: invalid boolean
AllowShortIfStatementsOnASingleLine: Never

Repeats for each usage of the Never keyword.

Make frame allocator more dynamic

We currently use a constant for the available memory size, which allows us to initialize the bitmap and so on at build time.

This is nice because it works but we should probably use the multiboot info to determine the memory size at runtime. This would allow to change the RAM config without having to change a constant and recompile the kernel.

I think we could achieve that by creating a bitmap in the first frame we can allocate (in frame_init()). We should double check but 4kb for a bitmap should be enough in most cases.

Make VFS work once paging is enabled

After having remapped the kernel, the VFS does not work anymore and it's not clear why. Maybe we need to map the multiboot module first? (This seems to help a bit but I am not able to list any files).

Handle huge page

In the paging code, there is no huge page support. It seems to be required, so it should be done at some point. I need to read more about this stuff.

Crash when calling /bin/init

When I run an unmodified version of this repo, the init program will crash:

Screenshot 2020-02-18 at 14 51 47

Do you have any idea what the problem might be or how it could be solved?

Compilation warnings in debug mode

To Fix

src/kernel/console.c:13:23: warning: initializer element is not constant [-Wpedantic]
   13 |                       (vtansi_parser_t){ VTSTATE_ESC, { { 0, 0 } }, 0 },
      |                       ^
src/kernel/console.c:13:23: note: (near initialization for 'vtc.ansiparser')
In file included from src/kernel/console.c:4:
libs/vtconsole/vtconsole.h:16:3: warning: initializer element is not constant [-Wpedantic]
   16 |   (vtattr_t) { false, VTC_DEFAULT_FOREGROUND, VTC_DEFAULT_BACKGROUND }
      |   ^
src/kernel/console.c:14:29: note: in expansion of macro 'VTC_DEFAULT_ATTR'
   14 |                     .attr = VTC_DEFAULT_ATTR,
      |                             ^~~~~~~~~~~~~~~~
libs/vtconsole/vtconsole.h:16:3: note: (near initialization for 'vtc.attr')
   16 |   (vtattr_t) { false, VTC_DEFAULT_FOREGROUND, VTC_DEFAULT_BACKGROUND }
      |   ^
src/kernel/console.c:14:29: note: in expansion of macro 'VTC_DEFAULT_ATTR'
   14 |                     .attr = VTC_DEFAULT_ATTR,
      |                             ^~~~~~~~~~~~~~~~
src/kernel/console.c:16:31: warning: initializer element is not constant [-Wpedantic]
   16 |                     .cursor = (vtcursor_t){ 0, 0 },
      |                               ^
src/kernel/console.c:16:31: note: (near initialization for 'vtc.cursor')

Fixed

src/net/dns.c: In function 'dns_request':
src/net/dns.c:73:29: warning: missing braces around initializer [-Wmissing-braces]
   73 |   struct sockaddr_in addr = {
      |                             ^
......
   76 |     .sin_addr = inet_addr2(interface->dns_ip),
      |                 {
   77 |   };
      |   }
src/fs/debug.c: In function 'debug_write':
src/fs/debug.c:48:25: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith]
   48 |     printf("%s", &buffer[offset]);
      |                         ^
src/fs/debug.c:48:25: warning: dereferencing 'void *' pointer

Review the syscall implementation

We shouldn't be using random registers when syscall'ing, see: https://wiki.osdev.org/Calling_Conventions.

I think it'd be nice to fix that but I am pretty sure we'll need to extract the syscall handler to a separate "thing", i.e. without using the ISR code. Even if we use an interrupt for syscalls, ISR/IRQ handling is different (it seems).

Could willOS fit on a floppy?

Even today the floppies are still being used, for example - as virtual floppies inside the coreboot open source BIOS. Just imagine: your wonderful OS could be a part of someone's BIOS build! (for coreboot supported motherboard, maybe you have or could get one - see https://www.coreboot.org/Supported_Motherboards )

@willdurand , If you already have a coreboot-supported motherboard, or a real chance to get one, - wouldn't it be cool to be able to launch your own OS straight from the BIOS chip? ;) With one simple command its possible to add any floppy to coreboot BIOS build - and then you see it as a boot entry! Multiple floppies could be added this way (as long as you have enough space left inside the BIOS flash chip, luckily LZMA compression could be used for the stored floppies to reduce their occupied size)

Add `fcntl.h` header

The fcntl.h header is often used in existing programs/libs. The kernel does not provide it but given that we provide unix-like open, read, write, close functions, we might want to have it. We could only define the constants we want to support for now (O_*).

Move `fd.*` files to `src/proc` and rename them to `descriptor.*`

fd stands for file descriptor but we'll have to handle socket descriptors too. Let's use the more generic term descriptor to represent the different descriptors. The fd code currently lives under src/fs but descriptors should not be filesystem-specific. I guess we can create a proc sub-folder ("proc" for "process").

Unmapping the ELF memory causes a page fault

Something we need to look into after my ELF fix PR is merged.
I've tried several times to do an elf_unload method but it always causes a page fault when I unmap any of the pages.
Any idea why that could be happening?

Configure prefix for `liballoc`

We use liballoc to manage the kernel heap memory. In addition to handling a bunch of annoying stuff for us, it exposes the following well-known functions: malloc, free, calloc and realloc. We use them in the kernel code. We cannot use them in "userland" programs because the libc does not offer these functions yet.

We should configure a PREFIX for liballoc (something like k_) and use the k_* functions in the kernel code. That should make it clear that we are using kernel heap memory.

This is needed because this project uses pretty much the same code for the libk and libc.

Create frame_t struct

We should create the following type to differentiate frame numbers vs physical addresses. I suppose creating an optional-like type would make sense.

typedef struct frame {
  bool has_value;
  uint64_t addr;
} frame_t;

The frame numbers will continue to use the uint64_t type.

Add INFO logs

It'd be handy to have INFO logs for things that we should log all the time, even in release mode.

In addition to that, let's rename core/debug.h to logger.h.

It would be great to have one logger per module too and makefile variables for each one so that we can enable debug logs for modules we want.

Remap the kernel

According to this: http://os.phil-opp.com/remap-the-kernel.html, which is a very helpful resource, I have to remap the kernel to do something better. I have to investigate because I am not sure to understand everything right now.

This should bring a (mostly) safe kernel stack and a working page table module, which is the path to add a virtual memory allocator \o/

Memory management

I wrote many things to deal with memory, like frames, pages, etc. The frame allocator seems to work and translating a phy address seems to work too. I can get a page, a frame, etc. I think unmapping works too. I can handle page faults (more ore less).

The problem is I have no clue about what to do next. It seems like I need some sort of allocator to build a heap so that I can write kmalloc/kfree and then I am done, but I don't really know how to reuse what I have so far.

The multiboot info contains memory areas, so I should probably use that as well. I actually pass such info to the mmu but...

In the phil'os tutorial, the idea was to remap the kernel. I don't have that, so maybe I should?

I am thinking about restarting from scratch because obviously I cant remember what I did 2 years ago

Implement `init` unimplemented commands

The init program contains stub functions to implement. The end state will be to replace most of the kshell features with init and later move init to user mode (kshell will stay in kernel mode).

  • cal
  • clear
  • date
  • exit
  • help
  • hostname
  • reboot
  • uptime
  • overflow

Note: I suggest creating multiple PRs (like one per command) instead of a big patch that will be hard to review.

ELF loading/unloading leaks frames

STR:

  1. in kshell, type cat /proc/meminfo
  2. run a program like cal
  3. type cat /proc/meminfo

I am not sure where it leaks but I expect the number of frames in (1) and (3) to be the same, but it's not the case:

Screen Shot 2020-11-25 at 22 16 21

Implement sockfs

Instead of maintaining some socket specific information in a descriptor and a probably a buffer in the net/socket module, we could probably create a sockfs FS that would take care of this. That way, we could revert the changes made on the file descriptors, use inodes for sockets, and move the "net" logic from syscalls to sockfs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.