Git Product home page Git Product logo

klange / toaruos Goto Github PK

View Code? Open in Web Editor NEW
5.9K 252.0 466.0 71.39 MB

A completely-from-scratch hobby operating system: bootloader, kernel, drivers, C library, and userspace including a composited graphical UI, dynamic linker, syntax-highlighting text editor, network stack, etc.

Home Page: https://toaruos.org/

License: University of Illinois/NCSA Open Source License

Makefile 0.44% Shell 0.26% Assembly 0.84% C 98.39% Python 0.07% Dockerfile 0.01%
operating-system toaruos operating-systems c c-library compositor window-manager baremetal os bootloader

toaruos's Introduction

ToaruOS

ToaruOS is a "complete" operating system for x86-64 PCs and experimental support for ARMv8.

While many independent, hobby, and research OSes aim to experiment with new designs, ToaruOS is intended as an educational resource, providing a representative microcosm of functionality found in major desktop operating systems.

The OS includes a kernel, bootloader, dynamic shared object linker, C standard library, its own composited windowing system, a dynamic bytecode-compiled programming language, advanced code editor, and dozens of other utilities and example applications.

There are no external runtime dependencies and all required source code, totalling roughly 100k lines of (primarily) C, is included in this repository, save for Kuroko, which lives separately.

Screenshot Demonstration of ToaruOS's UI and some applications.

History

I have been working on ToaruOS for over a decade now, and my goals have changed over the years.

When I first started the project in December 2010, my aim was to "learn by doing" - studying Unix-like systems by making one from scratch. I had been a contributor to Compiz, one of the first widely-used compositing window managers for X11, a few years prior, and somewhat naturally ToaruOS gained a GUI built on similar concepts early on.

For its original 1.0 release in 2015, ToaruOS was not the "completely from scratch" OS it has since become. Newlib provided the libc, and the GUI was built on Cairo, libpng, and Freetype. In the middle of 2018, I started a new project to replace these third-party components, which was eventually completed and merged to become ToaruOS 1.6.

Through out the project, ToaruOS has also attracted quite a few beginner OS developers who have tried to use it as a reference. ToaruOS's kernel, however, was a source of personal embarrassment for me, and in April 2021, after a long hiatus, I began work on a new one. The result was Misaka: a new 64-bit, SMP-enabled kernel. Misaka was merged in May and started the 1.99 series of beta releases leading up to ToaruOS 2.0.

Features

  • Dynamically linked userspace with shared libraries and dlopen.
  • Composited graphical UI with software acceleration and a late-2000s design inspiration.
  • VM integration for absolute mouse and automatic display sizing in VirtualBox and VMware Workstation.
  • Unix-like terminal interface including a feature-rich terminal emulator and several familiar utilities.
  • Optional third-party ports including GCC 10.3, Binutils, SDL1.2, Quake, and more.

Notable Components

  • Misaka (kernel), kernel/, a hybrid modular kernel, and the core of the operating system.
  • Yutani (window compositor), apps/compositor.c, manages window buffers, layout, and input routing.
  • Bim (text editor), apps/bim.c, is a Vim-inspired editor with syntax highlighting.
  • Terminal, apps/terminal.c, xterm-esque terminal emulator with 24-bit color support.
  • ld.so (dynamic linker/loader), linker/linker.c, loads dynamically-linked ELF binaries.
  • Esh (shell), apps/esh.c, supports pipes, redirections, variables, etc.
  • Kuroko (interpreter), kuroko/, a dynamic bytecode-compiled programming language.

Current Goals

The following projects are currently in progress:

  • Rewrite the network stack for greater throughput, stability, and server support.
  • Improve SMP performance with better scheduling and smarter userspace synchronization functions.
  • Support more hardware with new device drivers for AHCI, USB, virtio devices, etc.
  • Bring back ports from ToaruOS "Legacy", like muPDF and Mesa.
  • Improve POSIX coverage especially in regards to signals, synchronization primitives, as well as by providing more common utilities.
  • Continue to improve the C library which remains quite incomplete compared to Newlib and is a major source of issues with bringing back old ports.
  • Replace third-party development tools to get the OS to a state where it is self-hosting with just the addition of a C compiler.
  • Implement a C compiler toolchain in toarucc.

Building / Installation

Building With Docker

General users hoping to build ToaruOS from source are recommended to fork the repository on Github and make use of the Github CI pipeline.

For those looking to build locally on an appropriately configured Linux host with Docker, a build container is available. The ToaruOS repository should be used as a bind mount at /root/misaka and util/build-in-docker.sh can be run within this container to complete the compilation process:

git clone https://github.com/klange/toaruos
cd toaruos
git submodule update --init kuroko
docker pull toaruos/build-tools:1.99.x
docker run -v `pwd`:/root/misaka -w /root/misaka -e LANG=C.UTF-8 -t toaruos/build-tools:1.99.x util/build-in-docker.sh

After building like this, you can run the various utility targets (make run, etc.). Try make shell to run a ToaruOS shell using a serial port with QEMU.

Build Process Internals

The Makefile uses a Kuroko tool, auto-dep.krk, to generate additional Makefiles for the userspace applications and libraries, automatically resolving dependencies based on #include directives.

In an indeterminate order, the C library, kernel, userspace librares and applications are built, combined into a compressed archive for use as a ramdisk, and then packaged into an ISO9660 filesystem image.

Project Layout

  • apps - Userspace applications, all first-party.
  • base - Ramdisk root filesystem staging directory. Includes C headers in base/usr/include, as well as graphical resources for the compositor and window decorator.
  • boot - BIOS and EFI loader with interactive menus.
  • build - Auxiliary build scripts for platform ports.
  • kernel - The Misaka kernel.
  • kuroko - Submodule checkout of the Kuroko interpreter.
  • lib - Userspace libraries.
  • libc - C standard library implementation.
  • linker - Userspace dynamic linker/loader, implements shared library support.
  • modules - Loadable driver modules for the kernel.
  • util - Utility scripts, staging directory for the toolchain (binutils/gcc).
  • .make - Generated Makefiles.

Filesystem Layout

The root filesystem is set up as follows:

  • bin: First-party applications.
  • cdrom: Mount point for the CD, if available.
  • dev: Virtual device directory, generated by the kernel.
    • net: Network interface devices.
    • pex: Packet Exchange hub, lists accessible IPC services.
    • pts: PTY secondaries, endpoints for TTYs.
  • etc: Configuration files, startup scripts.
  • home: User directories.
  • lib: First-party libraries
    • kuroko: Kuroko modules.
  • mod: Loadable kernel modules.
  • proc: Virtual files that present kernel state.
    • 1, etc.: Virtual files with status information for individual processes.
  • src: Source files, see "Project Layout" section above.
  • tmp: Mounted as a read/write tmpfs normally.
  • usr: Userspace resources
    • bin: Third-party applications, normally empty until packages are installed.
    • include: Header files, including potentially ones from third-party packages.
    • lib: Third-party libraries. Should have libgcc_s.so by default.
    • share: Various resources.
      • bim: Syntax highlighting and themes for the text editor.
      • cursor: Mouse cursor sprites.
      • fonts: TrueType font files. Live CDs ship with Deja Vu Sans.
      • games: Dumping ground for game-related resource files, like Doom wads.
      • help: Documentation files for the Help Browser application.
      • icons: PNG icons, divided into further directories by size.
      • ttk: Spritesheet resources for the window decorator and widget library.
      • wallpapers: JPEG wallpapers.
  • var: Runtime files, including package manager manifest cache, PID files, some lock files, etc.

Running ToaruOS

VirtualBox and VMware Workstation

The best end-user experience with ToaruOS will be had in either of these virtual machines, as ToaruOS has support for their automatic display sizing and absolute mouse positioning.

Set up a new VM for an "other" 64-bit guest, supply it with at least 1GiB of RAM, attach the CD image, remove or ignore any hard disks, and select an Intel Gigabit NIC. Two or more CPUs are recommended, as well.

VirtualBox screenshot ToaruOS running in VirtualBox.

VMware screenshot ToaruOS running in VMware Workstation Player.

By default, the bootloader will pass a flag to the VirtualBox device driver to disable "Seamless" support as the implementation has a performance overhead. To enable Seamless mode, use the bootloader menu to check the "VirtualBox Seamless" option before booting. The menu also has options to disable automatic guest display sizing if you experience issues with this feature.

QEMU

Most development of ToaruOS happens in QEMU, as it provides the most flexibility in hardware and the best debugging experience. A recommended QEMU command line in an Ubuntu 20.04 host is:

qemu-system-x86_64 -enable-kvm -m 1G -device AC97 -cdrom image.iso -smp 2

Replace -enable-kvm with -accel hvm or -accel haxm as appropriate on host platforms without KVM, or remove it to try under QEMU's TCG software emulation.

Note that QEMU command line options are not stable and these flags may produce warnings in newer versions.

The option -M q35 will replace the PIIX chipset emulation with a newer one, which has the side effect of switching the IDE controller for a SATA one. This can result in faster boot times at the expense of ToaruOS not being able to read its own CD at runtime until I get around to finishing my AHCI driver.

Other

ToaruOS has been successfully tested on real hardware. If the native BIOS or EFI loaders fail to function, try booting with Grub. ToaruOS complies with the "Multiboot" and "Multiboot 2" specs so it may be loaded with either the multiboot or multiboot2 commands as follows:

multiboot2 /path/to/misaka-kernel root=/dev/ram0 migrate vid=auto start=live-session
module2 /path/to/ramdisk.igz
set gfxpayload=keep

Native photo ToaruOS running natively from a USB stick on a ThinkPad T410.

License

All first-party parts of ToaruOS are made available under the terms of the University of Illinois / NCSA License, which is a BSD-style permissive license. Unless otherwise specified, this is the original and only license for all files in this repository - just because a file does not have a copyright header does not mean it isn't under this license. ToaruOS is intended as an educational reference, and I encourage the use of my code, but please be sure you follow the requirements of the license. You may redistribute code under the NCSA license, as well as make modifications to the code and sublicense it under other terms (such as the GPL, or a proprietary license), but you must always include the copyright notice specified in the license as well as make the full text of the license (it's only a couple paragraphs) available to end-users.

While most of ToaruOS is written entirely by myself, be sure to include other authors where relevant, such as with Mike's audio subsystem or Dale's string functions.

Some components of ToaruOS, such as Kuroko or bim have different but compatible terms.

Community

Mirrors

ToaruOS is regularly mirrored to multiple Git hosting sites.

IRC

#toaruos on Libera (irc.libera.chat)

FAQs

Is ToaruOS self-hosting?

Individual applications and libraries can be built by installing the build-essential metapackage from the repository, which will pull in gcc and binutils. Sources are available in the /src directory on the live CD in a similar layout to this repository, and the auto-dep.krk utility script is also available.

For building ramdisks, finalized kernels, or CD images, some components are currently unavailable. In particular, the build script for ramdisks is still written in Python and depends on its tarfile module and zlib support. Previously, with a capable compiler toolchain, ToaruOS 1.x was able to build its own kernel, userspace, libraries, and bootloader, and turn these into a working ISO CD image through a Python script that performed a similar function to the Makefile.

ToaruOS is not currently capable of building most of its ports, due to a lack of a proper POSIX shell and Make implementation. These are eventual goals of the project.

Is ToaruOS a Linux distribution?

No, not at all. There is no code from Linux anywhere in ToaruOS, nor were Linux sources used as a reference material.

ToaruOS is a completely independent project, and all code in this repository - which is the entire codebase of the operating system, including its kernel, bootloaders, libraries, and applications - is original, written by myself and a handful of contributors over the course of ten years. The complete source history, going back to when ToaruOS was nothing more than a baremetal "hello world" can be tracked through this git repository.

When you say "complete"...

ToaruOS is complete in the sense that it covers the whole range of functionality for an OS: It is not "just a kernel" or "just a userspace".

ToaruOS is not complete in the sense of being "done".

Is ToaruOS POSIX-compliant?

While I aim to support POSIX interfaces well enough for software to be ported, strict implementation of the standard is not a major goal of the OS, and full compliance may even be undesirable.

Are contributions accepted?

ToaruOS is a personal project, not a community project. Contributions in the form of code should be discussed in advance. Ports and other work outside of the repo, however, are a great way to help out.

You can also help by contributing to Kuroko - which is part of why it's kept as a separate repository.

toaruos's People

Contributors

adamdicarlo avatar fabiensiron avatar farseer2 avatar fluxer avatar gerow avatar gil0mendes avatar graphitemaster avatar ivoah avatar jozefnagy1133 avatar klange avatar lioncash avatar martica avatar patricklucas avatar peter-harliman avatar shawnanastasio avatar srjek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

toaruos's Issues

Memory leak in syscall_chdir

Hi,
you have a memory leak here in the function syscall_chdir(). You do a kopen() get a fs_node_t but you dont free() it:

static int sys_chdir(char * newdir)
{
char * path = canonicalize_path(current_process->wd_name, newdir);
fs_node_t * chd = kopen(path, 0);
if (chd) {
if ((chd->flags & FS_DIRECTORY) == 0) {
return -1;
}
free(current_process->wd_name);
current_process->wd_name = malloc(strlen(path) + 1);
memcpy(current_process->wd_name, path, strlen(path) + 1);
return 0;
} else {
return -1;
}
}

Brgds
C.

1.0.0 Tracking Ticket

  • GCC port
  • Audio device drivers
  • Audio subsystem
  • ATAPI CD support (CD filesystem?)
  • Network stack
  • New wallpaper and character assets
  • Networking samples (telnet, etc.)
  • SDL backend + port
  • Stable ext2 writes
  • Dynamic loader
  • Finish Python port
  • Toolkit (Python?)
  • VFS permissions (basic)

Build a better sub-second timing API

Currently, the only way to get accurate sub-second timing information is through the uptime interface, which is disturbingly wrong (though slightly interesting from a plan9 perspective...)

There are several places where sub-second timing information would be useful:

  • Ensuring smooth animations in the compositor.
  • Support double clicks properly.
  • Video playback.

One of the POSIX timing APIs should be fully implemented with at least millisecond accuracy, but preferably microsecond accuracy (which is currently lacking). May require general upgrades to timekeeping.

Forgot -I flag

You forgot -I./kernel/include on line 156 of the Makefile.

Slow Pipes

Pipe support is extremely slow. Fix this.

x86-64 Port

ToaruOS has been x86 32-bit only for quite some time. A half-hearted attempt at an ARM port was made in 2013, but was not completed. Porting to x86-64 would provide an opportunity to discover and fix size issues, built out the general port infrastructure that Toaru's build tools have been lacking, and also provide an environment where all of the general-purpose drivers (mouse/keyboard/video) continue to function.

Things to think about:

  • Where are we making bad assumptions about pointer sizes?
  • What can we do to make the build scripts support multiple target architectures?
  • How can we make process structures more generic?
  • Where do we make assumptions about memory layout that may no longer be true as we move to higher-half with x86-64?

Installation / Setup Wizard

Now that we have working disk writes, we need a setup or installation tool.

It should be able to:

  • Run on first boot for production images.
  • Create a default user.
  • Setup timezone offsets (not currently supported anywhere).
  • Walk through the UI / controls.

In the future, we should build a full installation wizard that can be run from a bootable ISO to:

  • Format a disk.
  • Install files from a compressed archive.
  • Proceed with the setup above.

Fix const correctness for strings

There's plenty of const correctness issues throughout the kernel, but starting with static strong declarations would get us pretty far.

Mouse not working

Hi,
i booted up the system with qemu 2.0 yesterday and the mouse wont work anymore. Not sure if it is the qemu 2.x (also installed 2.1 to verify). But since you are only doing "ps2" port things, it should work. Will try to set up a qemu 1.x to verify if there is something "new" in qemu.

ATAPI Drivers

ATAPI drivers are required to read data from the CD drive.

ISO / CD Filesystem Support

The kernel lacks support for a filesystem read from directly from a CD.

Either, support an emulated Unix-like file system on the CD, or support enough to load a memdisk without having to use the bootloader.

Remove old boot logging

There are like 3 different methods for logging in the kernel, and some of them are never, ever used because they only print visible log messages to the serial line... before serial is enabled.

Get rid of the stored log – just make debug_print store to a temporary buffer until the VFS is available – and kill the boot log – it hasn't served a purpose since the kernel terminal was removed.

GUI dont running.

Hi, after make system, i create .img file with image-builder/create-image.sh after this process i type sudo qemu-system-i386 toaru-disk.img and qemu show me grub, i select Graphical Mode but nothing happened and restart system and i see grub again, what's wrong ?

Network Tracking Ticket

This is a task list ticket for tracking progress on the network stack.

  • Packet construction
    • Ethernet frames
    • IPv4 headers
    • UDP packets
      • UDP packet checksums
    • TCP packets
  • Core protocol support
    • ARP
    • DHCP
      • DHCP packet construction
      • DHCP packet interpretation (options)
    • DNS
      • Request generation
      • Response parsing
    • IRC
    • Telnet
  • Userspace interface

Support VirtualBox's BGE device in lfbvideo driver

It has a different PCI device ID, but is compatible with the eixsting driver:

18:15 <sortie>         bool is_qemu_bga = id.vendorid == 0x1234 && id.deviceid == 0x1111;
18:15 <sortie>         bool is_vbox_bga = id.vendorid == 0x80EE && id.deviceid == 0xBEEF;

I'm pretty sure the "Eldritch Horror" fallback means we already support these devices, but inefficiently (and possibly dangerously).

TCP: Reordering

Sometimes TCP packets come in in the wrong order. The stack needs to account for this.

Toolkit Todo List

  • Integrate with graphics library
  • Include support for multiple windows
  • Add objects for:
    • Menus
    • Labels
    • Sliders
    • Scrollbars
    • Checboxes / Radio Buttons
    • Input boxes
    • Tabviews?
  • Add support for graphical themes?
  • Need to integrate with font engine (freetype2 + shmemfonts)

GCC build error

While compiling GCC, after removing/updating any legacy build dependencies, I keep stumbling upon the error shown in the screenshot. I attempted to fix it many ways, but I keep on getting this error (marked in white).

error

Support Alt+F4 everywhere

Some applications lack a clear way to be closed, especially ones without titlebars. We should support Alt+F4 either at the compositor level or through something else (perhaps through the panel, like we do with some other window management functionality).

I cant build, help please.

I'm following this page to build kernel,

But when I execute ./.build-travis.sh, here is what happens:

bekir@ubuntu:~/klange/osdev$ sudo ./.build-travis.sh 
/home/build/osdev/toolchain ~/klange/osdev
~/klange/osdev
   CC   ./gui/ttk/lib/ttk-core.cIn file included from ./gui/ttk/lib/ttk-core.c:10:0:
./gui/ttk/ttk.h:7:19: fatal error: cairo.h: No such file or directory
compilation terminated.
   CC   ./gui/ttk/lib/ttk-core.c
/home/bekir/klange/osdev/userspace/gui/ttk/lib/ttk-core.o 1
['i686-pc-toaru-gcc', '-std=c99', '-U__STRICT_ANSI__', '-O3', '-m32', '-Wa,--32', '-g', '-I.', '-I/home/build/osdev/toolchain/local/i686-pc-toaru/include/freetype2', '-I/home/build/osdev/toolchain/local/i686-pc-toaru/include/cairo', '-I/home/build/osdev/toolchain/local/i686-pc-toaru/include/pixman-1', '-c', '-o', '/home/bekir/klange/osdev/userspace/gui/ttk/lib/ttk-core.o', './gui/ttk/lib/ttk-core.c']
Traceback (most recent call last):
  File "build.py", line 220, in <module>
    source_obj.run()
  File "build.py", line 108, in run
    raise Exception
Exception
make: *** [.userspace-check] Error 1
bekir@ubuntu:~/klange/osdev$ 

Message Passing

The kernel needs an efficient and safe message passing system for use in services.

newlib no longer builds

Host: Arch Linux x86_64, all packages up to date

newlib just stopped building recently, and I think it might be because it requires an old version of autoconf (currently have 2.69)

This is what comes out of the build script: https://gist.github.com/4593932

glogin improvements

There are several things that could use improvement in glogin:

  • Text input boxes need cursor when highlighted (preferably with blink)
  • Entire interface needs mouse support
  • The clock in the corner doesn't update (it's only a date, so this is pretty minor, but if you're logging in around midnight you might notice)
  • Yutani doesn't focus the login window when it is the only window (after logging out)
  • Add additional controls to login screen, such as "shut down"
  • Make design things configurable (transparency, layout, logo, etc.)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.