amjad50 / emerald Goto Github PK

make a better safe API for using the virtual_space, we should return a valid pointer or a struct that will always point to a valid pointer. It will handle dropping and so on.

Also fix the issue where we store the mmio value in APIC as a raw pointer, and maybe replace it with this concrete type (probably unsafecell or something, but gotta be safe)

Use a better locking, like test loop, that only uses load and only try compare_exchange once it can actually do it. Reduce contention.

Also other improvements

Tasks

• reduce contention in locking
• Keep track of time holding the lock
• Keep track of time needed to wait for the lock
• Assert several metrics (go back to reddit for the suggestions ppl mentioned)
• (Need to check if its good) Keep track of owner CPU in the Lock itself and use Usize and not Bool
  • We won't do this
• Add more functionalities and structs around locks
  • Rwlock

A simple program to perform ICMP messages

Probably add github pages as well

• mouse driver
• mouse movement updates
• mouse rendering

Use the functionalities of keyboard driver, mouse driver, graphics to display video frames on the graphics buffer from usermode and control the playback, just pause/play, forward, backward is enough.

For the mouse, we can have a progress bar that we can control with mouse.

Change all occurances of amjad_os to emerald, add new logo and also maybe ASCII art on OS boot on the shell process would be cool

Not really sure what to do tests for exactly

Think of fields to test, for now could be on:

• Filesystem in general
• Fat
• virtual space
• virtual memory
• allocations
• scheduling
• other?

Try to do extreme tests that catch edge cases.

Save up the logs from Console Early and Late and then save them in the filesystem after/during runtime of the OS.

We can have it either:

• remove old logs on every new boot
• create a new log file/append on every boot

shared memory between processes, for now, not sure if we need to implement memory mapped files and paging. Maybe not, as its kinda different, we just want to share memory between processes as a form of IPC #13

Currently there is an issue with dealing with ACPI, framebuffer and other memory regions that are fixed in physical space.

We need to have some kind of "virtual address allocator", where we can get a segment of N pages and map them to our desired physical space.
This is only used in startup and such.

Currently for ACPI, we have a very dumb method to reserve space in kernel virtual space based on the first address we get, but that's not enough XD. as in some cases this address is very large (several MB) so we can't just get a memory around some place random.

https://github.com/Amjad50/OS/blob/0a28e58e9e3d19e6c126bbde23bd2378bf422596/kernel/src/acpi/tables.rs#L22-L93

Idea

• The idea I have is to have a virtual page allocator, similar to the physical page allocator, it will contain a range of free blocks, then we can request a number of pages, get a region, then we can probably free in the future as well.
• Since unlike the physical page allocator, we are dealing with addresses pages that must be following each other, we might get fragmentation easily. So will have to look into how to minimize it.

Do note: if its not clear already, this is not allocating any memory at all, this is just virtual space, so only ram is used is to store the metadata about free ranges and that's it.

This will help us be able to get the details of the ACPI implementation, like the interrupt lines used and the sleep modes supported and the values we should pass to ACPI functions to shutdown/sleep etc.

This is by no means needed at this point or even necessary to do, there is ACPICA and other libraries we can use instead of doing from scratch, but I got interested in it, so we are here.

Now we have keyboard, add mouse input, and a way for devices + usermode to read that.

For this roadmap, we can focus on keyboard input, and make it accessible to the shell somehow in a console device for example. But in the future it will be accessible from GUI applications

A software that can perform tcp connections

In order to build a shell, we must somehow be able to search for programs in the disk and execute them.

• build ls

For now, this should work for qemu, I'm not sure how general we can be, but lets focus on that, and test on other platforms later and improve/make new drivers when needed.

This issue is first considered as part of the roadmap for making a shell, so probably we don't need all types of IPCs and just something like a pipe will be enough.
In that case, the rest can be moved to another issue

• #83
• Pipes #23
• Messages sharing (not sure how yet)

need to do 2 things:

• use the current HPET (for calibration) along with low overhear CPU instructions to get clock speed, to track the time on the kernel.
• we can use RTC to get the start time
• implement userspace sleep with that.

In the idea dump: https://github.com/users/Amjad50/projects/3/views/1?filterQuery=-milestone%3A%22Rust+Hello+World%22%2C%22Userspace+graphics%3A+Video+player%22&pane=issue&itemId=47273788

I talked about async kernel, and other implementations.

The goal of this issue/ticket, is just to improve performance, by smartly waiting for IO and not spin looping (which is what is still done until now).

We can change the implementation of the kernel to be fully async (rust async), and use those features to implement waiting.
Or we implement them manually.

This is more of a demo of the last item of this milestone.

Create a video showing what this can do.

Currently there are some issues that doesn't allow us to boot into UEFI.

1. The way we take the memory layout is, take the second "Available" memory, and it must be at least 128 MB. UEFI and even BIOS doesn't guarentee that the second memory area is the main one. So we need to be more dynamic.

Tasks

• Implement dynamic physical page allocator from the multiboot info

We have changed the target in rust to allow sse features: Amjad50/rust@0664391.

We need to enable some CPU flags for it to work.

I noticed in UEFI and BIOS, the state of the CPU is different, namely the MSR and C0,4 registers.
for example, as of writing this issue, with UEFI, the kernel cannot write to protected user pages, but with BIOS it can. (tested in qemu only).
Which resulted in a crash in the ELF loader for UEFI, because we are writing to a read-only user pages.

Would be good to have a standard known from boot which features we have support for.

Add debugging and profiling capabilities, and this will include stack unwinding so that we can create a flamegrpah kind of stuff, among other stuff

This reduces unintentional bugs, for example the order of initialization during main

Not sure if this is useful or needed, maybe it would make operations easier in fs/mod.rs, but an idea is to have a structure specific for paths in the kernel.

Find a way to send keypresses to usermode without going to the console

Of course also, if we spawn the player from the shell we want to hijack and take control of the keyboard device data

We can added special growing for the stack alone (interact with the exception handler)
or add mmap with a mode that allows any memory to grow (upon reserved size).

Kernel will just panic (just as we do now?, but maybe have it better, include info about the reason and so on for example).

The important part here is userspace programs, currently userspace performing weird memory stuff will panic the kernel XD

Depend on some part of #11.

Think of how to manage the heap, is its from the kernel with something like brk? or make the userspace allocate its own pages?

If we choose the second option, need to make sure the location where the userspace program choose to use is safe to use and won't get in some memory in the middle.

Add metrics to track user space processes metrics, including:

• cpu time (user and kernel).
• memory usage.
• number of syscalls.
• etc?

I'm not sure if I should do POSIX complience on this stage, but at least we can get some API to allow reading/writing to disk, creating processes.

For now, the idea is to have this in the kernel, similar to linux, not sure if its the best idea, if there are better, we can implement them instead.

Kinda continuation after #36 .

Have the VGA buffer be a static buffer owned by the kernel, that the console can use (like what we do in keyboard)

Beside that, have it so that the userspace can take control over it somehow, and stop the console.

The output of this would be to display something from usermode on the vga buffer.

Currently we only use max 128MB memory, this memory is mapped from 0 into the KERNEL_BASE and that's good since we can have something like 1:1 mapping and that's how physical2virtual function works.

But for more memory, we need to have a special physical page allocator, that doesn't have the memory virtually mapped, and thus we can't store the free pages as a linked list connected to one another by we can store the metadata about free pages in the heap somewhere since we have that.

The setup can be done as so.

• initial physical page allocator setup
• virtual memory setup
• [implied] heap setup (for us heap just works after virtual memory is initialized)
• allocate rest of ram into high physical page allocator
• Can use any amount of memory from ram

Addition to #31 , ability to change file size. (also includes seeking to after the file end)

currently the physical page allocator only allocate pages, and because we free them start to finish they are stored in reverse order, so it may be hard to easily get contiguous size of how many we want.

Currently we don't need this, but I think this will become very important later on when dealing with dma for example and other hardware devices that require physical memory.

We should find a better method to organize physical memory.

I like how linux does it, it stores it as tree structure, where you have small amount of 4K pages, so when you want 4K pages and you don't have any, you allocate from the 8K pages above, and so on, each step is 2x the step before it, so you allocate 1 block and split it into 2 for the steps below.
Not sure with this approach how many steps we should take though. And how we would handle freeing memory, would it merge back up?

This is a revamp to how we do logging, I think we can have this built dynamically and add components to the console, where it will start for example with only uart, then add to it, vga logger and file logger as additional entities. (all are writable objects)

Currently only read operations are supported for the filesystem

Have a serial/uart as a device, and console depend on it. Would be like tty in linux.

Here are the list of syscalls or functionalities we will need to implement shell and loading other processes:

• execute/spawn (fork, not sure if I need to implement it, its an awkward api)
• mmap? or maybe brk? or something else for heap (diff name)
• exit (process exit)

There is more things to add, such as mprotect/VirtualProtect, Futex/Mutex related stuff, others, but this is just to get something going.

We have a very basic API to deal with PCI, we should improve it.

For example, one of the main issue in the design now, is that we don't handle 32/64 memory BARs correctly I think.

There is no need to allocate time specifically for this issue, its just being recorded so that we notice the issue when implementing the next PCI device.

Currently, we only have IDE device, and its quite basic as it uses IO BARs

current working directory for the process, and file commands will understand the relative path.
And ability to change this path

Not sure exactly when to put this?

should we implement this very early and have like two variants (Basic, Advanced) display.

• one will be initialized before anything else (to allow to display logs), similar to EarlyConsole
• the other will use heap and whatnot.

Have only one, and set it up as soon as we can (probably after virtual memory).
We can print the logs with this method and for old logs, we can just keep it in the EarlyConsole for example in a static sized array big enough (few KBs) and copy it in the display once its ready.

Not sure exactly about the startup, but that's not the most important part. This issue focuses on getting easy API to draw to the screen.

The output of this issue is to not change anything really :D visually, currently on bios mode, we get Text mode graphics, and that's what we use.

So after this, we should get this same/similar output in display, but it would work in uefi/bios etc...
This would make it easier for future work on gui.