vvaltchev / tilck Goto Github PK

Builded the toolchain with a script. Changed it a bit to use pacman for my Artix Linux. The toolchain is builded and the package is installed. However, when the kernel compilation hits 53 percent, I run out of memory and Xorg killed.

Tilck

As it can be seen here: https://app.codecov.io/gh/vvaltchev/tilck/blob/master/kernel/fork.c, currently the following code path inside fork_dup_all_handles() is not covered by tests:

      if (rc < 0 || !dup_h) {

         enable_preemption();
         {
            for (u32 j = 0; j < i; j++)
               vfs_close(pi->handles[j]);
         }
         disable_preemption();
         return -ENOMEM;
      }

Now that we have gMock in Tilck, it would be relatively simple to just mock vfs_dup and vfs_close() so that inside a unit test we can trigger this code path. To do that, it would be necessary to make the fork_dup_all_handles() function STATIC instead of static, so that in unit tests, the function won't be static anymore. Also, it would be necessary add the mocked functions to a CMakeLists.txt file and to another file in the unit tests directory. For the exact instructions, see the comments in: https://github.com/vvaltchev/tilck/blob/master/tests/unit/mock_experiments.cpp

Clearly, because the fork_dup_all_handles() function takes a struct process * parameter, the unit test will have to allocate such a structure and set at least 2 handles in the pi->handles array.

The other/musl-gcc and other/musl-g++ template scripts are currently in the wrong place. The fix requires simply moving those two files to scripts/templates and replacing their paths in the other/cmake/gen_config_pre.cmake file:

Code snippet:

smart_config_file(
   ${CMAKE_SOURCE_DIR}/other/musl-gcc
   ${MUSL_GCC}
)

smart_config_file(
   ${CMAKE_SOURCE_DIR}/other/musl-g++
   ${MUSL_GXX}
)

Describe the bug
Unable to build_toolchain after unzipping project and installing dependencies like cmake, gcc, g++, clang...

Host configuration
Windows Subsystem for Linux
OS: Ubuntu 20.04 LTS on Windows 10 x86_64
Kernel: 5.10.16.3-microsoft-standard-WSL2
CPU: Intel i5-7200U (4) @ 2.711GHz
GPU: 2ce4:00:00.0 Microsoft Corporation Device 008e
Memory: 72MiB / 6271MiB

gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1 20.04.1)
g++ (Ubuntu 9.4.0-1ubuntu1 20.04.1) 9.4.0
clang version 10.0.0-4ubuntu1

Toolchain configuration
Run with no additional configuration or modifications.
cmake version 3.16.3
GNU Make 4.2.1

Tilck build configuration
2c5b642, Last Week, today's version (downloaded July 13 2023)

Type of issue (select one)

• Tilck build problem

Reproduction details
Downloaded project as *.zip and unzipped project.
Moved into directory, ran ./scripts/build_toolchain

root@LAPTOP-2GLCKP5T:/mnt/c/Users/fake pasta/Downloads/tilck-master/tilck-master# ./scripts/build_toolchain
./scripts/build_toolchain: line 36: cd: too many arguments

ARCH=i386
HOST_ARCH=x86_64
GCC_TC_VER=8.4.0
ARCH_GCC_TC_VER=stable-2020.02-2

./scripts/build_toolchain: line 287: /scripts/bash_includes/script_utils: No such file or directory
./scripts/build_toolchain: line 709: /scripts/tc/pkgs/*: No such file or directory
./scripts/build_toolchain: line 686: install_packages: command not found
root@LAPTOP-2GLCKP5T:/mnt/c/Users/fake pasta/Downloads/tilck-master/tilck-master#

Expected behavior
cmake begins building of dependencies for creating the disk image built using the other scripts.

Screenshots

root@LAPTOP-2GLCKP5T:/mnt/c/Users/fake pasta/Downloads/tilck-master/tilck-master# ./scripts/cmake_run
./scripts/cmake_run: line 10: cd: too many arguments
./scripts/cmake_run: line 12: /scripts/bash_includes/script_utils: No such file or directory
[cmake_run] Using '/mnt/c/Users/fake pasta/Downloads/tilck-master/tilck-master' as build directory
./scripts/cmake_run: line 105: generic_get_version_str: command not found
./scripts/cmake_run: line 106: fail_if_too_old: command not found
./scripts/cmake_run: line 110: generic_get_version_str: command not found
./scripts/cmake_run: line 111: fail_if_too_old: command not found
[cmake_run] Using 'cmake' as CMake executable
./scripts/cmake_run: line 134: cd: too many arguments
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/g++
-- Check for working CXX compiler: /usr/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:90 (message):
  Toolchain not found: run scripts/build_toolchain first.

-- Configuring incomplete, errors occurred!
See also "/mnt/c/Users/fake pasta/Downloads/tilck-master/tilck-master/CMakeFiles/CMakeOutput.log".
root@LAPTOP-2GLCKP5T:/mnt/c/Users/fake pasta/Downloads/tilck-master/tilck-master# ./scripts/build_toolchain
./scripts/build_toolchain: line 36: cd: too many arguments

ARCH=i386
HOST_ARCH=x86_64
GCC_TC_VER=8.4.0
ARCH_GCC_TC_VER=stable-2020.02-2

./scripts/build_toolchain: line 287: /scripts/bash_includes/script_utils: No such file or directory
./scripts/build_toolchain: line 709: /scripts/tc/pkgs/*: No such file or directory
./scripts/build_toolchain: line 686: install_packages: command not found
root@LAPTOP-2GLCKP5T:/mnt/c/Users/fake pasta/Downloads/tilck-master/tilck-master#

I read through that CMakeOutput.log and I wasn't able to find anything in it that helped figure out what the problem exactly is.

Additional context

I feel like this would be a better experience for people that are not well versed on c++(possibly first time contributors). For example, this slowed me down quite a bit. I know it's not really anyone's fault, but a warning would be nice. If you feel like it's not necessary, just close the issue.

Currently, https://uploader.codecov.io/latest/linux/codecov is downloaded twice: once by the command ./build/scripts/generate_kernel_coverage_report --codecov and once explicitly. It would be better to skip the 2nd download and just re-run the coverage tool from ./build/tmp/covecov.

@michelececcacci I guess this is for you to fix ;-)

Describe the bug
BusyBox build error prevents building the image.

Host configuration
Linux kernel: 5.15.80-1-lts
Arch: x86_64
Distro: Arch Linux

Reproduction details
Run:

 ./scripts/build_toolchain
make

Result:

ln: unable to access '/home/max/tilck/toolchain2/i386/busybox/busybox': No such file or directory

Although I didn’t notice any problems when I first started the script, and busybox even downloaded.
If you call:

 ./scripts/build_toolchain --help

You can see:

download_gcc_toolchain (*)          [ installed ]
download_cmake (*)                  [ skipped   ]
build_gnuefi (*)                    [ installed ][ i386 x86_64 ]
build_mtools (*)                    [ installed ][ host_x86_64 ]
build_zlib (*)                      [ installed ][ i386 ]
build_busybox (*)                   [ error     ]

Describe the bug
vladimir@Veriton:~/Project2/tilck$ ./scripts/build_toolchain -s build_tcc

ARCH=i386
HOST_ARCH=x86_64
GCC_TC_VER=8.4.0
ARCH_GCC_TC_VER=stable-2020.02-2

CI Detected: None [Assuming user machine]
[build_toolchain] Executing single function 'build_tcc'
NOTE: Skipping GCC TOOLCHAIN (i386 - musl)

*** TCC ***

+ local url=git://repo.or.cz/tinycc.git
+ local tag=15e9b7384e7091200aa33499b1ddcbf6e2f6a959
+ local tarname=tcc.tgz
+ local dirname=tcc
+ '[' -f /home/vladimir/Project2/tilck/toolchain2/cache/tcc.tgz ']'
+ '[' -z 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 ']'
+ git clone --branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 --depth 1 git://repo.or.cz/tinycc.git tcc
Cloning into 'tcc'...
warning: Could not find remote branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 to clone.
fatal: Remote branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 not found in upstream origin
+ tar cfz tcc.tgz tcc
tar: tcc: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
+ mv tcc.tgz /home/vladimir/Project2/tilck/toolchain2/cache
+ rm -rf tcc
+ set +x
Extracting tcc.tgz...
Extracting completed.
/home/vladimir/Project2/tilck/scripts/bash_includes/tc/tcc: line 40: cd: tcc: No such file or directory
sed: can't read Makefile: No such file or directory
Running command: ./configure --enable-cross --cross-prefix=i686-linux- --enable-static --cpu=i386 --prefix=/ --extra-ldflags='-Wl,--oformat=elf32-i386 -static' --crtprefix=/lib/i686-tilck-musl --libpaths=/lib/i686-tilck-musl &> configure.log
Running command: make cross-i386 &> build.log
i686-linux-strip: 'i386-tcc': No such file
[build_toolchain] Executing single function: DONE

Host configuration

• Linux distro and version
  Linux Veriton 5.13.0-41-generic 20.04.1-Ubuntu SMP Wed Apr 20 13:16:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

• Compiler (name and version)
  gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

• Tilck commit: please indicate the exact commit used to reproduce the problem
  b4ba5e7

Type of issue (select one)

• Toolchain build problem

P.S. tcc.tgz in cahe is empty.
P.P.S. It's desirable to have minimal error processing in scripts to exit early after the critical errors.
Best regards,
Vladimir

[ RUN ] exvfs.fseek
[ INFO ] random seed: 2385683618
Kernel assertion 'to_read == d->cluster_size' FAILED in /home/travis/build/vvaltchev/experimentOs/common/fat32_base.c at line 643
/home/travis/.travis/job_stages: line 57: 12802 Aborted (core dumped) MALLOC_CHECK_=2 ./build/gtests
The command "MALLOC_CHECK_=2 ./build/gtests" exited with 134.

Currently, exOS does not do any checking on the vaddrs that user applications pass to it through syscalls.
This clearly a problem since, an invalid vaddr can make the whole kernel to crash.

I think this is a great project. IMHO, I think in order for adoption, a distro with a package manager would be nice for people to use and play with, see what software works, what does not. "kick the tires". That way people don't have to start from scratch. I know you're primary aim is towards system programmers who can help you with the kernel but there are probably just as many non system programmers who want to see something like this project thrive. Just a thought.

As it can be seen here: https://app.codecov.io/gh/vvaltchev/tilck/blob/master/modules/tracing/ptype_buffer.c, both the functions don't have enough coverage.

It will be great to write unit tests for both the functions. Hints:

1. Both the functions are static and need to be made STATIC in order to be used from unit tests
2. Testing save_param_buffer() requires mocking copy_str_from_user() and copy_from_user(), in order to make them fail.
3. As an example of a similar issue, see: #120

Log:

[devshell] [RUN   ] getrusage
[devshell] ASSERT '++n < max' FAILED.
	On:	/__w/1/s/tests/system/test_misc.c:467
	ERRNO:	Invalid argument
[devshell] The command FAILED with 1
[devshell] [FAILED] getrusage (100 ms)

Full run log:
https://dev.azure.com/vkvaltchev/Tilck/_build/results?buildId=2007&view=logs&j=a6269aa3-3cc2-5e31-00e7-4d9e12bd7faf&t=ebbbc250-5186-538c-cb5d-d884e3da9992&l=1152

So, I was just trying out the kernel with busy box by following the guidelines provided in README file.
When I execute the command touch hello.txt from any directory I get the following error :

touch : hello.txt : Bad Address

In other words, it throws EFAULT error.
Maybe there is a bug in one of the system calls invoked by touch command.

Update:
After analyzing the source code, I could see that there is an issue with the parameters passed to sys_utimes system-call.
The second parameter u_times array which is of type timeval is NULL because of which copy_from_user is returning -1, so EFAULT is thrown.

Hi!
Oh man what an awesome project. Your blog posts were nice too :D
Quite a lot of questions spinning around in my head now, but mainly just wanted to drop in to say "hi, saw what you did there, i approve" :D

I do have a small bug to report though. I did some testing in UniPCemu and found out that bogoMips goes to zero for some reason
(EDIT: looks like it's caused by emulation slowness...__bogo_loops reaches value of 8, so that would do it).

This causes Tilck to freeze at kb8042 kernel module init, because the nop loop counter (ecx) will flip over to max value.
Forcing loops_per_us value to 1 enables Tilck to boot all the way to the shell prompt.

scripts/templates/qemu/run_qemu checks for presence of KVM in the system with which kvm:

I have KVM installed and the KVM module is loaded into the kernel:

$ lsmod | grep kvm
kvm_intel             335872  0
kvm                  1036288  1 kvm_intel
irqbypass              16384  1 kvm

But there is no kvm binary, so the check above fails.

The script also actually runs QEMU with -enable-kvm successfully and info kvm in the QEMU monitor outputs kvm support: enabled.

I understand that this is only a false warning, not a false error, but maybe there is another way of checking for absence of KVM that works on Arch Linux too?

Describe the bug
Kernel panic

Host configuration
Lenovo T440
BIOS:
UEFI/Legacy Boot [UEFI Only]
-CSM Support [No]

Toolchain configuration
Standard

Tilck build configuration
Commit number in attachment below

Type of issue (select one)

• Panic

Reproduction details
Boot, no actions, panic.

Screenshots

As it can be seen from https://app.codecov.io/gh/vvaltchev/tilck/blobs/master/kernel/misc.c, print_banner_line() and show_tilck_logo() don't have coverage, despite being always run on boot. That's simply because in coverage builds, the env variable TILCK_NO_LOGO is set. See https://github.com/vvaltchev/tilck/blob/master/other/coverage_build.yml.

Because those trivial code lines that don't really need automated testing (we always see their output on boot) and we don't care about their coverage, it would nice to just compile them out when TILCK_NO_LOGO is set. Doing that is pretty simple, considering that setting the env variable TILCK_NO_LOGO does nothing more than setting the default value of the CMake option KERNEL_SHOW_LOGO, which is already exported in <tilck_gen_headers/mod_console.h>.

Therefore, the fix would require just to use surround print_banner_line() and show_tilck_logo() with #if KERNEL_SHOW_LOGO and #endif, making sure there are no compile errors.

As it can be seen here: https://app.codecov.io/gh/vvaltchev/tilck/blob/master/kernel/auth.c, the trivial functions in kernel/auth.c don't have any test coverage. That's because they are so simple that it's obvious why they work BUT, at the same time, it would still be nice to have test coverage for them for two reasons:

1. It would increase Tilck's test coverage a little bit
2. It would be a great exercise for new contributors to learn how to implement a new system test

To test sys_getuid(), sys_setuid() and friends, it would be enough to add new funcs in tests/system/test_misc.c. Actually, even a single cmd_test_uids could be enough to test them all.

Notes and hints:

1. the new cmd function (shellcmd) must be registered in tests/system/cmds_table.h
2. because simply calling setuid() from C would just call the latest syscall version, in order to test older syscalls like sys_setgid16(), it would be necessary to use long syscall(long number, ...) with the right syscall number from kernel/arch/i386/arch_syscalls.c.

Steps I ran:

./scripts/build_toolchain --skip-install-pkgs
./scripts/cmake_run -DTERM_BIG_SCROLL_BUF=1 -DKERNEL_SHOW_LOGO=0
make -j12
I get this output:

/bin/sh: /root/tilck-master/toolchain2/host/gcc_8_4_0_i386_musl/bin/i686-linux-gcc: not found

Which is actually the command 'toolchain-wrapper' being run with the error message: ./toolchain-wrapper: cannot execute: required file not found

As you can see from codecov: https://app.codecov.io/gh/vvaltchev/tilck/blob/master/kernel/exit.c, the functions register_on_task_exit_cb() and unregister_on_task_exit_cb() don't have test coverage. It will be good to add a test for them. This can be done via a kernel self-test.

Is your feature request related to a problem? Please describe.
I cannot build this in a Docker container with no human interaction. It currently halts on apt install, waiting for me to accept installation of the dependencies.

Describe the solution you'd like
I would like a -y option or something like that to auto-accept / not require human interaction.

Describe alternatives you've considered
Rewriting the build script, which I might do eventually anyway. I ultimately want to make a Docker image whose build script has little to no branching execution, variables, etc., so that it is super readable what all is being done to build the kernel image.

[ INFO ] random seed: 2134373407
Buffers differ.
Last offset: 49084
Curr pos: 94598
read ret: 0
Linux buf: ffffffe0 15 ffffffa0 4c fffffffe 7f 00 00 ffffffbc ffffffaa 48 00 00 00 00 00 00 00 00 00 00 00 00 00 fffffff0 5b 12 02 00 00 00 00 00 00 00 00 00 00 00 00 ffffff8a 6e 49 00 00 00 00 00 ffffffc8 15 ffffffa0 4c fffffffe 7f 00 00 10 16 ffffffa0 4c fffffffe 7f 00 00
ExOS buf: 7c 00 00 00 00 00 00 00 40 6e 12 02 00 00 00 00 ffffffc0 ffffffb1 01 00 00 00 00 00 28 2c 12 02 00 00 00 00 ffffffbc ffffffaa 48 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffffc0 70 49 00 00 00 00 00 68 00 00 00 00 00 00 00 /home/travis/build/vvaltchev/experimentOs/unittests/exvfs_test.cpp:165: Failure
Failed
[ FAILED ] exvfs.fseek (34 ms)

When I had the first approach with the project I was searching for the contributing.md file without seeing the docs folder.

I think that move the file to the main folder would be better.

Another alternative could be to speicify the file in the README.md

Reproduction:

[parent] waiting the child to exit...
[kernel] Pid 12 will WAIT until pid 13 dies
*** DEBUG: attempt to write COW page at 0xbffff000 (addr: 0xbffffef0)
*** DEBUG: the page was not shared anymore. Making it writable.
Fork returned 0
############## I'm the child!
[thread signal]: under lock, signal_all!
[thread signal]: exit
****** [kernel thread] EXIT (pid: 8)
[validate stack] real stack page:       0xc01ff000
[validate stack] current->kernel_stack: 0xc441c000
************************ KERNEL PANIC ************************
Invalid kernel stack pointer.
File /home/travis/build/darthvader88/experimentOs/kernel/interrupts.c at line 67
Current process: 8 [KERNEL]
Interrupts: [ ]
*** KERNEL STACKTRACE ***
frame[0]: 0xc0102ee5
frame[1]: 0xc0105432
frame[2]: 0xc0103658
frame[3]: 0xc0100473
*** END STACKTRACE ***

As it can be seen here: https://app.codecov.io/gh/vvaltchev/tilck/blob/master/modules/tracing/tracing.c, the functions simple_wildcard_match() and set_traced_syscalls_int() have little test coverage, while it will be easy to write unit tests for them.

Just make the functions be STATIC instead of static, so that the functions will be visible in the unit tests binary. The goal here is to reach 100% line coverage for both the functions.

HINT: testing set_traced_syscalls_int() will require mocking handle_sys_trace_arg(), which is called by it.

I am getting an error when I try to build fbdoom on Ubuntu 20.04 x86_64.
Maybe I am missing some 16bit libraries, does anyone knows which ones I need to install ?

$ ./scripts/build_toolchain -s build_fbdoom

ARCH=i386
GCC_TC_VER=7.3.0
ARCH_GCC_TC_VER=stable-2018.11-1

CI Detected: None [Assuming user machine]
[build_toolchain] Executing single function 'build_fbdoom'
NOTE: Skipping FBDOOM
[build_toolchain] Executing single function: DONE

$ cat ./toolchain2/i386/fbDOOM/fbdoom/build.log
mkdir -p build
[Compiling i_video_fbdev.c]
[Compiling i_input_tty.c]
[Compiling i_main.c]
[Compiling dummy.c]
/tmp/tilck/scripts/build_scripts/i686-linux-gcc: 2: exec: -m32: not found
make: *** [Makefile:61: build/i_video_fbdev.o] Error 127
/tmp/tilck/scripts/build_scripts/i686-linux-gcc: 2: exec: -m32: not found
make: *** Waiting for unfinished jobs....
make: *** [Makefile:61: build/i_input_tty.o] Error 127
/tmp/tilck/scripts/build_scripts/i686-linux-gcc: 2: exec: -m32: not found
make: *** [Makefile:61: build/i_main.o] Error 127
/tmp/tilck/scripts/build_scripts/i686-linux-gcc: 2: exec: -m32: not found
make: *** [Makefile:61: build/dummy.o] Error 127

There are lightweight x server implementations like Xfree86 and Nano-x that could maybe be added as extra stuff that you have to configure with the build scripts. It would be impressive to see how well this would work and what applications would run. Although i think you would probably also need gtk and qt5 which would bloat it a bit. Just something to think about.

The file contains just debug-util functions to dump a FAT32 in-memory partition. It doesn't make sense to keep it in the kernel itself.
Note: its declaration in include/tilck/common/fat32_base.h has to be removed as well.

I included few different problems in one issue, because (may be) they relate to the same problem.

Describe the bug
ACPI related message appears only once.

Host configuration
Amilo UI 3520

Toolchain configuration
Standard

Tilck build configuration
See attachment below

Type of issue (select one)

• Incorrect behavior

Reproduction details

1. Boot with power cable is ON
   1.1. Switch power cable OFF after the boot completed
   1.2. The message:

ACPI: got GPE #29

1.3. Switch power cable ON/OFF/ON/etc. - no above message (not trapped or logged)
2. Boot with power cable is OFF
2.1. Switch power cable ON after the boot completed
2.2. The message:

ACPI: got GPE #29

2.3. Switch power cable OFF/ON/OFF/etc. - no above message (not trapped or logged)

Expected behavior
The message after each switch of the power cable.

Describe the bug
poweroff command is not completed

Host configuration
Amilo UI 3520

Toolchain configuration
Standard

Tilck build configuration
See attachment below

Type of issue (select one)

• Incorrect behavior

Reproduction details

1. Boot. Than issue: poweroff
   1.1. Executing, black screen, but power and WiFi indicators are light.
2. Boot. Than issue poweroff -f
   

Expected behavior
Black screen and no light indicators.

Describe the bug
ACPI related message with exact number appears only once.

Host configuration
Lenovo G410

Toolchain configuration
Standard

Tilck build configuration
See attachment above

Type of issue (select one)

• Incorrect behavior

Reproduction details

1. poweroff and poweroff -f work OK
2. After the boot:

ACPI: got GPE #2 (?)

ACPI: got GPE #27 (closing the screen cover)

Closing the screen cover again - no ACPI message

Expected behavior
The message after each closing the screen cover.

This is a follow up on our discussion on [0] from the other day. I was interested in the details and did some research and I wanted to share what I found, which I hope may be of interest to you as well.

I checked several answers in StackOverflow and Reddit and then I went forward straight to the source - the C99 standard [1] and I didn't find any clause explicitly saying that casting between pointer to data and pointer to function may yield undefined behavior. It was just not listed explicitly as allowed in the main normative text:

6.3.2.3 Pointers
1. A pointer to void may be converted to or from a pointer to any incomplete or object type. A pointer to any incomplete or object type may be converted to a pointer to void and back again; the result shall compare equal to the original pointer.

It is here that pointer to function types are not mentioned and I think this paragraph is what causes some people to consider this undefined behavior.

[...]
5. An integer may be converted to any pointer type (Petar: including pointer-to-function?). Except as previously specified, the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation.

I interpret this to mean that if you know the address of a function, the compiler should happily implicitly convert the integer literal to the desired pointer to function type:

// Create a pointer to the function at address 128:
int (*fp)() = 128;

And as expected, compiling this with gcc with -Wall -std=c99 yielded no warnings for me.

6. Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type.

Given 5. and 6., one could conclude that if a given compiler provides with an integer type large enough to hold a pointer value (i.e.. intptr_t exists), then converting a pointer-to-data to an integer and then converting that number to a pointer-to-function should be allowed.

8. A pointer to a function of one type may be converted to a pointer to a function of another type and back again; the result shall compare equal to the original pointer. If a converted pointer is used to call a function whose type is not compatible with the pointed-to type, the behavior is undefined.

This essentially is saying that you can store a pointer-to-function of one type in a pointer-to-function of a different type, as long as you don't try to call it through the wrong type.
(Initially I thought that this property (which I'm sure that you're well aware of) could save you from that cast, and that's why I'm mentioning it, but that's not the case. Anyways...)

Going a bit further, the Posix standard, which essentially operates on top of a subset of the C standard (I mean it requires things that in theory won't be true on all things that C targets), effectively requires that you to be able to convert between pointer-to-void and pointer-to-function in cases like dlsym [2]:

The ISO C standard does not require that pointers to functions can be cast back and forth to pointers to data. Indeed, the ISO C standard does not require that an object of type void * can hold a pointer to a function. Implementations supporting the XSI extension, however, do require that an object of type void * can hold a pointer to a function. The result of converting a pointer to a function into a pointer to another data type (except void *) is still undefined, however. Note that compilers conforming to the ISO C standard are required to generate a warning if a conversion from a void * pointer to a function pointer is attempted as in:

fptr = (int (*)(int))dlsym(handle, "my_function");

From [3]:

The standard defines two levels of conformance: POSIX conformance, which is a baseline set of interfaces required of a conforming system; and XSI Conformance, which additionally mandates a set of interfaces (the "XSI extension") which are only optional for POSIX conformance. XSI-conformant systems can be branded UNIX 03. (XSI conformance constitutes the Single UNIX Specification version 3 (SUSv3).)

And indeed, not only most C compilers allow it for typical (i.e. non-exotic and non-embedded targets), it is also mentioned in the C99 standard under Annex J (informative (Petar: note: not normative)) Portability issues, J.5 Common extensions:

J.5.7 Function pointer casts

1. A pointer to an object or to void may be cast to a pointer to a function, allowing data to be invoked as a function (6.5.4).
2. A pointer to a function may be cast to a pointer to an object or to void, allowing a function to be inspected or modified (for example, by a debugger) (6.5.4).

So, in summary, I believe that the reason that the C standard doesn't explicitly mandate this behavior has nothing to do with the strict aliasing rules, which were a late, poorly executed effort to catch-up in performance with Fortran, but with the fact that back then pure Harvard architectures [4] were something that C needed to support.

In some systems, there is much more instruction memory than data memory so instruction addresses are wider than data addresses.

[..]

Microcontrollers are characterized by having small amounts of program (flash memory) and data (SRAM) memory, and take advantage of the Harvard architecture to speed processing by concurrent instruction and data access. The separate storage means the program and data memories may feature different bit widths, for example using 16-bit-wide instructions and 8-bit-wide data. They also mean that instruction prefetch can be performed in parallel with other activities.

So in practice, what this means for Tilck is that as long as you only are targeting only modified Harvard architecture systems (main memory holds both instruction and data, but the CPU may have different internal data paths for instructions and data) and also keeping the ELF constructor section technique I think it's reasonable to say that you should be ok with a naive cast. Also I think that if you're targeting an architecture where indeed functions and data are in completely separate address spaces and use different bus widths, the current cast will be just as undefined behavior, as the naive one.

I don't have any preference about the code style, so you can leave it as it is, I wanted to share my findings, as I know you appreciate understanding things deeply like me ;)

Cheers,
Petar

Describe the bug
Unable to build on Ubuntu 22.10.

Host configuration

ARCH: i386
HOST_ARCH: x86_64
GCC_TC_VER: 8.4.0
ARCH_GCC_TC_VER: stable-2020.02-2

Toolchain configuration
I used only ./scripts/build_toolchain

Tilck build configuration
I used only make command only.
Commit: 2cb2bf7

Type of issue (select one)

• Toolchain build problem

Reproduction details

• Is the issue reproducible in a deterministic way? Yes

Expected behavior
tilck.img never builds.

Additional context
I attached my terminal output.
tilck_output.txt

Is your feature request related to a problem? Please describe.
I want to create a Tilck distro, and I need to be able to add files, such as shells and compilers, to the tilck.img produced from building. I have searched through the scripts for a while now, and it is not clear to me where files get copied in, and how to customize this.

Describe the solution you'd like
I think the user-friendly solution would be for there to be a "skeleton folder" that gets copied on top of the image's file system. Other than this, it might be nice to be able to just know / have some documentation on where the files get copied in the shell scripts so it can be easily modified.

Describe alternatives you've considered
I know when you build the Linux kernel, you just get a kernel, not a disk image. Would that be possible with this repository? If I just got a kernel executable, I could just configure Tilck as a GRUB menu entry (I think).

Additional context
N/A

Describe the bug
According to the comment in hwsleep.c:
...
/*
* We wanted to sleep > S3, but it didn't happen (by virtue of the
* fact that we are still executing!)
*
* Wait ten seconds, then try again. This is to get S4/S5 to work on
* all machines.
*
* We wait so long to allow chipsets that poll this reg very slowly
* to still read the right value. Ideally, this block would go
* away entirely.
*/
AcpiOsStall (10 * ACPI_USEC_PER_SEC);
...

It looks like we don't wait. I changed the code in hwsleep.c to check it:

...
    ACPI_DEBUG_PRINT ((ACPI_DB_INIT,
        "Entering sleep state [S%u]\n", SleepState));
    AcpiOsStall (10 * ACPI_USEC_PER_SEC); /* Temp changes */
    ACPI_DEBUG_PRINT ((ACPI_DB_INIT, /* Temp changes */
        "The second entering sleep state [S%u]\n", SleepState)); /* Temp changes */
    while (true) {}; /* Temp changes */

    /* Clear the SLP_EN and SLP_TYP fields */
...

And I see that we don't wait:

Host configuration
Linux Veriton 5.13.0-51-generic #58~20.04.1-Ubuntu SMP Tue Jun 14 11:29:12 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

Toolchain configuration
Standard.

Tilck build configuration

• Tilck commit: see screenshot above

Type of issue (select one)

• Incorrect behavior

Reproduction details
Reproducible on different hardware with code changes above.

Expected behavior
Wait requested time.

Screenshots
Screenshot is above.

Additional context
It looks like borrowed modules (for example, utosi.c, actypes.h, hwsleep.c, ... are out of date. Do you have any ideas/plans to synchronize these modules time to time?

Currently, the unit tests have to be run from project's main dir, because the fat32 tests use hard-coded relative paths to reach the build directory. It would be good instead to pass build directory's full path to the tests with a compile option.

As it can be seen here: https://app.codecov.io/gh/vvaltchev/tilck/blob/master/common/string_util.c, those functions do not have full coverage, while it will good to have that.

One of the technical problems here is that those functions have the same name as the standard libc functions (e.g. strcpy(), strcat() etc.). Therefore, they cannot be exported to unit tests directly. The solution to this problem is simple:

• add a tilck_ prefix to those functions
• under #if !defined(TESTING) && !defined(USERMODE_APP), create alias symbols using the GCC alias attribute.

Example:

int tilck_add(int a, int b)
{
	return a+b;
}

#if !defined(TESTING) && !defined(USERMODE_APP)
int add(int a, int b) __attribute__((alias("tilck_add"))); 
#endif

This way, it will be trivial to test those functions from C++, using they full name (tilck_strcmp, tilck_strcat, etc.). The goal is having 100% line coverage.

Steps to reproduce:

1. in tasklet_runner_kthread() use kernel_yield() instead of the kcond_wait()
2. enter quickly many commands. It likely a race condition.

************************ KERNEL PANIC ************************
[kfree] Heap not found for block: 0x00670000

Current process: 91 [USER]
Interrupts: [ 128 ]
eflags: 0x00000002 [ ], IOPL: 0
ss: 0x00000010, cs: 0xc7e50008, ds: 0xc7e50010, esp: 0xc7e58e08
eip: 0xc0105bfd, eax: 0xc0105bfd, ecx: 0x00000000, edx: 0xc7e10a78
ebx: 0x00000000, ebp: 0xc7e58e24, esi: 0x080482ad, edi: 0xbfffff24
Stacktrace (8 frames):
[0xc010143a] dump_stacktrace + 0x30
[0xc0105d13] panic + 0x143
[0xc01081cf] kfree2 + 0x200
[0xc01067d0] dfree_strarray + 0x2f
[0xc010a14b] sys_execve + 0x23e
[0xc010ae77] handle_syscall + 0xdb
[0xc0106db8] generic_interrupt_handler + 0x2ae
[0xc01010ce] asm_int_handler + 0x1b

Here https://github.com/vvaltchev/tilck/wiki/Micropython-on-Tilck scripts is written as script in both the code sections that contain it. I would have provided a PR myself, but apparently i can't submit PRs for the wiki so here we go.