Git Product home page Git Product logo

mlibc's Introduction

mlibc is a C standard library

Continuous Integration

Official Discord server: https://discord.gg/7WB6Ur3

AUR package (provides mlibc-gcc): https://aur.archlinux.org/packages/mlibc

Design of the library

Directory Purpose
options/ (More or less) OS-independent headers and code.
options/ is divided into subdirectories that can be enabled or disabled by ports.
sysdeps/ OS-specific headers and code.
sysdeps/ is divded into per-port subdirectories. Exactly one of those subdirectories is enabled in each build.
abis/ OS-specific interface headers ("ABI headers"). Those contain the constants and structs of the OS interface. For example, the numerical values of SEEK_SET or O_CREAT live here, as well as structs like struct stat. ABI headers are only allowed to contain constants, structs and unions but no function declarations or logic.
abis/ is divided into per-OS subdirectories but this division is for organizational purposes only. Ports can still mix headers from different abis/ subdirectories.

Porting mlibc to a new OS

Ports to new OSes are welcome. To port mlibc to another OS, the following changes need to be made:

  1. Add new sysdeps/ subdirectory sysdeps/some-new-os/ and a meson.build to compile it. Integreate sysdeps/some-new-os/meson.build into the toplevel meson.build.
  2. Create ABI headers in abis/some-new-os/. Add symlinks in sysdeps/some-new-os/include/abi-bits to your ABI headers. Look at existing ports to figure out the ABI headers required for the options enabled by sysdeps/some-new-os/meson.build.
  3. In sysdeps/some-new-os/, add code to implement (a subset of) the functions from options/internal/include/mlibc/internal-sysdeps.hpp. Which subset you need depends on the options that sysdeps/some-new-os/meson.build enables.

We recommend that new ports do not build from master as we occasionally make internal changes that cause out-of-tree sysdeps to break. Instead we recommend you pin a specific release (or commit), or to upstream your changes to this repository so that we can build them on our CI and thus any breakages will be fixed by us in-tree.

Build Configuration

The following custom meson options are accepted, in addition to the built-in options. The options below are booleans which default to false (see meson_options.txt).

  • headers_only: Only install headers; don't build libc.so or ld.so.
  • mlibc_no_headers: Don't install headers; only build libc.so and ld.so.
  • build_tests: Build the test suite (see below).
  • disable_x_option: Disable x component of mlibc functionality. See meson_options.txt for a full list of possible values for x. This may be used to e.g disable POSIX and glibc extensions.
  • linux_kernel_headers: Allows for directing mlibc to installed linux headers. These can be obtained easily, placed in a directory and this option set to the corresponding path. This is required if the linux option is enabled, i.e. when the linux option is not disabled.

The type of library to be built (static, shared, or both) is controlled by meson's default_library option. Passing -Ddefault_library=static effectively disables the dynamic linker.

We also support building with -Db_sanitize=undefined to use UBSan inside mlibc. Note that this does not enable UBSan for external applications which link against libc.so, but it can be useful during development to detect internal bugs (e.g when adding new sysdeps).

Running Tests

The mlibc test suite can be run under a Linux host. To do this, first install a set of kernel headers (as described here), then run from the project root:

meson setup -Dbuild_tests=true -Dlinux_kernel_headers=/path/to/kernel/headers/include build

This will create a build directory. Then, cd build and run the tests (showing output) with:

meson test -v

mlibc's People

Contributors

48cf avatar 64 avatar alula avatar andy-python-programmer avatar arsenarsen avatar atiep avatar avdgrinten avatar brainstackoverflow avatar cleanbaja avatar dennisbonke avatar electrodeyt avatar fedorlap2006 avatar fido2020 avatar geertiebear avatar ikbenlike avatar kyota-exe avatar mathewnd avatar matt8898 avatar menotdan avatar mintsuki avatar moodyhunter avatar netbsduser avatar no92 avatar positrontheory avatar qookei avatar qwinci avatar raidtheweb avatar streaksu avatar thomtl avatar toor avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mlibc's Issues

posix: Implement recursive mutexes

Our pthread_mutex struct already has a member to store the level of nested locks but it is not used yet. Add a sys_futex_tid function to get a per-thread ID. When locking a mutex, store the sys_futex_tid in the mutex (instead of 0/1, as we currently do).

rtdl: linker does not enter directories

Under Linux, libraries in subdirs of /usr/lib, for example, man-db in /usr/lib/man-db, are found and used as requested. When running managarm, the libraries in these directories are not found, and need to be copied over manually to /usr/lib.

Tested with man-db's man command

Crash when executing a symlink

When executing a symlink (in my test case, /usr/bin/cc pointing to /usr/bin/gcc) the following is printed

In function operator(), file ../../../src/mlibc/options/posix/generic/posix_stdlib.cpp:200
__ensure(!S_ISLNK(st.st_mode) && "TO

The full ensure line is as follows

__ensure(!S_ISLNK(st.st_mode) && "TODO: Use readlink in realpath()");

options/internal: SIGSEGV in locking code

While testing git on managarm, the following bug came to my attention.
By doing the following operations in order, the program can be killed with an SIGSEGV
To reproduce:

  • Have git installed and a git repo available, in this test case managarm
  • Do the required git setup, for managarm, this is touch /dev/null, and setting the global username and email (git will complain about this)
  • In the git repo, touch a file, and run git add .
  • Run git commit -m <message>, where message is the commit message (irrelevant for triggering this bug) and watch it crash.

Running addr2line on libc.so gives the following line

dennis@DENNIS-MAIN-LAPTOP-SIDUCTION:~/managarm-clone/build$ addr2line --e system-root/usr/lib/libc.so 
0x987d0
/home/dennis/managarm-clone/build/pkg-builds/mlibc/../../../src/mlibc/options/internal/include/mlibc/lock.hpp:24

which translates to

if(__atomic_compare_exchange_n(&_futex, &expected, 1,
							false, __ATOMIC_ACQUIRE, __ATOMIC_ACQUIRE)) {

where the first part of the if statement is line 24 in lock.hpp.

Unknown printf() specifiers

While testing the coreutils programs the following utilities required an unimplemented printf() specifier ('76')

  • seq
  • dd

The following program required the unimplemented printf() specifier ('39'), which converts to an '

  • xz

Furthermore, it seems we are lacking an implementation of %f

Linux and BSD compatibility issues

mlibc's ABI differs from Linux and the BSDs in the following aspects:

  • File type constants returned by stat() have the same values on Linux and the BSDs but differ in mlibc.

renameat() is unimplemented

Coreutils mv program calls renameat() to perform the move operation, this function is unimplemented. Implementing it would allow mv and programs relying on it to function.

printf formatting issues

While testing the coreutils programs on managarm the stat program crashed with the following error:

../../../src/mlibc/subprojects/frigg/include/frg/formatting.hpp:379: Assertion '!opts.left_justify' failed!

This can be reproduced by stating any regular file.

Running the following command gives the same error, on a different location, this time at line 459
readelf -I <executable>

getaddrinfo() improvements

As discussed on discord there are numerous improvements to be made to our current implementation of getaddrinfo()

  • Handle DNS ipv6 records
  • Parse /etc/resolv.conf for the nameserver
  • Parse /etc/hosts and use it in name resolution
  • Parse /etc/services to implement the services part of the function
  • Handle CNAME records correctly
  • Handle hints properly
  • Handle all flags properly
  • Sort output list according to RFC 3484

getaddrinfo() unimplemented

With the recent addition of tcp to managarm, it has become clear that we are lacking in name resolution. One major example is getaddrinfo(), required by at least git from my initial testing. We should strive to implement this as soon as possible

ld.so: Integrate into static build

Currently, we do not use the dynamic linker in static builds. That is problematic for two reasons:

  • dlopen() will not work
  • ld.so is responsible for allocating thread-local storage (TLS)

Fix this by linking the dynamic linker into the static library. Use the (undocumented) __ehdr_start symbol to find the ELF EHDR of the executable. Read the PHDR offset from the EHDR and use that to find the executable's PHDRs. Then proceed to do dynamic linking as usual (naturally, there will be no relocations, so this will mostly be a no-op).

As an additional benefit, this change will unify the initialization paths (for .init_array etc.) in the static and dynamic cases.

Change sysdeps contract to return error instead of setting errno

This allows us to call sysdeps as we like, without having to worry about corrupting errno.

As of e70ccf0, the following files are affected:

  • options/rtdl/generic/linker.cpp
  • options/glibc/generic/sys-ioctl.cpp
  • options/posix/generic/dirent-stubs.cpp
  • options/posix/generic/termios-stubs.cpp
  • options/posix/generic/fcntl-stubs.cpp
  • options/posix/generic/sys-select-stubs.cpp
  • options/posix/generic/sys-mman-stubs.cpp
  • options/posix/generic/posix_stdlib.cpp
  • options/posix/generic/pthread-stubs.cpp
  • options/posix/generic/sys-time-stubs.cpp
  • options/posix/generic/sched-stubs.cpp
  • options/posix/generic/sys-stat-stubs.cpp
  • options/posix/generic/unistd-stubs.cpp
  • options/posix/generic/sys-wait-stubs.cpp
  • options/posix/generic/posix_signal.cpp
  • options/posix/generic/sys-socket-stubs.cpp
  • options/internal/include/mlibc/sysdeps.hpp
  • options/internal/generic/debug.cpp
  • options/internal/generic/frigg.cpp
  • options/internal/generic/allocator.cpp
  • options/internal/gcc/guard-abi.cpp
  • options/internal/gcc/initfini.cpp
  • options/ansi/generic/stdio-stubs.cpp
  • options/ansi/generic/signal-stubs.cpp
  • options/ansi/generic/stdlib-stubs.cpp
  • options/ansi/generic/time-stubs.cpp
  • options/ansi/generic/file-io.cpp
  • options/linux/generic/poll-stubs.cpp
  • options/linux/generic/pty-stubs.cpp
  • options/linux/generic/sys-signalfd.cpp
  • options/linux/generic/sys-inotify-stubs.cpp
  • options/linux/generic/sys-epoll.cpp
  • options/linux/generic/sys-timerfd.cpp
  • options/linux/generic/sys-mount.cpp
  • sysdeps/qword/generic/file.cpp
  • sysdeps/managarm/rtdl-generic/support.cpp
  • sysdeps/managarm/generic/memory.cpp
  • sysdeps/managarm/generic/signals.cpp
  • sysdeps/managarm/generic/file.cpp
  • sysdeps/managarm/generic/fork-exec.cpp
  • sysdeps/managarm/generic/socket.cpp
  • sysdeps/managarm/generic/time.cpp
  • sysdeps/managarm/generic/mount.cpp
  • sysdeps/managarm/generic/ensure.cpp

inet_pton() unimplemented

The following programs, tested on managarm, require inet_pton() to properly function

  • curl, running curl <ip> and curl <url> crashes due to the inet_pton() function being unimplemented

mkfifo() unimplemented

While testing the coreutils programs the following utilities required mkfifo()

  • mkfifo

sync() unimplemented

While testing the coreutils programs the following utilities required sync()

  • sync

scandir() unimplemented

While testing on qword, dpkg failed to initialize when reading it's configuration directory.

link() unimplemented

While testing the coreutils programs the following utilities required link()

  • link

Git problems

While testing git on managarm, the following unimplemented functions are observed

  • pthread_setcancelstate(), required for most of the operations, including git commit and git log
  • rmdir(), required for git checkout, possibly only when directories need to be removed (see issue #50).
  • utime(), required for git add, this is implemented for managarm, but calls it with unsupported modes (meaning one of the fields is not UTIME_NOW).
  • with git add working(ish), git commit now throws an SIGSEGV in an internal mlibc header, before triggering the pthread_setcancelstate() function. This problem is tracked in a seperate issue, issue #111
  • fetching repo's with either git clone, git pull or git fetch (assuming git:// repo's) require handling of the /etc/services file to work (part of issue #99)
  • fetching, cloning or pulling repo's (http(s):// repo's) require curl (tracked in managarm/bootstrap-managarm#33)

This is a tracking issue for all git related problems in mlibc

textdomain() unimplemented

The following programs require textdomain() to be implemented:

  • gapplication from the package glib
  • gdbus from the package glib
  • gresource from the package glib
  • gsettings from the package glib

nohup fails to run

On managarm, when running any command with nohup, for example, nohup true, the following line is returned

(null): failed to render standard input unusable: No such file or directory (ENOENT)

This can be reproduced by running nohup true, although nohup <command> should work as well.

regcomp() unimplemented

The following programs, tested on managarm, require regcomp() to properly function

  • file, file works for executables, but for other files, regcomp() is called, thus severely limiting it's use.
  • flex, all programs from flex want regcomp(). Without it, even flex --help fails

freopen() unimplemented

While testing the coreutils programs the following utilities required freopen()

  • shuf
  • tsort
  • uniq

Add memory debugging options

Add an option to write some non-zero byte pattern to malloc()ed memory and/or memory obtained from sys_anon_allocate to simplify memory testing.

mknod() unimplemented

While testing managarm, the following program from coreutils crashed on the mknod() function being unimplemented

  • mknod

fdatasync() unimplemented

While testing the coreutils programs the following utilities required fdatasync()

  • shred

The following programs also requires fdatasync()

  • update-mime-info from the package shared-mime-info
  • xbps-pkgdb from the package xbps

gethostbyname() unimplemented

With the recent addition of tcp to managarm, it has become clear that we are lacking in name resolution. One major example is gethostbyname(), required by at least lynx and syslogd from my initial testing. We should strive to implement this as soon as possible

getentropy() unimplemented

During testing on managarm, the following program required getentropy()

  • wget, when fetching https:// links

setmntent() unimplemented

The following programs want setmntent() to function correctly:

  • df from the package coreutils
  • updatedb from the package findutils

program_invocation_name is NULL

A lot of programs get their own name by looking at program_invocation_name or program_invocation_short_name. Right now, those are NULL. Setting these correctly would fix this.

/usr/bin/mknod crashes

When running /usr/bin/mknod <filename> c 1 3 on managarm, which should be a valid command according to LFS, mknod crashes with the following error

In function strtoxmax, file ../../../src/mlibc/options/ansi/generic/inttypes-stubs.cpp:25 __ensure(base) failed

This can be reliably be reproduced by running the command specified above

rmdir() unimplemented

While testing the coreutils programs the following utilities required rmdir()

  • rmdir

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.