reswitched / libtransistor Goto Github PK
View Code? Open in Web Editor NEWOpen source toolchain for Switch development
License: ISC License
Open source toolchain for Switch development
License: ISC License
Here's what misson sent me :
An NRO file has four sections in it: .text
, .rodata
, .data
, and .bss
.
The NRO file itself is just a concatenation of these sections, with a small header. To make things easier, we just include this header in the .text section and start .text at the beginning of the file.
The NRO header is 0x80 bytes long, which is why you see us skip 0x80 bytes at the start of .text in crt0.nro.S
. The NRO header is generated by elf2nxo.py
there. Another important header is the MOD0 header. I don't think we really need to include it exactly, since I'm pretty sure it's only used by the Nintendo runtime linker (rtld
), but we include it anyway. It lives near the start of .data and describes, among other things, where the linking information lives.
As such, the flat NRO file looks like this:
.text
.rodata
.data
Special care needs to be taken to ensure that elf2nxo.py
places .rela.dyn
and .dynamic
where the linker expects. If anyone is better at link scripts than I am and can figure out how to get the linker to put those sections in .rodata
and .data
respectively, please so.
The NRO file on the disk is pretty much exactly the same as the NRO image that gets loaded. There's no compression, no sections get moved around, or anything. The last piece of the puzzle is to relocate all the code. This really should be done in C, but right now for us, that's part of NRO loading. You can see in Nxo.cpp in Mephisto the stupid linker I wrote, and we have a similar one written in JavaScript.
newlib
must have been updated recently, and with it came another implementation of libssp
. This causes problems because our libssp implementation is conflicting with newlib's, causing duplicate symbols. We should either deactivate newlib's, or remove ours.
However, newlib's libssp depends on the __constructor__
symbol, which we don't support right now. I guess we should fix that too.
Tagging @dvdfreitag since he's the one who did the initial libssp port.
https://github.com/reswitched/newlib/tree/master/newlib/libc/ssp is newlib's ssp.
somethings' failing to find an ip address after I load my switch with runnro ace.nro in pegaswitch. (latest libtransistor + pegaswitch, all tools up to date)
Will probably have to include newlib with -isystem
because I tried -Weverything
earlier and 90% of it was newlib headers.
First need to figure out how to talk to sfdnsres
, then we need to implement in libtransistor. Thanks to hthh, we have the names for all of the IResolver
commands up on SwitchBrew: http://switchbrew.org/index.php?title=Sockets_services#sfdnsres. We haven't updated SwIPC yet, but it will still be able to tell us what parameters all of the commands take: https://reswitched.github.io/SwIPC/ifaces.html#nn::socket::resolver::IResolver.
Working PegaSwitch example cases
libtransistor bindings
For reference, here are my bsd:u
PegaSwitch test cases: https://gist.github.com/misson20000/82f92209800787fedda758c4333c6c8c.
I suspect that it will follow a pretty similar pattern to how the familiar bsd socket API has been adapted for HIPC. Any parameters to the original function call will be copied verbatim as raw data, except those representing buffers or those that are pointers to output values. The first u32 in the response is what is returned from the bsd function call, the second is the new value for errno, and any remaining ones are additional output values.
Running any NRO that uses framebuffer makes it unloadable.
With the merge of a recent PR (I think the makefile one), I started seeing the following warning everywhere :
clang-5.0: warning: unknown platform, assuming -mfloat-abi=soft
As far as I know, the nintendo switch (more specifically, the tegra x1) supports hard floats. We should pass -mfloat-abi=hard
to the Makefile ?
The current Makefile solution really isn't doing it. It's hard to read, hard to maintain, hard to read build output for, and doesn't even handle dependencies properly.
I would really appreciate the input of someone who's better at linker wrangling than I am on how to get the linker to just put everything into three sections. The current nxodance solution is really stupid. We pretty much manually concatenate sections from the ELF file into the RX, R, and RW sections that wind up in the NRO. The particularly fragile part is that we try to predict where each section goes based off of its size and a hardcoded alignment, which breaks when the linker does things like adding a section that we don't expect in the middle of everything.
When I build by make
, following errors are caused by llvm-mc.
Environment
mac OS Sierra 10.12.5
LLVM version 5.0.0
$ make
mkdir -p build/lib
llvm-mc -arch=aarch64 lib/crt0.nro.S -filetype=obj -o build/lib/crt0.nro.o
lib/crt0.nro.S:1:1: error: unknown directive
.hidden NORELOC__filestart
^
lib/crt0.nro.S:3:1: error: unknown directive
.hidden NORELOC__nro_modhdr
^
lib/crt0.nro.S:5:1: error: unknown directive
.hidden NORELOC_TEXT_START_
^
lib/crt0.nro.S:6:1: error: unknown directive
.hidden NORELOC_TEXT_END_
^
lib/crt0.nro.S:7:1: error: unknown directive
.hidden NORELOC_RODATA_START_
^
lib/crt0.nro.S:8:1: error: unknown directive
.hidden NORELOC_DYNSTR_START_
^
lib/crt0.nro.S:9:1: error: unknown directive
.hidden NORELOC_DYNSYM_START_
^
lib/crt0.nro.S:10:1: error: unknown directive
.hidden NORELOC_RODATA_END_
^
lib/crt0.nro.S:11:1: error: unknown directive
.hidden NORELOC_DATA_START_
^
lib/crt0.nro.S:12:1: error: unknown directive
.hidden NORELOC_DYNAMIC_START_
^
lib/crt0.nro.S:13:1: error: unknown directive
.hidden NORELOC_GOT_START_
^
lib/crt0.nro.S:14:1: error: unknown directive
.hidden NORELOC_GOT_PLT_START_
^
lib/crt0.nro.S:15:1: error: unknown directive
.hidden NORELOC_DATA_END_
^
lib/crt0.nro.S:16:1: error: unknown directive
.hidden NORELOC_BSS_START_
^
lib/crt0.nro.S:17:1: error: unknown directive
.hidden NORELOC_BSS_END_
Should we get rid of those, and use intX_t
everywhere instead ?
Here is my Log building an Nro bigger than 20 MB , the issue dont happen with smaller files up to 16Mega.
Switch Crashes with Error Code: 2168-0003 (cant also load the file with Yuzu)
more than half of the times, calling hid_init() returns error 0xdc01
test_hid.c suffers of this for example
to continue brings a crash of the switch
( being able to test if a button is pushed is really usfeul during tests )
We need to figure out how to make exit
thread-safe. I'm getting kind-of doubtful this is possible to do entirely, but right now exit in another thread gives us a segfault. We might be able to at least give it a saner behavior.
~/libtransistor# make
mkdir -p /root/libtransistor/build/lib
llvm-mc -arch=aarch64 -triple aarch64-none-switch /root/libtransistor/lib/crt0.nro.S -filetype=obj -o /root/libtransistor/build/lib/crt0.nro.o
/root/libtransistor/lib/crt0.nro.S:42:21: error: expected compatible register, symbol or integer in range [0, 4095]
sub sp, sp, 0x10
Ubuntu trusty, Just installed llvm, clang, cmake, python-lz4, etc.
There currently is no license associated with libtransistor, which makes it default to All rights reserved. It seems libnx
is using ISC (basically "everyone can do whatever"). Should we follow suit ?
Looking at libnx
for inspiration, it seems that their newlib
is actually a clean newlib from upstream (well, from devkitPro), and all they do is populate a global structure of function pointers (for the syscalls). This is called early in their crt0.
By doing this, it is possible to keep newlib clean, making it easier to update, and creates a clear separation of concerns between the libc and the "OS support". Furthermore, it would avoid the "submodule dance" whenever we want to add a feature to the libc, where we have to update the newlib and libtransistor in order to propagate changes.
So I'm compiling this from an archlinux box on which I installed clang
, llvm
, lld
and the python deps. I'm currently stuck with the following error :
ld.lld -Bsymbolic --shared --emit-relocs --no-gc-sections --no-undefined -T link.T -o build/test/test_malloc.nro.s o build/test/test_malloc.o --whole-archive build/lib/libtransistor.nro.a --no-whole-archive newlib/aarch64-none-sw itch/newlib/libc.a
ld.lld: error: undefined symbol: __stack_chk_guard
>>> referenced by ipc.c:53 (lib/ipc.c:53)
>>> ipc.o:(ipc_marshal) in archive build/lib/libtransistor.nro.a
Full output : https://gist.github.com/5af916a59ce82bc73aabe91bf14eeaf7
It'd be nice if we had an auto-generated documentation of libtransistor (the core lib - newlib is posix-compliant and has its own docs) hosted on gh-pages, using travis and doxygen to auto build them on each commit.
@3Daniel expressed interest in implementing this.
Lots of people don't see it. It's the easiest way to get a simple project. We should do something so people find that.
Right now, ace_loader's readme is a bit barren. We should probably explain how it works, and how to use it, as it's likely to become the main way to use homebrew.
Hello,
When compiling on archlinux, I get the following error running make
:
roblabla@roblab ~/Dropbox/dev/src/c/libtransistor master $ make
mkdir -p build/lib
llvm-mc -arch=aarch64 -triple aarch64-none-switch lib/crt0.nro.S -filetype=obj -o build/lib/crt0.nro.o
mkdir -p build/lib
llvm-mc -arch=aarch64 -triple aarch64-none-switch lib/svc.S -filetype=obj -o build/lib/svc.o
mkdir -p build/lib
clang -g -fPIC -ffreestanding -fexceptions -target aarch64-none-linux-gnu -O0 -mtune=cortex-a53 -I include/ -I newlib/newlib/libc/include/ -I newlib/newlib/libc/sys/switch/include/ -Wall -c -o build/lib/ipc.o lib/ipc.c
lib/ipc.c:366:7: warning: unused variable 'c_descriptor_flags' [-Wunused-variable]
int c_descriptor_flags = (header1 >> 10) & 0xF;
^
lib/ipc.c:397:7: warning: unused variable 'raw_data_start' [-Wunused-variable]
int raw_data_start = h;
^
mkdir -p build/lib
llvm-mc -arch=aarch64 -triple aarch64-none-switch lib/tls.S -filetype=obj -o build/lib/tls.o
2 warnings generated.
mkdir -p build/lib
clang -g -fPIC -ffreestanding -fexceptions -target aarch64-none-linux-gnu -O0 -mtune=cortex-a53 -I include/ -I newlib/newlib/libc/include/ -I newlib/newlib/libc/sys/switch/include/ -Wall -c -o build/lib/util.o lib/util.c
In file included from lib/util.c:5:
In file included from include/libtransistor/ipc/bsd.h:4:
In file included from /usr/include/sys/types.h:156:
/usr/include/bits/stdint-intn.h:27:19: error: typedef redefinition with different types ('__int64_t' (aka 'long long') vs 'long')
typedef __int64_t int64_t;
^
/usr/lib/clang/5.0.0/include/stdint.h:107:24: note: previous definition is here
typedef __INT64_TYPE__ int64_t;
^
In file included from lib/util.c:5:
In file included from include/libtransistor/ipc/bsd.h:6:
In file included from /usr/include/netinet/in.h:22:
/usr/include/bits/stdint-uintn.h:27:20: error: typedef redefinition with different types ('__uint64_t' (aka 'unsigned long long') vs 'unsigned long')
typedef __uint64_t uint64_t;
^
/usr/lib/clang/5.0.0/include/stdint.h:109:25: note: previous definition is here
typedef __UINT64_TYPE__ uint64_t;
^
2 errors generated.
make: *** [Makefile:43: build/lib/util.o] Error 1
So I'm not totally sure, but are we supposed to be pulling in /usr/include
like this ? Shouldn't libtransistor be building independently of the host headers ?
This is heavily related to #58 : We'll need to figure out a way to share resources between multiple started NROs, off the top of my head : filesystem hierarchy and open file descriptors. This would allow us to port stuff like bash
without too much trouble, and it would "generalize" the way to pass file descriptors from the parent to the child (I'm thinking of stdin
/stdout
/stderr
from ace_loader to the started process).
I think the best would be to have the libtransistor_context contain an optional pointer to the file descriptor array (or NULL if we don't want to pass any fds), which the child would copy into its own array. Something similar would probably be done to share a filesystem hierarchy.
Not so often, HID init fails with 0xdc01. When this happens, it is not possible to use HID at all until console reset.
It seems to be tied to specific browser run, not to NRO runs.
We need to implement thread_local storage ! This will probably require some modifications to the linker script. This is a fairly advanced task.
There are a number of functions we don't have bindings for yet. Here is a list of all the functions in nn::socket::sf::IClient
and what we have support for.
Most of these aren't important to me, but I think we should implement Select
at least.
Here's the definition straight from SwIPC:
Unknown5(u32, bytes<0x18>, buffer<unknown, 0x21, 0>, buffer<unknown, 0x21, 0>, buffer<unknown, 0x21, 0>) -> (u32, u32, buffer<unknown, 0x22, 0>, buffer<unknown, 0x22, 0>, buffer<unknown, 0x22, 0>)
Relevant signatures and definitions:
int select(int nfds, fd_set *restrict readfds,
fd_set *restrict writefds, fd_set *restrict errorfds,
struct timeval *restrict timeout);
struct timeval {
long tv_sec; /* seconds */
long tv_usec; /* microseconds */
};
See bionic for fd_set definitions: https://android.googlesource.com/platform/bionic/+/7943df62f70f686b0c77532f6617b47255d75763/libc/include/sys/select.h.
This is my best guess at what all the parameters are:
Select(u32 nfds, bytes<0x18> timeout, buffer<fd_set, 0x21, 0> readfds_in, buffer<fd_set, 0x21, 0> writefds_in, buffer<fd_set, 0x21, 0> errorfds_in) -> (u32 ret, u32 errno, buffer<fd_set, 0x22, 0> readfds_out, buffer<fd_set, 0x22, 0> writefds_out, buffer<fd_set, 0x22, 0> errorfds_out)
I'm not exactly sure how struct timeval
can become 0x18 bytes long though.
For examples of PegaSwitch test cases, see ipcsock3.js: https://gist.github.com/misson20000/82f92209800787fedda758c4333c6c8c/
Currently, libtransistor doesn't pass the -nostdinc
flag to clang when compiling, which causes it to use the host's headers. When I tried to add -nostdinc
, I was greeted with errors about the use of stdbool.h, which isn't defined by newlib.
Here's the current flags I use to make clang happy :
.clang_arg("-nostdinc")
.clang_args(&["-isystem", "/usr/lib/clang/5.0.0/include"])
.clang_args(&["-isystem", "libtransistor/newlib/newlib/libc/include"])
.clang_args(&["-isystem", "libtransistor/newlib/newlib/libc/sys/switch/include"])
.clang_arg("-Ilibtransistor/include")
/usr/lib/clang/CLANG_VERSION/include
contains stdbool.h, among other headers, defining many built-in C types. The equivalent in GCC seems to be /usr/lib/gcc/TARGET_TRIPLET/GCC_VERSION/include
.
The lack of having a way to allocate contiguous type-5 memory is now blocking a number of hacks.
The proposed API is to pass a function pointer in libtransistor_context
for void *(*alloc_pages)(size_t min, size_t max, size_t *actual)
. For allocations such as those listed above, min
would equal max
. For allocating space for the heap, min
can be 0x1000 so that we can allocate any small blocks that may exist, since we're going to be mapping them anyway and can't use them for much else.
So yeah, just running make
somehow doesn't work for me. I get the following error:
make
mkdir -p build/lib
llvm-mc -arch=aarch64 -triple aarch64-none-switch lib/crt0.nro.S -filetype=obj -o build/lib/crt0.nro.o
llvm-mc: error: invalid target 'aarch64'.
make: *** [Makefile:47: build/lib/crt0.nro.o] Error 1
I'm running Solus Budgie (Linux) and all dependencies are installed properly. I also tried removing the aarch64-none-switch
that was added in order to fix building on Mac, but that didn't do the trick unfortunately.
If you have any ideas on how to fix this, I'd be grateful.
Thanks!
libcxx requires long double support, but newlib does not implement it. There is a stubbed implementation which just calls the double version of the function, for example: cosl.
There are a few options to resolve this issue:
_LDBL_EQ_DBL
is defined, and fake long double support with newlib's existing stubs. This will truncate all 128-bit floats to 64-bit floats.The .ctors
section contains an array of funtion pointers that the dynamic loader is supposed to call. This is necessary to get newlib's libssp working.
libtransistor claims to be ISC, yet:
https://github.com/reswitched/libtransistor/blob/master/include/libtransistor/fd.h#L5
"This comes straight from the linux kernel".
The Linux kernel is licensed under the GPLv2 license (only), therefore causing libtransistor and all the programs using it GPLv2 in effect.
As a solution, you could use parts of the FreeBSD kernel instead.
Right now, ace_loader sends "remapped" heap in the OverrideHeap loader section. This is bad for multiple reasons:
We should either find a way to properly unmap the whole browser, or settle with finding the biggest contiguous region in our heap.
A few users got burned by this already. We should check if uid = 0 and deny building if that's the case (maybe with an escape hatch flag ?).
Some services hand out handles to the user. The user may forget to close them. We should keep track on them, and automatically close them on exit. For one: it's saner, and for two, the current behavior of leaving them open causes crashes because HBL is stingy. See #110
bpc#1 will reboot the console: https://github.com/reswitched/pegaswitch/blob/master/exploit/main.js#L117
We should probably call setjmp very early in the start function.
When libtransistor NROs are started from hbl, they crash on exit, yielding error 2347-0026. This is an undocumented errno. Pinging @plutooo on this. Any hints as to what that error code means ?
Libnx sometimes returns from the entry point with a fucked up stack, which causes segfault and all other kind of goodies : https://github.com/switchbrew/libnx/blob/5b0de6c0546da3aa9af7cb6606fee48e091688d9/nx/source/runtime/env.c#L91
We should call the NRO in an asm thunk that backs up all registers before jumping to the entry point, and restoring the registers before returning.
In trying to load resources for an application, one will quickly encounter the problem of no filesystem functions having been implemented.
I'd like to implement a sort of virtual filesystem API, where you can mount filesystems provided by various modules and access files from there. Filesystem providers could include some sort of network filesystem, a read-only filesystem embedded in the binary, one that lives in memory, or, eventually, native filesystems such as the console's internal storage and SD card.
Thoughts? What exactly do we want this to look like?
"/usr/bin/clang"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: /home/connor/oss/libtransistor/build/compiler-rt/CMakeFiles/CMakeTmp
Run Build Command:"/usr/bin/make" "cmTC_1967b/fast"
make[1]: Entering directory '/home/connor/oss/libtransistor/build/compiler-rt/CMakeFiles/CMakeTmp'
/usr/bin/make -f CMakeFiles/cmTC_1967b.dir/build.make CMakeFiles/cmTC_1967b.dir/build
make[2]: Entering directory '/home/connor/oss/libtransistor/build/compiler-rt/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_1967b.dir/testCCompiler.c.o
/usr/bin/clang --target=aarch64-none-linux-gnu -g -fPIC -ffreestanding -fexceptions -fuse-ld=lld -fstack-protector-strong -O0 -mtune=cortex-a53 -target aarch64-none-linux-gnu -nostdlib -nostdlibinc -isystem /home/connor/oss/libtransistor/newlib/newlib/libc/sys/switch/include/ -isystem /home/connor/oss/libtransistor/newlib/newlib/libc/include/ -I/home/connor/oss/libtransistor/include/ -I /home/connor/oss/libtransistor/build/sdl2_install/include/ -D__SWITCH__=1 -o CMakeFiles/cmTC_1967b.dir/testCCompiler.c.o -c /home/connor/oss/libtransistor/build/compiler-rt/CMakeFiles/CMakeTmp/testCCompiler.c
clang-5.0: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
Linking C executable cmTC_1967b
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_1967b.dir/link.txt --verbose=1
/usr/bin/clang --target=aarch64-none-linux-gnu -g -fPIC -ffreestanding -fexceptions -fuse-ld=lld -fstack-protector-strong -O0 -mtune=cortex-a53 -target aarch64-none-linux-gnu -nostdlib -nostdlibinc -isystem /home/connor/oss/libtransistor/newlib/newlib/libc/sys/switch/include/ -isystem /home/connor/oss/libtransistor/newlib/newlib/libc/include/ -I/home/connor/oss/libtransistor/include/ -I /home/connor/oss/libtransistor/build/sdl2_install/include/ -D__SWITCH__=1 -fuse-ld=lld CMakeFiles/cmTC_1967b.dir/testCCompiler.c.o -o cmTC_1967b
clang-5.0: error: invalid linker name in argument '-fuse-ld=lld'
make[2]: *** [CMakeFiles/cmTC_1967b.dir/build.make:98: cmTC_1967b] Error 1
make[2]: Leaving directory '/home/connor/oss/libtransistor/build/compiler-rt/CMakeFiles/CMakeTmp'
make[1]: *** [Makefile:126: cmTC_1967b/fast] Error 2
make[1]: Leaving directory '/home/connor/oss/libtransistor/build/compiler-rt/CMakeFiles/CMakeTmp'
CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:14 (project)
-- Configuring incomplete, errors occurred!
See also "/home/connor/oss/libtransistor/build/compiler-rt/CMakeFiles/CMakeOutput.log".
See also "/home/connor/oss/libtransistor/build/compiler-rt/CMakeFiles/CMakeError.log".
make: *** [Makefile:92: /home/connor/oss/libtransistor/build/compiler-rt/Makefile] Error 1
build is on arch linux with latest llvm
and clang
installed.
The current implementation uses a pool of memory in .bss
. Although this is what many sysmodules do, it's kinda stupid. The better option would be to use svc_set_heap_size
. However, if we're running under the browser, I'm think this will move the heap somewhere else and segfault running browser code. Not sure about that, but I haven't managed to call it under the browser without crashing somehow.
So I was wondering how good an idea it'd be to generate bindings automatically from SwIPC (with something like reswitched/SwIPC#9). Basically, replace most of the stuff in the ipc/
folder with auto-generated things.
This would be a good litmus test to ensure that our SwIPC documentation is correct, while also removing a bunch of boilerplate.
@misson20000 how do you feel about this ?
The documentation PR was a good first step in the right direction, but it still needs a lot of work:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.