schrodingerzhu / snmalloc-rs Goto Github PK

View Code? Open in Web Editor NEW

103.0 6.0 13.0 125 KB

rust bindings of snmalloc

License: MIT License

Rust 100.00%

snmalloc-rs's Introduction

snmalloc-rs

Notice: MinGW Build is broken and may not be fixed in a near future. See this PR in the upstream.

MSVC/MinGW/Linux/MacOS:

FreeBSD:

snmalloc-rs provides a wrapper for microsoft/snmalloc to make it usable as a global allocator for rust. snmalloc is a research allocator. Its key design features are:

Memory that is freed by the same thread that allocated it does not require any synchronising operations.
Freeing memory in a different thread to initially allocated it, does not take any locks and instead uses a novel message passing scheme to return the memory to the original allocator, where it is recycled.
The allocator uses large ranges of pages to reduce the amount of meta-data required.

Some old benchmark results are available in the snmalloc paper. Some recent benchmark results are listed at bench_suite. There are three features defined in this crate:

debug: Enable the Debug mode in snmalloc.
~~1mib: Use the 1mib chunk configuration. From 0.2.17, this is set as a default feature~~ (removed since 0.3.0)
~~16mib: Use the 16mib chunk configuration.~~ (removed since 0.3.0)
~~cache-friendly: Make the allocator more cache friendly (setting CACHE_FRIENDLY_OFFSET to 64 in building the library).~~ (removed since 0.3.0)
native-cpu: Optimize snmalloc for the native CPU of the host machine. (this is not a default behavior since 0.2.14)
qemu: Workaround madvise problem of QEMU environment
~~stats: Enable statistics~~ (removed since 0.3.0)
local_dynamic_tls: Workaround cannot allocate memory in static tls block
build_cc: Use of cc crate instead of cmake (cmake still default) as builder (more platform agnostic)
~~usecxx20: Enable C++20 standard if available~~ (removed since 0.3.0)
usecxx17: Use C++17 standard
check: Enable extra checks to improve security, see upstream security docs. Note that the memcpy protection is not enabled in Rust.
win8compat: Improve compatibility for old Windows platforms (removing usages of VirtualAlloc2 and other new APIs)

To get the crates compiled, you need to choose either 1mib or 16mib to determine the chunk configuration

To use snmalloc-rs add it as a dependency:

# Cargo.toml
[dependencies]
snmalloc-rs = "0.3.4"

To set SnMalloc as the global allocator add this to your project:

#[global_allocator]
static ALLOC: snmalloc_rs::SnMalloc = snmalloc_rs::SnMalloc;

For MinGW Users

mingw version is only tested on nightly branch with MSYS environment. We are using dynamic linking method. Hence, please make sure the following libs are in your PATH:

winpthread
atomic
stdc++
gcc_s

Notice: since version 0.2.12, we no longer require you to provide additional environment variables for mingw target.

For Android Cross-Compilation

ANDROID_NDK must be provided as an environment variable
ANDROID_PLATFORM can be passed as an optional environment variable
ANDROID_ABI used by CMake is detected automatically
feature android-lld can be used to set the linker of snmalloc to lld
~~feature android-shared-std can be used to set the STL library of snmalloc to c++_shared (it uses c++_static by default)~~ (libstdc++ is no longer a dependency)

Changelog

0.3.4

Tracking upstream to version 0.6.2.

0.3.3

Tracking upstream to fix Linux PAL typo.

0.3.2

Tracking upstream to enable old Linux variants.

0.3.1

Fixes build_cc feature (broken in 0.3.0 release).
Fixes native-cpu feature (broken in 0.3.0 release).

0.3.0

Release to support snmalloc 0.6.0.

0.3.0-beta.1

Beta release to support snmalloc 2 0.6.0

snmalloc-rs's People

Contributors

Stargazers

Watchers

Forkers

mjp41 snf licenser kylixs anupdhml photoszzt etheryalos ryancinsight devnexen beyonddream-productions wpbrown lqd shepmaster

snmalloc-rs's Issues

Failing cross compilation aarch64-unknown-linux-musl

  -- Configuring done
  -- Generating /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build
     Called from: [1]	/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
  -- Generating done
  -- Build files have been written to: /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build
  running: "cmake" "--build" "." "--target" "snmallocshim-1mib-rust" "--config" "Release" "--"
  /usr/local/bin/cmake -S/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc -B/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build --check-build-system CMakeFiles/Makefile.cmake 0
  /usr/bin/make  -f CMakeFiles/Makefile2 snmallocshim-1mib-rust
  make[1]: Entering directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  /usr/local/bin/cmake -S/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc -B/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build --check-build-system CMakeFiles/Makefile.cmake 0
  /usr/local/bin/cmake -E cmake_progress_start /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles 1
  /usr/bin/make  -f CMakeFiles/Makefile2 CMakeFiles/snmallocshim-1mib-rust.dir/all
  make[2]: Entering directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  /usr/bin/make  -f CMakeFiles/snmallocshim-1mib-rust.dir/build.make CMakeFiles/snmallocshim-1mib-rust.dir/depend
  make[3]: Entering directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  cd /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc /cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build /target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/DependInfo.cmake --color=
  Dependee "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/DependInfo.cmake" is newer than depender "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/depend.internal".
  Dependee "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/depend.internal".
  Scanning dependencies of target snmallocshim-1mib-rust
  make[3]: Leaving directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  /usr/bin/make  -f CMakeFiles/snmallocshim-1mib-rust.dir/build.make CMakeFiles/snmallocshim-1mib-rust.dir/build
  make[3]: Entering directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  [100%] Building CXX object CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o
  /usr/local/bin/aarch64-linux-musl-g++  -DSNMALLOC_EXPORT="__attribute__((visibility(\"default\")))" -I/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/src  -ffunction-sections -fdata-sections -fPIC -O3 -DNDEBUG -fvisibility=hidden   -Wall -Wextra -Werror -Wundef -fno-exceptions -fno-rtti -g -fomit-frame-pointer -ftls-model=initial-exec -mcx16 -std=gnu++1z -o CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o -c /cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/src/override/rust.cc
  CMakeFiles/snmallocshim-1mib-rust.dir/build.make:85: recipe for target 'CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o' failed
  make[3]: Leaving directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  CMakeFiles/Makefile2:1189: recipe for target 'CMakeFiles/snmallocshim-1mib-rust.dir/all' failed
  make[2]: Leaving directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  CMakeFiles/Makefile2:1196: recipe for target 'CMakeFiles/snmallocshim-1mib-rust.dir/rule' failed
  make[1]: Leaving directory '/target/aarch64-unknown-linux-musl/release/build/snmalloc-sys-9a2717f15c76e1a1/out/build'
  Makefile:581: recipe for target 'snmallocshim-1mib-rust' failed

  --- stderr
  CMake Warning at CMakeLists.txt:79 (message):
    Not generating clangformat target, no clang-format tool found
  Call Stack (most recent call first):
    CMakeLists.txt:366 (clangformat_targets)


     Called from: [2]	/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
                  [1]	/cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
  CMake Warning:
    Manually-specified variables were not used by the project:

      CMAKE_ASM_COMPILER
      CMAKE_ASM_FLAGS


  aarch64-linux-musl-g++: error: unrecognized command line option '-mcx16'
  make[3]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o] Error 1
  make[2]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/all] Error 2
  make[1]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/rule] Error 2
  make: *** [snmallocshim-1mib-rust] Error 2
  thread 'main' panicked at '
  command did not execute successfully, got: exit status: 2

  build script failed, must exit now', /cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.45/src/lib.rs:894:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I built it with cross.

Failing static compilation on stable-x86_64-pc-windows-msvc

Trying to build any crate which depends on sn-malloc and uses it as its global allocater with RUSTFLAGS=-C target-feature=+crt-static on Windows leads to a link-time error:

   Compiling snmalloc-windows-test v0.1.0 (C:\Users\xxx\Documents\dev\rustspikes\snmalloc-windows-test)
error: linking with `link.exe` failed: exit code: 1120
  |
  = note: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX64\\x64\\link.exe" "/NOLOGO" "/NXCOMPAT" "/LIBPATH:C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\snmalloc_windows_test.snmalloc_windows_test.uvh4yiuq-cgu.0.rcgu.o" "C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\snmalloc_windows_test.snmalloc_windows_test.uvh4yiuq-cgu.1.rcgu.o" "C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\snmalloc_windows_test.snmalloc_windows_test.uvh4yiuq-cgu.2.rcgu.o" "C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\snmalloc_windows_test.snmalloc_windows_test.uvh4yiuq-cgu.3.rcgu.o" "/OUT:C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\snmalloc_windows_test.exe" "C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\snmalloc_windows_test.1hv601l7hgil9fss.rcgu.o" "/OPT:REF,ICF" "/DEBUG" "/NATVIS:C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\intrinsic.natvis" "/NATVIS:C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\liballoc.natvis" "/NATVIS:C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libcore.natvis" "/NATVIS:C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\etc\\libstd.natvis" "/LIBPATH:C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps" "/LIBPATH:C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\build\\snmalloc-sys-edccc1af11933945\\out\\./build/Release" "/LIBPATH:C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib" "C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\libsnmalloc_rs-ac271706809acdd3.rlib" "C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\libsnmalloc_sys-c9c44a07abfb04ce.rlib" "C:\\Users\\xxx\\Documents\\dev\\rustspikes\\snmalloc-windows-test\\target\\release\\deps\\liblibc-dacd56e8d1ed66b8.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libstd-b3b7d2754e4f3c4a.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libpanic_unwind-ae2d0fd7efecc707.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_demangle-190d7f5796f574cb.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libhashbrown-f81dd2aa8d51e1ef.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_alloc-9a56cd7574396589.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libunwind-938cf2d7f7936554.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcfg_if-3ed683af0e96cdb9.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liblibc-24993eff3af321f7.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\liballoc-7b7f5b91dda770f4.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\librustc_std_workspace_core-00274f5eadf75c2b.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcore-465f87485a6021ee.rlib" "C:\\Users\\xxx\\.rustup\\toolchains\\stable-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-pc-windows-msvc\\lib\\libcompiler_builtins-d9775021cc0867d6.rlib" "snmallocshim-1mib-rust.lib" "mincore.lib" "advapi32.lib" "ws2_32.lib" "userenv.lib" "libcmt.lib"
  = note: LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:library
          LINK : warning LNK4217: symbol '__acrt_iob_func' defined in 'libucrt.lib(_file.obj)' is imported by 'snmallocshim-1mib-rust.lib(rust.obj)' in function '"public: static void __cdecl snmalloc::PALWindows::error(char const * const)" (?error@PALWindows@snmalloc@@SAXQEBD@Z)'
          LINK : warning LNK4217: symbol 'fflush' defined in 'libucrt.lib(fflush.obj)' is imported by 'snmallocshim-1mib-rust.lib(rust.obj)' in function '"public: static void __cdecl snmalloc::PALWindows::error(char const * const)" (?error@PALWindows@snmalloc@@SAXQEBD@Z)'
          LINK : warning LNK4217: symbol 'abort' defined in 'libucrt.lib(abort.obj)' is imported by 'snmallocshim-1mib-rust.lib(rust.obj)' in function '"public: static void __cdecl snmalloc::PALWindows::error(char const * const)" (?error@PALWindows@snmalloc@@SAXQEBD@Z)'
          snmallocshim-1mib-rust.lib(rust.obj) : error LNK2019: unresolved external symbol __imp_puts referenced in function "public: static void __cdecl snmalloc::PALWindows::error(char const * const)" (?error@PALWindows@snmalloc@@SAXQEBD@Z)
          C:\Users\xxx\Documents\dev\rustspikes\snmalloc-windows-test\target\release\deps\snmalloc_windows_test.exe : fatal error LNK1120: 1 unresolved externals
          

error: aborting due to previous error

error: could not compile `snmalloc-windows-test`

To learn more, run the command again with --verbose.

warning in build_cc

@mjp41 saw new warning in updates to snmalloc using gcc, should I add any flags as suggested below in warning.

warning: In file included from snmalloc/src/override/../mem/slab.h:3,
warning: from snmalloc/src/override/../mem/chunkmap.h:9,
warning: from snmalloc/src/override/../mem/alloc.h:11,
warning: from snmalloc/src/override/../mem/globalalloc.h:4,
warning: from snmalloc/src/override/../mem/slowalloc.h:3,
warning: from snmalloc/src/override/malloc.cc:1,
warning: from snmalloc/src/override/rust.cc:2:
warning: snmalloc/src/override/../mem/freelist.h: In member function ‘void* snmalloc::Allocator<NeedsInitialisation, InitThreadAllocator, MemoryProvider, ChunkMap, IsQueueInline>::small_alloc_new_slab(snmalloc::sizeclass_t) [with snmalloc::ZeroMem zero_mem = snmalloc::NoZero; bool (* NeedsInitialisation)(void*) = snmalloc::needs_initialisation; void* (* InitThreadAllocator)(snmalloc::function_ref<void*(void*)>) = snmalloc::init_thread_allocator; MemoryProvider = snmalloc::MemoryProviderStateMixinsnmalloc::PALLinux; ChunkMap = snmalloc::DefaultChunkMap<>; bool IsQueueInline = true]’:
warning: snmalloc/src/override/../mem/freelist.h:120:20: warning: ‘b.snmalloc::FreeListBuilder::head[1].snmalloc::EncodeFreeObjectReference::reference’ may be used uninitialized in this function [-Wmaybe-uninitialized]
warning: 120 | return encode(local_key, reference);
warning: | ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
warning: snmalloc/src/override/../mem/freelist.h: In member function ‘void* snmalloc::Allocator<NeedsInitialisation, InitThreadAllocator, MemoryProvider, ChunkMap, IsQueueInline>::small_alloc_new_slab(snmalloc::sizeclass_t) [with snmalloc::ZeroMem zero_mem = snmalloc::YesZero; bool (* NeedsInitialisation)(void*) = snmalloc::needs_initialisation; void* (* InitThreadAllocator)(snmalloc::function_ref<void*(void*)>) = snmalloc::init_thread_allocator; MemoryProvider = snmalloc::MemoryProviderStateMixinsnmalloc::PALLinux; ChunkMap = snmalloc::DefaultChunkMap<>; bool IsQueueInline = true]’:
warning: snmalloc/src/override/../mem/freelist.h:120:20: warning: ‘b.snmalloc::FreeListBuilder::head[1].snmalloc::EncodeFreeObjectReference::reference’ may be used uninitialized in this function [-Wmaybe-uninitialized]
warning: 120 | return encode(local_key, reference);
warning: | ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
warning: snmalloc/src/override/../mem/freelist.h: In member function ‘void* snmalloc::Allocator<NeedsInitialisation, InitThreadAllocator, MemoryProvider, ChunkMap, IsQueueInline>::small_alloc_rare(snmalloc::sizeclass_t, size_t) [with snmalloc::ZeroMem zero_mem = snmalloc::YesZero; bool (* NeedsInitialisation)(void*) = snmalloc::needs_initialisation; void* (* InitThreadAllocator)(snmalloc::function_ref<void*(void*)>) = snmalloc::init_thread_allocator; MemoryProvider = snmalloc::MemoryProviderStateMixinsnmalloc::PALLinux; ChunkMap = snmalloc::DefaultChunkMap<>; bool IsQueueInline = true]’:
warning: snmalloc/src/override/../mem/freelist.h:120:20: warning: ‘b.snmalloc::FreeListBuilder::head[1].snmalloc::EncodeFreeObjectReference::reference’ may be used uninitialized in this function [-Wmaybe-uninitialized]
warning: 120 | return encode(local_key, reference);
warning: | ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
warning: snmalloc/src/override/../mem/freelist.h: In member function ‘void* snmalloc::Allocator<NeedsInitialisation, InitThreadAllocator, MemoryProvider, ChunkMap, IsQueueInline>::small_alloc_rare(snmalloc::sizeclass_t, size_t) [with snmalloc::ZeroMem zero_mem = snmalloc::NoZero; bool (* NeedsInitialisation)(void*) = snmalloc::needs_initialisation; void* (* InitThreadAllocator)(snmalloc::function_ref<void*(void*)>) = snmalloc::init_thread_allocator; MemoryProvider = snmalloc::MemoryProviderStateMixinsnmalloc::PALLinux; ChunkMap = snmalloc::DefaultChunkMap<>; bool IsQueueInline = true]’:
warning: snmalloc/src/override/../mem/freelist.h:120:20: warning: ‘b.snmalloc::FreeListBuilder::head[1].snmalloc::EncodeFreeObjectReference::reference’ may be used uninitialized in this function [-Wmaybe-uninitialized]
warning: 120 | return encode(local_key, reference);
warning: | ~~~~~~^~~~~~~~~~~~~~~~~~~~~~

Failed to cross compile arm target on x86 mac

Hi,

I am trying to cross compile to an arm binary from a x86 mac machine in github action job, but it's failing with the following error:

  /usr/local/Cellar/cmake/3.21.3/bin/cmake -E cmake_progress_start /Users/runner/work/roapi/roapi/target/aarch64-apple-darwin/debug/build/snmalloc-sys-77e000b0b98622f1/out/build/CMakeFiles 1
  /Applications/Xcode_12.4.app/Contents/Developer/usr/bin/make  -f CMakeFiles/Makefile2 CMakeFiles/snmallocshim-1mib-rust.dir/all
  /Applications/Xcode_12.4.app/Contents/Developer/usr/bin/make  -f CMakeFiles/snmallocshim-1mib-rust.dir/build.make CMakeFiles/snmallocshim-1mib-rust.dir/depend
  cd /Users/runner/work/roapi/roapi/target/aarch64-apple-darwin/debug/build/snmalloc-sys-77e000b0b98622f1/out/build && /usr/local/Cellar/cmake/3.21.3/bin/cmake -E cmake_depends "Unix Makefiles" /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc /Users/runner/work/roapi/roapi/target/aarch64-apple-darwin/debug/build/snmalloc-sys-77e000b0b98622f1/out/build /Users/runner/work/roapi/roapi/target/aarch64-apple-darwin/debug/build/snmalloc-sys-77e000b0b98622f1/out/build /Users/runner/work/roapi/roapi/target/aarch64-apple-darwin/debug/build/snmalloc-sys-77e000b0b98622f1/out/build/CMakeFiles/snmallocshim-1mib-rust.dir/DependInfo.cmake --color=
  /Applications/Xcode_12.4.app/Contents/Developer/usr/bin/make  -f CMakeFiles/snmallocshim-1mib-rust.dir/build.make CMakeFiles/snmallocshim-1mib-rust.dir/build
  [100%] Building CXX object CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o
  /usr/bin/c++ -DBACKTRACE_HEADER=\"execinfo.h\" -DCACHE_FRIENDLY_OFFSET=64 -DSNMALLOC_EXPORT="__attribute__((visibility(\"default\")))" -I/Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/src -ffunction-sections -fdata-sections -fPIC -arch arm64 -O3 -DNDEBUG -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -mmacosx-version-min=10.9 -fvisibility=hidden -Wsign-conversion -Wconversion -Wall -Wextra -Werror -Wundef -fno-exceptions -fno-rtti -g -fomit-frame-pointer -ftls-model=initial-exec -mcx16 -std=gnu++17 -MD -MT CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o -MF CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o.d -o CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o -c /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/src/override/rust.cc

  --- stderr
  CMake Warning (dev) at /usr/local/Cellar/cmake/3.21.3/share/cmake/Modules/CMakeDetermineCompiler.cmake:143 (set):
    Policy CMP0126 is not set: set(CACHE) does not remove a normal variable of
    the same name.  Run "cmake --help-policy CMP0126" for policy details.  Use
    the cmake_policy command to set the policy and suppress this warning.

    For compatibility with older versions of CMake, normal variable
    "CMAKE_C_COMPILER" will be removed from the current scope.
  Call Stack (most recent call first):
    /usr/local/Cellar/cmake/3.21.3/share/cmake/Modules/CMakeDetermineCCompiler.cmake:68 (_cmake_find_compiler_path)
    CMakeLists.txt:2 (project)
  This warning is for project developers.  Use -Wno-dev to suppress it.

     Called from: [3]	/usr/local/Cellar/cmake/3.21.3/share/cmake/Modules/CMakeDetermineCompiler.cmake
                  [2]	/usr/local/Cellar/cmake/3.21.3/share/cmake/Modules/CMakeDetermineCCompiler.cmake
                  [1]	/Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
  CMake Warning (dev) at /usr/local/Cellar/cmake/3.21.3/share/cmake/Modules/CMakeDetermineCompiler.cmake:143 (set):
    Policy CMP0126 is not set: set(CACHE) does not remove a normal variable of
    the same name.  Run "cmake --help-policy CMP0126" for policy details.  Use
    the cmake_policy command to set the policy and suppress this warning.

    For compatibility with older versions of CMake, normal variable
    "CMAKE_CXX_COMPILER" will be removed from the current scope.
  Call Stack (most recent call first):
    /usr/local/Cellar/cmake/3.21.3/share/cmake/Modules/CMakeDetermineCXXCompiler.cmake:66 (_cmake_find_compiler_path)
    CMakeLists.txt:2 (project)
  This warning is for project developers.  Use -Wno-dev to suppress it.

     Called from: [3]	/usr/local/Cellar/cmake/3.21.3/share/cmake/Modules/CMakeDetermineCompiler.cmake
                  [2]	/usr/local/Cellar/cmake/3.21.3/share/cmake/Modules/CMakeDetermineCXXCompiler.cmake
                  [1]	/Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
  CMake Warning at CMakeLists.txt:79 (message):
    Not generating clangformat target, no clang-format tool found
  Call Stack (most recent call first):
    CMakeLists.txt:366 (clangformat_targets)


     Called from: [2]	/Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
                  [1]	/Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.27/snmalloc/CMakeLists.txt
  CMake Warning:
    Manually-specified variables were not used by the project:

      CMAKE_ASM_COMPILER
      CMAKE_ASM_FLAGS


  clang: error: argument unused during compilation: '-mcx16' [-Werror,-Wunused-command-line-argument]
  make[3]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/src/override/rust.cc.o] Error 1
  make[2]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/all] Error 2
  make[1]: *** [CMakeFiles/snmallocshim-1mib-rust.dir/rule] Error 2
  make: *** [snmallocshim-1mib-rust] Error 2
  thread 'main' panicked at '
  command did not execute successfully, got: exit status: 2

Here is the build command I used to reproduce the issue:

cargo build --bin roapi-http --target aarch64-apple-darwin --features=simd

I can see that the -mcx16 flag is set in snmalloc-rs/snmalloc-sys/build.rs. Is this a bug in the cc crate? It looks like the build.flag_if_supported("-mcx16") flag test is performed with the host target instead of the specified target.

Here is the host environment provided by github action:

Darwin Mac-1633928908187.local 19.6.0 Darwin Kernel Version 19.6.0: Tue Aug 24 20:28:00 PDT 2021; root:xnu-6153.141.40~1/RELEASE_X86_64 x86_64

Full CI run error can be found at: https://github.com/roapi/roapi/pull/89/checks?check_run_id=3855785363

GitHub action task config:

  mac_build:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install Rust toolchain
        uses: actions-rs/toolchain@v1
        with:
          toolchain: nightly-${{ env.RUST_TC_NIGHTLY_VER }}
          target: aarch64-apple-darwin
          profile: minimal
          default: true
      - name: Build binary
        env:
          DEVELOPER_DIR: /Applications/Xcode.app/Contents/Developer
          MACOSX_DEPLOYMENT_TARGET: '10.9'
        run: |
          # set SDKROOT for C dependencies
          export SDKROOT=$(xcrun --sdk macosx --show-sdk-path)
          cd roapi-http && \
            cargo build --bin roapi-http --target aarch64-apple-darwin --features=simd

Strict Main Branch Protection

@mjp41 I enabled strict main branch protection following github's guidance. Not sure if this will bring trouble. Ping me and I will be happy to fix/change.

Segmentation fault EXC_BAD_ACCESS

* thread #32, name = 'tokio-runtime-worker', stop reason = EXC_BAD_ACCESS (code=1, address=0x40)
  * frame #0: 0x000000010105353e fuse-query`::rust_alloc(size_t, size_t) [inlined] snmalloc::EncodeFreeObjectReference::read(this=0x0000000000000040, entropy=0x0000000210022150) at freelist.h:122:32 [opt]
    frame #1: 0x000000010105353e fuse-query`::rust_alloc(size_t, size_t) [inlined] snmalloc::FreeObject::read_next(this=0x0000000000000040, entropy=0x0000000210022150) at freelist.h:158 [opt]
    frame #2: 0x000000010105353e fuse-query`::rust_alloc(size_t, size_t) [inlined] snmalloc::FreeListIter::take(this=<unavailable>, entropy=0x0000000210022150) at freelist.h:244 [opt]

failing compilation with musl

Hi,
I'm trying to build an application using snmalloc in with a statically linked musl libc instead of the gnu libc, during that we've been running in a few problems.

I am not sure 100% where the issue orriginates, from my research it seems to be related to the usage of C++ and global destructors.

The error boils down to:

/usr/local/bin/../lib/gcc/x86_64-linux-musl/9.2.0/../../../../x86_64-linux-musl/bin/ld: /target/x86_64-unknown-linux-musl/debug/deps/tremor-17be01d5a5f619ac: hidden symbol `__dso_handle' isn't defined

To make it easy I've created a minimal reproduction with nothing but snmalloc-rs and a mainline function:

https://github.com/Licenser/snmalloc-musl

The error can be triggered by simply running make, it will build a musl docker image and then try to compile the application in it.

Error on windows 7 builds

Placing this here since I'm not sure if it's on rust side,

Using snmalloc causes windows 7 waitforsingleobject error despite working on windows 10 normally

Mimalloc does not cause the same issue though.

@mjp41 any ideas where this error could be coming from, I tried injecting dll's and reinstalling vc_redist but windows 7 gave me same error despite windows 10 being fine with it.

Does not build for MSVC with stats feature flag

Building Custom Rule C:/Users/xxx/.cargo/registry/src/github.com-1ecc6299db9ec823/snmalloc-sys-0.2.17/snmalloc/CMakeLists.txt
  rust.cc
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\ostream(376,1): error C2220: the following warning is treated as an error [C:\Users\xxx\Documents\dev\rustspikes\wikidumpgrep\target\release\build\snmalloc-sys-9db952a15b16475a\out\build\snmallocshim-1mib-rust.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\ostream(369): message : while compiling class template member function 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' [C:\Users\xxx\Documents\dev\rustspikes\wikidumpgrep\target\release\build\snmalloc-sys-9db952a15b16475a\out\build\snmallocshim-1mib-rust.vcxproj]
C:\Users\xxx\.cargo\registry\src\github.com-1ecc6299db9ec823\snmalloc-sys-0.2.17\snmalloc\src\mem\../ds/csv.h(44): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::basic_ostream<char,std::char_traits<char>>::operator <<(unsigned __int64)' being compiled [C:\Users\xxx\Documents\dev\rustspikes\wikidumpgrep\target\release\build\snmalloc-sys-9db952a15b16475a\out\build\snmallocshim-1mib-rust.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\ostream(642): message : see reference to class template instantiation 'std::basic_ostream<char,std::char_traits<char>>' being compiled [C:\Users\xxx\Documents\dev\rustspikes\wikidumpgrep\target\release\build\snmalloc-sys-9db952a15b16475a\out\build\snmallocshim-1mib-rust.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include\ostream(376,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc [C:\Users\xxx\Documents\dev\rustspikes\wikidumpgrep\target\release\build\snmalloc-sys-9db952a15b16475a\out\build\snmallocshim-1mib-rust.vcxproj]

--- stderr
CMake Warning at CMakeLists.txt:70 (message):
  Not generating clangformat target, no clang-format tool found
Call Stack (most recent call first):
  CMakeLists.txt:326 (clangformat_targets)


CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_ASM_FLAGS
    CMAKE_ASM_FLAGS_RELEASE


thread 'main' panicked at '
command did not execute successfully, got: exit code: 1

build script failed, must exit now', C:\Users\xxx\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.44\src\lib.rs:885:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Update doc to introduce CI changes

Custom allocator options

@SchrodingerZhu I was going over the rust docs and came across the custom-allocator section. Do you know the difference between this and setting the global allocator?https://doc.rust-lang.org/1.5.0/book/custom-allocators.html

Build fails on Windows 10 with VS 2019 (VS 2017 is also installed)

Repro:

cargo new --bin snmt
add snmalloc 0.2.17 to dependencies
cargo check or cargo build

Result:

   Compiling snmalloc-sys v0.2.17
error: failed to run custom build command for `snmalloc-sys v0.2.17`

Caused by:
  process didn't exit successfully: `C:\work\nayato\snmt\target\debug\build\snmalloc-sys-2540178998b215d3\build-script-build` (exit code: 101) 
--- stdout
running: "cmake" "C:\\Users\\mgortman\\.cargo\\registry\\src\\github.com-1ecc6299db9ec823\\snmalloc-sys-0.2.17\\snmalloc" "-G" "Visual Studio 16 2019" "-Thost=x64" "-Ax64" "-DSNMALLOC_RUST_SUPPORT=ON" "-DCMAKE_CXX_FLAGS_RELEASE=/MD /O2 /Ob2 /DNDEBUG" "-DCMAKE_C_FLAGS_RELEASE=/MD /O2 /Ob2 /DNDEBUG" "-DCMAKE_INSTALL_PREFIX=C:\\work\\nayato\\snmt\\target\\debug\\build\\snmalloc-sys-919db581a6feec40\\out" "-DCMAKE_C_FLAGS= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_BUILD_TYPE=Release"

--- stderr
CMake Error: Could not create named generator Visual Studio 16 2019

Generators
  Visual Studio 15 2017 [arch] = Generates Visual Studio 2017 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 14 2015 [arch] = Generates Visual Studio 2015 project files.
                                 Optional [arch] can be "Win64" or "ARM".
.....

thread 'main' panicked at '
command did not execute successfully, got: exit code: 1

build script failed, must exit now', C:\Users\...\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.44\src\lib.rs:885:5
stack backtrace:
   0: backtrace::backtrace::trace_unsynchronized
             at C:\Users\VssAdministrator\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.46\src\backtrace\mod.rs:66
...

Tracking `snmalloc 2`

@mjp41 sorry for not catching up with upstream for a while. so do we need to remove cache-friendly flag now?

Follow upstream refactor for CHERI architecture

upstream is under active development for CHERIfication.

let us wait for a proper merge point for updates.

Change the `if` statement in `snmalloc-sys/build.rs` to be more futureproof

snmalloc-rs/snmalloc-sys/build.rs

Lines 30 to 36 in 089d298

 if cfg!(unix) { 

 println!("cargo:rustc-link-search=native={}", dst.display()); 

 println!("cargo:rustc-link-lib=dylib=stdc++"); 

 } else { 

 println!("cargo:rustc-link-search=native={}/{}", dst.display(), build_type); 

 println!("cargo:rustc-link-lib=dylib=mincore"); 

 }

This if statement should be changed to like the following:

    if cfg!(windows) {
        println!("cargo:rustc-link-search=native={}/{}", dst.display(), build_type);
        println!("cargo:rustc-link-lib=dylib=mincore");
    } else {
        println!("cargo:rustc-link-search=native={}", dst.display());
        println!("cargo:rustc-link-lib=dylib=stdc++");
    }

Since only windows targets have the mincore library.

MinGW Support

This issue is left for improvements for MinGW toolchains.

Adjust for CPU Target

rustc has its own CPU Target configuration.

Should we detect that, and use it as an argument to pass to the cmake?

Or just leave it as a feature flag?

@mjp41

native-cpu feature issue with 0.3.0

I'm unable to build with native-cpu feature, the output contains one failed test, i'm not sure how to output any other useful information.

build works without native-cpu

-- Performing Test SNMALLOC_LINKER_SUPPORT_NO_ALLOW_SHLIB_UNDEF Called from: [4] /usr/local/Cellar/cmake/3.23.1_1/share/cmake/Modules/Internal/CheckSourceCompiles.cmake [3] /usr/local/Cellar/cmake/3.23.1_1/share/cmake/Modules/Internal/CheckLinkerFlag.cmake [2] /usr/local/Cellar/cmake/3.23.1_1/share/cmake/Modules/CheckLinkerFlag.cmake [1] /.cargo/git/checkouts/snmalloc-rs-801a4b46a1ccfde5/3eeba56/snmalloc-sys/snmalloc/CMakeLists.txt -- Performing Test SNMALLOC_LINKER_SUPPORT_NO_ALLOW_SHLIB_UNDEF - Failed Called from: [4] /usr/local/Cellar/cmake/3.23.1_1/share/cmake/Modules/Internal/CheckSourceCompiles.cmake [3] /usr/local/Cellar/cmake/3.23.1_1/share/cmake/Modules/Internal/CheckLinkerFlag.cmake [2] /usr/local/Cellar/cmake/3.23.1_1/share/cmake/Modules/CheckLinkerFlag.cmake [1] /.cargo/git/checkouts/snmalloc-rs-801a4b46a1ccfde5/3eeba56/snmalloc-sys/snmalloc/CMakeLists.txt

Automating keeping snmalloc up to date.

For Project Verona @plietar built a github action that automatically raises an issue if the snmalloc submodule is out of date. This might be useful for this project too:

https://github.com/microsoft/verona/blob/master/.github/workflows/update-snmalloc.yaml

removal of libc dependency breaks compilation on systems with older libc

As reported upstream in microsoft/snmalloc#328
snmalloc needs a fix for older libc versions that dont have a getentropy function.

While upgrading snmalloc-rs from 0.2.26 to 0.2.27 we broke our builds as we need to build on an old libc system (2.17 to be exact) (don ask). While 0.2.26 brought its own libc, which was more up to date than the system one, compilation succeeded.

With 0.2.27 we now depend on the system libc, which broke our build.

I am working on a fix for upstream snmalloc to work on older libc systems, but wanted to report this to you anyways.

	if cfg!(unix) {
	println!("cargo:rustc-link-search=native={}", dst.display());
	println!("cargo:rustc-link-lib=dylib=stdc++");
	} else {
	println!("cargo:rustc-link-search=native={}/{}", dst.display(), build_type);
	println!("cargo:rustc-link-lib=dylib=mincore");
	}