xudong-huang / generator-rs Goto Github PK

View Code? Open in Web Editor NEW

286.0 14.0 35.0 358 KB

rust stackful generator library

License: Apache License 2.0

Rust 90.34% Assembly 9.66%

rust generator yield stackful-coroutines

generator-rs's People

Contributors

Stargazers

Watchers

generator-rs's Issues

Release new version

Can we get a new release with 628d0dc? The larger dependency list is preventing me from pulling the latest release into rust-analyzer.

Can't I create generator in generator recursively?

I'm trying to export this python's function to rust, but it seems like recursive infinity.
https://github.com/micromario/talisman-optimiser/blob/038b847b1c/bot.py#L476

I wrote the following code:

fn routes<'a>(accessories: &Accessories, rarity: Rarity) -> Generator<'a, (), Route> {
    fn helper<'a>(
        rarity: Rarity,
        count: i32,
        index: usize,
        current: Vec<i32>,
    ) -> Generator<'a, (), Route> {
        Gn::new_scoped(move |mut s| {
            if count == 0 {
                s.yield_with(Route::new(current.clone(), rarity));
            } else if index == *TALISMAN_REFORGES_LEN - 1 {
                let mut new = current.clone();
                new[index] += count;
                s.yield_with(Route::new(new, rarity));
            } else {
                if let Some(_) = TALISMAN_RAW[index][rarity as usize] {
                    let mut new = current.clone();
                    new[index] += 1;
                    for x in helper(rarity, count - 1, index, new) {
                        s.yield_with(x);
                    }
                }
                for x in helper(rarity, count, index + 1, current) {
                    s.yield_with(x);
                }
            }
            done!()
        })
    }

    return helper(
        rarity,
        accessories.get_count(rarity),
        0,
        vec![0; *TALISMAN_REFORGES_LEN],
    );
}

Maybe it is due to this crate is not start the generator from point at called yield last time I guess.

Any way to get stack start of generator?

Hello! How do I get stack start of generator? I need this for conservative roots in my GC

`Scope` API is unsound

Scope::new takes 2 *mut Option<_> parameters, but the rest of the Scope API assumes that those raw pointers are valid. For example, the public get_yield method does the following:

let para = unsafe { &mut *self.para };
para.take()

Calling Scope::new with invalid (null, misaligned or dangling) pointers is possible in safe Rust, and get_yield can also be called from safe Rust, therefore this API is unsound.

Either Scope::new should be marked unsafe, or it should take &'a mut Option<_> instead of raw pointers, adding a lifetime parameter to Scope.

support stable rust compile

need to use gcc crate for the asm compilation

Add suport for `ExactSizeIterator`

The Generator struct implements Iterator, but not ExactSizeIterator. It would be nice to be able to create one if you already have access to the number of elements you are going to generate.

Safe sendable `Generator` is unsound

Sadly sendable Generator is currently unsound because thread local variables may leak non-sendable values from within the our coroutine. Here's example:

#![forbid(unsafe_code)]

use std::{cell::Cell, rc::Rc, thread};

use generator::Gn;

const COUNT_ITERATIONS: u64 = 1000000000;

fn main() {
    thread_local! {
        static COUNTER: Cell<Option<Rc<Cell<u64>>>> = Cell::new(None);
    }

    let counter = Rc::new(Cell::new(0));
    COUNTER.with(|tl| tl.set(Some(Rc::clone(&counter))));

    let mut generator = Gn::new_scoped(move |mut s| {
        let counter = COUNTER.with(|tl| tl.take()).unwrap();
        s.yield_with(());
        println!("Rc {:p} in thread {:?}", counter, thread::current());
        for _ in 0..COUNT_ITERATIONS {
            counter.set(counter.get() + 1)
        }
    });
    generator.next().unwrap();

    let child = thread::spawn(move || {
        generator.next().unwrap();
    });

    println!("Rc {:p} in thread {:?}", counter, thread::current());
    for _ in 0..COUNT_ITERATIONS {
        counter.set(counter.get() + 1)
    }

    child.join().unwrap();
    assert_eq!(counter.get(), 2 * COUNT_ITERATIONS);
}

Output:

Rc 0x60000058d210 in thread Thread { id: ThreadId(1), name: Some("main"), .. }
Rc 0x60000058d210 in thread Thread { id: ThreadId(2), name: None, .. }
thread 'main' panicked at src/main.rs:37:5:
assertion `left == right` failed
  left: 1028478159
 right: 2000000000

See https://blaz.is/blog/post/lets-pretend-that-task-equals-thread/ and https://users.rust-lang.org/t/controversial-opinion-keeping-rc-across-await-should-not-make-future-send-by-itself/100554/12 for details.

Interesting! But where to start?

It looks like generator-rs implements concurrent programming in another way than the future.
Am I right?

I'd like to learn more about how it works. But I am not an expert in this area:(.
Where can I find some doc/reference about the way in which the generator is implemented?

Thanks a lot! :DDD

Bus error when running in debug mode with thread_rng() on Apple M1

When running the following program in debug mode, I get a bus error.

use generator::{Generator, Gn, done};
use rand::{thread_rng, Rng};

fn simple_generator<'a>() -> Generator<'a, (), i32>{
    Gn::new_scoped(move |mut s| {
        let v: i32 = thread_rng().gen();
        s.yield_(v);
        done!();
    })
}

fn main() {
    for v in simple_generator(){
        println!("{}", v);
    }
}

It runs fine in release mode, or when there is no thread_rng involved.

I'm on an M1 Mac, Ventura 13.0.1. Stable Rust (1.72.0).

Free function `yield_` is unsound

It can be used to transmute types. Example causing a segmentation fault (at least on my machine):

use generator::*;

fn main() {
    let g = Gn::new_scoped(|_| {
        yield_::<String, _>(());
    });

    for () in g {}
}

Cargo test --release failed

error: process didn't exit successfully: /Users/joyious/Workspace/sdag/generator-rs/target/release/deps/generator-fbaeb80c50e742bc (signal: 11, SIGSEGV: invalid memory reference)

Recursive generators

I would like to ask if this implementation of generators is appropriate to be used recursively.

It is not an uncommon pattern to use recursive generator functions for certain problems. Any algorithm with the shape of a tree/graph search problem could benefit from this pattern for making it lazy (only return more results when the caller calls next()).

Even if we were not calling the same function recursively, it is also a useful pattern to have a deep stack of generator functions, such that a high-level function may pull results from it as needed in a lazy manner. A web API with pagination for example.

From my own research of the source code, it looks like each call to Gn::new_scoped will dynamically allocate a new stack using mmap (for unix), which by default is of 4KB (0x1000 bytes). This means that each recursive call of a generator function will consume quite a lot of memory. Is this correct?

The default size can be reduced, but exceeding the stack causes a hard SIGSEGV and it would be quite inconvenient to tune the stack size for each function so that it is just large enough. I'm also getting failed to alloc sys stack: IoError(Os { code: 12, kind: Other, message: "Cannot allocate memory" }) panics by just recursing 6 times, which seems unreasonable.

Is there a better way of doing this? I have considered passing the Scope object down the stack, but I believe that the yields in deeper function calls would emit at the top level, which is not the behaviour that you would expect from a generator stack.

Since you probably know the ecosystem better. Is there any other generator library more suited to this? Such as genawaiter for example? Or the native Rust generators, which are still unstable?

Thank you in advance. And apologies if this is a bit too abstract to understand. I don't have time right now to give code examples, but feel free to ask questions and I will clarify as best I can.

大佬，问下汇编推荐什么书？

Compile failure on windows

For some reason, generator is starting to fail to compile on windows. This started recently on the Azure windows CI platform. Here is an example of a failed build:

https://dev.azure.com/tokio-rs/Tokio/_build/results?buildId=4652&view=logs&jobId=edbb8bb1-bc6c-5b89-6b1a-b043f7e49efc&j=edbb8bb1-bc6c-5b89-6b1a-b043f7e49efc&t=38b5b02f-30ce-5594-6cc9-d7432c882ad9

The error is:

error: failed to run custom build command for `generator v0.6.19`

Caused by:
  process didn't exit successfully: `D:\a\1\s\target\debug\build\generator-7edb320b2034b9ab\build-script-build` (exit code: 1)
--- stdout
TARGET = Some("i686-pc-windows-msvc")
running: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Enterprise\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX64\\x86\\ml.exe" "-safeseh" "-FoD:\\a\\1\\s\\target\\debug\\build\\generator-cb101e03a46c8720\\out\\src/detail/asm\\asm_i386_ms_pe_masm.o" "-c" "src/detail/asm\\asm_i386_ms_pe_masm.asm"
cargo:warning=Microsoft (R) Macro Assembler Version 14.16.27034.0
cargo:warning=Copyright (C) Microsoft Corporation.  All rights reserved.
cargo:warning=
 Assembling: src/detail/asm\asm_i386_ms_pe_masm.asm
MASM : fatal error A1000:cannot open file : src/detail/asm\asm_i386_ms_pe_masm.asm
exit code: 1

--- stderr


error occurred: Command "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Enterprise\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX64\\x86\\ml.exe" "-safeseh" "-FoD:\\a\\1\\s\\target\\debug\\build\\generator-cb101e03a46c8720\\out\\src/detail/asm\\asm_i386_ms_pe_masm.o" "-c" "src/detail/asm\\asm_i386_ms_pe_masm.asm" with args "ml.exe" did not execute successfully (status code exit code: 1).

Generator's Send trait should have bounds

Hi there, we (Rust group @sslab-gatech) are scanning crates on crates.io for potential soundness bugs. We noticed that Generator implements Send as long as the closure has a static lifetime. However, this should also probably be bounded by T: Send, otherwise it's possible to smuggle across non-Send types across thread boundaries.

Here's an example of a data race in safe Rust code through a Generator.

#![forbid(unsafe_code)]

use generator::Gn;
use std::{rc::Rc, thread};

fn main() {
    let rc = Rc::new(());

    let rc_clone = rc.clone();
    let mut generator = Gn::new_scoped(move |_| {
        return rc_clone;
    });

    let child = thread::spawn(move || {
        let smuggled_rc = generator.next().unwrap();

        println!("RC in thread: {:p}", smuggled_rc);
        for _ in 0..1000000000 {
            let x = smuggled_rc.clone();
        }
    });

    println!("RC in main: {:p}", rc);
    for _ in 0..1000000000 {
        let x = rc.clone();
    }

    child.join().unwrap();
    assert_eq!(Rc::strong_count(&rc), 2);
}

Output:

RC in main: 0x5587d9ed6a50
RC in thread: 0x5587d9ed6a50

Return Code: -4 (SIGILL)

Windows on arm64 support

i see that is no aarch64_windows.rs support,so want windows on arm64 plz!

What causes this, here we just panic to exit the func??

I am intermittently getting the error thread 'tests::stealer_takes_blocking' panicked at 'Box<dyn Any>', /Users/aozen/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.7.0/src/yield_.rs:60:9 while testing some code changes under Loom to evaluate Consume memory orderings. This usually passes all tests, but when it doesn't and I get this error, when I check the source code all this says is // here we just panic to exit the func, so that does mean? I can't tell from this if an error even exists, and here I have ASAN/LSAN/TSAN/Miri/Loom testing all showing this works correctly in most runs.

This is the code in question: https://github.com/Bajix/swap-queue-rs/tree/feature/consume-ordering

(probably) erroneous warning

When some of the values generated by a generator created by Gn are not consumed, a warning appears in the logs.

I assumed this warning is erroneous, and should at most be a trace!() if any.

use generator::*;

// This generator seems totally fine
fn generate_values(start: u8) -> impl Iterator<Item = u8> {
    Gn::new_scoped(move |mut s| { // The warning will only appears if it's a moved closure
        for i in start..20 {
            s.yield_(i);
        }
        done!()
    })
}

fn main() {
    env_logger::init();
    let _: Vec<_> = dbg!(
        generate_values(3).
        .take(3) // this will abort the generator early
        .collect());
}

# Cargo.toml
[dependencies]
log = "0.4"
env_logger = "0.7.1"

$ RUST_LOG=trace cargo run
[2020-07-24T15:45:33Z WARN  generator::gen_impl] generator is not done while drop
[src/main.rs:26] foo(3).take(3).collect() = [
    3,
    4,
    5,
]

`done()` is unsound and should be marked `unsafe`

done is equivalent to mem::uninitialized and thus can be used to trivially construct uninitialized instances of types.

The function is documented "don't use it directly, use done!() macro instead", but it's neither hidden nor marked as unsafe, both should probably be done.

This results in a segmentation fault on my machine:

use generator::{co_get_yield, co_set_para, Gn, done};

fn main() {
    done::<(String, Vec<u128>, Box<[u128]>)>();
}

Incorrect use of assume_init

I noticed by coincidence some incorrect uses of assume_init in this crate:

generator-rs/src/yield_.rs

Line 32 in 9f45261

unsafe { ret.assume_init() }

generator-rs/src/stack/unix.rs

Line 97 in cd677f7

let mut limit = unsafe { limit.assume_init() };

Quoting from the assume_init docs (emphasis mine):

It is up to the caller to guarantee that the MaybeUninit really is in an initialized state. Calling this when the content is not yet fully initialized causes immediate undefined behavior.

It does not matter whether the value is dropped, or never used, or anything like that. Calling assume_init when the data is not yet initialized is a safety violation in all cases.

getrlimit is easy to fix, it should use MaybeUninit::as_mut_ptr. The other function looks very suspicious but I do not understand enough of what is going on to suggest a fix.

(The other two uses of assume_init in this crate also look suspicious, but I can't tell where the values are coming from -- they sure look like they have not actually been properly initialized yet, though.)

take advantage of libfringe for context switch

take advantage of libfringe context switch

please ref Xudong-Huang/may#31

`test_inner_ref` test fail when compiling v0.6.21 (current master) in debug mode when used in a workspace

When running cargo test I get the following error

  process didn't exit successfully: `$HOME/$WORSKSPACE/target/debug/deps/lib-2737ca9cdad51a01` (signal: 11, SIGSEGV: invalid memory reference)

I can't reproduce it I just clone the project in a standalone directory. However if I clone it in my workspace, and only in debug mode, the test test_inner_ref fails with the above error. I tried to rebuild (cargo clean + cargo build) and still got the error. Given that this test uses unsafe code, I think that some assumption about alignment are wrong.

bootstrap_green_task uses C abi but InitFn is Rust abi

src/reg_context.rs

// first argument is task handle, second is thunk ptr
pub type InitFn = fn(usize, *mut usize) -> !;

src/gen_impl.rs

/// the init function passed to reg_context
fn gen_init(_: usize, f: *mut usize) -> ! {

ARMv7 support

Hello there, I'm trying to compile may but it looks like generator doesn't likes my Raspberry Pi v2.

I used this command to try to build it:

git clone https://github.com/Xudong-Huang/may
cd may && cargo run --example echo --release

This is the error log:

error[E0432]: unresolved import `self::asm`
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/detail/mod.rs:33:15
   |
33 | pub use self::asm::{Registers, swap_registers, initialize_call_frame, prefetch};
   |               ^^^ Could not find `asm` in `self`

error[E0433]: failed to resolve. Use of undeclared type or module `Registers`
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:15:28
   |
15 |         RegContext { regs: Registers::new() }
   |                            ^^^^^^^^^ Use of undeclared type or module `Registers`

error[E0433]: failed to resolve. Use of undeclared type or module `Registers`
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:63:23
   |
63 |         let mut cur = Registers::new();
   |                       ^^^^^^^^^ Use of undeclared type or module `Registers`

error[E0412]: cannot find type `Registers` in this scope
 --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:7:11
  |
7 |     regs: Registers,
  |           ^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `initialize_call_frame` in this scope
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:36:9
   |
36 |         initialize_call_frame(&mut self.regs, init, arg, start, stack);
   |         ^^^^^^^^^^^^^^^^^^^^^ not found in this scope

error[E0412]: cannot find type `Registers` in this scope
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:47:28
   |
47 |         let out_regs: &mut Registers = match *out_context {
   |                            ^^^^^^^^^ not found in this scope

error[E0412]: cannot find type `Registers` in this scope
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:50:23
   |
50 |         let in_regs: &Registers = match *in_context {
   |                       ^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `swap_registers` in this scope
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:56:18
   |
56 |         unsafe { swap_registers(out_regs, in_regs) }
   |                  ^^^^^^^^^^^^^^ not found in this scope

error[E0412]: cannot find type `Registers` in this scope
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:64:20
   |
64 |         let regs: &Registers = &to_context.regs;
   |                    ^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `swap_registers` in this scope
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/generator-0.6.1/src/reg_context.rs:66:18
   |
66 |         unsafe { swap_registers(&mut cur, regs) }
   |                  ^^^^^^^^^^^^^^ not found in this scope
error: aborting due to 10 previous errors

error: Could not compile `generator`.
warning: build failed, waiting for other jobs to finish...

This is my /proc/cpu/info

processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 1
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 2
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 3
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 38.40
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

Hardware        : BCM2835
Revision        : a21041
Serial          : 0000000095c87563

Kernel, distro, toolchain and Rust info:

root@rust:~/may# lsb_release -a && uname -a && cargo --version && rustc --version && rustup toolchain list
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.3 (stretch)
Release:        9.3
Codename:       stretch
Linux rust 4.9.59-v7+ #1047 SMP Sun Oct 29 12:19:23 GMT 2017 armv7l GNU/Linux
cargo 0.25.0-nightly (930f9d949 2017-12-05)
rustc 1.24.0-nightly (250b49205 2017-12-21)
nightly-armv7-unknown-linux-gnueabihf (default)

breaking change with 0.7.6 release

With the 0.7.6 release, my project qsv fails to compile with this error:

cargo t -F all_features
...                                                                                                                                                                           
error[E0107]: struct takes 2 lifetime arguments but 1 lifetime argument was supplied
  --> C:\Users\joeln\.cargo\registry\src\index.crates.io-6f17d22bba15001f\kiddo-4.2.0\src\common\generate_within_unsorted_iter.rs:44:32
   |
44 |                   mut gen_scope: Scope<'a, (), NearestNeighbour<A, T>>,
   |                                  ^^^^^ -- supplied 1 lifetime argument
   |                                  |
   |                                  expected 2 lifetime arguments
   |
  ::: C:\Users\joeln\.cargo\registry\src\index.crates.io-6f17d22bba15001f\kiddo-4.2.0\src\fixed\query\within_unsorted_iter.rs:18:5
   |
18 | /     generate_within_unsorted_iter!(
19 | |         (r#"Finds all elements within `dist` of `query`, using the specified
20 | | distance metric function.
21 | |
...  |
46 | | ```"#)
47 | |     );
   | |_____- in this macro invocation
   |
note: struct defined here, with 2 lifetime parameters: `'scope`, `'a`
  --> C:\Users\joeln\.cargo\registry\src\index.crates.io-6f17d22bba15001f\generator-0.7.6\src\scope.rs:16:12
   |
16 | pub struct Scope<'scope, 'a, A, T> {
   |            ^^^^^ ------  --
   = note: this error originates in the macro `generate_within_unsorted_iter` (in Nightly builds, run with -Z macro-backtrace for more info)
help: add missing lifetime argument
   |
44 |                 mut gen_scope: Scope<'a, 'a, (), NearestNeighbour<A, T>>,
   |                                        ++++

error[E0107]: struct takes 2 lifetime arguments but 1 lifetime argument was supplied
  --> C:\Users\joeln\.cargo\registry\src\index.crates.io-6f17d22bba15001f\kiddo-4.2.0\src\common\generate_within_unsorted_iter.rs:44:32
   |
44 |                   mut gen_scope: Scope<'a, (), NearestNeighbour<A, T>>,
   |                                  ^^^^^ -- supplied 1 lifetime argument
   |                                  |
   |                                  expected 2 lifetime arguments
   |
  ::: C:\Users\joeln\.cargo\registry\src\index.crates.io-6f17d22bba15001f\kiddo-4.2.0\src\float\query\within_unsorted_iter.rs:43:5
   |
43 | /     generate_float_within_unsorted_iter!(
44 | |         "
45 | | let mut tree: KdTree<f64, 3> = KdTree::new();
46 | | tree.add(&[1.0, 2.0, 5.0], 100);
47 | | tree.add(&[2.0, 3.0, 6.0], 101);"
48 | |     );
   | |_____- in this macro invocation
   |
note: struct defined here, with 2 lifetime parameters: `'scope`, `'a`
  --> C:\Users\joeln\.cargo\registry\src\index.crates.io-6f17d22bba15001f\generator-0.7.6\src\scope.rs:16:12
   |
16 | pub struct Scope<'scope, 'a, A, T> {
   |            ^^^^^ ------  --
   = note: this error originates in the macro `generate_within_unsorted_iter` which comes from the expansion of the macro `generate_float_within_unsorted_iter` (in Nightly builds, run with -Z macro-backtrace for more info)
help: add missing lifetime argument
   |
44 |                 mut gen_scope: Scope<'a, 'a, (), NearestNeighbour<A, T>>,
   |                                        ++++

For more information about this error, try `rustc --explain E0107`.

I fixed it by pinning generator to 0.7.5
jqnatividad/qsv@0f53a4b

Perhaps, 0.7.6 should be yanked and re-released as 0.8.0 instead?

Doesn’t build on recent nightlies anymore

error: the legacy LLVM-style asm! syntax is no longer supported

munmap_chunk(): invalid pointer

where am I wrong? why do i get an error

use generator::Gn;

pub struct TestStruct<'a> {
    pub name: Cow<'a, str>,
    pub value: i32
}

impl<'a> TestStruct<'a>  {
    pub fn new() -> TestStruct<'a> {
        TestStruct { name: Cow::Owned("test".to_string()), value: 0 }
    }
}

pub fn test_1gen<'a> () -> impl Iterator<Item =TestStruct<'a>>
{
    Gn::new_scoped(move |mut s| {
        for _i in (0..10).step_by(2)
            {
                let py = TestStruct::new();
                s.yield_(py);
            }
        generator::done!();
    })
}

pub fn test_2gen<'a>() -> impl Iterator<Item = TestStruct<'a>> {
    let iter_test = test_1gen();

    Gn::new_scoped(move |mut s| {
        for i in iter_test
            {
                s.yield_(i);
            }
        generator::done!();
    })
}


fn main() {
    for i in test_2gen() {
        println!("{}", i.name);
    }
}

munmap_chunk(): invalid pointer
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Rust 1.33-1.35 stable/nightly linux-gnu - x86_64

compile failure in nightly

rustc 1.71.0-nightly (c609da59d 2023-04-18)
win11

 symbol `ROOT_CONTEXT_P` is already defined
  --> C:\Users\caobi\.cargo\registry\src\mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd\generator-0.7.3\src\rt.rs:28:1
   |
28 | static mut ROOT_CONTEXT_P: *mut Context = ptr::null_mut();
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: could not compile `generator` (lib) due to previous error
warning: build failed, waiting for other jobs to finish...

release a new version?

Any chance of publishing a new version? I'd like to pick up #31 now that several folks on my team are actively using M1 Macs :-) Thanks!

enable stack guard by default

this will alloc an extra 4k bytes page to guard stack overflow

Trampolines twice in unix stack bootstrapping

Hi - I was single-stepping through this code to make sure I understood how it worked, and I discovered that, at least on the x64 unix code (https://github.com/Xudong-Huang/generator-rs/blob/master/src/detail/x86_64_unix.rs), the trampoline ends up running twice.

The first swap context sets stack pointer to sp[-4] (using the sp defined in initialize_call_frame), and then returns into bootstrap_green_task. that function then puts fptr from R14 into sp[-2], returns again into bootstrap_green_task which is at sp[-3]. the copies from r12, r13, and r14 run again, harmlessly, and then finally it returns into fptr.

I believe the only necessary correction is to change mov %r14, 8(%rsp) to mov %r14, (%rsp) in bootstrap_green_task. The assignment of sp[-3] to bootstrap_green_task in initialize_call_frame can then be eliminated as well.

Alternately, you can not put fptr in regs.gpr[r14] at all, and put it directly into sp[-3], and then eliminate the mov %r14 from the bootstrap entirely.

However, it is also possible I have no idea what I'm talking about, because I've only learned Intel assembly programming very recently.

Compile failure on Windows with Rust Nightly

error: could not compile `generator`.

Caused by:
  process didn't exit successfully: `rustc --crate-name generator --edition=2018 C:\Users\runneradmin\.cargo\registry\src\github.com-1ecc6299db9ec823\generator-0.6.21\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=374606f5dac045c8 -C extra-filename=-374606f5dac045c8 --out-dir D:\a\wgpu\wgpu\target\debug\deps -L dependency=D:\a\wgpu\wgpu\target\debug\deps --extern log=D:\a\wgpu\wgpu\target\debug\deps\liblog-a8bc277de1943a86.rmeta --extern winapi=D:\a\wgpu\wgpu\target\debug\deps\libwinapi-762dcc63287a8a1e.rmeta --cap-lints allow --cfg nightly` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)
warning: build failed, waiting for other jobs to finish...
error: build failed

See https://github.com/gfx-rs/wgpu/runs/1022537234

`Gn::new_scoped` and similar functions are unsound

Gn::new_scoped allows a coroutine to borrow non-'static data, but it doesn't (and can't) guarantee it will end before the lifetime expires since the coroutine may be leaked during a yield point. This is unsound because Rust guarantees that sync code either reaches the end of its execution (normally or with a panic doesn't matter) or the process ends. Leaking a coroutine prevents the code inside from actually ending normally or with a panic, thus it must be considered as if it is still executing (albeit without making any progress) until the end of the process. This however requires it to not borrow non-'static data, which is however what Gn::new_scoped allows to.

The API that most evidently relies on guarantee mentioned is std::thread::scope, and in fact here is an example of use-after-free due ot its assumptions being violated:

let foo = "foo".to_string();

let mut gen = generator::Gn::new_scoped(|mut s| {
    std::thread::scope(|s2| {
        s2.spawn(|| {
            std::thread::sleep(std::time::Duration::from_millis(500));
            println!("{foo}");
        });
        s.yield_(());
    });
    generator::done!();
});

gen.next();
std::mem::forget(gen);
drop(foo);
std::thread::sleep(std::time::Duration::from_millis(1000));

libfringe broken in latest rust nightly

need to investigate

Context switching performance

Hey,
playing with coroutines and libfringe, it turned out that a vital part of context switching performance lies in pop+jmp vs ret.
This comment on HN sheds some light

jmp rax (or any register really) uses the indirect jump prediction, while ret uses a special dedicated predictor that uses a stack (the Return Stack Buffer or RSB), populated by call instructions, to predict the return address. In the coroutine case, the ret does not jump to the address of the last call so it will be mispredicted many time.

This is still a thing with the modern CPUs, zen3 at least.
Changing two lines on ~master:

generator-rs/src/detail/asm/asm_x86_64_sysv_elf_gas.S

Line 43 in 5888dac

ret

pop %rax
jmp *%rax

x86_64 zen3

name	before ns/iter	after ns/iter	diff ns/iter	diff %	speedup
scoped_yield_bench	22	9	-13	-59.09%	x 2.44
single_yield_bench	25	10	-15	-60.00%	x 2.50
single_yield_with_bench	22	9	-13	-59.09%	x 2.44

perf output for one of the benches

	before	after
page-faults	779.883	6457	/sec
stalled-cycles-frontend	12.93%	0.08%	frontend cycles idle
stalled-cycles-backend	1.30%	48.56%	backend cycles idle
instructions	1.32	3.01	insn per cycle
branches	1.247	2.754	G/sec
branch-misses	6.61%	0.03%	of all branches

I don't have other hardware at hand right now, but can test this on Macbook M1 this week.

Current scoped API is unsound

use std::{mem::forget, sync::mpsc};

use generator::Gn;

fn main() {
    let mut s = "Happy".to_owned();
    let (tx1, rx1) = mpsc::sync_channel(1);
    let (tx2, rx2) = mpsc::sync_channel(1);
    {
        let s = s.as_str();
        let mut coroutine = Gn::<()>::new_scoped_local(|mut yielder| {
            std::thread::scope(|scope| {
                let _thrd = scope.spawn(move || {
                    rx1.recv().unwrap();
                    tx2.send(s.to_owned()).unwrap();
                });
                yielder.yield_with(());
            });
            panic!();
        });
        coroutine.resume();
        forget(coroutine);
    }
    s.clear();
    s.push_str("Sad");
    tx1.send(()).unwrap();
    let data_race_s = rx2.recv().unwrap();
    dbg!(&s, &data_race_s);
}

Prints out:

[src/main.rs:28:5] &s = "Sad"
[src/main.rs:28:5] &data_race_s = "Sadpy"

Similar to Amanieu/corosensei#27

rustc 1.27.0-nightly breaks the debug version

rustc 1.27.0-nightly generate unwanted code for naked functions, this affect the logic of swap_registers by using inline assembly.

// why save the rcx and rdx in stack? this will overwite something!
// the naked function should only use the asm block, debug version breaks
// since rustc 1.27.0-nightly, we have to use O2 level optimization
        :
        : "{rcx}"(out_regs), "{rdx}"(in_regs) 
        : "memory"
        : "volatile");

xudong-huang / generator-rs Goto Github PK

generator-rs's People

Contributors

Stargazers

Watchers

Forkers

generator-rs's Issues

Recommend Projects

Recommend Topics

Recommend Org