Git Product home page Git Product logo

rust2go's Introduction

Rust2Go

Crates.io

Rust2Go is a project that provides users with a simple and efficient way to call Golang from Rust with native async support.

Features

  • Sync and async calls from Rust to Golang
  • Efficient data exchange: no serialization or socket communication, but FFI
  • Simple interface design: no new invented IDL except for native rust

How to Use

  1. Define the structs and calling interfaces in restricted Rust syntax, and include generated code in the same file.
  2. Generate golang code with rust2go-cli --src src/user.rs --dst go/gen.go
  3. Write a build.rs for you project.
  4. You can then use generated implementation to call golang in your Rust project!

For detailed example, please checkout the example project.

Key Design

Detailed design details can be found in this article: Design and Implementation of a Rust-Go FFI Framework.

Why Fast?

In order to achieve the ultimate performance, this project is not based on communication, but on FFI to pass specially encoded data. In order to reduce memory operations to a minimum, data that satisfies a specific memory layout is passed directly by reference rather than copied.

For example, Vec<u8> and String is represented as a pointer and a length. However, structs like Vec<String> or Vec<Vec<u8>> require intermediate representation. In order to reduce the number of memory allocations to one, I use a precomputed size buffer to store these intermediate structures.

Memory Safety

On the Golang side, the data it receives is referenced from Rust. The Rust side will do its best to ensure the validity of this data during the call. So the Golang side can implement the handler arbitrarily, but manually deep copy when leaking data outside the function life cycle.

On the Rust side, it is needed to ensure that the slot pointer of the callback ffi operation, and the user parameters are valid when the future drops. This is archieved by implementing an atomic slot structure and providing a [drop_safe] attribute to require user passing parameters with ownership.

Toolchain Requirements

  • Golang: >=1.18
    • For >=1.18 && < 1.20: generate golang code with --go118
    • For >=1.20: generate golang code normally
  • Rust: >=1.75 if you want to use async

Milestones

Init Version

  • IDL(in rust) parse
  • Go code generation
  • Build script helper
  • Basic data types and convertion generation
  • Rust impl generation
  • Future and basic synchronization primitives used

Basic Ability Enhancement

  • More complicated data types support
  • Support user passing references
  • More elegant code generation implementation
  • Better build cache control
  • Golang interface support(separate user code from generated code)
  • Dynamic linking support
  • Golang helper library

Extended Features

  • Support calling rust from golang

Credit

This project is inspired by fcplug.

rust2go's People

Contributors

ihciah avatar bobozhengsir avatar suikammd avatar

Stargazers

Toni Ramírez avatar Dario Castañé avatar xukgo avatar  avatar CSF avatar  avatar ypcpy avatar Kotaro Inoue avatar kenkoooo avatar Kevin Yang avatar Mihail Zverev avatar Alex Kocharin avatar Jeff Carpenter avatar  avatar chan avatar 陈寒彤 avatar Tony Wu avatar Phillips Horselover avatar Hai-Hsin avatar Marcelo Henrique Neppel avatar Aoi avatar ringsaturn avatar like avatar yichya QC avatar

Watchers

 avatar  avatar

rust2go's Issues

Weird error when have more than two params

When my fn have more than two params, rustc complains like this:

error: expected one of `!`, `(`, `)`, `+`, `,`, `::`, or `<`, found `client_hello_id_version`
  --> src/ffi.rs:52:60
   |
52 |     fn set_client_hello_id(client_hello_id_client: String, client_hello_id_version: String);
   |                                                          - ^^^^^^^^^^^^^^^^^^^^^^^ expected one of 7 possible tokens
   |                                                          |
   |                                                          help: missing `,`

I tested in offcial example and encountered with same issue. I wonder if it's a bug and come here for help.

Passing reference in sync calls should also be considered unsafe

Thank you for creating this great crate!

I noticed that passing a reference in a sync call might cause a dangling pointer because the reference on the Go side might be saved, while it could have already been dropped on the Rust side.

Given that the design goal of this crate is to pursue extreme performance, this might be expected. However, I am wondering if synchronous calls should also be considered unsafe, similar to asynchronous calls.

Please have a look or try this simple example:
https://github.com/Heap-Hop/rust2go/tree/unsafe_sync

Heap-Hop@9734b24#diff-8aa8e83ff3b2189fd6397149a7b274f82daede3923191a3d0409f72a2f8bb78fR36

If I missed any mention of similar issues in the documentation, please correct me. Thanks again.

Support for cross compilation

First of all, really great work, thanks for putting this together!

I was wondering if you would be interested in accepting a patch to add support for running the go compilation step with cross compiler support.

My use case is that I would like to use rust2go to build a rust library that calls a go library that's then wrapped with JNI to be included inside of an Android application (yeah it's basically inception).

In order to achieve that I would need to be able to set certain environment variables (and possibly command line options) when the go build call is placed here: https://github.com/ihciah/rust2go/blob/master/rust2go/src/build.rs#L123.

You can check this cmake file and Makefile to understand more in detail what kind of configuration is needed to cross compile a golang shared library to be included inside of an android application.

Effectively you need to set GOOS, GOARCH, CC, CXX, CGO_CFLAGS, CGO_LDFLAGS and then add some extra parameters to the build command like: -tags linux.

I think the patch would not be too big, depending on how it's done. I see a few approaches for to doing it:

  1. Extend struct Builder to accept an additional set of parameters such as the extra environment variables and extra parameters to the go build call. The issue is that if the call to the go build changes, you need to careful of what are the default variables. Maybe there is an idiomatic way in rust to have a default set of environment variables that you can allow a user to extend or replace entirely.
  2. The most popular way to build a rust library in Android, AFAIK, is to use the mozilla rust-android-gradle plugin for it. Another approach could be to check the currently set environment variables and see if rust is being built from within the context of this kind of cross compilation (see: https://github.com/mozilla/rust-android-gradle/blob/master/plugin/src/main/kotlin/com/nishtahir/CargoBuildTask.kt#L208). Basically checking if something like CARGO_TARGET_* is set and extract from there the necessary things to map the rust specific targets into the golang syntax.

I think 2 would be nicer in that it would make the whole thing work out of the box without requiring changes to support cross compilation, however it's not clear to me if these CARGO_TARGET_* are standards or if it's something specific to the mozilla rust-android-gradle.

If you let me know which approach you think is best, I'm happy to draft a PR for it.

Weird bug without `#[repr(C)]`

According to offcial example, we should add rust2go::R2G attribute to structs that will be passed to Go side.

#[derive(Debug, rust2go::R2G)]
pub struct Http2SettingFfi {
    pub setting_id: u16,
    pub setting_val: u32,
}

However, when it comes to the example above, there will come a weird bug that Go side will exchange, well I wonder if I explain clearly, values of these two fields.

After thinking for a while, I come to the solution. It's simple: don't forget #[repr(C)], which assures that the order, size, and alignment of fields is exactly what you would expect from C that acting as FFI bridge between Rust and Go.

#[derive(Debug, rust2go::R2G)]
#[repr(C)]
pub struct Http2SettingFfi {
    pub setting_id: u16,
    pub setting_val: u32,
}

BTW, I also want to share my experience here: if you pass a string from Rust to Go, you'd better make a deep copy especially when it acts like config, etc, otherwise if original one is dropped within Rust side, you may encounter with unfarished behaviors.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.