Comments (16)
@bltavares heya! -- thanks for the question! I've been thinking about this as well, and I think there's more or less two takes on this:
The Short Haul
In the short run we could probabably get something working on Android, by creating bindings using cbindgen
(tutorial), and then manually binding that to Android.
Probably a first goal would be to get an in-memory version of Hypercore working first. But then as we build out the networking layer we can build that part out too.
This can probably already be experimented with -- probably by adding C support through cbbindgen
first (PRs would be super welcome!), and later adding more android-specific bits too.
The Long Haul
The manual binding part of the step above is probably going to be the biggest bottleneck for a good workflow.
Ideally we could create a similar experience to cbindgen
/ wasm-bindgen
, where all that's needed to create Android bindings is to slap an attribute on the thing we want to export.
This would require a good amount of work, but would be massively beneficial to the Rust ecosystem in general. Perhaps something like this exists already, but I haven't seen it yet.
Conclusion
Hopefully this sketches an idea of how to approach this. The story here is by no means smooth, but I think that it can be pulled off, and improved as progress is made. I'm super encouraging if you want to dive into Hypercore, and try and create C/Android bindings. It'd be super cool to see how far you could get! (I think quite far, actually!)
Is this something you would want to work on?
Further Reading
- Rust-Kotlin FFI Example
- Example Service for the above bindings
- android-rs-glue - seems discontinued, but might be a useful reference
from hypercore.
Hi,
I have a bit of experience using cbindgen to generate C headers for FFI and using then in Python, and I've been paying attention to this repo to check what would be the right time to have an FFI available for other languages (I really want dat working in Python!).
I don't have any Android/JNI experience, but maybe it would be viable to share some code with cbindgen/FFI?
from hypercore.
@bltavares: I looked into the JNA example and it seems pretty similar to what happens in Python and CFFI.
I keep pushing for FFI because the same interface can be used for any language that support FFI. For my use case there are solutions that are closer to Python (like PyO3), but I would like to give access to my lib to other languages too (like R or Julia) without having to write R-specific Rust bindings.
(BTW, sorry for hijacking the issue...)
from hypercore.
for sourmash-rust I've been keeping everything in the same crate, with an sourmash::ffi
module (and a sourmash::wasm
too). It is better to avoid putting FFI-specific things (like the #[no_mangle]
attribute) in the Rust code, and it was even recommended during RustConf
I think the FFI code should go in this repo, and then have separate Python and Java projects that use the FFI code. In the Python side you can use milksnake to make it easy to deal with setuptools and CFFI.
from hypercore.
Thanks for pointing out to the example. Looking further it seems to use JNA instead of JNI to avoid the complexity of making the functions match what the JVM expects as symbol names.
As I mentioned before, I don't have much experience with those and I had to search, coming to understand that JNA is slower than JNI, but I think it would be ok for this moment to be slow if it means sharing the C interface to kick of the FFI efforts.
I would like to experiment with either jni
or ffi
as a Java lib, and I'll probably put some time this week to get a hello world running.
(Sorry to be over-posting. I'm trying to coordinate efforts and get feedback as early on as possible to move to the right direction)
from hypercore.
Hi, just to keep everyone in the loop.
I was able to spend some time today fighting setting up a Gradle based project that uses a export "C" fn
from Rust.
It is very rough, but I'm already able to:
- Build a Java lib wrapping a
.so
- Use the Java lib from another Java CLI project
- Build an Android lib wrapping the Java lib
- This is mainly to be able to add Android specific dependencies and configs
- Compile the APK, but it crashes because it is missing the correct
x86_64-linux-android/libhypercore.so
, as I'm only packaging awin-x86-64/hypercore.dll
I'm currently developing on Windows, and cross-compilation is kind of a pain.
Next week I'm planning to use trust to build the desired .so
that will be wrapped.
Thinking about this, maybe the next steps for hypercore could be:
- Create a
hypercore::ffi
module (as already suggest) - Setup
trust
to build and publish each.so
for many platforms (Windows/Linux/Mac/Android/iOS) - Create a new project for each language binding (already proposed also), which uses the published bindings from GitHub Releases
Having the published .so
for all desired architectures would help a lot creating the bindings, specially as the tooling for cross-compilation is not easy to get right. What do you think?
from hypercore.
That is neat. I'll probably give it a try to use crossgen
and send a PR. First (next week probably), I'll fork the project and mess with the trust
template to see what would be an ideal setup and come back to this issue.
from hypercore.
I would like to help with this, certainly! But it is a type of work which I'm not used to, so I'm not sure how to take the first steps towards it.
The long haul approach would be quite powerful for the community as well, and I wonder if there would be more interest around this.
I'll read more about NDK and FFI on Rust, and see how much I can go forward, but I would love if other s could help as well, as this is quite new to me :)
from hypercore.
(To centralize information on the issue)
After a quick search, there are more projects related to Java/JNI that could help:
- https://github.com/sfackler/rust-jni-sys
- https://github.com/Geal/rust_on_mobile
- https://github.com/Dushistov/rust_swig
- https://github.com/Monnoroch/rust-jni
from hypercore.
@luizirber having a C API would be a fantastic first step. It'd be so good to allow Dat work with Python to allow people to run it in their data pipeline.
I'd be happy to help land any FFI patches needed -- if you'd like to start experimenting with this that would be fantastic! Any contributions to move the C API along would be great!
from hypercore.
Maybe we could make a meta-issue to track the implementation of the C-API.
If we could reimplement the nodejs API using the bindings, I think we would be able to use it for Python, Java and so on.
I was thinking more of the organization of it as I'm reading more about FFI, but please correct if I'm wrong:
- Create a
hypercore::ffi
module to expose the C interface without messing with the Rust-itc code - Create two new cargo proviles
ffi.release
andffi.debug
to produce thecdylib
s to be used - Expose the header files on the project as well
And for the projects:
- Create a new Python project that wraps around the
.so
(should this live in a new repo?) - Create a new Java project that wraps around the
.so
(should this live in a new repo?)- Create a Flutter plugin that consumes the Java project (this would be my end-goal)
Looking at the project, this seems to expose the main part of the Dat protocol, but not the dat cli. Should we also expose a C API for what would be the datrs
CLI, or each platform would also implement it's dat
API on top of hypercore's C API?
from hypercore.
Reading more about cargo
we wouldn't be able to override the build section using profiles, so probably we would need to create a new FFI project to integrate with cbindgen
and build.rs
.
Maybe using a workspace would be better if we want to keep the FFI part on the same project, but I'm not sure what is the best strategy for such kind of projects
from hypercore.
Considering this presentation it might not be possible to reuse the same C FFI for JNI integration for the Java part.
from hypercore.
@bltavares It looks like it should be possible (working production example of a C FFI targeting Android), but the JNI crate linked in the presentation looks really good too!
Perhaps it would be useful to try using jni
for the android bindings directly, and expose methods with ffi
. @bltavares would you like to experiment with using jni
?
@luizirber I like that approach a lot; thanks for linking all the resources too! -- a PR to setup an FFI structure would be an amazing start!
from hypercore.
Setup trust to build and publish each .so for many platforms (Windows/Linux/Mac/Android/iOS)
You can use crossgen
for this. Opened up an issue to target libraries specifically too yoshuawuyts/crossgen#11
from hypercore.
Found another, recently updated topic on internals: https://internals.rust-lang.org/t/idea-first-class-android-ndk-integration/5057
from hypercore.
Related Issues (20)
- Change stack from `failure` to `std::error::Error` HOT 2
- Feed.put with data yields weird results HOT 3
- Dynamic dispatch for storage? HOT 5
- Plan 0.12 release HOT 7
- Building latest version fails HOT 4
- Running benchmarks fails HOT 2
- Read key pair from disk HOT 3
- Upgrade sha2 to 0.8.x HOT 1
- Implement feed.audit() HOT 5
- Possible issue with signing data.
- Cannot declare a satisfiable feed reference HOT 2
- refactor binary interface types from `usize` to their specific sizes HOT 2
- Remove Feed Builder in favor of Feed simple create function HOT 7
- update random-access-* HOT 2
- README is incorrect about what the sample code prints HOT 1
- Dependabot can't resolve your Rust dependency files
- How do I read an existing hypercore on disk? HOT 3
- Networking: NOISE handshake and transport encryption HOT 7
- Async API on hypercore HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hypercore.