libp2p / go-libp2p-daemon Goto Github PK
View Code? Open in Web Editor NEWa libp2p-backed daemon wrapping the functionalities of go-libp2p for use in other languages
License: Other
a libp2p-backed daemon wrapping the functionalities of go-libp2p for use in other languages
License: Other
I was trying to PUT_VALUE
with the value from GET_PUBLIC_KEY
, and the key of Multihash(value, sha2_256)
. I needed to convert the key bytes to string in advance because there will be error when we feed a non-UTF8 encoded bytes to the string field. E.g.
dht_req = p2pd_pb.DHTRequest(
type= p2pd_pb.DHTRequest.PUT_VALUE,
key=key_bytes, # error: should be unicode string, instead of non UTF-8 encoded bytes
value=value,
)
`ValueError: b'/pk/\x12 B\xbe\x80\x01\xe9\xc8\x15\xf6D\xa0W\x89\xb2\x86\xb1\xe0\x99\xd5U\xa6\xf9\x99\xc9\xe1\x0e\xd0\xcbhx\xfbt\xb4' has type str, but isn't valid UTF-8 encoding. Non-UTF-8 strings must be converted to unicode objects before being added.
So the code snippet which performs put_value
looks like below in python:
value = client.get_public_key(peer_id)
mh_digest = multihash.digest(value, algo)
mh_bytes = mh_digest.encode()
key_bytes_in_str = "".join([chr(i) for i in mh_bytes]) # convert bytes to string
key = "/pk/" + key_bytes_in_str
client.put_value(key, value)
And I got error from the daemon: connect failed. msg=key did not contain valid multihash: multihash length inconsistent
. I investigated for a while and think the problem should be the encoding in Python protobuf. The key
in unicode will still be encode('utf-8')
, when encoded by pbmsg.SerializeToString()
, and turned into a malformatted multihash with some bigger unicode encoded to utf-8, where the content and length are changed. Therefore the daemon receives a malformatted key.
I am trying to recover the malformatted key before it is sent, and therefore the issue might be solved. Possible solutions might be
encode('utf-8')
to encode('iso-8859-1')
in the serialization. But it seems nowhere to configure it in protobuf Python. Might possibly override something to achieve this.I wonder if it makes sense to change the key
in protobuf from string
to bytes
, if it is possibly composed of non-utf-8 encoded text like the result of hashes?
Thank you!
google.protobuf.text_encoding.CEscape(b, as_utf8=False)
, serialize the request PB, unescape the serialized bytes back to the unescaped string, turn it to bytes again with encode('iso-8859-1')
, and send the bytes. But errors still occurs when the protobuf tries to deserialize the bytes: [libprotobuf ERROR google/protobuf/wire_format_lite.cc:626] String field 'p2pd.pb.DHTRequest.key' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
Hi there! I am playing around with the daemon and started writing a client in Rust. I tried opening the unix socket and sent protobuf encoded messages (identify and find peer), but the daemon does not seem to respond at all. Is there some initialization procedure I am missing or something?
I assume it to work this way:
Should it work this way?
For debugging the state of the daemon, it could be useful to add a hook to catch SIGUSR2
(as we plan to use SIGUSR1
for config reload) that prints statistics/diagnostics to stdout/stderr.
We can output at least:
dht.RoutingTable.Print()
For now, the DHT routing table would suffice, as we actually do need it kind of immediately.
Are there instructions / examples? Brief search did not get results (I did not try to build it though).
This project is remarkably useful for basic interop testing between go-libp2p and js-libp2p, but the go-libp2p version is out of date so we can't test recent features like the wan/lan dual-dht, yamux, etc.
A great little starter task would be to update go-libp2p in this project.
We want to set up our custom network so that machines under NAT can still connect to each other. We tried providing our bootstrap nodes, but they didn't return the public address of our under-NAT machines.
Then we tried providing nodes from this go-libp2p example and encountered some strange behavior: when providing only one of the peers nothing works (either we can't connect, or the peer doesn't return the public IP), but when providing both everything works just fine.
What could be the issue? Could the issue be with how we configure our nodes? If yes, what options do we need to specify, so the peers return the public addresses of other nodes?
I'm thinking if it makes sense to let the daemon register a network notifiee which has both Connected
and Disconnected
exposed to the clients in some way(e.g. through the socket which the client is listening). It will allow the clients to be called back right after connected and disconnected events. This is not necessary because we can do it through polling with list_peers
.
The advantage I can think of now is we can perform handshaking or blacklist checking right after connecting, and be updated when peers have disconnected. The drawback is that it incurs additional complexity to do the callbacks in both the daemon and client.
Error with v0.2.3
and go version 1.12:
build github.com/libp2p/go-libp2p-daemon/p2p-keygen: cannot find module for path crypto/ed25519
Per readme, 1.12 should be supported. Worked with go1.13
Once we feel good that things have settled in our spec, we should migrate it over to the main specs repo.
I tried using this link, https://github.com/libp2p/go-libp2p-daemon/blob/master/specs/CONFIG.md, to test how to running configuration, but I got error message.
$ p2pd -f ./DO.json 2021/02/16 11:52:18 failed to parse multiaddr "\"/unix/tmp/p2pd.sock\"": must begin with /
How would I get this issue resolved? And is it possible to have something like the below in configuration file?
{
"ID": "Qma3GsJmB47xYuyahPZPSadh1avvxfyYQwk8R3UnFrQ6aP"
"PublicKey": 'xxxxxxxxx',
"PrivateKey" : "yyyyyyyyyy",
}
Thanks
This would be useful for the use case where the daemon is built as an .so. Would be compatible with Linux, osx, and windows. It would also remove the need to make a windows compatible shared memory implementation.
Could not build go-libp2p-daemon
using instructions on MacOS Travis CI.
https://travis-ci.org/status-im/nim-libp2p/jobs/477444788#L87-L378
It looks like problem is in go-reuseport
package.
https://travis-ci.org/status-im/nim-libp2p/jobs/477444788#L332-L339
In order to leverage the daemon for purposes of integration testing and the testbed more generally, we need to have remote-dialable daemons. I'd vote for abstracting the listen path into a multiaddr (similarly, client listen paths should be multiaddrs) and adding support to dial unix sockets to go-multiaddr-net.
Expose the Peerstore to allow daemon clients to access that information, such as known protocols a peer supports.
I'm proposing an addition of protos to the PeerInfo protobuf. This would enable us to expose the known protocols of a peer to consuming clients during the LIST_PEERS request.
Is it possible to reset a stream which we get from STREAM_OPEN
or STREAM_HANDLER
? I scanned the codebase and couldn't find it out. It seems we can only close
the stream for now.
I am writing the interop tests for the DHT between js and go and I am having an issue with the keys.
Using js-ipfs
and go-ipfs
daemon, we are able to put / get any type of keys, while it doesn't seem to be the case with go-libp2p
. In this context, I think that we should be able to set custom validators/selectors as we are able to do when using the DHT module. Moreover, regarding testing the interop between get
and put
will be more complex since we would have to use a valid namespace such as ipns
and put a valid record.
Example in go tests for custom validators/selectors: libp2p/go-libp2p-kad-dht/dht_test.go#L51
Opening this issue to continue the discussion we started regarding what concurrency models that should be supported when targeting language level bindings.
Concurrency handled on the Go client side. Async calls from Java/Python/C etc that require a response are tracked with an id. There are two ways we could handle this:
1) Java/Python/C could poll the Go client until the matching response id is returned
2) Java/Python/C could provide a function pointer and the Go client could issue a callback when the response is ready.
I prefer option 2 here
There might be cases where we want to provide a synchronous mode of interaction. This might be useful for simulations where we want things to happen during each timestep for the purpose of research and reporting the health of the network given different simulated pathological scenarios.
Just a thought. I am curious what you guys think
Just wondering why Daemon interface operates with byte representation of multiaddress?
The interface appears pretty highlevel and the string representation would look more appropriate
p2pd output on start is:
Control socket: /tmp/p2pd.sock
Peer ID: QmcvboYSXDy3QBGsk21HbNdiXssq7S4EgRAMyPPAXmFCr1
Peer Addrs:
/p2p-circuit
/ip4/127.0.0.1/tcp/37091
/ip4/192.168.58.131/tcp/37091
/ip6/::1/tcp/45191
IDENTIFY
method response in binary form
08 00 22 52 0A 22 12 20 D8 B6 14 0E 76 4F 49 7A
0A F9 E1 C3 C5 88 E9 B0 5D 41 44 5A F9 6C 1E 8A
CC CB D2 86 3C DA AE 96 12 14 29 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 01 06 B0 87 12 02
A2 02 12 08 04 7F 00 00 01 06 90 E3 12 08 04 C0
A8 3A 83 06 90 E3
Peer ID in binary form
12 20 D8 B6 14 0E 76 4F 49 7A 0A F9 E1 C3 C5 88
E9 B0 5D 41 44 5A F9 6C 1E 8A CC CB D2 86 3C DA
AE 96
IPv6 address
29 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01 06 B0 87
Unknown address, i could not find any information about this type of address, could you please explain it?
A2 02
IPv4 address
04 7F 00 00 01 06 90 E3
IPv4 address
04 C0 A8 3A 83 06 90 E3
At the moment, the client application can not find out when the bootstrap process is completed enough to execute such commands as PROVIDE
or FIND_PROVIDERS
.
In such case it is possible to get error from daemon failed to find any peer in table
.
I understand that bootstrap process is working in the background. Is it possible to block result of PROVIDE
until DHT will be able to perform this request. Because client application is unable to know when its safe to call PROVIDE
.
CONNECT
request will stuck if go-libp2p-daemon
could not connect to peer, so i think some kind of timeout
required.
It seems like application-provided validator functions for pubsub are not implemented yet. I think we had discussed this is something want. Ethereum 2.0 will likely use it.
We've officially deprecated gx, time to move on to a standard go mod build!
Failure during go get ./...
in HEAD, on AppVeyor - https://ci.appveyor.com/project/nimbus/nim-libp2p/builds/27746839/job/dnpugbq12b2t298q#L579 :
# github.com/prometheus/client_golang/prometheus
..\goblin\pkg\mod\github.com\prometheus\[email protected]\prometheus\process_collector_windows.go:78:9: assignment mismatch: 2 variables but "golang.org/x/sys/windows".GetCurrentProcess returns 1 values
Upstream issue: prometheus/client_golang#642
This would be useful for many ongoing research efforts. Adding an option for QUIC would potentially get more teams testing/contributing to the daemon.
As the daemon grows our configuration flag options are likely to become cumbersome pretty quickly. I was talking with @bigs and @vasco-santos about needing to add support for specifying the multiaddrs the daemons libp2p node listens on. We could, and probably should, add a comma delimited string flag similar to how we add bootstrap addresses. Over time though, reading through all of the flags you have enabled for a node could get overwhelming.
I'd like to propose adding support for piping a config file to the daemon. For example:
$ cat config.json | p2p daemon
This would also have the nice benefit of being able to have node type configurations pre made for users; DHT booster, relay, rendezvous, etc.
If this sounds good, I can write up a spec for the json configuration.
As users demand more functionalities (e.g. #34), we're likely to see a proliferation of option flags for the daemon binary. While the daemon is marked as an experimental project, we should attempt to bring in structure to our option flags. Let's agree on an approach and lock it in.
Proposal:
go test
uses no casing or separators, and ipfs
uses hyphens. My proposal is to use hyphens.For transports, this can look as follows:
-tpt-enable-quic enables the QUIC transport
-tpt-enable-ws enables the WebSockets transport
-tpt-enable-tcp enables the TCP transport
-tpt-disable-defaults disables the default transports
So if I wanted to enable ONLY QUIC, I'd do the following:
p2pd -tpt-disable-defaults -tpt-enable-quic
WDYT? CC @vyzo @bigs @jrhea @cheatfate @mhchia
We use the listen
flag to control socket. There's also a noListenAddrs
which specifies "don't listen on any libp2p transports". There's also an option hostAddrs
which specifies which libp2p transports/addresses we should listen on.
We should change this to:
-control
, or -controlSocket
, etc. for the control address.listenAddrs
and noListenAddrs
for libp2p listen addresses.I see here present connect(peer, addresses)
and recently added feature trim()
(which drops connections to all connected peers), but i don't see any ability to disconnect from particular peer. Is this feature is missing or not needed?
Now that we officially deprecated secio as a connection encryption module for libp2p, we should remove it from the daemon. Making noise used by default and remove the option to use secio/noise at all.
Currently, we are running interop tests for libp2p using secio unless explicitly setting noise, which is not desirable.
We should this together with libp2p/js-libp2p-daemon#43 to update the interop tests to use the new daemons with noise
Opening a thread for the discussion of the control API for our daemon.
Two solid options:
Editor's note: I think we can get away by polling the filesystem/shared memory where our streams are created as opposed to polling the control API, which would make things simpler.
For example with CIRCUIT transport , the convertStreamInfo return error("no protocol with code 290"). Now I fix it by import circuit packge in client process sapce.
how about if daemon process return the string of addr and protocol to client?
p2pd -version
now prints a generic help, but it's impossible to know what version of the source code it was built from - preferably git hash + tagged release or something would be nice.
What license will this be released under? MIT, Apache 2? Do you mind adding that to the repo?
During devcon4 workshops, somebody suggested to use Named Pipes in Windows, as the Unix domain socket support is experimental. Suggested steps:
This should be easy once #24 is done, and would be really useful for things like peer-base/peer-pad#285.
go get ./...
go get: warning: modules disabled by GO111MODULE=auto in GOPATH/src;
ignoring go.mod;
see 'go help modules'
# github.com/libp2p/go-libp2p-quic-transport
../go-libp2p-quic-transport/stream.go:15:31: s.Stream.CancelRead(0) used as value
../go-libp2p-quic-transport/stream.go:18:29: s.Stream.CancelWrite(0) used as value
../go-libp2p-quic-transport/transport.go:22:62: undefined: quic.VersionMilestone0_10_0
The clean exit logic is not so clean after all, it panics on systemd shutdown:
May 03 08:51:16 ip-172-31-45-199 systemd[1]: Stopping libp2p relay...
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: 08:51:16.786 ERROR p2pd: error accepting connection: accept unix /var/run/p2pd/p2pd.sock: use of closed network connection daemon.go:142
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: 08:51:16.786 ERROR p2pd: error accepting connection: accept unix /var/run/p2pd/p2pd.sock: use of closed network connection daemon.go:142
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: panic: runtime error: invalid memory address or nil pointer dereference
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9e4287]
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: goroutine 2019 [running]:
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: github.com/libp2p/go-libp2p-daemon.(*Daemon).handleConn(0xc00013a540, 0x0, 0x0)
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: /home/ubuntu/go/src/github.com/libp2p/go-libp2p-daemon/conn.go:22 +0x37
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: created by github.com/libp2p/go-libp2p-daemon.(*Daemon).listen
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: /home/ubuntu/go/src/github.com/libp2p/go-libp2p-daemon/daemon.go:146 +0x98
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: panic: runtime error: invalid memory address or nil pointer dereference
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9e4287]
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: goroutine 2023 [running]:
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: github.com/libp2p/go-libp2p-daemon.(*Daemon).handleConn(0xc00013a540, 0x0, 0x0)
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: /home/ubuntu/go/src/github.com/libp2p/go-libp2p-daemon/conn.go:22 +0x37
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: created by github.com/libp2p/go-libp2p-daemon.(*Daemon).listen
May 03 08:51:16 ip-172-31-45-199 relay.sh[14745]: /home/ubuntu/go/src/github.com/libp2p/go-libp2p-daemon/daemon.go:146 +0x98
May 03 08:51:16 ip-172-31-45-199 systemd[1]: relay.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 03 08:51:16 ip-172-31-45-199 systemd[1]: relay.service: Failed with result 'exit-code'.
Why do we need a libp2p daemon and what do we want it to do?
In particular, daemons should clean up any unix sockets it may have created after trapping SIGINT.
Now that we have multiaddr support for control sockets, which allows us to use tcp, we should finally fix the tests on windows.
Tired of those red Xs!
I'm at d161fe580fc98d62a8ba961df5ac0e70a76fc71d
.
After go build
inside /p2pd
and then:
./p2pd -pubsub=true -pubsubRouter=gossipsub
I get this:
Control socket: /unix/tmp/p2pd.sock
Peer ID: QmPi2mpZuaq2w7ZDQxXXnGr4WwJo7tv8eoLs64roVFyZ9N
Peer Addrs:
panic: non-positive interval for NewTicker
goroutine 32 [running]:
time.NewTicker(0x0, 0xc00007dea0)
/go/1.12.1/libexec/src/time/tick.go:23 +0x190
github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).heartbeatTimer(0xc000048180)
/go/pkg/mod/github.com/libp2p/[email protected]/gossipsub.go:365 +0x14a
created by github.com/libp2p/go-libp2p-pubsub.(*GossipSubRouter).Attach
/go/pkg/mod/github.com/libp2p/[email protected]/gossipsub.go:76 +0x52
The README describes using go get to install. That method will require Go 1.11 since go-reuseport added Windows support and uses the net Control API. See #56.
In most of the cases daemon usage is fine, but sometimes your project could not spawn actual daemon executable. Is it possible to wrap the daemon inside of simple shared library, this library can export for example just two functions start(configuration)
and stop()
.
So when application loads shared library and calls to start(configuration)
daemon's main()
function will be executed in new thread (or in any other possible ways). In such case all the named sockets created will be inside of application, so it can be possible to use daemon in iOS (for example).
This is of course not priority issue, but it would be very handy to have such feature.
There's a "spec" in specs/CONFIG.md but it doesn't actually say how to use a config.
There's also a useless "go read the go docs" comment in the README.
We need to replace the Usage section in the README with a short section on how to get started:
Currently daemon
is not return any error/data to client
when CONNECT/STREAM_OPEN timeout is exceeded, so client
can stuck on waiting data from daemon
.
> ./p2pd
2021/03/15 11:40:50 if we explicitly specify a transport, we must also explicitly specify the listen addrs
Following the instructions doesn't work on MacOS with go1.13:
git clone https://github.com/libp2p/go-libp2p-daemon
cd go-libp2p-daemon/
git checkout c4c3f318
go get ./...
go: finding golang.org/x/text v0.3.0
go: finding github.com/jackpal/go-nat-pmp v1.0.1
go: finding github.com/koron/go-ssdp v0.0.0-20180514024734-4a0ed625a78b
go: finding golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522
# golang.org/x/xerrors
../../../go/pkg/mod/golang.org/x/[email protected]/adaptor_go1_13.go:16:14: undefined: errors.Frame
../../../go/pkg/mod/golang.org/x/[email protected]/format_go1_13.go:12:18: undefined: errors.Formatter
Upstream: golang/go#32246
and workaround: golang/go#32246 (comment)
golang.org/x/xerrors v0.0.0-20190528162220-0421b64034aa
We talked about this in Devcon4 workshops to add version support to IPC protocol.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.