pravahio / go-mesh Goto Github PK

View Code? Open in Web Editor NEW

24.0 24.0 5.0 14.67 MB

Realtime data exchange platform for Smart Cities

Home Page: https://pravah.io

License: MIT License

Go 99.02% Makefile 0.94% Dockerfile 0.04%

data realtime streaming-data

go-mesh's People

Contributors

Stargazers

Watchers

Forkers

jackbekket massimiliano76 bonedaddy nsashi amit2011

go-mesh's Issues

Account compatibility between RemoteAccess and Libp2p

There are two account right now, one used for libp2p and other one for RemoteAccess (ethereum for now). As these two accounts could be generated differently (RSA vs ECC) we need a way to make them compatible and future proof.

Proposal

Keep them independent of each other even if they are generated from the same generator. This would mean two key space one for libp2p and the other one for RemoteAccess.

Pros	Cons
New RA implementations are easy to adopt	Would have to export two account
Would be totally independent of each other hence no compatibility issues

Integrate AutoNAT to discover dialability of a node

bootstrapping leads to no connection due to many dead peers

During testing the bootstrap node is populated with many dead peers. As a result we don't get a healthy peer to connect too.

23:09:22.224 DEBUG svc-bootst: Found peer: {QmPmAGEYy3WbC6q7MzGkoXt9qXdv7PjjChrFCgyVY8VyPF: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmNv5MX4o4YAqsUSZiqx9rVcaroVa6KTcAjkuDGYwTsYPK: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmeLepdbZoAVL4SbMJhgSLwutH9H1BV96zmGGwbvzmTPSb: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmbSXsYy2JpxgZZzqH6MSkAH1jxvSrAyoZNJD7Hn7f9f3P: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmcTNbyvYWa6Ui8EXku2TgXBgqauK1dx3wVNUGeZ2PdisM: []} connection.go:81
23:09:22.224 DEBUG svc-bootst: Found peer: {QmZaXMgBa494QZY5XuMSL7mnKYdRyuZzaJSQsuU3bdqF85: []} connection.go:81
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmbSXsYy2JpxgZZzqH6MSkAH1jxvSrAyoZNJD7Hn7f9f3P: no addresses connection.go:87
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmNv5MX4o4YAqsUSZiqx9rVcaroVa6KTcAjkuDGYwTsYPK: no addresses connection.go:87
23:09:22.224 WARNI svc-bootst: Connection failed: failed to dial QmeLepdbZoAVL4SbMJhgSLwutH9H1BV96zmGGwbvzmTPSb: no addresses connection.go:87
23:09:22.224 WARNI svc-bootst: Connection failed: failed to dial QmPmAGEYy3WbC6q7MzGkoXt9qXdv7PjjChrFCgyVY8VyPF: no addresses connection.go:87
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmcTNbyvYWa6Ui8EXku2TgXBgqauK1dx3wVNUGeZ2PdisM: no addresses connection.go:87
23:09:22.225 DEBUG svc-bootst: Found peer: {QmVzDFh9iRh5g1DvYW5FCrd1DMETveUuR4HswCxEsAqGhu: []} connection.go:81
23:09:22.225 DEBUG svc-bootst: Found peer: {QmbWMvkxknPhYjNAJB4jtftx36Co5RqHUFfBi9Uk89N7aL: []} connection.go:81
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmZaXMgBa494QZY5XuMSL7mnKYdRyuZzaJSQsuU3bdqF85: no addresses connection.go:87
23:09:22.225 DEBUG svc-bootst: Found peer: {QmU9TwVrjqwvFdXsAa2QGxRvGVWZwXnuJpng7nVf6hiFmW: []} connection.go:81
23:09:22.225 DEBUG svc-bootst: Found peer: {Qmajpxt5vNAVAUTcmLG26zrYMXD45URDo7EMZY8MNVkJDS: []} connection.go:81
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmVzDFh9iRh5g1DvYW5FCrd1DMETveUuR4HswCxEsAqGhu: no addresses connection.go:87
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmbWMvkxknPhYjNAJB4jtftx36Co5RqHUFfBi9Uk89N7aL: no addresses connection.go:87
23:09:22.225 WARNI svc-bootst: Connection failed: failed to dial QmU9TwVrjqwvFdXsAa2QGxRvGVWZwXnuJpng7nVf6hiFmW: no addresses connection.go:87
23:09:22.225 DEBUG svc-bootst: Found peer: {QmXfDRfnWjqGYD5ZQ8iiAjsDYBibtBsPCA1CJew9KMg8kc: []} connection.go:81
23:09:22.225 DEBUG svc-bootst: Found peer: {QmTUcnRvfaf15G5uouskCrqiGGtRQKkn4eZ8NwQwHuU2Ni: []} connection.go:81
23:09:22.226 DEBUG svc-bootst: Found peer: {QmabH7BsDANPX3ghHwCnKuxFXLJF3fv7bBuR2ZTb4z2HPw: []} connection.go:81
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial Qmajpxt5vNAVAUTcmLG26zrYMXD45URDo7EMZY8MNVkJDS: no addresses connection.go:87
23:09:22.226 DEBUG svc-bootst: Found peer: {QmP7GrwHUNbBrW8FjMvZ9MusCUWScz1VtKqd4ohbCu1rim: []} connection.go:81
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial QmXfDRfnWjqGYD5ZQ8iiAjsDYBibtBsPCA1CJew9KMg8kc: no addresses connection.go:87
23:09:22.226 DEBUG svc-bootst: Found peer: {Qmbu4H9uVacrz2L5PRBEQkeQsS7TrZZHZBRPGtNMGKT1Ti: []} connection.go:81
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial QmTUcnRvfaf15G5uouskCrqiGGtRQKkn4eZ8NwQwHuU2Ni: no addresses connection.go:87
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial QmabH7BsDANPX3ghHwCnKuxFXLJF3fv7bBuR2ZTb4z2HPw: no addresses connection.go:87
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial QmP7GrwHUNbBrW8FjMvZ9MusCUWScz1VtKqd4ohbCu1rim: no addresses connection.go:87
23:09:22.226 WARNI svc-bootst: Connection failed: failed to dial Qmbu4H9uVacrz2L5PRBEQkeQsS7TrZZHZBRPGtNMGKT1Ti: no addresses connection.go:87

Solution
Auto Bootstrap.

Auto bootstrap does not happen

Bootstrapping only happens once during initialisation. Peer should automatically connect to new peers during the entire lifetime. Use go-libp2p-bootstrap.

update readme to include new py-pravah code

rename RemoteAccess to something more meaningful

RemoteAccess is misleading and should be changed to something more intuitive.

It is primarily an authentication remote server where one registers for subscription and publication.

Auth
RemoteAuth

Nice Readme with docs

Support multiple transport protocol

Currently go-mesh only support QUIC which is not yet finalised and ready for production. If we update QUIC almost all the running nodes are deprecated and needs to be updated.

Nodes with older and new QUIC can't connect and hence it bifurcate the network (which is frustrating)

Possible Solution: Support TCP based protocol along with QUIC (Any version). QUIC will be used for primary communication but during an update old nodes can switch to TCP and can later be updated to newer QUIC.

Support for raw JSON in the message

publish needs a protobuf serialiser with predefined data standard before sending the data to the wire.

There are multiple use cases where dynamic data standards are required and JSON could be a good way of sending this kind of data.

Challenges:

How will the end user distinguish between JSON and Protobuf message?
Deserialisation would be an issue if format is not known before. Need to identify it somehow.

mandate subscribers to register to the remote server

Currently subscribers don't have to register themselves with the remote server. This means we could not track them for how much data they consumed. It should be mandatory for them to register to the remote server before starting to consume data.

"Found peer" is without any public IP address

What's happening: bootstrap dht resolves the new peer but only gives private IPs which are undialable.

What's expected: bootstrap node should preserve the public IP as well and Found Peer should reflect that as well.

What's breaking: Given we only have private IPs, nodes won't be able to connect with each other. Subscribers can't connect with publishers which results in no data transmission.

Logs:

Found peer: {QmSnyucQUCbAjNEjoaLZtyhzTEB7Hms7Vn6jxWFhQdLbFN: [/ip4/127.0.0.1/udp/5502/quic /ip4/172.31.43.146/udp/5502/quic]}

go routine per topic should be terminated if that topic is unsubscribed

Currently, subscriber service has a global context. We would need a local context per topic which could be used to terminate go routines working on each topic subscriber.

go-mesh/service/subscriber/api.go

Lines 48 to 65 in cd97627

 go func(subs *pubsub.Subscription) { 

 for { 

 select { 

 case <-subService.ctx.Done(): 

 close(msg) 

 return 

 default: 

 } 

 m, err := subs.Next(subService.ctx) 

 if err != nil { 

 continue 

 } 

 msg <- m 

 } 

 }(sub)

Wrong error if config.json is not found

If config.json is missing when mesh -c config.json --en-sub is executed it throws

13:56:11.612 ERROR   mesh-cli: Need a rendezvous string main.go:109

instead it should say that config.json not found.

Enable cors for COVID-19 API

🙇‍♂️ Thank you for creating this API. Been wanting to play with the data using observable notebooks, but looks like cors is not enabled:

Test: https://observablehq.com/d/894750d4be37b21a

Enabling cross origin requests for the APIs will allow it to be used more widely in any web apps 🙏

unsubscribe is not supported

This would include:

Unregistering from the remote access (blockchain)
Close the subscription channel

	go func(subs *pubsub.Subscription) {
	for {
	select {
	case <-subService.ctx.Done():
	close(msg)
	return
	default:
	}

	m, err := subs.Next(subService.ctx)
	if err != nil {
	continue
	}

	msg <- m
	}

	}(sub)