Failed to get final advertise address about olric HOT 14 CLOSED

Hello,

After a short investigation, I found that the error is produced by memberlist package. It may be caused by your operating system configuration.

Check out the code:
https://github.com/hashicorp/go-sockaddr/blob/master/ifaddr.go#L27
https://github.com/hashicorp/memberlist/blob/master/net_transport.go#L148

// GetPrivateIP returns a string with a single IP address that is part of RFC
// 6890 and has a default route. If the system can't determine its IP address
// or find an RFC 6890 IP address, an empty string will be returned instead.
// This function is the eval equivalent of:
//
// // $ sockaddr eval -r '{{GetPrivateInterfaces | attr "address"}}' ///

It seems that GetPrivateIP returns an empty string as IP for your case. I think this is pretty hard to debug on my side. Very likely, the problem is triggered by your host system configuration.

If you want to find the root cause of the problem, I can help you.

from olric.

@buraksezer Yep, the new consistent hashing algorithms is great, I have already read this paper. Thanks again for everything you've done! 👍

from olric.

@asdf2014 I'm glad to hear that. You feel free to ask any question or give advice about Olric. I have very limited spare time for this project but I'm willing to improve it constantly.

from olric.

Hello, I just checked the tests on master branch and there is no problem on my end. Please give more information about your environment. Output of go env may be useful.

Olric is developed and tested on Linux and macOS.

from olric.

@buraksezer Thank you for your attention. The following code block is the result of running the go env command.

$ go env

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/benedictjin/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/code/gopath"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4z/h5ftkmm90vg5s7j0kv65j0gw0000gp/T/go-build152216936=/tmp/go-build -gno-record-gcc-switches -fno-common"

from olric.

@buraksezer Thanks for the explanation. It finally worked after I removed IfByRFC("6890", privateIfs) from GetPrivateInterfaces method.

from olric.

@buraksezer May I ask another question about the architecture of Olric? How can we ensure that the backup partition dose not appear in the member that owns the primary partition? And why don't we let the consistent project covers this logic?

from olric.

@buraksezer With the growth of the Olric cluster, should I increase the value of ReplicationFactor? And will the increasing operation hurt the Consistency of cluster?

from olric.

Hi, @buraksezer . When I look at these test cases in dmap_backup_test.go, I think the two Olric instance db1 and db2 are bundled into a cluster pair by the peer. Is there any chance to bundle more Olric clusters together?

from olric.

@buraksezer May I ask another question about the architecture of Olric? How can we ensure that the backup partition dose not appear in the member that owns the primary partition? And why don't we let the consistent project covers this logic?

FSCK component is responsible maintaining cluster consistency. Unfortunately it has some missing parts. consistent package should provide a reliable key distribution mechanism between members. Olric calls GetClosestNForPartition function to find home for backups. That function finds partition's home firstly, then finds closest(suitable for backup) members for the partition.

Here is the code:
https://github.com/buraksezer/consistent/blob/master/consistent.go#L303

As I said before, FSCK is responsible to check inconsistencies and fix problems(including edge-cases) in cluster but It needs more effort to work properly.

from olric.

@buraksezer With the growth of the Olric cluster, should I increase the value of ReplicationFactor? And will the increasing operation hurt the Consistency of cluster?

ReplicationFactor is a configuration parameter of consistent package. It's not about backup count on Olric cluster. It's about key distribution and average key count on a member. According to consistent documentation: Members are replicated on consistent hash ring. This number means that a member how many times replicated on the ring.

Please note that consistent package doesn't aware about keys. Olric distributes partitions among members and keys are distributed among partitions by using modulo operator.

hkey = hash(key) 
partID = hkey % config.partitionCount

Cluster configuration cannot be changed after initialization. So you should design your cluster properly. If your requirements change along the way, you need to reset your cluster with the new configuration.

I need to test Olric with some fake data to know the exact effect of ReplicationFactor on operation. Now, the whole project is just experimental.

from olric.

Hi, @buraksezer . When I look at these test cases in dmap_backup_test.go, I think the two Olric instance db1 and db2 are bundled into a cluster pair by the peer. Is there any chance to bundle more Olric clusters together?

An Olric node needs to be seeded at initialization phase. Even if you have a cluster with 100 nodes, you just need to give address one of the nodes. It should discover the other nodes automatically. memberlist package provides this functionality.

Olric only supports static configuration at startup. In future, it may support more way to discover node.

https://github.com/buraksezer/olric/blob/master/config.go#L106

By the way, I designed Olric by reading Hazelcast's code and documentation. Their operations and deployment guide is a great way to learn about running distributed systems on production.

https://hazelcast.com/resources/hazelcast-deployment-operations-guide/

from olric.

@buraksezer Thanks a lot. One more thing is that the ReplicationFactor parameters of all of members are same for now. Is there a chance assign a different ReplicationFactor to each members, and then we can fix the heterogeneity problem of machine.

from olric.

Normally consistent hashing algorithms doesn't care about heterogeneity. In Olric, I use a different algorithm to overcome that problem. It's defined in a paper by Googlers.

https://arxiv.org/abs/1608.01350
https://ai.googleblog.com/2017/04/consistent-hashing-with-bounded-loads.html

In order to manage heterogeneity problem, you can use different combinations of ReplicationFactor and Load parameters in consistent package. The following sample code from consistent package can be very useful for experimenting:

https://github.com/buraksezer/consistent/blob/master/_examples/load_distribution.go

from olric.