docker-archive / classicswarm Goto Github PK
View Code? Open in Web Editor NEWSwarm Classic: a container clustering system. Not to be confused with Docker Swarm which is at https://github.com/docker/swarmkit
License: Apache License 2.0
Swarm Classic: a container clustering system. Not to be confused with Docker Swarm which is at https://github.com/docker/swarmkit
License: Apache License 2.0
Hi,
Running the following installation procedures, I am running on error:
Building my vagrant machine
~/dev/docker_swarm> vagrant init ubuntu/trusty64
~/dev/docker_swarm> wget https://master.dockerproject.com/linux/amd64/docker-1.3.2-dev
~/dev/docker_swarm> vagrant up
~/dev/docker_swarm> vagrant ssh
From inside the vagrant machine
vagrant@vagrant-ubuntu-trusty-64:
$ sudo apt-get install golang-go git -y && sudo -E go get github.com/docker/swarm && sudo -E go install github.com/docker/swarm$ sudo /vagrant/docker-1.3.2-dev -H 0.0.0.0:2375 -d &
vagrant@vagrant-ubuntu-trusty-64:
vagrant@vagrant-ubuntu-trusty-64:$ swarm create$ swarm join --token=6f1104155948d4163cae6d316502af55 --addr=127.0.0.1:4243
6f1104155948d4163cae6d316502af55
vagrant@vagrant-ubuntu-trusty-64:
vagrant@vagrant-ubuntu-trusty-64:~$ swarm manage --token=6f1104155948d4163cae6d316502af55
And now the error:
INFO[0008] GET /v1.15/info
ERRO[0008] Node http://127.0.0.1:4243 is running an unsupported version of Docker Engine. Please upgrade.
see moby/moby#6962
/cc @SvenDowideit
Every time I run swarmd
it prints the following line to standard output:
2014/06/17 17:22:12 Couldn't automatically load credentials
It seems to come from the Tutum backend, specifically from a library pulled in as dependency by that backend. @fermayo could you fix it to avoid the pollution please?
Thanks.
Do we have a roadmap?
If I can see that I will have:
When I try to install libswarm I get:
$ go install github.com/docker/libswarm/swarmd
# github.com/docker/libswarm/backends
go/src/github.com/docker/libswarm/backends/ec2.go:345: c.instance.IPAddress undefined (type *ec2.Instance has no field or method IPAddress)
go/src/github.com/docker/libswarm/backends/ec2.go:436: c.instance.IPAddress undefined (type *ec2.Instance has no field or method IPAddress)
Currently many endpoints have a notImplementedHandler
with a message that the endpoint is not supported in clustering mode. Found this out trying to do a swarmed image pull
, which would have been much nicer than pull
in a loop.
Is this a case of "not implemented yet" or "swarm will not implement these APIs"?
If the latter, is the intended way to (for example) update images across the cluster to manually hit all the daemons?
Using @vieux's recommendation in #145, I'm attempting to use swarm manage --addr=<swarm listener> <dockerport1> <dockerport2>
, but it always fails with:
ERRO[0000] node was already added to the cluster
Tracking this error message down to here, I see it's using the Node
's ID
property to check if it's been added already. Tracing that to samalba/dockerclient/types.go, it looks like it populates the Node object by parsing the JSON response of issuing /info
... which has no ID field. This leads me to believe it is a blank string, and therefore you can never have more than one trying to use swarm this way.
I am not a Go developer, so I'm probably missing things here, but figured I'd bring this up just in case.
I've followed these steps:
without error. But on
I am seeing
(As you can see, I'm running the docker daemon without "--tlsverify". This is a test launch.)
I guess this is because I am making bad choices for swarm_ip and swarm_port (and maybe node_ip and node_port also). Googling for "no Host in request URL" doesn't look pretty.
Can anyone tell me what I need to change?
Thanks
When creating and managing multiple machines, it is confusing to identify which machine you have logged into after using machine ssh
. Setting the hostname to match the machine name would be useful in this scenario.
is-mbp-andrew:libswarm andrew$ go version
go version go1.3.1 darwin/386
is-mbp-andrew:libswarm andrew$ go build
# github.com/docker/libswarm
./message.go:50: s.Sender.CreateNestedReceiver undefined (type libchan.Sender has no field or method CreateNestedReceiver)
./message.go:65: s.Sender.CreateByteStream undefined (type libchan.Sender has no field or method CreateByteStream)
./message.go:82: cannot use imsg (type *internalMessage) as type *libchan.Message in argument to s.Sender.Send
./message.go:82: multiple-value s.Sender.Send() in single-value context
./message.go:95: cannot use imsg (type *internalMessage) as type int in argument to r.Receiver.Receive
./message.go:95: multiple-value r.Receiver.Receive() in single-value context
is-mbp-andrew:libswarm andrew$
The Watch
method of the discovery backend calls cluster.UpdateNodes
method to update the node list according the return value of the Fetch
method of the discovery service.
But it only inserts the nodes that aren't exist in the node list and doesn't touch the nodes that are "removed" from the discovery service.
If a node is removed from the discovery service, the swarm list
command returns the correct result because it calls the Fetch
method of the discovery service directly. But in fact the removed node is still in the node list of the cluster.
I have been thinking about cluster configuration lately.
My question is; does it make sense to keep some mutable state inside a container? If so, does it make sense to design a standard interface to state controls?
Example:
If we look at containers as lego bricks. You can either build a house solely out of stateless bricks. Whenever you want to switch on the lights, you swap out the dark lightbulb bricks with glowing lightbulb bricks. On the other hand, those lightbulb bricks might expose a switch that lets you turn them on and off.
What does the community think? Just wondering ๐ฑ
Hi, i have 3 machines running docker and docker-swarm, 2 with ubuntu server 14.04 and 1 with ubuntu desktop 14.04. The docker and swarm have the same version in all machines, docker 1.4.1 and docker-swarm 0.0.1.
I create a cluster with "swarm create" and start swarm join in 2 machines running ubuntu server, all without errors. When i start swarm manage in ubuntu desktop it's running ok, but when i try execute any command(ps, run, info...) i receive this error message:
"ERRO[0014] unable to find a node with port available"
Using the docker client i can access all machines from your ip:port and running commands directly on docker client works fine too. Someone know what is happening?
The addressing of servers is rather inconsistent, especially between docker & swarm.
tcp://1.2.3.4:2375
(here)1.2.3.4:2375
(here).swarm list
and docker info
commands output the format `http://1.2.3.4:2375http://
protocol prefixes, only tcp://
.tcp://1.2.3.4:2375
(and http://...
) is not a host (is named as such in $SWARM_HOST
and $DOCKER_HOST
). The format 1.2.3.4:2375
is not a host either. A host, or hostname, does not include protocol or port.swarm join
command uses --addr
for specifying the IP & PORTswarm manage
command uses --host
for specifying the IP & PORTAs you can see, the naming is all over the place (and this is just from a brief look as well). Half these are URLs & URIs, the other half might be considered addresses, and none of them are hosts, but most are labeled as such.
While I personally would prefer we not call things 'hosts' that aren't hosts, it's a little late to fix this naming issue. However I do think all this needs to be made consistent. The documentation can be fixed at any time, and the swarm code can be fixed before it is labeled as 'stable'.
I searched the code but I couldn't find any references to the links feature. What is swarm's networking model? This should be clearly defined to be able to build a distributed system.
Perhaps this is just a documentation bug.
The repository doesn't have a travis build.
We should add one.
What are the steps to build and test libswarm?
Trying to build swarm. Seems to fail on Dockerserver + Rax
[sg@whatup ~]# go get github.com/docker/libswarm/...
# github.com/docker/libswarm/backends
mygo/src/github.com/docker/libswarm/backends/dockerserver.go:429: cannot use "github.com/dotcloud/docker/api".APIVERSION (type "github.com/docker/docker/pkg/version".Version) as type "github.com/dotcloud/docker/pkg/version".Version in assignment
mygo/src/github.com/docker/libswarm/backends/rax.go:290: too few values in struct initializer
mygo/src/github.com/docker/libswarm/backends/rax.go:291: too few values in struct initializer
mygo/src/github.com/docker/libswarm/backends/rax.go:292: too few values in struct initializer
mygo/src/github.com/docker/libswarm/backends/rax.go:293: too few values in struct initializer
mygo/src/github.com/docker/libswarm/backends/rax.go:294: too few values in struct initializer
mygo/src/github.com/docker/libswarm/backends/rax.go:295: too few values in struct initializer
mygo/src/github.com/docker/libswarm/backends/rax.go:296: too few values in struct initializer
mygo/src/github.com/docker/libswarm/backends/rax.go:376: cannot use job (type *"github.com/dotcloud/docker/engine".Job) as type *"github.com/docker/docker/engine".Job in function argument
mygo/src/github.com/docker/libswarm/backends/rax.go:477: cannot use job (type *"github.com/dotcloud/docker/engine".Job) as type *"github.com/docker/docker/engine".Job in function argument
mygo/src/github.com/docker/libswarm/backends/rax.go:477: too many errors
Using Go version 1.2.2
Looks like TLS cert authentication is missing for both front and back end connections.
I presume this would be added with flags mirroring those of docker
?
On Ubuntu 14.04 immediately after installing:
root@docker-swarm-1:~# swarm create
unexpected fault address 0x7f2800000011
fatal error: fault
[signal 0xb code=0x1 addr=0x7f2800000011]
goroutine 3 [running]:
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
:0
goroutine 1 [select]:
github.com_docker_swarm_discovery_token.CreateCluster.pN61_github.com_docker_swarm_discovery_token.TokenDiscoveryService
/root/go/src/github.com/docker/swarm/discovery/token/token.go:85
main.$nested1
/root/go/src/github.com/docker/swarm/main.go:47
github.com_codegangsta_cli.Run.N34_github.com_codegangsta_cli.Command
/root/go/src/github.com/codegangsta/cli/command.go:101
github.com_codegangsta_cli.Run.pN30_github.com_codegangsta_cli.App
/root/go/src/github.com/codegangsta/cli/app.go:154
main.main
/root/go/src/github.com/docker/swarm/main.go:98
in some instances its just a "Segmentation fault (core dumped)"
I spun up a new VM on Digital Ocean via applications > docker 1.4.0, ssh'd in and ran this before hand so its a clean install, plenty of cpu and ram:
apt-get install gccgo-go
export GOPATH=/root/go/
go get -u github.com/docker/swarm
cp /root/go/bin/swarm /usr/bin/
The strangest thing about this is that if I give it some time it doesn't happen any more. I gave it a few minutes and the command successfully executed.
Executing the go get -u github.com/docker/swarm
command again causes the problem to occur again for some small but significant amount of time. Using either the new binary or the one in /usr/bin which isn't too surprising.
It's probably a bit to late now, but it should have been named "Shoal" which is a large number of fish swimming together :-)
In order to consume the swarm manage API using javascript, we need to enable CORS. It looks like it's already there but api.go has a hard coded false for enableCors
on line 360. Could this be made into a flag?
i tried running libswarm using following command
$ swarmd 'dockerserver tcp://localhost:4243' 'dockerclient tcp://localhost:4243'
Then tried to connect to the running server using following
$ docker -H localhost:4243 ps
which throws follwing error
https://gist.github.com/ric03uec/999f82756285d6657991
Tried restarting the server(as suggested by @cpuguy83 on irc) but didn't help.
System info
$ uname -a
Linux crossbow 3.8.0-35-generic #50~precise1-Ubuntu SMP Wed Dec 4 17:25:51 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f
$ docker -D info
Containers: 12
Images: 359
Storage Driver: aufs
Root Dir: /data/docker/aufs
Dirs: 385
Execution Driver: native-0.2
Kernel Version: 3.8.0-35-generic
Operating System: Ubuntu precise (12.04.4 LTS)
Debug mode (server): true
Debug mode (client): true
Fds: 61
Goroutines: 52
EventsListeners: 0
Init Path: /usr/bin/docker
Username: ric03uec
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
It would be very useful for my use case to run the swarm manage
server on a unix socket, without binding any ports at all (e.g. the case of running swarm
in a networkless container).
This also would also bring swarm
closer to parity with the docker daemon functionality.
When using swarm with TLS I get the error: " x509: certificate signed by unknown authority " when using "docker start" and it also happens with "docker logs".
The problem is that proxy function in utils.go doesn't use the tlsConfig.
This is what is missing is to configure the tls transport in the proxy function:
httpTransport := &http.Transport{
TLSClientConfig: tlsConfig,
}
client := &http.Client{Transport: httpTransport}
After making these changes "docker start" worked correctly.
api/api.go
first I needed to relay cluster variable to proxy function
func proxyContainer(c *context, w http.ResponseWriter, r *http.Request) {
container, err := getContainerFromVars(c, mux.Vars(r))
if err != nil {
httpError(w, err.Error(), http.StatusNotFound)
return
}
if err := proxy(c.cluster, container, w, r); err != nil {
httpError(w, err.Error(), http.StatusInternalServerError)
}
}
api/utils.go
from the cluster variable I read the TlsConfig
func proxy(c *cluster.Cluster, container *cluster.Container, w http.ResponseWriter, r *http.Request) error {
httpTransport := &http.Transport{
TLSClientConfig: c.TlsConfig,
}
// Use a new client for each request
client := &http.Client{Transport: httpTransport}
// RequestURI may not be sent to client
r.RequestURI = ""
parts := strings.SplitN(container.Node().Addr, "://", 2)
if len(parts) == 2 {
r.URL.Scheme = parts[0]
r.URL.Host = parts[1]
} else {
r.URL.Scheme = "http"
r.URL.Host = parts[0]
}
log.Debugf("[PROXY] --> %s %s", r.Method, r.URL)
resp, err := client.Do(r)
if err != nil {
return err
}
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
return nil
}
cluster/cluster.go
TlsConfig needed to be public for proxy function to be able to access it.
type Cluster struct {
sync.RWMutex
TlsConfig *tls.Config
eventHandlers []EventHandler
nodes map[string]*Node
}
....
if err := n.Connect(c.TlsConfig); err != nil {
Should the core swarmd code be an importable package the I can import in my applications?
It could be cool to have a cloudinit example in the readme, as that would help to understand on how to use Swarm.
Hi,
I am trying to create "delete" operation for docker in libswarm. I made following changes:
Still, i am not able to get this new verb operational.
I couldn't figure out how to use a TLS connection with the dockerclient backend, although some TLS stuff is mentioned in the source code. Is it possible?
POST /containers/create
in particular seems to return 200 OK
, but the correct code is 201 Created
(https://docs.docker.com/reference/api/docker_remote_api_v1.16/#create-a-container).
From a quick look this seems like a broader problem since none of the API status codes are being proxied at all.
This breaks drivers like dockerode
that expect the API spec to be followed.
#148/#160 work fine for the frontend connection, but this change seems to assume all of the TLS backend servers use the same TLS config as the swarm manage listener... This is not always the desired behavior, especially with heterogeneous backends that don't all use the same certificate set (in which case you basically can't use swarm?).
There seems to be no way to override this behavior; correct me if I'm wrong.
This problem is solved if there was a way specify the TLS key/cert/ca for docker daemons on a per-server basis, but it's not completely straightforward how that would work with the various discovery APIs...
It would be nice to have an adapter which exposes a tree of objects over HTTP, in a rest-ish interface which can easily be navigated by eg. curl.
โฐโ> ./build/bin/swarmd 'dockerserver' 'dockerclient tcp://x.x.x.x:4243'
copying task output from &beam.receiverWrapper{Receiver:(*libchan.PipeReceiver)(0x12a00e78)} to &beam.senderWrapper{Sender:(*libchan.PipeSender)(0x12aa3198)}
Starting Docker server...
Calling POST /containers/create
Error: expected status code 201, got 404:
No such image: busybox (tag: latest)
โฐโ> DOCKER_HOST="tcp://localhost:4243" docker run -d -t busybox
2014/06/25 12:24:27 EOF
Once the image is on the machine (manually run the container on the host first), docker run
work as expected.
It seems like some of this code needs to be in the libswarm dockerclient too:
https://github.com/dotcloud/docker/blob/master/api/client/commands.go#L1933-L1969
Hi,
I'm trying to use swarm for the first time, and I followed the README
but there is one error when I do "swarm create":
FATA[0000] Post /clusters: unsupported protocol scheme ""
uname -a :
Linux 3.14.1-031401-generic #201404141220 SMP Mon Apr 14 16:21:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
docker version:
Client version: 1.4.0
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 4595d4f
OS/Arch (client): linux/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f
docker -D info:
Containers: 10
Images: 87
Storage Driver: devicemapper
Pool Name: docker-8:5-6161142-pool
Pool Blocksize: 65.54 kB
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 9.783 GB
Data Space Total: 107.4 GB
Metadata Space Used: 9.712 MB
Metadata Space Total: 2.147 GB
Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 3.14.1-031401-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 8
Total Memory: 7.731 GiB
Name: Totote
ID: LPHH:4727:NJ63:DSPC:USZI:WPMJ:NPGU:IW7D:66II:OHDS:2SPP:FACK
Debug mode (server): false
Debug mode (client): true
Fds: 11
Goroutines: 10
EventsListeners: 0
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support
Any idea ?
Thank you,
Emeline
How easy or hard would it be to add a strategy where
Here is something we wrote internally.
https://github.com/CodeNow/mavis
We don't have any Go programmers but will try to contribute to this
I wrote the "Installation" section of the README a little while back and it might still work in some cases, but with the introduction of godeps
etc. I'm noting here that things should be rewritten to cover the more "modern" way of building libswarm e.g. you have to run make deps
and docker version
?
swarmd(initial-play) $ ./swarmd 'dockerserver unix://tmp/docker' 'dockerclient tcp://localhost:2375'
...
libswarm(initial-play) $ docker -H unix:///tmp/docker info
2014/06/22 22:47:41 Error: 404 page not found
Is there any particular reason that DISCOVERY_URL is not configurable? It would be nice if it was configurable similar to DOCKER_HOST. Would you accept a pull request for this?
$ go version
go version go1.3 darwin/amd64
$ uname -a
Darwin alis-macbook-air.fritz.box 13.3.0 Darwin Kernel Version 13.3.0: Tue Jun 3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64
$ go get github.com/docker/libswarm
package github.com/docker/libchan/data
imports github.com/docker/libchan/data
imports github.com/docker/libchan/data: cannot find package "github.com/docker/libchan/data" in any of:
/usr/local/Cellar/go/1.3/libexec/src/pkg/github.com/docker/libchan/data (from $GOROOT)
/Users/ali/Work/golang/src/github.com/docker/libchan/data (from $GOPATH)
Looks like these changes are covered by moby/swarmkit#115 -- or should I be doing something different to pull in an old version of libchan
?
/containers/json
prefixes Name
with the node name, but /containers/:id/json
doesn't, so currently it looks like there is no way to know with/containers/:id/json
which node you are running on.
This case is a trivial fix, but according to TODOs in api.go
the whole Name thing is unresolved anyway. So, is there an official word on the thinking behind how host info will ultimately be reported?
Just having a consistent Name
prefixing everywhere seems sensible to me -- but that's not what's currently in the code.
See postContainersCreate()
, https://github.com/docker/swarm/blob/master/api/api.go#L139:
if config.AttachStdout || config.AttachStdin || config.AttachStderr {
http.Error(w, "Attach is not supported in clustering mode, use -d.", http.StatusInternalServerError)
return
}
This is preventing swarm from creating attachable containers, period. I know swarm doesn't support the attach endpoint, but I don't think erroring out on the create config is the correct place to fail: It's reasonable to create an attachable container via swarm and attach to it later some other way. At least, it's reasonable enough for my tests to hit this!
I realize this is probably just a hack to dump a clean early error message for the CLI, but it's disruptive to API clients with different flows.
It looks like a key part of swarm is hosted on docker.com. The API to that service is looks to be closed source and undocumented. Is that code/spec available anywhere.
ec2 backend stucks at this message after that there is no activity
waiting for docker daemon on remote machine to be available.
From what I can tell so far the discovery seems to be based on a similar process to etcd bootstrapping in CoreOS. (The swarm discovery looks like it might even be using etcd
as its backend?)
I've had a lot of problems with that in the past when experimenting with CoreOS and would therefore like to have some way to statically define clusters, so that I can template that configuration myself.
I'm trying to aggregate backends on a cluster of coreos VMs. If I run swarmd like this, it works as expected:
./swarmd 'dockerserver tcp://0.0.0.0:3333' 'dockerclient tcp://node-1:2375'
However, if I try to aggregate instead, running docker -H tcp://0.0.0.0:3333 ps
just hangs and never returns:
./swarmd 'dockerserver tcp://0.0.0.0:3333' "aggregate 'dockerclient tcp://node-1:2375' 'dockerclient tcp://node-2:2375' 'dockerclient tcp://node-3:2375' 'dockerclient tcp://node-4:2375' 'dockerclient tcp://node:2375'"
Interestingly enough, if I use the simulator, things work as expected as well. If I try to aggregate two simulators, I see the same exact hang.
This is on master of libswarm. Here's the other debug output listed in the contributing docs:
bmorton@node-1 ~/swarmd $ uname -a
Linux node-1 3.17.2+ #2 SMP Sat Nov 8 00:22:33 UTC 2014 x86_64 Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz GenuineIntel GNU/Linux
bmorton@node-1 ~/swarmd $ docker version
Client version: 1.3.0
Client API version: 1.15
Go version (client): go1.3.2
Git commit (client): c78088f
OS/Arch (client): linux/amd64
Server version: 1.3.0
Server API version: 1.15
Go version (server): go1.3.2
Git commit (server): c78088f
bmorton@node-1 ~/swarmd $ docker -D info
Containers: 42
Images: 114
Storage Driver: btrfs
Execution Driver: native-0.2
Kernel Version: 3.17.2+
Operating System: CoreOS 494.0.0
Debug mode (server): false
Debug mode (client): true
Fds: 41
Goroutines: 55
EventsListeners: 0
Init SHA1: 25a6fdb579aa99537fd0d9488c553700693e73df
Init Path: /usr/libexec/docker/dockerinit
The other day I sat down with @aanand and learned some core libswarm stuff that I couldn't figure out on my own. It would be very helpful to have some documentation that covered what is the purpose of verbs, etc., and what does the overlying architecture of the system look like. I'm interested in giving this a go, just filing an issue so it's kept track of somewhere other than in my head.
Additionally, if #121 gets merged in, I would be happy to write up a detail section on how to use aggregate
backend to do the "connect to dockerserver
, run docker ps
from one host that is actually multiple hosts" thing.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.