kubernetes-sigs / kubespray Goto Github PK

Below is a list of tasks in order to automate cluster deployment tests :

• Test on the main linux distribs :
  • Fedora
  • Centos 7
  • RedHat 7
  • Ubuntu 1404
  • Ubuntu 1510
  • Debian 8
• One test per network plugin
• Idealy a cluster will be composed of
  • a master (master components only)
  • a master with node components (handles workloads)
  • a node

Test sequence:

• Cluster deployment
• Check all nodes ready
• Run a replica controller composed of 2 pods

kubectl run test --image=busybox --replicas=2 --command -- tail -f /dev/null
replicationcontroller "test" created

• Check that pods ip addresses are in the configured subnet

kubectl describe po test-34ozs | grep ^IP
IP:             10.233.16.2

kubectl exec test-34ozs -- ip a show dev eth0
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
    link/ether 02:42:0a:e9:2b:03 brd ff:ff:ff:ff:ff:ff
    inet 10.233.16.2/24 scope global eth0

• Ping between pods
• Deploy the kubedns app and browse the kubernetes api using the service name as follows

kubectl exec test-f31yx -- curl -k -u kube:changeme https://kubernetes

For an unknown reason, the kube-api service is not started when deploying the first time.

Configure Calico to connect to the api on secure port.

Currently, the script sends all manifests to master[0] in /etc/k8s/manifestsand then use kubectl to create the resources.
I think we all agree that is better to use API so the question is what do we do about resource manifests?

1. Use API only, the manifests don;t need to be stored
2. Use API only and store manifests in the deploy machine
3. Use API only and store manifests in one of the masters
4. (complete)

From my side

• I like to see the manifests so not the 1.
• 2 is fine, but deploy machine can be any. And if someone else from the team deploy, I may not have access to the manifests.
• I like 3 :), @Smana I know you don't, can you explain why?

fatal: [node1] => Failed to template {{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}: The ipaddr filter requires python-netaddr be installed on the ansible controller

According to projectcalico devs, the cbr0 is not required anymore with the var CALICO_IPAM=true.
To be tested.

Until now no graphs are displayed in the grafana dashboard. (404)

I just notice that the kubernetes service only have one endpoint in a multi master architecture.

Name:           kubernetes
Namespace:      default
Labels:         component=apiserver,provider=kubernetes
Selector:       <none>
Type:           ClusterIP
IP:         10.233.0.1
Port:           <unnamed>   443/TCP
Endpoints:      10.115.99.41:443
Session Affinity:   None
No events.

$(pgrep systemd > /dev/null && systemctl status > /dev/null);
has a return code of 3 when systemd is running but has a failed service.

Some users use ansible for the first time, I think a quick note/doc-link about options and configuration would be good.

failed: [10.206.41.5 -> localhost] => {"cmd": "rsync --delay-updates -F --compress --times --rsh 'ssh -S none -o StrictHostKeyChecking=no' --out-format='<>%i %n%L' "/tmp/releases/kubernetes/bin/kubelet" "[email protected]:/usr/local/bin/kubelet"", "failed": true, "rc": 23}
msg: rsync: mkstemp "/usr/local/bin/.kubelet.kPJDLA" failed: Permission denied (13)

Upgrade to Kubernetes 1.1 with the latest version of Calico-docker.
projectcalico/k8s-exec-plugin#92

kubernetes/kubernetes#18758

Try to find a generic way (linux distribution) to configure iptables.

I'm happy to help here, but I'm not very familiar with Ansible, so some guidance will be needed.

Playbook should have test

We noticed that it was tricky to notify the apiserver about changes (e.g. add when a new node is added #47 ).
We've decided that core kubernetes components like etcd or apiserver should be running as host services.

The docker role fail on ubuntu, the role works only on debian

Create a role "common" in charge of resolving any dependencies (python, package, binaries...).

We faced issues with synchronize module and using 'copy' had too much constraint.

I propose that each nodes download their dependencies, it will then remove the need to copy/rsync binaries from node to node.

I'll probably work again on the 'downloader' host later as it can be usefull in somecase (later)

I think it would be easier to manage only systemd init linux distribs.
All the main linux distribs have made that choice, even ubuntu is switching to it.

@ant31 : what is your opinion ? is it possible to use a systemd OS with travis ?

This is more for questions to @ant31 and @Smana.

I'm trying to use the apps (kube-dns, and others). I'm getting errors with all of them because the files to be fed into kubectl have unicode string in them. The problem appears to be that python structure dumps into the kube templates causes u'ddd' style unicode string. This isn't supported by kubectl. I've found a fix for them, but they will hit pretty much all of the k8s-XXX modules. I've added to_nice_json to many of the vars for printing into the files.

I was wondering if you have this working or haven't tried it recently.

Using systemd unit should be optionnal.

This include rework on following roles:

• etcd
• dnsmasq
• calico-node
• kubelet

ansible/ansible-modules-extras#1229

I see two options:

1. configure with ansible the firewall
2. Create a 'firewall' controller daemon that check services and ingress

In the file /etc/network-environment
The default ipv4 in took from the first interface only. But it should be possible to set ip from differents interface

DEFAULT_IPV4=

I had an issue on reboot with etcd as pod.

It looks like kubelet continue to try restarting same pods, and pods failed to start because etcd was not 'on'.

We should rollback and use an etcd not-managed by kubernetes.

I propose to run kubelet on all nodes. It will allow to use pod manifests to bootstrap services like etcd.

---   
- hosts: k8s-cluster
 roles: 
    - { role: docker, tags: docker }
    - { role: kubernetes/kubelet, tags: kubelet }
    - { role: etcd, tags: etcd }  
    - { role: dnsmasq, tags: dnsmasq }
    - { role: network_plugin, tags: ['calico', 'flannel', 'network'] }

- hosts: k8s-node
  roles:
    - { role: kubernetes/node, tags: node }

- hosts: kube-master
  roles:
    - { role: kubernetes/master, tags: master }

if possible to support both 1.9 / 2

My installs are now failing because kube and etcd users are missing.

I noticed nothing creates them. I can add them to the playbook in download.

Deploy multiple cluster in one ansible run.

http://kubernetes.io/v1.0/docs/admin/multi-cluster.html

kubernetes/kubernetes#18174

Kubenetes cluster's components should be deployed in HA mode when possible.

http://kubernetes.io/v1.0/docs/admin/high-availability.html

Currently, the deployment fails at the first deployment.
The apiserver is restarted multiple times and when the handlers are played i have the following output for an unknown reason:

NOTIFIED: [kubernetes/master | restart kube-apiserver] ************************ 
failed: [node2] => {"failed": true}
msg: Job for kube-apiserver.service failed. See 'systemctl status kube-apiserver.service' and 'journalctl -xn' for details.

The logs don't help very much

Jan 24 15:29:54 kubenode2 kube-apiserver[4192]: [restful] 2016/01/24 15:29:54 log.go:30: [restful/swagger] listing is av
Jan 24 15:29:54 kubenode2 kube-apiserver[4192]: [restful] 2016/01/24 15:29:54 log.go:30: [restful/swagger] https://10.11
Jan 24 15:29:55 kubenode2 kube-apiserver[4192]: F0124 15:29:55.343958    4192 controller.go:80] Unable to perform initia
Jan 24 15:29:55 kubenode2 systemd[1]: kube-apiserver.service: main process exited, code=exited, status=255/n/a
Jan 24 15:29:55 kubenode2 systemd[1]: Failed to start Kubernetes API Server.
Jan 24 15:29:55 kubenode2 systemd[1]: Unit kube-apiserver.service entered failed state.

I guess the error is caused by a missing controller-manager but i'm not sure

the task doesn't work if there is already a bridge with name for exemple 'lxcbr0'