oracle / terraform-kubernetes-installer Goto Github PK
View Code? Open in Web Editor NEWTerraform Installer for Kubernetes on Oracle Cloud Infrastructure
License: Other
Terraform Installer for Kubernetes on Oracle Cloud Infrastructure
License: Other
An extended etcd connectivity issue can lead to pods failing to start.
How: flannel uses an expiring (24 hours) etcd keys to manage the subnets allocated it worker nodes
e.g. subnet allocated to k8s-worker-ad1-0:
etcdctl ls /flannel/network/subnets
/flannel/network/subnets/10.99.82.0-24
When the worker nodes lose connectivity to etcd (e.g. when the etcd-lb is malfunctioning) the /flannel/network/subnets/10.99.82.0-24
key TTL expires and the key will be gone:
etcdctl ls /flannel/network/subnets
When the connectivity to etcd is restored, a new key is created and distributed to the flannel service on each worker node:
etcdctl ls /flannel/network/subnets
/flannel/network/subnets/10.99.43.0-24
At this point, you'll see a number of symptoms including that new pods will fail to start at complaining (presumably about the pods on the old network):
Failed to setup network for pod \"hello-2093073260-lk3f2_default(9f0a4b8a-90f0-11e6-b54b-080027242396)\" using network plugins \"cni\": \"cni0\" already has an IP address different from 10.99.43.1/24,
You'll also see the network namespace container for the pod (that "pause" pod that starts alongside other containers) with the related error:
Failed to start with docker id 99a811606b51 with error: API error (500): Cannot start container 99a811606b51cdbeddbea14af474f0df432278ac9f73baea5d8ecaf5453f521e: cannot join network of a non running container: c8a4a648e63d5b18dcd8fad6fd2d70f466584f2a749bb4245bb86a6da1ceea55
It may also have something these files are doing or not doing when a new subnet is allocated to the worker:
./instances/k8smaster/scripts/flannel.service
./instances/k8sworker/scripts/flannel.service
./instances/k8smaster/scripts/cni-bridge.service
./instances/k8smaster/scripts/cni-bridge.sh
The RPM packages for the version of Docker-Engine we depend on (17.03.1.ce
) appears to have been removed from some yum repos in favor of 17.06.2.ol
, which is causing the bootstrap to fail.
We need to update the default version to 17.06.2.ol
ASAP.
Based on some early internal feedback, we should let users control whether instances, load-balancers, subnets are public or private.
Oracle Cloud Infrastructure provider has been updated since this Installer was published and breaks the TF configs. The latest version I tried (v2.0.1) does not work for the instructions given for terraform-kubernetes-installer. I had to install an earlier version (v1.0.18) of the provider before the name change was done to successfully install with this installer.
0.10.8
2.02
label_prefix ="k8s-"
etcdAd1Count=1
etcdAd2Count=0
etcdAd3Count=0
k8sWorkerAd1Count=1
k8sWorkerAd2Count=1
k8sWorkerAd3Count=0
k8sMasterAd1Count=0
k8sMasterAd2Count=0
k8sMasterAd3Count=1
etcdShape="VM.Standard1.1"
k8sMasterShape="VM.Standard1.2"
k8sWorkerShape="VM.Standard1.4"
k8sMasterLBShape="100Mbps"
etcdLBShape="100Mbps"
creating a cluster using default access policy should give a public load balancer ip address
master_lb_ip = [
10.0.30.4
]
Currently, the installer does not modify any docker logging configuration. This can lead to full-disk usage errors if the cluster is left running long enough.
Need to add some basic log size limits to docker containers. Also add some log rotation capabilities.
client-to-server communication is already disallowed by ingress. The etcd LB is also private. However, we should also ensure that communication within the VCN is also secured using SSL.
The issue should be easy to reproduce with the following:
terraform init
terraform plan
. This works which seems to prove the credentials are correctterraform apply
.Error applying plan:
1 error(s) occurred:
* module.vcn.oci_core_virtual_network.CompleteVCN: 1 error(s) occurred:
* oci_core_virtual_network.CompleteVCN: Status: 404; Code: NotAuthorizedOrNotFound; OPC Request ID: /OMITTED; Message: Authorization failed or requested resource not found.
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
After the failure I ran terraform destroy
and it cleaned by successfully.
Any missing prerequisites ? Or is this a legitimate defect?
We're seeing some sort of pod communication issue when a second generation instance shapes are used for worker nodes (e.g. VM.DenseIO2.16, BM.GPU2.2) where we cannot connect to pod-ip:pod-port from a node other than the node the pod is running on even though it is part of the same Flannel overlay network.
Steps to reproduce
terraform apply
kubectl run my-nginx --image=nginx --replicas=1 --port=80
kubectl expose deployment my-nginx --port=80
curl http://<pod-ip>:80
and see the command hang. (note that you can ping ).curl http://<pod-ip>:80
and see the command succeed.Also note that the issue only occurs when either VXLAN (default) or host-gw are used as the backend mode for Flannel are used. udp backend mode appears to be working correctly, although this comes with the limitation of only being able to have nodes in a single AD.
$ terraform -v
$ <path-to-plugin>/terraform-provider-oci
It'd be good to have an example configuration that uses Oracle's Amazon S3 Compatibility API for for remote state storage.
terraform apply
after a successful build / validation.Should include the things we need to reproduce issues including:
terraform plan
scripts/cluster-check.sh
Step 3 says I need to create a file that has
providers {
oci = "<path_to_provider_binary>/terraform-provider-oci"
}
Ooookkkk.....what is a provider binary? That is not discussed anywhere that I can see unless I'm really missing something. Is this something that I downloaded in the previous packages, the code that I want to upload using kubernetes, or is it an option in the OCI dashboard?
Please, could you make this more clear?
Error applying plan:
9 error(s) occurred:
This message is a bit of a red herring, the problem isn't that domain_name and k8s_ver are missing, it is related to parsing outputs that aren't in the state file.
doesn't reproduce with v0.10.3 and higher.
Currently, when control_plane_subnet_access=private
a single set of public subnets (public-subnet-AD[1-3], are provisioned which are shared by the NAT instances as well as the k8s master LB (if k8s_master_lb_access=public
). As a consequence, this means that the NAT and the k8s master LB share the same security list.
This was thought to be OK with the assumption that there would not be any overlap between the ingress between the two (e.g. 22 for the NAT, 443 for the LB) . However, there could be corner cases wherein the aforementioned assumption doesn't hold true and thereby the topology is not desirable. (redacting the internal usecase from this public filing) As discussed with @jlamillan , If we could add another flag that separates the two subnets - that would be ideal.
We recently bumped the default version of Kubernetes (i.e. k8s_ver) from 1.7.4 to 1.7.10. Unfortunately, 1.7.10 yum packages appear to be available in less Yum repos, which leads to issues initializing the software.
Specifically, when $k8s_ver is not available in the Yum repository, we randomly pick up whatever version of the kubelet that kubernetes-cni pulls in to fulfill its dependency:
until yum install -y kubelet-${k8s_ver}-$RPM_TAG kubectl-${k8s_ver}-$RPM_TAG kubernetes-cni; do sleep 1 && echo -n ".";done
Create an optional parameter for VCN CIDR for the created VCN.
This would give users more flexibility to leverage VCN peering which requires non-overlapping VCN CIDRs.
Hi,
Is there a plan to add support for Calico or other CNI or is Flannel the only option that you will support in this repo?
Thanks,
If there is a restart of the VM instances, the processes on the etcd, master and worker VMs are not coming up. we should configure the processes that we start during the bootstrap process on the etcd, master and worker VMs as daemon process.
Currently, after a k8s cluster has been created, etcd nodes cannot successfully be replaced. This is a limitation due to how the etcd cluster is created. The current method is to use a discovery url. Switching this to a more convention list of etcd endpoints would allow for more flexible etcd nodes within the installer cluster.
Move from using discovery_url to configure the cluster to building a list of cluster ips, or dns names to allow for terraform node udpates or replacements.
This will required creating an LB managed by BMC in some form.
We currently do not support adding or removing members of the etcd cluster after it is provisioned.
We need to add support for dynamically scaling etcd cluster in and out at runtime.
Based on some internal feedback, it would be better if we automatically wrote out the instance private key file to generated folder by default instead of having the user manually do it as indicated in the README:
$ terraform output ssh_private_key > generated/instances_id_rsa
$ chmod 600 generated/instances_id_rsa
$ ssh -i generated/instances_id_rsa ...
Currently the etcd data directory is in the docker volume. we need to mount to an instance and use a block volume so that we could do snapshots of the block volume to recover from any failures.
Intermittently, the call to https://discovery.etcd.io can fail, which causes the following invalid discovery token to be generated in the generated
folder:
jemillanmbp:generated jessem$ cat discoverye3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Unable to generate token
This situation in turn leads to an invalid parameter being passed to -discovery
in terraform-kubernetes-installer/instances/etcd/cloud_init/bootstrap.template.sh
:
docker run -d \
-p 2380:2380 -p 2379:2379 \
-v /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt \
--net=host \
quay.io/coreos/etcd:${etcd_ver} \
/usr/local/bin/etcd \
-name $HOSTNAME \
-advertise-client-urls http://$IP_LOCAL:2379 \
-listen-client-urls http://$IP_LOCAL:2379,http://127.0.0.1:2379 \
-listen-peer-urls http://0.0.0.0:2380 \
-discovery "Unable to generate token"
This causes the cluster to fail to come up properly.
We are moving away from using the discovery URL, but in the meantime, we should not try to create a cluster if we failed to get a discovery url.
On running command
$ kubectl get cluster-info
I get an error - No such resource type "cluster-info".
Access to the k8s master insecure port is already disallowed by ingress. However, we should also ensure that communication within the VCN is also only on the secure port using SSL.
With private subnets, we see that the worker nodes come up in not ready state
worker nodes(1 or 2 or 3) are coming up in NotReady status.
kubectl get nodes
NAME STATUS AGE VERSION
k8s-worker-ad1-0.k8sworkerad1.k8sbmcs.oraclevcn.com NotReady 31m v1.7.4
k8s-worker-ad2-0.k8sworkerad2.k8sbmcs.oraclevcn.com NotReady 31m v1.7.4
k8s-worker-ad3-0.k8sworkerad3.k8sbmcs.oraclevcn.com NotReady 31m v1.7.4
Below in the master node journal entries.
Jan 02 17:37:44 k8s-master-ad1-0 kubelet[13086]: W0102 17:37:44.615767 13086 container_manager_linux.go:218] Running with swap on is not supported, please disable swap! This will be a fatal error by default starting in K8s v1.6! In the meantime, you can opt-in to making this a fatal error by enabling --experimental-fail-swap-on.
On the worker nodes journal entries, seeing the below
Jan 02 18:20:05 k8s-worker-ad1-0 kubelet[13436]: E0102 18:20:05.035951 13436 eviction_manager.go:238] eviction manager: unexpected err: failed GetNode: node 'k8s-worker-ad1-0.k8sworkerad1.k8sbmcs.oraclevcn.com' not found
Jan 02 18:20:15 k8s-worker-ad1-0 kubelet[13436]: E0102 18:20:15.073231 13436 eviction_manager.go:238] eviction manager: unexpected err: failed GetNode: node 'k8s-worker-ad1-0.k8sworkerad1.k8sbmcs.oraclevcn.com' not found
Jan 02 18:20:25 k8s-worker-ad1-0 kubelet[13436]: E0102 18:20:25.111192 13436 eviction_manager.go:238] eviction manager: unexpected err: failed GetNode: node 'k8s-worker-ad1-0.k8sworkerad1.k8sbmcs.oraclevcn.com' not found
Noticed that the kube-proxy is not coming up..for another 2 workers nodes are in not ready status now..[opc@k8s-master-ad1-0 ~]$ kubectl get po -n=kube-system
NAME READY STATUS RESTARTS AGE
kube-dns-2272871451-v3p1r 3/3 Running 0 16m
kube-proxy-k8s-worker-ad1-0.k8sworkerad1.k8sbmcs.oraclevcn.com 1/1 Running 0 14m
kube-proxy-k8s-worker-ad2-0.k8sworkerad2.k8sbmcs.oraclevcn.com 0/1 Unknown 0 14m
kube-proxy-k8s-worker-ad3-0.k8sworkerad3.k8sbmcs.oraclevcn.com 0/1 Pending 0 14m
kubernetes-dashboard-3313488171-1z911 1/1 Running 0 16m
tiller-deploy-2136207906-kct9c 1/1 Running 0 7m
10.8
2.0.2
1.8.0 and higher
commit f47ed4b
localhost kubelet: error: failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename#011#011#011#011Type#011#011Size#011Used#011Priority /dev/sda2 partition#0118420344#0110#011-1]
Nov 15 17:59:40 localhost systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
2.0.4 - 2017-11-2
I have Oracle Public Cloud environment. Does terraform-kubernetes-installer support Oracle Public Cloud(OCI-C)?
Thanks in advance!
Provide an option in the config to merge the etcd instance and k8smaster instance to the same node.
We need to move to Kubernetes 1.8 in the terraform-kubernetes-installer. This task also includes making whatever changes are necessary to the bootstrap scripts, and re-running all tests including Kubernetes conformance tests.
In addition, we should consider moving to the latest:
Since there is no kubeadm, so can't use kubeadm to upgrade k8s.
It will be best if can provide a instruction for upgrade k8s with this installer. Thanks!
When load balancer resources are limited in a single bmc tenancy, it'll limit the total number of k8s clusters that can be set up in a team to be used in testing and development, if every installation requires two load balancers, one for etcd, and one for the masters.
Currently, each cluster creates a brand new VCN, subnets, reoute tables, etc. for each cluster. It would be nice if we supported the ability to provision subsequent clusters into an an existing VCN.
Enable support for Persistent Volumes and Persistent Volume Claims
It looks like oci_core_images is now returning a GPU image as the first image in the list, which will cause an error unless the user is using the BM.GPU2.2 shape due to the way we set up our data
and resources
:
e.g. data:
data "oci_core_images" "ImageOCID" {
compartment_id = "${var.compartment_ocid}"
operating_system = "Oracle Linux"
operating_system_version = "7.4"
}
e.g. resource:
resource "oci_core_instance" "TFInstance" {
...
image = "${lookup(data.oci_core_images.ImageOCID.images[0], "id")}"
...
}
which leads to the following error:
* oci_core_instance.TFInstanceK8sMaster: Status: 400; Code: InvalidParameter; OPC Request ID: /XXXXX; Message: Shape VM.Standard1.8 is not valid for image ocid1.image.oc1.iad.aaaaaaaaflaxipg2k4l7mjudowij5vic76jjnqgzktw46pga4knjpqcrxw6a.
This goes to show that the oci_core_images image list can and will change underneath us. Therefore, I think we'd be better off passing the image name in directly in the data source so that it is fixed for the query and the live cluster:
i.e. something like:
data "baremetal_core_images" "ImageOCID" {
compartment_id = "${var.compartment_ocid}"
display_name = "Oracle-Linux-7.4-2017.08.25-0"
}
We should add support for all the flavors of Linux distributions BMC provides including Oracle Linux, CentOS, and Ubuntu.
$ terraform -v
Terraform v0.11.1
$ <path-to-plugin>/terraform-provider-oci
terraform-provider-oci 2.0.5
A worker node doesn't have ExternalIP. Can it be provisioned or configured after the fact?
Cluster set up with a fairly recent (last couple of days) version of the installer.
I am seeing the following:
14:10:00.460463 IP 10.0.42.2.39359 > 169.254.169.254.domain: 48005+ A? ioctl.org. (27)
14:10:00.460982 IP 169.254.169.254.domain > 10.0.42.2.39359: 48005 1/13/0 A 94.136.54.223 (254)
ie, that’s working.
Traffic on that node that’s targeting the upstream resolver from inside the pod:
14:10:03.873217 IP 10.99.38.7.30422 > 169.254.169.254.domain: 38833+ A? registry-1.docker.io. (38)
14:10:03.873263 IP 10.99.38.7.41485 > 169.254.169.254.domain: 40603+ AAAA? registry-1.docker.io. (38)
In fact, all traffic destined for off the cluster is not NATted correctly:
14:17:15.508146 IP 10.99.16.11 > 8.8.8.8: ICMP echo request, id 24576, seq 0, length 64
(This a tcpdump of the traffic from a node containing an arbitrary pod pinging that address.)
It looks like the docker-engine systemd configuration is overriding --ip-masq=false (and the other options there). That is, /usr/lib/systemd/system/docker.service
has the right ExecStart but /etc/systemd/system/docker.service.d/docker-sysconfig.conf
(which is package-installed) overrides it with (empty) values from /etc/sysconfig/docker*
Without the --ipmasq=false setting on dockerd, pod traffic can't correctly get out of the cluster.
we used the latest sre installer and noticed that the cluster is setup with etcd LBR as a private LBR
We are concerned that using this setup with a private etcd LBR, losing network connectivity to that AD with etcd could potentially bring down the service.
SRE installer should provide the user with an option to make it public, so the user can choose how they want it to be setup based on their cluster setup requirements.
We need to move away from bash init scripts for configuring the cluster to local Ansible playbooks.
One of many advantages of this is that we would no longer be tied to one OS.
Hi ,
Does oracle use some flavor of open-stack? If yes , I want to discuss the possibility of porting this project to a generic open-stack installation. What major changes will be needed to make this work on any latest open-stack installation?
Thanks.
https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
could be one option
10.8
2.02
commit f47ed4b
label_prefix ="k8s-"
etcdAd1Count=1
etcdAd2Count=0
etcdAd3Count=0
k8sWorkerAd1Count=1
k8sWorkerAd2Count=1
k8sWorkerAd3Count=0
k8sMasterAd1Count=0
k8sMasterAd2Count=0
k8sMasterAd3Count=1
etcdShape="VM.Standard1.1"
k8sMasterShape="VM.Standard1.2"
k8sWorkerShape="VM.Standard1.4"
k8sMasterLBShape="100Mbps"
etcdLBShape="100Mbps"
etcd_lb_enabled="false"
k8s_ver = "1.8.3"
etcd_docker_max_log_size = "250m"
etcd_docker_max_log_files = "2"
master_docker_max_log_size = "100m"
worker_docker_max_log_size = "75m"
worker_docker_max_log_files = "3"
disable_auto_retries="false"
systemd: [/etc/systemd/system/flannel.service:10] Unknown lvalue 'After' in section 'Service'
I am trying to install the kubernetes service-catalog into the k8s 1.7.4 provisioned by BMC terraform.
The link for installing service catalog is https://github.com/kubernetes-incubator/service-catalog/blob/master/docs/install.md
I am getting Error: cluster doesn't provide requestheader-client-ca-file
Azure's acs-engine also had the same issue and they fixed it. The link for that issue is Azure/acs-engine#1390
Add a parameter to support the enable/disable of using an etcd load balancer.
etcdLBEnabled = "false"
we should have a different ssh key for the NAT instances when using private subnets. Currently its shared with worker, master and etcd instances as well.
Created a new cluster and see empty / null values for some labels
node.info/node.id_prefix:
node.info/node.id_suffix:
node.info/node.shape:
It was fresh git clone on 11/2. The provider is terraform-provider-oci_v2.0.3
More details in slack #kubernetes-installer
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.