metral / corekube Goto Github PK
View Code? Open in Web Editor NEWCoreOS + Kubernetes + OpenStack - The simplest way to deploy a POC Kubernetes cluster using a Heat template
License: Apache License 2.0
corekube's People
Contributors
Stargazers
Watchers
corekube's Issues
multiple networks in private openstack?
Thank you for the blog post and this configuration.
When deploying this on a private openstack network, are 2 private networks necessary to create the additional ifaces? Could it all be done on one private network and public ip address accessing it?
Pointers welcome.
Best,
Tom
timeout for overlord and discovery
first, i wanted to say that this is a great project you have here. i work with your folks on the openstack-ansible side, and you rackspace folks are awesome with the community!
i'm running into a bit of an issue with overlord and discovery timing out, and i was wondering if this should work for private installs of openstack-ansible as well? i was trying to load the openstack.yml file without much luck, since CoreOS was reporting failed units and other things. this would be a huge win, since i'd like to demo some things around coreos for our folks internally. any ideas what could be causing the issues? if you need logs or more, just tell me what to grab and i will provide it for you.
thanks for everything, including the awesome project!
Unable to create stack
I'm running this command:
heat stack-create corekube --template-file corekube-cloudservers.yaml -P keyname=<id_name>
and getting the following error seconds afterwards:
ERROR: b'{"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "Failed to validate: One of the properties \\"network\\", \\"port\\" or \\"subnet\\" should be set for the specified network of server \\"0\\".", "traceback": null, "type": "StackValidationFailed"}, "title": "Bad Request"}'
I'm not sure why that would be missing... Is that a value I have to set before attempting to create the stack?
How to expose service with Cloud DNS?
hi. major n00b here.
I setup the heat stack on my rackspace cloud. I see overlord, master, discovery and 3 minions are setup. I've also verified overlord setup.
I deployed the guestbook example. I see that the service is running:
kubernetes-master ~ # /opt/bin/kubectl get services
NAME LABELS SELECTOR IP(S) PORT(S)
kubernetes component=apiserver,provider=kubernetes <none> 10.1.0.1 443/TCP
redis-master name=redis-master name=redis-master 10.1.254.206 6379/TCP
redis-slave name=redis-slave name=redis-slave 10.1.231.118 6379/TCP
How do I go about exposing the frontend service to reach it from a browser?
Private openstack network issues during install, net and mtu
This is the template that I had to use for a private openstack install. If you want I can do a PR with a new .yaml file.
I figured I would just put it here for historical purposes in case anyone runs into the issue I had.
I had to change the ProviderNet to Net.
corekube_network:
type: OS::Neutron::Net
properties:
name: "corekube"
and in the overlord switch up the mtu based upon my environment
- path: /run/flannel_docker_opts.env
permissions: '0755'
content: |
DOCKER_OPTS="--mtu=1404"
This is the full file that I used to get it running.
heat_template_version: 2013-05-23
description: >
Deploy Kubernetes on a CoreOS cluster that operates on an
OpenStack Powered Cloud (Openstack Kilo | Rackspace Private Cloud)
Maintainer - Mike Metral <[email protected]> | Github: @metral
parameters:
kubernetes_minion_count:
description: Number of CoreOS machines to deploy as Kubernetes Minion
type: number
default: 3
constraints:
- range:
min: 1
max: 12
description: Must be between 2 and 12 servers.
keyname:
type: string
description: Name of keypair to be used for compute instance
flavor:
type: string
default: m1.medium
constraints:
- allowed_values:
- m1.medium
- m1.large
- m1.xlarge
description: |
Must be a valid flavor
coreos_image:
type: string
description: Rackspace Cloud Servers CoreOS Stable (808.0.0) UUID
default: "CoreOS"
git_command:
type: string
description: Git repo checkout command
default: "/usr/bin/git clone https://github.com/metral/overlord ; /usr/bin/git -C overlord checkout -qf 199028526132ff9491d26c00b2fd9b9b9d7239a8"
flannel_url:
type: string
description: Flannel (0.5.5) Binary URL
default: "http://0afbabe78c1b1691fad4-21af6827a69c3ad910ccdb6595a59d19.r49.cf1.rackcdn.com/flanneld"
private_discovery_etcd_version:
type: string
description: Version of etcd used for Private Discovery node
default: "2.2.2"
discovery_net_interface:
type: string
description: Host network interface for which the Discovery node will operate over
default: "eth0"
kubernetes_net_interface:
type: string
description: Host network interface for which Kubernetes' overlay will operate over
default: "eth0"
timeout:
description: Wait condition timeout seconds
type: number
default: 600
floating-network-id:
type: string
label: Floating Network ID
description: UUID of the external network. The private network created by this stack will route to this network. Any floating IP addresses needed by this stack will also route to this network.
resources:
coreos_cluster_uuid:
type: OS::Heat::RandomString
stack_security:
type: OS::Neutron::SecurityGroup
properties:
name: "corekube"
rules:
- remote_ip_prefix: 0.0.0.0/0
protocol: tcp
port_range_min: 0
port_range_max: 65535
- remote_ip_prefix: 0.0.0.0/0
protocol: udp
port_range_min: 0
port_range_max: 65535
- remote_ip_prefix: 0.0.0.0/0
protocol: icmp
corekube_network:
type: OS::Neutron::Net
properties:
name: "corekube"
corekube_subnet:
type: OS::Neutron::Subnet
properties:
name: "corekube_subnet"
network_id:
get_resource: corekube_network
cidr: "10.10.12.0/24"
gateway_ip: "10.10.12.1"
allocation_pools: [{"start": "10.10.12.2", "end": "10.10.12.254"}]
dns_nameservers:
- 8.8.8.8
- 8.8.4.4
corekube_router:
type: OS::Neutron::Router
properties:
name: "corekube_router"
external_gateway_info:
enable_snat: true
network:
get_param: floating-network-id
corekube_router_interface:
type: OS::Neutron::RouterInterface
properties:
router_id:
get_resource: corekube_router
subnet_id:
get_resource: corekube_subnet
discovery_wait_handle:
type: OS::Heat::WaitConditionHandle
discovery_wait_condition:
type: OS::Heat::WaitCondition
properties:
handle: { get_resource: discovery_wait_handle }
timeout: { get_param: timeout }
discovery_floatingip:
type: OS::Neutron::FloatingIP
properties:
floating_network_id:
get_param: floating-network-id
discovery_port:
type: OS::Neutron::Port
properties:
security_groups:
- get_resource: stack_security
network_id:
get_resource: corekube_network
discovery_floatingip_association:
type: OS::Neutron::FloatingIPAssociation
properties:
floatingip_id:
get_resource: discovery_floatingip
port_id:
get_resource: discovery_port
discovery:
type: OS::Nova::Server
properties:
key_name: { get_param: keyname }
image: { get_param: coreos_image }
flavor: { get_param: flavor }
name: "discovery"
networks:
- port:
get_resource: discovery_port
user_data_format: RAW
config_drive: "true"
user_data:
str_replace:
template: |
#cloud-config
write_files:
- path: /run/get_discovery_interface_ip.sh
permissions: '0755'
content: |
#!/bin/bash
# Get's the IP of the interface that discovery will be
# accessible over
DISCOVERY_IF=%discovery_net_interface%
/usr/bin/ip -4 addr show $DISCOVERY_IF | /usr/bin/awk '/inet/ {print $2}' | /usr/bin/cut -d/ -f1 > /run/IP
/usr/bin/sed -i 's/^/IP=/' /run/IP
- path: /run/wcnotify.sh
permissions: '0755'
content: |
#!/bin/bash
%wc_notify% --data-binary '{"status": "SUCCESS"}'
coreos:
update:
group: stable
reboot-strategy: off
units:
- name: private-discovery-setup.service
command: start
content: |
[Unit]
After=network-online.target
Requires=network-online.target
[Service]
Type=forking
ExecStart=/usr/bin/bash /run/get_discovery_interface_ip.sh
- name: private-discovery.service
command: start
content: |
[Unit]
After=network-online.target private-discovery-setup.service
Requires=network-online.target private-discovery-setup.service
[Service]
EnvironmentFile=/run/IP
Restart=on-failure
RestartSec=5s
ExecStartPre=/usr/bin/docker pull quay.io/coreos/etcd:v%private_discovery_etcd_version%
ExecStart=/usr/bin/docker run -d --name discovery \
-p 2379:2379 -p 2380:2380 \
-v /usr/share/ca-certificates/:/etc/ssl/certs \
--net host quay.io/coreos/etcd:v%private_discovery_etcd_version% -name discovery \
-initial-advertise-peer-urls http://${IP}:2380,http://${IP}:7001 \
-listen-peer-urls http://${IP}:2380,http://${IP}:7001 \
-initial-cluster discovery=http://${IP}:2380,discovery=http://${IP}:7001 \
-advertise-client-urls http://${IP}:2379,http://${IP}:4001 \
-listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
- name: private-discovery-done.service
command: start
content: |
# Notifiy wait condition
[Unit]
After=network-online.target private-discovery.service
Requires=network-online.target private-discovery.service
[Service]
ExecStart=/run/wcnotify.sh
params:
"%wc_notify%": { get_attr: [discovery_wait_handle, curl_cli] }
"%discovery_net_interface%": { get_param: discovery_net_interface }
"%private_discovery_etcd_version%": { get_param: private_discovery_etcd_version}
overlord_wait_handle:
type: OS::Heat::WaitConditionHandle
overlord_wait_condition:
type: OS::Heat::WaitCondition
properties:
handle: {get_resource: overlord_wait_handle}
timeout: {get_param: timeout}
overlord_floatingip:
type: OS::Neutron::FloatingIP
properties:
floating_network_id:
get_param: floating-network-id
overlord_port:
type: OS::Neutron::Port
properties:
security_groups:
- get_resource: stack_security
network_id:
get_resource: corekube_network
overlord_floatingip_association:
type: OS::Neutron::FloatingIPAssociation
properties:
floatingip_id:
get_resource: overlord_floatingip
port_id:
get_resource: overlord_port
overlord:
type: OS::Nova::Server
properties:
key_name: { get_param: keyname }
image: { get_param: coreos_image }
flavor: { get_param: flavor }
name: "overlord"
networks:
- port:
get_resource: overlord_port
user_data_format: RAW
config_drive: "true"
user_data:
str_replace:
template: |
#cloud-config
write_files:
- path: /run/wcnotify.sh
permissions: '0755'
content: |
#!/bin/bash
%wc_notify% --data-binary '{"status": "SUCCESS"}'
- path: /run/flannel_docker_opts.env
permissions: '0755'
content: |
DOCKER_OPTS="--mtu=1404"
- path: /run/setup_etcd_cloudinit_conf.sh
permissions: '0755'
content: |
#!/bin/bash
ENV="/etc/environment"
# Test for RW access to $1
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV
exit 1
fi
# Setup environment target
sed -i -e '/^COREOS_PUBLIC_IPV4=/d' \
-e '/^COREOS_PRIVATE_IPV4=/d' \
"${ENV}"
# We spin loop until the the IP addresses are set
function get_ip () {
IF=$1
IP=
while [ 1 ]; do
IP=$(ifconfig $IF | awk '/inet / {print $2}')
if [ "$IP" != "" ]; then
break
fi
sleep .1
done
echo $IP
}
# Echo results of IP queries to environment file as soon as network interfaces
# get assigned IPs
echo COREOS_PUBLIC_IPV4=$(get_ip eth0) >> $ENV # Also assigned to same IP
echo COREOS_PRIVATE_IPV4=$(get_ip eth0) >> $ENV #eno1 should be changed to your device name
A=`cat /etc/environment | grep COREOS_PRIVATE_IPV4 | cut -f2 -d "="`
sed -i "s#=:#=${A}:#g" /run/systemd/system/etcd.service.d/20-cloudinit.conf
systemctl daemon-reload
coreos:
etcd:
name: overlord
discovery: http://%discovery%:2379/v2/keys/discovery/%uuid%
addr: $private_ipv4:4001
peer-addr: $private_ipv4:7001
update:
group: stable
reboot-strategy: off
units:
- name: setup-etcd-cloudinit-conf.service
command: start
runtime: true
content: |
[Unit]
Description=Setup environment with private (and public) IP addresses
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/run/setup_etcd_cloudinit_conf.sh
- name: etcd.service
command: start
- name: fleet.socket
command: start
content: |
[Socket]
# Talk to the API over a Unix domain socket (default)
ListenStream=/var/run/fleet.sock
# Talk to the API over an exposed port
ListenStream=10001
Service=fleet-local.service
[Install]
WantedBy=sockets.target
- name: fleet-local.service
command: start
content: |
# fleet-local is kicked off by fleet.socket after API port
# is opened
[Unit]
Description=fleet-local
Wants=etcd.service
Requires=etcd.service
[Service]
Environment=FLEET_PUBLIC_IP=$private_ipv4
Environment=FLEET_METADATA=kubernetes_role=overlord
ExecStart=/usr/bin/fleet
Restart=always
RestartSec=10s
- name: overlord.service
command: start
content: |
# Overlord / logic layer service to deploy kubernetes to
# the cluster
[Unit]
After=network-online.target etcd.service
Requires=network-online.target etcd.service
[Service]
WorkingDirectory=/root
Environment="DIR=overlord"
ExecStartPre=/usr/bin/rm -rf $DIR
ExecStartPre=%git_command%
ExecStart=/usr/bin/bash ${DIR}/build_run.sh
- name: overlord-done.service
command: start
content: |
# Notifiy wait condition
[Unit]
After=network-online.target overlord.service
Requires=network-online.target overlord.service
[Service]
ExecStart=/run/wcnotify.sh
params:
"%discovery%": { get_attr: [discovery, networks, corekube, 0] }
"%uuid%": { get_attr: [coreos_cluster_uuid, value] }
"%git_command%": { get_param: git_command }
"%wc_notify%": { get_attr: [overlord_wait_handle, curl_cli] }
kubernetes_master_floatingip:
type: OS::Neutron::FloatingIP
properties:
floating_network_id:
get_param: floating-network-id
kubernetes_master_port:
type: OS::Neutron::Port
properties:
security_groups:
- get_resource: stack_security
network_id:
get_resource: corekube_network
kubernetes_master_floatingip_association:
type: OS::Neutron::FloatingIPAssociation
properties:
floatingip_id:
get_resource: kubernetes_master_floatingip
port_id:
get_resource: kubernetes_master_port
kubernetes_master:
type: OS::Nova::Server
properties:
key_name: { get_param: keyname }
image: { get_param: coreos_image }
flavor: { get_param: flavor }
name: "kubernetes_master"
networks:
- port:
get_resource: kubernetes_master_port
user_data_format: RAW
config_drive: "true"
user_data:
str_replace:
template: |
#cloud-config
write_files:
- path: /run/setup_etcd_cloudinit_conf.sh
permissions: '0755'
content: |
#!/bin/bash
ENV="/etc/environment"
# Test for RW access to $1
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV
exit 1
fi
# Setup environment target
sed -i -e '/^COREOS_PUBLIC_IPV4=/d' \
-e '/^COREOS_PRIVATE_IPV4=/d' \
"${ENV}"
# We spin loop until the the IP addresses are set
function get_ip () {
IF=$1
IP=
while [ 1 ]; do
IP=$(ifconfig $IF | awk '/inet / {print $2}')
if [ "$IP" != "" ]; then
break
fi
sleep .1
done
echo $IP
}
# Echo results of IP queries to environment file as soon as network interfaces
# get assigned IPs
echo COREOS_PUBLIC_IPV4=$(get_ip eth0) >> $ENV # Also assigned to same IP
echo COREOS_PRIVATE_IPV4=$(get_ip eth0) >> $ENV #eno1 should be changed to your device name
A=`cat /etc/environment | grep COREOS_PRIVATE_IPV4 | cut -f2 -d "="`
sed -i "s#=:#=${A}:#g" /run/systemd/system/etcd.service.d/20-cloudinit.conf
systemctl daemon-reload
- path: /run/get_discovery_ip_port.sh
permissions: '0755'
content: |
#!/bin/bash
# Sets up environment file with the discovery node's IP &
# port so # that in Overlode's template
# [email protected] it can be passed
# in as an argument
/usr/bin/cat /run/systemd/system/etcd.service.d/20-cloudinit.conf | /usr/bin/grep -i discovery | /usr/bin/cut -f3 -d"=" | /usr/bin/awk -F '/v' '{print $1}' > /run/discovery_ip_port
/usr/bin/sed -i 's/^/DISCOVERY_IP_PORT=/' /run/discovery_ip_port
coreos:
etcd:
name: kubernetes_master
discovery: http://%discovery%:2379/v2/keys/discovery/%uuid%
addr: $private_ipv4:4001
peer-addr: $private_ipv4:7001
fleet:
public-ip: $private_ipv4
metadata: kubernetes_role=master
update:
group: stable
reboot-strategy: off
units:
- name: setup-etcd-cloudinit-conf.service
command: start
runtime: true
content: |
[Unit]
Description=Setup environment with private (and public) IP addresses
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/run/setup_etcd_cloudinit_conf.sh
- name: etcd.service
command: start
- name: fleet.service
command: start
- name: get_discovery_ip_port.service
command: start
runtime: true
content: |
# Runs get_discovery_ip_port.sh to have discovery IP &
# port ready for consumption by overlord when creating
# the Kubernetes' [email protected] template
[Unit]
After=network-online.target
Requires=network-online.target
[Service]
ExecStart=/usr/bin/bash /run/get_discovery_ip_port.sh
- name: flannel-install.service
command: start
content: |
# Installs flannel
[Unit]
After=network-online.target
Requires=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/wget -N -P /opt/bin %flannel_url%
ExecStart=/usr/bin/chmod +x /opt/bin/flanneld
- name: flannel.service
command: start
content: |
# Configures & starts flannel
[Unit]
After=network-online.target etcd.service flannel-install.service
Requires=network-online.target etcd.service flannel-install.service
[Service]
ExecStartPre=/usr/bin/etcdctl --endpoint=%discovery%:4001 mk /coreos.com/network/config '{"Network":"10.244.0.0/15", "Backend": {"Type": "vxlan"}}'
ExecStart=/opt/bin/flanneld -etcd-endpoints=http://%discovery%:4001 -iface=%kubernetes_net_interface%
Restart=always
RestartSec=5s
- name: flannel-env.path
command: start
content: |
# Ensures flannel env vars are set to use with Docker
[Path]
PathExists=/run/flannel/subnet.env
Unit=docker.service
- name: docker.service
command: start
content: |
# Starts new docker server that uses flannel
[Unit]
After=flannel-env.path network-online.target flannel.service
Requires=flannel-env.path network-online.target flannel.service
Description=Docker Application Container Engine
[Service]
EnvironmentFile=/run/flannel/subnet.env
ExecStartPre=/bin/mount --make-rprivate /
ExecStartPre=/usr/bin/systemctl kill docker.service
ExecStart=/usr/bin/docker -d --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
[Install]
WantedBy=multi-user.target
- name: generate-k8s-certs.service
command: start
content: |
[Unit]
Description=Generate Kubernetes API Server certificates
ConditionPathExists=!/srv/kubernetes/.certs.lock
Requires=network-online.target
After=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/etc/environment
ExecStartPre=-/usr/sbin/groupadd -r kube-cert
ExecStartPre=/usr/bin/wget -q -N -P /opt/bin https://gist.githubusercontent.com/metral/0731466d7b35d4020dfd/raw/2fb0c5858e5eaefdf6b45b9492f1d6c75a7815b0/make-ca-cert.sh
ExecStartPre=/usr/bin/chmod u=rwx,go= /opt/bin/make-ca-cert.sh
ExecStart=/opt/bin/make-ca-cert.sh ${COREOS_PRIVATE_IPV4} IP:10.1.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.freeletics.local
ExecStart=/opt/bin/make-ca-cert.sh ${COREOS_PRIVATE_IPV4} IP:10.1.0.1,IP:127.0.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.freeletics.local,DNS:localhost
RemainAfterExit=true
params:
"%discovery%": { get_attr: [discovery, networks, corekube, 0] }
"%uuid%": { get_attr: [coreos_cluster_uuid, value] }
"%flannel_url%": { get_param: flannel_url }
"%kubernetes_net_interface%": { get_param: kubernetes_net_interface }
kubernetes_minions:
type: "OS::Heat::ResourceGroup"
properties:
count: { get_param: kubernetes_minion_count }
resource_def:
type: OS::Nova::Server
properties:
key_name: { get_param: keyname }
image: { get_param: coreos_image }
flavor: { get_param: flavor }
name: kubernetes_minion_%index%
networks:
- network: { get_resource: corekube_network }
security_groups:
- get_resource: stack_security
user_data_format: RAW
config_drive: "true"
user_data:
str_replace:
template: |
#cloud-config
write_files:
- path: /run/setup_etcd_cloudinit_conf.sh
permissions: '0755'
content: |
#!/bin/bash
ENV="/etc/environment"
# Test for RW access to $1
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV
exit 1
fi
# Setup environment target
sed -i -e '/^COREOS_PUBLIC_IPV4=/d' \
-e '/^COREOS_PRIVATE_IPV4=/d' \
"${ENV}"
# We spin loop until the the IP addresses are set
function get_ip () {
IF=$1
IP=
while [ 1 ]; do
IP=$(ifconfig $IF | awk '/inet / {print $2}')
if [ "$IP" != "" ]; then
break
fi
sleep .1
done
echo $IP
}
# Echo results of IP queries to environment file as soon as network interfaces
# get assigned IPs
echo COREOS_PUBLIC_IPV4=$(get_ip eth0) >> $ENV # Also assigned to same IP
echo COREOS_PRIVATE_IPV4=$(get_ip eth0) >> $ENV #eno1 should be changed to your device name
A=`cat /etc/environment | grep COREOS_PRIVATE_IPV4 | cut -f2 -d "="`
sed -i "s#=:#=${A}:#g" /run/systemd/system/etcd.service.d/20-cloudinit.conf
systemctl daemon-reload
coreos:
etcd:
name: kubernetes_minion_%index%
discovery: http://%discovery%:2379/v2/keys/discovery/%uuid%
addr: $private_ipv4:4001
peer-addr: $private_ipv4:7001
fleet:
public-ip: $private_ipv4
metadata: kubernetes_role=minion
update:
group: stable
reboot-strategy: off
units:
- name: setup-etcd-cloudinit-conf.service
command: start
runtime: true
content: |
[Unit]
Description=Setup environment with private (and public) IP addresses
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/run/setup_etcd_cloudinit_conf.sh
- name: etcd.service
command: start
- name: fleet.service
command: start
- name: flannel-install.service
command: start
content: |
# Installs flannel
[Unit]
After=network-online.target
Requires=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/wget -N -P /opt/bin %flannel_url%
ExecStart=/usr/bin/chmod +x /opt/bin/flanneld
- name: flannel.service
command: start
content: |
# Configures & starts flannel
[Unit]
After=etcd.service flannel-install.service
Requires=etcd.service flannel-install.service
[Service]
ExecStart=/opt/bin/flanneld -etcd-endpoints=http://%discovery%:4001 -iface=%kubernetes_net_interface%
Restart=always
RestartSec=5s
- name: flannel-env.path
command: start
content: |
# Ensures flannel env vars are set to use with Docker
[Path]
PathExists=/run/flannel/subnet.env
Unit=docker.service
- name: docker.service
command: start
content: |
# Starts new docker server that uses flannel
[Unit]
After=flannel-env.path network-online.target flannel.service
Requires=flannel-env.path network-online.target flannel.service
Description=Docker Application Container Engine
[Service]
EnvironmentFile=/run/flannel/subnet.env
ExecStartPre=/bin/mount --make-rprivate /
ExecStartPre=/usr/bin/systemctl kill docker.service
ExecStart=/usr/bin/docker -d --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
[Install]
WantedBy=multi-user.target
params:
"%discovery%": { get_attr: [discovery, networks, corekube, 0] }
"%uuid%": { get_attr: [coreos_cluster_uuid, value] }
"%flannel_url%": { get_param: flannel_url }
"%kubernetes_net_interface%": { get_param: kubernetes_net_interface }
outputs:
discovery_ip:
value: { get_attr: [ discovery, first_address ] }
description: The IP of the Discovery
overlord_ip:
value: { get_attr: [ overlord, first_address ] }
description: The IP of the Overlord
master_ip:
value: { get_attr: [ kubernetes_master, first_address ] }
description: The IP of the Kubernetes Master
minion_ips:
value: { get_attr: [ kubernetes_minions, first_address ] }
description: The IP of the Kubernetes Minions
latest coreos image requires docker daemon instead of docker -d
ExecStart=/usr/bin/docker daemon --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
working file
heat_template_version: 2013-05-23
description: >
Deploy Kubernetes on a CoreOS cluster that operates on an
OpenStack Powered Cloud (Openstack Kilo | Rackspace Private Cloud)
Maintainer - Mike Metral <[email protected]> | Github: @metral
parameters:
kubernetes_minion_count:
description: Number of CoreOS machines to deploy as Kubernetes Minion
type: number
default: 3
constraints:
- range:
min: 1
max: 12
description: Must be between 2 and 12 servers.
keyname:
type: string
description: Name of keypair to be used for compute instance
flavor:
type: string
default: m1.medium
constraints:
- allowed_values:
- m1.medium
- m1.large
- m1.xlarge
description: |
Must be a valid flavor
coreos_image:
type: string
description: Rackspace Cloud Servers CoreOS Stable (808.0.0) UUID
default: "CoreOS"
git_command:
type: string
description: Git repo checkout command
default: "/usr/bin/git clone https://github.com/metral/overlord ; /usr/bin/git -C overlord checkout -qf 199028526132ff9491d26c00b2fd9b9b9d7239a8"
flannel_url:
type: string
description: Flannel (0.5.5) Binary URL
default: "http://0afbabe78c1b1691fad4-21af6827a69c3ad910ccdb6595a59d19.r49.cf1.rackcdn.com/flanneld"
private_discovery_etcd_version:
type: string
description: Version of etcd used for Private Discovery node
default: "2.2.2"
discovery_net_interface:
type: string
description: Host network interface for which the Discovery node will operate over
default: "eth0"
kubernetes_net_interface:
type: string
description: Host network interface for which Kubernetes' overlay will operate over
default: "eth0"
timeout:
description: Wait condition timeout seconds
type: number
default: 600
floating-network-id:
type: string
label: Floating Network ID
description: UUID of the external network. The private network created by this stack will route to this network. Any floating IP addresses needed by this stack will also route to this network.
resources:
coreos_cluster_uuid:
type: OS::Heat::RandomString
stack_security:
type: OS::Neutron::SecurityGroup
properties:
name: "corekube"
rules:
- remote_ip_prefix: 0.0.0.0/0
protocol: tcp
port_range_min: 0
port_range_max: 65535
- remote_ip_prefix: 0.0.0.0/0
protocol: udp
port_range_min: 0
port_range_max: 65535
- remote_ip_prefix: 0.0.0.0/0
protocol: icmp
corekube_network:
type: OS::Neutron::Net
properties:
name: "corekube"
corekube_subnet:
type: OS::Neutron::Subnet
properties:
name: "corekube_subnet"
network_id:
get_resource: corekube_network
cidr: "10.10.12.0/24"
gateway_ip: "10.10.12.1"
allocation_pools: [{"start": "10.10.12.2", "end": "10.10.12.254"}]
dns_nameservers:
- 8.8.8.8
- 8.8.4.4
corekube_router:
type: OS::Neutron::Router
properties:
name: "corekube_router"
external_gateway_info:
enable_snat: true
network:
get_param: floating-network-id
corekube_router_interface:
type: OS::Neutron::RouterInterface
properties:
router_id:
get_resource: corekube_router
subnet_id:
get_resource: corekube_subnet
discovery_wait_handle:
type: OS::Heat::WaitConditionHandle
discovery_wait_condition:
type: OS::Heat::WaitCondition
properties:
handle: { get_resource: discovery_wait_handle }
timeout: { get_param: timeout }
discovery_floatingip:
type: OS::Neutron::FloatingIP
properties:
floating_network_id:
get_param: floating-network-id
discovery_port:
type: OS::Neutron::Port
properties:
security_groups:
- get_resource: stack_security
network_id:
get_resource: corekube_network
discovery_floatingip_association:
type: OS::Neutron::FloatingIPAssociation
properties:
floatingip_id:
get_resource: discovery_floatingip
port_id:
get_resource: discovery_port
discovery:
type: OS::Nova::Server
properties:
key_name: { get_param: keyname }
image: { get_param: coreos_image }
flavor: { get_param: flavor }
name: "discovery"
networks:
- port:
get_resource: discovery_port
user_data_format: RAW
config_drive: "true"
user_data:
str_replace:
template: |
#cloud-config
write_files:
- path: /run/get_discovery_interface_ip.sh
permissions: '0755'
content: |
#!/bin/bash
# Get's the IP of the interface that discovery will be
# accessible over
DISCOVERY_IF=%discovery_net_interface%
/usr/bin/ip -4 addr show $DISCOVERY_IF | /usr/bin/awk '/inet/ {print $2}' | /usr/bin/cut -d/ -f1 > /run/IP
/usr/bin/sed -i 's/^/IP=/' /run/IP
- path: /run/wcnotify.sh
permissions: '0755'
content: |
#!/bin/bash
%wc_notify% --data-binary '{"status": "SUCCESS"}'
coreos:
update:
group: stable
reboot-strategy: off
units:
- name: private-discovery-setup.service
command: start
content: |
[Unit]
After=network-online.target
Requires=network-online.target
[Service]
Type=forking
ExecStart=/usr/bin/bash /run/get_discovery_interface_ip.sh
- name: private-discovery.service
command: start
content: |
[Unit]
After=network-online.target private-discovery-setup.service
Requires=network-online.target private-discovery-setup.service
[Service]
EnvironmentFile=/run/IP
Restart=on-failure
RestartSec=5s
ExecStartPre=/usr/bin/docker pull quay.io/coreos/etcd:v%private_discovery_etcd_version%
ExecStart=/usr/bin/docker run -d --name discovery \
-p 2379:2379 -p 2380:2380 \
-v /usr/share/ca-certificates/:/etc/ssl/certs \
--net host quay.io/coreos/etcd:v%private_discovery_etcd_version% -name discovery \
-initial-advertise-peer-urls http://${IP}:2380,http://${IP}:7001 \
-listen-peer-urls http://${IP}:2380,http://${IP}:7001 \
-initial-cluster discovery=http://${IP}:2380,discovery=http://${IP}:7001 \
-advertise-client-urls http://${IP}:2379,http://${IP}:4001 \
-listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
- name: private-discovery-done.service
command: start
content: |
# Notifiy wait condition
[Unit]
After=network-online.target private-discovery.service
Requires=network-online.target private-discovery.service
[Service]
ExecStart=/run/wcnotify.sh
params:
"%wc_notify%": { get_attr: [discovery_wait_handle, curl_cli] }
"%discovery_net_interface%": { get_param: discovery_net_interface }
"%private_discovery_etcd_version%": { get_param: private_discovery_etcd_version}
overlord_wait_handle:
type: OS::Heat::WaitConditionHandle
overlord_wait_condition:
type: OS::Heat::WaitCondition
properties:
handle: {get_resource: overlord_wait_handle}
timeout: {get_param: timeout}
overlord_floatingip:
type: OS::Neutron::FloatingIP
properties:
floating_network_id:
get_param: floating-network-id
overlord_port:
type: OS::Neutron::Port
properties:
security_groups:
- get_resource: stack_security
network_id:
get_resource: corekube_network
overlord_floatingip_association:
type: OS::Neutron::FloatingIPAssociation
properties:
floatingip_id:
get_resource: overlord_floatingip
port_id:
get_resource: overlord_port
overlord:
type: OS::Nova::Server
properties:
key_name: { get_param: keyname }
image: { get_param: coreos_image }
flavor: { get_param: flavor }
name: "overlord"
networks:
- port:
get_resource: overlord_port
user_data_format: RAW
config_drive: "true"
user_data:
str_replace:
template: |
#cloud-config
write_files:
- path: /run/wcnotify.sh
permissions: '0755'
content: |
#!/bin/bash
%wc_notify% --data-binary '{"status": "SUCCESS"}'
- path: /run/flannel_docker_opts.env
permissions: '0755'
content: |
DOCKER_OPTS="--mtu=1404"
- path: /run/setup_etcd_cloudinit_conf.sh
permissions: '0755'
content: |
#!/bin/bash
ENV="/etc/environment"
# Test for RW access to $1
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV
exit 1
fi
# Setup environment target
sed -i -e '/^COREOS_PUBLIC_IPV4=/d' \
-e '/^COREOS_PRIVATE_IPV4=/d' \
"${ENV}"
# We spin loop until the the IP addresses are set
function get_ip () {
IF=$1
IP=
while [ 1 ]; do
IP=$(ifconfig $IF | awk '/inet / {print $2}')
if [ "$IP" != "" ]; then
break
fi
sleep .1
done
echo $IP
}
# Echo results of IP queries to environment file as soon as network interfaces
# get assigned IPs
echo COREOS_PUBLIC_IPV4=$(get_ip eth0) >> $ENV # Also assigned to same IP
echo COREOS_PRIVATE_IPV4=$(get_ip eth0) >> $ENV #eno1 should be changed to your device name
A=`cat /etc/environment | grep COREOS_PRIVATE_IPV4 | cut -f2 -d "="`
sed -i "s#=:#=${A}:#g" /run/systemd/system/etcd.service.d/20-cloudinit.conf
systemctl daemon-reload
coreos:
etcd:
name: overlord
discovery: http://%discovery%:2379/v2/keys/discovery/%uuid%
addr: $private_ipv4:4001
peer-addr: $private_ipv4:7001
update:
group: stable
reboot-strategy: off
units:
- name: setup-etcd-cloudinit-conf.service
command: start
runtime: true
content: |
[Unit]
Description=Setup environment with private (and public) IP addresses
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/run/setup_etcd_cloudinit_conf.sh
- name: etcd.service
command: start
- name: fleet.socket
command: start
content: |
[Socket]
# Talk to the API over a Unix domain socket (default)
ListenStream=/var/run/fleet.sock
# Talk to the API over an exposed port
ListenStream=10001
Service=fleet-local.service
[Install]
WantedBy=sockets.target
- name: fleet-local.service
command: start
content: |
# fleet-local is kicked off by fleet.socket after API port
# is opened
[Unit]
Description=fleet-local
Wants=etcd.service
Requires=etcd.service
[Service]
Environment=FLEET_PUBLIC_IP=$private_ipv4
Environment=FLEET_METADATA=kubernetes_role=overlord
ExecStart=/usr/bin/fleet
Restart=always
RestartSec=10s
- name: overlord.service
command: start
content: |
# Overlord / logic layer service to deploy kubernetes to
# the cluster
[Unit]
After=network-online.target etcd.service
Requires=network-online.target etcd.service
[Service]
WorkingDirectory=/root
Environment="DIR=overlord"
ExecStartPre=/usr/bin/rm -rf $DIR
ExecStartPre=%git_command%
ExecStart=/usr/bin/bash ${DIR}/build_run.sh
- name: overlord-done.service
command: start
content: |
# Notifiy wait condition
[Unit]
After=network-online.target overlord.service
Requires=network-online.target overlord.service
[Service]
ExecStart=/run/wcnotify.sh
params:
"%discovery%": { get_attr: [discovery, networks, corekube, 0] }
"%uuid%": { get_attr: [coreos_cluster_uuid, value] }
"%git_command%": { get_param: git_command }
"%wc_notify%": { get_attr: [overlord_wait_handle, curl_cli] }
kubernetes_master_floatingip:
type: OS::Neutron::FloatingIP
properties:
floating_network_id:
get_param: floating-network-id
kubernetes_master_port:
type: OS::Neutron::Port
properties:
security_groups:
- get_resource: stack_security
network_id:
get_resource: corekube_network
kubernetes_master_floatingip_association:
type: OS::Neutron::FloatingIPAssociation
properties:
floatingip_id:
get_resource: kubernetes_master_floatingip
port_id:
get_resource: kubernetes_master_port
kubernetes_master:
type: OS::Nova::Server
properties:
key_name: { get_param: keyname }
image: { get_param: coreos_image }
flavor: { get_param: flavor }
name: "kubernetes_master"
networks:
- port:
get_resource: kubernetes_master_port
user_data_format: RAW
config_drive: "true"
user_data:
str_replace:
template: |
#cloud-config
write_files:
- path: /run/setup_etcd_cloudinit_conf.sh
permissions: '0755'
content: |
#!/bin/bash
ENV="/etc/environment"
# Test for RW access to $1
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV
exit 1
fi
# Setup environment target
sed -i -e '/^COREOS_PUBLIC_IPV4=/d' \
-e '/^COREOS_PRIVATE_IPV4=/d' \
"${ENV}"
# We spin loop until the the IP addresses are set
function get_ip () {
IF=$1
IP=
while [ 1 ]; do
IP=$(ifconfig $IF | awk '/inet / {print $2}')
if [ "$IP" != "" ]; then
break
fi
sleep .1
done
echo $IP
}
# Echo results of IP queries to environment file as soon as network interfaces
# get assigned IPs
echo COREOS_PUBLIC_IPV4=$(get_ip eth0) >> $ENV # Also assigned to same IP
echo COREOS_PRIVATE_IPV4=$(get_ip eth0) >> $ENV #eno1 should be changed to your device name
A=`cat /etc/environment | grep COREOS_PRIVATE_IPV4 | cut -f2 -d "="`
sed -i "s#=:#=${A}:#g" /run/systemd/system/etcd.service.d/20-cloudinit.conf
systemctl daemon-reload
- path: /run/get_discovery_ip_port.sh
permissions: '0755'
content: |
#!/bin/bash
# Sets up environment file with the discovery node's IP &
# port so # that in Overlode's template
# [email protected] it can be passed
# in as an argument
/usr/bin/cat /run/systemd/system/etcd.service.d/20-cloudinit.conf | /usr/bin/grep -i discovery | /usr/bin/cut -f3 -d"=" | /usr/bin/awk -F '/v' '{print $1}' > /run/discovery_ip_port
/usr/bin/sed -i 's/^/DISCOVERY_IP_PORT=/' /run/discovery_ip_port
coreos:
etcd:
name: kubernetes_master
discovery: http://%discovery%:2379/v2/keys/discovery/%uuid%
addr: $private_ipv4:4001
peer-addr: $private_ipv4:7001
fleet:
public-ip: $private_ipv4
metadata: kubernetes_role=master
update:
group: stable
reboot-strategy: off
units:
- name: setup-etcd-cloudinit-conf.service
command: start
runtime: true
content: |
[Unit]
Description=Setup environment with private (and public) IP addresses
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/run/setup_etcd_cloudinit_conf.sh
- name: etcd.service
command: start
- name: fleet.service
command: start
- name: get_discovery_ip_port.service
command: start
runtime: true
content: |
# Runs get_discovery_ip_port.sh to have discovery IP &
# port ready for consumption by overlord when creating
# the Kubernetes' [email protected] template
[Unit]
After=network-online.target
Requires=network-online.target
[Service]
ExecStart=/usr/bin/bash /run/get_discovery_ip_port.sh
- name: flannel-install.service
command: start
content: |
# Installs flannel
[Unit]
After=network-online.target
Requires=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/wget -N -P /opt/bin %flannel_url%
ExecStart=/usr/bin/chmod +x /opt/bin/flanneld
- name: flannel.service
command: start
content: |
# Configures & starts flannel
[Unit]
After=network-online.target etcd.service flannel-install.service
Requires=network-online.target etcd.service flannel-install.service
[Service]
ExecStartPre=/usr/bin/etcdctl --endpoint=%discovery%:4001 mk /coreos.com/network/config '{"Network":"10.244.0.0/15", "Backend": {"Type": "vxlan"}}'
ExecStart=/opt/bin/flanneld -etcd-endpoints=http://%discovery%:4001 -iface=%kubernetes_net_interface%
Restart=always
RestartSec=5s
- name: flannel-env.path
command: start
content: |
# Ensures flannel env vars are set to use with Docker
[Path]
PathExists=/run/flannel/subnet.env
Unit=docker.service
- name: docker.service
command: start
content: |
# Starts new docker server that uses flannel
[Unit]
After=flannel-env.path network-online.target flannel.service
Requires=flannel-env.path network-online.target flannel.service
Description=Docker Application Container Engine
[Service]
EnvironmentFile=/run/flannel/subnet.env
ExecStartPre=/bin/mount --make-rprivate /
ExecStartPre=/usr/bin/systemctl kill docker.service
ExecStart=/usr/bin/docker daemon --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
[Install]
WantedBy=multi-user.target
- name: generate-k8s-certs.service
command: start
content: |
[Unit]
Description=Generate Kubernetes API Server certificates
ConditionPathExists=!/srv/kubernetes/.certs.lock
Requires=network-online.target
After=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/etc/environment
ExecStartPre=-/usr/sbin/groupadd -r kube-cert
ExecStartPre=/usr/bin/wget -q -N -P /opt/bin https://gist.githubusercontent.com/metral/0731466d7b35d4020dfd/raw/2fb0c5858e5eaefdf6b45b9492f1d6c75a7815b0/make-ca-cert.sh
ExecStartPre=/usr/bin/chmod u=rwx,go= /opt/bin/make-ca-cert.sh
ExecStart=/opt/bin/make-ca-cert.sh ${COREOS_PRIVATE_IPV4} IP:10.1.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.freeletics.local
ExecStart=/opt/bin/make-ca-cert.sh ${COREOS_PRIVATE_IPV4} IP:10.1.0.1,IP:127.0.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.freeletics.local,DNS:localhost
RemainAfterExit=true
params:
"%discovery%": { get_attr: [discovery, networks, corekube, 0] }
"%uuid%": { get_attr: [coreos_cluster_uuid, value] }
"%flannel_url%": { get_param: flannel_url }
"%kubernetes_net_interface%": { get_param: kubernetes_net_interface }
kubernetes_minions:
type: "OS::Heat::ResourceGroup"
properties:
count: { get_param: kubernetes_minion_count }
resource_def:
type: OS::Nova::Server
properties:
key_name: { get_param: keyname }
image: { get_param: coreos_image }
flavor: { get_param: flavor }
name: kubernetes_minion_%index%
networks:
- network: { get_resource: corekube_network }
security_groups:
- get_resource: stack_security
user_data_format: RAW
config_drive: "true"
user_data:
str_replace:
template: |
#cloud-config
write_files:
- path: /run/setup_etcd_cloudinit_conf.sh
permissions: '0755'
content: |
#!/bin/bash
ENV="/etc/environment"
# Test for RW access to $1
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV
exit 1
fi
# Setup environment target
sed -i -e '/^COREOS_PUBLIC_IPV4=/d' \
-e '/^COREOS_PRIVATE_IPV4=/d' \
"${ENV}"
# We spin loop until the the IP addresses are set
function get_ip () {
IF=$1
IP=
while [ 1 ]; do
IP=$(ifconfig $IF | awk '/inet / {print $2}')
if [ "$IP" != "" ]; then
break
fi
sleep .1
done
echo $IP
}
# Echo results of IP queries to environment file as soon as network interfaces
# get assigned IPs
echo COREOS_PUBLIC_IPV4=$(get_ip eth0) >> $ENV # Also assigned to same IP
echo COREOS_PRIVATE_IPV4=$(get_ip eth0) >> $ENV #eno1 should be changed to your device name
A=`cat /etc/environment | grep COREOS_PRIVATE_IPV4 | cut -f2 -d "="`
sed -i "s#=:#=${A}:#g" /run/systemd/system/etcd.service.d/20-cloudinit.conf
systemctl daemon-reload
coreos:
etcd:
name: kubernetes_minion_%index%
discovery: http://%discovery%:2379/v2/keys/discovery/%uuid%
addr: $private_ipv4:4001
peer-addr: $private_ipv4:7001
fleet:
public-ip: $private_ipv4
metadata: kubernetes_role=minion
update:
group: stable
reboot-strategy: off
units:
- name: setup-etcd-cloudinit-conf.service
command: start
runtime: true
content: |
[Unit]
Description=Setup environment with private (and public) IP addresses
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/run/setup_etcd_cloudinit_conf.sh
- name: etcd.service
command: start
- name: fleet.service
command: start
- name: flannel-install.service
command: start
content: |
# Installs flannel
[Unit]
After=network-online.target
Requires=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/wget -N -P /opt/bin %flannel_url%
ExecStart=/usr/bin/chmod +x /opt/bin/flanneld
- name: flannel.service
command: start
content: |
# Configures & starts flannel
[Unit]
After=etcd.service flannel-install.service
Requires=etcd.service flannel-install.service
[Service]
ExecStart=/opt/bin/flanneld -etcd-endpoints=http://%discovery%:4001 -iface=%kubernetes_net_interface%
Restart=always
RestartSec=5s
- name: flannel-env.path
command: start
content: |
# Ensures flannel env vars are set to use with Docker
[Path]
PathExists=/run/flannel/subnet.env
Unit=docker.service
- name: docker.service
command: start
content: |
# Starts new docker server that uses flannel
[Unit]
After=flannel-env.path network-online.target flannel.service
Requires=flannel-env.path network-online.target flannel.service
Description=Docker Application Container Engine
[Service]
EnvironmentFile=/run/flannel/subnet.env
ExecStartPre=/bin/mount --make-rprivate /
ExecStartPre=/usr/bin/systemctl kill docker.service
ExecStart=/usr/bin/docker daemon --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
[Install]
WantedBy=multi-user.target
params:
"%discovery%": { get_attr: [discovery, networks, corekube, 0] }
"%uuid%": { get_attr: [coreos_cluster_uuid, value] }
"%flannel_url%": { get_param: flannel_url }
"%kubernetes_net_interface%": { get_param: kubernetes_net_interface }
outputs:
discovery_ip:
value: { get_attr: [ discovery, first_address ] }
description: The IP of the Discovery
overlord_ip:
value: { get_attr: [ overlord, first_address ] }
description: The IP of the Overlord
master_ip:
value: { get_attr: [ kubernetes_master, first_address ] }
description: The IP of the Kubernetes Master
minion_ips:
value: { get_attr: [ kubernetes_minions, first_address ] }
description: The IP of the Kubernetes Minions
Setup Error / Unexpected end of JSON input
Hello all,
I'm having some trouble deploying out to Rackspace (RackConnect v3) using the rackconnect-v3 yaml.
Here is the error I am getting from the overlord after deployment:
2015/05/22 17:08:44 ------------------------------------------------
2015/05/22 17:08:44 Current # of machines discovered: (4)
2015/05/22 17:08:44 ------------------------------------------------
2015/05/22 17:08:44 Found machine:
2015/05/22 17:08:44 -- ID: 3c59a19de5a848ffb7969d22e02f01b9
2015/05/22 17:08:44 -- IP: 172.19.1.10
2015/05/22 17:08:44 -- Metadata: (kubernetes_role => master)
2015/05/22 17:08:44 Created all unit files for: 3c59a19de5a848ffb7969d22e02f01b9
2015/05/22 17:08:44 ------------------------------------------------
2015/05/22 17:08:44 Found machine:
2015/05/22 17:08:44 -- ID: 6136462ad4e547608456f2ba20cb310e
2015/05/22 17:08:44 -- IP: 172.19.1.9
2015/05/22 17:08:44 -- Metadata: (kubernetes_role => minion)
2015/05/22 17:08:44 Created all unit files for: 6136462ad4e547608456f2ba20cb310e
2015/05/22 17:08:44 ------------------------------------------------
2015/05/22 17:08:44 Found machine:
2015/05/22 17:08:44 -- ID: c047453d369e4db7a8054a70deabc726
2015/05/22 17:08:44 -- IP: 172.19.1.7
2015/05/22 17:08:44 -- Metadata: (kubernetes_role => minion)
2015/05/22 17:08:44 Created all unit files for: c047453d369e4db7a8054a70deabc726
2015/05/22 17:08:44 ------------------------------------------------
2015/05/22 17:08:44 Found machine:
2015/05/22 17:08:44 -- ID: 824f77d9d59145919b33ae7e61d15186
2015/05/22 17:08:44 -- IP: 172.19.1.8
2015/05/22 17:08:44 -- Metadata: (kubernetes_role => minion)
2015/05/22 17:08:44 Created all unit files for: 824f77d9d59145919b33ae7e61d15186
2015/05/22 17:08:46 [Error] in github.com/metral/overlord/lib.Main[/gopath/src/github.com/metral/overlord/lib/utils.go:52] unexpected end of JSON input
Any help would be appreciated! Thanks for this repo.
Stack Build Error
Hi,
When i use the latest version of https://github.com/metral/corekube/blob/master/corekube-cloudservers.yaml i get an error when creating the stack (rackspace). Any idea?
Resource CREATE failed: resources.kubernetes_minions: Property error: resources[1].properties.networks[0].network: Error validating value '00000000-0000-0000-0000-000000000000': SSL certificate validation has failed: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failedThanks,
Sven
Error parsing template heat template
hey,
Im using openstack mitaka
heat version 1.1.0
I cant create stack from corekube-openstack.yaml
The error is:
Error parsing template mapping values are not allowed here
in "", line 3, column 13:
description: >
Rack Connect
Might be worth mentioning that I had to add two firewall rules to make this work with RackConnect. It works as-is without RackConnect, but in an RC environment the following rules are required:
Dedicated -> Cloud Source=Network [192.168.3.0/24] Destination Type [All] Protocol [All]
Dedicated -> Cloud Source=Network [10.244.0.0/15] Destination Type [All] Protocol [All]
Feel free to close this issue if RackConnect is too esoteric to mention in the Readme.
Thanks for corekube!
Resource CREATE failed: Invalid value for port 0
Hi, I am getting this error in HEAT when using latest corekube-openstack.yaml. It is related to security group param port_range_min: 0. When I change this parameter value to 1 all goes well :)
Resource CREATE failed: BadRequest: resources.stack_security: Invalid value for port 0
Versions:
- openstack - mitaka
- heat client - 0.2.8
Overlord doesn't finish completion due to failed unit in fleet
Overlord continues to wait for a node to finish running its service but it will indefinitely sit there waiting and not continue.
The reason is that a unit or multiple units failed on a node as shown when doing a fleetctl list-units
When you log into the failed node and issue a fleetctl status <unit_name> it gives an error due to not being able to pull the binaries from the Internet. This is due to the fact that systemd-networkd was restarted too many times and it exited from continuing to attempt restarting the service as shown in systemdctl status systemd-networkd. The issue has to do with the issuance of multiple network devices requiring a restart of systemd-networkd and its restarting too many times for it to be happy.
This issue happens every so often but not always.
The simple work around unfortunately is to destroy the stack all-together via Heat and create a new stack or restart the systemd-networkd unit as well as any other units on the failed nodes, and then observing of the logs of the overlord to make sure it completed
Failed to start flannel.service
I'm trying to use the public cloud heat template without any changes with the Rackspace orchestration service but the etcd cluster never comes up. Doing etcdctl cluster-health on the discovery machine says all is well. But doing the same on kubernetes master yeilds the following:
cluster may be unhealthy: failed to list members
Error: unexpected status code 404
Doing journalctl -f outputs this:
Feb 04 15:33:06 kubernetes-master etcdctl[2224]: Error: unexpected status code 404
Feb 04 15:33:06 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4
Feb 04 15:33:06 kubernetes-master systemd[1]: Failed to start flannel.service.
Feb 04 15:33:06 kubernetes-master systemd[1]: flannel.service: Unit entered failed state.
Feb 04 15:33:06 kubernetes-master systemd[1]: flannel.service: Failed with result 'exit-code'.
Feb 04 15:33:11 kubernetes-master systemd[1]: flannel.service: Service hold-off time over, scheduling restart.
Feb 04 15:33:11 kubernetes-master systemd[1]: Stopped flannel.service.
Feb 04 15:33:11 kubernetes-master systemd[1]: Starting flannel.service...
Feb 04 15:33:11 kubernetes-master etcdctl[2233]: Error: unexpected status code 404
Feb 04 15:33:11 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4
Feb 04 15:33:11 kubernetes-master systemd[1]: Failed to start flannel.service.
Feb 04 15:33:11 kubernetes-master systemd[1]: flannel.service: Unit entered failed state.
Feb 04 15:33:11 kubernetes-master systemd[1]: flannel.service: Failed with result 'exit-code'.
Feb 04 15:33:16 kubernetes-master systemd[1]: flannel.service: Service hold-off time over, scheduling restart.
Feb 04 15:33:16 kubernetes-master systemd[1]: Stopped flannel.service.
Feb 04 15:33:16 kubernetes-master systemd[1]: Starting flannel.service...
Feb 04 15:33:16 kubernetes-master etcdctl[2241]: Error: unexpected status code 404
Feb 04 15:33:16 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4
Feb 04 15:33:16 kubernetes-master systemd[1]: Failed to start flannel.service.
Feb 04 15:33:16 kubernetes-master systemd[1]: flannel.service: Unit entered failed state.
Feb 04 15:33:16 kubernetes-master systemd[1]: flannel.service: Failed with result 'exit-code'.
Feb 04 15:33:17 kubernetes-master kube-scheduler[1709]: I0204 15:33:17.309879 1709 event.go:206] Event(api.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns-v8-fprbi", UID:"f7a1fc1d-cb53-11e5-8199-bc764e11c1a6", APIVersion:"v1", ResourceVersion:"19", FieldPath:""}): reason: 'FailedScheduling' no nodes available to schedule pods
Feb 04 15:33:18 kubernetes-master systemd[1]: Starting Generate /run/coreos/motd...
Feb 04 15:33:18 kubernetes-master systemd[1]: Started Generate /run/coreos/motd.
Feb 04 15:33:22 kubernetes-master systemd[1]: flannel.service: Service hold-off time over, scheduling restart.
Feb 04 15:33:22 kubernetes-master systemd[1]: Stopped flannel.service.
Feb 04 15:33:22 kubernetes-master systemd[1]: Starting flannel.service...
Feb 04 15:33:22 kubernetes-master etcdctl[2257]: Error: unexpected status code 404
Feb 04 15:33:22 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4
Feb 04 15:33:22 kubernetes-master systemd[1]: Failed to start flannel.service.
Feb 04 15:33:22 kubernetes-master systemd[1]: flannel.service: Unit entered failed state.
Feb 04 15:33:22 kubernetes-master systemd[1]: flannel.service: Failed with result 'exit-code'.
Feb 04 15:33:27 kubernetes-master systemd[1]: flannel.service: Service hold-off time over, scheduling restart.
Feb 04 15:33:27 kubernetes-master systemd[1]: Stopped flannel.service.
Feb 04 15:33:27 kubernetes-master systemd[1]: Starting flannel.service...
Feb 04 15:33:27 kubernetes-master etcdctl[2265]: Error: unexpected status code 404
Feb 04 15:33:27 kubernetes-master systemd[1]: flannel.service: Control process exited, code=exited status=4
Feb 04 15:33:27 kubernetes-master systemd[1]: Failed to start flannel.service.
Inspecting just the etcd service looks just fine (again on the kubernetes master machine):
$ journalctl -u etcd.service
-- Logs begin at Thu 2016-02-04 15:25:00 UTC, end at Thu 2016-02-04 15:53:25 UTC. --
Feb 04 15:25:15 kubernetes-master systemd[1]: Started etcd.
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.306 INFO | Discovery via http://10.210.193.89:2379 using prefix discovery/1mn3MOFVbKnC02nbnhi2BZ7nGAXjPkSR.
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.313 INFO | Discovery found peers [http://10.209.104.221:7001]
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.313 INFO | Discovery fetched back peer list: [10.209.104.221:7001]
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.320 INFO | Send Join Request to http://10.209.104.221:7001/join
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.330 INFO | kubernetes_master joined the cluster via peer 10.209.104.221:7001
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.335 INFO | etcd server [name kubernetes_master, listen on :4001, advertised url http://10.209.102.21:4001]
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.335 INFO | peer server [name kubernetes_master, listen on :7001, advertised url http://10.209.102.21:7001]
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.336 INFO | kubernetes_master starting in peer mode
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.336 INFO | kubernetes_master: state changed from 'initialized' to 'follower'.
Feb 04 15:25:15 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:15.387 INFO | kubernetes_master: peer added: 'overlord'
Feb 04 15:25:16 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:16.785 INFO | kubernetes_master: peer added: 'kubernetes_minion_0'
Feb 04 15:25:18 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:18.402 INFO | kubernetes_master: peer added: 'kubernetes_minion_1'
Feb 04 15:25:19 kubernetes-master etcd[1256]: [etcd] Feb 4 15:25:19.179 INFO | kubernetes_master: peer added: 'kubernetes_minion_2'
For days I've been trying to diagnose the issue with no luck. Is this a template issue? I've also noticed any Docker commands just hang, but only on machines where the etcd cluster is not working.
CoreOS 835.9.0
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.