rancher / vagrant Goto Github PK

View Code? Open in Web Editor NEW

81.0 10.0 56.0 227 KB

Vagrant file to stand up a Local Rancher install with 3 nodes

Ruby 13.98% Shell 86.02%

vagrant's Introduction

localranchervagrant

Vagrant files to stand up a Local Rancher install with 3 nodes

This runs RancherOS as the base OS for the nodes which doesn't have the guest tools for Virtualbox installed

Start the cluster and the Rancher UI will become accessible on http://172.22.101.100

To see the contents of the registry cache proxy, navigate to http://172.22.101.100:5000/v2/_catalog

The default file will bring up a cattle environment. You can change this by editing orchestrator in the config file.

Usage

To use this you must have vagrant installed, which can be obtained from www.vagrantup.com

clone the directory and then run vagrant up

This has been tested with vagrant 1.9.1 and VirtualBox 5.0.32. If you experience issues with the networking it is likely related to running an older version.

Config

The config.yml contains any variables that you should need to change, below is a description of the variables and their values:

orchestrator - Possible values are cattle, kubernetes, mesos and swarm

This sets the orchestrator that will be used for the environment, as part of the process the Default environment is deleted and we create a new one with the name of the orchestrator.

network_mode - Possible values are isolated, airgap and normal

isolated set the Rancher Server and Rancher nodes to have no external internet access other than via a proxy server that runs on the master node. This is used to emulate environments where a proxy server is required to access the internet

airgap sets the Rancher Server and nodes to have no external access at all. All images required to stand up Rancher are downloaded to a private repo on master and pulled from there

sslenabled - Possible values are true and false

This uses a pre-generated certificate to terminate the connection to the Rancher server with SSL. This certificate is located in the /certs folder. If this is changed then the public key will need to be replaced in the configure_rancher_node.sh script otherwise the agent will error.

ssldns - Default value is server.rancher.vagrant

The setting for this needs to match the string that is stored in the SSL certificate that is used for termination.

version - Possible values latest, stable, v1.x.x where x.x is any release of Rancher Server

This is the version of Rancher Server that you want to be deployed into your environment

rancher_env_vars - Pass through additional environment variables to the Rancher server

agent_version - The version of the Rancher agent for the nodes to pull

ROS_version - The version of RancherOS for the nodes to run, possible values are 1.0.3 and 1.0.4

master - Settings for the master node that runs the proxy, registry mirror etc., this value should not be changed

cpus - Default 1 This is the number of vCPU's that the master node should have
memory - Default 1024 This is the amount of RAM to be allocated to the master node, if running on a machine with only 8GB this should be dropped to 512

server - Settings for the server node(s) that runs the Rancher Server, this value should not be changed

count - Default 1 This is the number of Rancher Servers to run, if you want to test HA then this should be set to 2 or above
cpus - Default 1 This is the number of vCPU's that each server node should have
memory - Default 2048 This is the amount of RAM to be allocated to each server node, if running on a machine with only 8GB this should be dropped to 1024

node - Settings for the rancher node(s) that run in the Rancher environment, this value should not be changed

count - Default 3 This is the number of nodes to run
cpus - Default 1 This is the number of vCPU's that each Rancher node should have
memory - Default 2048 This is the amount of RAM to be allocated to each Rancher node, if running on a machine with only 8GB this should be dropped to 1024

ip - This section defines the IP address ranges for the virtual machines

master - Default 172.22.101.100
server - Default 172.22.101.101
node - Default 172.22.101.111

linked_clones - Default value true Leave as this as it reduces disk footprint

net - Network Settings section, this should not be changed

private_nic_type - Default 82545EM this sometime needs to be changed to 82540EM This is the network card that is emulated in the virtual machine
network_type - Default private_network

If you want to expose the Virtual Machines directly to the network this can be set to public_network

keys - Subsection for defining keys to be used when enabling external_ssh. The public key will be placed onto all servers, the private key will be placed onto just the master node. You can then use the master node as a jump host to each of the remaining VM's, or access them directly with the ssh key

public_key - This should be set to the path of the public key that needs to be uploaded
private_key - This should be set to the path of the private key that needs to be uploaded

external_access - To expose the setup to an external network

enabled - Default value false, Change to true if you want to expose the master node to an external network`
ssh_port - Default value 2277, this is the port that the master node will be exposed on if you enabled external_ssh
http_port - set this value to the local port on the host to forward to port 80 on the master
https_port - set this value to the local port on the host to forward to port 443 on the master

Troubleshooting

VM's starting but not running any scripts - Try changing the private_nic_type

vagrant's People

Contributors

Stargazers

Watchers

vagrant's Issues

any chance it could involve ansible as privisioner?

Great initiative,

all those scripts are a bit hard to read and modify, any chance would have Ansible as provisioner?
I see here something, but not sure if I can combine the two: https://github.com/rancher/ansible-playbooks.

Thanks!

Restarting Vagrant project causes Rancher Server to fail with `Error starting daemon: error initializing graphdriver: \"/var/lib/docker\" contains several valid graphdrivers: overlay, overlay2; Please cleanup or explicitly choose storage driver (-s <DRIVER>)`

This Vagrant project works when I first run vagrant up, but it fails to work after reloading the project in any form, including restarting my laptop, allowing the laptop to sleep and wake, etc.

Host detais:
MacOS 10.13.4 High Sierra
Vagrant 2.0.4
VirtualBox 5.2.10

The root cause appears to be that the Docker daemon in the server-01 VM is confused by which graphdriver to use, as shown by the error below. Can you help figure out a workaround? This seems to be caused by rancher/os#2278 .

stefanl@stefanl:vagrant $ vagrant status
Current machine states:

master                    running (virtualbox)
server-01                 running (virtualbox)
node-01                   running (virtualbox)
node-02                   running (virtualbox)

stefanl@stefanl:vagrant $ vagrant ssh server-01
[rancher@server-01 ~]$ sudo docker ps
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
[rancher@server-01 ~]$ tail /var/log/docker.log 
time="2018-04-24T21:49:32.850635694Z" level=fatal msg="Error starting daemon: error initializing graphdriver: \"/var/lib/docker\" contains several valid graphdrivers: overlay, overlay2; Please cleanup or explicitly choose storage driver (-s <DRIVER>)"

Steps to reproduce:

Check out the code:

stefanl@stefanl:Vagrant $ git clone https://github.com/rancher/vagrant
Cloning into 'vagrant'...
remote: Counting objects: 347, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 347 (delta 0), reused 1 (delta 0), pack-reused 341
Receiving objects: 100% (347/347), 230.47 KiB | 4.80 MiB/s, done.
Resolving deltas: 100% (190/190), done.
stefanl@stefanl:Vagrant $ cd vagrant
stefanl@stefanl:vagrant $ git checkout 2.0
Branch 2.0 set up to track remote branch 2.0 from origin.
Switched to a new branch '2.0'

Start the project for the first time, and confirm that I can reach the Rancher Server:

stefanl@stefanl:vagrant $ vagrant up
...
Bringing machine 'master' up with 'virtualbox' provider...
Bringing machine 'server-01' up with 'virtualbox' provider...
Bringing machine 'node-01' up with 'virtualbox' provider...
Bringing machine 'node-02' up with 'virtualbox' provider...
...

    node-02: Status: Downloaded newer image for rancher/agent:master
    node-02: f87f8b6d02619d7e00477da0d722df42287a002d25690620672d46765ced5439

stefanl@stefanl:vagrant $ curl --head --insecure https://172.22.101.101/
HTTP/1.1 200 OK
Content-Type: application/json
Expires: Wed 24 Feb 1982 18:42:00 GMT
X-Api-Schemas: https://172.22.101.101/meta/schemas
Date: Tue, 24 Apr 2018 21:46:06 GMT

stefanl@stefanl:vagrant $

Restart the Vagrant project. The Rancher Server becomes unavailable.


stefanl@stefanl:vagrant $ vagrant reload
==> master: Attempting graceful shutdown of VM...
==> master: Machine booted and ready!
...
==> server-01: Attempting graceful shutdown of VM...
==> master: Machine booted and ready!
...


stefanl@stefanl:vagrant $ vagrant status
...

Current machine states:

master                    running (virtualbox)
server-01                 running (virtualbox)
node-01                   running (virtualbox)
node-02                   running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
stefanl@stefanl:vagrant $ curl --head --insecure https://172.22.101.101/
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 172.22.101.101:443 
stefanl@stefanl:vagrant $

NFS Dynamic PVC for K8s clusters, NFS Driver for cattle clusters

Automate configuration and installation of NFS Client Dynamic Provisioner for kubernetes clusters.

This will facilitate local dev/test of Helm Charts / Kubernetes Manifests that make using of Dynamic Persistent Volumes.

Additionally, automate configuration and installation of NFS Driver in cattle environments.

kubectl config is generated with https even though that is not available

Just opening so I don't forget, I plan to look into it further. Just changing the url in the server key from https to http solves it but it would be nice if we didn't have to

2.0 not working?

The 2.0 branch doesn't appear to work. After vagrant up I get this in a loop:

    server-01: + wget -T 5 -c https://localhost/ping
    server-01: Connecting to localhost (127.0.0.1:443)
    server-01: wget: error getting response: Connection reset by peer
    server-01: + sleep 5
    server-01: + true
    server-01: + wget -T 5 -c https://localhost/ping
    server-01: Connecting to localhost (127.0.0.1:443)
    server-01: wget: error getting response: Connection reset by peer
    server-01: + sleep 5
    server-01: + true
    server-01: + wget -T 5 -c https://localhost/ping
    server-01: Connecting to localhost (127.0.0.1:443)
    server-01: wget: error getting response: Connection reset by peer
    server-01: + sleep 5
    server-01: + true
    server-01: + wget -T 5 -c https://localhost/ping
    server-01: Connecting to localhost (127.0.0.1:443)
    server-01: wget: error getting response: Connection reset by peer
    server-01: + sleep 5
    server-01: + true

External Config

As new features are added, existing installations get out of sync and require full recreation or manual intervention to fix. Users shouldn't need to modify Bash or Ruby files.

Externalize the configuration parameters to an answers file, check-in reasonable default config, and add default config to .gitignore.

PKI

It would be nice, for demonstration and reference purposes, to secure the cluster as if it were a production system. This would also enable safe(r) operation on public networks.

Secure any/all of the following endpoints with user-provided or generated certs:

Rancher Servers / HAProxy (most important)
Docker Daemons
Registry Cache
Git Server
Local Registry

Validation tests don't work

Validation tests, even when pointed to a custom PROJECT_ID, perform a lookup for project with UUID=adminProject, which is the not-really-a-uuid assigned to the Default environment. We should either not delete the Default environment during provisioning or update the validation tests to not assume the environment exists. I am leaning towards the latter.

INTERNALERROR>   File "[redacted]/validation-tests/tests/v2_validation/cattlevalidationtest/core/conftest.py", line 71, in cleanup
INTERNALERROR>     account = rancher_client.list_project(uuid="adminProject")[0] 
INTERNALERROR> IndexError: list index out of range

cc @sangeethah @galal-hussein @soumyalj

VMWare Provisioner

Add support for provisioning machines with VMWare.

Bastion / Manhole

To enable sharing machine access with others (on LAN), it would be useful to make the master node a bastion host that is capable of accessing the servers/nodes with a generated keypair.

This could be extended to WAN access via #23

"Could not rename the directory .../VirtualBox VMs/ubuntu-14.04-amd64_1519329807399_87133' to '.../VirtualBox VMs/master' to save the settings file (VERR_ALREADY_EXISTS)

I'm trying to run the rancher-vagrant project for the first time, but it is not working for me. I'm not a Vagrant master, but I understand correctly, the project may be renaming the image incorrectly as shown below with the VERR_ALREADY_EXISTS error.

I'll look into this more, and I will contribute a fix if I can. But in the meanwhile, does anyone know what's going on here?

My Environment is fairly standard MacOS High Sierra host running with up-to-date Vagrant and Virtualbox software.

stefanl@stefanl:Vagrant $ vagrant --version
Vagrant 2.0.2
stefanl@stefanl:Vagrant $ VBoxManage --version
5.2.6r120293

And just to verify, the Ubuntu directory specified below doesn't exist already:

ls -ld "/Users/stefanl/VirtualBox VMs/ubuntu* /Users/stefanl/VirtualBox VMs/master*"
ls: /Users/stefanl/VirtualBox VMs/ubuntu* /Users/stefanl/VirtualBox VMs/master*: No such file or directory
stefanl@stefanl:rancher-vagrant $

But if I follow the procedure to get this installed, the installation fails.

stefanl@stefanl:Vagrant $ git clone https://github.com/rancher/vagrant.git rancher-vagrant
Cloning into 'rancher-vagrant'...
remote: Counting objects: 341, done.
remote: Total 341 (delta 0), reused 0 (delta 0), pack-reused 341
Receiving objects: 100% (341/341), 225.31 KiB | 3.58 MiB/s, done.
Resolving deltas: 100% (190/190), done.
stefanl@stefanl:Vagrant $ cd rancher-vagrant/
stefanl@stefanl:rancher-vagrant $ vagrant up
Config: {"orchestrator"=>"cattle", "network_mode"=>"normal", "sslenabled"=>"false", "ssldns"=>"server.rancher.vagrant", "version"=>"latest", "rancher_env_vars"=>"", "agent_version"=>"v1.2.5", "ROS_version"=>"1.0.3", "master"=>{"cpus"=>1, "memory"=>1024}, "server"=>{"count"=>1, "cpus"=>1, "memory"=>2048}, "node"=>{"count"=>3, "cpus"=>1, "memory"=>2048}, "ip"=>{"master"=>"172.22.101.100", "server"=>"172.22.101.101", "node"=>"172.22.101.111"}, "linked_clones"=>true, "net"=>{"private_nic_type"=>"82545EM", "network_type"=>"private_network"}, "keys"=>{"public_key"=>"", "private_key"=>""}, "external_access"=>{"enabled"=>false, "ssh_port"=>2277, "http_port"=>80, "https_port"=>443}}

Bringing machine 'master' up with 'virtualbox' provider...
Bringing machine 'server-01' up with 'virtualbox' provider...
Bringing machine 'node-01' up with 'virtualbox' provider...
Bringing machine 'node-02' up with 'virtualbox' provider...
Bringing machine 'node-03' up with 'virtualbox' provider...
==> master: Importing base box 'williamyeh/ubuntu-trusty64-docker'...
==> master: Matching MAC address for NAT networking...
==> master: Checking if box 'williamyeh/ubuntu-trusty64-docker' is up to date...
==> master: Setting the name of the VM: master
The name of your virtual machine couldn't be set because VirtualBox
is reporting another VM with that name already exists. Most of the
time, this is because of an error with VirtualBox not cleaning up
properly. To fix this, verify that no VMs with that name do exist
(by opening the VirtualBox GUI). If they don't, then look at the
folder in the error message from VirtualBox below and remove it
if there isn't any information you need in there.

VirtualBox error:

VBoxManage: error: Could not rename the directory '/Users/stefanl/VirtualBox VMs/ubuntu-14.04-amd64_1519329807399_87133' to '/Users/stefanl/VirtualBox VMs/master' to save the settings file (VERR_ALREADY_EXISTS)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component SessionMachine, interface IMachine, callee nsISupports
VBoxManage: error: Context: "SaveSettings()" at line 3105 of file VBoxManageModifyVM.cpp
stefanl@stefanl:rancher-vagrant $

node-01 in loop

There are some issues happening after you perform a vagrant up, server-01 keeps in loop in order to run a container, but after a while it works, but node-01 never starts and keeps in loop with the following information:

node-01: + CLUSTERID=
node-01: + '[' -n '' ']'
node-01: + sleep 5
node-01: + true
node-01: ++ docker run --rm -i stedolan/jq -r '.data[].id'
node-01: ++ vagrant.txt
node-01: + CLUSTERID=
node-01: + '[' -n '' ']'
node-01: + sleep 5
node-01: + true
node-01: ++ docker run --rm -i stedolan/jq -r '.data[].id'
node-01: ++ vagrant.txt

It was tested in Vagrant running on Windows, Linux and Mac OS, in all systems we get the same loop
vagrant.txt
.