SandBox for EYWA PoC

Distributed Virtual Router Networking /w OpenNebula Cluster (on Vagrant Environment)

(Note) Currently VirtualBox is not supporting nesting VT-X, so simulation VMs is slow.

(Note) In Windows, Configure "core.autocrlf = input" in you Git Env, because "^M"

1. Performance

• Load Balancing by Multiple VR (No Limit)

2. High Availability

• HA by Multiple VR (No Limit)

3. Traffic Engineering

• Save of Network Bandwidth
• Traffic Isolation by VxLAN (16,777,216)
• Multicast instead of Broadcast
• Decrease in Packet Floods
• Flat Address
• Multiple Gateway for Load-balancer of Inbound & Outbound Traffic

4. Cost

• Multiple VR instead of L4 Switch (Single Router)
• Scalable VR

1. Load Balancing

• Load Balancing by Unlimited VRs(Virtual Router)
• Scale-out
• Load balanced Inbound & Outbound Traffic

2. High Availability

• HA by Unlimited VRs

3. Traffic Engineering

• Save of Network Bandwidth
• Low latency

4. VM Migration

1. A large number of tenants

• A large number of VLANs
• Traffic Isolation by VxLAN (16,777,216)

2. Large layer 2 network

• 10.0.0.0/8 (16,777,216 IPs) per Tenant
• Multicast instead of Broadcast, by VxLAN
• Decrease in MAC Flooding, by VxLAN
• Eliminate Broadcast (ToDo)

• Support OpeNebula 4.6 & OpenNebula 4.10 (Default: OpenNebula 4.10)
• Edit 'Vagrantfile' to change OpenNebula version
• "opennebula_version" Parameter.
• (Optional) Tested OpenNebula Hypervisor: KVM
• (Optional) Tested OpenNeubla Storage: NFS Shared, Qcow2 (OpenNebula's "Default" Datastore)

• Public Repository: OneDrive
• Folder-Link: http://1drv.ms/1BcJiDq (Read-Only)
• Detail-Info: http://www.evernote.com/l/ACma0cNe1CdCKLQFaDn6U3ukySYCWS6tmeM/
• (Note) If puppet *.pp Download(wget) is failed, check link url. (and feeback to me)
• EYWA-Router Template
• "EYWA-Ubuntu-14.04_64.qcow2.gz"

• Tested on VirtualBox 4.3.26.r98988

• eth0: "NAT" for External.

• eth1: "Host-Only Adapter"
• 192.168.33.0/24 (GW: 192.168.33.2)
• Range: 192.168.33.101 ~ 192.168.33.200 (100 EA)
• Promiscuous: "Allow All"

• eth2: "Internal Network"
• 10.0.0.0/8 (GW: 10.0.0.1) (16777216 EA)
• Promiscuous: "Allow All"

• 239.0.0.0 ~ 239.0.1.155 (4096 EA)

• eth0
• External Network.
• 192.168.33.0/24 (GW/ 192.168.33.2)
• eth1
• Internal Network. (per Tenant)
• 10.0.0.1/8 (Default Gateway of Tenant's VMs)
• 10.0.0.x/8 (Internal IP-Address. exclude 10.0.0.1)

• eth0
• 10.0.0.x/8 (Default Gateway: 10.0.0.1)

• (Note) At least two host need.

host> vagrant up master (Mandatory)

host> vagrant up slave-1 (Mandatory)
host> vagrant up slave-2 (Optional)

• (Note) VNC Password: passw0rd

[master]
VNC Address: {Vagrant-Host-IP}:55910

[slave-1]
VNC Address: {Vagrant-Host-IP}:55911

[slave-2]
VNC Address: {Vagrant-Host-IP}:55912

• http://{Host-IP}:9869
• Admin ID/PW: oneadmin / passw0rd

• (Note) This is test of external traffic. (VM to Exter)
• Log in to Web-UI, by "oneadmin" user.
• Go to 'System' Tab -> 'Users' Tab.
• Click '+' Button.
• Create 'testuser' User. (Password is that you want.)

• After user created, then...
• Default two templates is generated automatic. (in "Templates" Tab)
  • (Note) The templates is '2-EYWA-Router' and '2-Ubuntu(EYWA)'.
  • (Note) '2' is User-ID.
• First EYWA-Virutal-Router('2-EYWA-Router-0') is automaticaly launched. (in 'Virtual Machines' Tab)

• After '2-EYWA-Router-0' is up(Status is RUNNING), Go to 'Virtual Resources' Tab -> 'Virtual Machines'.
• Create first VM.

• Click '+' Button in 'Virtual Machines' Tab.
• Create first VM(named with '2-Ubuntu(EYWA)-0') by '2-Ubuntu(EYWA)' Template.
• Please wait until the status of 'Status' is "RUNNING".

• Create Second VR.

• Click '+' Button in 'Virtual Machines' Tab.
• Add '2-EYWA-Router-1' with '2-EYWA-Router' template. (for Router's LB/HA)
• Please wait until the status of 'Status' is "RUNNING".

• Create Second VM(named '2-Ubuntu(EYWA)-1').

• Click '+' Button in 'Virtual Machines' Tab.
• Add VM('2-Ubuntu(EYWA)-1') with '2-Ubuntu(EYWA)' template. (for Routers's LB/HA)
• Please wait until the status of 'Status' is "RUNNING".

• Check all VM/VR for EYWA-Demo Environment.

• EYWA-VRs should be present on different hosts(VirtualBox VM). (Check 'Host' in 'Virtual Machine' Tab)
• EYWA-VMs should be present on different hosts(VirtualBox VM). (Check 'Host' in 'Virtual Machine' Tab)
• All Status must be "RUNNING"
• (Note) Sample list of the generated VM with virt-manager on VNC-Console.
  • '2-EYWA-Router-0' => 'one-0'
  • '2-Ubuntu(EYWA)-0' => 'one-1'
  • '2-EYWA-Router-1' => 'one-2'
  • '2-Ubuntu(EYWA)-1' => 'one-3'

• Test failure scenarios. (If Some VRs is Down/Fail...)

• (Note) VM Placement is Random (by OpenNebula), Therefore, it may not exactly match the following information.

• Connect from any Host to VMs, then check below, (SSH Path: Client -> VR -> VM)

  (SSH Key is already exist, so auto-login. If prompt for password, then 'CTRL+C' and retry.) 
  
  [on Any-Host]# ssh 192.168.33.101 (SSH Connect to 2-EYWA-Router-0)
  [on 2-EYWA-Router-0]# ip address
  [on 2-EYWA-Router-0]# ssh 10.0.0.3 (IP-Address of 2-Ubuntu(EYWA)-0)
  [on 2-Ubuntu(EYWA)-0]# route -n
  [on 2-Ubuntu(EYWA)-0]# arp -n
  
  [on Any-Host]# ssh 192.168.33.102 (SSH Connect to 2-EYWA-Router-1)
  [on 2-EYWA-Router-1]# ip address
  [on 2-EYWA-Router-1]# ssh 10.0.0.5 (IP-Address of 2-Ubuntu(EYWA)-1)
  [on 2-Ubuntu(EYWA)-1]# route -n
  [on 2-Ubuntu(EYWA)-1]# arp -n

  • All VRs's secondary IP-Address of Internal-NIC is "10.0.0.1" for VMs's Default-Gateway.
  • All VMs's Default-Gateway is "10.0.0.1".

• Test Outbound Ping on all EYWA-VMs. (with VNC-Console)

  [on 2-Ubuntu(EYWA)-0]# ping 8.8.8.8 (Test Outbound-Networking on VM)
  [on 2-Ubuntu(EYWA)-0]# arp -n (Get IP/Mac of Gateway-VR)
  (ARP Result)
     10.0.0.1 --> 02:00:0a:00:00:01
              OR
     10.0.0.1 --> 02:00:0a:00:00:04

  • SSH to another VM to Test Outbound Ping.

  [on 2-Ubuntu(EYWA)-1]# ping 8.8.8.8 (Test Outbound-Networking on VM)
  [on 2-Ubuntu(EYWA)-1]# arp -n (Get IP/Mac of Gateway-VR)
  (ARP Result of Default-Gateway)
     10.0.0.1 --> 02:00:0a:00:00:01
              OR
     10.0.0.1 --> 02:00:0a:00:00:04

• Before delete one EYWA-Router, via the VR remain and again ssh re-connecting to all VMs.

  • (Note) VR to delete is not use as Gateway. (If both is used, then delete any.)

• Run ping on all VMs.

  [on 2-Ubuntu(EYWA)-0] watch -d -n1 'arp -n1; ping -c1 8.8.8.8'
  [on 2-Ubuntu(EYWA)-1] watch -d -n1 'arp -n1; ping -c1 8.8.8.8'

• Delete(Trash) one of EYWA-Router that working as Default-Gateway by ARP-Result.

  • Select that EYWA-Virtual-Router on "Virtual Machines" Tab, then destroy it.

• Only briefly stop the ping, it works again soon.

  • After ARP Refresh, ping test of all VMs is resumed. (Failover)

• Check 'arp -n', the 10.0.0.1's MAC is changed to remained EYWA-Virtual-Router's MAC.

• Next, Add New Virtual-Router.

• Create new VR to test LB.
• Then check Gateway(10.0.0.1)'s MAC on all VMs.

• (Optional) In addition, try other test. (add / delete as you want)

• TODO (ASAP)

• TODO (ASAP)

(Note) VM is Slow, becase of Nested-Virtaulization.

(Note) VM/VR's root-password is '1234'.

On any-node(master/slave)

vagrant ssh master //or slave-1/slave-2
sudo -i //root-privileged
ssh root@{VR-IP(External)}
ssh root@{VM-IP}

• With OpenNebula, Not yet support Migration fo EYWA-VR/VM, and deployment strategy of EYWA-VR.

GitLab Repository: https://gitlab.com/call518/addon-eywa/tree/master

• Env.
• VR-1 IP: eth0:192.168.33.101, eth1:10.0.0.2,10.0.0.1 (on-master)
• VM-1 IP: eth0:10.0.0.3 (on slave-1)
• VM-2 IP: eth0:10.0.0.4 (on master)

master> ssh [email protected]
VR-1> ssh [email protected]
VM-2> (root's Prompt)

VM-2> arping 10.0.0.3

VM-2> ping -s 5000 10.0.0.3

VM-2> ping -s 5000 -M want 10.0.0.3