missioncriticalcloud / cosmic Goto Github PK

Cosmic is open source software designed to deploy and manage large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform.

License: Apache License 2.0

Python 7.26% Shell 0.51% Java 71.07% CSS 1.43% JavaScript 17.94% HTML 0.54% FreeMarker 0.01% XSLT 0.68% Groovy 0.58%

cosmic's People

Contributors

Stargazers

Watchers

Forkers

edwardbetts rowhit miguelaferreira jpwesselink sanderv32 liuyong240 maduhu andyrepton sspans haj priestd09 robbertjansw aawm agelstrong pickkaa ddegoede jqmtony

cosmic's Issues

Removing VPC network DNS values does not revert to default set when network created

When a new network in a VPC is created the dns1 and dns2 values are empty and instances gets the gateway IP as their resolver as expected. When setting a resolver and then removing it, the dns1 value is set as 8.8.8.8 and dns2 is set as 208.67.220.220 and not setting dns1 as the gateway IP.

To illustrate this in an example:

Create a network in VPC with CIDR 10.0.0.0/24 with 10.0.0.1 as gateway
Check API and UI, both show DNS1 value as empty, /etc/resolver.conf on an instance shows nameserver 10.0.0.1 (as expected)
Set DNS1 to 1.1.1.1
Check API and UI, both show DNS1 value as 1.1.1.1
Delete DNS1 value in UI or API
Refresh network settings page in UI if using UI, it now shows DNS1 = 8.8.8.8 and DNS2 = 208.67.220.220

Feature request: Allow network ACL lists to be renamed

Currently it's not possible to rename a network ACL; it would be nice if this was possible without needing to recreate the ACL list.

Internal VPC should have network_overview sent on start

Most likely due to not having a public NIC, the 'Default Internal VPC' doesn't get a network_overview sent which results in keepalived not starting. When you create the first tier everything works.

We should make this the same as with the other VPCs and send the overview on startup.

Creating snapshot on KVM seems to block router operations

This snapshot job was created, sequence: 4494310953139132674:

2016-06-22 16:02:20,341 DEBUG [c.c.a.t.Request] (API-Job-Executor-71:ctx-a83500fc job-173319 ctx-f3446e61) Seq 195-4494310953139132674: Sending  { Cmd , MgmtId: 2886730125, via: 195(hypervisor), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/a8b493fe-7f50-3435-9bd0-61899d6ad490/60ad938a-afc8-4a50-be80-25bd950a4d41/f7391fd7-7881-43b0-a714-e5ae4287a83b","volume":{"uuid":"60ad938a-afc8-4a50-be80-25bd950a4d41","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"a8b493fe-7f50-3435-9bd0-61899d6ad490","id":39,"poolType":"NetworkFilesystem"
<cut>

Other operations (like spinning a VM) are waiting for that:

2016-06-22 21:57:25,912 DEBUG [o.a.c.n.t.BasicNetworkTopology] (Work-Job-Executor-15:ctx-ce669ba8 job-174279/job-174282 ctx-bd5226e9) Applying dhcp entry in network Ntwk[1189|Guest|52]

2016-06-22 21:57:25,946 DEBUG [c.c.a.t.Request] (Work-Job-Executor-15:ctx-ce669ba8 job-174279/job-174282 ctx-bd5226e9) Seq 195-4494310953139145246: Waiting for Seq 4494310953139132674 Scheduling:  { Cmd , MgmtId: 2886730125, via: 19
5(hypervisor), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:77:5e:00:03","vmIpAddress":"10.136.252.92","vmName":"vmname134","defaultRouter":"10.136.252
.1","duid":"00:03:00:01:02:00:77:5e:00:03","isDefault":true,"executeInSequence":true,"accessDetails":{"router.name":"r-6047-VM","router.guest.ip":"10.136.252.1","router.ip":"169.254.2.26","zone.network.type":"Advanced"},"wait":0}}] 
}

2016-06-22 22:27:11,241 WARN  [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-ae52240f) Task (job-174282) has been pending for 1785 seconds

2016-06-22 22:27:25,947 DEBUG [c.c.a.m.AgentAttache] (Work-Job-Executor-15:ctx-ce669ba8 job-174279/job-174282 ctx-bd5226e9) Seq 195-4494310953139145246: Waited too long.
2016-06-22 22:27:25,947 INFO  [c.c.u.e.CSExceptionErrorCode] (Work-Job-Executor-15:ctx-ce669ba8 job-174279/job-174282 ctx-bd5226e9) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2016-06-22 22:27:25,947 WARN  [c.c.a.m.AgentAttache] (Work-Job-Executor-15:ctx-ce669ba8 job-174279/job-174282 ctx-bd5226e9) Seq 195-4494310953139145246: Timed out on Seq 195-4494310953139145246:  { Cmd , MgmtId: 2886730125, via: 195(hypervisor), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:77:5e:00:03","vmIpAddress":"10.136.252.92","vmName":"vmtest134","defaultRouter":"10.136.252.1","duid":"00:03:00:01:02:00:77:5e:00:03","isDefault":true,"executeInSequence":true,"accessDetails":{"router.name":"r-6047-VM","router.guest.ip":"10.136.252.1","router.ip":"169.254.2.26","zone.network.type":"Advanced"},"wait":0}}] }
2016-06-22 22:27:25,948 DEBUG [c.c.a.m.AgentAttache] (Work-Job-Executor-15:ctx-ce669ba8 job-174279/job-174282 ctx-bd5226e9) Seq 195-4494310953139145246: Cancelling.
2016-06-22 22:27:25,948 WARN  [c.c.n.r.NetworkHelperImpl] (Work-Job-Executor-15:ctx-ce669ba8 job-174279/job-174282 ctx-bd5226e9) Timed Out
com.cloud.exception.OperationTimedoutException: Commands 4494310953139145246 to Host 195 timed out after 1800
        at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:429)
        at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:454)
        at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:972)
        at com.cloud.network.router.NetworkHelperImpl.sendCommandsToRouter(NetworkHelperImpl.java:165)
        at org.apache.cloudstack.network.topology.AdvancedNetworkVisitor.visit(AdvancedNetworkVisitor.java:87)
        at com.cloud.network.rules.DhcpEntryRules.accept(DhcpEntryRules.java:62)
        at org.apache.cloudstack.network.topology.BasicNetworkTopology.applyRules(BasicNetworkTopology.java:391)
        at org.apache.cloudstack.network.topology.AdvancedNetworkTopology.applyDhcpEntry(AdvancedNetworkTopology.java:173)
        at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:938)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1113)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1239)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1175)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:907)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4451)
        at sun.reflect.GeneratedMethodAccessor382.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:106)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4607)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:555)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:503)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-06-22 22:27:25,949 WARN  [o.a.c.n.t.BasicNetworkTopology] (Work-Job-Executor-15:ctx-ce669ba8 job-174279/job-174282 ctx-bd5226e9) Unable to apply dhcp entry on disconnected router r-6047-VM
com.cloud.exception.AgentUnavailableException: Resource [Host:195] is unreachable: Host 195: Unable to send commands to virtual router 
        at com.cloud.network.router.NetworkHelperImpl.sendCommandsToRouter(NetworkHelperImpl.java:168)
        at org.apache.cloudstack.network.topology.AdvancedNetworkVisitor.visit(AdvancedNetworkVisitor.java:87)
        at com.cloud.network.rules.DhcpEntryRules.accept(DhcpEntryRules.java:62)
        at org.apache.cloudstack.network.topology.BasicNetworkTopology.applyRules(BasicNetworkTopology.java:391)
        at org.apache.cloudstack.network.topology.AdvancedNetworkTopology.applyDhcpEntry(AdvancedNetworkTopology.java:173)
        at com.cloud.network.element.VirtualRouterElement.addDhcpEntry(VirtualRouterElement.java:938)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1113)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1239)
        at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1175)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:907)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4451)
        at sun.reflect.GeneratedMethodAccessor382.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Why would this be blocking at all?

Starting VM fails with NSX error 409

When starting a VM sometimes it will throw an exception confronting you with an NSX error 409:

Message: com.cloud.utils.rest.CloudstackRESTException: Unexpecetd status code: 409
Stack: com.cloud.network.nicira.NiciraNvpApiException: com.cloud.utils.rest.CloudstackRESTException: Unexpecetd status code: 409
        at com.cloud.network.nicira.NiciraNvpApi.updateWithUri(NiciraNvpApi.java:210)
        at com.cloud.network.nicira.NiciraNvpApi.updateLogicalSwitchPortAttachment(NiciraNvpApi.java:367)
        at com.cloud.network.resource.wrapper.NiciraNvpCreateLogicalSwitchPortCommandWrapper.execute(NiciraNvpCreateLogicalSwitchPortCommandWrapper.java:39)
        at com.cloud.network.resource.wrapper.NiciraNvpCreateLogicalSwitchPortCommandWrapper.execute(NiciraNvpCreateLogicalSwitchPortCommandWrapper.java:21)
        at com.cloud.network.resource.NiciraNvpRequestWrapper.execute(NiciraNvpRequestWrapper.java:53)
        at com.cloud.network.resource.NiciraNvpResource.executeRequest(NiciraNvpResource.java:196)
        at com.cloud.network.utils.CommandRetryUtility.retry(CommandRetryUtility.java:59)

This happens when there is a VIF already attached to the LPORT:

2017-12-11 06:55:08.158 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2017-12-11 06:55:08.158 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 >> "[\r][\n]"
2017-12-11 06:55:08.158 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 >> "{"type":"VifAttachment","vif_uuid":"aa6543bf-c466-43e2-8c40-3fcf440f558e"}"
2017-12-11 06:55:08.173 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << "HTTP/1.1 409 Conflict[\r][\n]"
2017-12-11 06:55:08.173 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << "Date: Mon, 11 Dec 2017 05:55:07 GMT[\r][\n]"
2017-12-11 06:55:08.173 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << "Content-Type: text/html[\r][\n]"
2017-12-11 06:55:08.173 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << "Server: NVP/4.2.4.42965[\r][\n]"
2017-12-11 06:55:08.173 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << "Content-Length: 114[\r][\n]"
2017-12-11 06:55:08.173 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << "[\r][\n]"
2017-12-11 06:55:08.173 DEBUG [o.a.h.wire] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << "VIF 'aa6543bf-c466-43e2-8c40-3fcf440f558e' already attached to logical port 'aa6543bf-c466-43e2-8c40-3fcf440f558e'"
2017-12-11 06:55:08.173 DEBUG [o.a.h.headers] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << HTTP/1.1 409 Conflict
2017-12-11 06:55:08.173 DEBUG [o.a.h.headers] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << Date: Mon, 11 Dec 2017 05:55:07 GMT
2017-12-11 06:55:08.173 DEBUG [o.a.h.headers] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << Content-Type: text/html
2017-12-11 06:55:08.173 DEBUG [o.a.h.headers] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << Server: NVP/4.2.4.42965
2017-12-11 06:55:08.173 DEBUG [o.a.h.headers] (logid: 09d1506b) (ctx: d5cb175d) http-outgoing-24 << Content-Length: 114
2017-12-11 06:55:08.173 DEBUG [o.a.h.i.e.MainClientExec] (logid: 09d1506b) (ctx: d5cb175d) Connection can be kept alive indefinitely
2017-12-11 06:55:08.174 DEBUG [c.c.n.n.NiciraRestClient] (logid: 09d1506b) (ctx: d5cb175d) Status of last request: HTTP/1.1 409 Conflict
2017-12-11 06:55:08.174 WARN  [c.c.n.r.w.NiciraNvpCreateLogicalSwitchPortCommandWrapper] (logid: 09d1506b) (ctx: d5cb175d) modifyLogicalSwitchPort failed after switchport was created, removing switchport

Which in turn happens after an unclean shutdown of the VM and/or incomplete DB changes.

Steps to reproduce:

Create VPC
Create network in VPC
Create VM in network
Destroy VM on HV
Stop VM in DB
Remove VIF attachment from nicira_nvp_nic_map in DB
Try to start VM

Unable to use LoadBalancer VIP from within VPC tier

At the moment it is not possible to reach a public IP with a Load Balancer rule from within the VPC where it is attached however this seems to work if the public IP is configured with a Port Forward rule to a single virtual machine.

As a system engineer I would like to be able to use the same functionality for Load Balancer rules to avoid single points of failures.

Issue raised in name of @rezgaa

Consider not restarting network on offering change

A flag is set to true when we change the offering. Reconsider!

13:58 $ git diff
diff --git a/cosmic-core/server/src/main/java/com/cloud/network/NetworkServiceImpl.java b/cosmic-core/server/src/main/java/com/cloud/network/NetworkServiceImpl.java
index 0e9b3ba8c..f1a0da512 100644
--- a/cosmic-core/server/src/main/java/com/cloud/network/NetworkServiceImpl.java
+++ b/cosmic-core/server/src/main/java/com/cloud/network/NetworkServiceImpl.java
@@ -1601,6 +1601,7 @@ public class NetworkServiceImpl extends ManagerBase implements NetworkService {
                     throw new InvalidParameterValueException("Can't upgrade from network offering " + oldNtwkOff.getUuid() + " to " + networkOffering.getUuid()
                             + "; check logs for more information");
                 }
+                // Why?
                 restartNetwork = true;
                 networkOfferingChanged = true;

Connection timed out

I'm trying to build. However Maven spits out a couple of errors like these:

[INFO] Scanning for projects...
Downloading from beta-nexus: https://beta-nexus.mcc.schubergphilis.com/content/groups/public/org/springframework/boot/spring-boot-starter-parent/2.0.1.RELEASE/spring-boot-starter-parent-2.0.1.RELEASE.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for cloud.cosmic:cosmic:6.1.12-SNAPSHOT: Could not transfer artifact org.springframework.boot:spring-boot-starter-parent:pom:2.0.1.RELEASE from/to beta-nexus (https://beta-nexus.mcc.schubergphilis.com/content/groups/public): Connect to beta-nexus.mcc.schubergphilis.com:443 [beta-nexus.mcc.schubergphilis.com/85.222.237.248] failed: Connection timed out (Connection timed out) and 'parent.relativePath' points at no local POM @ line 35, column 1

I tried to connect to this host manually and this also failed. Is it correct this host is down? Any idea when, if ever, it's coming back?
Thanks in advance..

Suggestion: empty network should not block deleting VPC

At the moment when you try to delete a VPC it will fail if it contains an empty network.

Ideally this shouldn't fail if the network or networks in the VPC are empty, but should fail when those networks contain instances.

Feature: Snapshots for VMs in KVM Hypervisors

Backport this feature from CloudStack.

PR: apache/cloudstack#977
JIRA issue: https://issues.apache.org/jira/browse/CLOUDSTACK-8746

VPC restart + cleanup destroys NSX switch if it is believed to be empty

During a VPC restart + cleanup, if a network tier is currently up, but if all existing VMs in that tier are stopped or previous VMs are destroyed, the tier is not added to the router during the cleanup and the lswitch is removed.

In some circumstances the lswitch has VIFs attached from private HV, so we should not destroy tiers if they are currently implemented.

Updating networkdomain of a VPC tier does not update DNSMasq / Metadata config

Example:

update network networkdomain=new-domain.nl id=44050d06-6df0-46ac-ac4d-4b4da82e91cc

On router /etc/dnsmasq.d/cloud.conf is unchanged:

root@r-18-VM:/var/cache/cloud/processed# cat /etc/dnsmasq.d/cloud.conf
dhcp-hostsfile=/etc/dhcphosts.txt
dhcp-optsfile=/etc/dhcpopts.txt
log-dhcp
interface=eth2
dhcp-range=interface:eth2,set:interface-eth2-0,10.3.1.1,static
dhcp-option=tag:interface-eth2-0,15,domain.nl
dhcp-option=tag:interface-eth2-0,6,10.3.1.1
dhcp-option=tag:interface-eth2-0,3,10.3.1.1
dhcp-option=tag:interface-eth2-0,1,255.255.255.0
interface=eth3
dhcp-range=interface:eth3,set:interface-eth3-1,10.3.2.1,static
dhcp-option=tag:interface-eth3-1,15,domain.nl
dhcp-option=tag:interface-eth3-1,6,10.3.2.1
dhcp-option=tag:interface-eth3-1,3,10.3.2.1
dhcp-option=tag:interface-eth3-1,1,255.255.255.0

The expected result is that option 15 gets updated in Dnsmasq.

Also check the metadata in /var/www/html/metadata to be correct.

Deleting a networkacllist applied on privategateway fails without error

If a networkacllist is applied to a private gateway and I try to delete the that networkacllist the job eventually timeout, but does not give an error stating that the networkacllist is still in use

IPtables jump to SOURCE_NAT_LIST should check for SNAT

On routers without a public interface, or with an public interface without source nat, should not add the jump to SOURCE_NAT_LIST.

root@r-40978-VM:~# iptables-restore < /tmp/rules.save 
iptables-restore v1.4.14: Couldn't load target `SOURCE_NAT_LIST':No such file or directory

Error occurred at line: 1613
Try `iptables-restore -h' or 'iptables-restore --help' for more information.

Unable to resize a volume if the instance is not running

Hi,

it seems that with the implementation of the libvirt bindings an issue was introduced were a disk cannot be resized if the instance is not running.

2018-06-26 11:03:57.343 ERROR [c.c.f.j.i.AsyncJobManagerImpl] (logid: 47344f9c) (job: 4704433) Unexpected exception java.lang.RuntimeException: Unexpected exception at com.cloud.storage.VolumeApiServiceImpl.resizeVolume(VolumeApiServiceImpl.java:1479) at com.cloud.storage.VolumeApiServiceImpl.resizeVolume(VolumeApiServiceImpl.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:34) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy214.resizeVolume(Unknown Source) at com.cloud.api.command.user.volume.ResizeVolumeCmd.execute(ResizeVolumeCmd.java:124) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:121) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:89) at com.cloud.framework.jobs.impl.AsyncJobManagerImpl$1.runInContext(AsyncJobManagerImpl.java:258) at com.cloud.common.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:37) at com.cloud.common.managed.context.DefaultManagedContext$1.call(DefaultManagedContext.java:35) at com.cloud.common.managed.context.DefaultManagedContext.callWithContext(DefaultManagedContext.java:81) at com.cloud.common.managed.context.DefaultManagedContext.runWithContext(DefaultManagedContext.java:32) at com.cloud.common.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:34) at com.cloud.framework.jobs.impl.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:204) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.cloud.legacymodel.exceptions.CloudException: Caught exception while handling a VmWorkJob at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:104) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2963) at sun.reflect.GeneratedMethodAccessor545.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy214.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:89) ... 12 common frames omitted Caused by: com.cloud.legacymodel.exceptions.CloudRuntimeException: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'i-126-22797-VM' at com.cloud.storage.VolumeApiServiceImpl.orchestrateResizeVolume(VolumeApiServiceImpl.java:1620) at com.cloud.storage.VolumeApiServiceImpl.orchestrateResizeVolume(VolumeApiServiceImpl.java:2939) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:88) ... 24 common frames omitted

Create test for protocol in ACL

See PR #826 where if the protocol is in the ACL the script throws an 'KeyError' exception.

Cosmic autoscaling for K8S

In order to support a form of autoscaling for K8S we need to implement a VM Group. This group will make sure that a desired number of VMs is running with a specified launch configuration.

VM Group:

Assigned VPC and tier
Desired number of VMs
Actual number of VMs
List of current VMs
Assigned loadbalancer rules
Launch configuration (defined below)

Launch Configuration:

All possible values a VM can have (http://apidoc.cosmiccloud.io/root_admin/deployVirtualMachine.html)
Optional: Versioned config per VM

Invalid iptables rules generated when protocol id is used

When adding allow rule in ingress/egress for a protocol id, like 47, this results in invalid json and as a result iptables loading fails:

        "ingress_rules": [
            {
                "allowed": false,
                "cidr": "ACCEPT",
                "protocol": 47,
                "type": "protocol"
            },

-A ACL_OUTBOUND_eth2 -p 47 -d ACCEPT -j DROP

root@r-39254-VM:~# iptables-restore < /tmp/rules.save
iptables-restore v1.4.14: host/network `ACCEPT' not found
Error occurred at line: 30
Try `iptables-restore -h' or 'iptables-restore --help' for more information.

ACL does not seem to be applied to loadbalancer config

Scenario:

VPC
Tier
1 or more VMs
Acquire public ip address
Config loadbalancer say port 80 to one or more VM
Confirm this works OK
Set default_deny acl (or custom acl blocking your ip)
Acl doesn't do anything, it is open for all

Most likely related to the way loadbalancing works (HAproxy).

NullPointerException returned when changing VPN Customer Gateway settings

When trying to modify an existing VPN Customer Gateway via the API NullPointerException: is returned:

$ cloudmonkey list vpncustomergateways id=cc9942b9-e953-4f2b-88e6-05467b6c1075
count = 1
vpncustomergateway:
name = terraform-foo
id = cc9942b9-e953-4f2b-88e6-05467b6c1075
account = ...
cidrlist = 10.0.1.0/24
domain = SBP
domainid = f7842270-6097-45af-ad1e-a678e06d0f22
dpd = False
esplifetime = 86400
esppolicy = aes256-sha1
forceencap = False
gateway = 1.2.3.4
ikelifetime = 86400
ikepolicy = aes256-sha1;modp1024
ipsecpsk = test


$ cloudmonkey update vpncustomergateway id=cc9942b9-e953-4f2b-88e6-05467b6c1075 cidrlist=10.0.1.0/24 gateway=1.2.3.4 esppolicy=aes256-sha1 ikepolicy='aes256-sha1;modp2048' ipsecpsk=test
Async job 35181e0a-e1b8-43b3-9204-79952c79cc4c failed
Error 530, NullPointerException:
accountid = 1ac7a8ae-c639-4f8c-b934-a01e23be7cab
cmd = com.cloud.api.command.user.vpn.UpdateVpnCustomerGatewayCmd
created = 2019-02-24T16:18:13+0100
jobid = 35181e0a-e1b8-43b3-9204-79952c79cc4c
jobprocstatus = 0
jobresult:
errorcode = 530
errortext = NullPointerException:
jobresultcode = 530
jobresulttype = object
jobstatus = 2
userid = 1fac3ec1-0ae7-40d7-85b5-83d073c77aac

Can't replicate this when modifying properties of the VPN Customer Gateway via the UI, after saving and refreshing the new settings are visible (and returned correctly via the API).

DHCP gateway wrong in dnsmasq

the gateway for eth3 is not correct in /etc/dnsmasq.d/cloud.conf

root@r-40978-VM:~# cat /etc/dnsmasq.d/cloud.conf 
dhcp-hostsfile=/etc/dhcphosts.txt
dhcp-optsfile=/etc/dhcpopts.txt
log-dhcp
interface=eth2
dhcp-range=interface:eth2,set:interface-eth2-0,10.136.75.65,static
dhcp-option=tag:interface-eth2-0,15,enga.local
dhcp-option=tag:interface-eth2-0,6,10.136.75.65
dhcp-option=tag:interface-eth2-0,3,10.136.75.65
dhcp-option=tag:interface-eth2-0,1,255.255.255.192
interface=eth3
dhcp-range=interface:eth3,set:interface-eth3-1,10.136.75.1,static
dhcp-option=tag:interface-eth3-1,15,enga.local
dhcp-option=tag:interface-eth3-1,6,10.136.75.129
dhcp-option=tag:interface-eth3-1,3,10.136.75.129
dhcp-option=tag:interface-eth3-1,1,255.255.255.192

root@r-40978-VM:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 0e:00:a9:fe:02:e9 brd ff:ff:ff:ff:ff:ff
    inet 169.254.2.233/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 06:1a:36:00:08:83 brd ff:ff:ff:ff:ff:ff
    inet 10.135.246.197/25 brd 10.135.246.255 scope global eth1
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:50:a2:00:37 brd ff:ff:ff:ff:ff:ff
    inet 10.136.75.65/26 brd 10.136.75.127 scope global eth2
       valid_lft forever preferred_lft forever
8: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:3c:ac:00:58 brd ff:ff:ff:ff:ff:ff
    inet 10.136.75.1/26 brd 10.136.75.63 scope global eth3
       valid_lft forever preferred_lft forever

Ability to create a redundant internal VPC

The reasons this wouldn't work are now resolved, so we should allow redundant internal VPCs.

Not specifying isolated network gateway results in gateway set to DHCP server interface

When creating an isolated network and not setting a default gateway /etc/dnsmasq.d/cloud.conf will have the line:

dhcp-option=tag:interface-eth0,3,0.0.0.0

0.0.0.0 maps to self and will result in sending the DHCP server interface as the gateway.

This line should be:

dhcp-option=tag:interface-eth0,3

Deleting a non attached volume remains in state deleted

Looks like the expunge job does not change the state of a volume that was never attached after it is deleted, probably because the volume was never actually created on disk.

Failed static route apply is not removed from DB

Applying a static route fails with error:

[EXCEPTION] CloudRuntimeException: Failed to apply static routes in vpc [VPC [918-CUST_t_CX_SBP]

But the route is still in the DB as can be seen in the GUI:

After this the route cannot be deleted anymore:

root@r-40969-VM:~# replay list
total 16
-rw-r--r-- 1 root root 1736 Jul 18 17:42 network_acl.json.69be969d-9d4a-48dd-8979-a1d6a9de03e0.gz
-rw-r--r-- 1 root root 1738 Jul 18 17:42 network_acl.json.f62d0893-a56b-424a-9fce-d05b6b329fdc.gz
-rw-r--r-- 1 root root  547 Jul 18 18:01 static_routes.json.c7576a51-8d6c-45f6-99e6-4ae1d712f9af.gz
-rw-r--r-- 1 root root  552 Jul 18 18:01 static_routes.json.bafdae6e-a3e8-4131-b592-ae86edfe5f45.gz
root@r-40969-VM:~# jsoncat static_routes.json.c7576a51-8d6c-45f6-99e6-4ae1d712f9af.gz
{
    "routes": [

... output removed

        {
            "cidr": "10.132.1.128/25", 
            "ip_address": "10.135.246.139", 
            "revoke": false
        }
    ], 
    "type": "staticroutes"
}
root@r-40969-VM:~# jsoncat static_routes.json.bafdae6e-a3e8-4131-b592-ae86edfe5f45.gz
{
    "routes": [

... output removed

        {
            "cidr": "10.132.1.128/25", 
            "ip_address": "10.135.246.139", 
            "revoke": true
        }
    ], 
    "type": "staticroutes"
}

Route is not applied:

root@r-40969-VM:~# ip route
default via 10.135.246.131 dev eth1 
10.132.4.0/25 via 10.135.246.139 dev eth1 
10.132.156.0/27 via 10.135.246.140 dev eth1 
10.132.240.0/21 via 10.135.246.139 dev eth1

So there appear to be two issues:

a route that failed to apply to the routervm is still in the database even though an Exception is reported.
a route in DB that does not exist on the routervm cannot be deleted anymore.

Return category name in listApis response

Please could the category be included in the listApis response. The category can then be used to group related calls (for use with an API client).

List all load balancer instances always requires ID

It'd be nice to support listall=true and not require ID

HAProxy does not support UDP load balancing, but the API allows UDP

When creating a load balancer rule the protocol is optional, and defaults to TCP, in fact even if one specifies protocol=udp, which the documentation states is valid, it is changed into TCP.

Fact is that HAProxy cannot load balance UDP, only TCP as the first two lines state:

HAProxy: The Reliable, High Performance TCP/HTTP Load Balancer ( http://www.haproxy.org/ )

So either we look into using nginx ( https://www.nginx.com/blog/announcing-udp-load-balancing/ ) or remove UDP from the documentation and return an error is protocol=udp is passed to the API.

Change back to semantic versioning

Using Flyway as a DB migrator, we don't need the extra version number anymore.

Failed startup

after running first time:
mvn -pl :cloud-client-ui jetty:run
I got following error:

[ERROR] Failed startup of context org.mortbay.jetty.plugin.Jetty6PluginWebAppContext@8851ec{/cloud-client-ui,/home/tarmo/work/cosmic/cosmic-client/src/main/webapp}
com.cloud.utils.exception.CloudRuntimeException
at com.cloud.flyway.FlywayDB.check(FlywayDB.java:29)
at com.cloud.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:34)
at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.jetty.plugin.Jetty6PluginWebAppContext.doStart(Jetty6PluginWebAppContext.java:115)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.plugin.Jetty6PluginServer.start(Jetty6PluginServer.java:132)
at org.mortbay.jetty.plugin.AbstractJettyMojo.startJetty(AbstractJettyMojo.java:454)
at org.mortbay.jetty.plugin.AbstractJettyMojo.execute(AbstractJettyMojo.java:396)
at org.mortbay.jetty.plugin.AbstractJettyRunMojo.execute(AbstractJettyRunMojo.java:210)
at org.mortbay.jetty.plugin.Jetty6RunMojo.execute(Jetty6RunMojo.java:184)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: javax.naming.NameNotFoundException; remaining name 'jdbc/cosmic'
at org.mortbay.naming.NamingContext.lookup(NamingContext.java:634)
at org.mortbay.naming.NamingContext.lookup(NamingContext.java:665)
at org.mortbay.naming.NamingContext.lookup(NamingContext.java:665)
at org.mortbay.naming.NamingContext.lookup(NamingContext.java:680)
at org.mortbay.naming.java.javaRootURLContext.lookup(javaRootURLContext.java:112)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at com.cloud.flyway.FlywayDB.check(FlywayDB.java:26)
... 43 more

Prevent live migrating to host that doesn't meet tag or affinity group

Disassociation of public ip will not reset its ACL

This results in an "empty" ACL when the ACL linked got removed in the mean time, or is not available due to permissions. We need to to reset the ACL when we disassociate it.

Investigate creation/removal of duplicate lswitch id of private network

Listing all loadbalancer rules of all projects fails

🐵 > list loadbalancerrules listall=true projectid=-1
Error 530: None
{
  "cserrorcode": 9999,
  "errorcode": 530,
  "uuidList": []
}

2017-09-08 15:35:04.289 ERROR [c.c.a.ApiServer] (logid: 3b41de26) (ctx: b3d84f9a) Unhandled exception executing api command: queryAsyncJobResult
com.google.gson.JsonParseException: Expecting object found: "mnt"
        at com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
        at com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:498)
        at com.google.gson.Gson.fromJson(Gson.java:467)
        at com.google.gson.Gson.fromJson(Gson.java:417)
        at com.google.gson.Gson.fromJson(Gson.java:389)
        at com.cloud.api.ApiSerializerHelper.fromSerializedString(ApiSerializerHelper.java:53)
        at com.cloud.api.query.dao.AsyncJobJoinDaoImpl.newAsyncJobResponse(AsyncJobJoinDaoImpl.java:56)
        at sun.reflect.GeneratedMethodAccessor411.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)

Clustered Mgt server: on upgrade another mgt server is detected with the same ip

After which it shuts itself down. A simple systemctl restart tomcat solves the issue.

2017-06-20 17:27:26.544 ERROR [c.c.c.ClusterManagerImpl] Detected that another management node with the same IP a.x.y.z is already running, please check your cluster configuration
2017-06-20 17:27:26.544 ERROR [c.c.s.l.CloudStackExtendedLifeCycle] Failed to configure ClusterManagerImpl
javax.naming.ConfigurationException: Detected that another management node with the same IP a.x.y.z is already running, please check your cluster configuration
        at com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:350)
        at com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:301)
        at com.cloud.spring.lifecycle.CloudStackExtendedLifeCycle$1.with(CloudStackExtendedLifeCycle.java:70)
        at com.cloud.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:84)
        at com.cloud.spring.lifecycle.CloudStackExtendedLifeCycle.configure(CloudStackExtendedLifeCycle.java:66)
        at com.cloud.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:37)
        at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:175)
        at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:50)
        at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:348)
        at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:151)
        at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:114)
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:879)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545)
        at com.cloud.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:123)
        at com.cloud.spring.module.model.impl.DefaultModuleDefinitionSet.lambda$loadContexts$1(DefaultModuleDefinitionSet.java:101)
        at com.cloud.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:220)
        at com.cloud.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:225)
        at com.cloud.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:225)
        at com.cloud.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:207)
        at com.cloud.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:96)
        at com.cloud.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:60)
        at com.cloud.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:19)
        at com.cloud.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:52)
        at com.cloud.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:42)
        at com.cloud.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:33)
        at com.cloud.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:32)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
        at org.apache.catalina.core.StandardContext.reload(StandardContext.java:4088)
        at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:425)
        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1342)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
        at java.lang.Thread.run(Thread.java:748)

Root disk controller not always correctly displayed

If a VM has a datadisk that is created before the VM instance, then the controller type of that disk is sometimes shown as the Root disk controller:

even though the root volume has an IDE controller configured:

I believe this is due to the user_vm_view ordering

Remove "Firewall" methods

As discussed with @remibergsma, the firewall functionality is redundant as Cosmic no longer supports isolated networks, only VPC networks and Private networks, so the following can be removed:

createFirewallRule
deleteFirewallRule
listFirewallRules
updateFirewallRule
createEgressFirewallRule
deleteEgressFirewallRule
listEgressFirewallRules
updateEgressFirewallRule

/cc @ddegoede

Metadata local-hostname should have FQDN instead of short hostname

Creating issue from e-mail received to keep track of it.

Hi guys,

For a while now, we’ve seen cloud-init behaving oddly when starting up new centos6 templates, specifically setting the domain of the machine to be .localdomain. Then, this needs to be manually fixed for people before they can move onto their configuration. We spent some time on Friday really digging into this, and eventually in the code we found that in the metadata of the machine, the current ‘standard’ for cloud-init is for the domain to be appended to the machine name, from the network domain. At the moment however, cloudstack/cosmic is only providing the short name of the machine, and then the default of ‘localdomain’ is appended.

https://github.com/number5/cloud-init/blob/master/cloudinit/sources/__init__.py#L182

In short, on the router vms, the file local-hostname in /var/www/html/metadata/$instance_IP/local-hostname needs to have the network domain appended to it. An example is vm app26, which is in a network of bla.local, at the moment the file only has ‘app26’ when it needs to have ‘app26.bla.local’

I hope this makes sense, please let me know if I can provide any further information. Could this be added to the backlog to be fixed in a new version of cosmic please?

Initial start of SSVM sometimes fails resulting in failed builds

We see builds hanging waiting on the templates to become ready. This never happens, because the deployment/start of the ssvm fails somehow.

On the KVM agent the VM is never started. All it gets is a stopCommand to clean it up:

2016-10-01 03:55:33.179 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] com.cloud.agent.service.Agent            : Request:Seq 2-2335397881768312842:  { Cmd , MgmtId: 90520732674657, via: 2, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"checkBeforeCleanup":false,"isProxy":false,"vmName":"s-2-VM","executeInSequence":false,"wait":0}}] }
2016-10-01 03:55:33.179 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] com.cloud.agent.service.Agent            : Processing command: com.cloud.agent.api.StopCommand
2016-10-01 03:55:33.179 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.k.r.LibvirtComputingResource       : Processing cmd com.cloud.agent.api.StopCommand
2016-10-01 03:55:33.179 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.kvm.resource.LibvirtConnection     : Looking for libvirtd connection at: qemu:///system
2016-10-01 03:55:33.181 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.kvm.resource.LibvirtConnection     : Can not find KVM connection for Instance: s-2-VM, continuing.
2016-10-01 03:55:33.181  WARN (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.kvm.resource.LibvirtConnection     : Can not find a connection for Instance s-2-VM. Assuming the default connection.
2016-10-01 03:55:33.181 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.kvm.resource.LibvirtConnection     : Looking for libvirtd connection at: qemu:///system
2016-10-01 03:55:33.184 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.k.r.LibvirtComputingResource       : Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 's-2-VM'
2016-10-01 03:55:33.185 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.k.r.LibvirtComputingResource       : Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 's-2-VM'
2016-10-01 03:55:33.185 DEBUG (logid: ade9abca) 10333 --- [agentRequest-Handler-1] com.cloud.agent.service.Agent            : Request:Seq 2-2335397881768312843:  { Cmd , MgmtId: 90520732674657, via: 2, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"http://cloudstack.apt-get.eu/systemvm/4.6/systemvm64template-4.6.0-kvm.qcow2.bz2","uuid":"f470b845-698c-4d5b-af58-3682b75f58e2","id":3,"format":"QCOW2","accountId":1,"checksum":"c059b0d051e0cd6fbe9d5d4fc40c7e5d","hvm":false,"displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.22.1:/data/storage/secondary/MCCT-SHARED-1","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://cloudstack.apt-get.eu/systemvm/4.6/systemvm64template-4.6.0-kvm.qcow2.bz2","uuid":"f470b845-698c-4d5b-af58-3682b75f58e2","id":3,"format":"QCOW2","accountId":1,"checksum":"c059b0d051e0cd6fbe9d5d4fc40c7e5d","hvm":false,"displayText":"SystemVM Template (KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"812ea6a3-7ad0-30f4-9cab-01e3f2985b98","id":1,"url":"NetworkFilesystem://192.168.22.1/data/storage/primary/MCCT-KVM-1/?ROLE=Primary&STOREUUID=812ea6a3-7ad0-30f4-9cab-01e3f2985b98","poolType":"NetworkFilesystem","host":"192.168.22.1","path":"/data/storage/primary/MCCT-KVM-1","port":2049}},"name":"routing-3","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"options2":{},"wait":10800}}] }
2016-10-01 03:55:33.185 DEBUG (logid: ade9abca) 10333 --- [agentRequest-Handler-1] com.cloud.agent.service.Agent            : Processing command: org.apache.cloudstack.storage.command.CopyCommand
2016-10-01 03:55:33.185 DEBUG (logid: ade9abca) 10333 --- [agentRequest-Handler-1] c.c.h.k.r.LibvirtComputingResource       : Processing cmd org.apache.cloudstack.storage.command.CopyCommand
2016-10-01 03:55:33.185 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.k.r.LibvirtComputingResource       : Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 's-2-VM'
2016-10-01 03:55:33.185 DEBUG (logid: 3bd85172) 10333 --- [agentRequest-Handler-4] c.c.h.k.r.LibvirtComputingResource       : Executing: /opt/cosmic/agent/./scripts/vm/network/security_group.py destroy_network_rules_for_vm --vmname s-2-VM

On the mgt server side we see that it wants to spin a new SSVM:

2016-10-01 03:55:32.698 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Zone 1 is ready to launch secondary storage VM
2016-10-01 03:55:32.698 INFO  [c.c.s.s.SecondaryStorageManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Scanning secondary storage pool 1
2016-10-01 03:55:32.699 INFO  [c.c.s.s.SecondaryStorageManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Found less (0) secondary storage VMs than image stores (1) in dcId=1, starti
ng 1 new VMs
2016-10-01 03:55:32.699 INFO  [c.c.s.s.SecondaryStorageManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Resizing secondary storage pool (dcId=1) with action com.cloud.vm.AfterScanA
ction@3fd95ab1[action=EXPAND,value=1]
2016-10-01 03:55:32.699 DEBUG [c.c.s.SystemVmManagerBase] (logid: 4a2cdd27) (ctx: ca3f04b1) Expanding pool [iteration 1/1]
2016-10-01 03:55:32.700 INFO  [c.c.s.s.SecondaryStorageManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) No stopped secondary storage vm is available, need to allocate a new seconda
ry storage vm
2016-10-01 03:55:32.714 DEBUG [c.c.s.s.SecondaryStorageManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Assign secondary storage vm from a newly started instance for request from d
ata center : 1

It provisions a new one:

2016-10-01 03:55:32.733 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Found existing network configuration for offering [Network Offering [1-Public-System-Public-Network]: Ntwk[200|Public|1]
2016-10-01 03:55:32.733 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Releasing lock for Acct[61609720-8779-11e6-8d58-5254001daa61-system]
2016-10-01 03:55:32.735 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2016-10-01 03:55:32.735 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Releasing lock for Acct[61609720-8779-11e6-8d58-5254001daa61-system]
2016-10-01 03:55:32.736 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Found existing network configuration for offering [Network Offering [2-Management-System-Management-Network]: Ntwk[201|Management|2]
2016-10-01 03:55:32.736 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Releasing lock for Acct[61609720-8779-11e6-8d58-5254001daa61-system]
2016-10-01 03:55:32.742 DEBUG [c.c.v.VirtualMachineManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Allocating entries for VM: VM[SecondaryStorageVm|s-2-VM]
2016-10-01 03:55:32.743 DEBUG [c.c.v.VirtualMachineManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Allocating nics for VM[SecondaryStorageVm|s-2-VM]
2016-10-01 03:55:32.744 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Allocating nic for vm VM[SecondaryStorageVm|s-2-VM] in network Ntwk[200|Public|1] with requested profile NicProfile[0-0-null-null-null
2016-10-01 03:55:32.752 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Allocating nic for vm VM[SecondaryStorageVm|s-2-VM] in network Ntwk[202|Control|3] with requested profile null
2016-10-01 03:55:32.753 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 47775c69) (ctx: 723322da) Submitting Async Job class org.apache.cloudstack.framework.jobs.impl.VmWorkJobVO for object type VmWorkJobQueue
2016-10-01 03:55:32.754 DEBUG [o.a.c.e.o.NetworkOrchestrator] (logid: 4a2cdd27) (ctx: ca3f04b1) Allocating nic for vm VM[SecondaryStorageVm|s-2-VM] in network Ntwk[201|Management|2] with requested profile null
2016-10-01 03:55:32.757 DEBUG [c.c.v.VirtualMachineManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Allocating disks for VM[SecondaryStorageVm|s-2-VM]
2016-10-01 03:55:32.761 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 47775c69) (ctx: 723322da) Sync job-14 execution on object VmWorkJobQueue.1
2016-10-01 03:55:32.767 DEBUG [c.c.v.VirtualMachineManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Allocation completed for VM: VM[SecondaryStorageVm|s-2-VM]
2016-10-01 03:55:32.769 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (logid: 4a2cdd27) (ctx: ca3f04b1) received secondary storage vm alert

That seems to complete just fine (deployment/resources/etc):

2016-10-01 03:55:32.767 DEBUG [c.c.v.VirtualMachineManagerImpl] (logid: 4a2cdd27) (ctx: ca3f04b1) Allocation completed for VM: VM[SecondaryStorageVm|s-2-VM]

This is the start job:

2016-10-01 03:55:33.406 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (logid: d4f62c28) (ctx: 38e4a9ae) (job: 13/job: 15) Add job-15 into job monitoring
2016-10-01 03:55:33.410 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 3bd85172) (ctx: 38e4a9ae) (job: 13/job: 15) Executing AsyncJobVO {id:15, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520732674657, completeMsid: null, lastUpdated: null, lastPolled: null, created: Sat Oct 01 03:55:32 CEST 2016}

This however fails:

2016-10-01 03:55:33.939 ERROR [c.c.v.VmWorkJobHandlerProxy] (logid: 3bd85172) (ctx: f0c74c94) (job: 13/job: 15) Invocation exception, caused by: java.lang.ClassCastException: net
.sf.cglib.proxy.Enhancer$EnhancerFactoryData cannot be cast to net.sf.cglib.proxy.Factory
2016-10-01 03:55:33.939 INFO  [c.c.v.VmWorkJobHandlerProxy] (logid: 3bd85172) (ctx: f0c74c94) (job: 13/job: 15) Rethrow exception java.lang.ClassCastException: net.sf.cglib.proxy.Enhancer$EnhancerFactoryData cannot be cast to net.sf.cglib.proxy.Factory
2016-10-01 03:55:33.939 DEBUG [c.c.v.VmWorkJobDispatcher] (logid: 3bd85172) (job: 13/job: 15) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 2, job origin: 13
2016-10-01 03:55:33.940 ERROR [c.c.v.VmWorkJobDispatcher] (logid: 3bd85172) (job: 13/job: 15) Unable to complete AsyncJobVO {id:15, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520732674657, completeMsid: null, lastUpdated: null, lastPolled: null, created: Sat Oct 01 03:55:32 CEST 2016}, job origin:13
com.cloud.exception.CloudException: Caught exception while handling a VmWorkJob
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:104)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:3477)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:89)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.runInContext(AsyncJobManagerImpl.java:260)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:39)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:38)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:84)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:35)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:36)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:206)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassCastException: net.sf.cglib.proxy.Enhancer$EnhancerFactoryData cannot be cast to net.sf.cglib.proxy.Factory
        at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.getTarget(AsyncCallbackDispatcher.java:79)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:160)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:756)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1187)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1028)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1936)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1769)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:88)
        ... 14 common frames omitted
2016-10-01 03:55:33.954 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 3bd85172) (job: 13/job: 15) Complete async job-15, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyACJjb20uY2xvdWQuZXhjZXB0aW9uLkNsb3VkRXhjZXB0aW9ueeiOG_HkRMcCAAJMAAtjc0Vycm9yQ29kZXQAE0xqYXZhL2xhbmcvSW50ZWdlcjtMAAZpZExpc3R0ABVMamF2YS91dGlsL0FycmF5TGlzdDt4cgATamF2YS5sYW5nLkV4Y2VwdGlvbtD9Hz4aOxzEAgAAeHIAE2phdmEubGFuZy5UaHJvd2FibGXVxjUnOXe4ywMABEwABWNhdXNldAAVTGphdmEvbGFuZy9UaHJvd2FibGU7TAANZGV0YWlsTWVzc2FnZXQAEkxqYXZhL2xhbmcvU3RyaW5nO1sACnN0YWNrVHJhY2V0AB5bTGphdmEvbGFuZy9TdGFja1RyYWNlRWxlbWVudDtMABRzdXBwcmVzc2VkRXhjZXB0aW9uc3QAEExqYXZhL3V0aWwvTGlzdDt4cHNyABxqYXZhLmxhbmcuQ2xhc3NDYXN0RXhjZXB0aW9ugAAFzs5n5VwCAAB4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhxAH4AA3EAfgAMdABcbmV0LnNmLmNnbGliLnByb3h5LkVuaGFuY2VyJEVuaGFuY2VyRmFjdG9yeURhdGEgY2Fubm90IGJlIGNhc3QgdG8gbmV0LnNmLmNnbGliLnByb3h5LkZhY3Rvcnl1cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABpzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABkwACGZpbGVOYW1lcQB-AAZMAAptZXRob2ROYW1lcQB-AAZ4cAAAAE90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmFzeW5jLkFzeW5jQ2FsbGJhY2tEaXNwYXRjaGVydAAcQXN5bmNDYWxsYmFja0Rpc3BhdGNoZXIuamF2YXQACWdldFRhcmdldHNxAH4AEAAAAKB0ADZvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS52b2x1bWUuVm9sdW1lU2VydmljZUltcGx0ABZWb2x1bWVTZXJ2aWNlSW1wbC5qYXZhdAAUY3JlYXRlQmFzZUltYWdlQXN5bmNzcQB-ABAAAAL0cQB-ABZxAH4AF3QAHWNyZWF0ZVZvbHVtZUZyb21UZW1wbGF0ZUFzeW5jc3EAfgAQAAAEo3QAPW9yZy5hcGFjaGUuY2xvdWRzdGFjay5lbmdpbmUub3JjaGVzdHJhdGlvbi5Wb2x1bWVPcmNoZXN0cmF0b3J0ABdWb2x1bWVPcmNoZXN0cmF0b3IuamF2YXQADnJlY3JlYXRlVm9sdW1lc3EAfgAQAAAEBHEAfgAccQB-AB10AAdwcmVwYXJlc3EAfgAQAAAHkHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHNxAH4AEAAABulxAH4AInEAfgAjcQB-ACRzcQB-ABD____-dAAkc3VuLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAdTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2Uwc3EAfgAQAAAAPnEAfgAncQB-ACh0AAZpbnZva2VzcQB-ABAAAAArdAAoc3VuLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgArc3EAfgAQAAAB8nQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ACtzcQB-ABAAAABYdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9ic3EAfgAQAAANlXEAfgAicQB-ACNxAH4ANXNxAH4AEAAAAFl0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4AEAAAAQR0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDF0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQB-ABAAAAAndAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-ABAAAAAmdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfgAQAAAAVHQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4ARXQAD2NhbGxXaXRoQ29udGV4dHNxAH4AEAAAACNxAH4ASHEAfgBFdAAOcnVuV2l0aENvbnRleHRzcQB-ABAAAAAkdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-AEFxAH4AQnNxAH4AEAAAAM5xAH4APHEAfgA9cQB-AEJzcQB-ABAAAAH_dAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-AEZzcQB-ABAAAAEKdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgBCc3EAfgAQAAAEdnQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgAQAAACaXQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4AV3EAfgBCc3EAfgAQAAAC6XQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgBCc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAh4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4AY3h0ACtDYXVnaHQgZXhjZXB0aW9uIHdoaWxlIGhhbmRsaW5nIGEgVm1Xb3JrSm9idXEAfgAOAAAAD3NxAH4AEAAAAGhxAH4AM3EAfgA0cQB-ADVzcQB-ABAAAA2VcQB-ACJxAH4AI3EAfgA1c3EAfgAQAAAAWXEAfgA4cQB-ADlxAH4AOnNxAH4AEAAAAQRxAH4APHEAfgA9cQB-AD5zcQB-ABAAAAAncQB-AEBxAH4AQXEAfgBCc3EAfgAQAAAAJnEAfgBEcQB-AEVxAH4ARnNxAH4AEAAAAFRxAH4ASHEAfgBFcQB-AElzcQB-ABAAAAAjcQB-AEhxAH4ARXEAfgBLc3EAfgAQAAAAJHEAfgBNcQB-AEFxAH4AQnNxAH4AEAAAAM5xAH4APHEAfgA9cQB-AEJzcQB-ABAAAAH_cQB-AFBxAH4AUXEAfgBGc3EAfgAQAAABCnEAfgBTcQB-AFRxAH4AQnNxAH4AEAAABHZxAH4AVnEAfgBXcQB-AFhzcQB-ABAAAAJpcQB-AFpxAH4AV3EAfgBCc3EAfgAQAAAC6XEAfgBccQB-AF1xAH4AQnEAfgBheHNyABFqYXZhLmxhbmcuSW50ZWdlchLioKT3gYc4AgABSQAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAABCzc3EAfgBiAAAAAHcEAAAAAHg
2016-10-01 03:55:33.955 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 3bd85172) (job: 13/job: 15) Publish async job-15 complete on message bus
2016-10-01 03:55:33.955 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 3bd85172) (job: 13/job: 15) Wake up jobs related to job-15
2016-10-01 03:55:33.955 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 3bd85172) (job: 13/job: 15) Update db status for job-15
2016-10-01 03:55:33.956 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 3bd85172) (job: 13/job: 15) Wake up jobs joined with job-15 and disjoin all subjobs created from job- 15
2016-10-01 03:55:33.961 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (logid: 3bd85172) (job: 13/job: 15) Done executing com.cloud.vm.VmWorkStart for job-15
2016-10-01 03:55:33.963 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (logid: 3bd85172) (job: 13/job: 15) Remove job-15 from job monitoring

Code shows it's rethrowing exception (VmWorkJobHandlerProxy.java:104):

                // legacy CloudStack code relies on checked exception for error handling
                // we need to re-throw the real exception here
                if (cause != null && cause instanceof Exception) {
                    s_logger.info("Rethrow exception " + cause);
                    throw new CloudException("Caught exception while handling a VmWorkJob", cause);
                }

Probably this is the real problem:

Caused by: java.lang.ClassCastException: net.sf.cglib.proxy.Enhancer$EnhancerFactoryData cannot be cast to net.sf.cglib.proxy.Factory
        at org.apache.cloudstack.framework.async.AsyncCallbackDispatcher.getTarget(AsyncCallbackDispatcher.java:79)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:160)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:756)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1187)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1028)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1936)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1769)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:88)

Method createBaseImageAsync hints on the systemvm template. Although the cpvm does work fine. It also shows the failed SSVM als being 'stopped':

When I click 'start" all is well:

After this, the templates get ready and the build continue just fine.

Full logs here:

management.log.txt
agent-kvm2.log.txt
agent-kvm1.log.txt

Use of SHA-1 in CertificateHelper.java

CertificateHelper.java uses SHA-1:

final MessageDigest md = MessageDigest.getInstance("SHA-1");

SHA-1 is insecure and it is recommended to switch to a more secure newer version (e.g. SHA-256).

Ability to see and change the default route on a VPC

Right now the default route is set when using a public interface. We would like to make this visible to the end user, and provide the ability to change it.

Validate input for algorithm in loadbalancer rules

RoundRobin makes HAProxy fail to start. Let's add validation on these calls.

When using a network offering with remote gateway, it's not possible to deploy a VM as gateway

I'd like to have the option to use a VM (with for example Cisco vASA) as a gateway instead of the RouterVM.

Currently the only way to do this is by using a remote gateway, however this prevents the gateway IP from being handed out to a VM (as it's considered an excluded IP).

In a typical deployment I'd like to reserve IP addresses ending with .1 and .2 for my ASA cluster, setting .1 as gateway. The RVMs can then use .3 and .4.

One way I can see this work is to set an IP exclusion list for .1 and .2, gateway to .1 and have an option to explicitly bypass the exclusion list and gateway IP restriction for my firewall instances and NICs.

On keepalived failover we saw static routes not applied

Logs show:

Oct 31 10:09:44 r-68201-nl2 Keepalived_vrrp[1644]: Netlink: error: Network is unreachable, type=(24), seq=1540825043, pid=0

which implies the interface was not yet up. Needs investigation.

Feature: Specify the MAC address of a VM's NIC

Specify the MAC address of a VM's NIC during its deployment or when adding a new NIC to it.

PR: apache/cloudstack#2143
JIRA issue: https://issues.apache.org/jira/browse/CLOUDSTACK-9949

PXE boot option per network tier on the RVM

Hi,

I'd like the possibility to configure PXE on a per tier bases in de DHCP options (66,67)
Currently got it running by adding the below config manually en restarting dnsmasq.

Example config:
[root@r-****** ~]# cat /etc/dnsmasq.d/pxe.conf
dhcp-option=tag:interface-eth5,67,
dhcp-option=tag:interface-eth5,66,

And adding the MAC to dhcphosts:
[root@r-****** ~]# cat /etc/dhcphosts.txt
...
...
00:00:00:00:00:00,set:11_11_11_11,11.11.11.11,hostname,infinite

Would be nice :)

Thanks!

Switch public_ip ACL to default_deny

This is more secure

Allow to change to a VPC offering with less services than the current one

Currently it doesn't allow this.

Instead of comparing the VPC services with the current offering, it should check the services in use by the network tiers (network offering) and match those with the new VPC offering.

Should SourceNat, VPN be deleted, we also need to release the public ip address.

Ping /me or @ddegoede for more info if needed

No access to Slack channel for external emails (not @schubergphilis.com)

The Slack channel requires a @schubergphilis.com email address to join. Is there a community forum or somewhere else I could use for installation questions?

Request support for haproxy PROXY-protocol on loadbalancer backends

https://www.haproxy.com/blog/using-haproxy-with-the-proxy-protocol-to-better-secure-your-database/
https://www.haproxy.com/blog/preserve-source-ip-address-despite-reverse-proxies/
https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html
https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

HAProxy and other applications (such as nginx) support the PROXY-protocol to pass through the original source connection details to backend servers.

This becomes very helpful when, for example, nginx terminates TLS and needs to know the client's source-ip. Because the connection is encrypted end-to-end, HAProxy cannot add X-Forwarded-For headers.

The PROXY-protocol can be enabled by setting the following option on HAProxy backends.

send-proxy (enjoys wide support)
OR
send-proxy-v2

Because the receiving application needs explicit support for the PROXY-protocol, this should be a feature which can be enabled per loadbalancer but is disabled by default.