globalnoc / flowspacefirewall Goto Github PK

Looking at the code, it appears that ofp_queue_get_config_requests get handled in OFControllerChannelHandler.java by the method processOFQueueGetRequest. I see two issues with the handler:

• ofp_queue_get_config_request are responded to with ofp_get_config_reply instead of ofp_queue_get_config_reply
• The reply is not using the same XID as the request

I am seeing this in the 1.0.5-dev branch. This is non-critical for GENI, because we haven't exposed queues to our experimenters.

The main issue is that the response will violate the principle of least surprise if someone does send a queue get config message, and it will interact poorly with some OF libraries that handle symmetric messages by keying off of the XID.

Disclaimer: I know very little about creating schemas, but I think this is possible...

It would be useful to restrict the range element's start and end attributes according to the VLAN range you allow. The current error message of:

Exception in thread "main" java.lang.IllegalArgumentException: VLAN ID 0 is out of range for valid vlan tags
    at edu.iu.grnoc.flowspace_firewall.VLANRange.setVlanAvail(VLANRange.java:69)
    at edu.iu.grnoc.flowspace_firewall.ConfigParser.parseConfig(ConfigParser.java:178)
    at edu.iu.grnoc.flowspace_firewall.FlowSpaceFirewall.init(FlowSpaceFirewall.java:402)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.initModules(FloodlightModuleLoader.java:439)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromList(FloodlightModuleLoader.java:356)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromList(FloodlightModuleLoader.java:365)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromConfig(FloodlightModuleLoader.java:203)
    at net.floodlightcontroller.core.Main.main(Main.java:55)

is actually pretty obvious, but catching these kinds of error during the xmllint stage might help prevent downtime.

I was playing around with Floodlight static flow pusher on top of FSFW, and I was able to create 18 flows in an ovs instance when my slice limit was set to 10:

{
    "DPID": "d0:0d:de:ad:be:a7:da:d5", 
    "connected": true, 
    "flow_rate": 1111.1111111111111, 
    "max_flow_rule": 1, 
    "max_flows": 10, 
    "max_packet_in_rate": 1, 
    "packet_in_rate": 0.016272031924424873, 
    "total_flows": 18
}

I sent flows that did the following:

• Match VLAN ID 100
• Match MAC address (incrementing the src MAC by 1 for each flow)
• Output to a specific port (2)

My slice included ports 1 and 2, and VLAN 100 on both ports. By not specifying the in_port, I believe that FSFW split my flowmods into two flowmods, which match on in_port 1 or in_port 2. I have not investigated what caused the 18 flows (flow accounting issues vs. race conditions, etc).

Here is the resulting flowtable as reported by Open vSwitch:
https://gist.github.com/gth828r/e2efcdea6e87d647c6b1#file-gistfile1-txt

The current properties file distributed in 1.2 and on the website at http://globalnoc.iu.edu/running-flowspace-firewall/overriding-the-default-floodlight-config.html has a few minor issues.

1. Some of the commas and line continuation characters are missing from the config lines that set the list of modules that are assigned to floodlight.modules. Properly including some of those modules helps to remove some of the startup warnings.
2. We should explicitly remove the inclusion of net.floodlightcontroller.devicemanager.internal.DefaultEntityClassifier from floodlight.modules. It causes issues on startup when properly included, and leaving it as is in the config file is confusing (and conflicts with item 1 above!)
3. The path to flushSwitchesOnReconnect is incorrect, and therefore the variable is not being set (and it causes warnings on startup). The correct full path in the version of floodlight that the code is based on is net.floodlightcontroller.core.internal.FloodlightProvider.flushSwitchesOnReconnect.
4. Some debug modules are included by default, but in an operational setting those likely shouldn't be used. This is just a minor optimization.

Below is a diff of the changes I am proposing:

--- /users/tupty/floodlight_jar/floodlightdefault.properties    2014-07-18 10:21:55.000000000 -0400
+++ /etc/fsfw/floodlight.properties 2014-07-18 10:51:20.000000000 -0400
@@ -1,17 +1,13 @@
 floodlight.modules=\
-net.floodlightcontroller.jython.JythonDebugInterface,\
 net.floodlightcontroller.counter.CounterStore,\
 net.floodlightcontroller.storage.memory.MemoryStorageSource,\
 net.floodlightcontroller.core.internal.FloodlightProvider,\
-net.floodlightcontroller.debugcounter.DebugCounter,\
-net.floodlightcontroller.debugevent.DebugEvent,\
 net.floodlightcontroller.perfmon.PktInProcessingTime,\
-edu.iu.grnoc.flowspace_firewall.FlowSpaceFirewall
-org.sdnplatform.sync.internal.SyncManager
-org.sdnplatform.sync.internal.SyncManager
+edu.iu.grnoc.flowspace_firewall.FlowSpaceFirewall,\
+org.sdnplatform.sync.internal.SyncManager,\
+org.sdnplatform.sync.internal.SyncManager,\
 org.sdnplatform.sync.internal.SyncTorture
-net.floodlightcontroller.devicemanager.internal.DefaultEntityClassifier
 org.sdnplatform.sync.internal.SyncManager.authScheme=CHALLENGE_RESPONSE
 org.sdnplatform.sync.internal.SyncManager.keyStorePath=/etc/floodlight/auth_credentials.jceks
 org.sdnplatform.sync.internal.SyncManager.dbPath=/var/lib/floodlight/
-net.floodlightcontroller.core.FloodlightProvider.flushSwitchesOnReconnect = false
+net.floodlightcontroller.core.internal.FloodlightProvider.flushSwitchesOnReconnect = false

I strongly suggest thoroughly testing these changes before accepting them as-is. I have verified that in my setup, FSFW still starts up properly, and it doesn't include any messages other than what modules were loaded. I didn't verify much beyond that.

I am using a snapshot of master from Friday (f9ab98d), and I am trying the following test:

• Enable a slice
• Install flows into data plane switch, very high idle timeout
• Start sending traffic in the data plane, do not stop
• Remove the slice in fsfw.xml file
• Do a sudo service fsfw reload

When I do this, the data plane traffic continues to succeed, and I never see the flows timeout. I have seen this when the slice is in managed tag mode as well as when managed tag mode is disabled for the slice. I have not started looking into the packet captures and logs yet, but I will attach them here so you all can start looking while I look as well.

The packet captures are only of the southbound direction from FSFW, and the logs are all in debug mode for FSFW. I will also include the fsfw.xml files in both cases before the slice was removed.

OFControllerChannelHandler.java has a method processOFConfigRequest that directly handles GET_CONFIG_REQUEST messages from the controller. That method constructs the GET_CONFIG_REPLY.

The GET_CONFIG_REPLY should have the same transaction ID as the request, but the transaction ID is never set in the processOFConfigRequest method. Controllers that expect a response think that it never arrives, since the message that does arrive doesn't have a matching transaction ID.

SlicerStatusResource.java has the following:
results.put("max_flows_per_sec", myProxy.getSlicer().getMaxFlows());
results.put("max_flow_rule", myProxy.getSlicer().getMaxFlowRate());

I believe those lines should actually be:
results.put("max_flows_per_sec", myProxy.getSlicer().getMaxFlowRate());
results.put("max_flow_rule", myProxy.getSlicer().getMaxFlows());

When using managed tag mode, the total_len field in packet-in messages is incorrect when the packet-in reaches the upstream controller. Note that the length field (tracking the size of the OF packet) is set correctly, and just the total_len (tracking the size of the data frame encapsulated in the OF packet) is incorrect.

As an example, I took a packet capture of a packet-in as it came in from the switch and as it went out to the controller with the switch under managed tag mode for the slice.

Here it is coming in:

Note that the byte count in the lower left matches the total_len field in the ofp_packet_in header.

Now, here is a snapshot of a packet leaving:

Again, note the total_len field as compared to the byte count in the lower left. This time, they are off by four.

I was doing testing, and I noticed that FSFW was installing a default-drop rule when I didn't expect it to. Looking at the code, I noticed that the accessor for install_default_drop at https://github.com/GlobalNOC/FlowSpaceFirewall/blob/master/src/main/java/edu/iu/grnoc/flowspace_firewall/SwitchConfig.java#L32 is flush_on_connect.

Currently, I believe that Flowspace Firewall uses an EPERM error in the case where a flow is outside of the controller's flowspace in addition to when the controller has surpassed the number of flows allocated to it's slice. In the latter case, I think it would make more sense to return an OFPFMFC_ALL_TABLES_FULL error message.

Flowspace firewall currently polls the downstream device for flow stats at a rate of once every ten seconds. It would be nice if the rate at which flow stats requests were sent out was configurable. Some controlled environments with specific applications and high end switches might want to query more quickly, while other environments might benefit from having a slower polling rate.

I think that one of the changes that @thompsbp made in 89bab59 was lost along the way. In particlar, VLANSlicer.java line 456 in both master and 1.0.5-dev currently are set to log at the error level, whereas 89bab59 has a change to that message to be at the debug level, which makes more sense to me. Currently, I am seeing many entries in my log that look like this:

ERROR [e.i.g.f.VLANSlicer] Packet Out Action: OUTPUT

Currently, I believe it is only possible to define slices by hand-modifying the fsfw.xml file. It would be nice if the REST API included a way to create and delete slices in order to support orchestration tools.

When letting FSFW expand an OFPP_ALL port in a packet-out, it includes the in_port of the packet-out. This might be true of flowmods as well... I have not looked into that yet.

According to page 19 of the spec:

OFPP_ALL = 0xfffc, /* All physical ports except input port. */

The port expansion that FSFW does for an ALL port should intentionally exclude the in_port from the expansion.

1. If packet_in_rate is not defined for a switch in the config file, then a null pointer exception is thrown with the following stack trace:

Exception in thread "main" java.lang.NullPointerException
    at edu.iu.grnoc.flowspace_firewall.ConfigParser.parseConfig(ConfigParser.java:155)
    at edu.iu.grnoc.flowspace_firewall.FlowSpaceFirewall.init(FlowSpaceFirewall.java:402)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.initModules(FloodlightModuleLoader.java:439)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromList(FloodlightModuleLoader.java:356)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromList(FloodlightModuleLoader.java:365)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromConfig(FloodlightModuleLoader.java:203)
    at net.floodlightcontroller.core.Main.main(Main.java:55)

1. If flow_rate is not defined for a switch in the config file, then a null pointer exception is thrown with the following stack trace:

Exception in thread "main" java.lang.NullPointerException
    at edu.iu.grnoc.flowspace_firewall.ConfigParser.parseConfig(ConfigParser.java:153)
    at edu.iu.grnoc.flowspace_firewall.FlowSpaceFirewall.init(FlowSpaceFirewall.java:402)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.initModules(FloodlightModuleLoader.java:439)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromList(FloodlightModuleLoader.java:356)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromList(FloodlightModuleLoader.java:365)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromConfig(FloodlightModuleLoader.java:203)
    at net.floodlightcontroller.core.Main.main(Main.java:55)

1. If max_flows is not defined for a switch in the config file, then a null pointer exception is thrown with the following stack trace:

Exception in thread "main" java.lang.NullPointerException
    at edu.iu.grnoc.flowspace_firewall.ConfigParser.parseConfig(ConfigParser.java:151)
    at edu.iu.grnoc.flowspace_firewall.FlowSpaceFirewall.init(FlowSpaceFirewall.java:402)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.initModules(FloodlightModuleLoader.java:439)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromList(FloodlightModuleLoader.java:356)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromList(FloodlightModuleLoader.java:365)
    at net.floodlightcontroller.core.module.FloodlightModuleLoader.loadModulesFromConfig(FloodlightModuleLoader.java:203)
    at net.floodlightcontroller.core.Main.main(Main.java:55)

I believe that in all cases, running xmllint against the provided schema validates the config file, which makes this error hard to diagnose without looking at the code.

for basic controllers that can operate in a single vlan add the ability to append/remove the vlan tags for a slice. The goal here is for some slices they only need a single vlan tag (ie... ip over sdn) and let FSFW do all of the vlan tag/translations needed without the end controller needing the smarts to do tis.

Sometimes when debugging strange situations, it would be nice to have a timestamped log of the xid mapping. This can be useful when looking at packet captures between the controller and FSFW, and FSFW and the substrate. It makes it easier to correlate messages between the packet captures.

It would be nice if FSFW could install default drop rules for switch platforms that don't currently drop unmatched traffic by default. It would also be nice if this was an option that was configurable per switch defined in the fsfw.xml config file.

I accidentally discovered this problem when I was running some test cases that worked without managed tag mode, but then I switched over to managed tag mode. The relevant pieces of my setup procedure were as follows:

• Install fsfw
• Configure fsfw slices ahead of time (with managed tag mode on)
• Start fsfw service
• Turn up an ovs instance and point it at FSFW
• Add the ports to the ovs instance to support the topology I desire

If you are curious, you can see the last two steps in the script at https://gist.github.com/gth828r/f92081f20c2b5d2cf1fe

This approach worked just fine when I wasn't using managed tag mode, but when I did use managed tag mode, I consistently hit https://github.com/GlobalNOC/FlowSpaceFirewall/blob/master/src/main/java/edu/iu/grnoc/flowspace_firewall/VLANSlicer.java#L267 because the portId was being reported as zero.

I specifically saw this when my controller was sending a packet-out to the ALL port for an arp request packet.

Doing a:

sudo service fsfw reload

fixes the issue.

If a flow_mod is installed with a match that wildcards the VLAN ID, FSFW will apply the slice's VLAN tag for a given port if tag management is enabled. If a flow_removed message is generated, the VLAN tag is still present in the match that is returned to the controller.

I am not actually sure about this one, but you all would know better. In VLANSlicer.java (and perhaps other places) there is logic to check if certain values (link in_port or dl_vlan) are wildcarded.

Some examples:

        //if you wildcarded the vlan then tough we are blowing up now
        //we require an input vlan
        if(match.getDataLayerVirtualLan() == 0){
            log.debug("Slice: " + this.getSliceName() + ":" + this.sw.getStringId() + " Flow rule VLAN wildcarded.  Denied: " + flowMod.toString());
            return flowMods;
        }

        //wildcarded input port?
        if(match.getInputPort() == 0){

        //we require an input vlan
        if(match.getDataLayerVirtualLan() == 0){
            log.debug("VLAN is wildcarded");
            return false;
        }

Is this method of checking for wildcards safe? For example, if I wrote a controller that set the VLAN bit in the wildcard bitmask, but set the actual value of the VLAN ID to 123, could that potentially bypass the checks listed above and send my VLAN-wildcarded flow to the switch?

I am not an experienced floodlight developer, so I am not actually sure.