SDN - Using POX Openflow controller to program a Juniper EX switch Pt.2

The last blog entry I demonstrated how to configure a Juniper EX9200 switch to communicate with an Openflow controller (POX). The flow entry logic was very basic. It turned all the ports on the Openflow switch into a dumb hub. Flood out all ports except for the source.

In this post I'll demonstrate how to create a more unique flow entry.  This is based on the 10-tuple OpenFlow v1.0 header fields which are:

Ingress Port, Ethec Src, Ether Dst, Ether Type, Vlan ID, IP Dst, IP Src, TCP Dst, TCP Src, and IP Proto

First we need to figure out what a host does on a layer 2 network. If Host A wants to talk to Host B, it will first send out an ARP packet asking what is the mac address of Host B based on the IP address. After recieving a reply, Host A will send a unicast frame with the destnation Mac of Host B. Host B in turn will send unicast packets back to Host A. This means we need to program three types of flows.  One for ARP and two for unicast packets for each direction.

Below is the output in action:

jnpr@EX9200-RE0> show openflow flows detail   

jnpr@EX9200-RE0>

Next I startup the openflow controller:
jnpr@ubuntu:~/OPENFLOW-CONTROLLER$ ./pox.py misc.static
POX 0.2.0 (carp) / Copyright 2011-2013 James McCauley, et al.
INFO:core:POX 0.2.0 (carp) is up.
INFO:openflow.of_01:[3c-8a-b0-0d-c7-c0 1] connected
INFO:misc.static:sending to DPID ARP flow creation 
INFO:misc.static:sending to DPID flow creation->
INFO:misc.static:sending to DPID flow creation<-

On the EX switch:

jnpr@EX9200-RE0> show openflow flows detail   
Flow name: flow-16842752
Table ID: 1     Flow ID: 16842752           
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0     
Match: Input port: 1    
       Ethernet src addr: wildcard         
       Ethernet dst addr: wildcard         
       Input vlan id: 100               Input VLAN priority: wildcard
       Ether type: 0x800  
       IP ToS: wildcard                 IP protocol: wildcard
       IP src addr: wildcard            IP dst addr: 10.1.1.3/32     
       Source port: 0                   Destination port: 0     
Action: Output port 2,

Flow name: flow-33619968
Table ID: 1     Flow ID: 33619968           
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0     
Match: Input port: 2    
       Ethernet src addr: wildcard         
       Ethernet dst addr: wildcard         
       Input vlan id: 100               Input VLAN priority: wildcard
       Ether type: 0x800  
       IP ToS: wildcard                 IP protocol: wildcard
       IP src addr: wildcard            IP dst addr: 10.1.1.2/32     
       Source port: 0                   Destination port: 0     
Action: Output port 1,

Flow name: flow-83951616
Table ID: 1     Flow ID: 83951616           
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0     
Match: Input port: wildcard
       Ethernet src addr: wildcard         
       Ethernet dst addr: wildcard         
       Input vlan id: wildcard          Input VLAN priority: wildcard
       Ether type: 0x806  
       IP ToS: 0x0                      IP protocol: wildcard
       IP src addr: wildcard            IP dst addr: wildcard        
       Source port: 0                   Destination port: 0     
Action: Output port 65531,


jnpr@EX9200-RE0> show openflow flows          
Switch                 Flow      Number of packets    Priority Number of Number of
Name                   ID                                            match    action
oftest-92k             16842752      248                     32768    6         1       
oftest-92k             33619968      248                     32768    6         1       
oftest-92k             83951616        4                       32768    4         1      


Python code for POX

jnpr@ubuntu:~/OPENFLOW-CONTROLLER/pox/misc$ cat static.py

---------------------
from pox.core import core
from pox.lib.packet.ipv4 import ipv4
from pox.lib.packet.arp import arp
import pox.lib.packet as pkt
import pox.openflow.libopenflow_01 as of
import re

log = core.getLogger()

class Sdn (object):
  def __init__ (self, connection):
    self.connection = connection
    connection.addListeners(self)


    # FIRST OPENFLOW RULE - ARP
    #create an Openflow using flow table modification
    arp = of.ofp_flow_mod()
    #Define a match structure for your flow rules to follow
    #if ether type is 0x0806 or Arp
    arp.match.dl_type = 0x0806
    # Add an action to send to flood out all ports except the source
    arp.actions.append(of.ofp_action_output(port = of.OFPP_FLOOD))
    # program flow on switch
    self.connection.send(arp)
    #Send debug message for controller
    log.info("sending to DPID ARP flow creation  " )

    # FIRST OPENFLOW RULE
    #create an Openflow using flow table modification
    msg = of.ofp_flow_mod()
    #Define a match structure for your flow rules to follow
    msg.match.in_port = 1
    msg.match.dl_vlan = 100
    msg.match.dl_type = 0x0800
    msg.match.nw_dst = "10.1.1.3/32"
    # Add an action to send to the specified port
    msg.actions.append(of.ofp_action_output(port = 2))
    # Send message to switch
    self.connection.send(msg)
    #Debug message for controller
    log.info("sending to DPID flow creation->" )

    # SECOND OPENFLOW RULE
    #create an Openflow using flow table modification
    msg2 = of.ofp_flow_mod()
    #Define a match structure for your flow rule
    msg2.match.in_port = 2
    msg2.match.dl_vlan = 100
    msg2.match.dl_type = 0x0800
    msg2.match.nw_dst = "10.1.1.2/32"
    # Add an action to send to the specified port
    msg2.actions.append(of.ofp_action_output(port = 1))
    # Send message to switch
    self.connection.send(msg2)
#Debug message for controller
    log.info("sending to DPID flow creation<-" )


def launch ():
  """
  Starts the component
  """
  def start_switch (event):
    log.debug("Controlling DPID %s" % (event.connection,))
    Sdn(event.connection)
  core.openflow.addListenerByName("ConnectionUp", start_switch)

SDN - Using POX Openflow controller to program a Juniper EX switch Pt.1

Openflow support is now available on Juniper EX switches in version 13.3 of code. I decided to explore and test this out. I first needed to find an Openflow compatible Juniper platform. The EX9200 supports Openflow, but you have to add the openflow image to JUNOS

jnpr@EX9200-RE0>request system software add <jsdn-package-name>

After installation you should see it as a module:

 jnpr@EX9200-RE0# run show version
Hostname: EX9200-RE0
Model: ex9204
Junos: 13.3R1.6
JUNOS Base OS boot [13.3R1.6]
JUNOS Base OS Software Suite [13.3R1.6]
JUNOS 64-bit Kernel Software Suite [13.3R1.6]
[TRUNCATED]
JUNOS py-base-i386 [13.3R1.6]
JUNOS SDN Software Suite [13.3R1.6]

There are a few possible operational commands:

jnpr@EX9200-RE0> show openflow ?
Possible completions:
  capability           Show feature and configuration capability
  controller           Show controller information and connection status
  filters              Show filter information
  flows                Show flow information
  interfaces           Show interface information
  statistics           Show statistics commands
  summary              Show openflow information summary
  switch               Show switch instance description information

To configure openflow, you need to:

1) Create interfaces

2) Associate those interfaces to an OPENFLOW resource group (ie. a virtualized switch)

3) Point to the Openflow controller to receive commands.


First, the creation of interfaces. This is very basic, it's similar to creating normal switching interfaces:

jnpr@EX9200-RE0# show interfaces
xe-2/0/0 {
    unit 0 {
        family ethernet-switching {
            vlan {
                members v100;
            }
        }
    }
}
xe-2/0/1 {
    unit 0 {
        family ethernet-switching {
            interface-mode access;
            vlan {
                members v100;
            }
        }
    }
}

[edit]
jnpr@EX9200-RE0# show vlans
v100 {
    vlan-id 100;
}

Next, you need to place these interfaces into an Openflow resource group (ie a virtualized switch) and map those interfaces to a port-id that Openflow can understand.

jnpr@EX9200-RE0# show protocols openflow
switch OF-SWITCH-92k {
    default-action {
        packet-in;
    }
    interfaces {
        xe-2/0/0.0 port-id 1;
        xe-2/0/1.0 port-id 2;
    }
}


Then you'll need to point to an openflow controller.

set protocols openflow switch OF-SWITCH-92k controller address 10.161.11.77

By default, openflow communicates over TCP using port 6633

Once committed, the EX will continously try to communicate with the openflow server.

jnpr@EX9200-RE0# run show openflow controller                                            
Openflowd controller information:
Controller socket: 12
Controller IP address: 10.161.11.77
Controller protocol: tcp
Controller port: 6633
Controller connection state: down
Number of connection attempt: 10
Controller role: equal

If the EX was correctly communicating with the controller, the connection state should say "up".

jnpr@EX9200-RE0# run show openflow switch
Switch Name:        OF-SWITCH-92k                                             
Switch ID:          0                  Switch DPID:    00:00:3c:8a:b0:0d:c7:c0
Flow mod received:  6                  Vendor received:      0             
Packets sent:       841                Packets received:     845           
Echo req sent:      833                Echo req received:    0             
Echo reply sent:    0                  Echo reply received:  833           
Port Status sent:   0                  Port mod received:    0             
Barrier request:    0                  Barrier reply:        0             
Error msg sent:     0                  Error msg received:   0  

The DPID or DataPathIdentifier is the physical switch that will be programmed. We'll see where this comes into play later.

I created an Ubuntu VM and installed the POX openflow controller. It's Python based which is awesome since it's a language I can understand. There's Floodlight (Java) or Trema (Ruby), so choose the flavor that you're comfortable with.

To simply start with there is a very basic POX python module found under the forwarding directory. This is the module we'll start with just to demonstrate how this works.

jnpr@ubuntu:~/OPENFLOW-CONTROLLER$ ./pox.py log.level --DEBUG forwarding.hub
POX 0.2.0 (carp) / Copyright 2011-2013 James McCauley, et al.
INFO:forwarding.hub:Hub running.
DEBUG:core:POX 0.2.0 (carp) going up...
DEBUG:core:Running on CPython (2.7.5+/Feb 27 2014 19:37:08)
DEBUG:core:Platform is Linux-3.11.0-12-generic-x86_64-with-Ubuntu-13.10-saucy
INFO:core:POX 0.2.0 (carp) is up.
DEBUG:openflow.of_01:Listening on 0.0.0.0:6633
INFO:forwarding.hub:Hubifying 3c-8a-b0-0d-c7-c0  <<<< DPID of the EX Switch

What this does to the EX switch basically turns those two interfaces in a small hub.

On POX the gist of the python script is this:


  msg = of.ofp_flow_mod()
  msg.actions.append(of.ofp_action_output(port = of.OFPP_FLOOD))

In Openflow terminology:

OFPP_FLOOD - output all openflow ports except the input port and those with flooding disabled

And that's it.

On the EX the Openflow rule looks like this:

jnpr@EX9200-RE0# run show openflow flows detail
Flow name: flow-65536
Table ID: 1     Flow ID: 65536            
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0   
Match: Input port: wildcard
       Ethernet src addr: wildcard       
       Ethernet dst addr: wildcard       
       Input vlan id: wildcard          Input VLAN priority: wildcard
       Ether type: wildcard
       IP ToS: 0x0                      IP protocol: 0x0 
       IP src addr: 0.0.0.0/32          IP dst addr: 0.0.0.0/32    
       Source port: 0                   Destination port: 0   
Action: Output port 65531,


The Action: Output port 65531  basically means everything available openflow interface but where the source of the packet came in on.


I setup a few tester ports to send constant traffic at a very low rate. ~1 pps


jnpr@EX9200-RE0# run show openflow statistics interfaces 
Switch Name: OF-SWITCH-92k                                                     
Interface Name: xe-2/0/0.0       Port Number: 1   
Num of rx pkts: 12                         Num of tx pkts: 12                  
Num of rx bytes: 17952                     Num of tx bytes: 17952               
Num of rx error: 0                         Num of tx error:0                   
Number of packets dropped by RX: 0                   
Number of packets dropped by TX: 0                   
Number of rx frame error:        0                   
Number of rx overrun error:      0                   
Number of CRC error:             0                   
Number of collisions:            0                   

Switch Name: OF-SWITCH-92k                                                     
Interface Name: xe-2/0/1.0       Port Number: 2   
Num of rx pkts: 12                         Num of tx pkts: 12                  
Num of rx bytes: 17952                     Num of tx bytes: 17952               
Num of rx error: 0                         Num of tx error:0                   
Number of packets dropped by RX: 0                   
Number of packets dropped by TX: 0                   
Number of rx frame error:        0                   
Number of rx overrun error:      0                   
Number of CRC error:             0                   
Number of collisions:            0                   

In my next blog entry I'll tweak a POX module to create flow rule entries that coincide more with SDN programming than turning your expensive switch into an expensive hub.

SDN - A look at Openflow running on white box switches.

In my quest to understand SDN, I attended a presentation on Openflow running on whitebox switches by Pica8 Open Networking. Their goal is to commoditize switching hardware and allow the control plane decisions to be made by a controller using Openflow. Pica8 has a cheap whitebox switch that runs Open Vswitch. The switches can be setup through Zero Touch provisioning, meaning you take an unconfigured box, drop it into the network and it will communicate with a server and automagically configure itself. They run a very light weight open source operating system with hardly any functionality which keeps costs low. I've read on the internet that their switches sell at half the cost of other vendor equipment. 

         While this may sound enticing, you get what you pay for. The size of their TCAM is around 1K-2K entries for some of their switches and 10k entries for others. This is pretty low! Why is the TCAM important? Think of it as the table size for an ACL entry. This is the 5 tuple flow where you "program" the switch based on the source and destination IP address, source and destination port and protocol. This is the gist of how Openflow programs the switches in your datacenter.

The low number of entries means that Pica8 has to limit the size of a data center. They resolve this limitation by recommending grouping switches into different "clusters". (Note: clusters is my name for it, as they called it a unit of calculation) They've calculated that a typical cluster can scale to a maximum of 12 racks of servers with two TOR switches, two AGG switches and two CORE boxes. When you want to expand, you create another "cluster" of switches/servers and interconnect them through the CORE boxes. This seems to be a waste of ports at the AGG layer.

          I believe the reason for this is based on the TCAM problem. If the AGG layer runs out of TCAM space, you are forced to build another cluster. I can imagine that each cluster is managed by a separate SDN controller. Theoretically a single SDN controller could manage all the clusters, but they didn't really talk about it.

           

          Pica8 at the time of writing has 200 customers, but ZERO deployments. Which means that even though SDN and Openflow is a cool technology, no customer in their right mind is going to put this in their production environment until the technology is more mature.

Now some of the problems with this implementation is that you literally have to program your network. You first have to create drop flow profiles. If you don't want IPv6 traversing your switch, create a drop flow. No multicast, create a drop flow. Then you create your forwarding entry. Need to go from a VM on port 1 to a VM on port 2. Create a forwarding entry. Now imagine that you have 48 ports per switch and two TOR switches, that right there means 96 entries in a single direction. Add the reverse direction and you have 192 entries to program. As you can see this could get very tedious. 

    

         Hopefully someone has a controller that can do this automatically. Once you run out of TCAM space you'll have to move VMs to a new cluster. Which brings me to the second issue, which is support for VMotion. Moving VMs between servers requires you to reprogram flows. But VMotion will have to integrate with the SDN Controller for this to work. This is also a flow based mechanism which means that this can be susceptible to DDOS attacks. Just send a bunch of arps and the switch will punt this to the controller. Get enough ARPs and you can overwhelm a controller and bring down a cluster.

Next they discussed network diagnostics. This was a very interesting topic. Where do you put this? On a normal switch you have counters and can retrieve them typically through SNMP. But on an Openflow switch, the hardware is supposed to be dumb. You need to put this on the controller. But how do you access a switch's counters without compromising performance. Do you retrieve this though Openflow? Is there another Northbound connection that will be both lightweight and scalable? Also Pica8 mentioned that some counters such as ingress and egress port statistics were not easily accessible. Another issue was when an upstream AGG switch did not have a proper flow entry it blackholed the packet and sent flow control packets down stream to the TOR, which filled up the buffers, thus preventing no packets to be forwarded from the TOR. They had to drop down to the switch's debug level to figure this out. In this scenario, where is the troubleshooting? You think it's a TOR issue, but in fact it's an AGG issue. This is a big concern I have about SDN, lack of diagnostics to troubleshoot issues. There is not enough visibility into the network to trace down the problem.

          While Openflow is an interesting technology this implementation is not yet mature and requires a lot of customization. Because of the limitation in a switches hardware this is not a scalable solution. You also need an intelligent controller that can automate your flow entries in a simple manner. 

Now to resolve this hardware issue I can imagine building a switch like building a bare metal server. Make the parts swappable. Running out of TCAM? Pull out the current one and install a new one just like you can swap out RAM and CPU Cores. White boxes need to be built so that their network connections stay in place and you swap out the FRUs around it. There also needs to be a way to get the optics to the point where they are tri-rate like copper links. Need to upgrade from 1G to 10G to 40G to 100G? Just update the flash. However this will also need some kind of black plane to upgrade the switch fabric.. Commoditized hardware needs to be built modularly. But this may be a Chassis based switch, not a TOR. However technology is constantly shrinking things down while packing more punch so I can imagine that eventually this will happen.

SDN - Unbundling platforms via Cumulus Networks

I went to a Meetup the other night and saw a presentation by JR Rivers (Co-founder) of Cumulus Networks.

He had an interesting point that most Enterprise networks have hardware that run a single OS. For instance a customer Data Center will run Cisco hardware running Cisco IOS. As it is, the Network Admin is stuck using that vendor and cannot move to another vendor without a forklift upgrade.

Who wants to reconnect links?

JR's proposition is to basically take what's currently going on in the server market and redefine the networking industry.

If you don't like your current vendor's OS, then you can re-image the gear and run the Network OS flavor of your choice.

I can see why Cumulus has taken this approach. In order to disrupt an incumbent, you need a way to remove them from the environment without disrupting the environment.

I can see this working on whitebox gear, but not really much on incumbent gear. A question from the audience was raised, who does the support? JR said that they would support software related issues. But what happens if Cumulus OS is running on top of a Cisco box? Good luck trying to call Cisco. They would automatically void the warranty in a situation like this. The one good thing about having a single vendor environment is that there is support. You know that the vendor will fix their bugs. In this new environment, you're not sure who's going to take responsibility to fix the bug. Also Cumulus is currently only targeting TOR boxes, so this unbundling of platforms will currently only happen in the Data Center.

The value that Cumulus adds is that their OS is Linux. With Linux, DevOps is very familiar with this OS and can program both the APP and the network, hence the SDN part. If Cisco doesn't have a feature in IOS that you need, as a customer you have to ask for a feature request, give your SE a dollar value tied to this feature and then wait x months for the new version to come out. Of course that software upgrade costs money.

As a Demo, JR showed us a BASH script that solved a problem. His point, he created the feature himself. It didn't cost him any money or require him to upgrade his OS.

Now for some vendors like Cisco you could do scripting off box using screen scrapes. But other vendors already have this capability. For instance, Arista has Python built into their OS, so JR's value add doesn't really hold a lot of weight. Juniper has a scripting language called SLAX which is based on XML/XLST.

A problem I see is that the OS is Linux and Security is a big problem. The reason why some vendors don't want to open up a scripting language on their box is because of this very reason. If a hacker is able to take control of your OS, they could create malicious code to do all sorts of things. I wouldn't want to be the Network Admin who has to figure out why their network is all of a sudden DDOSing their own servers and other networks. Which is why Juniper removed Perl from their early implementation of Junos and also why they support SLAX which basically prevents users from messing around with the kernel.

An audience member asked about JR's take on OpenFlow.  He was mostly against this. His response was that there would be lots of problems getting all the vendors to support this. This is an incorrect assumption. Networking vendors always update their OSes to support new protocols. As a vendor, you have to adapt to new things or you die.

I believe JR's reluctance is the threat OpenFlow can create. A SDN controller using OpenFlow can basically take the brains out of a switch. If you have 100 switches, you basically have 100 OSes,  a separate OS running on each switch. With Openflow, you can theoretically control all 100 switches with a single SDN controller and not rely on the underlying OS to do the decision making. Program the whole network, not just each individual switch.

While Cumulus does make an interesting point in unbundling platforms, SDN using a controller that is able to program your whole network is way more interesting.