How to install Contrail on a single node Ubuntu system.

I wanted to install and test out a contrail set on a single node system.

I first started with a clean Ubuntu server. Contrail seems to have support on 12.04LTS version of code (12.04.03 to be precise). So I downloaded the amd64-iso file and did a fresh install on a system.

After it came up I did some package adds to get the machine prepped.

First I made sure I was root.
sudo su
passwd
I gave root a password. This is to be used later when Contrail tries to install packages using root.

This are the files I noticed while looking at different blogs on the OpenContrail.org website. I'm not sure if all these dependencies are needed. This is what worked for me. (I'm pretty sure git isn't needed here, but I use git for other projects so I did it anyways.)

apt-get install  -y git-core ant build-essential pkg-config linux-headers-3.2.0-35-virtual
apt-get install -y scons git python-lxml wget gcc patch make unzip flex bison g++ libssl-dev autoconf automake libtool pkg-config vim python-dev python-setuptools python-paramiko

apt-get update


Then I went onto the Juniper website and downloaded the software.

You DON'T have to install Openstack first to get this going.

This package has Contrail plus the Openstack Havana version built in.

I ftp or scp this file onto my ubuntu server.

I place it in the /tmp directory.

Next I install the package.

dpkg -i contrail-install-packages_1.05.1-234~havana_all.deb

The packages get placed in a contrail directory.

cd /opt/contrail/contrail_packages

Then I run the setup shell script.

./setup.sh

After this, you'll need to create or modify a testbed.py script. This tells contrail how install the compute, storage and control nodes. Since this is an all-in-one system, I going to clone the single box example and modify it.

cd /opt/contrail/utils/fabfile/testbeds/

cp testbed_singlebox_example.py testbed.py

Next I edit the file.

vi testbed.py

In order to execute this file, you need to back up a few directories (I know it's lame, the command doesn't seem to execute in the correct directory.)

cd /opt/contrail/utils

Next you will issue the fabric command

fab install_contrail

fab setup_all

After a few minutes, the scripts will run and more packages will be installed and configured to the specifications of your testbed file.

The server will reboot automatically.

After it comes back up sudo su when you login again

Then you need to source your authentication files.

keystonerc  and openstackrc  are located in /etc/contrail

source /etc/contrail/keystonerc
source /etc/contrail/openstackrc

Here are some commands  you can issue after installation to check the state of openstack

openstack-status

nova-manage service list

root@ubuntu:/home/user# openstack-status
== Nova services ==
openstack-nova-api:           active
openstack-nova-compute:       active
openstack-nova-network:       inactive (disabled on boot)
openstack-nova-scheduler:     active
openstack-nova-volume:        inactive (disabled on boot)
openstack-nova-conductor:     active
== Glance services ==
openstack-glance-api:         active
openstack-glance-registry:    active
== Keystone service ==
openstack-keystone:           active
== Cinder services ==
openstack-cinder-api:         active
openstack-cinder-scheduler:   active
openstack-cinder-volume:      inactive (disabled on boot)
== Support services ==
mysql:                        active
libvirt-bin:                  active
rabbitmq-server:              inactive (disabled on boot)
memcached:                    inactive (disabled on boot)
== Keystone users ==

+----------------------------------+---------+---------+---------------------+
|                id                |   name  | enabled |        email        |
+----------------------------------+---------+---------+---------------------+
| 2fa7b037efe1437a9045eab35f446511 |  admin  |   True  |  admin@example.com  |
| 6bb262f464a546b391b30879f1e8f10b |  cinder |   True  |  cinder@example.com |
| fdedd2c1a26d44ab9e83f000383cedf3 |   demo  |   True  |   demo@example.com  |
| cdc338c569af42cf87ba7bc7e7e161a8 |  glance |   True  |  glance@example.com |
| 2e28f88537064d97bfed64ad62f2fc66 | neutron |   True  | neutron@example.com |
| c09776b2f84744198d46c0361bfc1070 |   nova  |   True  |   nova@example.com  |
+----------------------------------+---------+---------+---------------------+
== Glance images ==
ID                                   Name                           Disk Format          Container Format     Size         
------------------------------------ ------------------------------ -------------------- -------------------- --------------
== Nova instance flavors ==
m1.medium: Memory: 4096MB, VCPUS: 2, Root: 40GB, Ephemeral: 0Gb, FlavorID: 3, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.tiny: Memory: 512MB, VCPUS: 1, Root: 1GB, Ephemeral: 0Gb, FlavorID: 1, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.large: Memory: 8192MB, VCPUS: 4, Root: 80GB, Ephemeral: 0Gb, FlavorID: 4, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.xlarge: Memory: 16384MB, VCPUS: 8, Root: 160GB, Ephemeral: 0Gb, FlavorID: 5, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.small: Memory: 2048MB, VCPUS: 1, Root: 20GB, Ephemeral: 0Gb, FlavorID: 2, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
== Nova instances ==

root@ubuntu:/home/user# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth ubuntu                               internal         enabled    :-)   2014-09-18 17:26:26
nova-console     ubuntu                               internal         enabled    :-)   2014-09-18 17:26:26
nova-scheduler   ubuntu                               internal         enabled    :-)   2014-09-18 17:26:26
nova-conductor   ubuntu                               internal         enabled    :-)   2014-09-18 17:26:27
nova-compute     ubuntu                               nova             enabled    :-)   2014-09-18 17:26:26


You are now ready to login to the Horizon Dashboard to configure your Openstack platform.

http://x.x.x.x/horizon

To log into your contrail controller change the port number to 8143

https://x.x.x.x:8143/

Next time I'll show how create a MP-BGP connection to a MX gateway to all traffic from the virtual network to connect to a physical network.

Network Automation is as easy as Py

as in PyEZ. PyEZ is a micro-framework to remotely manage and automate Juniper devices. It works with Python and allows you to pull Junos specific features into an abstraction layer.  This is great because you don't have to do any screen scraping to pull out any fields. I installed this module on my Mac to test this out.

The documentation is located here is great because you can look at the apis on how to create your script. The first script I wanted to test out is how to pull information from a router.

PyEZ can use YAML which is a human readable format markup language. I created a yaml file to extract the fields I was looking for. 

Here's my yaml file.

vrf.yml
VRF:
  get: routing-instances/instance
  args_key: name
  view: VRFView

VRFView:
 fields:
  instance_name: name
  instance_type: instance-type
  rd_type: route-distinguisher/rd-type
  vrf_target: vrf-target/community
  interface: interface/name

My script will parse VRFs on a router. I created two routing instances for this demo.

jnpr@R1# show routing-instances
VRF1 {
    instance-type vrf;
    interface lo0.1;
    route-distinguisher 1.1.1.1:100;
    vrf-target target:100:100;
}
VRF2 {
    instance-type vrf;
    interface lo0.2;
    route-distinguisher 1.1.1.1:101;
    vrf-target target:100:101;
}

Now I can test this in python.

$ python
Python 2.7.5 (default, Mar  9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

First I import all the necessary libraries.

>>> from pprint import pprint
>>> from jnpr.junos import Device
>>> from jnpr.junos.op.routes import RouteTable
>>> from lxml import etree
>>> from jnpr.junos.factory import loadyaml
>>> globals().update( loadyaml('vrf.yml') )

Then I open a connection to a junos device

>>> dev = Device('hostname_or_ip', user='username', password='password')
>>> dev.open()
Device(x.x.x.x)


Next I create a table
>>> tbl = VRF(dev)



Then get the fields for the table
>>> tbl.get(values=True) #make sure to pass values=True
VRF:x.x.x.x: 2 items






Now I can iterate through the table an print the contents.

>>> for item in tbl:
...     print 'instance_name:', item.instance_name
...     print 'instance_type:', item.instance_type
...     print 'rd_type:', item.rd_type
...     print 'vrf_target:', item.vrf_target
...     print 'interface:', item.interface
...
instance_name: VRF1
instance_type: vrf
rd_type: 1.1.1.1:101
vrf_target: target:100:101
interface: lo0.1
instance_name: VRF2
instance_type: vrf
rd_type: 1.1.1.1:102
vrf_target: target:100:102
interface: lo0.2

Now I can then manipulate the tables and look at individual fields.

>>> find = tbl['VRF1']

>>> find.interface
'lo0.1'

Now imagine a router with a hundred VRFs. I can now parse through this router remotely and automate operations.

SDN - Using POX Openflow controller to program a Juniper EX switch Pt.1

Openflow support is now available on Juniper EX switches in version 13.3 of code. I decided to explore and test this out. I first needed to find an Openflow compatible Juniper platform. The EX9200 supports Openflow, but you have to add the openflow image to JUNOS

jnpr@EX9200-RE0>request system software add <jsdn-package-name>

After installation you should see it as a module:

 jnpr@EX9200-RE0# run show version
Hostname: EX9200-RE0
Model: ex9204
Junos: 13.3R1.6
JUNOS Base OS boot [13.3R1.6]
JUNOS Base OS Software Suite [13.3R1.6]
JUNOS 64-bit Kernel Software Suite [13.3R1.6]
[TRUNCATED]
JUNOS py-base-i386 [13.3R1.6]
JUNOS SDN Software Suite [13.3R1.6]

There are a few possible operational commands:

jnpr@EX9200-RE0> show openflow ?
Possible completions:
  capability           Show feature and configuration capability
  controller           Show controller information and connection status
  filters              Show filter information
  flows                Show flow information
  interfaces           Show interface information
  statistics           Show statistics commands
  summary              Show openflow information summary
  switch               Show switch instance description information

To configure openflow, you need to:

1) Create interfaces

2) Associate those interfaces to an OPENFLOW resource group (ie. a virtualized switch)

3) Point to the Openflow controller to receive commands.


First, the creation of interfaces. This is very basic, it's similar to creating normal switching interfaces:

jnpr@EX9200-RE0# show interfaces
xe-2/0/0 {
    unit 0 {
        family ethernet-switching {
            vlan {
                members v100;
            }
        }
    }
}
xe-2/0/1 {
    unit 0 {
        family ethernet-switching {
            interface-mode access;
            vlan {
                members v100;
            }
        }
    }
}

[edit]
jnpr@EX9200-RE0# show vlans
v100 {
    vlan-id 100;
}

Next, you need to place these interfaces into an Openflow resource group (ie a virtualized switch) and map those interfaces to a port-id that Openflow can understand.

jnpr@EX9200-RE0# show protocols openflow
switch OF-SWITCH-92k {
    default-action {
        packet-in;
    }
    interfaces {
        xe-2/0/0.0 port-id 1;
        xe-2/0/1.0 port-id 2;
    }
}


Then you'll need to point to an openflow controller.

set protocols openflow switch OF-SWITCH-92k controller address 10.161.11.77

By default, openflow communicates over TCP using port 6633

Once committed, the EX will continously try to communicate with the openflow server.

jnpr@EX9200-RE0# run show openflow controller                                            
Openflowd controller information:
Controller socket: 12
Controller IP address: 10.161.11.77
Controller protocol: tcp
Controller port: 6633
Controller connection state: down
Number of connection attempt: 10
Controller role: equal

If the EX was correctly communicating with the controller, the connection state should say "up".

jnpr@EX9200-RE0# run show openflow switch
Switch Name:        OF-SWITCH-92k                                             
Switch ID:          0                  Switch DPID:    00:00:3c:8a:b0:0d:c7:c0
Flow mod received:  6                  Vendor received:      0             
Packets sent:       841                Packets received:     845           
Echo req sent:      833                Echo req received:    0             
Echo reply sent:    0                  Echo reply received:  833           
Port Status sent:   0                  Port mod received:    0             
Barrier request:    0                  Barrier reply:        0             
Error msg sent:     0                  Error msg received:   0  

The DPID or DataPathIdentifier is the physical switch that will be programmed. We'll see where this comes into play later.

I created an Ubuntu VM and installed the POX openflow controller. It's Python based which is awesome since it's a language I can understand. There's Floodlight (Java) or Trema (Ruby), so choose the flavor that you're comfortable with.

To simply start with there is a very basic POX python module found under the forwarding directory. This is the module we'll start with just to demonstrate how this works.

jnpr@ubuntu:~/OPENFLOW-CONTROLLER$ ./pox.py log.level --DEBUG forwarding.hub
POX 0.2.0 (carp) / Copyright 2011-2013 James McCauley, et al.
INFO:forwarding.hub:Hub running.
DEBUG:core:POX 0.2.0 (carp) going up...
DEBUG:core:Running on CPython (2.7.5+/Feb 27 2014 19:37:08)
DEBUG:core:Platform is Linux-3.11.0-12-generic-x86_64-with-Ubuntu-13.10-saucy
INFO:core:POX 0.2.0 (carp) is up.
DEBUG:openflow.of_01:Listening on 0.0.0.0:6633
INFO:forwarding.hub:Hubifying 3c-8a-b0-0d-c7-c0  <<<< DPID of the EX Switch

What this does to the EX switch basically turns those two interfaces in a small hub.

On POX the gist of the python script is this:


  msg = of.ofp_flow_mod()
  msg.actions.append(of.ofp_action_output(port = of.OFPP_FLOOD))

In Openflow terminology:

OFPP_FLOOD - output all openflow ports except the input port and those with flooding disabled

And that's it.

On the EX the Openflow rule looks like this:

jnpr@EX9200-RE0# run show openflow flows detail
Flow name: flow-65536
Table ID: 1     Flow ID: 65536            
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0   
Match: Input port: wildcard
       Ethernet src addr: wildcard       
       Ethernet dst addr: wildcard       
       Input vlan id: wildcard          Input VLAN priority: wildcard
       Ether type: wildcard
       IP ToS: 0x0                      IP protocol: 0x0 
       IP src addr: 0.0.0.0/32          IP dst addr: 0.0.0.0/32    
       Source port: 0                   Destination port: 0   
Action: Output port 65531,


The Action: Output port 65531  basically means everything available openflow interface but where the source of the packet came in on.


I setup a few tester ports to send constant traffic at a very low rate. ~1 pps


jnpr@EX9200-RE0# run show openflow statistics interfaces 
Switch Name: OF-SWITCH-92k                                                     
Interface Name: xe-2/0/0.0       Port Number: 1   
Num of rx pkts: 12                         Num of tx pkts: 12                  
Num of rx bytes: 17952                     Num of tx bytes: 17952               
Num of rx error: 0                         Num of tx error:0                   
Number of packets dropped by RX: 0                   
Number of packets dropped by TX: 0                   
Number of rx frame error:        0                   
Number of rx overrun error:      0                   
Number of CRC error:             0                   
Number of collisions:            0                   

Switch Name: OF-SWITCH-92k                                                     
Interface Name: xe-2/0/1.0       Port Number: 2   
Num of rx pkts: 12                         Num of tx pkts: 12                  
Num of rx bytes: 17952                     Num of tx bytes: 17952               
Num of rx error: 0                         Num of tx error:0                   
Number of packets dropped by RX: 0                   
Number of packets dropped by TX: 0                   
Number of rx frame error:        0                   
Number of rx overrun error:      0                   
Number of CRC error:             0                   
Number of collisions:            0                   

In my next blog entry I'll tweak a POX module to create flow rule entries that coincide more with SDN programming than turning your expensive switch into an expensive hub.

SDN - A look at Openflow running on white box switches.

In my quest to understand SDN, I attended a presentation on Openflow running on whitebox switches by Pica8 Open Networking. Their goal is to commoditize switching hardware and allow the control plane decisions to be made by a controller using Openflow. Pica8 has a cheap whitebox switch that runs Open Vswitch. The switches can be setup through Zero Touch provisioning, meaning you take an unconfigured box, drop it into the network and it will communicate with a server and automagically configure itself. They run a very light weight open source operating system with hardly any functionality which keeps costs low. I've read on the internet that their switches sell at half the cost of other vendor equipment. 

         While this may sound enticing, you get what you pay for. The size of their TCAM is around 1K-2K entries for some of their switches and 10k entries for others. This is pretty low! Why is the TCAM important? Think of it as the table size for an ACL entry. This is the 5 tuple flow where you "program" the switch based on the source and destination IP address, source and destination port and protocol. This is the gist of how Openflow programs the switches in your datacenter.

The low number of entries means that Pica8 has to limit the size of a data center. They resolve this limitation by recommending grouping switches into different "clusters". (Note: clusters is my name for it, as they called it a unit of calculation) They've calculated that a typical cluster can scale to a maximum of 12 racks of servers with two TOR switches, two AGG switches and two CORE boxes. When you want to expand, you create another "cluster" of switches/servers and interconnect them through the CORE boxes. This seems to be a waste of ports at the AGG layer.

          I believe the reason for this is based on the TCAM problem. If the AGG layer runs out of TCAM space, you are forced to build another cluster. I can imagine that each cluster is managed by a separate SDN controller. Theoretically a single SDN controller could manage all the clusters, but they didn't really talk about it.

           

          Pica8 at the time of writing has 200 customers, but ZERO deployments. Which means that even though SDN and Openflow is a cool technology, no customer in their right mind is going to put this in their production environment until the technology is more mature.

Now some of the problems with this implementation is that you literally have to program your network. You first have to create drop flow profiles. If you don't want IPv6 traversing your switch, create a drop flow. No multicast, create a drop flow. Then you create your forwarding entry. Need to go from a VM on port 1 to a VM on port 2. Create a forwarding entry. Now imagine that you have 48 ports per switch and two TOR switches, that right there means 96 entries in a single direction. Add the reverse direction and you have 192 entries to program. As you can see this could get very tedious. 

    

         Hopefully someone has a controller that can do this automatically. Once you run out of TCAM space you'll have to move VMs to a new cluster. Which brings me to the second issue, which is support for VMotion. Moving VMs between servers requires you to reprogram flows. But VMotion will have to integrate with the SDN Controller for this to work. This is also a flow based mechanism which means that this can be susceptible to DDOS attacks. Just send a bunch of arps and the switch will punt this to the controller. Get enough ARPs and you can overwhelm a controller and bring down a cluster.

Next they discussed network diagnostics. This was a very interesting topic. Where do you put this? On a normal switch you have counters and can retrieve them typically through SNMP. But on an Openflow switch, the hardware is supposed to be dumb. You need to put this on the controller. But how do you access a switch's counters without compromising performance. Do you retrieve this though Openflow? Is there another Northbound connection that will be both lightweight and scalable? Also Pica8 mentioned that some counters such as ingress and egress port statistics were not easily accessible. Another issue was when an upstream AGG switch did not have a proper flow entry it blackholed the packet and sent flow control packets down stream to the TOR, which filled up the buffers, thus preventing no packets to be forwarded from the TOR. They had to drop down to the switch's debug level to figure this out. In this scenario, where is the troubleshooting? You think it's a TOR issue, but in fact it's an AGG issue. This is a big concern I have about SDN, lack of diagnostics to troubleshoot issues. There is not enough visibility into the network to trace down the problem.

          While Openflow is an interesting technology this implementation is not yet mature and requires a lot of customization. Because of the limitation in a switches hardware this is not a scalable solution. You also need an intelligent controller that can automate your flow entries in a simple manner. 

Now to resolve this hardware issue I can imagine building a switch like building a bare metal server. Make the parts swappable. Running out of TCAM? Pull out the current one and install a new one just like you can swap out RAM and CPU Cores. White boxes need to be built so that their network connections stay in place and you swap out the FRUs around it. There also needs to be a way to get the optics to the point where they are tri-rate like copper links. Need to upgrade from 1G to 10G to 40G to 100G? Just update the flash. However this will also need some kind of black plane to upgrade the switch fabric.. Commoditized hardware needs to be built modularly. But this may be a Chassis based switch, not a TOR. However technology is constantly shrinking things down while packing more punch so I can imagine that eventually this will happen.

My research into SDN and Openvswitch (OVS)

So I've been researching SDN with openvswitch (OVS) and I'm not impressed. I'm all for network virtualization and SDN, but this implementation is rather poor. For one, why are they building antiquated technologies.

On their features page http://openvswitch.org they mention:

• STP (IEEE 802.1D-1998)

WTF? Why? It's even an older version of spanning tree to boot. If you're going to put spanning tree in OVS why not use the much faster RSTP instead. What's the intention of STP? Backwards compatibility? So are you going to deploy brand new technology (i.e. OVS and Openstack) on a network that has old switches?

If you're going to create an SDN network, why not use a new technology like Shortest Path Bridging or TRILL that allows all paths to be active. Better yet, program the network so all paths are active so when a failure occurs, reroute around the failure. Isn't that what SDN is all about?

Also OVS is flow based. Flow setup rate for GRE tunnels are low ~24K connects/sec utilizing >80% of the CPU. And this makes it vulnerable to DOS attacks. Compromise a VM and have the instance create BUM frames (Broadcast, unknown Unicast, and Multicast) will bring OVS to it's knees. And if you're going to use STP on OVS, beware of a hacker crafting packets with a superior BPDU. You wouldn't want to have ports go into a blocking state for no reason.

There has to be a different approach to doing this without copying old technology and porting it to something new. 

SDN - Unbundling platforms via Cumulus Networks

I went to a Meetup the other night and saw a presentation by JR Rivers (Co-founder) of Cumulus Networks.

He had an interesting point that most Enterprise networks have hardware that run a single OS. For instance a customer Data Center will run Cisco hardware running Cisco IOS. As it is, the Network Admin is stuck using that vendor and cannot move to another vendor without a forklift upgrade.

Who wants to reconnect links?

JR's proposition is to basically take what's currently going on in the server market and redefine the networking industry.

If you don't like your current vendor's OS, then you can re-image the gear and run the Network OS flavor of your choice.

I can see why Cumulus has taken this approach. In order to disrupt an incumbent, you need a way to remove them from the environment without disrupting the environment.

I can see this working on whitebox gear, but not really much on incumbent gear. A question from the audience was raised, who does the support? JR said that they would support software related issues. But what happens if Cumulus OS is running on top of a Cisco box? Good luck trying to call Cisco. They would automatically void the warranty in a situation like this. The one good thing about having a single vendor environment is that there is support. You know that the vendor will fix their bugs. In this new environment, you're not sure who's going to take responsibility to fix the bug. Also Cumulus is currently only targeting TOR boxes, so this unbundling of platforms will currently only happen in the Data Center.

The value that Cumulus adds is that their OS is Linux. With Linux, DevOps is very familiar with this OS and can program both the APP and the network, hence the SDN part. If Cisco doesn't have a feature in IOS that you need, as a customer you have to ask for a feature request, give your SE a dollar value tied to this feature and then wait x months for the new version to come out. Of course that software upgrade costs money.

As a Demo, JR showed us a BASH script that solved a problem. His point, he created the feature himself. It didn't cost him any money or require him to upgrade his OS.

Now for some vendors like Cisco you could do scripting off box using screen scrapes. But other vendors already have this capability. For instance, Arista has Python built into their OS, so JR's value add doesn't really hold a lot of weight. Juniper has a scripting language called SLAX which is based on XML/XLST.

A problem I see is that the OS is Linux and Security is a big problem. The reason why some vendors don't want to open up a scripting language on their box is because of this very reason. If a hacker is able to take control of your OS, they could create malicious code to do all sorts of things. I wouldn't want to be the Network Admin who has to figure out why their network is all of a sudden DDOSing their own servers and other networks. Which is why Juniper removed Perl from their early implementation of Junos and also why they support SLAX which basically prevents users from messing around with the kernel.

An audience member asked about JR's take on OpenFlow.  He was mostly against this. His response was that there would be lots of problems getting all the vendors to support this. This is an incorrect assumption. Networking vendors always update their OSes to support new protocols. As a vendor, you have to adapt to new things or you die.

I believe JR's reluctance is the threat OpenFlow can create. A SDN controller using OpenFlow can basically take the brains out of a switch. If you have 100 switches, you basically have 100 OSes,  a separate OS running on each switch. With Openflow, you can theoretically control all 100 switches with a single SDN controller and not rely on the underlying OS to do the decision making. Program the whole network, not just each individual switch.

While Cumulus does make an interesting point in unbundling platforms, SDN using a controller that is able to program your whole network is way more interesting.