Doing reverse lookups of information in order to find things in Openstack

One of the most annoying things I find in Openstack is the lack of use of user friendly names. It's very tedious to reverse lookup information for troubleshooting and configuration.

For example, say I want to find out which tap port on my VM is connected to an openvswitch bridge. I would have to look up the VM in nova and find the corresponding IP assigned to it.

[root@centos-6-5-openstack ~(keystone_admin)]# nova list

+--------------------------------------+------+--------+------------+-------------+---------------------+

| ID                                   | Name | Status | Task State | Power State | Networks            |

+--------------------------------------+------+--------+------------+-------------+---------------------+

| 8b3ecae9-7b59-4b6f-a0a0-bbd5ab5aa629 | aa   | ACTIVE | -          | Running     | net1=192.168.251.10 |

| a679f6f9-bd7d-4e31-95a6-9776c44f0a4a | bb   | ACTIVE | -          | Running     | net1=192.168.251.12 |

Then I would list the ports in neutron.

[root@centos-6-5-openstack ~(keystone_admin)]# neutron port-list

+--------------------------------------+-------------+-------------------+---------------------------------------------------------------------------------------+

| id                                   | name        | mac_address       | fixed_ips                                                                             |

+--------------------------------------+-------------+-------------------+---------------------------------------------------------------------------------------+

| 014d6bba-7ab9-4028-9cd9-dbe27bd51b83 |             | fa:16:3e:1b:19:7d | {"subnet_id": "8942189b-8fbf-44e9-adbb-d62dd0d27015", "ip_address": "192.168.250.10"} |

| 1d3246d7-adb1-4b18-a911-b36562c278e0 |             | fa:16:3e:bd:e9:69 | {"subnet_id": "0147052f-2a0b-4a5b-9930-1d4e258e0c1a", "ip_address": "192.168.251.10"} |

| 40b988aa-c016-4e85-b9cf-beeb4332ab04 |             | fa:16:3e:e1:7e:57 | {"subnet_id": "0147052f-2a0b-4a5b-9930-1d4e258e0c1a", "ip_address": "192.168.251.12"} |

| cb867d96-a4ed-449d-a35a-f3670a14e710 |             | fa:16:3e:ce:a5:7e | {"subnet_id": "0147052f-2a0b-4a5b-9930-1d4e258e0c1a", "ip_address": "192.168.251.11"} |

| d267267b-5630-43df-8a07-05793fd831d0 |             | fa:16:3e:61:84:91 | {"subnet_id": "8942189b-8fbf-44e9-adbb-d62dd0d27015", "ip_address": "192.168.250.11"} |

| f434c257-5bbf-42a6-9136-072d1104fa19 |               | fa:16:3e:24:18:08 | {"subnet_id": "0147052f-2a0b-4a5b-9930-1d4e258e0c1a", "ip_address": "192.168.251.1"}  |

+--------------------------------------+-------------+-------------------+---------------------------------------------------------------------------------------+

Next I would need to look at the mac address and do a lookup in ip address

After coming through the list I can then verify the tap address with ip address show tapxxxx

[root@centos-6-5-openstack ~(keystone_admin)]# ip a show tap40b988aa-c0

30: tap40b988aa-c0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500

    link/ether fe:16:3e:e1:7e:57 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::fc16:3eff:fee1:7e57/64 scope link 

       valid_lft forever preferred_lft forever

And now I can do something useful like see which bridge the tap interface belongs to and verify the mac in the mac table.

[root@centos-6-5-openstack ~(keystone_admin)]# brctl show

bridge name bridge id STP enabled interfaces

qbr1d3246d7-ad 8000.4a04d0b90d18 no qvb1d3246d7-ad

tap1d3246d7-ad

qbr40b988aa-c0 8000.de752effa37c no qvb40b988aa-c0

tap40b988aa-c0

qbr7316a274-16 8000.8671bda4d699 no qvb7316a274-16

tap7316a274-16

qbrd06fccba-ad 8000.4613c3848432 no qvbd06fccba-ad

tapd06fccba-ad

[root@centos-6-5-openstack ~(keystone_admin)]# brctl showmacs qbr40b988aa-c0

port no mac addr is local? ageing timer

  1 de:75:2e:ff:a3:7c yes   0.00

  2 fe:16:3e:e1:7e:57 yes   0.00

or look at the openvswitch table and find out which bridge it belongs to

[root@centos-6-5-openstack ~(keystone_admin)]# ovs-vsctl list-ifaces br-int

int-br-ex

qr-87202226-14

qr-ba3e1c3b-49

qr-f2b01b2b-ad

qr-f434c257-5b

qvo1d3246d7-ad

qvo40b988aa-c0

qvo7316a274-16

qvod06fccba-ad

tapafe229a7-2e

tapca265ec6-29

tapcb867d96-a4

tape4751034-2c

This looks like a to do scripting project to get this information. 

Here I could create a table that would give me:

Nova ID

VM name

VM IP

VM Mac address

Neutron ID

tap interface

bridge mac table

openvswitch bridge

Creating my first OpenStack cloud platform and connecting it to a physical network.

In my quest to learn more about network virtualization I decided to learn more about OpenStack as a cloud platform. So I started from scratch and tried to install Openstack.

I took an old server which had two NIC ports and installed Centos 6.5. I then installed openstack all in one server from the following:

http://openstack.redhat.com/Quickstart

I decided to use the havana version because icehouse just came out and I wanted to wait until after the bugs were fixed.

After doing a bunch of yum installs I had it working. I was actually able to create a few VMs using the Cirros Linux image and created a simple network. I was able to ping from one VM to the other.

However the documentation was not clear on how to connect your VMs to the outside world through a physical network.

After searching around for a few days, I found this:

http://openstack.redhat.com/Neutron_with_existing_external_network

I decided to use eth0 as my management ip to access Openstack and use eth1 as the port for the VMs to connect to the physical network.

I also learned that OpenStack uses OpenVswitch and Neutron for network connectivity. You have to learn a little bit about OpenVswitch on a server.

This led me to make the following changes.
under  /etc/sysconfig/network-scripts

I duplicated (to backup) and changed the following files.

[root@centos-6-5-openstack network-scripts]# more ifcfg-eth1
DEVICE=eth1
TYPE=OVSPort
UUID=100083c1-6174-4a59-b3d0-09081eef106a
ONBOOT=yes
DEVICETYPE=ovs
HWADDR=00:30:48:F9:B9:C9
OVS_BRIDGE=br-ex
NAME="System eth1"

[root@centos-6-5-openstack network-scripts]# more ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.250.11
NETMASK=255.255.255.0
DNS=198.6.1.1
ONBOOT=yes

Then I tied the eth1 interface to the bridge "br-ex"

"br-ex" was automatically created on my server, but if it's not there, you should create a bridge (ova-vsctl add-br <NAME>)  and associate the physical interface (eth1) to that bridge.

[root@centos-6-5-openstack network-scripts]# port br-ex eth1

[root@centos-6-5-openstack network-scripts]# ovs-vsctl show
f6d22683-9529-48b6-b9d0-5e49cb720d44
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-d267267b-56"
            Interface "qg-d267267b-56"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
        Port "eth1"
            Interface "eth1"
    Bridge br-int

I also had to make a change to a file:

Add to the /etc/neutron/plugin.ini file these lines:
network_vlan_ranges = physnet1
bridge_mappings = physnet1:br-ex

I had to make sure that eth1 is a port for bridge br-ex not br-int. For some reason it was not doing this, so I removed the port from br-int using ovs-vsctl del-port command

Red hat wants you to restart nework services after making the changes. However when I issued that command, in ifconfig I would see that eth1 still had an ip address and br-ex was not updated. So I rebooted the server.

I also used the following to troubleshoot
tcpdump -nei br-ex
tcpdump -nei eth1

After reboot all was well.

root@centos-6-5-openstack network-scripts]# ifconfig
br-ex     Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C9
          inet addr:192.168.250.11  Bcast:192.168.250.255  Mask:255.255.255.0
          inet6 addr: fe80::5822:16ff:fe25:91a/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4834 (4.7 KiB)  TX bytes:1336 (1.3 KiB)

br-int    Link encap:Ethernet  HWaddr BE:9D:E4:A4:B0:44
          inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:213 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:16262 (15.8 KiB)  TX bytes:468 (468.0 b)

eth0      Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C8
          inet addr:10.161.32.230  Bcast:10.161.39.255  Mask:255.255.248.0
          inet6 addr: fe80::230:48ff:fef9:b9c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14400 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1686756 (1.6 MiB)  TX bytes:7627901 (7.2 MiB)
          Interrupt:18 Memory:d8020000-d8040000

eth1      Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C9
          inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:905 errors:0 dropped:0 overruns:0 frame:0
          TX packets:978 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:82360 (80.4 KiB)  TX bytes:87771 (85.7 KiB)
          Interrupt:19 Memory:d8060000-d8080000

int-br-ex Link encap:Ethernet  HWaddr 5A:EC:7F:16:11:35
          inet6 addr: fe80::58ec:7fff:fe16:1135/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:190 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4752 (4.6 KiB)  TX bytes:14550 (14.2 KiB)


From my physical gateway switch/router 192.168.250.1 I tried pinging the br-ex interface 192.168.250.11

Next I created the tenant subnet, vm and the openstack router.

First I had to source my admin credentials

source /root/keystonerc_admin

Next I followed the instructions on creating a public network. I added a floating ip.

http://openstack.redhat.com/Floating_IP_range

By default packstack creates a demo public network, with a subnet that I was not using. So I cleared the gw of this first.

# neutron router-gateway-clear router1
Then deleted the subnet
# neutron subnet-delete public

Next I recreated a new subnet
neutron subnet-create --name public --enable_dhcp=False --allocation-pool=start=192.168.250.10,end=192.168.250.20 --gateway=192.168.250.1 public 192.168.250.0/24

Then created a router using Horizon.
Then I tied the router to the public subnet.

neutron router-gateway-set test-rtr public

[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron subnet-show public
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "192.168.250.10", "end": "192.168.250.20"} |
| cidr             | 192.168.250.0/24                                     |
| dns_nameservers  |                                                      |
| enable_dhcp      | False                                                |
| gateway_ip       | 192.168.250.1                                        |
| host_routes      |                                                      |
| id               | 8942189b-8fbf-44e9-adbb-d62dd0d27015                 |
| ip_version       | 4                                                    |
| name             | public                                               |
| network_id       | 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d                 |
| tenant_id        | 3a10de8a82444118865a6398b336ee68                     |
+------------------+------------------------------------------------------+

From Horizon, my setup looks simple:

One thing that threw me off during the whole process was trying to figure out if the openstack Gateway IP was working or not.

I noticed that the status kept saying down.

[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron port-show router-gw | grep status
| status                | DOWN  

After googling it, I found out that this was a bug and a display issue. So you can't tell if the gateway ip is actually working or not because the status is broken.


From my VM I am now able to access my physical switch

SDN - A look at Openflow running on white box switches.

In my quest to understand SDN, I attended a presentation on Openflow running on whitebox switches by Pica8 Open Networking. Their goal is to commoditize switching hardware and allow the control plane decisions to be made by a controller using Openflow. Pica8 has a cheap whitebox switch that runs Open Vswitch. The switches can be setup through Zero Touch provisioning, meaning you take an unconfigured box, drop it into the network and it will communicate with a server and automagically configure itself. They run a very light weight open source operating system with hardly any functionality which keeps costs low. I've read on the internet that their switches sell at half the cost of other vendor equipment. 

         While this may sound enticing, you get what you pay for. The size of their TCAM is around 1K-2K entries for some of their switches and 10k entries for others. This is pretty low! Why is the TCAM important? Think of it as the table size for an ACL entry. This is the 5 tuple flow where you "program" the switch based on the source and destination IP address, source and destination port and protocol. This is the gist of how Openflow programs the switches in your datacenter.

The low number of entries means that Pica8 has to limit the size of a data center. They resolve this limitation by recommending grouping switches into different "clusters". (Note: clusters is my name for it, as they called it a unit of calculation) They've calculated that a typical cluster can scale to a maximum of 12 racks of servers with two TOR switches, two AGG switches and two CORE boxes. When you want to expand, you create another "cluster" of switches/servers and interconnect them through the CORE boxes. This seems to be a waste of ports at the AGG layer.

          I believe the reason for this is based on the TCAM problem. If the AGG layer runs out of TCAM space, you are forced to build another cluster. I can imagine that each cluster is managed by a separate SDN controller. Theoretically a single SDN controller could manage all the clusters, but they didn't really talk about it.

           

          Pica8 at the time of writing has 200 customers, but ZERO deployments. Which means that even though SDN and Openflow is a cool technology, no customer in their right mind is going to put this in their production environment until the technology is more mature.

Now some of the problems with this implementation is that you literally have to program your network. You first have to create drop flow profiles. If you don't want IPv6 traversing your switch, create a drop flow. No multicast, create a drop flow. Then you create your forwarding entry. Need to go from a VM on port 1 to a VM on port 2. Create a forwarding entry. Now imagine that you have 48 ports per switch and two TOR switches, that right there means 96 entries in a single direction. Add the reverse direction and you have 192 entries to program. As you can see this could get very tedious. 

    

         Hopefully someone has a controller that can do this automatically. Once you run out of TCAM space you'll have to move VMs to a new cluster. Which brings me to the second issue, which is support for VMotion. Moving VMs between servers requires you to reprogram flows. But VMotion will have to integrate with the SDN Controller for this to work. This is also a flow based mechanism which means that this can be susceptible to DDOS attacks. Just send a bunch of arps and the switch will punt this to the controller. Get enough ARPs and you can overwhelm a controller and bring down a cluster.

Next they discussed network diagnostics. This was a very interesting topic. Where do you put this? On a normal switch you have counters and can retrieve them typically through SNMP. But on an Openflow switch, the hardware is supposed to be dumb. You need to put this on the controller. But how do you access a switch's counters without compromising performance. Do you retrieve this though Openflow? Is there another Northbound connection that will be both lightweight and scalable? Also Pica8 mentioned that some counters such as ingress and egress port statistics were not easily accessible. Another issue was when an upstream AGG switch did not have a proper flow entry it blackholed the packet and sent flow control packets down stream to the TOR, which filled up the buffers, thus preventing no packets to be forwarded from the TOR. They had to drop down to the switch's debug level to figure this out. In this scenario, where is the troubleshooting? You think it's a TOR issue, but in fact it's an AGG issue. This is a big concern I have about SDN, lack of diagnostics to troubleshoot issues. There is not enough visibility into the network to trace down the problem.

          While Openflow is an interesting technology this implementation is not yet mature and requires a lot of customization. Because of the limitation in a switches hardware this is not a scalable solution. You also need an intelligent controller that can automate your flow entries in a simple manner. 

Now to resolve this hardware issue I can imagine building a switch like building a bare metal server. Make the parts swappable. Running out of TCAM? Pull out the current one and install a new one just like you can swap out RAM and CPU Cores. White boxes need to be built so that their network connections stay in place and you swap out the FRUs around it. There also needs to be a way to get the optics to the point where they are tri-rate like copper links. Need to upgrade from 1G to 10G to 40G to 100G? Just update the flash. However this will also need some kind of black plane to upgrade the switch fabric.. Commoditized hardware needs to be built modularly. But this may be a Chassis based switch, not a TOR. However technology is constantly shrinking things down while packing more punch so I can imagine that eventually this will happen.

My research into SDN and Openvswitch (OVS)

So I've been researching SDN with openvswitch (OVS) and I'm not impressed. I'm all for network virtualization and SDN, but this implementation is rather poor. For one, why are they building antiquated technologies.

On their features page http://openvswitch.org they mention:

• STP (IEEE 802.1D-1998)

WTF? Why? It's even an older version of spanning tree to boot. If you're going to put spanning tree in OVS why not use the much faster RSTP instead. What's the intention of STP? Backwards compatibility? So are you going to deploy brand new technology (i.e. OVS and Openstack) on a network that has old switches?

If you're going to create an SDN network, why not use a new technology like Shortest Path Bridging or TRILL that allows all paths to be active. Better yet, program the network so all paths are active so when a failure occurs, reroute around the failure. Isn't that what SDN is all about?

Also OVS is flow based. Flow setup rate for GRE tunnels are low ~24K connects/sec utilizing >80% of the CPU. And this makes it vulnerable to DOS attacks. Compromise a VM and have the instance create BUM frames (Broadcast, unknown Unicast, and Multicast) will bring OVS to it's knees. And if you're going to use STP on OVS, beware of a hacker crafting packets with a superior BPDU. You wouldn't want to have ports go into a blocking state for no reason.

There has to be a different approach to doing this without copying old technology and porting it to something new.