How to use cURL to spin up an OpenStack VM

I know this is a little long but there are dependencies on how this works. Hopefully you can understand it. The last cURL command is the piece that puts it all together and spins up a VM. The last command will spin up a instance called "instance1", use a flavor and an cirros image and place it in the "VN2" network

LAPTOP:SCRIPTS$cat openstack-create.sh
TOKEN=""

#Do a POST to GET the AUTH TOKEN
TOKEN=$(curl -i \
  -H "Content-Type: application/json" \
  -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "admin",
          "domain": { "id": "default" },
          "password": "password",
          "tenantName": "admin"
        }
      }
    },
    "scope": {
      "project": {
        "name": "demo",
        "domain": { "id": "default" }
      }
    }
  }
}' \
  http://192.168.1.4:5000/v3/auth/tokens | grep X-Subject-Token: | sed -e "s/X-Subject-Token: //")

#STORE TOKEN
header='X-Auth-Token: '$TOKEN
#echo "$header"

#NOW USE the AUTH TOKEN TO CHECK TENANTS
curl -X GET http://192.168.1.4:5000/v2.0/tenants/ -H "$header" | python -m json.tool

#CHECK IMAGES FOR TENANT/PROJECT CALLED DEMO
# TENANT ID FOR DEMO = 6d0ebb466f01454e897e518289b3b785
echo "IMAGES"
curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/images | python -m json.tool


# CHECK FLAVORS
echo "FLAVORS"

curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/flavors | python -m json.tool

# CHECK SERVERS
echo "SERVERS"

curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/servers | python -m json.tool

# CHECK NETWORK ID - THIS IS NEUTRON AND HAS IT'S OWN PORT NUMBER 9696
echo "NETWORKS"

curl -H "Content-Type: application/json" -H "$header" -X GET http://192.168.1.4:9696/v2.0/networks | python -m json.tool

#CREATE VM
echo "CREATE VM"

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -H "X-Auth-Project-Id: demo" -H "$header" -X POST http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/servers -d '{"server": {"name": "instance1", "imageRef": "d5f30886-e8ce-4e89-9709-c49a40717b5b", "flavorRef": "2", "max_count": 1, "min_count": 1,"key_name": "xyz", "networks": [{"network": "bd912f99-c345-4f3e-8026-88e1dde42255", "uuid": "bd912f99-c345-4f3e-8026-88e1dde42255"}]}}'

-----------------------

I don’t execute the script and print it out as it would not look too pretty.

The most annoying part of this is all the curly brackets I had to dig around for the formatting of "networks" part as this was the hardest part.

I found out the formatting was like this:

networks: [{"port": String, "fixed_ip": String, "uuid": String, "network": String}, {"port": String, "fixed_ip": String, "uuid": String, "network": String}, ...]

Here's the api that I found useful:

http://developer.openstack.org/api-ref-compute-v2.html

Below is how it would look like if you executed the commands directly on the compute node. Note how image-list has the id for "imageRef"

root@openstack:/# nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name      | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 1  | m1.tiny   | 512       | 1    | 0         |      | 1     | 1.0         | True      |
| 2  | m1.small  | 2048      | 20   | 0         |      | 1     | 1.0         | True      |
| 3  | m1.medium | 4096      | 40   | 0         |      | 2     | 1.0         | True      |
| 4  | m1.large  | 8192      | 80   | 0         |      | 4     | 1.0         | True      |
| 5  | m1.xlarge | 16384     | 160  | 0         |      | 8     | 1.0         | True      |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
root@openstack:/# nova image-list
+--------------------------------------+--------------+--------+--------------------------------------+
| ID                                   | Name         | Status | Server                               |
+--------------------------------------+--------------+--------+--------------------------------------+
| d894be99-6c35-46be-b0bf-01149d724aec | c1           | ACTIVE |                                      |
| d5f30886-e8ce-4e89-9709-c49a40717b5b | cirros       | ACTIVE |                                      |
| 10a33e93-84da-43cb-862b-fbf0b0ea9d64 | ubuntu-cloud | ACTIVE |                                      |
+--------------------------------------+--------------+--------+--------------------------------------+
root@openstack:/# nova keypair-list
+------+-------------------------------------------------+
| Name | Fingerprint                                     |
+------+-------------------------------------------------+
| xyz  | 88:c3:9d:32:d0:33:c9:1d:c3:9a:ed:cc:51:0c:55:30 |
+------+-------------------------------------------------+
root@openstack:/# nova network-list
+--------------------------------------+-----------------------------+------+
| ID                                   | Label                       | Cidr |
+--------------------------------------+-----------------------------+------+
| 4c626c1c-26e2-45a3-8fa6-336fb297ffea | west-network                | None |
| 3d608137-9e84-452a-9b9b-dd055f21a095 | right-nw                    | None |
| cfcd7fd5-4dae-4b0d-bf77-41df699e7bfb | vn-blue                     | None |
| c72cad3d-8f41-48a9-8ebc-73f6f4d8be87 | VN1                         | None |
| 80d71671-0b00-422e-911f-cb599e5b462a | external-floating-ip        | None |
| a8d51e11-34a7-4e72-bd91-7e51d7fbdc6c | default-virtual-network     | None |
| c9a0a923-d583-4513-b2b6-c2c6d805ff18 | management-network-poc-demo | None |
| f7681c05-30e9-4951-b2fb-1796a0b7c41f | left-vn                     | None |
| 3a156ca5-54de-4654-b344-5e75a1a4369c | ip-fabric                   | None |
| bd912f99-c345-4f3e-8026-88e1dde42255 | VN2                         | None |
| a77a4bec-51d8-4a6f-8e93-01d8e2d21512 | management                  | None |
| d4b022f2-88ff-4d63-84cb-c79b76153799 | network1                    | None |
| 632269aa-74ac-4306-820d-3b27835f5951 | __link_local__              | None |
| f2bc533f-ba1d-412d-a788-875670883e9b | svc-vn-mgmt                 | None |
| 1072c78f-f464-4a6f-91a5-0caac6cf2f26 | east-network                | None |
| 77c40c19-07d1-4416-86d5-1a00b2545a88 | public-fip                  | None |
+--------------------------------------+-----------------------------+------+
root@openstack:/#

How Contrail communicates with the underlay

Contrail typically consists of a cluster of nodes. The three main nodes are the config, control and compute. The config node is where the Openstack Horizon and Contrail Controller exists. The Control node is used to form a MP-BGP session to a gateway router. The Compute node hosts all the VMs and virtual networks.

You can think of Contrail as a PE router as pretty much this is what a gateway router perceives the other end of the connection. Contrail uses a vRouter and when you configure virtual networks you have the ability to add a route-target to that virtual network. On the Gateway router you would create VRFs to associate with the corresponding virtual networks and prefixes can be exchanged. Data plane traffic will traverse through an MPLS tunnel between Contrail and the Gateway router. It's at the gateway router where you would "leak" the received Contrail virtual network into the main routing instance of the gateway router.

Here I use an Juniper MX as the gateway router. When I first setup contrail I used the testbed.py script to add the mx gateway router.

It's called ext_router = [ip address]

Then in contrail webui I should see the BGP session. You can however add this post contrail installation.

On the MX, I configure an iBGP session to connect with the Contrail control node.

user@router# show protocols
mpls {
    interface all;
}
bgp {
    group IBGP-CONTRAIL {
        type internal;
        local-address 192.168.10.11;
        family inet-vpn {
            unicast;
        }
        neighbor 192.168.10.2;
    }
}

Then in Contrail config node I create a virtual network and add a route target.

I create a corresponding VRF on the MX with the route target.

user@router# show routing-instances
VRF1 {
    instance-type vrf;
    interface lt-3/0/0.3;
    route-distinguisher 1.1.1.1:101;
    vrf-target target:64512:101;
    routing-options {
        static {
            route 0.0.0.0/0 next-hop 192.168.12.1;
        }
    }
} 

I check to see the BGP session established.

user@router# run show bgp summary                                     
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l3vpn.0         
                       8          8          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.10.2          64512       5724       6292       0       3 1d 23:08:56 Establ
  bgp.l3vpn.0: 8/8/8/0
  VRF1.inet.0: 3/3/3/0

The Virtual Network IP addresses for the VMs will be sent.

user@router# run show route receive-protocol bgp 192.168.10.2 

VRF1.inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
  Prefix          Nexthop           MED     Lclpref    AS path
* 11.1.1.1/32             192.168.10.3                 100        ?
* 11.1.1.5/32             192.168.10.3                 100        ?
* 11.1.1.7/32             192.168.10.3                 200        ?

bgp.l3vpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
  Prefix          Nexthop           MED     Lclpref    AS path
  192.168.10.3:7:11.1.1.1/32                   
*                         192.168.10.3                 100        ?
  192.168.10.3:7:11.1.1.5/32                   
*                         192.168.10.3                 100        ?
  192.168.10.3:7:11.1.1.7/32                   
*                         192.168.10.3                 200        ?

mpls.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0                  *[MPLS/0] 1w1d 04:10:08, metric 1
                      Receive
1                  *[MPLS/0] 1w1d 04:10:08, metric 1
                      Receive
2                  *[MPLS/0] 1w1d 04:10:08, metric 1
                      Receive
13                 *[MPLS/0] 1w1d 04:10:08, metric 1
                      Receive
299904             *[VPN/170] 1d 23:14:00
                    > to 192.168.11.1 via lt-3/0/0.1, Pop     
299936             *[VPN/170] 1d 12:47:38
                      receive table VRF1.inet.0, Pop     
299952             *[VPN/170] 1d 12:47:38
                    > to 192.168.12.1 via lt-3/0/0.3, Pop     

bgp.l3vpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.3:7:11.1.1.1/32               
                   *[BGP/170] 1d 12:46:01, localpref 100, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 24
192.168.10.3:7:11.1.1.5/32               
                   *[BGP/170] 1d 12:46:01, localpref 100, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 18
192.168.10.3:7:11.1.1.7/32               
                   *[BGP/170] 1d 12:26:14, localpref 200, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 28


Note the dynamic MPLS GRE tunnel is created. You will need to create one on the MX.

user@router# show chassis
fpc 3 {
    pic 0 {
        tunnel-services;
    }
}

user@router# show routing-options
static {
    route 0.0.0.0/0 next-hop 10.161.1.1;
}
autonomous-system 64512;
dynamic-tunnels {
    dynamic_overlay_tunnels {
        source-address 192.168.10.11;
        gre;
        destination-networks {
            192.168.10.0/24;
        }
    }
}



PoC-Demo.inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 1d 13:05:36
                    > to 192.168.12.1 via lt-3/0/0.3
11.1.1.1/32        *[BGP/170] 1d 13:03:59, localpref 100, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 24
11.1.1.5/32        *[BGP/170] 1d 13:03:59, localpref 100, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 18
11.1.1.7/32        *[BGP/170] 1d 12:44:12, localpref 200, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 28
192.168.12.0/24    *[Direct/0] 1d 13:05:36
                    > via lt-3/0/0.3
192.168.12.2/32    *[Local/0] 1d 13:05:36
                      Local via lt-3/0/0.3

LT interfaces are created to allow the virtual network traffic to communicate between the VRF and the main routing instance. You could also use RIB groups and Policies to do the same thing.

    lt-3/0/0 {
        unit 2 {
            encapsulation ethernet;
            peer-unit 3;
            family inet {
                address 192.168.12.1/24;
            }
        }
        unit 3 {
            encapsulation ethernet;
            peer-unit 2;
            family inet {
                address 192.168.12.2/24;
            }
        }
    }

You then need to make sure the interface that is connecting to the Contrail network is using MPLS.


interfaces {

    ge-3/1/1 {
        unit 0 {
            family inet {
                address 192.168.10.11/24;
            }
            family mpls;
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 1.1.1.1/32;
            }
            family iso {
                address 49.0002.0010.0100.1001.00;
            }
        }
    }
}

One thing you should be aware of is the next-hop of the route advertised by contrail points to the IP address of the Compute Node and not the control node.
user@router# run show route 11.1.1.1/32 detail

VRF1.inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
11.1.1.1/32 (1 entry, 1 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 192.168.10.3:7  <<< Contrail's RD
                Next hop type: Indirect
                Address: 0x94f4a28
                Next-hop reference count: 3
                Source: 192.168.10.2
                Next hop type: Router, Next hop index: 660
                Next hop: via gr-3/0/0.32770, selected
                Label operation: Push 24
                Label TTL action: prop-ttl
                Session Id: 0xd
                Protocol next hop: 192.168.10.3  <<<< IP of compute node
                Push 24
                Indirect next hop: 0x9574410 1048574 INH Session ID: 0xe
                State: <Secondary Active Int Ext ProtectionCand>
                Local AS: 64512 Peer AS: 64512
                Age: 1d 13:54:24     Metric2: 0
                Validation State: unverified
                Task: BGP_64512.192.168.10.2+34735
                Announcement bits (1): 1-KRT
                AS path: ?
                Communities: target:64512:101   << RT from contrail
                Import Accepted
                VPN Label: 24
                Localpref: 100
                Router ID: 192.168.10.2         <<<< IP of control node
                Primary Routing Table bgp.l3vpn.0

My attempt at setting up ssh key pairs for an Openstack VM

In my quest to learn more about Openstack I've decided to test out the ssh key-pair authentication method. I'm not much of a Unix guy so this attempt may be the wrong approach. But hey, learning is all about experimenting, so my failures may one day lead to success.

 I've read the RDO quick install on setting up the key-pair, but I could not get it to work using the Horizon webui. The documentation is a little sparse with no examples. It says I should be able to access the VM from my host. But after a few attempts I couldn't get it to work. So I've decided to try it a different way.

First I had to figure out how to ssh from my host to the VM.

pinging the VM didn't work.

[root@centos-6-5-openstack .ssh]$ ping 192.168.251.12
PING 192.168.251.12 (192.168.251.12) 56(84) bytes of data.
^C
--- 192.168.251.12 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4659ms

Then I remembered that I had to use network namespaces.

[root@centos-6-5-openstack .ssh(keystone_admin)]# ip netns list
qrouter-ed4afc1b-06ab-417e-a7e2-d5be13b822af
qdhcp-4dc834f5-e759-4d79-acf0-780768f1fa86
qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48
qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7
qdhcp-dc49c1a5-07d0-4225-bea5-02316aec3a42

[root@centos-6-5-openstack .ssh(keystone_admin)]# ip netns exec qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7 ip a
31: tapcb867d96-a4: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:ce:a5:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.251.11/24 brd 192.168.251.255 scope global tapcb867d96-a4
    inet6 fe80::f816:3eff:fece:a57e/64 scope link
       valid_lft forever preferred_lft forever
35: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

[root@centos-6-5-openstack .ssh(keystone_admin)]# ip netns exec qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7 ping 192.168.251.12
PING 192.168.251.12 (192.168.251.12) 56(84) bytes of data.
64 bytes from 192.168.251.12: icmp_seq=1 ttl=64 time=3.33 ms
64 bytes from 192.168.251.12: icmp_seq=2 ttl=64 time=0.436 ms
^C
--- 192.168.251.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1065ms
rtt min/avg/max/mdev = 0.436/1.885/3.334/1.449 ms

Awesome. That worked. So next I tried sshing to the VM.





[root@centos-6-5-openstack .ssh(keystone_admin)]# ip netns exec qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7 ssh -l cirros 192.168.251.12
The authenticity of host '192.168.251.12 (192.168.251.12)' can't be established.
RSA key fingerprint is 80:bc:58:4c:04:a6:a7:a4:0e:58:e1:0b:8d:55:e0:45.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.251.12' (RSA) to the list of known hosts.
cirros@192.168.251.12's password:
$

Good I'm in.

$ exit







Next I looked for a public key I already generated from my host machine

[root@centos-6-5-openstack .ssh(keystone_admin)]# ls
authorized_keys  id_rsa  id_rsa.pub  known_hosts










So now all I did was scp the file to the authorized_keys directory of the VM

[root@centos-6-5-openstack .ssh(keystone_admin)]# ip netns exec qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7 scp id_rsa.pub cirros@192.168.251.12:.ssh/authorized_keys
cirros@192.168.251.12's password:
id_rsa.pub                                                                              100%  407     0.4KB/s   00:00   

Now I can ssh with the key-pair without having to type in my password.

[root@centos-6-5-openstack .ssh(keystone_admin)]# ip netns exec qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7 ssh -l cirros 192.168.251.12

$ whoami
cirros

I'm still going to try to figure this out. Hopefully I'll be able to work this out the regular way.

Neutron Network Namespaces

Warning: The following is a rant of my observations about Openstack neutron.

They shouldn't call it Network Namespaces. They should call it overly complex naming convention.
I've been trying to wrap my head around the networking portion, aka Neutron, of OpenStack. What I've noticed is how ridiculously lousy the Horizon GUI is when trying to troubleshoot any issues. Almost anything useful is accessible through the CLI. What's the purpose of Cloud computing if you have to fallback to the cli to do anything?

When I build a VM, say VM1 on network net1 , I have no visibility about the virtual interfaces that it uses.

For example: Say I want to ping the virtual interface the VM created from the Openstack server. It should be pretty simple right? You should be able to do something like ping <network-name> ip address.

In order to do this you have to use this command:

ip netns - which I believe is short for network namespace

First, you need to know what the networkUUID.

You can list the network objects that were created by the namespace as follows:

# ip netns list
qrouter-ed4afc1b-06ab-417e-a7e2-d5be13b822af
qdhcp-4dc834f5-e759-4d79-acf0-780768f1fa86
qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48
qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7
qdhcp-dc49c1a5-07d0-4225-bea5-02316aec3a42

The structure of the networkUUID looks like this

qdhcp-<networkUUID>

WHAT the hell? That's not a useful name. That's an ID.

You can find this under this directory:

[root@centos-6-5-openstack ~(keystone_admin)]# ls /var/lib/neutron/dhcp/
0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48  a5958652-7348-436f-8aff-2c9ebd7dd9f7
4dc834f5-e759-4d79-acf0-780768f1fa86  dc49c1a5-07d0-4225-bea5-02316aec3a42

and then peer into it's contents

[root@centos-6-5-openstack ~(keystone_admin)]# cat /var/lib/neutron/dhcp/0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48/host
fa:16:3e:b8:98:40,host-20-20-0-11.openstacklocal,20.20.0.11
fa:16:3e:b5:6f:26,host-20-20-0-10.openstacklocal,20.20.0.10

You can also try to do this via this command:

[root@centos-6-5-openstack ~(keystone_admin)]# ip netns exec qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 ip a
30: tapafe229a7-2e: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:b8:98:40 brd ff:ff:ff:ff:ff:ff
    inet 20.20.0.11/24 brd 20.20.0.255 scope global tapafe229a7-2e
    inet6 fe80::f816:3eff:feb8:9840/64 scope link
       valid_lft forever preferred_lft forever
33: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

It won't give show you the VM's ip, but should show you the subnet it's on.

One other place to look:

[root@centos-6-5-openstack ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+-------------------------------------------------------+
| id                                   | name    | subnets                                               |
+--------------------------------------+---------+-------------------------------------------------------+
| 0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 | net1   | d4bf516f-135c-4c7f-ba7e-363cb6c7d307 20.20.0.0/24     |
| 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d | public  | 8942189b-8fbf-44e9-adbb-d62dd0d27015 192.168.250.0/24 |
| dc49c1a5-07d0-4225-bea5-02316aec3a42 | private | 6d67e2de-7f63-454d-9e88-fe33e6121b7b 10.0.0.0/24      |
+--------------------------------------+---------+-------------------------------------------------------+

So once I have the network id I can issue the following command:

[root@centos-6-5-openstack ~(keystone_admin)]# ip netns exec qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 ping 20.20.0.10
PING 20.20.0.10 (20.20.0.10) 56(84) bytes of data.
64 bytes from 20.20.0.10: icmp_seq=1 ttl=64 time=29.5 ms
64 bytes from 20.20.0.10: icmp_seq=2 ttl=64 time=15.0 ms
^C
--- 20.20.0.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1534ms
rtt min/avg/max/mdev = 15.079/22.301/29.523/7.222 ms

This is such a pain. What if the number of tenants, networks and VMs is huge? Why can't Openstack utilize the network name as part of the network UUid?

SDN - A look at Openflow running on white box switches.

In my quest to understand SDN, I attended a presentation on Openflow running on whitebox switches by Pica8 Open Networking. Their goal is to commoditize switching hardware and allow the control plane decisions to be made by a controller using Openflow. Pica8 has a cheap whitebox switch that runs Open Vswitch. The switches can be setup through Zero Touch provisioning, meaning you take an unconfigured box, drop it into the network and it will communicate with a server and automagically configure itself. They run a very light weight open source operating system with hardly any functionality which keeps costs low. I've read on the internet that their switches sell at half the cost of other vendor equipment. 

         While this may sound enticing, you get what you pay for. The size of their TCAM is around 1K-2K entries for some of their switches and 10k entries for others. This is pretty low! Why is the TCAM important? Think of it as the table size for an ACL entry. This is the 5 tuple flow where you "program" the switch based on the source and destination IP address, source and destination port and protocol. This is the gist of how Openflow programs the switches in your datacenter.

The low number of entries means that Pica8 has to limit the size of a data center. They resolve this limitation by recommending grouping switches into different "clusters". (Note: clusters is my name for it, as they called it a unit of calculation) They've calculated that a typical cluster can scale to a maximum of 12 racks of servers with two TOR switches, two AGG switches and two CORE boxes. When you want to expand, you create another "cluster" of switches/servers and interconnect them through the CORE boxes. This seems to be a waste of ports at the AGG layer.

          I believe the reason for this is based on the TCAM problem. If the AGG layer runs out of TCAM space, you are forced to build another cluster. I can imagine that each cluster is managed by a separate SDN controller. Theoretically a single SDN controller could manage all the clusters, but they didn't really talk about it.

           

          Pica8 at the time of writing has 200 customers, but ZERO deployments. Which means that even though SDN and Openflow is a cool technology, no customer in their right mind is going to put this in their production environment until the technology is more mature.

Now some of the problems with this implementation is that you literally have to program your network. You first have to create drop flow profiles. If you don't want IPv6 traversing your switch, create a drop flow. No multicast, create a drop flow. Then you create your forwarding entry. Need to go from a VM on port 1 to a VM on port 2. Create a forwarding entry. Now imagine that you have 48 ports per switch and two TOR switches, that right there means 96 entries in a single direction. Add the reverse direction and you have 192 entries to program. As you can see this could get very tedious. 

    

         Hopefully someone has a controller that can do this automatically. Once you run out of TCAM space you'll have to move VMs to a new cluster. Which brings me to the second issue, which is support for VMotion. Moving VMs between servers requires you to reprogram flows. But VMotion will have to integrate with the SDN Controller for this to work. This is also a flow based mechanism which means that this can be susceptible to DDOS attacks. Just send a bunch of arps and the switch will punt this to the controller. Get enough ARPs and you can overwhelm a controller and bring down a cluster.

Next they discussed network diagnostics. This was a very interesting topic. Where do you put this? On a normal switch you have counters and can retrieve them typically through SNMP. But on an Openflow switch, the hardware is supposed to be dumb. You need to put this on the controller. But how do you access a switch's counters without compromising performance. Do you retrieve this though Openflow? Is there another Northbound connection that will be both lightweight and scalable? Also Pica8 mentioned that some counters such as ingress and egress port statistics were not easily accessible. Another issue was when an upstream AGG switch did not have a proper flow entry it blackholed the packet and sent flow control packets down stream to the TOR, which filled up the buffers, thus preventing no packets to be forwarded from the TOR. They had to drop down to the switch's debug level to figure this out. In this scenario, where is the troubleshooting? You think it's a TOR issue, but in fact it's an AGG issue. This is a big concern I have about SDN, lack of diagnostics to troubleshoot issues. There is not enough visibility into the network to trace down the problem.

          While Openflow is an interesting technology this implementation is not yet mature and requires a lot of customization. Because of the limitation in a switches hardware this is not a scalable solution. You also need an intelligent controller that can automate your flow entries in a simple manner. 

Now to resolve this hardware issue I can imagine building a switch like building a bare metal server. Make the parts swappable. Running out of TCAM? Pull out the current one and install a new one just like you can swap out RAM and CPU Cores. White boxes need to be built so that their network connections stay in place and you swap out the FRUs around it. There also needs to be a way to get the optics to the point where they are tri-rate like copper links. Need to upgrade from 1G to 10G to 40G to 100G? Just update the flash. However this will also need some kind of black plane to upgrade the switch fabric.. Commoditized hardware needs to be built modularly. But this may be a Chassis based switch, not a TOR. However technology is constantly shrinking things down while packing more punch so I can imagine that eventually this will happen.