The Network Arborist: 2014

Thursday, December 18, 2014

Gathering and graphing snmp stats on a Palo Alto Networks Firewall

So I was task to the challenge of gathering cpu utilization and active sessions on a Palo Alto Networks Firewall.

There are two CPUs on a Firewall. There is a management plane cpu and a data plane cpu.

The OIDs are below.

Active sessions: .1.3.6.1.4.1.25461.2.1.2.3.3

MGMT Utilization: .1.3.6.1.2.1.25.3.3.1.2.1

Data Plane Utilization: .1.3.6.1.2.1.25.3.3.1.2.2

The first step is to enable snmp on the Firewall.

Under

Device/Setup/Operations/Miscellaneous/SNMP Setup

enter your community string and pick the version.

Now I'm using unsecured snmp v2 because I didn't know how to use snmpv3 on the Splunk Application that I'm going to use to generate charts.

on an ubuntu vm I tested it.

admin@ubuntu-poc-vm:~$ snmpget 10.48.64.112 -v 2c -c public .1.3.6.1.2.1.25.3.2.1.3.2
iso.3.6.1.2.1.25.3.2.1.3.2 = STRING: "Slot-1 Data Processor"
admin@ubuntu-poc-vm:~$ snmpget 10.48.64.112 -v 2c -c public .1.3.6.1.2.1.25.3.2.1.3.1
iso.3.6.1.2.1.25.3.2.1.3.1 = STRING: "Management Processor"
admin@ubuntu-poc-vm:~$ snmpget 10.48.64.112 -v 2c -c public .1.3.6.1.2.1.25.3.3.1.2.1
iso.3.6.1.2.1.25.3.3.1.2.1 = INTEGER: 23
admin@ubuntu-poc-vm:~$ snmpwalk 10.48.64.112 -v 2c -c public .1.3.6.1.2.1.25.3.3.1.2.2
iso.3.6.1.2.1.25.3.3.1.2.2 = INTEGER: 5

Next I setup Splunk on a Windows 7 VM. It was pretty easy as you go to their website and download the app for your particular flavor of OS.

Once installed you hop onto the webui and change your credentials.

Next you need to install the SNMP Modular Input app. It's free.

Then you need to go Settings > Data Inputs > SNMP > Add New

Here I created a new input for each SNMP OID that I wanted to query.

There's a reason for this. I could have added a list of OIDs using comma delimited but I had a hard time trying parsing the data I wanted to graph. If someone has a better method let me know.

Last I used set source type to Manual and the actual source type to "snmp_ta" which is the SNMP Modular app.

Next I went to Manage App and looked for snmp_ta app and edited the permissions. I made the app visible.

A new icon appears in the dashboard and now I can double click it to examine the data being polled.

I first click on data summary and select the source tab and choose your source.

I should now see all the data that SNMP Modular Input queried from the firewall.

Now it's time to manipulate the data and create some nice graphs. First I have to manipulate the search fields.

I add a pipe and enter fields value.

This will specifically give me the value what is return from the SNMP OID.

Next I choose the Visualization tab and click on Pivot. I will only have 1 field to use which is the value field.

Next I choose the graph I want to create. I chose line chart.

Then on the Y Axis I make sure the field says #value and I can label this as Data Plane CPU.

Then on the X Axis side I choose _time which will graph the data collected over time.

Make sure the Null Values say connected to give me a nice line graph instead of a bunch of dots.

Last I save this panel and give it a name so I can put it on my Dashboard.

Now you may say, Big deal, all this work to do that. I can just spin up Solar Winds and it's really easy. No need to create search queries and add pipes and then do all this to create one graph. Well the reason for using Splunk is that Palo Alto Networks has a nice plug-in (it's free) that works directly with Splunk. So with one tab I can check on all the traffic, threats and wildfire data collected and on the other tab I can look at the CPU Utilization and Session counts. This gives me one single pane of glass instead of having to jump onto different management tools to give me the same information.

Monday, November 24, 2014

Vmotion over a VXLAN overlay.

In an old Post I showed how to create a simple VXLAN tunnel to stretch Layer 2 over an Layer 3 network. Now I'll show how VMWare VMotion works over this same type of setup. Note: I researched this and it warned that VXLAN was not a solution for doing VMotion over VXLAN. The latency could be high over a DCI so VMotion may not always work. This is just a proof of concept.

From what I've read, vmotion over vxlan on the hypervisor is not supported in ESXi 5.1 or earlier. I'm not sure about ESXi 5.5. In any case it doesn't matter what version of ESXi you have, because in this scenario the VXLAN tunnel starts and stops at the SPINE layer. The Hypervisor is NOT using VXLAN at all and is communicating to the underlay through a normal vlan tag. Even the leaf switch is communicating to the spine over a vlan tag.

In this setup I have a NetApp SAN that will serve as the NFS datastore for the VMs. The SAN network is using VLAN 10, while the VMs Data path is using VLAN 100.

On VMWare I created a NFS Datastore. The IP address of SAN NetApp is 100.10.1.254.

Each Host Serveris is seperated by a L3 network as in the above topology.

Host 1 has an VMKernal IP of 100.10.1.4 and Host 2 has an IP of 100.10.1.5 for communicating with NetApp.

Vmotion is enabled on both dvSwitches.

I created a distributed switch for the NFS network and added a 1G NIC as an uplink. This was done on both Hosts.

A second dvSwitch was created for the VMs Data path. This is using a 10G NIC for the uplink.

On NetApp

Vol1 was created which allows Read/Write access to the VMware hosts.

NetApps Network Interface is configured with the IP 100.10.1.254 and uses a vlan tag of 10.

On Juniper Leaf switch.

vlan 10 and vlan 100 are created

jnpr@QFX5100-48S-5# show vlans

v10 {
vlan-id 10;
}
v100 {
vlan-id 100;
}

Interfaces are created and vlans added.

set interfaces xe-0/0/2 description TO-VMW-145-vmnic9

set interfaces xe-0/0/2 unit 0 family ethernet-switching interface-mode trunk
set interfaces xe-0/0/2 unit 0 family ethernet-switching vlan members v100
set interfaces ge-0/0/47 description TO-VMW-145-vmnic3-for-NETAPP
set interfaces ge-0/0/47 unit 0 family ethernet-switching interface-mode trunk
set interfaces ge-0/0/47 unit 0 family ethernet-switching vlan members v10
set interfaces et-0/0/50 description TO-SPINE1
set interfaces et-0/0/50 unit 0 family ethernet-switching interface-mode trunk
set interfaces et-0/0/50 unit 0 family ethernet-switching vlan members all

The 1GE link is used for the NFS and the 10G is used for Data between the Leaf and Host1. The uplink to Spine1 is over a 10G link and has vlan member all.

On the Juniper Spine1:

Again two vlans are created, but here is where the mapping of vlan to VXLAN tunnel is created.

jnpr@EX9200-1# show vlans

v10 {
    vlan-id 10;
    l3-interface irb.10;
    vxlan {
        vni 10;
        multicast-group 239.1.1.10;
        encapsulate-inner-vlan;
        decapsulate-accept-inner-vlan;
    }
}
v100 {
    vlan-id 100;
    l3-interface irb.0;
    vxlan {
        vni 100;
        multicast-group 239.1.1.100;
        encapsulate-inner-vlan;
        decapsulate-accept-inner-vlan;
    }
}

Core interface is L3.

set interfaces et-2/0/0 description TO-CORE1

set interfaces et-2/0/0 unit 0 family inet address 192.168.24.4/24
set interfaces et-2/0/0 unit 0 family iso

Link towards Leaf1 is a trunk with the 2 vlans.

set interfaces et-2/2/1 description TO-QFX5100-48S-5

set interfaces et-2/2/1 unit 0 family ethernet-switching interface-mode trunk
set interfaces et-2/2/1 unit 0 family ethernet-switching vlan members v100
set interfaces et-2/2/1 unit 0 family ethernet-switching vlan members v10

The NetApp sits only off of Spine 1 and traffic is switched there from Leaf1.

set interfaces ge-7/0/0 description TO-NETAPP

set interfaces ge-7/0/0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ge-7/0/0 unit 0 family ethernet-switching vlan members v10

IRBs are created for the two vlans. Vlan 100's IRB is the default gateway for the Data Path of the VMs. IRB for vlan 10 is used just to make sure we can ping the NetApp and the VMware VMKernal IP.

set interfaces irb unit 0 family inet address 100.1.1.1/24

set interfaces irb unit 10 family inet address 100.10.1.200/24

PIM and IGP routing protocols are created for connectivity.

set protocols isis interface all

set protocols isis interface fxp0.0 disable
set protocols isis interface lo0.0 passive
set protocols pim rp static address 192.168.0.1
set protocols pim interface lo0.0 mode bidirectional-sparse
set protocols pim interface et-2/0/0.0 mode bidirectional-sparse
set protocols lldp interface all

ON the Remote Spine2 switch the configuration is almost the same. The only difference is that there is no direct NetApp connection, so NFS needs to be tunneled through VXLAN so the VMKernal on Host 2 can access the storage.

jnpr@EX9200-2# show vlans

jnpr@EX9200-2# show protocols | display set

set protocols isis reference-bandwidth 40g
set protocols isis interface et-2/0/0.0
set protocols isis interface all
set protocols isis interface fxp0.0 disable
set protocols isis interface lo0.0 passive
set protocols pim rp static address 192.168.0.1
set protocols pim interface lo0.0 mode bidirectional-sparse
set protocols pim interface et-2/0/0.0 mode bidirectional-sparse
set protocols lldp interface all

LLDP is turned on the Juniper switches and dvSwitches so we can monitor the underlay.

On Leaf1 we can see both dvSwitches

jnpr@QFX5100-48S-5# run show lldp neighbors

Local Interface    Parent Interface    Chassis Id          Port info          System Name
xe-0/0/2           -                   00:05:33:48:70:b9   port1
xe-0/0/2           -                   00:50:56:b9:0b:b3   eth1               South-VM
et-0/0/50          -                   4c:96:14:6b:bb:c0   TO-QFX5100-48S-5   EX9200-1
ge-0/0/47          -                                       port 2 on dvSwitch dvSwitch-NFS-v10 (etherswitch) localhost.jnpr.net
xe-0/0/2           -                                       port 2 on dvSwitch dvSwitch-v100 (etherswitch) localhost.jnpr.net

And a Host running lldpd. We can also see the uplink to Spine1.

jnpr@QFX5100-48S-5# run show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)

Ethernet switching table : 5 entries, 5 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical
    name                address             flags              interface
    v10                 00:50:56:6f:2c:31   D             -   ge-0/0/47.0
    v10                 02:a0:98:2c:06:cc   D             -   et-0/0/50.0

The things to note here are the two macs. The NetApp uses 02:a0:98:2c:06:cc which is being learned from the uplink to the Spine. And 00:50:56:6f:2c:31 is the NIC of Host 1.

On Spine 1

jnpr@EX9200-1# run show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)

Ethernet switching table : 5 entries, 5 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical                NH        RTR
    name                address             flags              interface              Index     ID
    v10                 00:50:56:69:37:db   D             -   vtep.32769
    v10                 00:50:56:6f:2c:31   D             -   et-2/2/1.0
    v10                 02:a0:98:2c:06:cc   D             -   ge-7/0/0.0
    v10                 4c:96:14:f2:b6:e0   D             -   vtep.32769
    v10                 a8:d0:e5:f7:bf:f0   D             -   vtep.32769

We can see that macs are being learned over the VTEP or VXLAN tunnel.

Mac 00:50:56:69:37:db is the NIC of Host2.

Spine 2 we can see NETAPP is being learned through the VXLAN tunnel.

# run show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)

Ethernet switching table : 3 entries, 3 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical                NH        RTR
    name                address             flags              interface              Index     ID
    v10                 00:50:56:69:37:db   D             -   et-2/2/1.0
    v10                 02:a0:98:2c:06:cc   D             -   vtep.32769           <<< NETAPP
    v10                 4c:96:14:6b:bb:f0   D             -   vtep.32769

Leaf 2

jnpr@QFX5100-48S-6> show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)

Ethernet switching table : 4 entries, 4 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical
    name                address             flags              interface
    v10                 00:50:56:69:37:db   D             -   ge-0/0/47.0
    v10                 02:a0:98:2c:06:cc   D             -   et-0/0/50.0
    v10                 4c:96:14:6b:bb:f0   D             -   et-0/0/50.0
    v10                 a8:d0:e5:f7:bf:f0   D             -   et-0/0/50.0

Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical
    name                address             flags              interface
    v100                00:50:56:b9:55:58   D             -   xe-0/0/2.0
    v100                4c:96:14:6b:bb:f0   D             -   et-0/0/50.0

Now we are ready to do vMotion.

You can see from Ubuntu the mac address is on eth1 which is going to vlan 100 on Leaf2

jnpr@vmotion-ubuntu:~$ ifconfig

eth1      Link encap:Ethernet HWaddr 00:50:56:b9:55:58
          inet addr:100.1.1.30 Bcast:100.1.1.255 Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:feb9:5558/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:8633 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7023 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1200614 (1.2 MB) TX bytes:905670 (905.6 KB)

On Spine2 it's a local mac.
jnpr@EX9200-2# run show ethernet-switching table vlan-id 100
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical                NH        RTR
    name                address             flags              interface              Index     ID
    v100                00:50:56:b9:55:58   D             -   et-2/2/1.0

On VMWare I choose the Vmotion ubuntu VM.

And Select Migrate and chooose Change Host and select Host1 as the Destination. Select the Priority of your choice and click Finish.

Migration completed!

On Leaf1 I can see the mac move.
jnpr@QFX5100-48S-5# run show ethernet-switching table vlan-id 100

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)

Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical
    name                address             flags              interface
    v100                00:50:56:b9:55:58   D             -   xe-0/0/2.0
    v100                4c:96:14:6b:bb:f0   D             -   et-0/0/50.0

On Spine 1 it is local.

jnpr@EX9200-1# run show ethernet-switching table vlan-id 100

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)

Ethernet switching table : 1 entries, 1 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical                NH        RTR
    name                address             flags              interface              Index     ID
    v100                00:50:56:b9:55:58   D             -   et-2/2/1.0

And on Spine 2 it now sits across the VTEP.

jnpr@EX9200-2# run show ethernet-switching table vlan-id 100

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)

Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical                NH        RTR
    name                address             flags              interface              Index     ID
    v100                00:50:56:b9:55:58   D             -   vtep.32769
    v100                4c:96:14:6b:bb:f0   D             -   vtep.32769

One caveat I found was that vmotion sends large frames. IP mcast + VXLAN adds overhead. So when I setup my network with the default configuration I could not initiate vmotion. I had to change the MTU size on Spine to 9000 and then it worked.

So the conclusion is, you don't need VXLAN support on the Hypervisor if you want to do vMotion over a Layer 2 stretch technology. You can run VXLAN Tunnels between two Juniper Ex9200 Spine switches and map vlans to these tunnels. The Overlay is unaware of what is happening in the Underlay.

Friday, November 21, 2014

Automating a Palo Alto Networks Firewall using Python

In my last post I used a python script to extract information from a Palo Alto Networks Firewall using pan-python. In this post I'll illustrate how to configure a firewall using the API.

Palo Alto Networks uses XML as the data structure for it's representation of the configuration file. Automating a firewall takes three steps.

Creating the xml file
Pushing the xml file to the firewall
Committing the candidate configuration

I created a sample xml configuration file that would add an IP address to a sub-interface with a vlan tag of 1 on interface ethernet1/3.

<entry name="ethernet1/3">
<layer3>
    <units>
      <entry name="ethernet1/3.1">
        <tag>1</tag>
        <ip>
          <entry name="30.4.1.2/24"/>
        </ip>
      </entry>
      </entry>
    </units>
</layer3>
</entry>

This is going to be placed into a text file called sub-int.xml which I'll use later.

In my script I read from the file and and place it into a variable called data. I strip the newlines so that I don't have separate each line into an array.

Last you need to commit the config. One thing about the api is that the commit call needs an xml element <commit/>

When I tried it without a cmd ie. xapi.commit(), I got the following error.

pan.xapi.PanXapiError: Missing value for parameter "cmd".

This was confusing at first, until I spoke with a Palo Alto networks Solutions Architect about it and he explained that you need to tell it which type of commit you want. There are a few options such as commit, commit partial and commit full. I think there should be a default setting. Commit without any input should mean a normal commit. Maybe I'll modify a git cloned repository.

script
-----------
import pan.xapi
from cred import get_pan_credentials
credentials = get_pan_credentials()

print credentials
xapi = pan.xapi.PanXapi(**credentials)

xpath = "/config/devices/entry/network/interface/ethernet"

#open xml file and read it into a variable called data.
with open ("sub-int.xml", "r") as myfile:
    data=myfile.read().replace('\n', '')

#set the config using the above xpath
xapi.set(xpath,element=data)

#commit the config. Make sure to add the xml command.
xapi.commit('<commit/>')

-------------

Here's the resulting screen cap:

Monday, November 3, 2014

Accessing a PAN firewall through an api with python

I found out that someone (Kevin Steves) made a python module to access Palo Alto Networks API. You can clone the repository from github.

git clone https://github.com/kevinsteves/pan-python.git

The module is very thorough and the documentation is pretty good. It was simple to test out and I was able to get it working in a few hours.

From what I can tell the module makes a REST call and exports the results into XML. This makes automation a lot easier as the data structure can be easily parsed.

My first script looks like this:

import pan.xapi

from cred import get_pan_credentials

credentials = get_pan_credentials()

print credentials

xapi = pan.xapi.PanXapi()

xapi.op(cmd='show system info', cmd_xml=True)

print xapi.xml_result()

----------

It makes a call to my credentials file

$ more cred.py

def get_pan_credentials():

 cred =  {}

 cred['api_username'] = "admin"

 cred['api_password'] = "admin"

 cred['hostname'] = "192.168.1.1"

 return cred

------------

The script in action.

$ python test1.py 

{'api_key': 'LUFRPT14MW5xOEo1R09KVlBZNnpnemh0VHRBOWl6TGM9bXcwM3JHUGVhRlNiY0dCR0srNERUQT09', 'hostname': '192.168.1.1', 'api_password': 'admin', 'api_username': 'admin'}

<system><hostname>PA-7050</hostname><ip-address>192.168.1.1</ip-address><netmask>255.255.255.0</netmask><default-gateway>192.168.1.254</default-gateway><ipv6-address>unknown</ipv6-address><ipv6-link-local-address>fe80::290:fbff:fe4d:175c/64</ipv6-link-local-address><ipv6-default-gateway /><mac-address>00:90:fb:4d:17:5c</mac-address><time>Mon Nov  3 23:28:08 2014</time>

<uptime>0 days, 11:39:35</uptime>

<devicename>PA-7050</devicename>

<family>7000</family><model>PA-7050</model><serial>015128030274</serial><sw-version>6.0.6</sw-version>

<global-protect-client-package-version>0.0.0</global-protect-client-package-version>

<app-version>466-2435</app-version>

<app-release-date>2014/10/28  20:28:09</app-release-date>

<av-version>1408-1880</av-version>

<av-release-date>2014/11/03  04:00:02</av-release-date>

<threat-version>466-2435</threat-version>

<threat-release-date>2014/10/28  20:28:09</threat-release-date>

<wildfire-version>0</wildfire-version>

<wildfire-release-date>unknown</wildfire-release-date>

<url-filtering-version>0000.00.00.000</url-filtering-version>

<global-protect-datafile-version>0</global-protect-datafile-version>

<global-protect-datafile-release-date>unknown</global-protect-datafile-release-date><logdb-version>6.0.6</logdb-version>

<platform-family>7000</platform-family>

<logger_mode>False</logger_mode>

<vpn-disable-mode>off</vpn-disable-mode>

<operational-mode>normal</operational-mode>

<multi-vsys>off</multi-vsys>

</system>

Sunday, October 12, 2014

Not all APIs are created equally.

Automation is starting to become the new catch phrase in the networking Industry. It seems like 2014 is the year that marketing groups from different vendors have been touting APIs on their products. Skeptical networking engineers however have claimed that an API does not mean that their jobs are getting easier. In fact it’s been making their jobs a little harder.

Before a networking engineer could strictly focus on pure networking. But now, network engineers are increasingly required to know more and more on how to code or at least know how to read code.

Just because you have an API doesn’t mean all the devices can and will play nice with each other.

We can see this by looking at three different platforms.

OpenStack Contrail Junos

Now let’s look at their API for data retrieval/configuration

REST REST Netconf

Ok now you have two platforms that use one type of API and a third platform that uses a different API.

Let's look at the resulting Data Structure response

JSON JSON XML

Again we have two platforms that have the same Data Structure and a third with a different Data Structure.

You might say, ok, at least two of these platforms have the same API and with the same data structure things should be good for both of them right? Actually as they say, the devil is in the details.

I can illustrate this just by looking at a simple IPv4 subnet.

On Openstack the data abstracted looks like this

{ "networks": [ { "contrail:subnet_ipam": [ { "subnet_cidr": "12.1.1.0/24", } ] } ] }

On Contrail it looks like this

{"virtual-network":{ "network_ipam_refs":[ { "attr": { "ipam_subnets": [ { "subnet": { "ip_prefix": "12.1.1.0", "ip_prefix_len": 24 }, } ] }, } ], } }

You can see that one platform combines the subnet with the mask while the other one separates it. For a DevOps engineers and Network Engineers this is annoying. It’s like having to learn different Network Operating systems. The goal of an API should be to allow a simplified abstraction layer.

APIs need to be standardized. Openflow is a good attempt at this. Openflow requires the underlay to have a common protocol in order to allow a controller to programmatically configure them. The networking industry has done a great job at standardizing protocols but a sorry job at creating a common API standard. Maybe the IETF needs to jump in on this. A standardized API could ultimately make our jobs that much more easier.

Friday, October 10, 2014

How to use cURL to spin up an OpenStack VM

I know this is a little long but there are dependencies on how this works. Hopefully you can understand it. The last cURL command is the piece that puts it all together and spins up a VM. The last command will spin up a instance called "instance1", use a flavor and an cirros image and place it in the "VN2" network

LAPTOP:SCRIPTS$cat openstack-create.sh
TOKEN=""

#Do a POST to GET the AUTH TOKEN
TOKEN=$(curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "admin",
          "domain": { "id": "default" },
          "password": "password",
          "tenantName": "admin"
        }
      }
    },
    "scope": {
      "project": {
        "name": "demo",
        "domain": { "id": "default" }
      }
    }
}
}' \
http://192.168.1.4:5000/v3/auth/tokens | grep X-Subject-Token: | sed -e "s/X-Subject-Token: //")

#STORE TOKEN
header='X-Auth-Token: '$TOKEN
#echo "$header"

#NOW USE the AUTH TOKEN TO CHECK TENANTS
curl -X GET http://192.168.1.4:5000/v2.0/tenants/ -H "$header" | python -m json.tool

#CHECK IMAGES FOR TENANT/PROJECT CALLED DEMO
# TENANT ID FOR DEMO = 6d0ebb466f01454e897e518289b3b785
echo "IMAGES"
curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/images | python -m json.tool

# CHECK FLAVORS
echo "FLAVORS"

curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/flavors | python -m json.tool

# CHECK SERVERS
echo "SERVERS"

curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/servers | python -m json.tool

# CHECK NETWORK ID - THIS IS NEUTRON AND HAS IT'S OWN PORT NUMBER 9696
echo "NETWORKS"

curl -H "Content-Type: application/json" -H "$header" -X GET http://192.168.1.4:9696/v2.0/networks | python -m json.tool

#CREATE VM
echo "CREATE VM"

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -H "X-Auth-Project-Id: demo" -H "$header" -X POST http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/servers -d '{"server": {"name": "instance1", "imageRef": "d5f30886-e8ce-4e89-9709-c49a40717b5b", "flavorRef": "2", "max_count": 1, "min_count": 1,"key_name": "xyz", "networks": [{"network": "bd912f99-c345-4f3e-8026-88e1dde42255", "uuid": "bd912f99-c345-4f3e-8026-88e1dde42255"}]}}'

-----------------------

I don’t execute the script and print it out as it would not look too pretty.

The most annoying part of this is all the curly brackets I had to dig around for the formatting of "networks" part as this was the hardest part.

I found out the formatting was like this:

networks: [{"port": String, "fixed_ip": String, "uuid": String, "network": String}, {"port": String, "fixed_ip": String, "uuid": String, "network": String}, ...]

Here's the api that I found useful:

http://developer.openstack.org/api-ref-compute-v2.html

Below is how it would look like if you executed the commands directly on the compute node. Note how image-list has the id for "imageRef"

Thursday, October 2, 2014

Is it the end of the TOR as we know it?

There are two driving forces that may break the TOR (Top of Rack) switch market.

The first one is the spiraling fall of switch prices. Competition among white box switch makers Cumulus and Pica8, incumbents such as Juniper and Cisco and new startups Arista are pushing margins ever so slim. Pretty much all the vendors sell the same exact TOR switch. Currently it’s a 40x10GE port switch with 4x40G uplinks with a Broadcom chip set. The only difference really is the Operating System. All this competition basically means a race to the bottom. I see a few outcomes of this market.

If you’re an enterprise like Facebook or Google, you have the technical savvy and economics of scale, to OEM just the physical hardware components and put on it a stripped down version of linux that does specifically what you need. Facebook already has a blue print for one called the Wedge. This means, none of the vendors can sell to these customers.

Next you have a customer that is DevOps savvy, but doesn’t want to build their own switches. These can be startups or large customers that have a good engineering team. They probably buy a TOR switch such as Pica8 or Cumulus and automate on top of these boxes. They don’t need much support.

Last you have a customer who just needs support so they just buy a brand named switch. These can be large Enterprises that are not technical enough to program their switches. These are the Coca-Colas and traditional companies where technology is not in their wheel house. I mean does Coca-Cola need SDN?

The second driving force is on the physical hardware side. Intel has announced the roadmap of the Rack Scale Architecture (RSA). They want to dominate the cloud computing infrastructure by providing mega scale data centers with a highly scalable solution. To understand what this means we have to see the trends of the current data center.

Currently in a typical rack you have several servers. Each server is usually kept in a server tray or server blade. If you break it down, a server has the following components. A CPU, Memory, Storage (HDD or SSD) and a Network Card. You might have some other peripherals, but you don’t really need them. All of these components interconnect via a PCIe fiber connection. Now this PCIe fiber connection is the key component.

If you disaggregate servers, you can create pools of resources. You can have a pool or tray of CPUs, a pool of Memory and a pool of storage. When you need to upgrade, you pull out the tray and swap in a new tray of resources. To connect all these together, Intel created silicon photonics PCIe (SiPh). These have speeds between 40-50Gbps in a single optical fiber. If you think about it, a majority of traffic runs east and west in a datacenter. With SiPh you can send traffic, intra-Rack at 50Gbs.

You don’t need a TOR switch because you can do this using a virtual switch and SiPh. You can bypass ethernet connections by connecting directly to another logical server over the PCIe connection. Why waste time going out an ethernet port up to a ToR switch and go back down to another NIC to get to a server when you can directly. You can shave even more nano seconds of networking speed.

Now you can say that traffic going between logical servers could eat up CPU, therefore you need a ToR switch to speed this up. If you need hardware acceleration, then Intel could just place a Fulcrum Switch Chip in the rack as a resource. North/South traffic or Inter-Rack traffic would egress the RSA via uplink ethernet connections. Again with RSA, you could create a separate Ethernet Resource pool of NIC for interconnects. Some people may wonder why? I mean have separate servers does the job just nicely. Also with all the resources pooled together, you really need DevOps team that can program the Rack. Well if I was Intel I'd target the Financial Industry. They would pay for the shaving of speeds that RSA could save. They have an Army of DevOps who can do this.

Tuesday, September 30, 2014

Using python to connect remotely to Openstack

In order to beef up my DevOps skills I decided to see if I could connect through a REST API using python. I found out that there are some python modules built specifically for this.

One such module is called novaclient. You would need pip on your machine to do this.

sudo easy_install pip

Then you can use pip to install the nova client.

sudo pip install python-novaclient

If you want to look at the details of the python module it can be found here:

https://github.com/openstack/python-novaclient

This is where I look more into the code and found out it’s capabilities

You can pass parameters to the “client” function.

def __init__(self, username=None, api_key=None, project_id=None,

auth_url=None, insecure=False, timeout=None,

proxy_tenant_id=None, proxy_token=None, region_name=None,

endpoint_type='publicURL', extensions=None,

service_type='compute', service_name=None,

volume_service_name=None, timings=False, bypass_url=None,

os_cache=False, no_cache=True, http_log_debug=False,

auth_system='keystone', auth_plugin=None, auth_token=None,

cacert=None, tenant_id=None, user_id=None,

connection_pool=False, session=None, auth=None,

completion_cache=None):

First I built a login function that will pass these parameters which I can call from my main script.

login.py

-----------

def get_nova_credentials():

 cred =  {}

 cred['username'] = "admin"

 cred['api_key'] = “password”

 cred['auth_url'] = "http://<openstack-ip>:5000/v2.0"

 cred['project_id'] = "demo"

 cred['service_type'] = "compute"

 return cred

Now in my main script I can import the client and my credentials

server-list.py

-----------------

#!/usr/bin/env python

import novaclient

from novaclient.v1_1 import client

from login import get_nova_credentials

credentials = get_nova_credentials()

#Pass credentials to the client function.

nova = client.Client(**credentials)

#grab the list of servers and print out the id, names and status

vms = nova.servers.list(detailed=True)

for vm in vms:

  print vm.id, vm.name, vm.status

—————

script in action.

laptop$ python server-list.py 

f5801333-5d81-496c-b257-e589ca36e944 Cirros-VM2 ACTIVE

098270e0-e5fb-4ea6-a1f1-2dfca11a409d Cirros-VM1 ACTIVE

So what's the big deal? Why do this when you can use the Horizon web ui?

Now that I have a basic understanding of this module, I can start automating things such as building VMs, virtual networks, etc in a scaled and precise manner.