Using cURL to edit and delete RESTful API Objects in a Palo Alto Networks Firewall

XML is a very hard language to understand when you are first working with it. For example when you refer to an element like:

<ship>Titanic</ship>

you refer to everything including the start tag and the end tag. The information inside the tags is text.

You need to keep this in mind when trying to reference things.

Another confusing thing is xpath and attributes. Take this XML example:

<rules>
  <entry name="rule1">
   <from>
       <member>
            Trust
       </member>
   </from>
  </entry>
  <entry name="rule2">
   <from>
       <member>
            UnTrust
       </member>
   </from>
  </entry>

<rules>

If you want to reference something say 'rule2' then what you want is the attribute value. You would use entry[@name='rule2'] entry is the element, name is the attribute and 'rule2' is the attribute value

If you want to reference the text value within an element then you would use element[text()='value'].

For example if you want to reference 'Trust' you can use member[text()='Trust']

This leads to why it could get a little confusing when trying to edit and delete specific values using the Palo Alto Networks API. Let's look at the following rule.

To delete a source-user member named 'acme\bob' in a group of source users
, use the below xpath:

xpath=/config/devices/entry[@name='<domain>']/vsys/entry[@name='<vsysname>']/rulebase/security/rules/entry[@name='<rulename>']/source-user/member[text()='acme\bob']


$curl -k "https://192.168.1.1/api/?type=config&action=delete&xpath=/config/devices/entry\[@name='localhost.localdomain'\]/vsys/entry\[@name='vsys1'\]/rulebase/security/rules/entry\[@name='deny-rule1'\]/source-user/member\[text()='acme\bob'\]&key=<API-KEY>"

<response status="success" code="20"><msg>command succeeded</msg>


If you want to edit a member value, then you need to reference the original member value with member\[text()='<value>'\] and then use the element parameter for the modified member text value: element=<xml code>

for example using curl

$ curl -k "https://192.168.1.1/api/?type=config&action=edit&xpath=/config/devices/entry\[@name='localhost.localdomain'\]/vsys/entry\[@name='vsys1'\]/rulebase/security/rules/entry\[@name='deny-rule1'\]/source-user/member\[text()='acme\bob'\]&element=<member>acme\calvin</member>&key=<API-KEY>"

<response status="success" code="20"><msg>command succeeded</msg></response>

Below are all xpath expressions you can use when accessing the Palo Alto Networks api.

Examples:

/source-user/member[position()<4]
Selects the first three member elements that are children of the source-user element

$ curl -k "https://192.168.1.1/api/?type=config&action=get&xpath=/\[@name='localhost.localdomain'\]/vsys/entry\[@name='vsys1'\]/rulebase/security/rules/entry\[@name='rule1'\]/source-user/member\[position()<4\]&key=<API-KEY>"
<response status="success" code="19"><result total-count="3" count="3">
  <member admin="admin" time="2015/10/21 13:42:11">acme\amy</member>
  <member admin="admin" time="2015/10/21 13:42:11">acme\bob</member>
  <member admin="admin" time="2015/10/21 13:42:11">acme\calvin</member>

/source-user/member[2] Selects the second member element that is the child of the source-user element


$ curl -k "https://192.168.1.1/api/?type=config&action=get&xpath=/\[@name='localhost.localdomain'\]/vsys/entry\[@name='vsys1'\]/rulebase/security/rules/entry\[@name='rule1'\]/source-user/member\[2\]&key=<API-KEY>" <response status="success" code="19"><result total-count="1" count="1">
  <member admin="admin" time="2015/10/21 13:42:11">acme\bob</member>

Automating a Palo Alto Networks Firewall using Python

In my last post I used a python script to extract information from a Palo Alto Networks Firewall using pan-python. In this post I'll illustrate how to configure a firewall using the API.

Palo Alto Networks uses XML as the data structure for it's representation of the configuration file. Automating a firewall takes three steps.

• Creating the xml file
• Pushing the xml file to the firewall
• Committing the candidate configuration

 I created a sample xml configuration file that would add an IP address to a sub-interface with a vlan tag of 1 on interface ethernet1/3.

<entry name="ethernet1/3">
  <layer3>
    <units>
      <entry name="ethernet1/3.1">
        <tag>1</tag>
        <ip>
          <entry name="30.4.1.2/24"/>
        </ip>
      </entry>
      </entry>
    </units>
  </layer3>
</entry>

This is going to be placed into a text file called sub-int.xml which I'll use later.

In my script I read from the file and and place it into a variable called data. I strip the newlines so that I don't have separate each line into an array.

Last you need to commit the config. One thing about the api is that the commit call needs an xml element <commit/>

When I tried it without a cmd ie. xapi.commit(), I got the following error.


pan.xapi.PanXapiError: Missing value for parameter "cmd".

This was confusing at first, until I spoke with a Palo Alto networks Solutions Architect about it and he explained that you need to tell it which type of commit you want. There are a few options such as commit, commit partial and commit full. I think there should be a default setting.  Commit without any input should mean a normal commit. Maybe I'll modify a git cloned repository.


script
-----------
import pan.xapi
from cred import get_pan_credentials
credentials = get_pan_credentials()

print credentials
xapi = pan.xapi.PanXapi(**credentials)

xpath = "/config/devices/entry/network/interface/ethernet"

#open xml file and read it into a variable called data.
with open ("sub-int.xml", "r") as myfile:
    data=myfile.read().replace('\n', '')

#set the config using the above xpath
xapi.set(xpath,element=data)

#commit the config. Make sure to add the xml command.
xapi.commit('<commit/>')



-------------

Here's the resulting screen cap:

Accessing a PAN firewall through an api with python

I found out that someone (Kevin Steves) made a python module to access Palo Alto Networks API. You can clone the repository from github.

git clone https://github.com/kevinsteves/pan-python.git

The module is very thorough and the documentation is pretty good. It was simple to test out and I was able to get it working in a few hours.

From what I can tell the module makes a REST call and exports the results into XML. This makes automation a lot easier as the data structure can be easily parsed.

My first script looks like this:

import pan.xapi

from cred import get_pan_credentials

credentials = get_pan_credentials()

print credentials

xapi = pan.xapi.PanXapi()

xapi.op(cmd='show system info', cmd_xml=True)

print xapi.xml_result()

----------

It makes a call to my credentials file

$ more cred.py

def get_pan_credentials():

 cred =  {}

 cred['api_username'] = "admin"

 cred['api_password'] = "admin"

 cred['hostname'] = "192.168.1.1"

 return cred

------------

The script in action.

$ python test1.py 

{'api_key': 'LUFRPT14MW5xOEo1R09KVlBZNnpnemh0VHRBOWl6TGM9bXcwM3JHUGVhRlNiY0dCR0srNERUQT09', 'hostname': '192.168.1.1', 'api_password': 'admin', 'api_username': 'admin'}

<system><hostname>PA-7050</hostname><ip-address>192.168.1.1</ip-address><netmask>255.255.255.0</netmask><default-gateway>192.168.1.254</default-gateway><ipv6-address>unknown</ipv6-address><ipv6-link-local-address>fe80::290:fbff:fe4d:175c/64</ipv6-link-local-address><ipv6-default-gateway /><mac-address>00:90:fb:4d:17:5c</mac-address><time>Mon Nov  3 23:28:08 2014</time>

<uptime>0 days, 11:39:35</uptime>

<devicename>PA-7050</devicename>

<family>7000</family><model>PA-7050</model><serial>015128030274</serial><sw-version>6.0.6</sw-version>

<global-protect-client-package-version>0.0.0</global-protect-client-package-version>

<app-version>466-2435</app-version>

<app-release-date>2014/10/28  20:28:09</app-release-date>

<av-version>1408-1880</av-version>

<av-release-date>2014/11/03  04:00:02</av-release-date>

<threat-version>466-2435</threat-version>

<threat-release-date>2014/10/28  20:28:09</threat-release-date>

<wildfire-version>0</wildfire-version>

<wildfire-release-date>unknown</wildfire-release-date>

<url-filtering-version>0000.00.00.000</url-filtering-version>

<global-protect-datafile-version>0</global-protect-datafile-version>

<global-protect-datafile-release-date>unknown</global-protect-datafile-release-date><logdb-version>6.0.6</logdb-version>

<platform-family>7000</platform-family>

<logger_mode>False</logger_mode>

<vpn-disable-mode>off</vpn-disable-mode>

<operational-mode>normal</operational-mode>

<multi-vsys>off</multi-vsys>

</system>

Using python to connect remotely to Openstack

In order to beef up my DevOps skills I decided to see if I could connect through a REST API using python. I found out that there are some python modules built specifically for this.

One such module is called novaclient. You would need pip on your machine to do this.

sudo easy_install pip

Then you can use pip to install the nova client.

sudo pip install python-novaclient

If you want to look at the details of the python module it can be found here:

https://github.com/openstack/python-novaclient

This is where I look more into the code and found out it’s capabilities

You can pass parameters to the “client” function.

def __init__(self, username=None, api_key=None, project_id=None,

auth_url=None, insecure=False, timeout=None,

proxy_tenant_id=None, proxy_token=None, region_name=None,

endpoint_type='publicURL', extensions=None,

service_type='compute', service_name=None,

volume_service_name=None, timings=False, bypass_url=None,

os_cache=False, no_cache=True, http_log_debug=False,

auth_system='keystone', auth_plugin=None, auth_token=None,

cacert=None, tenant_id=None, user_id=None,

connection_pool=False, session=None, auth=None,

completion_cache=None):

First I built a login function that will pass these parameters which I can call from my main script.

login.py

-----------

def get_nova_credentials():

 cred =  {}

 cred['username'] = "admin"

 cred['api_key'] = “password”

 cred['auth_url'] = "http://<openstack-ip>:5000/v2.0"

 cred['project_id'] = "demo"

 cred['service_type'] = "compute"

 return cred

Now in my main script I can import the client and my credentials

server-list.py

-----------------

#!/usr/bin/env python

import novaclient

from novaclient.v1_1 import client

from login import get_nova_credentials

credentials = get_nova_credentials()

#Pass credentials to the client function.

nova = client.Client(**credentials)

#grab the list of servers and print out the id, names and status

vms = nova.servers.list(detailed=True)

for vm in vms:

  print vm.id, vm.name, vm.status

—————

script in action.

laptop$ python server-list.py 

f5801333-5d81-496c-b257-e589ca36e944 Cirros-VM2 ACTIVE

098270e0-e5fb-4ea6-a1f1-2dfca11a409d Cirros-VM1 ACTIVE

So what's the big deal? Why do this when you can use the Horizon web ui?

Now that I have a basic understanding of this module, I can start automating things such as building VMs, virtual networks, etc in a scaled and precise manner.

Exploring the REST API on contrail.

I noticed on github that there was a tutorial on how to access the REST API on Juniper Contrail.

https://juniper.github.io/contrail-vnc/api-doc/html/tutorial_with_rest.html#

The easiest way to access this is by using cURL. Contrail uses tcp port 8082 for accessing it's REST API.

The url http:/contrail-ip/virtual-networks prints out a list of configured virtual networks on Contrail.

$ curl -X GET -H "Content-Type: application/json; charset=UTF-8" http://172.16.1.4:8082/virtual-networks | python -mjson.tool

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  1229  100  1229    0     0   2129      0 --:--:-- --:--:-- --:--:--  2129

{

    "virtual-networks": [

        {

            "fq_name": [

                "default-domain",

                "default-project",

                "__link_local__"

            ],

            "href": "http://172.16.1.4:8082/virtual-network/4092df7b-997a-4ee7-a5cc-46d5db1187d4",

            "uuid": "4092df7b-997a-4ee7-a5cc-46d5db1187d4"

        },

        {

            "fq_name": [

                "default-domain",

                "default-project",

                "default-virtual-network"

            ],

            "href": "http://172.16.1.4:8082/virtual-network/34579f9a-064e-4048-96a7-a30355c54e44",

            "uuid": "34579f9a-064e-4048-96a7-a30355c54e44"

        },

        {

            "fq_name": [

                "default-domain",

                "default-project",

                "ip-fabric"

            ],

            "href": "http://172.16.100.104:8082/virtual-network/db7d1afe-bcaa-456b-b33a-9a36f6d176fe",

            "uuid": "db7d1afe-bcaa-456b-b33a-9a36f6d176fe"

        },

        {

            "fq_name": [

                "default-domain",

                "demo",

                "Network1"

            ],

            "href": "http://172.16.1.4:8082/virtual-network/380c0f4e-34b6-4901-98b2-61a8ed7afd7d",

            "uuid": "380c0f4e-34b6-4901-98b2-61a8ed7afd7d"

        },

        {

            "fq_name": [

                "default-domain",

                "demo",

                "Network2"

            ],

            "href": "http://172.16.100.104:8082/virtual-network/ffe03354-aaa2-4305-b615-654e14111134",

            "uuid": "ffe03354-aaa2-4305-b615-654e14111134"

        },

        {

            "fq_name": [

                "default-domain",

                "demo",

                "Network3"

            ],

            "href": "http://172.16.1.4:8082/virtual-network/93317422-eea4-4b77-88cb-5aaac3bb58b6",

            "uuid": "93317422-eea4-4b77-88cb-5aaac3bb58b6"

        }

    ]

}

However cURL, at least for me, isn’t capable of abstracting the data structures programatically. It's more like screen scraping. So I dug around the internet and noticed that python has a curl module.

http://pycurl.sourceforge.net/doc/quickstart.html#

Now I can use python to execute cURL to pull data from a REST API. 

There are a few parts to this script.

The first part is a function to issue the cURL command.

The second part is to abstract the route-target from a virtual network created in the Contrail controller.

This could then be used later on with another script to create a template configuration and program the underlay gateway router with a VRF.

In this script I cheated a little by grabbing a specific virtual-network’s url address from the previous curl command. Then parsed the data to grab the information I was looking for.

—————————————

import pycurl

import StringIO

import json

#Function to issue cURL command

def get_url(WEB):

  buf = StringIO.StringIO()

  c = pycurl.Curl()

  c.setopt(c.URL, WEB)

  c.setopt(c.WRITEFUNCTION, buf.write)

  c.perform()

  body = buf.getvalue()

  network = json.loads(body)

  buf.close()

  return network

#URL of virtual network on contrail

SITE = 'http://172.16.1.4:8082/virtual-network/380c0f4e-34b6-4901-98b2-61a8ed7afd7d'

objects = get_url(SITE)

#pretty print the json results

print json.dumps(objects, sort_keys=True, indent=4)

#This part is to grab the path of the Virtual Network name and RT from contrail

print "Name: ", objects['virtual-network']['name'], " RT: ", objects['virtual-network']['route_target_list']['route_target'][0]

Script in action:

——————

$ python contrail.py 

{

    "virtual-network": {

        "fq_name": [

            "default-domain", 

            "demo", 

            "Network1"

        ], 

        "href": "http://172.16.1.4:8082/virtual-network/380c0f4e-34b6-4901-98b2-61a8ed7afd7d", 

        "id_perms": {

            "created": "2014-09-19T19:19:11.650288", 

            "description": null, 

            "enable": true, 

            "last_modified": "2014-09-27T05:22:30.453524", 

            "permissions": {

                "group": "cloud-admin-group", 

                "group_access": 7, 

                "other_access": 7, 

                "owner": "cloud-admin", 

                "owner_access": 7

            }, 

            "uuid": {

                "uuid_lslong": 11002964217786203517, 

                "uuid_mslong": 4038619794410719489

            }

        }, 

        "instance_ip_back_refs": [

            {

                "attr": null, 

                "href": "http://172.16.1.4:8082/instance-ip/17b19dc0-b177-4df7-955b-57b8a87caf28", 

                "to": [

                    "17b19dc0-b177-4df7-955b-57b8a87caf28"

                ], 

                "uuid": "17b19dc0-b177-4df7-955b-57b8a87caf28"

            }, 

            {

                "attr": null, 

                "href": "http://172.16.1.4:8082/instance-ip/d525ddce-3542-4986-b8d1-56f3831d8678", 

                "to": [

                    "d525ddce-3542-4986-b8d1-56f3831d8678"

                ], 

                "uuid": "d525ddce-3542-4986-b8d1-56f3831d8678"

            }

        ], 

        "is_shared": false, 

        "name": "Network1", 

        "network_ipam_refs": [

            {

                "attr": {

                    "ipam_subnets": [

                        {

                            "default_gateway": "100.1.1.254", 

                            "subnet": {

                                "gw": "100.1.1.254", 

                                "ip_prefix": "100.1.1.0", 

                                "ip_prefix_len": 24

                            }

                        }

                    ]

                }, 

                "href": "http://172.16.1.4:8082/network-ipam/1f24fa35-b7bf-4d0f-8185-746f58e234c9", 

                "to": [

                    "default-domain", 

                    "demo", 

                    "Network1-ipam"

                ], 

                "uuid": "1f24fa35-b7bf-4d0f-8185-746f58e234c9"

            }

        ], 

        "network_policy_refs": [

            {

                "attr": {

                    "sequence": {

                        "major": 0, 

                        "minor": 0

                    }, 

                    "timer": null

                }, 

                "href": "http://172.16.1.4:8082/network-policy/0a1c1776-a323-4c00-959a-aada50b91be8", 

                "to": [

                    "default-domain", 

                    "demo", 

                    "net1<->net2"

                ], 

                "uuid": "0a1c1776-a323-4c00-959a-aada50b91be8"

            }

        ], 

        "parent_href": "http://172.16.1.4:8082/project/3af66afe-3284-40cc-8b04-85c69af512c7", 

        "parent_type": "project", 

        "parent_uuid": "3af66afe-3284-40cc-8b04-85c69af512c7", 

        "route_target_list": {

            "route_target": [

                "target:64512:100"

            ]

        }, 

        "router_external": false, 

        "routing_instances": [

            {

                "href": "http://172.16.1.4:8082/routing-instance/06165761-3b55-417d-acf0-0ad27a9010d0", 

                "to": [

                    "default-domain", 

                    "demo", 

                    "Network1", 

                    "Network1"

                ], 

                "uuid": "06165761-3b55-417d-acf0-0ad27a9010d0"

            }

        ], 

        "uuid": "380c0f4e-34b6-4901-98b2-61a8ed7afd7d", 

        "virtual_machine_interface_back_refs": [

            {

                "attr": null, 

                "href": "http://172.16.1.4:8082/virtual-machine-interface/2aefccdc-bf34-4d3b-bd37-41f716274883", 

                "to": [

                    "e432ad9a-8f6e-4f7c-813b-18ada76bfd64", 

                    "2aefccdc-bf34-4d3b-bd37-41f716274883"

                ], 

                "uuid": "2aefccdc-bf34-4d3b-bd37-41f716274883"

            }, 

            {

                "attr": null, 

                "href": "http://172.16.1.4:8082/virtual-machine-interface/f20bb145-49cd-43bf-a6e5-9d9c50794244", 

                "to": [

                    "2e775d6f-8043-41df-9295-d1b8d8f705a2", 

                    "f20bb145-49cd-43bf-a6e5-9d9c50794244"

                ], 

                "uuid": "f20bb145-49cd-43bf-a6e5-9d9c50794244"

            }

        ], 

        "virtual_network_properties": {

            "extend_to_external_routers": null, 

            "forwarding_mode": "l2_l3", 

            "network_id": 4, 

            "vxlan_network_identifier": null

        }

    }

}

Name:  Network1  RT:  target:64512:100

As you can see I was able to pull the "name" of the virtual network and the route-target of the virtual network. Later I can then create a script template to build the VRF on the Router Gateway like a MX.  That's one step closer to network automation.

Network Automation is as easy as Py

as in PyEZ. PyEZ is a micro-framework to remotely manage and automate Juniper devices. It works with Python and allows you to pull Junos specific features into an abstraction layer.  This is great because you don't have to do any screen scraping to pull out any fields. I installed this module on my Mac to test this out.

The documentation is located here is great because you can look at the apis on how to create your script. The first script I wanted to test out is how to pull information from a router.

PyEZ can use YAML which is a human readable format markup language. I created a yaml file to extract the fields I was looking for. 

Here's my yaml file.

vrf.yml
VRF:
  get: routing-instances/instance
  args_key: name
  view: VRFView

VRFView:
 fields:
  instance_name: name
  instance_type: instance-type
  rd_type: route-distinguisher/rd-type
  vrf_target: vrf-target/community
  interface: interface/name

My script will parse VRFs on a router. I created two routing instances for this demo.

jnpr@R1# show routing-instances
VRF1 {
    instance-type vrf;
    interface lo0.1;
    route-distinguisher 1.1.1.1:100;
    vrf-target target:100:100;
}
VRF2 {
    instance-type vrf;
    interface lo0.2;
    route-distinguisher 1.1.1.1:101;
    vrf-target target:100:101;
}

Now I can test this in python.

$ python
Python 2.7.5 (default, Mar  9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

First I import all the necessary libraries.

>>> from pprint import pprint
>>> from jnpr.junos import Device
>>> from jnpr.junos.op.routes import RouteTable
>>> from lxml import etree
>>> from jnpr.junos.factory import loadyaml
>>> globals().update( loadyaml('vrf.yml') )

Then I open a connection to a junos device

>>> dev = Device('hostname_or_ip', user='username', password='password')
>>> dev.open()
Device(x.x.x.x)


Next I create a table
>>> tbl = VRF(dev)



Then get the fields for the table
>>> tbl.get(values=True) #make sure to pass values=True
VRF:x.x.x.x: 2 items






Now I can iterate through the table an print the contents.

>>> for item in tbl:
...     print 'instance_name:', item.instance_name
...     print 'instance_type:', item.instance_type
...     print 'rd_type:', item.rd_type
...     print 'vrf_target:', item.vrf_target
...     print 'interface:', item.interface
...
instance_name: VRF1
instance_type: vrf
rd_type: 1.1.1.1:101
vrf_target: target:100:101
interface: lo0.1
instance_name: VRF2
instance_type: vrf
rd_type: 1.1.1.1:102
vrf_target: target:100:102
interface: lo0.2

Now I can then manipulate the tables and look at individual fields.

>>> find = tbl['VRF1']

>>> find.interface
'lo0.1'

Now imagine a router with a hundred VRFs. I can now parse through this router remotely and automate operations.

OP script - look up a vendor mac oui directly from a juniper switch using slax and cURL

I found a couple of cool things in Juniper scripting.

There is the ability to retrieve data over an internet connection using curl. From Juniper - cURL is a tool that uses the libcurl library and permits data transfers using a number of protocols, including FTP, FTPS, HTTP, HTTPS, SCP, and SMTP.

BTW I'm using cURL on an EX switch that is running 12.3R3.4. I didn't see this on 11.4 code.

This got me thinking. Whenever I'm troubleshooting a switch, usually due to some errant device in the network, I have to look up the IEEE Mac OUI address. This entails copying each mac address in the mac table of the script. Then I can go to a website like www.macvendors.com and paste the MACOUI to retrieve the vendor of the mac. This could be a pain if you had to parse through hundreds of mac addresses.

Luckily, macvendors.com has an API. 

So what if we used the power of curl and scripting to do this for you. Instead of having to open up a separate web browser, have the switch fetch the info over the internet and do it for you.

See the difference below.

{master:0}[edit]

user@SWITCH# run show ethernet-switching table                  

Ethernet-switching table: 9 entries, 6 learned, 0 persistent entries

  VLAN             MAC address       Type         Age Interfaces

  v250              *                 Flood          - All-members

  v150              *                 Flood          - All-members

  v150              00:00:05:00:00:00 Learn          0 ge-0/0/47.0

  v150              00:21:59:c7:09:41 Learn          0 ge-0/0/47.0

  v150              00:24:dc:d3:1a:10 Learn          0 ge-0/0/45.0

  v100              *                 Flood          - All-members

  v100              00:00:03:00:00:00 Learn          0 ge-0/0/45.0

  v100              00:24:dc:d3:1a:10 Learn          0 ge-0/0/45.0

  v100              a8:d0:e5:5a:59:08 Learn          0 ge-0/0/45.0

Now as an OP script:

{master:0}[edit]

user@SWITCH# run op mac-resolve                   

Vlan           Mac address Interfaces       Age Vendor 

v150  00:00:05:00:00:00 ge-0/0/47.0 0 XEROX CORPORATION

v150  00:21:59:c7:09:41 ge-0/0/47.0 0 Juniper Networks

v150  00:24:dc:d3:1a:10 ge-0/0/45.0 0 Juniper Networks

v100  00:00:03:00:00:00 ge-0/0/45.0 0 XEROX CORPORATION

v100  00:24:dc:d3:1a:10 ge-0/0/45.0 0 Juniper Networks

v100  a8:d0:e5:5a:59:08 ge-0/0/45.0 0 Juniper Networks


Now I can imagine if you really wanted to do some kind of accounting, you could create a Web page of equipment and users in your network. You could create scripts to populate this database. For example, DHCP users would have both their user logins and mac addresses. Then you could build onto this script by extracting the user login and show the user who is connected to the interface above.

The source code

-----------------

version 1.1;

ns curl extension = "http://xml.libslax.org/curl";

ns junos= "http://xml.juniper.net/junos/*/junos";

ns xnm= "http://xml.juniper.net/xnm/1.1/xnm";

ns jcs= "http://xml.juniper.net/junos/commit-scripts/1.0";

import "../import/junos.xsl";

match / {

    <op-script-results> {

        var $mac-info =  <command> "show ethernet-switching table";

        var $mac-table = jcs:invoke($mac-info);

        <output> "Vlan\tMac address\t\tInterfaces\tAge\tVendor ";

        for-each($mac-table//mac-table-entry) {

          if (current()/mac-address != "*") {

        var $test = current()/mac-address;

        var $str = substring ($test,1,8);

          var $url = "http://api.macvendors.com/" _ $str;

        var $options := {

        <url> $url;

        <method> "get";

        }

        var $curl = curl:open();

        var $results = curl:perform($curl,$options);

        var $int = current()/mac-interfaces-list;

        var $int2 = translate ($int, "\t\n\r", "");

        <output> current()/mac-vlan _"\t" _ current()/mac-address _"\t" _ $int2 

_ "\t"_ current()/mac-age _ "\t"_ $results/raw-data;

        expr curl:close($curl);

            }

        }

    }

}

Using CURL to access Openstack

So I'm exploring the API for Openstack to see how I can extract information from an Openstack platform.

I found the examples can be found here.  While the API is located here.

This could be useful as in the future I could script a networking device (Switch, Router, Firewall, etc) to automate and provision services whenever a new tenant is built.

The idea is to self provision itself. For example when a new tenant is created, a switch can access Openstack, find out what network name was configured, which vlan id assigned and a what ip subnet configured.

So the first step is to authenticate to the Openstack Server and retrieve a temporary token.

host1#curl -d '{"auth":{"passwordCredentials":{"username": "admin","password": "mypassword"},"tenantName": "admin"}}' -H "Content-Type: application/js
on" http://x.x.x.x:5000/v2.0/tokens | python -mjson.tool

What comes back is a json response that needs to be parsed.  I piped the python module "python -mjson.tool"  to parse the contents.

What I get in return looks similar to this:

{
    "access": {
        "metadata": {
            "is_admin": 0,
            "roles": [
                "3e2b3771096e4a95b39f3832c19679df"
            ]
        },
        "serviceCatalog": [
            {
                "endpoints": [
                    {

             ....
            }
        ],
 "token": {
            "expires": "2014-07-10T19:31:10Z",
            "id": "14213fjgiod2341vcrt46b9674f",
            "issued_at": "2014-07-09T19:31:10.700433",
            "tenant": {
                "description": "admin tenant",
                "enabled": true,
                "id": "3a10de8a82444118865a6398b336ee68",
                "name": "admin"
            }
        },
        "user": {
            "id": "be246bdf3e01493d8d75bf3938e1bffc",
            "name": "admin",
            "roles": [
                {
                    "name": "admin"
                }
            ],
            "roles_links": [],
            "username": "admin"
        }
    }
}

Now you can see that the token "expires" and this will mean that you will need to request tokens every so often for security purposes.

After this you'll need the token id so you can make further requests.

Next assign your token to a variable.

host1# MyToken=14213fjgiod2341vcrt46b9674f

host1# echo $MyToken
14213fjgiod2341vcrt46b9674f

Then you can use the token to request further information:
host1#curl -s  -H "X-Auth-Token: $MyToken"  http://x.x.x.x:9696/v2.0/networks | python -mjson.tool
{
    "networks": [
        {
            "admin_state_up": true,
            "id": "0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48",
            "name": "Network1",
            "provider:network_type": "local",
            "provider:physical_network": null,
            "provider:segmentation_id": null,
            "router:external": false,
            "shared": false,
            "status": "ACTIVE",
            "subnets": [
                "d4bf516f-135c-4c7f-ba7e-363cb6c7d307"
            ],
            "tenant_id": "3a10de8a82444118865a6398b336ee68"
        }
}

With this information I can do a get on the subnet:

/v2.0/subnets/​{subnet_id}

 # curl -s  -H "X-Auth-Token: $MyToken"  http://x.x.x.x:9696/v2.0/subnets/d4bf516f-135c-4c7f-ba7e-363cb6c7d307 | python -mjson.tool
{
    "subnet": {
        "allocation_pools": [
            {
                "end": "20.20.0.20",
                "start": "20.20.0.10"
            }
        ],
        "cidr": "20.20.0.0/24",
        "dns_nameservers": [
            "198.6.1.1"
        ],
        "enable_dhcp": true,
        "gateway_ip": "20.20.0.1",
        "host_routes": [],
        "id": "d4bf516f-135c-4c7f-ba7e-363cb6c7d307",
        "ip_version": 4,
        "name": "Network1",
        "network_id": "0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48",
        "tenant_id": "3a10de8a82444118865a6398b336ee68"
    }
}

So with this information I can basically extract the cidr and  gateway_ip to populate a gateway router.

SDN - Using POX Openflow controller to program a Juniper EX switch Pt.2

The last blog entry I demonstrated how to configure a Juniper EX9200 switch to communicate with an Openflow controller (POX). The flow entry logic was very basic. It turned all the ports on the Openflow switch into a dumb hub. Flood out all ports except for the source.

In this post I'll demonstrate how to create a more unique flow entry.  This is based on the 10-tuple OpenFlow v1.0 header fields which are:

Ingress Port, Ethec Src, Ether Dst, Ether Type, Vlan ID, IP Dst, IP Src, TCP Dst, TCP Src, and IP Proto

First we need to figure out what a host does on a layer 2 network. If Host A wants to talk to Host B, it will first send out an ARP packet asking what is the mac address of Host B based on the IP address. After recieving a reply, Host A will send a unicast frame with the destnation Mac of Host B. Host B in turn will send unicast packets back to Host A. This means we need to program three types of flows.  One for ARP and two for unicast packets for each direction.

Below is the output in action:

jnpr@EX9200-RE0> show openflow flows detail   

jnpr@EX9200-RE0>

Next I startup the openflow controller:
jnpr@ubuntu:~/OPENFLOW-CONTROLLER$ ./pox.py misc.static
POX 0.2.0 (carp) / Copyright 2011-2013 James McCauley, et al.
INFO:core:POX 0.2.0 (carp) is up.
INFO:openflow.of_01:[3c-8a-b0-0d-c7-c0 1] connected
INFO:misc.static:sending to DPID ARP flow creation 
INFO:misc.static:sending to DPID flow creation->
INFO:misc.static:sending to DPID flow creation<-

On the EX switch:

jnpr@EX9200-RE0> show openflow flows detail   
Flow name: flow-16842752
Table ID: 1     Flow ID: 16842752           
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0     
Match: Input port: 1    
       Ethernet src addr: wildcard         
       Ethernet dst addr: wildcard         
       Input vlan id: 100               Input VLAN priority: wildcard
       Ether type: 0x800  
       IP ToS: wildcard                 IP protocol: wildcard
       IP src addr: wildcard            IP dst addr: 10.1.1.3/32     
       Source port: 0                   Destination port: 0     
Action: Output port 2,

Flow name: flow-33619968
Table ID: 1     Flow ID: 33619968           
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0     
Match: Input port: 2    
       Ethernet src addr: wildcard         
       Ethernet dst addr: wildcard         
       Input vlan id: 100               Input VLAN priority: wildcard
       Ether type: 0x800  
       IP ToS: wildcard                 IP protocol: wildcard
       IP src addr: wildcard            IP dst addr: 10.1.1.2/32     
       Source port: 0                   Destination port: 0     
Action: Output port 1,

Flow name: flow-83951616
Table ID: 1     Flow ID: 83951616           
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0     
Match: Input port: wildcard
       Ethernet src addr: wildcard         
       Ethernet dst addr: wildcard         
       Input vlan id: wildcard          Input VLAN priority: wildcard
       Ether type: 0x806  
       IP ToS: 0x0                      IP protocol: wildcard
       IP src addr: wildcard            IP dst addr: wildcard        
       Source port: 0                   Destination port: 0     
Action: Output port 65531,


jnpr@EX9200-RE0> show openflow flows          
Switch                 Flow      Number of packets    Priority Number of Number of
Name                   ID                                            match    action
oftest-92k             16842752      248                     32768    6         1       
oftest-92k             33619968      248                     32768    6         1       
oftest-92k             83951616        4                       32768    4         1      


Python code for POX

jnpr@ubuntu:~/OPENFLOW-CONTROLLER/pox/misc$ cat static.py

---------------------
from pox.core import core
from pox.lib.packet.ipv4 import ipv4
from pox.lib.packet.arp import arp
import pox.lib.packet as pkt
import pox.openflow.libopenflow_01 as of
import re

log = core.getLogger()

class Sdn (object):
  def __init__ (self, connection):
    self.connection = connection
    connection.addListeners(self)


    # FIRST OPENFLOW RULE - ARP
    #create an Openflow using flow table modification
    arp = of.ofp_flow_mod()
    #Define a match structure for your flow rules to follow
    #if ether type is 0x0806 or Arp
    arp.match.dl_type = 0x0806
    # Add an action to send to flood out all ports except the source
    arp.actions.append(of.ofp_action_output(port = of.OFPP_FLOOD))
    # program flow on switch
    self.connection.send(arp)
    #Send debug message for controller
    log.info("sending to DPID ARP flow creation  " )

    # FIRST OPENFLOW RULE
    #create an Openflow using flow table modification
    msg = of.ofp_flow_mod()
    #Define a match structure for your flow rules to follow
    msg.match.in_port = 1
    msg.match.dl_vlan = 100
    msg.match.dl_type = 0x0800
    msg.match.nw_dst = "10.1.1.3/32"
    # Add an action to send to the specified port
    msg.actions.append(of.ofp_action_output(port = 2))
    # Send message to switch
    self.connection.send(msg)
    #Debug message for controller
    log.info("sending to DPID flow creation->" )

    # SECOND OPENFLOW RULE
    #create an Openflow using flow table modification
    msg2 = of.ofp_flow_mod()
    #Define a match structure for your flow rule
    msg2.match.in_port = 2
    msg2.match.dl_vlan = 100
    msg2.match.dl_type = 0x0800
    msg2.match.nw_dst = "10.1.1.2/32"
    # Add an action to send to the specified port
    msg2.actions.append(of.ofp_action_output(port = 1))
    # Send message to switch
    self.connection.send(msg2)
#Debug message for controller
    log.info("sending to DPID flow creation<-" )


def launch ():
  """
  Starts the component
  """
  def start_switch (event):
    log.debug("Controlling DPID %s" % (event.connection,))
    Sdn(event.connection)
  core.openflow.addListenerByName("ConnectionUp", start_switch)

SDN - Using POX Openflow controller to program a Juniper EX switch Pt.1

Openflow support is now available on Juniper EX switches in version 13.3 of code. I decided to explore and test this out. I first needed to find an Openflow compatible Juniper platform. The EX9200 supports Openflow, but you have to add the openflow image to JUNOS

jnpr@EX9200-RE0>request system software add <jsdn-package-name>

After installation you should see it as a module:

 jnpr@EX9200-RE0# run show version
Hostname: EX9200-RE0
Model: ex9204
Junos: 13.3R1.6
JUNOS Base OS boot [13.3R1.6]
JUNOS Base OS Software Suite [13.3R1.6]
JUNOS 64-bit Kernel Software Suite [13.3R1.6]
[TRUNCATED]
JUNOS py-base-i386 [13.3R1.6]
JUNOS SDN Software Suite [13.3R1.6]

There are a few possible operational commands:

jnpr@EX9200-RE0> show openflow ?
Possible completions:
  capability           Show feature and configuration capability
  controller           Show controller information and connection status
  filters              Show filter information
  flows                Show flow information
  interfaces           Show interface information
  statistics           Show statistics commands
  summary              Show openflow information summary
  switch               Show switch instance description information

To configure openflow, you need to:

1) Create interfaces

2) Associate those interfaces to an OPENFLOW resource group (ie. a virtualized switch)

3) Point to the Openflow controller to receive commands.


First, the creation of interfaces. This is very basic, it's similar to creating normal switching interfaces:

jnpr@EX9200-RE0# show interfaces
xe-2/0/0 {
    unit 0 {
        family ethernet-switching {
            vlan {
                members v100;
            }
        }
    }
}
xe-2/0/1 {
    unit 0 {
        family ethernet-switching {
            interface-mode access;
            vlan {
                members v100;
            }
        }
    }
}

[edit]
jnpr@EX9200-RE0# show vlans
v100 {
    vlan-id 100;
}

Next, you need to place these interfaces into an Openflow resource group (ie a virtualized switch) and map those interfaces to a port-id that Openflow can understand.

jnpr@EX9200-RE0# show protocols openflow
switch OF-SWITCH-92k {
    default-action {
        packet-in;
    }
    interfaces {
        xe-2/0/0.0 port-id 1;
        xe-2/0/1.0 port-id 2;
    }
}


Then you'll need to point to an openflow controller.

set protocols openflow switch OF-SWITCH-92k controller address 10.161.11.77

By default, openflow communicates over TCP using port 6633

Once committed, the EX will continously try to communicate with the openflow server.

jnpr@EX9200-RE0# run show openflow controller                                            
Openflowd controller information:
Controller socket: 12
Controller IP address: 10.161.11.77
Controller protocol: tcp
Controller port: 6633
Controller connection state: down
Number of connection attempt: 10
Controller role: equal

If the EX was correctly communicating with the controller, the connection state should say "up".

jnpr@EX9200-RE0# run show openflow switch
Switch Name:        OF-SWITCH-92k                                             
Switch ID:          0                  Switch DPID:    00:00:3c:8a:b0:0d:c7:c0
Flow mod received:  6                  Vendor received:      0             
Packets sent:       841                Packets received:     845           
Echo req sent:      833                Echo req received:    0             
Echo reply sent:    0                  Echo reply received:  833           
Port Status sent:   0                  Port mod received:    0             
Barrier request:    0                  Barrier reply:        0             
Error msg sent:     0                  Error msg received:   0  

The DPID or DataPathIdentifier is the physical switch that will be programmed. We'll see where this comes into play later.

I created an Ubuntu VM and installed the POX openflow controller. It's Python based which is awesome since it's a language I can understand. There's Floodlight (Java) or Trema (Ruby), so choose the flavor that you're comfortable with.

To simply start with there is a very basic POX python module found under the forwarding directory. This is the module we'll start with just to demonstrate how this works.

jnpr@ubuntu:~/OPENFLOW-CONTROLLER$ ./pox.py log.level --DEBUG forwarding.hub
POX 0.2.0 (carp) / Copyright 2011-2013 James McCauley, et al.
INFO:forwarding.hub:Hub running.
DEBUG:core:POX 0.2.0 (carp) going up...
DEBUG:core:Running on CPython (2.7.5+/Feb 27 2014 19:37:08)
DEBUG:core:Platform is Linux-3.11.0-12-generic-x86_64-with-Ubuntu-13.10-saucy
INFO:core:POX 0.2.0 (carp) is up.
DEBUG:openflow.of_01:Listening on 0.0.0.0:6633
INFO:forwarding.hub:Hubifying 3c-8a-b0-0d-c7-c0  <<<< DPID of the EX Switch

What this does to the EX switch basically turns those two interfaces in a small hub.

On POX the gist of the python script is this:


  msg = of.ofp_flow_mod()
  msg.actions.append(of.ofp_action_output(port = of.OFPP_FLOOD))

In Openflow terminology:

OFPP_FLOOD - output all openflow ports except the input port and those with flooding disabled

And that's it.

On the EX the Openflow rule looks like this:

jnpr@EX9200-RE0# run show openflow flows detail
Flow name: flow-65536
Table ID: 1     Flow ID: 65536            
Priority: 32768   Idle timeout(in sec):0        Hard timeout(in sec): 0   
Match: Input port: wildcard
       Ethernet src addr: wildcard       
       Ethernet dst addr: wildcard       
       Input vlan id: wildcard          Input VLAN priority: wildcard
       Ether type: wildcard
       IP ToS: 0x0                      IP protocol: 0x0 
       IP src addr: 0.0.0.0/32          IP dst addr: 0.0.0.0/32    
       Source port: 0                   Destination port: 0   
Action: Output port 65531,


The Action: Output port 65531  basically means everything available openflow interface but where the source of the packet came in on.


I setup a few tester ports to send constant traffic at a very low rate. ~1 pps


jnpr@EX9200-RE0# run show openflow statistics interfaces 
Switch Name: OF-SWITCH-92k                                                     
Interface Name: xe-2/0/0.0       Port Number: 1   
Num of rx pkts: 12                         Num of tx pkts: 12                  
Num of rx bytes: 17952                     Num of tx bytes: 17952               
Num of rx error: 0                         Num of tx error:0                   
Number of packets dropped by RX: 0                   
Number of packets dropped by TX: 0                   
Number of rx frame error:        0                   
Number of rx overrun error:      0                   
Number of CRC error:             0                   
Number of collisions:            0                   

Switch Name: OF-SWITCH-92k                                                     
Interface Name: xe-2/0/1.0       Port Number: 2   
Num of rx pkts: 12                         Num of tx pkts: 12                  
Num of rx bytes: 17952                     Num of tx bytes: 17952               
Num of rx error: 0                         Num of tx error:0                   
Number of packets dropped by RX: 0                   
Number of packets dropped by TX: 0                   
Number of rx frame error:        0                   
Number of rx overrun error:      0                   
Number of CRC error:             0                   
Number of collisions:            0                   

In my next blog entry I'll tweak a POX module to create flow rule entries that coincide more with SDN programming than turning your expensive switch into an expensive hub.