The Network Arborist: automation

Showing posts with label automation. Show all posts

Wednesday, October 21, 2015

Using cURL to edit and delete RESTful API Objects in a Palo Alto Networks Firewall

XML is a very hard language to understand when you are first working with it. For example when you refer to an element like:

<ship>Titanic</ship>

you refer to everything including the start tag and the end tag. The information inside the tags is text.

You need to keep this in mind when trying to reference things.

Another confusing thing is xpath and attributes. Take this XML example:

<rules>
<entry name="rule1">
<from>
<member>
Trust
</member>
</from>
</entry>
<entry name="rule2">
<from>
<member>
UnTrust
</member>
</from>
</entry>

<rules>

If you want to reference something say 'rule2' then what you want is the attribute value. You would use entry[@name='rule2'] entry is the element, name is the attribute and 'rule2' is the attribute value

If you want to reference the text value within an element then you would use element[text()='value'].

For example if you want to reference 'Trust' you can use member[text()='Trust']

This leads to why it could get a little confusing when trying to edit and delete specific values using the Palo Alto Networks API. Let's look at the following rule.

To delete a source-user member named 'acme\bob' in a group of source users
, use the below xpath:

xpath=/config/devices/entry[@name='<domain>']/vsys/entry[@name='<vsysname>']/rulebase/security/rules/entry[@name='<rulename>']/source-user/member[text()='acme\bob']

$curl -k "https://192.168.1.1/api/?type=config&action=delete&xpath=/config/devices/entry\[@name='localhost.localdomain'\]/vsys/entry\[@name='vsys1'\]/rulebase/security/rules/entry\[@name='deny-rule1'\]/source-user/member\[text()='acme\bob'\]&key=<API-KEY>"

<response status="success" code="20"><msg>command succeeded</msg>

If you want to edit a member value, then you need to reference the original member value with member\[text()='<value>'\] and then use the element parameter for the modified member text value: element=<xml code>

for example using curl

$ curl -k "https://192.168.1.1/api/?type=config&action=edit&xpath=/config/devices/entry\[@name='localhost.localdomain'\]/vsys/entry\[@name='vsys1'\]/rulebase/security/rules/entry\[@name='deny-rule1'\]/source-user/member\[text()='acme\bob'\]&element=<member>acme\calvin</member>&key=<API-KEY>"

<response status="success" code="20"><msg>command succeeded</msg></response>

Below are all xpath expressions you can use when accessing the Palo Alto Networks api.

Examples:

/source-user/member[position()<4]
Selects the first three member elements that are children of the source-user element

$ curl -k "https://192.168.1.1/api/?type=config&action=get&xpath=/\[@name='localhost.localdomain'\]/vsys/entry\[@name='vsys1'\]/rulebase/security/rules/entry\[@name='rule1'\]/source-user/member\[position()<4\]&key=<API-KEY>"
<response status="success" code="19"><result total-count="3" count="3">
<member admin="admin" time="2015/10/21 13:42:11">acme\amy</member>
<member admin="admin" time="2015/10/21 13:42:11">acme\bob</member>
<member admin="admin" time="2015/10/21 13:42:11">acme\calvin</member>

/source-user/member[2] Selects the second member element that is the child of the source-user element

$ curl -k "https://192.168.1.1/api/?type=config&action=get&xpath=/\[@name='localhost.localdomain'\]/vsys/entry\[@name='vsys1'\]/rulebase/security/rules/entry\[@name='rule1'\]/source-user/member\[2\]&key=<API-KEY>" <response status="success" code="19"><result total-count="1" count="1">
<member admin="admin" time="2015/10/21 13:42:11">acme\bob</member>

Sunday, October 18, 2015

Using cURL to access the RESTful API of a Palo Alto Networks Firewall

There may be a situation where you would need to access the API of a Palo Alto Networks firewall. Some IT administrators may be more comfortable using cURL to access an API than a scripting language like PYTHON. Here's a few examples of how to perform some tasks using cURL.

The first thing you need to do is get an API key. How do you get an API Key? You query the firewall itself.

?type=keygen is the parameter to use.

curl -k "https://<firewall ip>/api/?type=keygen&user=<username>&password=<password>"

Make sure you wrap your url with Quotes. I tried this before and would get an error 400 response.

You will get a response in xml. The value for key is the API key.

curl -k "https://192.168.1.1/api/?type=keygen&user=admin&password=admin"

<key>

1234....

</key>

</result>

</response>

Then you can query the firewall using the API key instead of passing a username and password.

curl -k "https://<firewall ip>/api/?type=<api type>&<parameters>&key=<api-key>"

To get a list of api types and what commands to pass, you can use the REST API browser on the firewall. Just bring up a browser and authenticate against your firewall. After that you can add the path /api to the url. https://<firewall ip>/api

Here you have the different api "types" you can query the firewall. Each of these types have some caveats as to the way you send your

For our first example, lets execute an operational command. From the cli the equivalent command would be "show system info".

If you drill down the Operational Commands to show system and info you will see the REST API URL at the bottom of the page. This is the url path you would copy to your curl command.

user@ubuntu-vm:~/API/curl$ curl -k "https://192.168.1.1/api/?

type=op&cmd=<show><system><info></info></system></show>&key=<API-KEY>"

<ip-address>192.168.1.1</ip-address>

<default-gateway>192.168.1.254</default-gateway>

... TRUNCATED...

Notice that the command is in XML format. The firewall takes input and outputs in XML.

This is important to note as you'll need to know the XPATH for most queries.

The following is to query the hostname with. You will need to add "action=get" as one of the parameters to read the hostname.

user@ubuntu-vm:~/API/curl$ curl -k "https://192.168.1.1/api/?type=config&action=get&xpath=/config/devices/entry\[@name='localhost.localdomain'\]/deviceconfig/system/hostname&key=<API-KEY>"

Notice that I had to put the literal '\[' for special characters like the square brackets. cURL would complain if I didn't have this.

curl: (3) [globbing] bad range specification in column 78

Now that we did a get. Let's try a set. When we do a set command, we need an extra parameter "element" that will contain the xml of the new content.

user@ubuntu-vm:~/API/curl$ curl -k "https://192.168.1.1/api/?type=config&action=set&xpath=/config/devices/entry\[@name='localhost.localdomain'\]/deviceconfig/system&element=<hostname>test</hostname>&key=<API-KEY>"

<response status="success" code="20"><msg>command succeeded</msg></response>

Doing a get command again shows us the hostname is changed.

user@ubuntu-vm:~/API/curl$curl -k "https://192.168.1.1/api/?type=config&action=get&xpath=/config/devices/entry\[@name='localhost.localdomain'\]/deviceconfig/system/hostname&key=<API-KEY>"

Next you need to commit the set command.

user@ubuntu-vm:~/API/curl$curl -k "https://192.168.1.1/api/?type=commit&cmd=<commit></commit>&key=<API-KEY>"

<response status="success" code="19"><result><msg><line>Commit job enqueued with jobid 699</line></msg><job>69

The response code will return a job id.

The last API type I'll cover are logs. Log retrieval is an asynchronous process so you need to query the particular log type first which will create a job id in the response and then query the FW again with the job id. Here’s an example using a threat log.

The first one is to generate the job id.

curl -k "https://<FW-IP>/api/?type=log&log-type=threat&key=<API-KEY>

The second will show you the results after the job task is finished.

curl -k "https://<FW-IP>/api/?type=log&action=get&jobid=<JOB-ID>&key=<API-KEY>

user@ubuntu-vm:~$ curl -k "https://192.168.1.1/api/?type=log&log-type=threat&key=<API-KEY>"

<response status="success" code="19"><result><msg><line>query job enqueued with jobid 3605</line></msg><job>3605</job></result></response>

user@ubuntu-vm:~$ curl -k "https://192.168.1.1/api/?type=log&action=get&jobid=3605&key=<API-KEY>"

<job>

<cached-logs>33</cached-logs>

</job>

<log>

... TRUNCATED ...

</log>

<meta>

<hostname>localhost.localdomain</hostname>

<vsys>

<display-name>vsys1</display-name>

</entry>

</vsys>

</entry>

</devices>

</meta>

Wednesday, October 14, 2015

Upload files to BOX over FTPS

I wanted to upload some files from a linux machine to my BOX account. However I wanted to do this all via the command line. There are scenarios where all I have access to is the cli and I didn't want to transfer the files to another machine that had a web browser. Besides there are times when using the cli is more efficient.

First I tried ftp. So I found on the internets that BOX supports ftp.

However since I have an account with SSO I need to create an external password. You have to do this within the BOX portal of your account settings.

I recommend a randomly generated long password (> 12 Chars) for this and use a password manager to keep track of it. Don't use your SSO password. Don't be lazy! There are probably hackers out there who are already trying to bruteforce attack accounts using this attack vector. If they get this password, they get your SSO password.

You probably want to click the Log out of all Box applications using this account check box. This will prevent this external password from being used on other types of apps like mobile devices.

Side note: I like the fact that BOX keeps track of your login activity with the application you are trying to use, ie FTP server, Windows Chrome, etc and the geo location of the login.

user@ubuntu:~$ ftp ftp.box.com
Connected to ftp.box.com.
220 Service ready for new user.
Name (ftp.box.com:user): <username>
331 User name okay, need password for <username>
Password:
530 Box: Company.com does not allow regular FTP; use FTPS instead. (Both "explicit" and "implicit" FTPS are supported.)

Doh. Fail.

Duh! We need a secure connection.

Ok. I haven't used FTPS before. I've used SFTP, but not this other method.

FTPS uses SSL.

So after consulting with the internets I found I can use lftp.

user@ubuntu:~$ sudo apt-get install lftp
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
lftp

My second try:

user@ubuntu:~$ lftp -u user@company.com ftp.box.com
Password:
lftp user@company.com@ftp.box.com:~> ls
drwx------ 1 owner group 0 Sep 24 17:41 PoCs
drwx------ 1 owner group 0 Aug 27 10:18 Demos
drwx------ 1 owner group 0 Aug 27 10:19 Docs

interesting to see the shell.

lftp user@company.com@ftp.box.com:/> cd PoCs
lftp user@company.com@ftp.box.com:/PoCs> put file.zip
47408 bytes transferred
lftp user@company.com@ftp.box.com:/PoCs> exit

Success!

I can see that lftp has a parameter for sending a command. Which means I can automate this process using a shell script.

lftp -e 'unix command' -u <username>,<password> ftp.box.com

user@company.com:~$ lftp -e 'ls; bye' -u user,password ftp.box.com
lftp user@company.com@ftp.box.com:~> ls
drwx------ 1 owner group 0 Sep 24 17:41 PoCs
drwx------ 1 owner group 0 Aug 27 10:18 Demos
drwx------ 1 owner group 0 Aug 27 10:19 Docs
-rw------- 1 owner group 47408 Sep 27 10:39 file.zip
user@company.com:~$

Tuesday, October 13, 2015

Checking file hashes against Palo Alto Network's Wildfire to find their verdicts

I had a list of files I needed to check to see if they were malware. There are a few ways to approach this. The first is to upload each file manually one at a time onto the wild fire portal and have wildfire check to see if they're either benign or malicious.

Or automate this to into the simplest and most efficient way. I chose the latter.

The first thing I did was do a hash checksum of all the files. I chose md5 hashes for this example, but you can do sha256 as that is supported by wildfire as well.

On a linux machine I issued the following command on some files.

user@ubuntu-vm:~/$ md5sum *
1842b2365bf67121462d0cc026fb9300 Test.pdf
cd75d3e263ff0d1d13aad24cdb9f2593 flashplayer19_ha_install.exe

Now that I have those hashes I put them into a text file. I also added a few hashes I knew were malware just verify my script.

Next I used pan-python to create my script that would query Wild Fire. Now there are two ways to approach this. One is to send each hash one at a time and get the results back for each query. If you send each request one at a time, you'll use up your daily allotment of API queries
the more hashes you have. The other method is to send a bulk query. This would reduce the number of queries you have, but you only get to do a bulk of 500 hashes at a time.

Here's the code that I used.


import csv
import pan.wfapi


apikey = 'YOUR_API_KEY'

# loop through all the hashes and put this into a List

with open("sample-hashes.txt", "r") as ins:
  rows = csv.reader(ins)
  L= []
  for hash in rows:
    print hash[0]
    L.append(hash[0])

#make an API query to WF and do a bulk check
  WF = pan.wfapi.PanWFapi(api_key = apikey)
  WF.verdicts(hashes=L)
  print('hashes %s submitted' % L)

#print the xml response
  xml_response = WF.response_body
  print xml_response

I imported the python csv library to read my file that contained all my hashes. The reason I used csv was because I orginally recieved the hashes as a csv file with each filename associated with the hash value as you can see above there are filenames after each hash. So in this demo, I cropped out all the other fields and only used the hashes. If the file was delimited, I could read only the row with the hash value.

Then I imported the pan-python library that would allow me to run the query.

The loop is used so I can append to a list with all the hashes I found in the text file.
Then I would make a query to wildfire using the API key.

The api key can be found in your wildfire account on the wildfire portal.


$ python bulk-wf.py
1842b2365bf67121462d0cc026fb9300
bf1373d10842e96c85bf73a97ddec699
36944ab907576c10d217911ee6acc3c9
4b20dd78c13433f4ec47853bfecddc61
7309a9b75819dfd1496391fc75016d90
hashes ['1842b2365bf67121462d0cc026fb9300', 'ad9e1502d3fd341608fa4730a1609f8d', 'bf1373d10842e96c85bf73a97ddec699', '36944ab907576c10d217911ee6acc3c9', '4b20dd78c13433f4ec47853bfecddc61', '7309a9b75819dfd1496391fc75016d90'] submitted

<wildfire>
    <get-verdict-info>
        <sha256>6864e1fa5e0145c2f1ce6f403a3554fe9576287929b0e9e4e5fadb50915bb65e</sha256>
        <verdict>1</verdict>
        <md5>36944ab907576c10d217911ee6acc3c9</md5>
    </get-verdict-info>
    <get-verdict-info>
        <sha256>93cbeff02c16e7e09e41aa94ee37a3dae51849f14d335485fc936297b400ce04</sha256>
        <verdict>1</verdict>
        <md5>4b20dd78c13433f4ec47853bfecddc61</md5>
    </get-verdict-info>
    <get-verdict-info>
        <sha256>5683cc43393bfd01b5533a3c710c39d62387cbd5bdf9588f8b3c1dc13933473c</sha256>
        <verdict>1</verdict>
        <md5>7309a9b75819dfd1496391fc75016d90</md5>
    </get-verdict-info>
    <get-verdict-info>
        <sha256>fd54decc2b89c9ca00f4e6de39a1a9d677bd1c5a8ceb6d6b5b25eedc8d332e28</sha256>
        <verdict>0</verdict>
        <md5>1842b2365bf67121462d0cc026fb9300</md5>
    </get-verdict-info>
<\wildfire>

A verdict of 1 means the file is malicious, while a verdict of 0 means the file is benign.

If the verdict was -102 that means that the file is unkown and I would have to upload it to wildfire to have it further examine.
I could make some more logic in my script to print the actual verdict, but it would require me to manipulate the response code. I would need something like the library xmltodict so I could specifically retrieve and do a evaluation only the verdicts.

Now this is good to cut down the amount of traffic that would be needed to be sent across a network and evaluate known good or known bad files in a very fast way. Then only when you need to upload the unkowns you can then limit that number of files to a smaller group.

Friday, November 21, 2014

Automating a Palo Alto Networks Firewall using Python

In my last post I used a python script to extract information from a Palo Alto Networks Firewall using pan-python. In this post I'll illustrate how to configure a firewall using the API.

Palo Alto Networks uses XML as the data structure for it's representation of the configuration file. Automating a firewall takes three steps.

Creating the xml file
Pushing the xml file to the firewall
Committing the candidate configuration

I created a sample xml configuration file that would add an IP address to a sub-interface with a vlan tag of 1 on interface ethernet1/3.

<entry name="ethernet1/3">
<layer3>
    <units>
      <entry name="ethernet1/3.1">
        <tag>1</tag>
        <ip>
          <entry name="30.4.1.2/24"/>
        </ip>
      </entry>
      </entry>
    </units>
</layer3>
</entry>

This is going to be placed into a text file called sub-int.xml which I'll use later.

In my script I read from the file and and place it into a variable called data. I strip the newlines so that I don't have separate each line into an array.

Last you need to commit the config. One thing about the api is that the commit call needs an xml element <commit/>

When I tried it without a cmd ie. xapi.commit(), I got the following error.

pan.xapi.PanXapiError: Missing value for parameter "cmd".

This was confusing at first, until I spoke with a Palo Alto networks Solutions Architect about it and he explained that you need to tell it which type of commit you want. There are a few options such as commit, commit partial and commit full. I think there should be a default setting. Commit without any input should mean a normal commit. Maybe I'll modify a git cloned repository.

script
-----------
import pan.xapi
from cred import get_pan_credentials
credentials = get_pan_credentials()

print credentials
xapi = pan.xapi.PanXapi(**credentials)

xpath = "/config/devices/entry/network/interface/ethernet"

#open xml file and read it into a variable called data.
with open ("sub-int.xml", "r") as myfile:
    data=myfile.read().replace('\n', '')

#set the config using the above xpath
xapi.set(xpath,element=data)

#commit the config. Make sure to add the xml command.
xapi.commit('<commit/>')

-------------

Here's the resulting screen cap:

Monday, November 3, 2014

Accessing a PAN firewall through an api with python

I found out that someone (Kevin Steves) made a python module to access Palo Alto Networks API. You can clone the repository from github.

git clone https://github.com/kevinsteves/pan-python.git

The module is very thorough and the documentation is pretty good. It was simple to test out and I was able to get it working in a few hours.

From what I can tell the module makes a REST call and exports the results into XML. This makes automation a lot easier as the data structure can be easily parsed.

My first script looks like this:

import pan.xapi

from cred import get_pan_credentials

credentials = get_pan_credentials()

print credentials

xapi = pan.xapi.PanXapi()

xapi.op(cmd='show system info', cmd_xml=True)

print xapi.xml_result()

----------

It makes a call to my credentials file

$ more cred.py

def get_pan_credentials():

 cred =  {}

 cred['api_username'] = "admin"

 cred['api_password'] = "admin"

 cred['hostname'] = "192.168.1.1"

 return cred

------------

The script in action.

$ python test1.py 

{'api_key': 'LUFRPT14MW5xOEo1R09KVlBZNnpnemh0VHRBOWl6TGM9bXcwM3JHUGVhRlNiY0dCR0srNERUQT09', 'hostname': '192.168.1.1', 'api_password': 'admin', 'api_username': 'admin'}

<system><hostname>PA-7050</hostname><ip-address>192.168.1.1</ip-address><netmask>255.255.255.0</netmask><default-gateway>192.168.1.254</default-gateway><ipv6-address>unknown</ipv6-address><ipv6-link-local-address>fe80::290:fbff:fe4d:175c/64</ipv6-link-local-address><ipv6-default-gateway /><mac-address>00:90:fb:4d:17:5c</mac-address><time>Mon Nov  3 23:28:08 2014</time>

<uptime>0 days, 11:39:35</uptime>

<devicename>PA-7050</devicename>

<family>7000</family><model>PA-7050</model><serial>015128030274</serial><sw-version>6.0.6</sw-version>

<global-protect-client-package-version>0.0.0</global-protect-client-package-version>

<app-version>466-2435</app-version>

<app-release-date>2014/10/28  20:28:09</app-release-date>

<av-version>1408-1880</av-version>

<av-release-date>2014/11/03  04:00:02</av-release-date>

<threat-version>466-2435</threat-version>

<threat-release-date>2014/10/28  20:28:09</threat-release-date>

<wildfire-version>0</wildfire-version>

<wildfire-release-date>unknown</wildfire-release-date>

<url-filtering-version>0000.00.00.000</url-filtering-version>

<global-protect-datafile-version>0</global-protect-datafile-version>

<global-protect-datafile-release-date>unknown</global-protect-datafile-release-date><logdb-version>6.0.6</logdb-version>

<platform-family>7000</platform-family>

<logger_mode>False</logger_mode>

<vpn-disable-mode>off</vpn-disable-mode>

<operational-mode>normal</operational-mode>

<multi-vsys>off</multi-vsys>

</system>

Sunday, October 12, 2014

Not all APIs are created equally.

Automation is starting to become the new catch phrase in the networking Industry. It seems like 2014 is the year that marketing groups from different vendors have been touting APIs on their products. Skeptical networking engineers however have claimed that an API does not mean that their jobs are getting easier. In fact it’s been making their jobs a little harder.

Before a networking engineer could strictly focus on pure networking. But now, network engineers are increasingly required to know more and more on how to code or at least know how to read code.

Just because you have an API doesn’t mean all the devices can and will play nice with each other.

We can see this by looking at three different platforms.

OpenStack Contrail Junos

Now let’s look at their API for data retrieval/configuration

REST REST Netconf

Ok now you have two platforms that use one type of API and a third platform that uses a different API.

Let's look at the resulting Data Structure response

JSON JSON XML

Again we have two platforms that have the same Data Structure and a third with a different Data Structure.

You might say, ok, at least two of these platforms have the same API and with the same data structure things should be good for both of them right? Actually as they say, the devil is in the details.

I can illustrate this just by looking at a simple IPv4 subnet.

On Openstack the data abstracted looks like this

{ "networks": [ { "contrail:subnet_ipam": [ { "subnet_cidr": "12.1.1.0/24", } ] } ] }

On Contrail it looks like this

{"virtual-network":{ "network_ipam_refs":[ { "attr": { "ipam_subnets": [ { "subnet": { "ip_prefix": "12.1.1.0", "ip_prefix_len": 24 }, } ] }, } ], } }

You can see that one platform combines the subnet with the mask while the other one separates it. For a DevOps engineers and Network Engineers this is annoying. It’s like having to learn different Network Operating systems. The goal of an API should be to allow a simplified abstraction layer.

APIs need to be standardized. Openflow is a good attempt at this. Openflow requires the underlay to have a common protocol in order to allow a controller to programmatically configure them. The networking industry has done a great job at standardizing protocols but a sorry job at creating a common API standard. Maybe the IETF needs to jump in on this. A standardized API could ultimately make our jobs that much more easier.

Friday, October 10, 2014

How to use cURL to spin up an OpenStack VM

I know this is a little long but there are dependencies on how this works. Hopefully you can understand it. The last cURL command is the piece that puts it all together and spins up a VM. The last command will spin up a instance called "instance1", use a flavor and an cirros image and place it in the "VN2" network

LAPTOP:SCRIPTS$cat openstack-create.sh
TOKEN=""

#Do a POST to GET the AUTH TOKEN
TOKEN=$(curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "admin",
          "domain": { "id": "default" },
          "password": "password",
          "tenantName": "admin"
        }
      }
    },
    "scope": {
      "project": {
        "name": "demo",
        "domain": { "id": "default" }
      }
    }
}
}' \
http://192.168.1.4:5000/v3/auth/tokens | grep X-Subject-Token: | sed -e "s/X-Subject-Token: //")

#STORE TOKEN
header='X-Auth-Token: '$TOKEN
#echo "$header"

#NOW USE the AUTH TOKEN TO CHECK TENANTS
curl -X GET http://192.168.1.4:5000/v2.0/tenants/ -H "$header" | python -m json.tool

#CHECK IMAGES FOR TENANT/PROJECT CALLED DEMO
# TENANT ID FOR DEMO = 6d0ebb466f01454e897e518289b3b785
echo "IMAGES"
curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/images | python -m json.tool

# CHECK FLAVORS
echo "FLAVORS"

curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/flavors | python -m json.tool

# CHECK SERVERS
echo "SERVERS"

curl -H "X-Auth-Project-Id: demo" -H "$header" -X GET http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/servers | python -m json.tool

# CHECK NETWORK ID - THIS IS NEUTRON AND HAS IT'S OWN PORT NUMBER 9696
echo "NETWORKS"

curl -H "Content-Type: application/json" -H "$header" -X GET http://192.168.1.4:9696/v2.0/networks | python -m json.tool

#CREATE VM
echo "CREATE VM"

curl -i -H "Accept: application/json" -H "Content-Type: application/json" -H "X-Auth-Project-Id: demo" -H "$header" -X POST http://192.168.1.4:8774/v2/6d0ebb466f01454e897e518289b3b785/servers -d '{"server": {"name": "instance1", "imageRef": "d5f30886-e8ce-4e89-9709-c49a40717b5b", "flavorRef": "2", "max_count": 1, "min_count": 1,"key_name": "xyz", "networks": [{"network": "bd912f99-c345-4f3e-8026-88e1dde42255", "uuid": "bd912f99-c345-4f3e-8026-88e1dde42255"}]}}'

-----------------------

I don’t execute the script and print it out as it would not look too pretty.

The most annoying part of this is all the curly brackets I had to dig around for the formatting of "networks" part as this was the hardest part.

I found out the formatting was like this:

networks: [{"port": String, "fixed_ip": String, "uuid": String, "network": String}, {"port": String, "fixed_ip": String, "uuid": String, "network": String}, ...]

Here's the api that I found useful:

http://developer.openstack.org/api-ref-compute-v2.html

Below is how it would look like if you executed the commands directly on the compute node. Note how image-list has the id for "imageRef"

Tuesday, September 30, 2014

Using python to connect remotely to Openstack

In order to beef up my DevOps skills I decided to see if I could connect through a REST API using python. I found out that there are some python modules built specifically for this.

One such module is called novaclient. You would need pip on your machine to do this.

sudo easy_install pip

Then you can use pip to install the nova client.

sudo pip install python-novaclient

If you want to look at the details of the python module it can be found here:

https://github.com/openstack/python-novaclient

This is where I look more into the code and found out it’s capabilities

You can pass parameters to the “client” function.

def __init__(self, username=None, api_key=None, project_id=None,

auth_url=None, insecure=False, timeout=None,

proxy_tenant_id=None, proxy_token=None, region_name=None,

endpoint_type='publicURL', extensions=None,

service_type='compute', service_name=None,

volume_service_name=None, timings=False, bypass_url=None,

os_cache=False, no_cache=True, http_log_debug=False,

auth_system='keystone', auth_plugin=None, auth_token=None,

cacert=None, tenant_id=None, user_id=None,

connection_pool=False, session=None, auth=None,

completion_cache=None):

First I built a login function that will pass these parameters which I can call from my main script.

login.py

-----------

def get_nova_credentials():

 cred =  {}

 cred['username'] = "admin"

 cred['api_key'] = “password”

 cred['auth_url'] = "http://<openstack-ip>:5000/v2.0"

 cred['project_id'] = "demo"

 cred['service_type'] = "compute"

 return cred

Now in my main script I can import the client and my credentials

server-list.py

-----------------

#!/usr/bin/env python

import novaclient

from novaclient.v1_1 import client

from login import get_nova_credentials

credentials = get_nova_credentials()

#Pass credentials to the client function.

nova = client.Client(**credentials)

#grab the list of servers and print out the id, names and status

vms = nova.servers.list(detailed=True)

for vm in vms:

  print vm.id, vm.name, vm.status

—————

script in action.

laptop$ python server-list.py 

f5801333-5d81-496c-b257-e589ca36e944 Cirros-VM2 ACTIVE

098270e0-e5fb-4ea6-a1f1-2dfca11a409d Cirros-VM1 ACTIVE

So what's the big deal? Why do this when you can use the Horizon web ui?

Now that I have a basic understanding of this module, I can start automating things such as building VMs, virtual networks, etc in a scaled and precise manner.

Sunday, September 28, 2014

Exploring the REST API on contrail.

I noticed on github that there was a tutorial on how to access the REST API on Juniper Contrail.

https://juniper.github.io/contrail-vnc/api-doc/html/tutorial_with_rest.html#

The easiest way to access this is by using cURL. Contrail uses tcp port 8082 for accessing it's REST API.

The url http:/contrail-ip/virtual-networks prints out a list of configured virtual networks on Contrail.

$ curl -X GET -H "Content-Type: application/json; charset=UTF-8" http://172.16.1.4:8082/virtual-networks | python -mjson.tool

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  1229  100  1229    0     0   2129      0 --:--:-- --:--:-- --:--:--  2129

{

    "virtual-networks": [

{

            "fq_name": [

                "default-domain",

                "default-project",

                "__link_local__"

],

            "href": "http://172.16.1.4:8082/virtual-network/4092df7b-997a-4ee7-a5cc-46d5db1187d4",

            "uuid": "4092df7b-997a-4ee7-a5cc-46d5db1187d4"

},

{

            "fq_name": [

                "default-domain",

                "default-project",

                "default-virtual-network"

],

            "href": "http://172.16.1.4:8082/virtual-network/34579f9a-064e-4048-96a7-a30355c54e44",

            "uuid": "34579f9a-064e-4048-96a7-a30355c54e44"

},

{

            "fq_name": [

                "default-domain",

                "default-project",

                "ip-fabric"

],

            "href": "http://172.16.100.104:8082/virtual-network/db7d1afe-bcaa-456b-b33a-9a36f6d176fe",

            "uuid": "db7d1afe-bcaa-456b-b33a-9a36f6d176fe"

},

{

            "fq_name": [

                "default-domain",

                "demo",

                "Network1"

],

            "href": "http://172.16.1.4:8082/virtual-network/380c0f4e-34b6-4901-98b2-61a8ed7afd7d",

            "uuid": "380c0f4e-34b6-4901-98b2-61a8ed7afd7d"

},

{

            "fq_name": [

                "default-domain",

                "demo",

                "Network2"

],

            "href": "http://172.16.100.104:8082/virtual-network/ffe03354-aaa2-4305-b615-654e14111134",

            "uuid": "ffe03354-aaa2-4305-b615-654e14111134"

},

{

            "fq_name": [

                "default-domain",

                "demo",

                "Network3"

],

            "href": "http://172.16.1.4:8082/virtual-network/93317422-eea4-4b77-88cb-5aaac3bb58b6",

            "uuid": "93317422-eea4-4b77-88cb-5aaac3bb58b6"

}

]

}

However cURL, at least for me, isn’t capable of abstracting the data structures programatically. It's more like screen scraping. So I dug around the internet and noticed that python has a curl module.

http://pycurl.sourceforge.net/doc/quickstart.html#

Now I can use python to execute cURL to pull data from a REST API. 

There are a few parts to this script.

The first part is a function to issue the cURL command.

The second part is to abstract the route-target from a virtual network created in the Contrail controller.

This could then be used later on with another script to create a template configuration and program the underlay gateway router with a VRF.

In this script I cheated a little by grabbing a specific virtual-network’s url address from the previous curl command. Then parsed the data to grab the information I was looking for.

—————————————

import pycurl

import StringIO

import json

#Function to issue cURL command

def get_url(WEB):

  buf = StringIO.StringIO()

  c = pycurl.Curl()

  c.setopt(c.URL, WEB)

  c.setopt(c.WRITEFUNCTION, buf.write)

  c.perform()

  body = buf.getvalue()

  network = json.loads(body)

  buf.close()

  return network

#URL of virtual network on contrail

SITE = 'http://172.16.1.4:8082/virtual-network/380c0f4e-34b6-4901-98b2-61a8ed7afd7d'

objects = get_url(SITE)

#pretty print the json results

print json.dumps(objects, sort_keys=True, indent=4)

#This part is to grab the path of the Virtual Network name and RT from contrail

print "Name: ", objects['virtual-network']['name'], " RT: ", objects['virtual-network']['route_target_list']['route_target'][0]

Script in action:

——————

$ python contrail.py 

{

    "virtual-network": {

        "fq_name": [

            "default-domain", 

            "demo", 

            "Network1"

],

        "href": "http://172.16.1.4:8082/virtual-network/380c0f4e-34b6-4901-98b2-61a8ed7afd7d", 

        "id_perms": {

            "created": "2014-09-19T19:19:11.650288", 

            "description": null, 

            "enable": true, 

            "last_modified": "2014-09-27T05:22:30.453524", 

            "permissions": {

                "group": "cloud-admin-group", 

                "group_access": 7, 

                "other_access": 7, 

                "owner": "cloud-admin", 

                "owner_access": 7

},

            "uuid": {

                "uuid_lslong": 11002964217786203517, 

                "uuid_mslong": 4038619794410719489

}

},

        "instance_ip_back_refs": [

{

                "attr": null, 

                "href": "http://172.16.1.4:8082/instance-ip/17b19dc0-b177-4df7-955b-57b8a87caf28", 

                "to": [

                    "17b19dc0-b177-4df7-955b-57b8a87caf28"

],

                "uuid": "17b19dc0-b177-4df7-955b-57b8a87caf28"

},

{

                "attr": null, 

                "href": "http://172.16.1.4:8082/instance-ip/d525ddce-3542-4986-b8d1-56f3831d8678", 

                "to": [

                    "d525ddce-3542-4986-b8d1-56f3831d8678"

],

                "uuid": "d525ddce-3542-4986-b8d1-56f3831d8678"

}

],

        "is_shared": false, 

        "name": "Network1", 

        "network_ipam_refs": [

{

                "attr": {

                    "ipam_subnets": [

{

                            "default_gateway": "100.1.1.254", 

                            "subnet": {

                                "gw": "100.1.1.254", 

                                "ip_prefix": "100.1.1.0", 

                                "ip_prefix_len": 24

}

}

]

},

                "href": "http://172.16.1.4:8082/network-ipam/1f24fa35-b7bf-4d0f-8185-746f58e234c9", 

                "to": [

                    "default-domain", 

                    "demo", 

                    "Network1-ipam"

],

                "uuid": "1f24fa35-b7bf-4d0f-8185-746f58e234c9"

}

],

        "network_policy_refs": [

{

                "attr": {

                    "sequence": {

                        "major": 0, 

                        "minor": 0

},

                    "timer": null

},

                "href": "http://172.16.1.4:8082/network-policy/0a1c1776-a323-4c00-959a-aada50b91be8", 

                "to": [

                    "default-domain", 

                    "demo", 

                    "net1<->net2"

],

                "uuid": "0a1c1776-a323-4c00-959a-aada50b91be8"

}

],

        "parent_href": "http://172.16.1.4:8082/project/3af66afe-3284-40cc-8b04-85c69af512c7", 

        "parent_type": "project", 

        "parent_uuid": "3af66afe-3284-40cc-8b04-85c69af512c7", 

        "route_target_list": {

            "route_target": [

                "target:64512:100"

]

},

        "router_external": false, 

        "routing_instances": [

{

                "href": "http://172.16.1.4:8082/routing-instance/06165761-3b55-417d-acf0-0ad27a9010d0", 

                "to": [

                    "default-domain", 

                    "demo", 

                    "Network1", 

                    "Network1"

],

                "uuid": "06165761-3b55-417d-acf0-0ad27a9010d0"

}

],

        "uuid": "380c0f4e-34b6-4901-98b2-61a8ed7afd7d", 

        "virtual_machine_interface_back_refs": [

{

                "attr": null, 

                "href": "http://172.16.1.4:8082/virtual-machine-interface/2aefccdc-bf34-4d3b-bd37-41f716274883", 

                "to": [

                    "e432ad9a-8f6e-4f7c-813b-18ada76bfd64", 

                    "2aefccdc-bf34-4d3b-bd37-41f716274883"

],

                "uuid": "2aefccdc-bf34-4d3b-bd37-41f716274883"

},

{

                "attr": null, 

                "href": "http://172.16.1.4:8082/virtual-machine-interface/f20bb145-49cd-43bf-a6e5-9d9c50794244", 

                "to": [

                    "2e775d6f-8043-41df-9295-d1b8d8f705a2", 

                    "f20bb145-49cd-43bf-a6e5-9d9c50794244"

],

                "uuid": "f20bb145-49cd-43bf-a6e5-9d9c50794244"

}

],

        "virtual_network_properties": {

            "extend_to_external_routers": null, 

            "forwarding_mode": "l2_l3", 

            "network_id": 4, 

            "vxlan_network_identifier": null

}

}

}

Name:  Network1  RT:  target:64512:100

As you can see I was able to pull the "name" of the virtual network and the route-target of the virtual network. Later I can then create a script template to build the VRF on the Router Gateway like a MX. That's one step closer to network automation.