Showing posts with label neutron. Show all posts
Showing posts with label neutron. Show all posts

Wednesday, September 24, 2014

How Contrail communicates with the underlay

Contrail typically consists of a cluster of nodes. The three main nodes are the config, control and compute. The config node is where the Openstack Horizon and Contrail Controller exists. The Control node is used to form a MP-BGP session to a gateway router. The Compute node hosts all the VMs and virtual networks.

You can think of Contrail as a PE router as pretty much this is what a gateway router perceives the other end of the connection. Contrail uses a vRouter and when you configure virtual networks you have the ability to add a route-target to that virtual network. On the Gateway router you would create VRFs to associate with the corresponding virtual networks and prefixes can be exchanged. Data plane traffic will traverse through an MPLS tunnel between Contrail and the Gateway router. It's at the gateway router where you would "leak" the received Contrail virtual network into the main routing instance of the gateway router.

Here I use an Juniper MX as the gateway router. When I first setup contrail I used the testbed.py script to add the mx gateway router.

It's called ext_router = [ip address]

Then in contrail webui I should see the BGP session. You can however add this post contrail installation.


On the MX, I configure an iBGP session to connect with the Contrail control node.

user@router# show protocols
mpls {
    interface all;
}
bgp {
    group IBGP-CONTRAIL {
        type internal;
        local-address 192.168.10.11;
        family inet-vpn {
            unicast;
        }
        neighbor 192.168.10.2;
    }
}

Then in Contrail config node I create a virtual network and add a route target.




I create a corresponding VRF on the MX with the route target.

user@router# show routing-instances
VRF1 {
    instance-type vrf;
    interface lt-3/0/0.3;
    route-distinguisher 1.1.1.1:101;
    vrf-target target:64512:101;
    routing-options {
        static {
            route 0.0.0.0/0 next-hop 192.168.12.1;
        }
    }


I check to see the BGP session established.

user@router# run show bgp summary                                     
Groups: 1 Peers: 1 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
bgp.l3vpn.0         
                       8          8          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
192.168.10.2          64512       5724       6292       0       3 1d 23:08:56 Establ
  bgp.l3vpn.0: 8/8/8/0
  VRF1.inet.0: 3/3/3/0

The Virtual Network IP addresses for the VMs will be sent.

user@router# run show route receive-protocol bgp 192.168.10.2 

VRF1.inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
  Prefix          Nexthop           MED     Lclpref    AS path
* 11.1.1.1/32             192.168.10.3                 100        ?
* 11.1.1.5/32             192.168.10.3                 100        ?
* 11.1.1.7/32             192.168.10.3                 200        ?

bgp.l3vpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
  Prefix          Nexthop           MED     Lclpref    AS path
  192.168.10.3:7:11.1.1.1/32                   
*                         192.168.10.3                 100        ?
  192.168.10.3:7:11.1.1.5/32                   
*                         192.168.10.3                 100        ?
  192.168.10.3:7:11.1.1.7/32                   
*                         192.168.10.3                 200        ?

mpls.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0                  *[MPLS/0] 1w1d 04:10:08, metric 1
                      Receive
1                  *[MPLS/0] 1w1d 04:10:08, metric 1
                      Receive
2                  *[MPLS/0] 1w1d 04:10:08, metric 1
                      Receive
13                 *[MPLS/0] 1w1d 04:10:08, metric 1
                      Receive
299904             *[VPN/170] 1d 23:14:00
                    > to 192.168.11.1 via lt-3/0/0.1, Pop     
299936             *[VPN/170] 1d 12:47:38
                      receive table VRF1.inet.0, Pop     
299952             *[VPN/170] 1d 12:47:38
                    > to 192.168.12.1 via lt-3/0/0.3, Pop     

bgp.l3vpn.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.10.3:7:11.1.1.1/32               
                   *[BGP/170] 1d 12:46:01, localpref 100, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 24
192.168.10.3:7:11.1.1.5/32               
                   *[BGP/170] 1d 12:46:01, localpref 100, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 18
192.168.10.3:7:11.1.1.7/32               
                   *[BGP/170] 1d 12:26:14, localpref 200, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 28


Note the dynamic MPLS GRE tunnel is created. You will need to create one on the MX.

user@router# show chassis
fpc 3 {
    pic 0 {
        tunnel-services;
    }
}

user@router# show routing-options
static {
    route 0.0.0.0/0 next-hop 10.161.1.1;
}
autonomous-system 64512;
dynamic-tunnels {
    dynamic_overlay_tunnels {
        source-address 192.168.10.11;
        gre;
        destination-networks {
            192.168.10.0/24;
        }
    }
}



PoC-Demo.inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 1d 13:05:36
                    > to 192.168.12.1 via lt-3/0/0.3
11.1.1.1/32        *[BGP/170] 1d 13:03:59, localpref 100, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 24
11.1.1.5/32        *[BGP/170] 1d 13:03:59, localpref 100, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 18
11.1.1.7/32        *[BGP/170] 1d 12:44:12, localpref 200, from 192.168.10.2
                      AS path: ?, validation-state: unverified
                    > via gr-3/0/0.32770, Push 28
192.168.12.0/24    *[Direct/0] 1d 13:05:36
                    > via lt-3/0/0.3
192.168.12.2/32    *[Local/0] 1d 13:05:36
                      Local via lt-3/0/0.3

LT interfaces are created to allow the virtual network traffic to communicate between the VRF and the main routing instance. You could also use RIB groups and Policies to do the same thing.

    lt-3/0/0 {
        unit 2 {
            encapsulation ethernet;
            peer-unit 3;
            family inet {
                address 192.168.12.1/24;
            }
        }
        unit 3 {
            encapsulation ethernet;
            peer-unit 2;
            family inet {
                address 192.168.12.2/24;
            }
        }
    }

You then need to make sure the interface that is connecting to the Contrail network is using MPLS.


interfaces {

    ge-3/1/1 {
        unit 0 {
            family inet {
                address 192.168.10.11/24;
            }
            family mpls;
        }
    }
    lo0 {
        unit 0 {
            family inet {
                address 1.1.1.1/32;
            }
            family iso {
                address 49.0002.0010.0100.1001.00;
            }
        }
    }
}

One thing you should be aware of is the next-hop of the route advertised by contrail points to the IP address of the Compute Node and not the control node.
user@router# run show route 11.1.1.1/32 detail

VRF1.inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
11.1.1.1/32 (1 entry, 1 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 192.168.10.3:7  <<< Contrail's RD
                Next hop type: Indirect
                Address: 0x94f4a28
                Next-hop reference count: 3
                Source: 192.168.10.2
                Next hop type: Router, Next hop index: 660
                Next hop: via gr-3/0/0.32770, selected
                Label operation: Push 24
                Label TTL action: prop-ttl
                Session Id: 0xd
                Protocol next hop: 192.168.10.3  <<<< IP of compute node
                Push 24
                Indirect next hop: 0x9574410 1048574 INH Session ID: 0xe
                State: <Secondary Active Int Ext ProtectionCand>
                Local AS: 64512 Peer AS: 64512
                Age: 1d 13:54:24     Metric2: 0
                Validation State: unverified
                Task: BGP_64512.192.168.10.2+34735
                Announcement bits (1): 1-KRT
                AS path: ?
                Communities: target:64512:101   << RT from contrail
                Import Accepted
                VPN Label: 24
                Localpref: 100
                Router ID: 192.168.10.2         <<<< IP of control node
                Primary Routing Table bgp.l3vpn.0

Thursday, September 18, 2014

How to install Contrail on a single node Ubuntu system.

I wanted to install and test out a contrail set on a single node system.

I first started with a clean Ubuntu server. Contrail seems to have support on 12.04LTS version of code (12.04.03 to be precise). So I downloaded the amd64-iso file and did a fresh install on a system.

After it came up I did some package adds to get the machine prepped.

First I made sure I was root.
sudo su
passwd
I gave root a password. This is to be used later when Contrail tries to install packages using root.

This are the files I noticed while looking at different blogs on the OpenContrail.org website. I'm not sure if all these dependencies are needed. This is what worked for me. (I'm pretty sure git isn't needed here, but I use git for other projects so I did it anyways.)

apt-get install  -y git-core ant build-essential pkg-config linux-headers-3.2.0-35-virtual
apt-get install -y scons git python-lxml wget gcc patch make unzip flex bison g++ libssl-dev autoconf automake libtool pkg-config vim python-dev python-setuptools python-paramiko

apt-get update


Then I went onto the Juniper website and downloaded the software.




You DON'T have to install Openstack first to get this going.



This package has Contrail plus the Openstack Havana version built in.

I ftp or scp this file onto my ubuntu server.

I place it in the /tmp directory.

Next I install the package.

dpkg -i contrail-install-packages_1.05.1-234~havana_all.deb

The packages get placed in a contrail directory.

cd /opt/contrail/contrail_packages

Then I run the setup shell script.

./setup.sh

After this, you'll need to create or modify a testbed.py script. This tells contrail how install the compute, storage and control nodes. Since this is an all-in-one system, I going to clone the single box example and modify it.

cd /opt/contrail/utils/fabfile/testbeds/

cp testbed_singlebox_example.py testbed.py

Next I edit the file.

vi testbed.py


In order to execute this file, you need to back up a few directories (I know it's lame, the command doesn't seem to execute in the correct directory.)

cd /opt/contrail/utils

Next you will issue the fabric command

fab install_contrail

fab setup_all


After a few minutes, the scripts will run and more packages will be installed and configured to the specifications of your testbed file.

The server will reboot automatically.

After it comes back up sudo su when you login again

Then you need to source your authentication files.

keystonerc  and openstackrc  are located in /etc/contrail

source /etc/contrail/keystonerc
source /etc/contrail/openstackrc

Here are some commands  you can issue after installation to check the state of openstack

openstack-status

nova-manage service list

root@ubuntu:/home/user# openstack-status
== Nova services ==
openstack-nova-api:           active
openstack-nova-compute:       active
openstack-nova-network:       inactive (disabled on boot)
openstack-nova-scheduler:     active
openstack-nova-volume:        inactive (disabled on boot)
openstack-nova-conductor:     active
== Glance services ==
openstack-glance-api:         active
openstack-glance-registry:    active
== Keystone service ==
openstack-keystone:           active
== Cinder services ==
openstack-cinder-api:         active
openstack-cinder-scheduler:   active
openstack-cinder-volume:      inactive (disabled on boot)
== Support services ==
mysql:                        active
libvirt-bin:                  active
rabbitmq-server:              inactive (disabled on boot)
memcached:                    inactive (disabled on boot)
== Keystone users ==

+----------------------------------+---------+---------+---------------------+
|                id                |   name  | enabled |        email        |
+----------------------------------+---------+---------+---------------------+
| 2fa7b037efe1437a9045eab35f446511 |  admin  |   True  |  admin@example.com  |
| 6bb262f464a546b391b30879f1e8f10b |  cinder |   True  |  cinder@example.com |
| fdedd2c1a26d44ab9e83f000383cedf3 |   demo  |   True  |   demo@example.com  |
| cdc338c569af42cf87ba7bc7e7e161a8 |  glance |   True  |  glance@example.com |
| 2e28f88537064d97bfed64ad62f2fc66 | neutron |   True  | neutron@example.com |
| c09776b2f84744198d46c0361bfc1070 |   nova  |   True  |   nova@example.com  |
+----------------------------------+---------+---------+---------------------+
== Glance images ==
ID                                   Name                           Disk Format          Container Format     Size         
------------------------------------ ------------------------------ -------------------- -------------------- --------------
== Nova instance flavors ==
m1.medium: Memory: 4096MB, VCPUS: 2, Root: 40GB, Ephemeral: 0Gb, FlavorID: 3, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.tiny: Memory: 512MB, VCPUS: 1, Root: 1GB, Ephemeral: 0Gb, FlavorID: 1, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.large: Memory: 8192MB, VCPUS: 4, Root: 80GB, Ephemeral: 0Gb, FlavorID: 4, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.xlarge: Memory: 16384MB, VCPUS: 8, Root: 160GB, Ephemeral: 0Gb, FlavorID: 5, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.small: Memory: 2048MB, VCPUS: 1, Root: 20GB, Ephemeral: 0Gb, FlavorID: 2, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
== Nova instances ==

root@ubuntu:/home/user# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth ubuntu                               internal         enabled    :-)   2014-09-18 17:26:26
nova-console     ubuntu                               internal         enabled    :-)   2014-09-18 17:26:26
nova-scheduler   ubuntu                               internal         enabled    :-)   2014-09-18 17:26:26
nova-conductor   ubuntu                               internal         enabled    :-)   2014-09-18 17:26:27
nova-compute     ubuntu                               nova             enabled    :-)   2014-09-18 17:26:26


You are now ready to login to the Horizon Dashboard to configure your Openstack platform.

http://x.x.x.x/horizon



To log into your contrail controller change the port number to 8143

https://x.x.x.x:8143/



Next time I'll show how create a MP-BGP connection to a MX gateway to all traffic from the virtual network to connect to a physical network.

Thursday, May 15, 2014

Neutron Network Namespaces

Warning: The following is a rant of my observations about Openstack neutron.

They shouldn't call it Network Namespaces. They should call it overly complex naming convention.
I've been trying to wrap my head around the networking portion, aka Neutron, of OpenStack. What I've noticed is how ridiculously lousy the Horizon GUI is when trying to troubleshoot any issues. Almost anything useful is accessible through the CLI. What's the purpose of Cloud computing if you have to fallback to the cli to do anything?

When I build a VM, say VM1 on network net1 , I have no visibility about the virtual interfaces that it uses.

For example: Say I want to ping the virtual interface the VM created from the Openstack server. It should be pretty simple right? You should be able to do something like ping <network-name> ip address.

In order to do this you have to use this command:

ip netns - which I believe is short for network namespace

First, you need to know what the networkUUID.

You can list the network objects that were created by the namespace as follows:

# ip netns list
qrouter-ed4afc1b-06ab-417e-a7e2-d5be13b822af
qdhcp-4dc834f5-e759-4d79-acf0-780768f1fa86
qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48
qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7
qdhcp-dc49c1a5-07d0-4225-bea5-02316aec3a42

The structure of the networkUUID looks like this

qdhcp-<networkUUID>

WHAT the hell? That's not a useful name. That's an ID.

You can find this under this directory:

[root@centos-6-5-openstack ~(keystone_admin)]# ls /var/lib/neutron/dhcp/
0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48  a5958652-7348-436f-8aff-2c9ebd7dd9f7
4dc834f5-e759-4d79-acf0-780768f1fa86  dc49c1a5-07d0-4225-bea5-02316aec3a42

and then peer into it's contents

[root@centos-6-5-openstack ~(keystone_admin)]# cat /var/lib/neutron/dhcp/0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48/host
fa:16:3e:b8:98:40,host-20-20-0-11.openstacklocal,20.20.0.11
fa:16:3e:b5:6f:26,host-20-20-0-10.openstacklocal,20.20.0.10

You can also try to do this via this command:

[root@centos-6-5-openstack ~(keystone_admin)]# ip netns exec qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 ip a
30: tapafe229a7-2e: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:b8:98:40 brd ff:ff:ff:ff:ff:ff
    inet 20.20.0.11/24 brd 20.20.0.255 scope global tapafe229a7-2e
    inet6 fe80::f816:3eff:feb8:9840/64 scope link
       valid_lft forever preferred_lft forever
33: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

It won't give show you the VM's ip, but should show you the subnet it's on.

One other place to look:

[root@centos-6-5-openstack ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+-------------------------------------------------------+
| id                                   | name    | subnets                                               |
+--------------------------------------+---------+-------------------------------------------------------+
| 0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 | net1   | d4bf516f-135c-4c7f-ba7e-363cb6c7d307 20.20.0.0/24     |
| 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d | public  | 8942189b-8fbf-44e9-adbb-d62dd0d27015 192.168.250.0/24 |
| dc49c1a5-07d0-4225-bea5-02316aec3a42 | private | 6d67e2de-7f63-454d-9e88-fe33e6121b7b 10.0.0.0/24      |
+--------------------------------------+---------+-------------------------------------------------------+

So once I have the network id I can issue the following command:

[root@centos-6-5-openstack ~(keystone_admin)]# ip netns exec qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 ping 20.20.0.10
PING 20.20.0.10 (20.20.0.10) 56(84) bytes of data.
64 bytes from 20.20.0.10: icmp_seq=1 ttl=64 time=29.5 ms
64 bytes from 20.20.0.10: icmp_seq=2 ttl=64 time=15.0 ms
^C
--- 20.20.0.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1534ms
rtt min/avg/max/mdev = 15.079/22.301/29.523/7.222 ms

This is such a pain. What if the number of tenants, networks and VMs is huge? Why can't Openstack utilize the network name as part of the network UUid?



Tuesday, May 6, 2014

Doing reverse lookups of information in order to find things in Openstack

One of the most annoying things I find in Openstack is the lack of use of user friendly names. It's very tedious to reverse lookup information for troubleshooting and configuration.

For example, say I want to find out which tap port on my VM is connected to an openvswitch bridge. I would have to look up the VM in nova and find the corresponding IP assigned to it.

[root@centos-6-5-openstack ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| 8b3ecae9-7b59-4b6f-a0a0-bbd5ab5aa629 | aa   | ACTIVE | -          | Running     | net1=192.168.251.10 |

| a679f6f9-bd7d-4e31-95a6-9776c44f0a4a | bb   | ACTIVE | -          | Running     | net1=192.168.251.12 |

Then I would list the ports in neutron.

[root@centos-6-5-openstack ~(keystone_admin)]# neutron port-list
+--------------------------------------+-------------+-------------------+---------------------------------------------------------------------------------------+
| id                                   | name        | mac_address       | fixed_ips                                                                             |
+--------------------------------------+-------------+-------------------+---------------------------------------------------------------------------------------+
| 014d6bba-7ab9-4028-9cd9-dbe27bd51b83 |             | fa:16:3e:1b:19:7d | {"subnet_id": "8942189b-8fbf-44e9-adbb-d62dd0d27015", "ip_address": "192.168.250.10"} |
| 1d3246d7-adb1-4b18-a911-b36562c278e0 |             | fa:16:3e:bd:e9:69 | {"subnet_id": "0147052f-2a0b-4a5b-9930-1d4e258e0c1a", "ip_address": "192.168.251.10"} |
| 40b988aa-c016-4e85-b9cf-beeb4332ab04 |             | fa:16:3e:e1:7e:57 | {"subnet_id": "0147052f-2a0b-4a5b-9930-1d4e258e0c1a", "ip_address": "192.168.251.12"} |
| cb867d96-a4ed-449d-a35a-f3670a14e710 |             | fa:16:3e:ce:a5:7e | {"subnet_id": "0147052f-2a0b-4a5b-9930-1d4e258e0c1a", "ip_address": "192.168.251.11"} |
| d267267b-5630-43df-8a07-05793fd831d0 |             | fa:16:3e:61:84:91 | {"subnet_id": "8942189b-8fbf-44e9-adbb-d62dd0d27015", "ip_address": "192.168.250.11"} |
| f434c257-5bbf-42a6-9136-072d1104fa19 |               | fa:16:3e:24:18:08 | {"subnet_id": "0147052f-2a0b-4a5b-9930-1d4e258e0c1a", "ip_address": "192.168.251.1"}  |

+--------------------------------------+-------------+-------------------+---------------------------------------------------------------------------------------+

Next I would need to look at the mac address and do a lookup in ip address

After coming through the list I can then verify the tap address with ip address show tapxxxx

[root@centos-6-5-openstack ~(keystone_admin)]# ip a show tap40b988aa-c0
30: tap40b988aa-c0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:16:3e:e1:7e:57 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fee1:7e57/64 scope link 
       valid_lft forever preferred_lft forever


And now I can do something useful like see which bridge the tap interface belongs to and verify the mac in the mac table.

[root@centos-6-5-openstack ~(keystone_admin)]# brctl show
bridge name bridge id STP enabled interfaces
qbr1d3246d7-ad 8000.4a04d0b90d18 no qvb1d3246d7-ad
tap1d3246d7-ad
qbr40b988aa-c0 8000.de752effa37c no qvb40b988aa-c0
tap40b988aa-c0
qbr7316a274-16 8000.8671bda4d699 no qvb7316a274-16
tap7316a274-16
qbrd06fccba-ad 8000.4613c3848432 no qvbd06fccba-ad
tapd06fccba-ad
[root@centos-6-5-openstack ~(keystone_admin)]# brctl showmacs qbr40b988aa-c0
port no mac addr is local? ageing timer
  1 de:75:2e:ff:a3:7c yes   0.00
  2 fe:16:3e:e1:7e:57 yes   0.00

or look at the openvswitch table and find out which bridge it belongs to

[root@centos-6-5-openstack ~(keystone_admin)]# ovs-vsctl list-ifaces br-int
int-br-ex
qr-87202226-14
qr-ba3e1c3b-49
qr-f2b01b2b-ad
qr-f434c257-5b
qvo1d3246d7-ad
qvo40b988aa-c0
qvo7316a274-16
qvod06fccba-ad
tapafe229a7-2e
tapca265ec6-29
tapcb867d96-a4
tape4751034-2c


This looks like a to do scripting project to get this information. 

Here I could create a table that would give me:

Nova ID
VM name
VM IP
VM Mac address
Neutron ID
tap interface
bridge mac table
openvswitch bridge

Monday, May 5, 2014

Creating my first OpenStack cloud platform and connecting it to a physical network.

In my quest to learn more about network virtualization I decided to learn more about OpenStack as a cloud platform. So I started from scratch and tried to install Openstack.

I took an old server which had two NIC ports and installed Centos 6.5. I then installed openstack all in one server from the following:

http://openstack.redhat.com/Quickstart

I decided to use the havana version because icehouse just came out and I wanted to wait until after the bugs were fixed.

After doing a bunch of yum installs I had it working. I was actually able to create a few VMs using the Cirros Linux image and created a simple network. I was able to ping from one VM to the other.

However the documentation was not clear on how to connect your VMs to the outside world through a physical network.

After searching around for a few days, I found this:

http://openstack.redhat.com/Neutron_with_existing_external_network

I decided to use eth0 as my management ip to access Openstack and use eth1 as the port for the VMs to connect to the physical network.

I also learned that OpenStack uses OpenVswitch and Neutron for network connectivity. You have to learn a little bit about OpenVswitch on a server.

This led me to make the following changes.
under  /etc/sysconfig/network-scripts

I duplicated (to backup) and changed the following files.

[root@centos-6-5-openstack network-scripts]# more ifcfg-eth1
DEVICE=eth1
TYPE=OVSPort
UUID=100083c1-6174-4a59-b3d0-09081eef106a
ONBOOT=yes
DEVICETYPE=ovs
HWADDR=00:30:48:F9:B9:C9
OVS_BRIDGE=br-ex
NAME="System eth1"

[root@centos-6-5-openstack network-scripts]# more ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.250.11
NETMASK=255.255.255.0
DNS=198.6.1.1
ONBOOT=yes

Then I tied the eth1 interface to the bridge "br-ex"

"br-ex" was automatically created on my server, but if it's not there, you should create a bridge (ova-vsctl add-br <NAME>)  and associate the physical interface (eth1) to that bridge.

[root@centos-6-5-openstack network-scripts]# port br-ex eth1

[root@centos-6-5-openstack network-scripts]# ovs-vsctl show
f6d22683-9529-48b6-b9d0-5e49cb720d44
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-d267267b-56"
            Interface "qg-d267267b-56"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
        Port "eth1"
            Interface "eth1"
    Bridge br-int

I also had to make a change to a file:

Add to the /etc/neutron/plugin.ini file these lines:
network_vlan_ranges = physnet1
bridge_mappings = physnet1:br-ex

I had to make sure that eth1 is a port for bridge br-ex not br-int. For some reason it was not doing this, so I removed the port from br-int using ovs-vsctl del-port command

Red hat wants you to restart nework services after making the changes. However when I issued that command, in ifconfig I would see that eth1 still had an ip address and br-ex was not updated. So I rebooted the server.

I also used the following to troubleshoot
tcpdump -nei br-ex
tcpdump -nei eth1

After reboot all was well.

root@centos-6-5-openstack network-scripts]# ifconfig
br-ex     Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C9
          inet addr:192.168.250.11  Bcast:192.168.250.255  Mask:255.255.255.0
          inet6 addr: fe80::5822:16ff:fe25:91a/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4834 (4.7 KiB)  TX bytes:1336 (1.3 KiB)

br-int    Link encap:Ethernet  HWaddr BE:9D:E4:A4:B0:44
          inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:213 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:16262 (15.8 KiB)  TX bytes:468 (468.0 b)

eth0      Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C8
          inet addr:10.161.32.230  Bcast:10.161.39.255  Mask:255.255.248.0
          inet6 addr: fe80::230:48ff:fef9:b9c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14400 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1686756 (1.6 MiB)  TX bytes:7627901 (7.2 MiB)
          Interrupt:18 Memory:d8020000-d8040000

eth1      Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C9
          inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:905 errors:0 dropped:0 overruns:0 frame:0
          TX packets:978 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:82360 (80.4 KiB)  TX bytes:87771 (85.7 KiB)
          Interrupt:19 Memory:d8060000-d8080000

int-br-ex Link encap:Ethernet  HWaddr 5A:EC:7F:16:11:35
          inet6 addr: fe80::58ec:7fff:fe16:1135/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:190 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4752 (4.6 KiB)  TX bytes:14550 (14.2 KiB)


From my physical gateway switch/router 192.168.250.1 I tried pinging the br-ex interface 192.168.250.11

Next I created the tenant subnet, vm and the openstack router.

First I had to source my admin credentials

source /root/keystonerc_admin

Next I followed the instructions on creating a public network. I added a floating ip.

http://openstack.redhat.com/Floating_IP_range

By default packstack creates a demo public network, with a subnet that I was not using. So I cleared the gw of this first.

# neutron router-gateway-clear router1
Then deleted the subnet
# neutron subnet-delete public

Next I recreated a new subnet
neutron subnet-create --name public --enable_dhcp=False --allocation-pool=start=192.168.250.10,end=192.168.250.20 --gateway=192.168.250.1 public 192.168.250.0/24

Then created a router using Horizon.
Then I tied the router to the public subnet.

neutron router-gateway-set test-rtr public

[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron subnet-show public
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "192.168.250.10", "end": "192.168.250.20"} |
| cidr             | 192.168.250.0/24                                     |
| dns_nameservers  |                                                      |
| enable_dhcp      | False                                                |
| gateway_ip       | 192.168.250.1                                        |
| host_routes      |                                                      |
| id               | 8942189b-8fbf-44e9-adbb-d62dd0d27015                 |
| ip_version       | 4                                                    |
| name             | public                                               |
| network_id       | 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d                 |
| tenant_id        | 3a10de8a82444118865a6398b336ee68                     |
+------------------+------------------------------------------------------+

From Horizon, my setup looks simple:



One thing that threw me off during the whole process was trying to figure out if the openstack Gateway IP was working or not.

I noticed that the status kept saying down.

[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron port-show router-gw | grep status
| status                | DOWN  

After googling it, I found out that this was a bug and a display issue. So you can't tell if the gateway ip is actually working or not because the status is broken.


From my VM I am now able to access my physical switch