For some reason I could not delete a VM in my Openstack Cloud Platform. I tried to terminate the instance in the Horizon GUI but that did not work. I also tried to do it from the cli via:
# nova delete <uuid>
That did not work either. So I researched in on the web and noticed that the instances are kept in two places. The first place is in the nova directory:
/var/lib/nova/instances
[root@sdnnode2 instances]# ls
51426435-9f72-4077-91a4-6c89ea53894f 58fd794e-048a-4135-b064-17fb6bb3682b bbc44d49-62f7-4cdc-8e7f-564190c6f043 locks _base d0ce9950-752c-406f-91e0-3bd0ff0f488d snapshots
looking into those uuid directory names contained the following:
[root@sdnnode2 51426435-9f72-4077-91a4-6c89ea53894f]# ls
console.log disk libvirt.xml
However deleting these files still would not delete the VM from appearing in the Horizon GUI. These are the files that Openstack would use to spin up the VM.
The vm existed in MySQL.
I found a handy sql_guide to brush me up on SQL commands. Then I had to access the SQL server with the correct credentials. If you don't remember it you can find it in the nova.conf file located in:
/etc/nova/nova.conf
[root@sdnnode2 nova]# cat nova.conf | grep sql
sql_connection = mysql://nova:<mypassword>@10.161.38.4/nova
Then to access the SQL server:
[root@sdnnode2 nova]# sudo mysql -u nova -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 166
Server version: 5.1.73 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
show databases gives us.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| nova |
| test |
+--------------------+
mysql> use nova;
Database changed
mysql> show tables;
+--------------------------------------------+
| Tables_in_nova |
+--------------------------------------------+
| agent_builds |
| aggregate_hosts |
| aggregate_metadata |
| aggregates |
| block_device_mapping |
| bw_usage_cache |
<TRUNCATED>
| volume_usage_cache |
| volumes |
+--------------------------------------------+
115 rows in set (0.00 sec)
Here we can then find the instance uuid. First you could figure out the fields in a particular table:
mysql> describe key_pairs;
+-------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+----------------+
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| deleted_at | datetime | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(255) | YES | | NULL | |
| user_id | varchar(255) | YES | MUL | NULL | |
| fingerprint | varchar(255) | YES | | NULL | |
| public_key | mediumtext | YES | | NULL | |
| deleted | int(11) | YES | | NULL | |
+-------------+--------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)
Then you can list the objects that are in the field you are searching.
mysql> select name from key_pairs;
+----------+
| name |
+----------+
| my-key |
| root-key |
+----------+
2 rows in set (0.00 sec)
I found out there are 5 tables you need to modify to get delete the VM and the should be done in this order:
security_group_instance_association
block_device_mapping
instance_info_caches
fixed_ips
instances
So I needed to find the uuid of the VM I was going to delete.
mysql> select uuid,hostname from instances where deleted=0;
+--------------------------------------+-----------+
| uuid | hostname |
+--------------------------------------+-----------+
| bbc44d49-62f7-4cdc-8e7f-564190c6f043 | right-vm1 |
| d0ce9950-752c-406f-91e0-3bd0ff0f488d | left-vm1 |
| 51426435-9f72-4077-91a4-6c89ea53894f | test-vm |
| 58fd794e-048a-4135-b064-17fb6bb3682b | left-vm2 |
+--------------------------------------+-----------+
4 rows in set (0.00 sec)
I then started my process. I checked to see if the instance was not already marked deleted by checking for the condition of "deleted=0";
mysql> select instance_uuid from security_group_instance_association where deleted=0;
Empty set (0.00 sec)
I didn't have a security group association so I proceeded with the next step.
mysql> DELETE FROM block_device_mapping WHERE instance_uuid = "51426435-9f72-4077-91a4-6c89ea53894f";
Query OK, 1 row affected (0.00 sec)
I then followed the other steps
mysql> select instance_uuid from instance_info_caches where deleted=0;
DELETE FROM instance_info_caches WHERE instance_uuid = "51426435-9f72-4077-91a4-6c89ea53894f";
Query OK, 1 row affected (0.00 sec)
mysql>select instance_uuid from fixed_ips where deleted=0;
Empty set (0.00 sec)
I didn't have a fixed_ip so I went to the last step, which was to changed the "deleted" field.
You need to make sure the deleted number matches the id.
mysql> select id,deleted,uuid from instances;
+----+---------+--------------------------------------+
| id | deleted | uuid |
+----+---------+--------------------------------------+
| 31 | 31 | 028ff14b-55f2-4aa2-8fd9-37d70ff26a3b |
| 11 | 11 | 4c60dd0a-ae81-45e0-931b-c996e89b10d7 |
| 47 | 47 | 4ca2687f-35a4-4248-9777-24d9317e4c20 |
| 25 | 25 | 512747be-b608-423e-ac81-54f2ebcd7e58 |
| 35 | 0 | 51426435-9f72-4077-91a4-6c89ea53894f |
mysql> UPDATE instances SET deleted = 40 where uuid = "51426435-9f72-4077-91a4-6c89ea53894f";Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select id,deleted,uuid from instances;
+----+---------+--------------------------------------+
| id | deleted | uuid |
+----+---------+--------------------------------------+
| 31 | 31 | 028ff14b-55f2-4aa2-8fd9-37d70ff26a3b |
| 11 | 11 | 4c60dd0a-ae81-45e0-931b-c996e89b10d7 |
| 47 | 47 | 4ca2687f-35a4-4248-9777-24d9317e4c20 |
| 25 | 25 | 512747be-b608-423e-ac81-54f2ebcd7e58 |
| 35 | 35 | 51426435-9f72-4077-91a4-6c89ea53894f |
Finally the instance was no longer in my Openstack server.
[root@sdnserver1 nova]# nova show 51426435-9f72-4077-91a4-6c89ea53894f
ERROR: No server with a name or ID of '51426435-9f72-4077-91a4-6c89ea53894f' exists.
Showing posts with label VM. Show all posts
Showing posts with label VM. Show all posts
Wednesday, August 20, 2014
Thursday, May 15, 2014
Neutron Network Namespaces
Warning: The following is a rant of my observations about Openstack neutron.
They shouldn't call it Network Namespaces. They should call it overly complex naming convention.
I've been trying to wrap my head around the networking portion, aka Neutron, of OpenStack. What I've noticed is how ridiculously lousy the Horizon GUI is when trying to troubleshoot any issues. Almost anything useful is accessible through the CLI. What's the purpose of Cloud computing if you have to fallback to the cli to do anything?
When I build a VM, say VM1 on network net1 , I have no visibility about the virtual interfaces that it uses.
For example: Say I want to ping the virtual interface the VM created from the Openstack server. It should be pretty simple right? You should be able to do something like ping <network-name> ip address.
In order to do this you have to use this command:
ip netns - which I believe is short for network namespace
First, you need to know what the networkUUID.
You can list the network objects that were created by the namespace as follows:
# ip netns list
qrouter-ed4afc1b-06ab-417e-a7e2-d5be13b822af
qdhcp-4dc834f5-e759-4d79-acf0-780768f1fa86
qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48
qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7
qdhcp-dc49c1a5-07d0-4225-bea5-02316aec3a42
The structure of the networkUUID looks like this
qdhcp-<networkUUID>
WHAT the hell? That's not a useful name. That's an ID.
You can find this under this directory:
[root@centos-6-5-openstack ~(keystone_admin)]# ls /var/lib/neutron/dhcp/
0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 a5958652-7348-436f-8aff-2c9ebd7dd9f7
4dc834f5-e759-4d79-acf0-780768f1fa86 dc49c1a5-07d0-4225-bea5-02316aec3a42
and then peer into it's contents
[root@centos-6-5-openstack ~(keystone_admin)]# cat /var/lib/neutron/dhcp/0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48/host
fa:16:3e:b8:98:40,host-20-20-0-11.openstacklocal,20.20.0.11
fa:16:3e:b5:6f:26,host-20-20-0-10.openstacklocal,20.20.0.10
You can also try to do this via this command:
[root@centos-6-5-openstack ~(keystone_admin)]# ip netns exec qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 ip a
30: tapafe229a7-2e: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:b8:98:40 brd ff:ff:ff:ff:ff:ff
inet 20.20.0.11/24 brd 20.20.0.255 scope global tapafe229a7-2e
inet6 fe80::f816:3eff:feb8:9840/64 scope link
valid_lft forever preferred_lft forever
33: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
It won't give show you the VM's ip, but should show you the subnet it's on.
One other place to look:
[root@centos-6-5-openstack ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+-------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+-------------------------------------------------------+
| 0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 | net1 | d4bf516f-135c-4c7f-ba7e-363cb6c7d307 20.20.0.0/24 |
| 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d | public | 8942189b-8fbf-44e9-adbb-d62dd0d27015 192.168.250.0/24 |
| dc49c1a5-07d0-4225-bea5-02316aec3a42 | private | 6d67e2de-7f63-454d-9e88-fe33e6121b7b 10.0.0.0/24 |
+--------------------------------------+---------+-------------------------------------------------------+
So once I have the network id I can issue the following command:
[root@centos-6-5-openstack ~(keystone_admin)]# ip netns exec qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 ping 20.20.0.10
PING 20.20.0.10 (20.20.0.10) 56(84) bytes of data.
64 bytes from 20.20.0.10: icmp_seq=1 ttl=64 time=29.5 ms
64 bytes from 20.20.0.10: icmp_seq=2 ttl=64 time=15.0 ms
^C
--- 20.20.0.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1534ms
rtt min/avg/max/mdev = 15.079/22.301/29.523/7.222 ms
This is such a pain. What if the number of tenants, networks and VMs is huge? Why can't Openstack utilize the network name as part of the network UUid?
They shouldn't call it Network Namespaces. They should call it overly complex naming convention.
I've been trying to wrap my head around the networking portion, aka Neutron, of OpenStack. What I've noticed is how ridiculously lousy the Horizon GUI is when trying to troubleshoot any issues. Almost anything useful is accessible through the CLI. What's the purpose of Cloud computing if you have to fallback to the cli to do anything?
When I build a VM, say VM1 on network net1 , I have no visibility about the virtual interfaces that it uses.
For example: Say I want to ping the virtual interface the VM created from the Openstack server. It should be pretty simple right? You should be able to do something like ping <network-name> ip address.
In order to do this you have to use this command:
ip netns - which I believe is short for network namespace
First, you need to know what the networkUUID.
You can list the network objects that were created by the namespace as follows:
# ip netns list
qrouter-ed4afc1b-06ab-417e-a7e2-d5be13b822af
qdhcp-4dc834f5-e759-4d79-acf0-780768f1fa86
qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48
qdhcp-a5958652-7348-436f-8aff-2c9ebd7dd9f7
qdhcp-dc49c1a5-07d0-4225-bea5-02316aec3a42
The structure of the networkUUID looks like this
qdhcp-<networkUUID>
WHAT the hell? That's not a useful name. That's an ID.
You can find this under this directory:
[root@centos-6-5-openstack ~(keystone_admin)]# ls /var/lib/neutron/dhcp/
0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 a5958652-7348-436f-8aff-2c9ebd7dd9f7
4dc834f5-e759-4d79-acf0-780768f1fa86 dc49c1a5-07d0-4225-bea5-02316aec3a42
and then peer into it's contents
[root@centos-6-5-openstack ~(keystone_admin)]# cat /var/lib/neutron/dhcp/0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48/host
fa:16:3e:b8:98:40,host-20-20-0-11.openstacklocal,20.20.0.11
fa:16:3e:b5:6f:26,host-20-20-0-10.openstacklocal,20.20.0.10
You can also try to do this via this command:
[root@centos-6-5-openstack ~(keystone_admin)]# ip netns exec qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 ip a
30: tapafe229a7-2e: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:b8:98:40 brd ff:ff:ff:ff:ff:ff
inet 20.20.0.11/24 brd 20.20.0.255 scope global tapafe229a7-2e
inet6 fe80::f816:3eff:feb8:9840/64 scope link
valid_lft forever preferred_lft forever
33: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
It won't give show you the VM's ip, but should show you the subnet it's on.
One other place to look:
[root@centos-6-5-openstack ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+-------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+-------------------------------------------------------+
| 0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 | net1 | d4bf516f-135c-4c7f-ba7e-363cb6c7d307 20.20.0.0/24 |
| 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d | public | 8942189b-8fbf-44e9-adbb-d62dd0d27015 192.168.250.0/24 |
| dc49c1a5-07d0-4225-bea5-02316aec3a42 | private | 6d67e2de-7f63-454d-9e88-fe33e6121b7b 10.0.0.0/24 |
+--------------------------------------+---------+-------------------------------------------------------+
So once I have the network id I can issue the following command:
[root@centos-6-5-openstack ~(keystone_admin)]# ip netns exec qdhcp-0b6ed891-a9ae-4c5a-a7f9-36e851bf1d48 ping 20.20.0.10
PING 20.20.0.10 (20.20.0.10) 56(84) bytes of data.
64 bytes from 20.20.0.10: icmp_seq=1 ttl=64 time=29.5 ms
64 bytes from 20.20.0.10: icmp_seq=2 ttl=64 time=15.0 ms
^C
--- 20.20.0.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1534ms
rtt min/avg/max/mdev = 15.079/22.301/29.523/7.222 ms
This is such a pain. What if the number of tenants, networks and VMs is huge? Why can't Openstack utilize the network name as part of the network UUid?
Monday, May 5, 2014
Creating my first OpenStack cloud platform and connecting it to a physical network.
In my quest to learn more about network virtualization I decided to learn more about OpenStack as a cloud platform. So I started from scratch and tried to install Openstack.
I took an old server which had two NIC ports and installed Centos 6.5. I then installed openstack all in one server from the following:
http://openstack.redhat.com/Quickstart
I decided to use the havana version because icehouse just came out and I wanted to wait until after the bugs were fixed.
After doing a bunch of yum installs I had it working. I was actually able to create a few VMs using the Cirros Linux image and created a simple network. I was able to ping from one VM to the other.
However the documentation was not clear on how to connect your VMs to the outside world through a physical network.
After searching around for a few days, I found this:
http://openstack.redhat.com/Neutron_with_existing_external_network
I decided to use eth0 as my management ip to access Openstack and use eth1 as the port for the VMs to connect to the physical network.
I also learned that OpenStack uses OpenVswitch and Neutron for network connectivity. You have to learn a little bit about OpenVswitch on a server.
This led me to make the following changes.
under /etc/sysconfig/network-scripts
I duplicated (to backup) and changed the following files.
[root@centos-6-5-openstack network-scripts]# more ifcfg-eth1
DEVICE=eth1
TYPE=OVSPort
UUID=100083c1-6174-4a59-b3d0-09081eef106a
ONBOOT=yes
DEVICETYPE=ovs
HWADDR=00:30:48:F9:B9:C9
OVS_BRIDGE=br-ex
NAME="System eth1"
[root@centos-6-5-openstack network-scripts]# more ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.250.11
NETMASK=255.255.255.0
DNS=198.6.1.1
ONBOOT=yes
Then I tied the eth1 interface to the bridge "br-ex"
"br-ex" was automatically created on my server, but if it's not there, you should create a bridge (ova-vsctl add-br <NAME>) and associate the physical interface (eth1) to that bridge.
[root@centos-6-5-openstack network-scripts]# port br-ex eth1
[root@centos-6-5-openstack network-scripts]# ovs-vsctl show
f6d22683-9529-48b6-b9d0-5e49cb720d44
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "qg-d267267b-56"
Interface "qg-d267267b-56"
type: internal
Port phy-br-ex
Interface phy-br-ex
Port "eth1"
Interface "eth1"
Bridge br-int
I also had to make a change to a file:
Add to the /etc/neutron/plugin.ini file these lines:
network_vlan_ranges = physnet1
bridge_mappings = physnet1:br-ex
I had to make sure that eth1 is a port for bridge br-ex not br-int. For some reason it was not doing this, so I removed the port from br-int using ovs-vsctl del-port command
Red hat wants you to restart nework services after making the changes. However when I issued that command, in ifconfig I would see that eth1 still had an ip address and br-ex was not updated. So I rebooted the server.
I also used the following to troubleshoot
tcpdump -nei br-ex
tcpdump -nei eth1
After reboot all was well.
root@centos-6-5-openstack network-scripts]# ifconfig
br-ex Link encap:Ethernet HWaddr 00:30:48:F9:B9:C9
inet addr:192.168.250.11 Bcast:192.168.250.255 Mask:255.255.255.0
inet6 addr: fe80::5822:16ff:fe25:91a/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:100 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4834 (4.7 KiB) TX bytes:1336 (1.3 KiB)
br-int Link encap:Ethernet HWaddr BE:9D:E4:A4:B0:44
inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:213 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:16262 (15.8 KiB) TX bytes:468 (468.0 b)
eth0 Link encap:Ethernet HWaddr 00:30:48:F9:B9:C8
inet addr:10.161.32.230 Bcast:10.161.39.255 Mask:255.255.248.0
inet6 addr: fe80::230:48ff:fef9:b9c8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14400 errors:0 dropped:0 overruns:0 frame:0
TX packets:11354 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1686756 (1.6 MiB) TX bytes:7627901 (7.2 MiB)
Interrupt:18 Memory:d8020000-d8040000
eth1 Link encap:Ethernet HWaddr 00:30:48:F9:B9:C9
inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:905 errors:0 dropped:0 overruns:0 frame:0
TX packets:978 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:82360 (80.4 KiB) TX bytes:87771 (85.7 KiB)
Interrupt:19 Memory:d8060000-d8080000
int-br-ex Link encap:Ethernet HWaddr 5A:EC:7F:16:11:35
inet6 addr: fe80::58ec:7fff:fe16:1135/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:100 errors:0 dropped:0 overruns:0 frame:0
TX packets:190 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4752 (4.6 KiB) TX bytes:14550 (14.2 KiB)
From my physical gateway switch/router 192.168.250.1 I tried pinging the br-ex interface 192.168.250.11
Next I created the tenant subnet, vm and the openstack router.
First I had to source my admin credentials
source /root/keystonerc_admin
Next I followed the instructions on creating a public network. I added a floating ip.
http://openstack.redhat.com/Floating_IP_range
By default packstack creates a demo public network, with a subnet that I was not using. So I cleared the gw of this first.
# neutron router-gateway-clear router1
Then deleted the subnet
# neutron subnet-delete public
Next I recreated a new subnet
neutron subnet-create --name public --enable_dhcp=False --allocation-pool=start=192.168.250.10,end=192.168.250.20 --gateway=192.168.250.1 public 192.168.250.0/24
Then created a router using Horizon.
Then I tied the router to the public subnet.
neutron router-gateway-set test-rtr public
[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron subnet-show public
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "192.168.250.10", "end": "192.168.250.20"} |
| cidr | 192.168.250.0/24 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 192.168.250.1 |
| host_routes | |
| id | 8942189b-8fbf-44e9-adbb-d62dd0d27015 |
| ip_version | 4 |
| name | public |
| network_id | 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d |
| tenant_id | 3a10de8a82444118865a6398b336ee68 |
+------------------+------------------------------------------------------+
From Horizon, my setup looks simple:
One thing that threw me off during the whole process was trying to figure out if the openstack Gateway IP was working or not.
I noticed that the status kept saying down.
[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron port-show router-gw | grep status
| status | DOWN
After googling it, I found out that this was a bug and a display issue. So you can't tell if the gateway ip is actually working or not because the status is broken.
From my VM I am now able to access my physical switch
I took an old server which had two NIC ports and installed Centos 6.5. I then installed openstack all in one server from the following:
http://openstack.redhat.com/Quickstart
I decided to use the havana version because icehouse just came out and I wanted to wait until after the bugs were fixed.
After doing a bunch of yum installs I had it working. I was actually able to create a few VMs using the Cirros Linux image and created a simple network. I was able to ping from one VM to the other.
However the documentation was not clear on how to connect your VMs to the outside world through a physical network.
After searching around for a few days, I found this:
http://openstack.redhat.com/Neutron_with_existing_external_network
I decided to use eth0 as my management ip to access Openstack and use eth1 as the port for the VMs to connect to the physical network.
I also learned that OpenStack uses OpenVswitch and Neutron for network connectivity. You have to learn a little bit about OpenVswitch on a server.
This led me to make the following changes.
under /etc/sysconfig/network-scripts
I duplicated (to backup) and changed the following files.
[root@centos-6-5-openstack network-scripts]# more ifcfg-eth1
DEVICE=eth1
TYPE=OVSPort
UUID=100083c1-6174-4a59-b3d0-09081eef106a
ONBOOT=yes
DEVICETYPE=ovs
HWADDR=00:30:48:F9:B9:C9
OVS_BRIDGE=br-ex
NAME="System eth1"
[root@centos-6-5-openstack network-scripts]# more ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.250.11
NETMASK=255.255.255.0
DNS=198.6.1.1
ONBOOT=yes
Then I tied the eth1 interface to the bridge "br-ex"
"br-ex" was automatically created on my server, but if it's not there, you should create a bridge (ova-vsctl add-br <NAME>) and associate the physical interface (eth1) to that bridge.
[root@centos-6-5-openstack network-scripts]# port br-ex eth1
[root@centos-6-5-openstack network-scripts]# ovs-vsctl show
f6d22683-9529-48b6-b9d0-5e49cb720d44
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "qg-d267267b-56"
Interface "qg-d267267b-56"
type: internal
Port phy-br-ex
Interface phy-br-ex
Port "eth1"
Interface "eth1"
Bridge br-int
I also had to make a change to a file:
Add to the /etc/neutron/plugin.ini file these lines:
network_vlan_ranges = physnet1
bridge_mappings = physnet1:br-ex
I had to make sure that eth1 is a port for bridge br-ex not br-int. For some reason it was not doing this, so I removed the port from br-int using ovs-vsctl del-port command
Red hat wants you to restart nework services after making the changes. However when I issued that command, in ifconfig I would see that eth1 still had an ip address and br-ex was not updated. So I rebooted the server.
I also used the following to troubleshoot
tcpdump -nei br-ex
tcpdump -nei eth1
After reboot all was well.
root@centos-6-5-openstack network-scripts]# ifconfig
br-ex Link encap:Ethernet HWaddr 00:30:48:F9:B9:C9
inet addr:192.168.250.11 Bcast:192.168.250.255 Mask:255.255.255.0
inet6 addr: fe80::5822:16ff:fe25:91a/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:100 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4834 (4.7 KiB) TX bytes:1336 (1.3 KiB)
br-int Link encap:Ethernet HWaddr BE:9D:E4:A4:B0:44
inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:213 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:16262 (15.8 KiB) TX bytes:468 (468.0 b)
eth0 Link encap:Ethernet HWaddr 00:30:48:F9:B9:C8
inet addr:10.161.32.230 Bcast:10.161.39.255 Mask:255.255.248.0
inet6 addr: fe80::230:48ff:fef9:b9c8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:14400 errors:0 dropped:0 overruns:0 frame:0
TX packets:11354 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1686756 (1.6 MiB) TX bytes:7627901 (7.2 MiB)
Interrupt:18 Memory:d8020000-d8040000
eth1 Link encap:Ethernet HWaddr 00:30:48:F9:B9:C9
inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:905 errors:0 dropped:0 overruns:0 frame:0
TX packets:978 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:82360 (80.4 KiB) TX bytes:87771 (85.7 KiB)
Interrupt:19 Memory:d8060000-d8080000
int-br-ex Link encap:Ethernet HWaddr 5A:EC:7F:16:11:35
inet6 addr: fe80::58ec:7fff:fe16:1135/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:100 errors:0 dropped:0 overruns:0 frame:0
TX packets:190 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4752 (4.6 KiB) TX bytes:14550 (14.2 KiB)
From my physical gateway switch/router 192.168.250.1 I tried pinging the br-ex interface 192.168.250.11
Next I created the tenant subnet, vm and the openstack router.
First I had to source my admin credentials
source /root/keystonerc_admin
Next I followed the instructions on creating a public network. I added a floating ip.
http://openstack.redhat.com/Floating_IP_range
By default packstack creates a demo public network, with a subnet that I was not using. So I cleared the gw of this first.
# neutron router-gateway-clear router1
Then deleted the subnet
# neutron subnet-delete public
Next I recreated a new subnet
neutron subnet-create --name public --enable_dhcp=False --allocation-pool=start=192.168.250.10,end=192.168.250.20 --gateway=192.168.250.1 public 192.168.250.0/24
Then created a router using Horizon.
Then I tied the router to the public subnet.
neutron router-gateway-set test-rtr public
[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron subnet-show public
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "192.168.250.10", "end": "192.168.250.20"} |
| cidr | 192.168.250.0/24 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 192.168.250.1 |
| host_routes | |
| id | 8942189b-8fbf-44e9-adbb-d62dd0d27015 |
| ip_version | 4 |
| name | public |
| network_id | 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d |
| tenant_id | 3a10de8a82444118865a6398b336ee68 |
+------------------+------------------------------------------------------+
From Horizon, my setup looks simple:
One thing that threw me off during the whole process was trying to figure out if the openstack Gateway IP was working or not.
I noticed that the status kept saying down.
[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron port-show router-gw | grep status
| status | DOWN
After googling it, I found out that this was a bug and a display issue. So you can't tell if the gateway ip is actually working or not because the status is broken.
From my VM I am now able to access my physical switch
Tuesday, March 4, 2014
Use (EVPN) Ethernet Virtual Private Network for Data Center Interconnections (DCI)
As Enterprises build Data Centers at different locations for disaster recovery and traffic distribution, there is a need to interconnect them transparently. Stretching Layer 2 across a WAN poses some challenges.
1) Workload Mobility aka VM migration from one DC to another.
2) Fast convergence in a multi homed environment.
3) Load balancing across multiple active paths between data centers.
The Trombone effect when migrating VMs across a WAN.
When VM1 is moved from one Hypervisor in DC1 to the other Hypervisor in DC2, the default GW for VM1 still resides on DC1. When VM1 sends traffic to VM2, the traffic will traverse the core before tromboning back to DC2.
EVPN solves this. EVPN is a similar technology to VPLS except that mac addresses are learned and exchanged through the control plane using BGP as the transport protocol. A new BGP family is introduced called EVPN.
bgp {
group IBGP {
local-address 1.1.1.1;
family evpn {
signaling;
}
neighbor 2.2.2.2;
}
}
First an understanding of how EVPN works.
In a multi-tenant environment, each tenant will correspond to an EVPN instance (EVI). Route Distinguishers are used to distinguish between each EVI and Route Targets are used to share learned mac addresses between EVIs.
For mac learning, each PE router snoops for DHCP and/or ARP(IPv4)/ND(IPv6) packets for a particular EVI. The PE can then advertise the locally learned MAC address to remote PE nodes through MP-iBGP. MAC addresses are aggregated and a MAC prefix is advertised rather than advertising every single MAC address, thus allowing the ability to scale thousands of MAC addresses. When a remote PE receives this bgp update it will extract the mac address and build a table with the next-hop pointing to the LSP of the advertising PE. Because this is BGP, policies can be created to filter and manipulate forwarding decisions.
When a local PE router sees an ARP request for an IP address and if the PE router has the MAC address binding for that IP address across the wan, the PE router performs a proxy ARP and responds to the ARP Request and can make the forwarding decision locally. This reduces (BUM) flooding (Broadcast, Unknown Unicast and Multicast) across WAN links.
Gateway IP and MAC addresses syncing in EVPN allows the host to use the nearest gateway to route traffic. You do this by creating IRBs on both PEs using different GW IP addresses. To accomplish this IRBs (IP + MAC addresses) are advertised using a BGP extended community. When VM1 migrates to DC2, it sends packets to the mac address associated to GW IP address of DC1. The IRB in DC2 notices that the destination mac address for these packets is across the WAN, so it does the routing locally. When the arp entry for the GW in VM1 expires, the VM will arp again and the IRB in DC2 will send a reply to VM1 with it's updated mac address.
Another thing that happens when VM migration is performed in an EVPN network, the MAC address of the VM is now advertised in DC2, the PE in DC2 updates their mac table table while the PE in DC1 withdraws the entry.
To address fast convergence in a multi homed environment, a concept called an Ethernet Segment is introduced. The set of links connecting to two or more local PE routers are called an Ethernet Segment. Each segment has an unique identifier called an ESI. An ethernet tag is also used to identify each broadcast domain such a vlan. When an Ethernet segment fails, the local PE withdraws the corresponding Ethernet "route" from BGP which triggers all remote PE routers to update their forwarding tables to update the corresponding next-hop to the backup PE.
EVPN introduces Split Horizon. BUM flooding aka, Broadcast, Unknown unicast or Multicast traffic are encapsulated in a MPLS packet with the Ethernet Segment Identifier. This allows the Egress PE to make a forwarding decision and prevents loops, because the PEs know where the packet originated from.
This in turn makes it possible to forward traffic over multiple active links through the WAN and allows for the ability to load balance.
With these advantages EVPN makes it a viable choice for interconnecting Data Centers.
1) Workload Mobility aka VM migration from one DC to another.
2) Fast convergence in a multi homed environment.
3) Load balancing across multiple active paths between data centers.
The Trombone effect when migrating VMs across a WAN.
When VM1 is moved from one Hypervisor in DC1 to the other Hypervisor in DC2, the default GW for VM1 still resides on DC1. When VM1 sends traffic to VM2, the traffic will traverse the core before tromboning back to DC2.
EVPN solves this. EVPN is a similar technology to VPLS except that mac addresses are learned and exchanged through the control plane using BGP as the transport protocol. A new BGP family is introduced called EVPN.
bgp {
group IBGP {
local-address 1.1.1.1;
family evpn {
signaling;
}
neighbor 2.2.2.2;
}
}
First an understanding of how EVPN works.
In a multi-tenant environment, each tenant will correspond to an EVPN instance (EVI). Route Distinguishers are used to distinguish between each EVI and Route Targets are used to share learned mac addresses between EVIs.
For mac learning, each PE router snoops for DHCP and/or ARP(IPv4)/ND(IPv6) packets for a particular EVI. The PE can then advertise the locally learned MAC address to remote PE nodes through MP-iBGP. MAC addresses are aggregated and a MAC prefix is advertised rather than advertising every single MAC address, thus allowing the ability to scale thousands of MAC addresses. When a remote PE receives this bgp update it will extract the mac address and build a table with the next-hop pointing to the LSP of the advertising PE. Because this is BGP, policies can be created to filter and manipulate forwarding decisions.
When a local PE router sees an ARP request for an IP address and if the PE router has the MAC address binding for that IP address across the wan, the PE router performs a proxy ARP and responds to the ARP Request and can make the forwarding decision locally. This reduces (BUM) flooding (Broadcast, Unknown Unicast and Multicast) across WAN links.
Gateway IP and MAC addresses syncing in EVPN allows the host to use the nearest gateway to route traffic. You do this by creating IRBs on both PEs using different GW IP addresses. To accomplish this IRBs (IP + MAC addresses) are advertised using a BGP extended community. When VM1 migrates to DC2, it sends packets to the mac address associated to GW IP address of DC1. The IRB in DC2 notices that the destination mac address for these packets is across the WAN, so it does the routing locally. When the arp entry for the GW in VM1 expires, the VM will arp again and the IRB in DC2 will send a reply to VM1 with it's updated mac address.
Another thing that happens when VM migration is performed in an EVPN network, the MAC address of the VM is now advertised in DC2, the PE in DC2 updates their mac table table while the PE in DC1 withdraws the entry.
To address fast convergence in a multi homed environment, a concept called an Ethernet Segment is introduced. The set of links connecting to two or more local PE routers are called an Ethernet Segment. Each segment has an unique identifier called an ESI. An ethernet tag is also used to identify each broadcast domain such a vlan. When an Ethernet segment fails, the local PE withdraws the corresponding Ethernet "route" from BGP which triggers all remote PE routers to update their forwarding tables to update the corresponding next-hop to the backup PE.
EVPN introduces Split Horizon. BUM flooding aka, Broadcast, Unknown unicast or Multicast traffic are encapsulated in a MPLS packet with the Ethernet Segment Identifier. This allows the Egress PE to make a forwarding decision and prevents loops, because the PEs know where the packet originated from.
This in turn makes it possible to forward traffic over multiple active links through the WAN and allows for the ability to load balance.
With these advantages EVPN makes it a viable choice for interconnecting Data Centers.
Subscribe to:
Posts (Atom)