Data Center and Lab best practices

I saw my buddies Mark Honer of VM Ware and Dan Luderville from Gigamon speak at a Mirapath Sponsored Data Center Event. Now while I'm not much of a Data Center Infrastructure technologist specializing on the physical Underlay (Layer 1),  I do like to sit in on events like these to see where the trend is going and what problems these guys are seeing.  Mark is a very funny guy and likes to add humor in his presentations. One of the topics of interest is the Data Center Best Practices. We do these in our lab, so while this may seem common sense, you'd be surprised at how many other labs and data centers I've seen that doesn't.


I'll illustrate a few of them.

1) Use copper and fiber cables with unique serial number and lengths at both ends for tracing connections. Now you don't need serial numbers for inventory purposes. Most cables don't come like this, but Mirapath can do this as part of your order. Saves a lot of time especially when the distance is more than 1 meter. The cool thing is that Mirapath keeps a record of your serial number so when you order more, you don't have to remember where you last left off.

2. Use blanks in spaces were equipment are not installed. First it keeps the airflow going and two it prevents people from using your equipment as a shelf.

3. Using colored cables.

You should standardize around a particular color for each kind of connection.

We use white for our management connections and red for our console connections. Whenever we see some other color connected to the switch, we know someone has thrown in a rouge device and can easily track down who did it.

One of these kids don't look like the other

4. Buy special colored power cables for critical devices. If all your power cables are the same, which usually come in black, someone might pull your critical device from the network because they might not know which cable goes to which device. Now if you have some that are red, then they know not to unplug that device.

We call these the 'your fired' cables. We've had no incidents of accidental pulling of cables when we had these installed.

5. Invest in a nice labeler. You don't know how much time you can save if you just label things correctly. I used to spend many hours having to physically go back and forth between my desk and the rack just to figure out which hardware device I'm trying to find in the lab. If I had a labeler at that time to mark the devices with names, ips and basic information, I wouldn't have wasted my time.

These techniques are good for the small and young companies who are growing. The right best practices can prevent your company into turning into one of these.

Creating my first OpenStack cloud platform and connecting it to a physical network.

In my quest to learn more about network virtualization I decided to learn more about OpenStack as a cloud platform. So I started from scratch and tried to install Openstack.

I took an old server which had two NIC ports and installed Centos 6.5. I then installed openstack all in one server from the following:

http://openstack.redhat.com/Quickstart

I decided to use the havana version because icehouse just came out and I wanted to wait until after the bugs were fixed.

After doing a bunch of yum installs I had it working. I was actually able to create a few VMs using the Cirros Linux image and created a simple network. I was able to ping from one VM to the other.

However the documentation was not clear on how to connect your VMs to the outside world through a physical network.

After searching around for a few days, I found this:

http://openstack.redhat.com/Neutron_with_existing_external_network

I decided to use eth0 as my management ip to access Openstack and use eth1 as the port for the VMs to connect to the physical network.

I also learned that OpenStack uses OpenVswitch and Neutron for network connectivity. You have to learn a little bit about OpenVswitch on a server.

This led me to make the following changes.
under  /etc/sysconfig/network-scripts

I duplicated (to backup) and changed the following files.

[root@centos-6-5-openstack network-scripts]# more ifcfg-eth1
DEVICE=eth1
TYPE=OVSPort
UUID=100083c1-6174-4a59-b3d0-09081eef106a
ONBOOT=yes
DEVICETYPE=ovs
HWADDR=00:30:48:F9:B9:C9
OVS_BRIDGE=br-ex
NAME="System eth1"

[root@centos-6-5-openstack network-scripts]# more ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.250.11
NETMASK=255.255.255.0
DNS=198.6.1.1
ONBOOT=yes

Then I tied the eth1 interface to the bridge "br-ex"

"br-ex" was automatically created on my server, but if it's not there, you should create a bridge (ova-vsctl add-br <NAME>)  and associate the physical interface (eth1) to that bridge.

[root@centos-6-5-openstack network-scripts]# port br-ex eth1

[root@centos-6-5-openstack network-scripts]# ovs-vsctl show
f6d22683-9529-48b6-b9d0-5e49cb720d44
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-d267267b-56"
            Interface "qg-d267267b-56"
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
        Port "eth1"
            Interface "eth1"
    Bridge br-int

I also had to make a change to a file:

Add to the /etc/neutron/plugin.ini file these lines:
network_vlan_ranges = physnet1
bridge_mappings = physnet1:br-ex

I had to make sure that eth1 is a port for bridge br-ex not br-int. For some reason it was not doing this, so I removed the port from br-int using ovs-vsctl del-port command

Red hat wants you to restart nework services after making the changes. However when I issued that command, in ifconfig I would see that eth1 still had an ip address and br-ex was not updated. So I rebooted the server.

I also used the following to troubleshoot
tcpdump -nei br-ex
tcpdump -nei eth1

After reboot all was well.

root@centos-6-5-openstack network-scripts]# ifconfig
br-ex     Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C9
          inet addr:192.168.250.11  Bcast:192.168.250.255  Mask:255.255.255.0
          inet6 addr: fe80::5822:16ff:fe25:91a/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4834 (4.7 KiB)  TX bytes:1336 (1.3 KiB)

br-int    Link encap:Ethernet  HWaddr BE:9D:E4:A4:B0:44
          inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:213 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:16262 (15.8 KiB)  TX bytes:468 (468.0 b)

eth0      Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C8
          inet addr:10.161.32.230  Bcast:10.161.39.255  Mask:255.255.248.0
          inet6 addr: fe80::230:48ff:fef9:b9c8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14400 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1686756 (1.6 MiB)  TX bytes:7627901 (7.2 MiB)
          Interrupt:18 Memory:d8020000-d8040000

eth1      Link encap:Ethernet  HWaddr 00:30:48:F9:B9:C9
          inet6 addr: fe80::230:48ff:fef9:b9c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:905 errors:0 dropped:0 overruns:0 frame:0
          TX packets:978 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:82360 (80.4 KiB)  TX bytes:87771 (85.7 KiB)
          Interrupt:19 Memory:d8060000-d8080000

int-br-ex Link encap:Ethernet  HWaddr 5A:EC:7F:16:11:35
          inet6 addr: fe80::58ec:7fff:fe16:1135/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:100 errors:0 dropped:0 overruns:0 frame:0
          TX packets:190 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4752 (4.6 KiB)  TX bytes:14550 (14.2 KiB)


From my physical gateway switch/router 192.168.250.1 I tried pinging the br-ex interface 192.168.250.11

Next I created the tenant subnet, vm and the openstack router.

First I had to source my admin credentials

source /root/keystonerc_admin

Next I followed the instructions on creating a public network. I added a floating ip.

http://openstack.redhat.com/Floating_IP_range

By default packstack creates a demo public network, with a subnet that I was not using. So I cleared the gw of this first.

# neutron router-gateway-clear router1
Then deleted the subnet
# neutron subnet-delete public

Next I recreated a new subnet
neutron subnet-create --name public --enable_dhcp=False --allocation-pool=start=192.168.250.10,end=192.168.250.20 --gateway=192.168.250.1 public 192.168.250.0/24

Then created a router using Horizon.
Then I tied the router to the public subnet.

neutron router-gateway-set test-rtr public

[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron subnet-show public
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "192.168.250.10", "end": "192.168.250.20"} |
| cidr             | 192.168.250.0/24                                     |
| dns_nameservers  |                                                      |
| enable_dhcp      | False                                                |
| gateway_ip       | 192.168.250.1                                        |
| host_routes      |                                                      |
| id               | 8942189b-8fbf-44e9-adbb-d62dd0d27015                 |
| ip_version       | 4                                                    |
| name             | public                                               |
| network_id       | 1f438ff9-43cf-4eb4-8b92-a385dc1dff8d                 |
| tenant_id        | 3a10de8a82444118865a6398b336ee68                     |
+------------------+------------------------------------------------------+

From Horizon, my setup looks simple:

One thing that threw me off during the whole process was trying to figure out if the openstack Gateway IP was working or not.

I noticed that the status kept saying down.

[root@centos-6-5-openstack network-scripts(keystone_admin)]# neutron port-show router-gw | grep status
| status                | DOWN  

After googling it, I found out that this was a bug and a display issue. So you can't tell if the gateway ip is actually working or not because the status is broken.


From my VM I am now able to access my physical switch