Data Center and Lab best practices

I saw my buddies Mark Honer of VM Ware and Dan Luderville from Gigamon speak at a Mirapath Sponsored Data Center Event. Now while I'm not much of a Data Center Infrastructure technologist specializing on the physical Underlay (Layer 1),  I do like to sit in on events like these to see where the trend is going and what problems these guys are seeing.  Mark is a very funny guy and likes to add humor in his presentations. One of the topics of interest is the Data Center Best Practices. We do these in our lab, so while this may seem common sense, you'd be surprised at how many other labs and data centers I've seen that doesn't.


I'll illustrate a few of them.

1) Use copper and fiber cables with unique serial number and lengths at both ends for tracing connections. Now you don't need serial numbers for inventory purposes. Most cables don't come like this, but Mirapath can do this as part of your order. Saves a lot of time especially when the distance is more than 1 meter. The cool thing is that Mirapath keeps a record of your serial number so when you order more, you don't have to remember where you last left off.

2. Use blanks in spaces were equipment are not installed. First it keeps the airflow going and two it prevents people from using your equipment as a shelf.

3. Using colored cables.

You should standardize around a particular color for each kind of connection.

We use white for our management connections and red for our console connections. Whenever we see some other color connected to the switch, we know someone has thrown in a rouge device and can easily track down who did it.

One of these kids don't look like the other

4. Buy special colored power cables for critical devices. If all your power cables are the same, which usually come in black, someone might pull your critical device from the network because they might not know which cable goes to which device. Now if you have some that are red, then they know not to unplug that device.

We call these the 'your fired' cables. We've had no incidents of accidental pulling of cables when we had these installed.

5. Invest in a nice labeler. You don't know how much time you can save if you just label things correctly. I used to spend many hours having to physically go back and forth between my desk and the rack just to figure out which hardware device I'm trying to find in the lab. If I had a labeler at that time to mark the devices with names, ips and basic information, I wouldn't have wasted my time.

These techniques are good for the small and young companies who are growing. The right best practices can prevent your company into turning into one of these.

Juniper olive console port issue

So I decided to try out the Juniper olive I've been hearing about. I looked online and noticed how people were able to create a virtual Junos device on a virtualization platform like VMWare ESXi. The one problem that all the users were having was that the images were meant to be installed onto an actual router and configuring the device meant connecting to it via a console port. The problem there is that in a virtual platform, there is no physical access to the console port.

What you end up seeing on the video console is this message:

If only you could modify the image so that the /boot/loader.conf file so instead of reading:

console="comconsole"

it could be modify to project on the video console.

console="comconsole, vidconsole"

I found out that VMWare has the ability to allow remote serial connectivity through telnet. First you add a serial port.

Then in the pull down menu, you choose network.

Next you select Direction and choose Server. In the port URL you'll need to map the port number to a new value that is not a well know port number. I chose 10026. You add the url

telnet://:<your_port_number>

Then after you spin up the VM. You access the new VM by telneting to the Host Server's IP and the custom port number you created. It may take a while, but eventually you'll get a prompt.

My_Mac% telnet 10.11.39.145 10026
Trying 10.11.39.145...
Connected to 10.11.39.145.
Escape character is '^]'.


Amnesiac (ttyd0)

login:root



Next you can edit the /boot/loader.conf file or add an IP address and turn on telnet/ssh service on the Virtual Junos so you can directly access the device through a management port.

Use Zero Touch Provisioning (ZTP) to auto-configure and upgrade new or replacement switches in a datacenter.

A Typical Data Center can host 10s if not hundreds of Top of the Rack (TOR) Switches. Managing and configuring each one of these can become a tedious task. Replacing a switch that goes out of service is just as time consuming. ZTP is an automation method that reduces the amount of time, minimizes errors and the need for a Network Engineer to be on location. You would only need a junior engineer or technician to re-cable links, rack the units and power them on without having to console in or add any configuration.


HOW ZTP WORKS

ZTP uses a combination of DHCP and TFTP/FTP/HTTP servers for dynamically allocating ip addresses, uploading configuration and upgrading switch software images. Juniper EX and QFX switches automatically default to ZTP on boot up and basically become DHCP clients.

To start you would configure a DHCP server, modifying the dhcpd.conf file by adding a few options: DHCP option 43 with vendor specific information sub options and DHCP option 150 or 66 which contains the address of the TFTP server. On the TFTP or FTP server you would archive all your switches' configurations and software images.

On your linux server the dhcpd.conf file would look similar to this:

host <EX SWITCH NAME> {
hardware ethernet 4c:96:14:e5:a3:41; ## MAC address of the management interface, you can also use the dynamic IP allocation and also we can use any of the network port's (MAC add# chassis mac +1) for ZTP
  fixed-address 100.1.1.90;     # Switch's irb ip address
  option option-150 100.1.1.1; # TFTP Server address to download config and image
  option host-name "EX4300-1";
  option VENDOR_OP.image-file-name "jinstall-ex-4300-13.2X51-D20.2-domestic-signed.tgz";
  option VENDOR_OP.config-file-name "PE3713320070.conf";
  option VENDOR_OP.transfer-mode "tftp";
  option VENDOR_OP.image-file-type "symlink";
}

ZTP works on untagged interfaces on any ports on the switch (data ports or management ports).

If you were to console into the box during a ZTP sequence it would look like this:

---------------------------------------

Committing autoinstall config                                                 
                                                                              
FIRST THE SWITCH WILL TRY DHCP OVER THE MANAGMENT address. (VME or ME)
                                                                              
Auto Image Upgrade: DHCP OFFER Client vme.0: Invalid config, no file server information. OFFER REJECTED.                                                                              

If no server is reachable it will try all the interfaces that are up on the switch using the default vlan and a temporary irb.

It will then check the DHCP Options that are  passed between the server and the switch, noting the TFTP server IP address, the configuration file name and software image to be installed.

Auto Image Upgrade: DHCP Options for client interface irb.0:                  
ConfigFile: PE3713320070.conf ImageFile: jinstall-ex-4300-13.2X51-D20.2-domesti
c-signed.tgz Gateway: 100.1.1.1DHCP Server: 100.1.1.20 File Server: 100.1.1.1 O
ptions state: All options set                                                                              
                                                                              
Auto Image Upgrade: DHCP Client Bound interfaces: irb.0   vme.0                                                                                 
                                                                              
Auto Image Upgrade: DHCP Client Unbound interfaces: ge-0/0/0.0   ge-0/0/1.0   g
e-0/0/2.0   ge-0/0/3.0   ge-0/0/4.0   ge-0/0/5.0   ge-0/0/6.0   ge-0/0/7.0   ge
-0/0/8.0   ge-0/0/9.0   ge-0/0/10.0   ge-0/0/11.0   ge-0/0/12.0   ge-0/0/13.0 
 ge-0/0/14.0   ge-0/0/15.0   ge-0/0/16.0   ge-0/0/17.0   ge-0/0/18.0   ge-0/0/1
9.0   ge-0/0/20.0   ge-0/0/21.0   ge-0/0/22.0   ge-0/0/23.0   ge-0/0/24.0   ge-
0/0/25.0   ge-0/0/26.0   ge-0/0/27.0   ge-0/0/28.0   ge-0/0/29.0   ge-0/0/30.0
  ge-0/0/31.0   ge-0/0/32.0   ge-0/0/33.0   ge-0/0/34.0   ge-0/0/35.0   ge-0/0/
36.0   ge-0/0/37.0   ge-0/0/38.0   ge-0/0/39.0   ge-0/0/40.0   ge-0/0/41.0   ge
-0/0/42.0   ge-0/0/43.0   ge-0/0/44.0   ge-0/0/45.0   ge-0/0/46.0   ge-0/0/47.0
                                                                                 
                                                                              
Auto Image Upgrade: To stop, on CLI apply "delete chassis auto-image-upgrade" 
and commit                                                                              

The EX switch will then parse the dhcp response
                                                                              
Auto Image Upgrade: Active on client interface: irb.0                                                                              
                                                                              
Auto Image Upgrade: Interface::   "irb"                                       

Auto Image Upgrade: Server::      "100.1.1.1"                                 

Auto Image Upgrade: Image File::  "jinstall-ex-4300-13.2X51-D20.2-domestic-sign
ed.tgz"                                                                       

Auto Image Upgrade: Server File:: "PE3713320070.conf"                         

Auto Image Upgrade: Gateway::     "100.1.1.254"                                 

Auto Image Upgrade: Protocol::    "tftp"                                      
                                                                            
The EX switch will then download the config file and the software image. 
                                                                              
Auto Image Upgrade: Start fetching PE3713320070.conf file from server 100.1.1.1
 through irb using tftp                                                       
                                                                              
                                                                              
Auto Image Upgrade: File PE3713320070.conf fetched from server 100.1.1.1 throug
h irb                                                                         
                                                                              
                                                                              
Auto Image Upgrade: Start fetching jinstall-ex-4300-13.2X51-D20.2-domestic-sign
ed.tgz file from server 100.1.1.1 through irb using tftp              

If the installed version on the switch and the version on the tftp server are the same, then the upgrade process aborts.

Auto Image Upgrade: Aborting image installation of jinstall-ex-4300-13.2X51-D21
.1-domestic-signed.tgz received from 100.1.1.1 through irb: Installed and fetch
ed image version same                                                         
                       
If the images are not the same, the EX switch will auto upgrade.

Auto Image Upgrade: File jinstall-ex-4300-13.2X51-D20.2-domestic-signed.tgz fet
ched from server 100.1.1.1 through irb   
                                                                              
Auto Image Upgrade: To install /var/tmp/jinstall-ex-4300-13.2X51-D20.2-domestic
-signed.tgz image fetched from server 100.1.1.1 through irb                   
                                                                                                                                                             
WARNING!!! On successful image installation, system will reboot automatically 

Auto Image Upgrade: Installation of /var/tmp/jinstall-ex-4300-13.2X51-D20.2-dom
estic-signed.tgz image fetched from server 100.1.1.1 through irb is done, proce
eding for reboot of system                                                    
                                                                              
                                                                              
Broadcast Message from root@EX4300-1                                          
        (no tty) at 5:47 UTC...                                               
                                                                              
Auto image Upgrade: Stopped                                                   
                                                                              
                                                                              
*** System shutdown message from root@EX4300-1 ***                          

System going down in 1 minute                                                 

*** FINAL System shutdown message from root@EX4300-1 ***                    

System going down IMMEDIATELY     

### AFTER REBOOT
EX4300-1 (ttyu0)

login: jnpr
Password:

--- JUNOS 13.2X51-D20.2 built 2014-04-29 08:43:38 UTC
{master:0}
jnpr@EX4300-1>

The EX is now running the new version of code and the downloaded configuration file and is ready for production.
Here's the config I see on the switch after reboot. It matches the config I saved on the TFTP server.

------------
jnpr@EX4300-1# show
## Last changed: 2014-07-20 07:56:01 UTC
version 13.2X51-D21.1;
/*
 * dhcpd-generated /var/etc/dhcpd.options.conf
 * Version: JDHCPD release 13.2X51-D21.1 built by builder on 2014-05-29 13:06:11 UTC
 * Written: Sun Jul 20 07:49:45 2014
 */
system {
    host-name EX4300-1;
    root-authentication {
        encrypted-password "$1$byLFhlG6$my6QnZANcF7DqD9m9Op5s."; ## SECRET-DATA
    }
    login {
        user jnpr {
            uid 2005;
            class super-user;
            authentication {
                encrypted-password "$1$FNz57vVN$lQYXYBuxDKlPwtTBFQXWa0"; ## SECRET-DATA
            }
        }
    }
    services {                         
        ssh;
        telnet;
        web-management {
            http;
        }
    }
    syslog {
        user * {
            any emergency;
        }
        host 100.1.1.72 {
            any any;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
    ntp {
        server 100.1.1.73;            
    }
}
interfaces {
    ge-0/0/0 {
        unit 0 {
            family ethernet-switching {
                vlan {
                    members default;
                }
            }
        }
    }
    irb {
        unit 0 {
            family inet {
                address 100.1.1.90/24;
            }
        }
    }
}
vlans {
    default {
        vlan-id 1;
        l3-interface irb.0;
    }
}

dhcpd.conf file on your unix box
--------------------------------

#STARTING OPTIONS
option subnet-mask 255.255.255.0;
option routers 100.1.1.1;   # Default GW
option option-150 code 150 = ip-address;

#Vendor Specific Option
option space VENDOR_OP;        #Define the Vendor Specific Option called VENDOR_OP
option VENDOR_OP-encapsulation code 43 = encapsulate VENDOR_OP;
option VENDOR_OP.image-file-name code 0 = text;
option VENDOR_OP.config-file-name code 1 = text;
option VENDOR_OP.image-file-type code 2 = text;
option VENDOR_OP.transfer-mode code 3 = text;

# DHCP IP Pool for your PCs, etc.

subnet 100.1.1.0 netmask 255.255.255.0 {
  range 100.1.1.50 100.1.1.60;
  option routers 100.1.1.1;
  option broadcast-address 100.1.1.255;
  option subnet-mask 255.255.255.0;
  option domain-name-servers 8.8.8.8;
  option domain-name "mydomain.net";
}

### EX Switch entries

host EX4300-1 {
hardware ethernet 4c:96:14:e5:a3:41; ## MAC address of the management interface, you can also use the dynamic IP allocation
 and also we can use any of the network port's (MAC add# chassis mac +1) for ZTP
  fixed-address 100.1.1.90;     # Switch's irb ip address to be assigned
  option routers 100.1.1.1;     # Default GW in case tftp is on another subnet
  option option-150 200.1.1.1; # TFTP Server address to download config and image
  option host-name "EX4300-1";
  option VENDOR_OP.image-file-name "jinstall-ex-4300-13.2X51-D21.1-domestic-signed.tgz";
  option VENDOR_OP.config-file-name "PE3713320070.conf";
  option VENDOR_OP.transfer-mode "tftp";
  option VENDOR_OP.image-file-type "symlink";
  option log-servers 100.1.1.72;
  option ntp-servers 100.1.1.73;
}