Data Center and Lab best practices

I saw my buddies Mark Honer of VM Ware and Dan Luderville from Gigamon speak at a Mirapath Sponsored Data Center Event. Now while I'm not much of a Data Center Infrastructure technologist specializing on the physical Underlay (Layer 1),  I do like to sit in on events like these to see where the trend is going and what problems these guys are seeing.  Mark is a very funny guy and likes to add humor in his presentations. One of the topics of interest is the Data Center Best Practices. We do these in our lab, so while this may seem common sense, you'd be surprised at how many other labs and data centers I've seen that doesn't.


I'll illustrate a few of them.

1) Use copper and fiber cables with unique serial number and lengths at both ends for tracing connections. Now you don't need serial numbers for inventory purposes. Most cables don't come like this, but Mirapath can do this as part of your order. Saves a lot of time especially when the distance is more than 1 meter. The cool thing is that Mirapath keeps a record of your serial number so when you order more, you don't have to remember where you last left off.

2. Use blanks in spaces were equipment are not installed. First it keeps the airflow going and two it prevents people from using your equipment as a shelf.

3. Using colored cables.

You should standardize around a particular color for each kind of connection.

We use white for our management connections and red for our console connections. Whenever we see some other color connected to the switch, we know someone has thrown in a rouge device and can easily track down who did it.

One of these kids don't look like the other

4. Buy special colored power cables for critical devices. If all your power cables are the same, which usually come in black, someone might pull your critical device from the network because they might not know which cable goes to which device. Now if you have some that are red, then they know not to unplug that device.

We call these the 'your fired' cables. We've had no incidents of accidental pulling of cables when we had these installed.

5. Invest in a nice labeler. You don't know how much time you can save if you just label things correctly. I used to spend many hours having to physically go back and forth between my desk and the rack just to figure out which hardware device I'm trying to find in the lab. If I had a labeler at that time to mark the devices with names, ips and basic information, I wouldn't have wasted my time.

These techniques are good for the small and young companies who are growing. The right best practices can prevent your company into turning into one of these.

Juniper olive console port issue

So I decided to try out the Juniper olive I've been hearing about. I looked online and noticed how people were able to create a virtual Junos device on a virtualization platform like VMWare ESXi. The one problem that all the users were having was that the images were meant to be installed onto an actual router and configuring the device meant connecting to it via a console port. The problem there is that in a virtual platform, there is no physical access to the console port.

What you end up seeing on the video console is this message:

If only you could modify the image so that the /boot/loader.conf file so instead of reading:

console="comconsole"

it could be modify to project on the video console.

console="comconsole, vidconsole"

I found out that VMWare has the ability to allow remote serial connectivity through telnet. First you add a serial port.

Then in the pull down menu, you choose network.

Next you select Direction and choose Server. In the port URL you'll need to map the port number to a new value that is not a well know port number. I chose 10026. You add the url

telnet://:<your_port_number>

Then after you spin up the VM. You access the new VM by telneting to the Host Server's IP and the custom port number you created. It may take a while, but eventually you'll get a prompt.

My_Mac% telnet 10.11.39.145 10026
Trying 10.11.39.145...
Connected to 10.11.39.145.
Escape character is '^]'.


Amnesiac (ttyd0)

login:root



Next you can edit the /boot/loader.conf file or add an IP address and turn on telnet/ssh service on the Virtual Junos so you can directly access the device through a management port.

Another way to delete a VM instance in Openstack

For some reason I could not delete a VM in my Openstack Cloud Platform. I tried to terminate the instance in the Horizon GUI but that did not work. I also tried to do it from the cli via:

# nova delete <uuid>

That did not work either. So I researched in on the web and noticed that the instances are kept in two places. The first place is in the nova directory:

/var/lib/nova/instances
[root@sdnnode2 instances]# ls
51426435-9f72-4077-91a4-6c89ea53894f 58fd794e-048a-4135-b064-17fb6bb3682b  bbc44d49-62f7-4cdc-8e7f-564190c6f043  locks _base d0ce9950-752c-406f-91e0-3bd0ff0f488d  snapshots

looking into those uuid directory names contained the following:

[root@sdnnode2 51426435-9f72-4077-91a4-6c89ea53894f]# ls
console.log  disk  libvirt.xml

However deleting these files still would not delete the VM from appearing in the Horizon GUI. These are the files that Openstack would use to spin up the VM.

The vm existed in MySQL.

I found a handy sql_guide to brush me up on SQL commands. Then I had to access the SQL server with the correct credentials. If you don't remember it you can find it in the nova.conf file located in:

/etc/nova/nova.conf

[root@sdnnode2 nova]# cat nova.conf | grep sql
sql_connection = mysql://nova:<mypassword>@10.161.38.4/nova

Then to access the SQL server:
[root@sdnnode2 nova]# sudo mysql -u nova -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 166
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

show databases gives us.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| nova               |
| test               |
+--------------------+



mysql> use nova;

 Database changed
mysql> show tables;
+--------------------------------------------+
| Tables_in_nova                             |
+--------------------------------------------+
| agent_builds                               |
| aggregate_hosts                            |
| aggregate_metadata                         |
| aggregates                                 |
| block_device_mapping                       |
| bw_usage_cache                             |
<TRUNCATED>
| volume_usage_cache                         |
| volumes                                    |
+--------------------------------------------+
115 rows in set (0.00 sec)


Here we can then find the instance uuid. First you could figure out the fields in a particular table:


mysql> describe key_pairs;
+-------------+--------------+------+-----+---------+----------------+
| Field       | Type         | Null | Key | Default | Extra          |
+-------------+--------------+------+-----+---------+----------------+
| created_at  | datetime     | YES  |     | NULL    |                |
| updated_at  | datetime     | YES  |     | NULL    |                |
| deleted_at  | datetime     | YES  |     | NULL    |                |
| id          | int(11)      | NO   | PRI | NULL    | auto_increment |
| name        | varchar(255) | YES  |     | NULL    |                |
| user_id     | varchar(255) | YES  | MUL | NULL    |                |
| fingerprint | varchar(255) | YES  |     | NULL    |                |
| public_key  | mediumtext   | YES  |     | NULL    |                |
| deleted     | int(11)      | YES  |     | NULL    |                |
+-------------+--------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)



Then you can list the objects that are in the field you are searching.

mysql> select name from key_pairs;
+----------+
| name     |
+----------+
| my-key   |
| root-key |
+----------+
2 rows in set (0.00 sec)


I found out there are 5 tables you need to modify to get delete the VM and the should be done in this order:
security_group_instance_association
block_device_mapping
instance_info_caches
fixed_ips
instances

So I needed to find the uuid of the VM I was going to delete.

mysql> select uuid,hostname from instances where deleted=0;
+--------------------------------------+-----------+
| uuid                                 | hostname  |
+--------------------------------------+-----------+
| bbc44d49-62f7-4cdc-8e7f-564190c6f043 | right-vm1 |
| d0ce9950-752c-406f-91e0-3bd0ff0f488d | left-vm1  |
| 51426435-9f72-4077-91a4-6c89ea53894f | test-vm    |
| 58fd794e-048a-4135-b064-17fb6bb3682b | left-vm2  |
+--------------------------------------+-----------+
4 rows in set (0.00 sec)

I then started my process. I checked to see if the instance was not already marked deleted by checking for the condition of "deleted=0";

mysql> select instance_uuid from security_group_instance_association where deleted=0;
Empty set (0.00 sec)

I didn't have a security group association so I proceeded with the next step.

mysql>  DELETE FROM block_device_mapping WHERE instance_uuid = "51426435-9f72-4077-91a4-6c89ea53894f";


Query OK, 1 row affected (0.00 sec)


I then followed the other steps

mysql> select instance_uuid from instance_info_caches where deleted=0;
DELETE FROM instance_info_caches WHERE instance_uuid = "51426435-9f72-4077-91a4-6c89ea53894f";
Query OK, 1 row affected (0.00 sec)

mysql>select instance_uuid from fixed_ips where deleted=0;
Empty set (0.00 sec)

I didn't have a fixed_ip so I went to the last step, which was to changed the "deleted" field.
You need to make sure the deleted number matches the id.

mysql> select id,deleted,uuid from instances;
+----+---------+--------------------------------------+
| id | deleted | uuid                                 |
+----+---------+--------------------------------------+
| 31 |      31 | 028ff14b-55f2-4aa2-8fd9-37d70ff26a3b |
| 11 |      11 | 4c60dd0a-ae81-45e0-931b-c996e89b10d7 |
| 47 |      47 | 4ca2687f-35a4-4248-9777-24d9317e4c20 |
| 25 |      25 | 512747be-b608-423e-ac81-54f2ebcd7e58 |
| 35 |      0 | 51426435-9f72-4077-91a4-6c89ea53894f |

mysql> UPDATE instances SET deleted = 40 where uuid = "51426435-9f72-4077-91a4-6c89ea53894f";Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select id,deleted,uuid from instances;
+----+---------+--------------------------------------+
| id | deleted | uuid                                 |
+----+---------+--------------------------------------+
| 31 |      31 | 028ff14b-55f2-4aa2-8fd9-37d70ff26a3b |
| 11 |      11 | 4c60dd0a-ae81-45e0-931b-c996e89b10d7 |
| 47 |      47 | 4ca2687f-35a4-4248-9777-24d9317e4c20 |
| 25 |      25 | 512747be-b608-423e-ac81-54f2ebcd7e58 |
| 35 |      35 | 51426435-9f72-4077-91a4-6c89ea53894f |


Finally the instance was no longer in my Openstack server.

[root@sdnserver1 nova]# nova show 51426435-9f72-4077-91a4-6c89ea53894f
ERROR: No server with a name or ID of '51426435-9f72-4077-91a4-6c89ea53894f' exists.

VXLAN for Layer 2 stretch over L3 network

I showed how EVPN with MPLS is used to stretch Layer 2 across Data Centers. Now I'll show how to stretch Layer 2 using VXLAN as the tunneling protocol. I'm not going to setup EVPN with VXLAN as that is a different technical method. This is a simple point to point VTEP setup to show how it works.

With Juniper EX9200s you can map vlans into a specific VXLAN tunnel or VTEP (Vxlan Tunnel End Point) The original L2 frame gets encapsulated into a VXLAN header. The outer header is an IP frame.

 

This allows it to cross a L3 network while retaining the original L2 frame. Communication to setup the tunnel is done through multicast.

 

Note: VXLAN tunnels can originate from the Hypervisor itself using vShield in VMware. This is another method if you want to have the tunnels originate in the underlay.

The EXs will need to be configured for PIM and an RP will be needed to build the Multicast tree. Multicast is used to interconnect the different VTEPs. It's used to optimize network traffic. Only End points listening for the multicast traffic will be forwarded frames. Other devices in the network will not receive this traffic.

 Each VTEP will need to have two things.

1) A VXLAN Network Identifier (aka VNI) which is like a dlci in Frame relay or vc-id in Point to Point Psuedo-wires in MPLS. 

2) An IP multicast address

When a L2 packet hits the switch, it will be encapsulated into an ip mulicast address + a vxlan header. This packet will then go to the RP to do the replication  to all the "receivers". In our case we only have 2 end points so the RP will only see 2 receivers.

Here's config snippets of how this is built.

EX1

First build the interface connecting to the LEAF switch.

set interfaces et-2/2/1 description TO-LEAF1

set interfaces et-2/2/1 unit 0 family ethernet-switching interface-mode trunk
set interfaces et-2/2/1 unit 0 family ethernet-switching vlan members v100

Add the core facing interface

set interfaces et-2/0/0 description TO-CORE1
set interfaces et-2/0/0 unit 0 family inet address 192.168.24.4/24
set interfaces et-2/0/0 unit 0 family iso

setup the EX to use a tunneling resource

set chassis fpc 9 pic 1 tunnel-services

Add your IGP flavor of choice and it's related config to exchange L3 information through the network.

set interfaces lo0 unit 0 family inet address 4.4.4.4/32
set interfaces lo0 unit 0 family iso address 49.0001.0040.0400.4004.00
set protocols isis interface all
set protocols isis interface fxp0.0 disable
set protocols isis interface lo0.0 passive

Configure PIM and point it to the RP

set protocols pim rp static address 192.168.0.1
set protocols pim interface lo0.0 mode bidirectional-sparse
set protocols pim interface et-2/0/0.0 mode bidirectional-sparse

Then map your VLAN into a VTEP

set vlans v100 vlan-id 100
set vlans v100 l3-interface irb.0
set vlans v100 vxlan vni 1
set vlans v100 vxlan multicast-group 239.1.1.1
set vlans v100 vxlan encapsulate-inner-vlan
set vlans v100 vxlan decapsulate-accept-inner-vlan

Then setup the switch to use it's ip address as the source of the tunnel

set switch-options vtep-source-interface lo0.0

On the RP you would only need to setup your IPs addresses and PIM configuration

set chassis fpc 1 pic 2 tunnel-services
set interfaces et-2/0/0 description TO-EX2
set interfaces et-2/0/0 unit 0 family inet address 192.168.35.3/24
set interfaces et-2/0/0 unit 0 family iso
set interfaces et-3/2/0 description TO-CORE1
set interfaces et-3/2/0 unit 0 family inet address 192.168.23.3/24
set interfaces et-3/2/0 unit 0 family iso
set interfaces et-3/2/1 description TO-CORE1
set interfaces et-3/2/1 unit 0 family inet address 192.168.123.3/24
set interfaces et-3/2/1 unit 0 family iso
set interfaces lo0 unit 0 family inet address 3.3.3.3/32 primary
set interfaces lo0 unit 0 family inet address 192.168.0.1/32
set interfaces lo0 unit 0 family iso address 49.0001.0030.0300.3003.00
set protocols isis interface all
set protocols isis interface fxp0.0 disable
set protocols isis interface lo0.0 passive
set protocols pim rp local family inet address 192.168.0.1
set protocols pim interface all mode bidirectional-sparse
set protocols pim interface fxp0.0 disable


You would create a similar VTEP on the remote EX

set chassis fpc 3 pic 0 tunnel-services
set interfaces et-2/0/0 description TO-CORE2
set interfaces et-2/0/0 unit 0 family inet address 192.168.35.5/24
set interfaces et-2/0/0 unit 0 family iso
set interfaces et-2/2/1 unit 0 family ethernet-switching interface-mode trunk
set interfaces et-2/2/1 unit 0 family ethernet-switching vlan members v100
set interfaces irb unit 0 family inet address 100.1.1.2/24
set interfaces lo0 unit 0 family inet address 5.5.5.5/32
set interfaces lo0 unit 0 family iso address 49.0001.0050.0500.5005.00
set protocols isis reference-bandwidth 40g
set protocols isis interface et-2/0/0.0
set protocols isis interface all
set protocols isis interface fxp0.0 disable
set protocols isis interface lo0.0 passive
set protocols pim rp static address 192.168.0.1
set protocols pim interface lo0.0 mode bidirectional-sparse
set protocols pim interface et-2/0/0.0 mode bidirectional-sparse
set protocols lldp interface all
set switch-options vtep-source-interface lo0.0
set vlans v100 vlan-id 100
set vlans v100 l3-interface irb.0
set vlans v100 vxlan vni 1
set vlans v100 vxlan multicast-group 239.1.1.1
set vlans v100 vxlan encapsulate-inner-vlan
set vlans v100 vxlan decapsulate-accept-inner-vlan

-----------------------

Once that is done you can check the connectivity

EX1

Check the PIM state and see if the Mcast Join was sent to the RP and if the multicast route is seen by the pim neighbor


user@EX1# run show pim join detail
Instance: PIM.master Family: INET
R = Rendezvous Point Tree, S = Sparse, W = Wildcard

Group: 239.1.1.1
    Source: *
    RP: 192.168.0.1
    Flags: sparse,rptree,wildcard
    Upstream interface: et-2/0/0.0           
    Downstream neighbors:
        Interface: Pseudo-VXLAN          

Group: 239.1.1.1
    Source: 4.4.4.4
    Flags: sparse,spt
    Upstream interface: Local                
    Downstream neighbors:
        Interface: pe-9/1/0.32770        
        Interface: Pseudo-VXLAN          

Group: 239.1.1.1
    Source: 5.5.5.5
    Flags: sparse,spt
    Upstream interface: et-2/0/0.0           
    Downstream neighbors:
        Interface: Pseudo-VXLAN      

Once traffic is flowing from the LEAF switches,  you can then check the vtep to see traffic statistics

user@EX1# run show vlans   

Routing instance        VLAN name             Tag          Interfaces
default-switch          v100                  100     
                                                                                  et-2/2/1.0*
                                                                                  vtep.32768*
  
user@EX1# run show interfaces vtep.32768 detail
  Logical interface vtep.32768 (Index 324) (SNMP ifIndex 604) (Generation 239)
    Flags: Up SNMP-Traps Encapsulation: ENET2
    VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 5.5.5.5, L2 Routing Instance: default-switch, L3 Routing Instance: default
    Traffic statistics:
     Input  bytes  :            508486320
     Output bytes  :            509589960
     Input  packets:               498516
     Output packets:               499598
    Local statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Transit statistics:
     Input  bytes  :            508486320              8158304 bps
     Output bytes  :            509589960              8158280 bps
     Input  packets:               498516                  999 pps
     Output packets:               499598                  999 pps
    Protocol eth-switch, MTU: 1600, Generation: 331, Route table: 6
      Flags: Trunk-Mode

The EX is acting as a switch so you can see the mac table and find out where the macs are learned


user@EX1# run show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC)


Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
    Vlan                MAC                 MAC         Age    Logical                NH        RTR
    name                address             flags              interface              Index     ID
    v100                00:00:05:ed:ad:49   D             -   et-2/2/1.0           <<<< local
    v100                00:00:05:ed:ae:01   D             -   vtep.32768         <<< over the vxlan tunnel 

Here are some useful vxlan commands to check when the number of vteps grow larger

user@EX1# run show ethernet-switching vxlan-tunnel-end-point source   
Logical System Name       Id  SVTEP-IP         IFL   L3-Idx
<default>                 0   4.4.4.4          lo0.0    0 
    L2-RTT                   Bridge Domain              VNID     MC-Group-IP
    default-switch           v100+100                   1        239.1.1.1     

user@EX1# run show ethernet-switching vxlan-tunnel-end-point remote   
Logical System Name       Id  SVTEP-IP         IFL   L3-Idx
<default>                 0   4.4.4.4          lo0.0    0 
 RVTEP-IP         IFL-Idx   NH-Id
 5.5.5.5          324       597     
    VNID          MC-Group-IP     
    1             239.1.1.1      

user@EX1# run show ethernet-switching vxlan-tunnel-end-point remote mac-table

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
           SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

Logical system   : <default>
Routing instance : default-switch
 Bridging domain : v100+100, VLAN : 100, VNID : 1
   MAC                 MAC      Logical          Remote VTEP
   address             flags    interface        IP address
   00:00:05:ed:ae:01   D        vtep.32768       5.5.5.5     

On the RP, you'll need to make sure Multicast is working.

CORE2
user@CORE2# run show pim join detail
Instance: PIM.master Family: INET
R = Rendezvous Point Tree, S = Sparse, W = Wildcard

Group: 239.1.1.1
    Source: *
    RP: 192.168.0.1
    Flags: sparse,rptree,wildcard
    Upstream interface: Local                
    Downstream neighbors:
        Interface: et-2/0/0.0            
        Interface: et-3/2/0.0            

Group: 239.1.1.1
    Source: 4.4.4.4
    Flags: sparse
    Upstream interface: et-3/2/1.0           
    Downstream neighbors:
        Interface: et-3/2/0.0 (pruned)

Group: 239.1.1.1
    Source: 5.5.5.5
    Flags: sparse,spt
    Upstream interface: et-2/0/0.0           
    Downstream neighbors:              
        Interface: et-3/2/0.0            


Now the question you might ask is why use EVPN + MPLS as there are a lot of configuration invovled. Well that's like comparing apples to oranges. VXLAN and MPLS should be compared a little more as they are both transport mechanisms. MPLS is far more superior to VXLAN as MPLS can do traffic engineering, bandwidth reservation and ~ 50 ms convergence on failure, while VXLAN relies on the underlying IGP for much of the decision making.