Juniper MC-LAG configuration and behavior

A customer had an unusual requirement. Their spine switches didn't have any out of band management connectivity and they were not yet going to run any IP protocols so we couldn't use a loopback and reditribute that into an IGP. Their spine switches were also utilizing MC-LAG. The problem was how to access the switches to manage them. We basically setup in-band ip addresses on the MC-LAG. These addresses would have to be reachable through static routes. The problem we encountered was, in an MC-LAG which member would be received as the owner of the ip address. There is an option called status-control which does this. I ran a test and found out that it seems to work opposite of what our expected behavior.

First EX1's status-control is set to standby and EX2 is active.

jnpr@EX1# set interfaces ae0 aggregated-ether-options mc-ae status-control standby

jnpr@EX2# set interfaces ae0 aggregated-ether-options mc-ae status-control active

I put IRBs on both the MC-LAG Spine switches and the QFX leaf on vlan-id 100. 100.1.1.1 is the MC-LAG's ip and 100.1.1.100 is the QFX just for this test.  From the QFX I try to access the Spine.

jnpr@QFX5100-LEAF# run show arp no-resolve  
MAC Address       Address         Interface     Flags
00:00:5e:00:01:02 10.161.39.254   vme.0                none
4c:96:14:6b:bb:f0 100.1.1.1       ae0.0                none
4c:96:14:f2:b6:e3 192.168.1.1     em2.32768            none
Total entries: 4

{master:0}[edit]
jnpr@QFX5100-LEAF# run telnet 100.1.1.1
Trying 100.1.1.1...
Connected to 100.1.1.1 
Escape character is '^]'.

EX1 (ttyp1)

login: ^C
Client aborted login
Connection closed by foreign host.

I'm in EX1?!


Then I change the status control

jnpr@EX1# set interfaces ae0 aggregated-ether-options mc-ae status-control active 

jnpr@EX2# set interfaces ae0 aggregated-ether-options mc-ae status-control standby 

{master:0}[edit]
jnpr@QFX5100-LEAF# run show arp no-resolve  
MAC Address       Address         Interface     Flags
00:00:5e:00:01:02 10.161.39.254   vme.0                none
a8:d0:e5:f7:bf:f0 100.1.1.1       ae0.0                none
4c:96:14:f2:b6:e3 192.168.1.1     em2.32768            none
Total entries: 6

{master:0}[edit]
jnpr@QFX5100-LEAF# run telnet 100.1.1.1      
Trying 100.1.1.1...
Connected to 100.1.1.1
Escape character is '^]'.

EX2 (ttyp1)

login: ^C
Client aborted login

Weird. Not sure why this behavior seems backwards.

So the next issue is how do you access the other MC-LAG member? There are two ways. You can either access it via the ip address that is using the iccp connection. Or if you have the resources you can basically have two MC-LAGs per Spine switch and make one of them standby on one IRB and the other standby on a different IRB say 101 so both Chassises are IP reachable for management.

MC-LAG configuration example


EX1
set chassis redundancy graceful-switchover
set chassis aggregated-devices ethernet device-count 2
set interfaces et-2/0/1 description TO-LEAF
set interfaces et-2/0/1 ether-options 802.3ad ae0
set interfaces et-2/2/1 description TO-LEAF
set interfaces et-2/2/1 ether-options 802.3ad ae0
set interfaces xe-3/1/0 description ICCP
set interfaces xe-3/1/0 unit 0 family inet address 200.1.1.1/30
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-priority 100
set interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:00:00:05
set interfaces ae0 aggregated-ether-options lacp admin-key 1
set interfaces ae0 aggregated-ether-options mc-ae mc-ae-id 1
set interfaces ae0 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae0 aggregated-ether-options mc-ae chassis-id 0
set interfaces ae0 aggregated-ether-options mc-ae mode active-active
set interfaces ae0 aggregated-ether-options mc-ae status-control standby
set interfaces ae0 unit 0 multi-chassis-protection 200.1.1.2 interface xe-9/1/1.0
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members all
set interfaces irb unit 100 family inet address 100.1.1.1/24
set vlans v100 vlan-id 100
set vlans v100 l3-interface irb.100


EX2
set chassis redundancy graceful-switchover
set chassis aggregated-devices ethernet device-count 2
set interfaces et-2/0/1 description TO-LEAF
set interfaces et-2/0/1 ether-options 802.3ad ae0
set interfaces et-2/2/1 description TO-LEAF
set interfaces et-2/2/1 ether-options 802.3ad ae0
set interfaces xe-3/1/0 description ICCP
set interfaces xe-3/1/0 unit 0 family inet address 200.1.1.2/30
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-priority 100
set interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:00:00:05
set interfaces ae0 aggregated-ether-options lacp admin-key 1
set interfaces ae0 aggregated-ether-options mc-ae mc-ae-id 1
set interfaces ae0 aggregated-ether-options mc-ae redundancy-group 1
set interfaces ae0 aggregated-ether-options mc-ae chassis-id 1
set interfaces ae0 aggregated-ether-options mc-ae mode active-active
set interfaces ae0 aggregated-ether-options mc-ae status-control active
set interfaces ae0 unit 0 multi-chassis-protection 200.1.1.1 interface xe-3/1/1.0
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members all
set interfaces irb unit 100 family inet address 100.1.1.1/24
set vlans v100 vlan-id 100
set vlans v100 l3-interface irb.100


EX1
# run show interfaces ae0 extensive
Physical interface: ae0 (MC-AE-1, active), Enabled, Physical link is Up
  Interface index: 186, SNMP ifIndex: 561, Generation: 189
  Link-level type: Ethernet, MTU: 1518, Speed: 40Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
  Source filtering: Disabled, Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 1bps
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Current address: 4c:96:14:6b:bb:c0, Hardware address: 4c:96:14:6b:bb:c0
  Last flapped   : 2014-07-25 17:13:08 PDT (1d 06:39 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :                 2141                    0 bps
   Output bytes  :             22808481                 2616 bps
   Input  packets:                   30                    0 pps
   Output packets:               313340                    5 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0, Policed discards: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0
  Ingress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    0                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                                0                    0                    0
  Egress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                            39562                39562                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                           253116               253116                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    1                   expedited-forwarding
    2                   assured-forwarding
    3                   network-control

  Logical interface ae0.0 (Index 348) (SNMP ifIndex 563) (Generation 177)
    Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Statistics        Packets        pps         Bytes          bps
    Bundle:
        Input :             0          0             0            0
        Output:        309312          4      28634313         2008
    Adaptive Statistics:
        Adaptive Adjusts:          0
        Adaptive Scans  :          0
        Adaptive Updates:          0
    Link:
      et-2/0/1.0
        Input :             0          0             0            0
        Output:        313340          4      29939385         2008
    LACP info:        Role     System             System      Port    Port  Port
                             priority          identifier  priority  number   key
      et-2/0/1.0     Actor        100  00:00:00:00:00:05       127       1     1
      et-2/0/1.0   Partner        127  4c:96:14:f2:b6:e0       127       2     1
    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx
      et-2/0/1.0            111233      111816            0            0
    Marker Statistics:   Marker Rx     Resp Tx   Unknown Rx   Illegal Rx
      et-2/0/1.0                 0           0            0            0
    Protocol eth-switch, MTU: 1518, Generation: 229, Route table: 6
      Flags: Trunk-Mode

EX2

show interfaces ae0 extensive
Physical interface: ae0 (MC-AE-1, active), Enabled, Physical link is Up
  Interface index: 219, SNMP ifIndex: 501, Generation: 351
  Link-level type: Ethernet, MTU: 1518, Speed: 40Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
  Source filtering: Disabled, Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 1bps
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Current address: a8:d0:e5:f7:bf:c3, Hardware address: a8:d0:e5:f7:bf:c3
  Last flapped   : 2014-07-25 17:13:10 PDT (1d 06:39 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :               153427                    0 bps
   Output bytes  :             13628185                 1832 bps
   Input  packets:                 2414                    0 pps
   Output packets:               193271                    1 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0, Policed discards: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0
  Ingress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    0                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                                0                    0                    0
  Egress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                      90356345631          90356345631                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                           402965               402965                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    1                   expedited-forwarding
    2                   assured-forwarding
    3                   network-control

  Logical interface ae0.0 (Index 343) (SNMP ifIndex 18551) (Generation 128345)
    Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Statistics        Packets        pps         Bytes          bps
    Bundle:
        Input :             0          0             0            0
        Output:        189232          1      13150169          512
    Adaptive Statistics:
        Adaptive Adjusts:          0
        Adaptive Scans  :          0
        Adaptive Updates:          0
    Link:
      et-2/2/1.0
        Input :             0          0             0            0
        Output:        193272          1      14459095          512
    LACP info:        Role     System             System      Port    Port  Port
                             priority          identifier  priority  number   key
      et-2/2/1.0     Actor        100  00:00:00:00:00:05       127   32769     1
      et-2/2/1.0   Partner        127  4c:96:14:f2:b6:e0       127       1     1
    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx
      et-2/2/1.0            111288      111983            0            0
    Marker Statistics:   Marker Rx     Resp Tx   Unknown Rx   Illegal Rx
      et-2/2/1.0                 0           0            0            0
    Protocol eth-switch, MTU: 1518, Generation: 37237, Route table: 3
      Flags: Trunk-Mode



LEAF

{master:0}[edit]
jnpr@QFX5100-LEAF#run show interfaces ae0 extensive
Physical interface: ae0, Enabled, Physical link is Up
  Interface index: 659, SNMP ifIndex: 550, Generation: 150
  Description: TO-EX2
  Link-level type: Ethernet, MTU: 1514, Speed: 80Gbps, BPDU Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 0
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Current address: 4c:96:14:f2:b7:a0, Hardware address: 4c:96:14:f2:b7:a0
  Last flapped   : 2014-07-25 16:49:38 PDT (1d 07:04 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :       12286985366334                 2832 bps
   Output bytes  :       12315040186254                 2208 bps
   Input  packets:         179102021645                    3 pps
   Output packets:         179048324703                    2 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0, Policed discards: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 4, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 12 supported, 5 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0 best-effort                    0             95319185                    0
    3 fcoe                           0                    0                    0
    4 no-loss                        0                    0                    0
    7 network-cont                   0               242393                    0
    8 mcast                          0         178948686348                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    3                   fcoe
    4                   no-loss
    7                   network-control
    8                   mcast

  Logical interface ae0.0 (Index 557) (SNMP ifIndex 553) (Generation 167)
    Flags: SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Statistics        Packets        pps         Bytes          bps
    Bundle:
        Input :             0          0             0            0
        Output:          3338          0        183668            0
    Link:
      et-0/0/50.0
        Input :             0          0             0            0
        Output:          5685          0       1470451            0
      et-0/0/51.0
        Input :             0          0             0            0
        Output:          4057          0       1386350            0
    LACP info:        Role     System             System      Port    Port  Port
                             priority          identifier  priority  number   key
      et-0/0/50.0    Actor        127  4c:96:14:f2:b6:e0       127       1     1
      et-0/0/50.0  Partner        100  00:00:00:00:00:05       127   32769     1
      et-0/0/51.0    Actor        127  4c:96:14:f2:b6:e0       127       2     1
      et-0/0/51.0  Partner        100  00:00:00:00:00:05       127       1     1
    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx
      et-0/0/50.0           111855      111309            0            0
      et-0/0/51.0           111853      111308            0            0
    Marker Statistics:   Marker Rx     Resp Tx   Unknown Rx   Illegal Rx
      et-0/0/50.0                0           0            0            0
      et-0/0/51.0                0           0            0            0
    Protocol eth-switch, MTU: 1514, Generation: 181, Route table: 3
      Flags: Trunk-Mode