Eric Stewart: Running Off At The Mouth

Assuming you’ve read at least one of the previous Nexus 7700 posts, you know my task for our 100Gb project is to upgrade two Catalyst 6500 switches to two Nexus 7710s.  This will involve supporting Virtual Port Channels on the 7710s.  It’s taken a lot of my concentration these days to get these two switches up and running; they’re still not passing production data yet, but hopefully soon.

Part of the upgrade process involves not only replacing the 6500s, but moving some connections off of a couple of Nexus 5596 switches that are in a vPC pair onto the 7710s.  I was in the process of configuring the vPC interfaces for this portion of the upgrade (keeping the physical interfaces “shutdown” but putting the interfaces into “port channels” and then telling the port channels that they were part of a vPC connection) when I stumbled across a somewhat annoying behavior, and that triggered this post.

There’s a lot of ground to cover in this post, so, sorry if it gets a little long.

So What Are We Talking About Here?

A “port channel”, or more accurately, “EtherChannel” link is a method by which multiple links between two devices can be “bonded” together to provide additional bandwidth and redundancy; even if a single connection in an EtherChannel group breaks/goes down, the overall link stays up and there is (essentially) no interruption in service (just a loss of bandwidth).  At the very beginning, before two ports can properly form an EtherChannel connection (usually using a control protocol – LACP becoming the defacto standard these days), they must have (with few exceptions, like “description”) identical configurations.  Once the channel is configured/formed, a virtual interface that represents the combined links is created – and will have much of the same configuration as the member ports.  In addition, (most) changes made to the “PortChannel” interface are duplicated to the member interfaces.

Thing is, there still is a single point of failure: the switch itself.  Wouldn’t it be great if you could get this behavior but actually use two switches on either end instead of just one?

Cisco’s Two Ways To Do That

There’s VSS (Virtual Switching System) which essentially combines two switches into one virtual device.  From what I’ve seen of this, I hope I never have to touch it; for the most part, you configure only one device even though there are two individual devices.  It seems to be some kind of “master/slave” configuration, where the slave can become the master if connectivity to the the original master is lost.

vPC (Virtual Port Channels) allow two independent switches to present to “the other end” a single virtual interface between them.  In short, you configure the vPC interconnectivity of the two switches, and when forming “PortChannel” interfaces, tell the virtual interface that it is actually a member of a vPC.  This provides the switch/chassis redundancy that’s missing from regular EtherChannel connections.

So What Do You Need?

Well, at the very least, two (preferably EtherChanneled) links between the two devices. One as a “keepalive” link (in our case, a layer 3 point-to-point port channel, with connections on separate blades, directly connected between the two switches):

dc-7700-1# show run
...
vpc domain 77
  peer-keepalive destination 10.0.0.102 source 10.0.0.101 vrf default
...
interface port-channel1
  description vPC peer-keepalive
  ip address 10.0.0.101/30

(assume that .102 is the other 7700 and has a similar config on it’s end).  And one to carry the actual data between the two switches (in our case, a trunk port channel, again directly connected between the switches and with connections on separate blades):

interface port-channel77
  description vPC Peer Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan     ! every VLAN you need vPC support for,
  switchport trunk allowed vlan add ! or ALL of them
  spanning-tree port type network
  vpc peer-link

Mind you, my configuration isn’t live yet and I’m learning about this as I go along.  It’s very possible in your situation that you would need a third link that would carry just the VLANs that will never be used on vPC links.  Follow my often sanitized and trimmed configuration suggestions at your own risk.

So How Do I Use It?

Set your physical interface up to be a port-channel member:

interface Ethernet10/1/1
  lacp rate fast
  channel-group 1081 mode active

Then in the port-channel interface config, indicate that it’s a vpc port-channel (this number should match on both vPC switches, but don’t have to relate in any way to the port-channel designation):

interface port-channel1081
  vpc 1021

There’s commands you can use to verify that the configuration is properly “consistent” between the two vPC member switches; I use them below to illustrate a minor annoyance I’ve run across.

Once you’ve got this all set up, you can have any EtherChannel capable device (vPC or not) on the other end; my first working case at $JOB involved using this to provide a dual-uplink to a downstream switch that just saw the two connections as a properly bonded EtherChannel connection.

What Annoyance Is That?

Well, as I’m prepping the switch as a replacement to existing equipment, I don’t have live connections to test everything with yet.  We also have monitoring checks that inform us of oddities in optical connections, so I’m leaving the physical connections in “shutdown” state while I’m working on the configuration.  The annoyance is that, if a vPC port channel is down, it doesn’t actually do any consistency checking. (Also, some of this may depend on exactly when you put a physical interface involved into “shutdown”.)  However, it appears to do it on the first switch the vPC port channel is configured on, per Po1071 – 1073 below, and Po1074 below that:

dc-7700-1# show vpc brief 
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 77  
Peer status                       : peer adjacency formed ok      
vPC keep-alive status             : peer is alive                 
Configuration consistency status  : success 
Per-vlan consistency status       : success                       
Type-2 consistency status         : success 
vPC role                          : primary                       
Number of vPCs configured         : 5   
Peer Gateway                      : Enabled
Peer gateway excluded VLANs       : -
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans    
--   ----   ------ --------------------------------------------------
1    Po77   up     1,(snip)

vPC status
----------------------------------------------------------------------
id   Port      Status Consistency Reason                  Active vlans
--   ----      ------ ----------- ------                  ------------
1024 Po1024    up     success     success                    (snip)
1071 Po1071    down*  success     success                    -               
1072 Po1072    down*  success     success                    -               
1073 Po1073    down*  success     success                    -               
1074 Po1074    down*  Not         Consistency Check Not      -               
                      Applicable  Performed

And then on the other one:

dc-7700-2# show vpc bri
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 77  
Peer status                       : peer adjacency formed ok      
vPC keep-alive status             : peer is alive                 
Configuration consistency status  : success 
Per-vlan consistency status       : success                       
Type-2 consistency status         : success 
vPC role                          : secondary                     
Number of vPCs configured         : 5   
Peer Gateway                      : Enabled
Peer gateway excluded VLANs       : -
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans    
--   ----   ------ --------------------------------------------------
1    Po77   up     1,(snip)

vPC status
----------------------------------------------------------------------
id   Port      Status Consistency Reason                  Active vlans
--   ----      ------ ----------- ------                  ------------
1024 Po1024    up     success     success                    (snip)
1071 Po1071    down*  Not         Consistency Check Not      -               
                      Applicable  Performed                                  
1072 Po1072    down*  Not         Consistency Check Not      -               
                      Applicable  Performed                                  
1073 Po1073    down*  Not         Consistency Check Not      -               
                      Applicable  Performed                                  
1074 Po1074    down*  success     success                    -

So you can’t be 100% sure that you typed everything correctly unless you bring all the interfaces up and actually have a live connection on the other end.  This is, at best, inconvenient, because it would be nice if you could have some assurance that, even with the interface “down”, it was properly consistent across devices.  At worst, it’s misleading, since on the first device the port was created/configured on, it says it actually is consistent.

So that’s pretty much it for now; don’t be surprised if there’s edits to this page later as I learn more!