Eric Stewart: Running Off At The Mouth

The 100Gb project at $JOB, for those who haven’t read the previous articles, involves six 7700 chassis: two 7706 chassis taking on the role of our campus backbone, two 7706 chassis going into two of our busiest nodes (one each, paired with an older Nexus 7000 as a backup), and for our data center, two Nexus 7710 chassis.  It’s those last two that I will be responsible for, and what most of my articles will refer to.

Factoid common to both 100Gb and 40Gb breakout: it applies to SR connectivity only.  The result is multiple 10GBASE-SR connections, each of which should be compatible with any 10GBASE-SR optic for the other end.

100Gb Breakout: Why, and Why Not Yet

These 7710s were ordered with two 12 port 100Gb blades, two supervisors, and one 40Gb blade.  According to the data sheet, the 100Gb CPAK-100G-SR10 is capable of 10x10Gb operation; at Cisco Live, I asked for clarification from one of the presenters about 100Gb breakout capability and he mentioned 8x10Gb as a possibility.  Command options suggest 8x10Gb is the reality, except that when you try the command:

switch(config)# interface breakout module 1 port 2 map ?

 10g-8x Breaks out a 100G high BW front panel port into 8 10G ports

switch(config)# interface breakout module 1 port 2 map 10g-8x
ERROR: Breakout is not support on interface Ethernet1/2

We’ve been told it’s not available yet.  I understand very little about the cabling that would be involved in using a 100Gb port as a breakout, aside from the fact that it would involve MTP-24 OM3 cabling (multimode cable containing 24 strands of fiber in two rows of twelve at the connector; the SR10 optic would use 20 – 10 for transmit, 10 for receive).  Exactly which strands do which in the MTP-24 is for another time and the standard (if it exists) is very new, and how the barrels (the connection going into the cartridge and the splitter internal to the cartridge) are configured, how the pin outs are done with a given cable, and how the cartridges that the 10Gb connections plug into is wired can vary from vendor to vendor quite a bit.

The plan involves putting the two 7710s together into a vPC pair, and as far as we can tell, will involve multiple connections between the two.  One of those connections, the “keepalive”, doesn’t need to be a full 100Gb – but should be redundant, including blade redundant.  In addition, our firewall connectivity involves a 30Gb (3x10Gb) connection (to each 7710), and the plan was to make that fully redundant as well.  Using two broken out ports (one on each 100Gb blade) and another on the 40Gb blade, we could have the redundancy we needed.

Well, those plans went out the window when we saw the ERROR above (the keepalive will be 100Gb and the other connections will all be homed on the 40Gb blade – separate “parent” ports – until 100Gb breakout is available).  And so ends the 100Gb breakout portion of this post.

40Gb Breakout: What I Think I Know/What I’ve Seen

But, as for the 40Gb breakout, that’s working.  And it’s fairly simple – in theory, anyway.

40Gb (40GBASE-SR4 QSFP) connections use an MTP-12 connection – the cables use 12 strands, which the plugs have in a single line.  For these short run lines, the transmit and receive works something like:

01   02   03   04   05   06   07   08   09   10   11   12
Rx1  Rx2  Rx3  Rx4  NA   NA   NA   NA   Tx4  Tx3  Tx2  Tx1

Note the pin number designations; “7” and “12” are on the other side

Given the right cable (Panduit makes a nice reversible key/gender changeable cable for this), you can, if need be, simply flip one end of the cable over to change the transmit/receive configuration.  We worked with OCC to get the cartridges wired just right, with six LC connector pairs set up vertically:

01  02  03  04  05  06
12  11  10  09  08  07

This means that for a 4x10Gb connection, two plugs are actually unused (pairs 5/8 and 6/7).

On the 7710, prior to running the breakout command, the port appears like any other port (IE, “Eth10/9”).

switch# show int status | inc 10/9
Eth10/9 -- disabled routed auto auto QSFP-40G-SR

But, run:

switch(config)# interface breakout module 10 port 9 map 10g-4x
switch(config)#

It may take a bit, but seeing no errors is a good thing.  And the port is broken out:

switch# show int status | inc 10/9
Eth10/9/1 -- disabled routed auto auto QSFP-40G-SR
Eth10/9/2 -- disabled routed auto auto QSFP-40G-SR
Eth10/9/3 -- disabled routed auto auto QSFP-40G-SR
Eth10/9/4 -- disabled routed auto auto QSFP-40G-SR

LIU with cartridge installed (not yet sealed back up)

No shut the ports (and maybe put them in switchport mode), and connect one to the other using a standard OM3 LC to LC cable:

switch# show int status | inc 10/9
Eth10/9/1 -- notconnec 1 auto auto QSFP-40G-SR
Eth10/9/2 -- notconnec 1 auto auto QSFP-40G-SR
Eth10/9/3 -- connected 1 full 10G  QSFP-40G-SR
Eth10/9/4 -- connected 1 full 10G  QSFP-40G-SR

And there ya go.

Things that can bite you in the butt:

• Someone telling you that the first and last ports will be the “dark” ones.
• Being fully aware of the “1” and “6” labels indicating the top of the cartridge, but still putting it in to the LIU upside down.
• Needing to flip your MTP 12 cable because it’s going straight through instead of crossover.

And that’s the quick and dirty write-up of 40Gb breakout to 4x10Gb connectivity.

EDIT 20140724: Right now the activity light is amber, even when at least two of the interfaces are up.  I’ll be playing with it later today to see if I can get it to change colors.