There are trends of high density 10GbE connectivity in the data center that are increasing the needs for some to use 40GbE interfaces for uplink connectivity.  Because 40GbE requires a new type of optic, called a QSFP+, I’ve had many questions from customers and myself regarding the connectivity and cabling options.  Oddly enough,  it took talking to at least 5 Cisco Engineers that span San Jose to NYC to compile this data, so if you’ve like to correct or add anything here, please feel free to comment below.  

The specific questions and research I was doing was related to the Cisco Nexus 3000 series switches, namely the 3064 and 3016.  I state that because there is the chance that the QSFP+ could operate differently if using different switch types – and that’s per the TME of optic.  For those new to the Nexus 3000 series, the 3064 has 48 front facing ports of SFP+ (1G/10G) and 4 x 40GbE QSFP+ that could be used as uplinks.  The 3016 has 16 40GbE QSFP+ ports.   

The Nexus 7000 is constantly evolving and there seems to be more and more design parameters that have to be taken into consideration when designing Data Center networks with these switches.  I’m not going to go into each of the different areas from a technical standpoint, but rather try and point out as many of those so called “gotchas” that need to be known upfront when purchasing, designing, and deploying Nexus 7000 series switches.

Before we get started, here is a quick summary of current hardware on the market for the Nexus 7000.

1. Supervisor 1
   
2. Fabric Modules (FAB1, FAB2)
3. M1 Linecards (48 Port 10/100/1000, 48 Port 1G SFP, 32 Port 10G, 8 port 10G)
4. F1 Linecards (32 Port 1G/10G, F2 linecards, 48 Port 1G/10G)
5. Fabric Extenders (2148, 2224, 2248, 2232)
6. Chassis (7009, 7010, 7018)

Instead of writing about all of these design considerations, I thought I’d break it down into a Q & A format, as that’s typically how I end up getting these questions anyway.  I’ve ran into all of these questions over the past few weeks (many more than once), so hopefully this will be a good starting point, for myself as I tend to forget, and many others out there, to check compatibility issues between the hardware, software, features, and licenses of the Nexus 7000.  The goal is to keep the answers short and to the point.

Q: 
What are the throughput capabilities and differences of the two fabric modules (FAB1 & FAB2)?

A: 
It is important to note each chassis supports up to five (5) fabric modules.  Each FAB1 has a maximum throughput of 46Gbps/slot meaning the total per slot bandwidth available when there are five (5) FAB1s in a single chassis would be 230Gbps.  Each FAB2 has a maximum throughput of 110Gbps/slot meaning the total per slot bandwidth available when there are five (5) FAB2s in a single chassis would be 550Gbps.  The next question goes into this a bit deeper and how the MAXIMUM theoretical per slot bandwidth comes down based on which particular linecards are being used.  In other words, the max bandwidth per slot is really dependent on the fabric connection of the linecard being used.