Currently in Migration - Jason Edelman's Old Blog
  • Home
  • About
  • Contact

QSFP+ Fiber Connectivity

3/30/2012

19 Comments

 
There are trends of high density 10GbE connectivity in the data center that are increasing the needs for some to use 40GbE interfaces for uplink connectivity.  Because 40GbE requires a new type of optic, called a QSFP+, I’ve had many questions from customers and myself regarding the connectivity and cabling options.  Oddly enough,  it took talking to at least 5 Cisco Engineers that span San Jose to NYC to compile this data, so if you’ve like to correct or add anything here, please feel free to comment below.  

The specific questions and research I was doing was related to the Cisco Nexus 3000 series switches, namely the 3064 and 3016.  I state that because there is the chance that the QSFP+ could operate differently if using different switch types – and that’s per the TME of optic.  For those new to the Nexus 3000 series, the 3064 has 48 front facing ports of SFP+ (1G/10G) and 4 x 40GbE QSFP+ that could be used as uplinks.  The 3016 has 16 40GbE QSFP+ ports.   

While it is common to state each QSFP+ supports native 40GbE or 4 individual 10GE interfaces, it’s not entirely true if you wanted to mix and match ports.  For example, on a 3016, if you wanted 14 interfaces configured for native 40GbE and the other 2 physical interfaces configured for 8 individual 10GbE interfaces, that is not supported (per the documentation that I have).  On the Nexus 3016, there are only 3 valid modes of operation.  These are1. all ports would have to operate at 10GbE, 2. all ports would have to operate at 40GbE, or 3. 8 ports at 40GbE (half) and 32 ports at 10GbE (half).  This is one point to verify when you have “flexibility” with hardware and optics.  Taken from a late 2011 Cisco slide deck, “At FCS the chassis can be configured in the following modes (with a reload): 16x40 GE ports, 32x10 GE ports and 8x40GE ports, 64x10GE ports.”  However, I actually haven’t been able to find that on Cisco.com, so if anyone knows if it’s changed, please let me know.  Since Cisco stated “at FCS,” I imagine and hope this caveat will go away in the short future.

If you simply wanted to cable up two Nexus 3000s with 40GbE, the options are multi mode fiber or Twinax copper.  I only cover the fiber throughout this post as that is where most of the questions are at this time.  If you’re looking into Twinax, the information out there is pretty solid, but if you still have questions, feel free to contact me.

Picture
So, since you are now using fiber, how do we connect these switches into the network?  You will first need to insert the QSFP+ optic similar to how you insert a fiber optic for standard 1G or 10G connectivity.  For the Nexus 3000, only multi mode fiber is available, so the Cisco part number needed is QSFP-40G-SR4.  This is the equivalent of the GLC-SX-MM or SFP-10G-SR, for 1G and 10G, respectively.  The connector type is no longer LC, but now is a MPO (multi-fiber push-on) connector. 


It is interesting to note that these cables actually have 12 fiber strands internal to them (not all being used) to achieve 40GbE.  Distance limitations are 100m using OM3 and 150m using OM4 fiber.  Because these cables are MPO connectors, have 12 strands, and are ribbon cables for native 40GbE, they will not be able to leverage any of your existing fiber optic cable plant.  Be prepared to home run these cable where needed throughout the data center.  

Picture
Picture
Picture
However, you may not always need native 40GbE between two switches.  Instead, you may opt to configure multiple 10GbE interfaces instead.  In this case, the QSFP-40G-SR4 is still needed, but the cable selection is different than what was previously shown above and the ability to use current cable plants becomes a possibility.  The cable required here would have an MPO connector on one end that would connect into the QSFP port and then “break out” into 4 individual fiber links on the other end.   These breakout cables terminate with LC male connectors.  

Picture
This is great that they terminate with LC male connectors because this allows customers to leverage the current cable infrastructure assuming existing patch panels have LC interfaces and OM3/OM4 fiber is in use throughout the data center.  These breakout cables are also nice if you want to attach a northbound switch that only supports 10GbE interfaces.  You can easily direct connect or jump through a panel in the data center to connect the Nexus 3000 via multiple 10GbE interfaces to a Nexus 7000 (or any other switch with 10GbE-only interfaces), for example.

Here is the caveat in my opinion. 

What if you want to inter-connect two different Nexus 3000s together via a 20GbE port-channel using MMF through the QSFP+ interfaces?  You would end up with each end having LC connectors with male interfaces.  Last time I checked, a male needs a female to make a connection.  Remember, this isn’t a problem if these switches are on different sides of the data center and you are going through a panel.  This is only an issue if you’re in small data center or colo and need to home-run these connections.  I suppose you can use fiber couplers that have two female sides to inter-connect each of the breakout cables, but that just seems messy and has more moving parts than desired.  But hey, there really doesn’t seem to be any other way around this.  If I’ve missed something, feel free to comment below.

For a design I was working recently that resembled what I was describing above, it wasn’t worth dealing with the ‘mess’ of cabling, couplers, and QSFP+ optics on a 3016, and since 40GbE wasn’t technically required, we chose to use 3064s in the Core with all native 10GE rather than 3016s with QSFP+’s.  Just some food for thought.
19 Comments
Aj
5/15/2012 06:51:41 am

Hello Jason,

Did you have any luck bringing up two Nexus 3064 switches tied together with QSFP-40G-SR4 with MPO fiber connector cable? I have a similar set up and port does not come up.

Thank you,
AJ

Reply
Jason Edelman link
5/15/2012 12:29:52 pm

Have you tried changing the port?

http://www.cisco.com/en/US/docs/switches/datacenter/nexus3000/sw/layer2/503_U3_1/b_Cisco_n3k_Layer_2_Switching_Config_503_U31_chapter_00100.html#task_4AF0989035354997AEDCBF4DC39E7E55

Reply
AJ
5/16/2012 12:40:38 am

Hardware profile is set for 48x10 and 4x40. Link would not even come up and I am using QSFP 40GB SR4 optic with MPO connector. Looks like there are different flavors of MPO connecto cables Like Piar flipped and Striaght. Not sure which one works for this set up.

Reply
Jason Edelman link
5/16/2012 10:58:42 pm

I don't have any equipment to test with, but please let me know what you find out.

Reply
Chris
11/28/2012 08:46:43 am

I have just gotten 2x 3064s up w/ QSFP+ in both modes (16x10G, and 4x 40G) ... so it is possible to get it working

Reply
Rod Hardesty
3/7/2013 01:31:38 am

If a QSFP+ port uses only two of the four SFP+ connections is it ok to leave the other two SFP+ connectors unterminated?

Reply
Jason Edelman link
3/11/2013 07:25:21 am

Rod - I do not see why not. Feel free to use them although it may be more capacity than you really need :)

Reply
Eddie Hing link
7/2/2013 03:34:39 pm

Hi, if I understand your question correctly, we would make an assembly with one MPO to two MPO's, each having two pairs of receive/ transmit wired through. We may have to make an educated guess on the pinouts but I'm quite confident we could work it out

Reply
Susmita Adhikari link
9/4/2013 12:59:51 am

Interesting read. We have been hearing similar break out issues from some of our customers. However, they are using a 40G QSFP LR optics, which outputs 4xCWDM signal via duplex SMF fiber. We have searched for the available solution but there are none. We are now working into making our own solution with the use of a multiplexer and demultipler so that you can physically break the 40G duplex fiber into 4 duplex fibers running at 10G.

Reply
David Jonta
10/10/2013 07:17:27 am

I really interested in one scenario ... Could you connect QSFP (40Gig native) port to 4x SFP+ on the other side and bundle those 4 ports as port channel in static mode? Will the link go up and will this solution work?

Reply
Christopher Shoop link
2/20/2014 10:17:43 am

This is a good blog.

The 12F MTP/MPO cable should be terminated so that the following fibers are paired:

1-12 (blue + aqua)
2-11 (orange + pink)
3-10 (green + purple)
4-9 (brown + yellow)
5-8 (slate + black)
6-7 (white + red)

If any of you need help with cables, cassettes, panels or design/install you can reach me at 513-779-5448. We also offer the MTP-LC breakout cables and QSFP+ to QSFP+ cables.

Reply
Mark
6/8/2014 06:33:19 pm

David, looks like QSFP+ breakout cables could work with 3rd party switches like Dell MXLs:

http://humairahmed.com/blog/?p=5455

Reply
Anthony Samuels
6/10/2014 05:08:33 am

I had a great experience in dealing with Jade Communications. Their customer service was outstanding and the quality of work was excellent. I will be recommending them to family and friends in Boca Raton. Visit their website at http://www.jade-com.com/ to check out more about them.

Reply
James
7/9/2014 10:06:18 pm

I have a Nexus 3064 with QFSP and MTP cable for 4 x 10G breakout. Switch will connect to host SFP+ interfaces. What we find is that -

The 1st and 4th 10G port on the QSFP work fine independently but Ports 2 and 3 only work if both ports 2 and 3 are connected, i.e. Port 2 or 3 will not work independently. If both are connected then we get link lights on both host SFP+ interfaces, if we disconnect port 2 then port 3 will also disconnect. Has anyone else observed this behaviour , is it correct, is it documented somewhere?.

Thanks!

Reply
George
7/26/2014 07:58:51 pm

Is it possible to connect a NAS linux server to several PC clients using a 40GB QSFP PCI card on the server and a 10GB SFP PCI card on the clients. A QSFP to 4xSFP breakout cable would be used to connect the client PCs to the server. Will this setup work? Also, on the server, will I see 4 x ethernet ports when doing a 'ifconfig' linux command on the server?

Reply
Venkata S. Dendukuri
8/19/2014 03:09:29 pm

Hi All,
with QSFP-40G-SR4 MPO connector, Can i cross connect to other switches, Ex: i have Nexus 5K switches A, B, C and D and I have populated QSFP-40G-SR4 modules in each switch ( I have ordered 4x2 QSFP modules), i have insufficient modules to interconnect between A, B, C and D Switches.
In the above scenario, Can i use 4x 10G of QSFP-40G-SR4 to B, C and D switches and similarly from other switches, Please reply at the earliest

Reply
GN
8/19/2014 08:13:45 pm

Hi all,

Any idea if there is a breakout cable MPO into 4x 10GE male LR (long range) instead of SR ?

Reply
eddie link
8/19/2014 08:42:16 pm

If you mean MPO to 4x LC duplex, then yes, can be single mode (LR) or Multi Mode (SR).

Reply
Steve mercatili
2/13/2015 12:38:42 am

I'm looking for the same. Doesn't exist for Cisco. I see it for Arista

Reply



Leave a Reply.

    Author

    Jason Edelman, Founder of Network to Code, focused on training and services for emerging network technologies. CCIE 15394.  VCDX-NV 167.


    Enter your email address:

    Delivered by FeedBurner


    Top Posts

    The Future of Networking and the Network Engineer

    OpenFlow, vPath, and SDN

    Network Virtualization vs. SDN

    Nexus 7000 FAQ

    Possibilities of OpenFlow/SDN Applications 

    Loved, Hated, but Never Ignored #OpenFlow #SDN

    Software Defined Networking: Cisco Domination to Market Education

    OpenFlow, SDN, and Meraki

    CAPWAP and OpenFlow - thinking outside the box

    Introduction to OpenFlow...for Network Engineers


    Categories

    All
    1cloudroad
    2011
    2960
    40gbe
    7000
    Arista
    Aruba
    Big Switch
    Brocade
    Capwap
    Christmas
    Cisco
    Controller
    Data Center
    Dell Force10
    Embrane
    Extreme
    Fex
    Hadoop
    Hp
    Ibm
    Isr G2
    Juniper
    Limited Lifetime Warranty
    Meraki
    Multicast
    N7k
    Nexus
    Nicira
    Ons
    Opendaylight
    Openflow
    Openstack
    Presidio
    Qsfp
    Quick Facts
    Routeflow
    Sdn
    Sdn Ecosystem
    Security
    Ucs


    Archives

    May 2015
    April 2015
    February 2015
    January 2015
    December 2014
    November 2014
    October 2014
    September 2014
    August 2014
    June 2014
    May 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012
    January 2012
    December 2011
    November 2011


    RSS Feed


    View my profile on LinkedIn
Photo used under Creative Commons from NASA Goddard Photo and Video