2024年7月31日发(作者：郎晨辰)
ConnectX®-3 Ethernet Single and Dual SFP+
Port Adapter Card User Manual
P/N:
MCX312A-XCBT, MCX312B-XCBT, MCX311A-XCAT
Rev 1.7
Rev 1.7ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE
PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE
OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE
CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX
TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX
TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE
DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,
PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND
RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
Tel: (408) 970-3400
Mellanox Technologies, Ltd.
Beit Mellanox
PO Box 586 Yokneam 20692
Israel
Tel: +972 (0)74 723 7200
© Copyright 2012. Mellanox Technologies. All rights reserved.
Mellanox®, Mellanox logo®, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®,
SwitchX®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of Mellanox Technologies, Ltd. Connect-
IB™, FabricIT™, MLNX-OS™, ScalableHPC™, TestX™, Unbreakable-Link™, UFM™ and Unified Fabric Manager™ are
trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
2
Mellanox Technologies
Document Number: 3755
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Intended Audience. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Technical Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Firmware and Software Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1 Supported Network Protocol Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1.1 10 Gigabit Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2 Adapter Cards Covered in this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Finding the MAC and Serial Number on the Adapter Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Chapter 2 Adapter Card Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1 I/O Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.1 Ethernet SFP+ Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.2 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.3 LED Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.4 I2C-compatible Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 SFP+ Power Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.1 System Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.2 SPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.3 EEPROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Chapter 3 Ethernet Adapter Card Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 Identify the Card in Your System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.1 On Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.2 On Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4 Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.4.1 Inserting the Optical Transceiver Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.4.2 Cable Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Chapter 4 Driver Software and Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1 Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1.1 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1.2 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2 FlexBoot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3 RDMA over Converged Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3.1 Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4 Updating Adapter Card Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Appendix A Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
A.1MCX311A-XCAT Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Mellanox Technologies
3
Rev 1.7
A.2MCX312[AB]-XCBT Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
A.3Board Mechanical Drawing and Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
A.4Regulatory Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Appendix B Interface Connectors Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
B.1I2C-compatible Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
B.2PCI Express x8 Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
B.3SFP+ Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Appendix C Replacing a Tall Bracket With a Short Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
C.1Replacing a Bracket. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
C.2Removing the Existing Bracket from the Adapter Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
C.3Installing the New Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Appendix D Avertissements de sécurité d’installation (Warnings in French). . . . . . . . . . . . . . 49
Appendix E Sicherheitshinweise (Warnings in German). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish). . . . . . . . . 53
4
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
List of Tables
Table 1:
Table 2:
Table 3:
Table 4:
Table 5:
Table 6:
Table 7:
Table 8:
Table 9:
Table 10:
Table 11:
Table 12:
Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Documents List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Single and Dual-port 10 Gigabit Ethernet Adapter Card . . . . . . . . . . . . . . . . . . . . . . . . 11
Physical and Logical Link Indication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Jumper Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
MCX312A-XCBT VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
MCX312B-XCBT VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
MCX311A-XCAT VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
MCX311A-XCAT Specifications Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
MCX312A-XCBT Specifications Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
I2C-compatible Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
SFP+ Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mellanox Technologies
5
Rev 1.7
List of Figures
Figure 1:
Figure 2:
Figure 3:
Figure 4:
Figure 5:
Figure 6:
Figure 7:
Figure 8:
Figure 9:
Card Product Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Board Labels for MCX312B-XCBT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
MCX312A-XCBT Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
MCX312B-XCBT Card. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
MCX311A-XCAT Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Dual-port Bracket for MCX312A-XCBT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Dual-port Bracket for MCX312B-XCBT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Single-port Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
I2C-compatible Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Figure 10:PCI Device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Figure 11:Module With Locking Mechanism Closed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Figure 12:Module With Locking Mechanism Open . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Figure 13:Device Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Figure 14:Support Download Assistant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Figure 15:Mechanical Drawing of the Dual-port MCX311A-XCAT Adapter Card. . . . . . . . . . . . 39
Figure 16:Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card . . . . . . . . . . . 40
Figure 17:Mechanical Drawing of the Dual-port MCX312B-XCBT . . . . . . . . . . . . . . . . . . . . . . . 40
Figure 18:Compatible Connector Plug and Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Figure 19:PCIe Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Figure 20:Rear View of Module With Pin Placement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Figure 21:Bracket Screws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Figure 22:Placing the Bracket on the Card (MCX312A-XCBT). . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Figure 23:Placing the Bracket on the Card (MCX312B-XCBT). . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Figure 24:Placing the Bracket on the Card (MCX311A-XCAT) . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Revision History
This document was printed on November 14, 2012.
Table 1 - Revision History Table
Date
November 2012
Rev
1.7
Comments/Changes
•Added MCX312B-XCBT to the following locations:
- Section 1.2, “Adapter Cards Covered in this Manual,” on
page 11
- Figure 4, “MCX312B-XCBT Card,” on page 15
- Figure 7, “Dual-port Bracket for MCX312B-XCBT,” on
page 16
-
Table 7, “MCX312B-XCBT VPD Layout,” on
page 21
- Figure 17, “Mechanical Drawing of the Dual-port MCX312B-
XCBT,” on page 40
October 20121.6•
•
•
•
•
Added single-port SFP+ card to all necessary locations
Fixed Table 5, “Jumper Configuration,” on page 19
Updated figures in Section 4.1.2, “Windows,” on page 31 by pro-
viding ConnectX-3 examples
Removed all references to 40 Gigabit Ethernet from the User
Manual
Added operational and non-operational temperature and humid-
ity level to the following tables:
- Table 10, “MCX312A-XCBT Specifications Table,” on
page 38
Minor edits
Updated LED functions in Section 2.1.3, “LED Assignment,” on
page 17
Formatted specification tables in Appendix A, “Specifications,”
on page 37
October 2012
August 2012
1.5
1.4
January 20121.2•
•
•
October 2011
July 2011
July 2011
1.1
1.0
0.10
Added new OPNs based on ConnectX-3 Step A1 devices
Minor edits
First Release
Mellanox Technologies
7
Rev 1.7
About this Manual
This User Manual describes Mellanox Technologies ConnectX®-3 10 Gigabit Ethernet Single
Dual SFP+ port PCI Express x4 or x8 adapter cards. It provides details as to the interfaces of the
board, specifications, required software and firmware for operating the board, and relevant docu-
mentation.
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with Ethernet networks and architecture specifications.
Related Documentation
Table 2 - Documents List
Mellanox Firmware Tools (MFT) User Manual
Document no. 2204UG
IEEE Std 802.3 Specification
PCI Express 3.0 Specifications
User Manual describing the set of MFT firmware management tools for a
single node.
See
=> Support => Download Firmware Tools
This is the IEEE Ethernet specification
/getieee802
Industry Standard PCI Express 3.0 Base and Card Electromechanical
Specifications
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used
to mean PCI Express.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact us
through the following methods.
•URL:
=> Support
•E-mail: support@
•Tel: +1.408.916.0055
Customers who purchased Mellanox M-1 Global Support Services, please see your contract for
details regarding Technical Support.
8
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seek assistance through their reseller.
Firmware and Software Updates
The Mellanox support downloader contains software, firmware and knowledge database informa-
tion for Mellanox products. Access the data base from the Mellanox Support web page,
=> Support
or use the following link to go directly to the Mellanox Support Download Assistant page,
/supportdownloader/.
Mellanox Technologies
9
Rev 1.7Overview
1Overview
This document is a User Manual for Mellanox Technologies Ethernet cards based on the Con-
nectX®-3 EN integrated circuit device. The cards described in this manual have the following
main features:
•IEEE Std. 802.3 compliant
•PCI Express 3.0 (1.1 and 2.0 compatible) through an x4 or x8 edge connector up to 8GT/s
•10 Gb/s Ethernet
•Single and Dual SFP+ ports for connection Ethernet traffic
•RDMA over Converged Ethernet (RoCE)
•End-to-end QoS and congestion control
•Hardware-based I/O virtualization
•TCP/UDP/IP stateless off-load
•RoHS-R6 compliant
•Two bracket heights: short and tall
1.1Supported Network Protocol Standards
1.1.110 Gigabit Ethernet
10 Gigabit Ethernet is an Ethernet network protocol standard for LANs specifying data transfer
rate of 10Gb/s. Mellanox adapters comply with the following IEEE 802.3* standards:
•IEEE Std 802.3-2008 Ethernet
•IEEE Std 802.3ae 10 Gigabit Ethernet
•IEEE Std 802.3ad Link Aggregation and Failover
10
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
1.2Adapter Cards Covered in this Manual
Table 3 lists the single and dual port 10 Gigabit Ethernet adapter cards described in this manual.
Table 3 - Single and Dual-port 10 Gigabit Ethernet Adapter Card
Ordering Part Number
(OPN)
PCI Express
SERDES Speed
Data Transmis-
sion Rate/
# of ports
RoHS
IC
Part Number
Device ID (decimal)
MCX312A-XCAT
(Legacy product)
PCIe 3.0 x4/x8
8GT/s
10 GigE dual-port
SFP+
R-6MT27518A0-FCCR-XE
•
•
4099 for Physical
Function
4100 for Virtual
Function
4099 for non-virtu-
alized servers
4100 for virtual
machines
4099 for Physical
Function
4100 for Virtual
Function
4099 for Physical
Function
4100 for Virtual
Function
MCX311A-XCAT PCIe 3.0 x4
8GT/s
10 GigE single-
port SFP+
R-6MT27518A1-FCCR-XE
•
•
MCX312A-XCBTPCIe 3.0 x4/x8
8GT/s
10 GigE dual-port
SFP+
R-6MT27518A1-FCCR-XE
•
•
MCX312B-XCBTPCIe 3.0 x4/x8
8GT/s
10 GigE dual-port
SFP+
R-6MT27518A1-FCCR-XE
•
•
1.3Finding the MAC and Serial Number on the Adapter Card
Each Mellanox adapter card has a label on the print side that shows the card serial number and the
card MAC for the Ethernet protocol.
Figure 1: Card Product Label (Example)
Mellanox Technologies
11
Rev 1.7Overview
Figure 2: Board Labels for MCX312B-XCBT
1.4Safety Warnings
Below is a list of safety warnings in English. For safety warnings in other languages, please refer
to the appendices in this user manual.
lation Instructions
Read all installation instructions before connecting the equipment to the power source.
-temperature
This equipment should not be operated in an area with an ambient temperature exceed-
ing the maximum recommended: 55°C (131°F).
To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ven-
tilation openings.
Lightning - Electrical Hazard
During periods of lightning activity, do not work on the equipment or connect or dis-
connect cables.
Cable Connecting/Disconnecting
Some copper cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Refer to the cable manufacturer for spe-
cial warnings and instructions.
12
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
ent Installation
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
ent Disposal
Disposal of this equipment should be in accordance to all national laws and regula-
tions.
7.
Local and National Electrical Codes
This equipment should be installed in compliance with local and national electrical
codes.
ous Radiation Exposure
Caution – Use of controls or adjustment or performance of procedures other than
those specified herein may result in hazardous radiation exposure.
CLASS 1 LASER PRODUCT and reference to the most recent laser standards:
IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+
A2:2001
Mellanox Technologies
13
Rev 1.7Adapter Card Interfaces
2
2.1
Adapter Card Interfaces
I/O Interfaces
Each adapter card includes the following interfaces:
•SFP+ port(s)
•PCI Express (PCIe) x4 or x8 edge connector
•I/O panel LEDs
•I
2
C-compatible connector (for debug)
Figure 3: MCX312A-XCBT Card
14
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Figure 4: MCX312B-XCBT Card
Figure 5: MCX311A-XCAT Card
The adapter cards include special circuits to protect from ESD shocks to the card/server when
plugging copper cables.
Mellanox Technologies
15
Rev 1.7Adapter Card Interfaces
Figure 6: Dual-port Bracket for MCX312A-XCBT
Figure 7: Dual-port Bracket for MCX312B-XCBT
16
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Figure 8: Single-port Bracket
For dual-port cards, Port 1 connects to connector 1 of the device, while Port 2 connects to connec-
tor 2 of the device.
2.1.1Ethernet SFP+ Interface
The network ports of the ConnectX®-3 adapter cards are compliant with the IEEE 802.3 Ethernet
standards listed in Section 1.1.1, “10 Gigabit Ethernet,” on page 10. The SFP+ port has one Tx/Rx
pair of SerDes. Ethernet traffic is transmitted through the cards' SFP+ connectors.
2.1.2PCI Express Interface
The ConnectX®-3 adapter cards support PCI Express 3.0 (1.1 and 2.0 compatible) through an x4
or x8 edge connector. The device can be either a master initiating the PCI Express bus operations
or a slave responding to PCI bus operations.
2.1.3LED Assignment
There are two I/O LEDs per port in dual-port designs and one bicolor LED in single-port designs,
green and yellow, located on the I/O panel. See Table 4 for different LED functions.
Mellanox Technologies
17
Rev 1.7Adapter Card Interfaces
Table 4 - Physical and Logical Link Indication
LED Function
Green - physical link •
•
•
•
Constant on indicates a good physical link
If neither LED is lit, then the physical link has not been established
A blinking yellow indicates activity (data transfer)
Stays off when there is no activity
Yellow -
logical (data activity
link)
The short bracket has the same port and LED footprints as the tall bracket.
2.1.4I
2
C-compatible Interface
A three-pin header on the adapter cards is provided as the I
2
C-compatible interface. See Figure 16,
“Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card,” on page 40 for the loca-
tion on the board.
Figure 9: I
2
C-compatible Connector
2.2Power
All adapter cards receive 12V and 3.3V power from the PCI Express edge connector. All other
required power voltages are generated by on-board switch mode regulators. See “Specifications”
on page 37.
2.3SFP+ Power Level
The adapter cards support power levels according to SFF Committee SFF-8431 Specification for
SFP+ (Small Form-factor Pluggable) Transceiver.
2.4Memory
The adapter cards support multiple memory devices through the PCIe, SPI (Flash) and I
2
C inter-
faces.
18
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
2.4.1System Memory
The adapter cards utilize the PCI Express interface to store and access Ethernet fabric connection
information and packet data on the system memory.
2.4.2SPI
Each of the adapter cards includes one 16MB SPI Flash device (M25PX16-VMN6P device by ST
Microelectronics) accessible via the SPI interface of the ConnectX®-3 EN device.
When a jumper is inserted into the drill holes in the adapter it indicates to the device that the on-
board Flash device should not be used for boot instructions. Contact your Mellanox support repre-
sentative should you need to use this jumper. Table 5 provides information on this jumper. The
jumper location on the board is illustrated in Appendix A.3, “Board Mechanical Drawing and
Dimensions,” on page 39.
Table 5 - Jumper Configuration
DescriptionOption
Card Default
Configuration
Flash present/
not present
connection open – Flash presentconnection open –
Flash present
connection shorted – Flash not present
2.4.3EEPROM
Each board incorporates an EEPROM that is accessible through the I
2
C-compatible interface. The
EEPROM capacity is 4Kb. The EEPROM is used for storing the Vital Product Data (VPD).
The PCI VPD (Vital Product Data) layout for each of the adapter cards complies with the format
defined in the PCI 3.0 Specification, Appendix I.
Table 6 - MCX312A-XCBT VPD Layout
Offset
(Decimal)
0
1
2
3
27
Item
Large Resource Type ID
String Tag (0x02)
Length [7:0] LSB
Length [15:8] MSB
Data
Large Resource Type VPD-R
Tag (0x10)
Value
0x82
0x18
0x0
CX312A - Con-
nectX-3 SFP+
0x90
STR
FormatDescription
Mellanox Technologies
19
Rev 1.7
Offset
(Decimal)
28
29
30
32
33
54
56
57
59
61
62
86
88
89
105
107
108
109
110
111
112
114
115
121
123
124
148
150
151
256
258
259
512
514
515
768
770
771
1024
1026
1027
1280
Adapter Card Interfaces
Item
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Part Number
VPD Keyword
Length
Revision
VPD Keyword
Length
SerialNumber
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Large Resource Type VPD-W
Tag (0x11)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Value
0x4F
0x00
PN
0x15
PN
EC
0x2
RV
SN
0x18
SN
V0
0x10
PCIe Gen3 x8
RV
0x1
0,107
0x91
0x8F
0xF
V1
0x6
N/A
YA
0x18
N/A
RW
0x69
FormatDescription
STR
%STR_SPC
STR
Add in Card Part Number
Engineering Change Level of
the card (rev)
PCB revision
Serial Number
“00..00XXXX..XX”
Misc Information
%STR
STR
%STR_SPC
STR
STR_SPC
STR
%CS0
STR
STR_SPC
STR
STR_SPC
STR
STR_ZERO
EFI Driver version
Asset Tag
“N/A”
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR
STR_ZERO
RW
0xfd
STR
STR_ZERO
RW
0xfd
STR
STR_ZERO
RW
0xfd
STR
STR_ZERO
RWSTR
20
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual
Offset
(Decimal)
1282
1283
1536
1538
1539
1792
1794
1795
2048
2050
2051
2304
2306
2307
2560
2562
2563
2816
2818
2819
3072
3074
3075
3328
3330
3331
3584
3586
3587
3840
3842
3843
4095
Rev 1.7
Item
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Small Resource Type END
Tag (0x11)
Value
0xfd
FormatDescription
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfc
STR_ZERO
0x78
STR
STR
STR
STR
STR
STR
STR
STR
STR
STR
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Table 7 - MCX312B-XCBT VPD Layout
Offset
(Decimal)
0
1
2
Item
Large Resource Type ID
String Tag (0x02)
Length [7:0] LSB
Length [15:8] MSB
Value
0x82
0x18
0x0
FormatDescription
Mellanox Technologies
21
Rev 1.7Adapter Card Interfaces
Table 7 - MCX312B-XCBT VPD Layout
Offset
(Decimal)
3
27
28
29
30
32
33
54
56
57
59
61
62
86
88
89
105
107
108
109
110
111
112
114
115
121
123
124
148
150
151
256
258
259
512
514
515
768
770
771
Item
Data
Large Resource Type VPD-R
Tag (0x10)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Part Number
VPD Keyword
Length
Revision
VPD Keyword
Length
SerialNumber
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Large Resource Type VPD-W
Tag (0x11)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
RW
0xfd
STR_ZEROReserved (0x00)
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Value
CX312B - Con-
nectX-3 SFP+
0x90
0x4F
0x00
PN
0x15
PN
EC
0x2
RV
SN
0x18
SN
V0
0x10
PCIe Gen3 x8
RV
0x1
0,107
0x91
0x8F
0xF
V1
0x6
N/A
YA
0x18
N/A
RW
0x69
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
STR_SPC
STR
“N/A”
Remaining read/write area
STR_SPC
STRAsset Tag
STREFI Driver version
%CS0
STR_SPC
STR
%STR_SPC
STR
“00..00XXXX..XX”
Misc Information
%STR
STR
PCB revision
Serial Number
%STR_SPC
STREngineering Change Level of
the card (rev)
STRAdd in Card Part Number
Format
STR
Description
22
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Table 7 - MCX312B-XCBT VPD Layout
Offset
(Decimal)
1024
1026
1027
1280
1282
1283
1536
1538
1539
1792
1794
1795
2048
2050
2051
2304
2306
2307
2560
2562
2563
2816
2818
2819
3072
3074
3075
3328
3330
3331
3584
3586
3587
3840
3842
3843
4095
Item
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Small Resource Type END
Tag (0x11)
0x78
RW
0xfc
STR_ZEROReserved (0x00)
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Value
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Format
STR
Description
Remaining read/write area
Mellanox Technologies
23
Rev 1.7Adapter Card Interfaces
Table 8 - MCX311A-XCAT VPD Layout
Offset
(Decimal)
0
1
2
3
27
28
29
30
32
33
54
56
57
59
61
62
86
88
89
105
107
108
109
110
111
112
114
115
121
123
124
148
150
151
256
258
259
512
Item
Large Resource Type ID
String Tag (0x02)
Length [7:0] LSB
Length [15:8] MSB
Data
Large Resource Type VPD-R
Tag (0x10)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Part Number
VPD Keyword
Length
Revision
VPD Keyword
Length
SerialNumber
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Large Resource Type VPD-W
Tag (0x11)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD KeywordRW
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Value
0x82
0x18
0x0
CX311A - Con-
nectX-3 SFP+
0x90
0x4F
0x00
PN
0x15
PN
EC
0x2
RV
SN
0x18
SN
V0
0x10
PCIe Gen3 x4
RV
0x1
0,107
0x91
0x8F
0xF
V1
0x6
N/A
YA
0x18
N/A
RW
0x69
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
STR_SPC
STR
“N/A”
Remaining read/write area
STR_SPC
STRAsset Tag
STREFI Driver version
%CS0
STR_SPC
STR
%STR_SPC
STR
“00..00XXXX..XX”
Misc Information
%STR
STR
PCB revision
Serial Number
%STR_SPC
STREngineering Change Level of
the card (rev)
STRAdd in Card Part Number
STR
FormatDescription
24
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Table 8 - MCX311A-XCAT VPD Layout
Offset
(Decimal)
514
515
768
770
771
1024
1026
1027
1280
1282
1283
1536
1538
1539
1792
1794
1795
2048
2050
2051
2304
2306
2307
2560
2562
2563
2816
2818
2819
3072
3074
3075
3328
3330
3331
3584
3586
3587
3840
3842
3843
4095
Item
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Small Resource Type END
Tag (0x11)
0x78
RW
0xfc
STR_ZEROReserved (0x00)
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Value
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
FormatDescription
Mellanox Technologies
25
Rev 1.7Adapter Card Interfaces
26
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
3
3.1
Ethernet Adapter Card Installation
Hardware Requirements
A system with a PCI Express x4 or x8 slot is required for installing the card.
3.2Installation Instructions
Read all installation instructions before connecting the equipment to the power source.
To change a tall bracket to a short bracket see Replacing a Tall Bracket With a Short
Bracket on page 46.
The adapter cards listed above are standard PCI Express cards, each with a standard x8 edge con-
nector. The cards require a PCI Express x4 or x8. Please consult the host machine documentation
for instructions on how to install a PCI Express card.
If the card is installed in a PCI slot with less lanes than the card requires then the adapter
card will not provide the optimum data transfer.
3.3Identify the Card in Your System
1.
2.
3.
4.
Open Device Manager on the server. Click start => Run, and then enter “”.
Expand System Devices and locate your Mellanox ConnectX-3 adapter card.
Select Properties to display the adapter card properties window.
Click the Details tab and select Device Instance Id
(Windows 2003) or
Hardware Ids (Windows 2008/R2) from the Property pull-down menu.
3.3.1On Windows
Mellanox Technologies
27
Rev 1.7Ethernet Adapter Card Installation
Figure 10: PCI Device
the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the
display example above, notice the sub-string “PCIVEN_15B3&DEV_1003”: VEN is equal
to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 –
this is a valid Mellanox Technologies PCI Device ID.
If the PCI device does not have a Mellanox adapter ID, return to Step 4 to check another
device.
The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi-
tory at /read/PC/15b3.
3.3.2On Linux
Get the device location on the PCI bus by running lspci and locating lines with the string “Mella-
nox Technologies”:
> lspci |grep -i Mellanox
27:00.0 Network controller: Mellanox Technologies MT27500 Fam-
ily [ConnectX-3]
28
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
3.4Cables and Modules
Please refer to “Mellanox Products Approved Cable Lists” at /related-
docs/user_manuals/Mellanox_approved_
3.4.1Inserting the Optical Transceiver Module
To insert the module into the cage:
1.
2.
Open the module’s locking mechanism – see Figure 11 and Figure 12.
Make sure that the male connectors on the module will align with the female connectors
inside of the cage. Also check that there is no dirt or foreign matter in the module or in the
cage.
Figure 11: Module With Locking Mechanism Closed
Figure 12: Module With Locking Mechanism Open
3.
4.
Insert the module into the adapter card module cage.
Close the locking Mechanism.
To remove the module from the cage:
1.
2.
Unlock the locking mechanism by opening the handle.
Pull the module out of the cage.
3.4.2Cable Installation
1.
2.
All cables can be inserted or removed with the unit powered on. To insert a cable, press the
connector into the port receptacle until the connector is firmly seated.
After inserting a cable into a port, the GREEN LED indicator will light when the physical
connection is established (that is, when the unit is powered on and a cable is plugged into the
port with the other end of the connector plugged into a functioning port).
After plugging in a cable, lock the connector using the latching mechanism particular to the
cable vendor. When a logical connection is made the YELLOW LED will come on. When
data is being transferred the yellow led will blink.
3.
Mellanox Technologies
29
Rev 1.7Ethernet Adapter Card Installation
When installing cables make sure that the latches engage.
Always install and remove cables by pushing or pulling the cable and connector in a
straight line with the card.
4.
5.
Care should be taken as not to impede the air exhaust flow through the ventilation holes.
Cable lengths which allow for routing horizontally around to the side of the chassis before
bending upward or downward in the rack should be used
To remove a cable, disengage the locks and slowly pull the connector away from the port
receptacle. Both LED indicators will turn off when the cable is unseated.
Cables, especially long copper cables, can weigh a substantial amount. Make sure that
the weight of the cable is supported on its own and is not hanging from the adapter
card.
30
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
4
4.1
Driver Software and Firmware
Driver Software
4.1.1Linux
For Linux, download and install the latest MLNX_EN driver software package available via the
Mellanox web site at: => Products => Software/Drivers => Ethernet
SW/Drivers => ConnectX®-3 EN 10GigE Linux Driver => Download. Follow the installation
instructions included in the download package (also available from the download page). To ensure
that communication has been established follow the instructions below.
Check the link status
First check the network interface name by running the “ifconfig –a” command
To assign an IP address to the interface run:
> ifconfig eth
where 'x' is the OS assigned interface number.
To check driver and device information run:
> ethtool -i eth
Example:
> ethtool -i eth2
driver:mlx4_en (MT_1020110019_CX-3)
version: 1.5.6.33 (Oct 2011)
firmware-version:2.10.0000
bus-info: 0000:07:00.0
4.1.2Windows
For Windows, download the MLNX EN – Ethernet driver. Download this package from the Mella-
nox web site at:
=> Products => Software/Drivers => Ethernet SW/
Drivers => ConnectX® EN 10GigE Windows Driver => Download. Follow the installation
instructions included in the download package (also available from the download page). To ensure
that communication has been established follow the instructions below.
Mellanox Technologies
31
Rev 1.7Driver Software and Firmware
To display a summary of network adapter software-, firmware- and hardware-related information
such as driver version, firmware version, bus interface, adapter identity, and network port link
information, perform the following steps:
Step 1: Display the Device Manager
Figure 13: Device Manager
Step 2: Right-click a Mellanox ConnectX Ethernet adapter (under “Network adapters list) and left-
click Properties.
32
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Step 3: Select the Information tab from the Properties sheet (example).
To save this information for debug purposes, click Save To File and provide the output
file name.
4.2FlexBoot
FlexBoot enables remote boot over Ethernet, Boot over Ethernet (BoE) or Boot over iSCSI (Bo-
iSCSI). This technology is based on the Preboot Execution Environment (PXE) standard specifica-
tion, and FlexBoot software is based on the open source iPXE project (see ). For
more information go to
=> Products => Software/Drivers => Ethernet
Software/Drivers => FlexBoot => Download.
4.3RDMA over Converged Ethernet
ConnectX®-3 connected to an Ethernet fabric provides all of the basic NIC functionality plus
RDMA over Converged Ethernet (RoCE). RoCE utilizes advances in Data Center Bridging (DCB)
to enable efficient and low cost implementations of RDMA over Ethernet, supporting the entire
breadth of RDMA and low latency features. This includes reliable connected service, datagram
service, RDMA and send/receive semantics, atomic operations, user level multicast, user level I/O
access, kernel bypass, and zero copy. This capability is enabled when using the Mellanox OFED or
WinOF VPI drivers.
Mellanox Technologies
33
Rev 1.7Driver Software and Firmware
ConnectX®-3 EN with RoCE based network management is the same as that for any Ethernet and
DCB-based network management, eliminating the need for IT managers to learn new technolo-
gies.
4.3.1Hardware and Software Requirements
Software:
•Mellanox OFED 1.5.1 or WinOF 2.1.2 or later. To download these packages go to
=> Products => Software/Drivers => InfiniBand & VPI SW/Driver.
Hardware:
•ConnectX®-3 EN adapter card
4.4Updating Adapter Card Firmware
Each card is shipped with the latest version of qualified firmware at the time of manufacturing.
Firmware is updated occasionally, and the most recent firmware can be obtained from:
=> Support => Download Firmware. Check that the firmware on your
card is the latest found on the Mellanox site, if not update to the latest version found on the Mella-
nox web site.
Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firm-
ware Tools (MFT) package. This package is available for download, along with its user manual,
from the Mellanox Firmware Tools page. See => Support => Download
Firmware Tools.
A firmware binaries table lists a binary file per adapter card. The file name of each such binary is
composed by combining the firmware name, the firmware release version, and the card part num-
ber. Please contact Mellanox System Support if you cannot find the firmware binary for your
adapter card.
The following steps describe how to retrieve the PSID (firmware identification) and programmed
firmware version of your adapter card. They also describe how to update the card with the latest
firmware version available.
ve the PSID and firmware version:
l the MFT package. The package is available at => Products
=> Software/Drivers => Ethernet Software/Drivers => Firmware Tools. Make sure to down-
load the package corresponding to your computer’s operating system.
: mst start.
the Mellanox mst device name using the command "mst status". The mst device name will be of
the form: /dev/mst/mt4099_pci_cr0.
the PSID (firmware identification) and programmed firmware version using the command.
34
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
> flint -d /dev/mst/mt4099_pci_cr0 q
Image type: ConnectX
FW Version: 2.9.4000
Device ID: 4099
Chip Revision: 0
Description: Node Port1
Port2 Sys image
GUIDs: 000002c900000200 000002c900000201
000002c900000202 000002c900000203
MACs:
000002c90200 000002c90201
Board ID: (MT_1020110019)
VSD:
PSID: MT_1020110019
e the programmed firmware version with the latest available.
to Mellanox’s web site: /supportdownloader.
See Figure 14.
your card PSID to display the latest firmware.
Figure 14: Support Download Assistant
a newer firmware version exists for your adapter card, update the firmware as follows:
ad the firmware image zip file from the Download Center (see Step 2a above).
the firmware image.
the firmware image. Enter:
Mellanox Technologies
35
Rev 1.7Driver Software and Firmware
> flint -d /dev/mst/mt4099_pci_cr0 -i
the computer.
: mst start.
that the card firmware was updated successfully.
> flint -d /dev/mst/mt4099_pci_cr0 q
Image type: ConnectX
FW Version: 2.9.4100
Device ID: 4099
...
36
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Appendix A: Specifications
A.1MCX311A-XCAT Specifications
Table 9 - MCX311A-XCAT Specifications Table
Size: 2.11in. x 4.18 in. (53.59mm x 106.06 mm)
Physical
Connector: SFP+ 10Gb/s
Ethernet: XAUI/10GBASE-KX4 – 4X 10GigE port, XFI/SFI/10GBASE-KR – 1X
10GigE port, SGMII/1000BASE-KX – 1X 1GigE port
Protocol Support
Data Rate: 1/10Gb/s – Ethernet
PCI Express Gen3: SERDES @ 8.0GT/s, 4 lanes (2.0 and 1.1 compatible)
Voltage: 12V, 3.3V
Typ Power: Passive Cables 3.47W
Max Power: Passive Cables 4.84W
Power and
Environmental
Temperature: Operational 0°C to 55°C
Non-operational 0°C to 70°C
Humidity: 90% relative humidity
a
Air Flow: 200LFM
b
EMC: Refer to the following link:
/related-docs/user_manuals/Regulatory_and_Compliance_
Regulatory
Safety: IEC/EN 60950-1:2006
ETSI EN 300 019-2-2
IEC 60068-2- 64, 29, 32
RoHS: RoHS-R6
Cable Support
Please refer to "Mellanox Products Approved Cable Lists" at:
/related-docs/user_manuals/Mellanox_approved_
both operational and non-operational states
flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet.
Mellanox Technologies
37
Rev 1.7
A.2MCX312[AB]-XCBT Specifications
Table 10 - MCX312A-XCBT Specifications Table
Size: 2.71in. x 5.6in. (68.90 mm x 142.25 mm)
Physical
Connector: SFP+ 10Gb/s
Ethernet: 10GBASE-CX4, 10GBASE-R, and 1000BASE-R
Protocol Support
Data Rate: 1/10Gb/s – Ethernet
PCI Express Gen3: SERDES @ 8.0GT/s, 4 lanes (2.0 and 1.1 compatible)
PCI Express Gen2: SERDES @ 5.0GT/s, 8 lanes (1.1 compatible)
Voltage: 12V, 3.3V
Typ Power: PCIe x4 Gen3
Passive Cables 3.83W
PCIe x8 Gen2
Passive Cables 5.31W
Power and
Environmental
Max Power: PCIe x4 Gen3
Passive Cables 4.55W
PCIe x8 Gen2
Passive Cables 6.20W
Temperature: Operational 0°C to 55°C
Non-operational 0°C to 70°C
Humidity: 90% relative humidity
a
Air Flow: 200LFM
b
EMC: Refer to the following link:
/related-docs/user_manuals/Regulatory_and_Compliance_
Regulatory
Safety: IEC/EN 60950-1:2006
ETSI EN 300 019-2-2
IEC 60068-2- 64, 29, 32
RoHS: RoHS-R6
Cable Support
Please refer to "Mellanox Products Approved Cable Lists" at:
/related-docs/user_manuals/Mellanox_approved_
both operational and non-operational states
flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet.
38
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
A.3Board Mechanical Drawing and Dimensions
All dimensions are in millimeters.
All the mechanical tolerances are +/- 0.1mm
Figure 15: Mechanical Drawing of the Dual-port MCX311A-XCAT Adapter Card
J1 - I
2
C Connector
Flash Jumper
142.24
40.5
49.09
22.61
53.59
45.95
12.75
Mellanox Technologies
39
Rev 1.7
Figure 16: Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card
J1 - I
2
C Connector
Flash Jumper
142.25
64.4
50.00
56.15
68.90
45.96
12.75
Figure 17: Mechanical Drawing of the Dual-port MCX312B-XCBT
142.24
40.84
49.1
53.59
45.95
40
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
A.4Regulatory Statements
For regulatory statements for all ConnectX®-3 cards please refer to:
/related-docs/user_manuals/Regulatory_and_Compliance_
Mellanox Technologies
41
Rev 1.7
Appendix B: Interface Connectors Pinout
B.1I
2
C-compatible Connector Pinout
Figure 18: Compatible Connector Plug and Pinout
Table 11 - I2C-compatible Connector Pinout
Connector Pin Number
1
2
3
4
5
Signal Name
SPSDA
SPSCL
GND
NC
NC
B.2PCI Express x8 Connector Pinout
The adapter cards use a standard PCI Express x4 or x8 edge connector and the PCI Express x8
standard pinout according to the PCI Express 3.0 specification.
42
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Figure 19: PCIe Connector Pinout
Mellanox Technologies
43
Rev 1.7
B.3SFP+ Connector Pinout
Figure 20: Rear View of Module With Pin Placement
44
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Table 12 - SFP+ Connector Pinout
Pin
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Symbol Name
VeeT
TX_Fault
TX_Disable
SDA
SCL
MOD_ABS
RS0
RX_LOS
RS1
VeeR
VeeR
RD-
RD+
VeeR
VccR
VccT
VeeT
TD+
TD-
VeeT
Description
Transmitter Ground (Common with Receiver Ground)
a
Transmitter Fault.
b
Transmitter Disable. Laser output disabled on high or open.
c
2-wire Serial Interface Data Line
d
2-wire Serial Interface Clock Line
d
Module Absent. Grounded within the module
d
No connection required
Loss of Signal indication. Logic 0 indicates normal operation.
e
No connection required
Receiver Ground (Common with Transmitter Ground)
a
Receiver Ground (Common with Transmitter Ground)
a
Receiver Inverted DATA out. AC Coupled
Receiver Non-inverted DATA out. AC Coupled
Receiver Ground (Common with Transmitter Ground)
a
Receiver Power Supply
Transmitter Power Supply
Transmitter Ground (Common with Receiver Ground)
a
Transmitter Non-Inverted DATA in. AC Coupled.
Transmitter Inverted DATA in. AC Coupled.
Transmitter Ground (Common with Receiver Ground)
a
t ground is internally isolated from chassis ground.
b.T
FAULT
is an open collector/drain output, which should be pulled up with a 4.7k – 10k Ohms resistor
on the host board if intended for use. Pull up voltage should be between 2.0V to Vcc + 0.3V. A high
output indicates a transmitter fault caused by either the TX bias current or the TX output power exceed-
ing the preset alarm thresholds. A low output indicates normal operation. In the low state, the output is
pulled to <0.8V.
c. Laser output disabled on TDIS >2.0V or open, enabled on TDIS <0.8V
be pulled up with 4.7kΩ – 10kΩ on host board to a voltage between 2.0V and 3.6V. MOD_ABS
pulls line low to indicate module is plugged in.
is open collector output. Should be pulled up with 4.7kΩ – 10kΩ on host board to a voltage
between 2.0V and 3.6V. Logic 0 indicates normal operation; logic 1 indicates loss of signal.
Mellanox Technologies
45
Rev 1.7
Appendix C: Replacing a Tall Bracket With a Short Bracket
This section provides instructions on how to remove the tall bracket of a standard Mellanox Tech-
nologies adapter card and replace it with a short one.
C.1Replacing a Bracket
To replace the bracket you will need the following parts:
•the new bracket of the proper height
•one new square gasket for each of the ports
•the 2 screws saved from the removal of the bracket
•the 2 fiber washers saved from the removal of the bracket
C.2Removing the Existing Bracket from the Adapter Card
Figure 21: Bracket Screws
Screws
1.
2.
Remove the two screws holding the bracket in place.
The bracket comes loose from the card.
Be careful not to put stress on the LEDs.
the two screws and the two fiber washers.
46
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
C.3
1.
Installing the New Bracket
Place the bracket onto the card until the screw holes line up.
Do not force the bracket onto the card. You may have to gently push the LEDs using a
small screwdriver to align the LEDs with the holes in the bracket.
2. Screw on the bracket using the screws and washers saved from the procedure above step 1.
Figure 22: Placing the Bracket on the Card (MCX312A-XCBT)
Mellanox Technologies
47
Rev 1.7
Figure 23: Placing the Bracket on the Card (MCX312B-XCBT)
Figure 24: Placing the Bracket on the Card (MCX311A-XCAT)
3.
4.
Make sure that the LEDs are aligned onto the bracket holes.
Use a torque driver to apply up to 2 lbs-in torque on the screws.
48
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Appendix D: Avertissements de sécurité d’installation
(Warnings in French)
ctions d’installation
Lisez toutes les instructions d’installation avant de brancher le matériel à la source
d’alimentation électrique.
érature excessive
Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante
dépassant le maximum recommandé de 55°C (131°F). Un flux d’air de 200LFM à
cette température ambiante maximale est nécessaire. En outre, pour garantir un bon
écoulement de l’air, laissez au moins 8 cm (3 pouces) d’espace libre autour des ouver-
tures de ventilation.
– dangers électriques
Pendant un orage, il ne faut pas utiliser le matériel et il ne faut pas brancher ou
débrancher les câbles.
ement/débranchement des câbles en cuivre
Les câbles en cuivre sont lourds et ne sont pas flexibles, il faut donc faire très attention
en les branchant et en les débranchant des connecteurs. Consultez le fabricant des
câbles pour connaître les mises en garde et les instructions spéciales.
lation du matériel
Ce matériel ne doit être installé, remplacé ou entretenu que par du personnel formé et
qualifié.
ation du matériel
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et
réglementations nationales en vigueur.
Mellanox Technologies
49
Rev 1.7
électriques locaux et nationaux
Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux.
8. Exposition au rayonnement grave
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de
procédures autres que ce qui est spécifié dans les présentes peut engendrer une
exposition au rayonnement grave.
PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus
récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825-
1:1994+A1:1996+ A2:2001
50
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Appendix E: Sicherheitshinweise (Warnings in German)
lationsanleitungen
Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung
anschließen.
2.Übertemperatur
Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der
maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Es ist ein Luft-
strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich. Außerdem
sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen
einwandfreien Luftstrom zu gewährleisten.
Gewitter - Elektrische Gefahr
Arbeiten Sie während eines Gewitters und Blitzschlag nicht am Gerät, schließen Sie
keine Kabel an oder ab.
ießen/Trennen von -Kupferkabel
Kupferkabel sind schwer und nicht flexible. Deshalb müssen sie vorsichtig an die
Anschlüsse angebracht bzw. davon getrennt werden. Lesen Sie die speziellen Warnun-
gen und Anleitungen des Kabelherstellers.
5. Geräteinstallation
Diese Gerät sollte nur von geschultem und qualifiziertem Personal installiert, ausge-
tauscht oder gewartet werden.
äteentsorgung
Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim-
mungen erfolgen.
ale und nationale elektrische Bestimmungen
t
Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen
Bestimmungen installiert werden.
Mellanox Technologies
51
Rev 1.7
This equipment should be installed in compliance with local and national elec-
trical codes.
enkontak
Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von
Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon-
takt führen..
Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards :
ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+
A2:2001
52
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Appendix F: Advertencias de seguridad para la instalación
(Warnings in Spanish)
cciones de instalación
Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de
instalación.
alentamiento
No se debe utilizar el equipo en un área con una temperatura ambiente superior a la
máxima recomendada: 55°C(131°F). Además, para garantizar una circulación de aire
adecuada, se debe dejar como mínimo un espacio de 8 cm (3 pulgadas) alrededor de las
aberturas de ventilación.
hay rayos: peligro de descarga eléctrica
No utilizar el equipo ni conectar o desconectar cables durante períodos de actividad de
rayos.
ón y desconexión del cable Copper
Dado que los cables de cobre son pesados y no son flexibles, su conexión a los conec-
tores y su desconexión se deben efectuar con mucho cuidado. Para ver advertencias o
instrucciones especiales, consultar al fabricante del cable.
ación de equipos
La instalación, el reemplazo y el mantenimiento de este equipo estarán a cargo única-
mente de personal capacitado y competente.
ación de equipos
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y
reglamentaciones nacionales.
Mellanox Technologies
53
Rev 1.7
7.Códigos eléctricos locales y nacionales
Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales.
ción a niveles de radiación peligrosos
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de
los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos.
PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes:
IEC 60825-1:2007/03 y EN 60825-1:2007
54
Mellanox Technologies
2024年7月31日发(作者：郎晨辰)
ConnectX®-3 Ethernet Single and Dual SFP+
Port Adapter Card User Manual
P/N:
MCX312A-XCBT, MCX312B-XCBT, MCX311A-XCAT
Rev 1.7
Rev 1.7ConnectX®-3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE
PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE
OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE
CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX
TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX
TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE
HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE
DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO,
PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND
RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
Tel: (408) 970-3400
Mellanox Technologies, Ltd.
Beit Mellanox
PO Box 586 Yokneam 20692
Israel
Tel: +972 (0)74 723 7200
© Copyright 2012. Mellanox Technologies. All rights reserved.
Mellanox®, Mellanox logo®, BridgeX®, ConnectX®, CORE-Direct®, InfiniBridge®, InfiniHost®, InfiniScale®, PhyX®,
SwitchX®, Virtual Protocol Interconnect® and Voltaire® are registered trademarks of Mellanox Technologies, Ltd. Connect-
IB™, FabricIT™, MLNX-OS™, ScalableHPC™, TestX™, Unbreakable-Link™, UFM™ and Unified Fabric Manager™ are
trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
2
Mellanox Technologies
Document Number: 3755
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Table of Contents
Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6
Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7
About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8
Intended Audience. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Related Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Technical Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Firmware and Software Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1 Supported Network Protocol Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1.1 10 Gigabit Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2 Adapter Cards Covered in this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Finding the MAC and Serial Number on the Adapter Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Chapter 2 Adapter Card Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1 I/O Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.1 Ethernet SFP+ Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.2 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.3 LED Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.4 I2C-compatible Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 SFP+ Power Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.1 System Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.2 SPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.3 EEPROM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Chapter 3 Ethernet Adapter Card Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 Identify the Card in Your System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.1 On Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.2 On Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4 Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.4.1 Inserting the Optical Transceiver Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.4.2 Cable Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Chapter 4 Driver Software and Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1 Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1.1 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1.2 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2 FlexBoot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3 RDMA over Converged Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3.1 Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4 Updating Adapter Card Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Appendix A Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
A.1MCX311A-XCAT Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Mellanox Technologies
3
Rev 1.7
A.2MCX312[AB]-XCBT Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
A.3Board Mechanical Drawing and Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
A.4Regulatory Statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Appendix B Interface Connectors Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
B.1I2C-compatible Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
B.2PCI Express x8 Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
B.3SFP+ Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Appendix C Replacing a Tall Bracket With a Short Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
C.1Replacing a Bracket. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
C.2Removing the Existing Bracket from the Adapter Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
C.3Installing the New Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Appendix D Avertissements de sécurité d’installation (Warnings in French). . . . . . . . . . . . . . 49
Appendix E Sicherheitshinweise (Warnings in German). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish). . . . . . . . . 53
4
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
List of Tables
Table 1:
Table 2:
Table 3:
Table 4:
Table 5:
Table 6:
Table 7:
Table 8:
Table 9:
Table 10:
Table 11:
Table 12:
Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Documents List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Single and Dual-port 10 Gigabit Ethernet Adapter Card . . . . . . . . . . . . . . . . . . . . . . . . 11
Physical and Logical Link Indication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Jumper Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
MCX312A-XCBT VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
MCX312B-XCBT VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
MCX311A-XCAT VPD Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
MCX311A-XCAT Specifications Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
MCX312A-XCBT Specifications Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
I2C-compatible Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
SFP+ Connector Pinout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Mellanox Technologies
5
Rev 1.7
List of Figures
Figure 1:
Figure 2:
Figure 3:
Figure 4:
Figure 5:
Figure 6:
Figure 7:
Figure 8:
Figure 9:
Card Product Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Board Labels for MCX312B-XCBT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
MCX312A-XCBT Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
MCX312B-XCBT Card. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
MCX311A-XCAT Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Dual-port Bracket for MCX312A-XCBT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Dual-port Bracket for MCX312B-XCBT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Single-port Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
I2C-compatible Connector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Figure 10:PCI Device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Figure 11:Module With Locking Mechanism Closed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Figure 12:Module With Locking Mechanism Open . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Figure 13:Device Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Figure 14:Support Download Assistant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Figure 15:Mechanical Drawing of the Dual-port MCX311A-XCAT Adapter Card. . . . . . . . . . . . 39
Figure 16:Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card . . . . . . . . . . . 40
Figure 17:Mechanical Drawing of the Dual-port MCX312B-XCBT . . . . . . . . . . . . . . . . . . . . . . . 40
Figure 18:Compatible Connector Plug and Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Figure 19:PCIe Connector Pinout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Figure 20:Rear View of Module With Pin Placement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Figure 21:Bracket Screws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Figure 22:Placing the Bracket on the Card (MCX312A-XCBT). . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Figure 23:Placing the Bracket on the Card (MCX312B-XCBT). . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Figure 24:Placing the Bracket on the Card (MCX311A-XCAT) . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Revision History
This document was printed on November 14, 2012.
Table 1 - Revision History Table
Date
November 2012
Rev
1.7
Comments/Changes
•Added MCX312B-XCBT to the following locations:
- Section 1.2, “Adapter Cards Covered in this Manual,” on
page 11
- Figure 4, “MCX312B-XCBT Card,” on page 15
- Figure 7, “Dual-port Bracket for MCX312B-XCBT,” on
page 16
-
Table 7, “MCX312B-XCBT VPD Layout,” on
page 21
- Figure 17, “Mechanical Drawing of the Dual-port MCX312B-
XCBT,” on page 40
October 20121.6•
•
•
•
•
Added single-port SFP+ card to all necessary locations
Fixed Table 5, “Jumper Configuration,” on page 19
Updated figures in Section 4.1.2, “Windows,” on page 31 by pro-
viding ConnectX-3 examples
Removed all references to 40 Gigabit Ethernet from the User
Manual
Added operational and non-operational temperature and humid-
ity level to the following tables:
- Table 10, “MCX312A-XCBT Specifications Table,” on
page 38
Minor edits
Updated LED functions in Section 2.1.3, “LED Assignment,” on
page 17
Formatted specification tables in Appendix A, “Specifications,”
on page 37
October 2012
August 2012
1.5
1.4
January 20121.2•
•
•
October 2011
July 2011
July 2011
1.1
1.0
0.10
Added new OPNs based on ConnectX-3 Step A1 devices
Minor edits
First Release
Mellanox Technologies
7
Rev 1.7
About this Manual
This User Manual describes Mellanox Technologies ConnectX®-3 10 Gigabit Ethernet Single
Dual SFP+ port PCI Express x4 or x8 adapter cards. It provides details as to the interfaces of the
board, specifications, required software and firmware for operating the board, and relevant docu-
mentation.
Intended Audience
This manual is intended for the installer and user of these cards.
The manual assumes basic familiarity with Ethernet networks and architecture specifications.
Related Documentation
Table 2 - Documents List
Mellanox Firmware Tools (MFT) User Manual
Document no. 2204UG
IEEE Std 802.3 Specification
PCI Express 3.0 Specifications
User Manual describing the set of MFT firmware management tools for a
single node.
See
=> Support => Download Firmware Tools
This is the IEEE Ethernet specification
/getieee802
Industry Standard PCI Express 3.0 Base and Card Electromechanical
Specifications
Document Conventions
When discussing memory sizes, MB and MBytes are used in this document to mean size in mega
bytes. The use of Mb or Mbits (small b) indicates size in mega bits. In this document PCIe is used
to mean PCI Express.
Technical Support
Customers who purchased Mellanox products directly from Mellanox are invited to contact us
through the following methods.
•URL:
=> Support
•E-mail: support@
•Tel: +1.408.916.0055
Customers who purchased Mellanox M-1 Global Support Services, please see your contract for
details regarding Technical Support.
8
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Customers who purchased Mellanox products through a Mellanox approved reseller should first
seek assistance through their reseller.
Firmware and Software Updates
The Mellanox support downloader contains software, firmware and knowledge database informa-
tion for Mellanox products. Access the data base from the Mellanox Support web page,
=> Support
or use the following link to go directly to the Mellanox Support Download Assistant page,
/supportdownloader/.
Mellanox Technologies
9
Rev 1.7Overview
1Overview
This document is a User Manual for Mellanox Technologies Ethernet cards based on the Con-
nectX®-3 EN integrated circuit device. The cards described in this manual have the following
main features:
•IEEE Std. 802.3 compliant
•PCI Express 3.0 (1.1 and 2.0 compatible) through an x4 or x8 edge connector up to 8GT/s
•10 Gb/s Ethernet
•Single and Dual SFP+ ports for connection Ethernet traffic
•RDMA over Converged Ethernet (RoCE)
•End-to-end QoS and congestion control
•Hardware-based I/O virtualization
•TCP/UDP/IP stateless off-load
•RoHS-R6 compliant
•Two bracket heights: short and tall
1.1Supported Network Protocol Standards
1.1.110 Gigabit Ethernet
10 Gigabit Ethernet is an Ethernet network protocol standard for LANs specifying data transfer
rate of 10Gb/s. Mellanox adapters comply with the following IEEE 802.3* standards:
•IEEE Std 802.3-2008 Ethernet
•IEEE Std 802.3ae 10 Gigabit Ethernet
•IEEE Std 802.3ad Link Aggregation and Failover
10
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
1.2Adapter Cards Covered in this Manual
Table 3 lists the single and dual port 10 Gigabit Ethernet adapter cards described in this manual.
Table 3 - Single and Dual-port 10 Gigabit Ethernet Adapter Card
Ordering Part Number
(OPN)
PCI Express
SERDES Speed
Data Transmis-
sion Rate/
# of ports
RoHS
IC
Part Number
Device ID (decimal)
MCX312A-XCAT
(Legacy product)
PCIe 3.0 x4/x8
8GT/s
10 GigE dual-port
SFP+
R-6MT27518A0-FCCR-XE
•
•
4099 for Physical
Function
4100 for Virtual
Function
4099 for non-virtu-
alized servers
4100 for virtual
machines
4099 for Physical
Function
4100 for Virtual
Function
4099 for Physical
Function
4100 for Virtual
Function
MCX311A-XCAT PCIe 3.0 x4
8GT/s
10 GigE single-
port SFP+
R-6MT27518A1-FCCR-XE
•
•
MCX312A-XCBTPCIe 3.0 x4/x8
8GT/s
10 GigE dual-port
SFP+
R-6MT27518A1-FCCR-XE
•
•
MCX312B-XCBTPCIe 3.0 x4/x8
8GT/s
10 GigE dual-port
SFP+
R-6MT27518A1-FCCR-XE
•
•
1.3Finding the MAC and Serial Number on the Adapter Card
Each Mellanox adapter card has a label on the print side that shows the card serial number and the
card MAC for the Ethernet protocol.
Figure 1: Card Product Label (Example)
Mellanox Technologies
11
Rev 1.7Overview
Figure 2: Board Labels for MCX312B-XCBT
1.4Safety Warnings
Below is a list of safety warnings in English. For safety warnings in other languages, please refer
to the appendices in this user manual.
lation Instructions
Read all installation instructions before connecting the equipment to the power source.
-temperature
This equipment should not be operated in an area with an ambient temperature exceed-
ing the maximum recommended: 55°C (131°F).
To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ven-
tilation openings.
Lightning - Electrical Hazard
During periods of lightning activity, do not work on the equipment or connect or dis-
connect cables.
Cable Connecting/Disconnecting
Some copper cables are heavy and not flexible, as such they should be carefully
attached to or detached from the connectors. Refer to the cable manufacturer for spe-
cial warnings and instructions.
12
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
ent Installation
This equipment should be installed, replaced, or serviced only by trained and qualified
personnel.
ent Disposal
Disposal of this equipment should be in accordance to all national laws and regula-
tions.
7.
Local and National Electrical Codes
This equipment should be installed in compliance with local and national electrical
codes.
ous Radiation Exposure
Caution – Use of controls or adjustment or performance of procedures other than
those specified herein may result in hazardous radiation exposure.
CLASS 1 LASER PRODUCT and reference to the most recent laser standards:
IEC 60 825-1:1993 + A1:1997 + A2:2001 and EN 60825-1:1994+A1:1996+
A2:2001
Mellanox Technologies
13
Rev 1.7Adapter Card Interfaces
2
2.1
Adapter Card Interfaces
I/O Interfaces
Each adapter card includes the following interfaces:
•SFP+ port(s)
•PCI Express (PCIe) x4 or x8 edge connector
•I/O panel LEDs
•I
2
C-compatible connector (for debug)
Figure 3: MCX312A-XCBT Card
14
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Figure 4: MCX312B-XCBT Card
Figure 5: MCX311A-XCAT Card
The adapter cards include special circuits to protect from ESD shocks to the card/server when
plugging copper cables.
Mellanox Technologies
15
Rev 1.7Adapter Card Interfaces
Figure 6: Dual-port Bracket for MCX312A-XCBT
Figure 7: Dual-port Bracket for MCX312B-XCBT
16
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Figure 8: Single-port Bracket
For dual-port cards, Port 1 connects to connector 1 of the device, while Port 2 connects to connec-
tor 2 of the device.
2.1.1Ethernet SFP+ Interface
The network ports of the ConnectX®-3 adapter cards are compliant with the IEEE 802.3 Ethernet
standards listed in Section 1.1.1, “10 Gigabit Ethernet,” on page 10. The SFP+ port has one Tx/Rx
pair of SerDes. Ethernet traffic is transmitted through the cards' SFP+ connectors.
2.1.2PCI Express Interface
The ConnectX®-3 adapter cards support PCI Express 3.0 (1.1 and 2.0 compatible) through an x4
or x8 edge connector. The device can be either a master initiating the PCI Express bus operations
or a slave responding to PCI bus operations.
2.1.3LED Assignment
There are two I/O LEDs per port in dual-port designs and one bicolor LED in single-port designs,
green and yellow, located on the I/O panel. See Table 4 for different LED functions.
Mellanox Technologies
17
Rev 1.7Adapter Card Interfaces
Table 4 - Physical and Logical Link Indication
LED Function
Green - physical link •
•
•
•
Constant on indicates a good physical link
If neither LED is lit, then the physical link has not been established
A blinking yellow indicates activity (data transfer)
Stays off when there is no activity
Yellow -
logical (data activity
link)
The short bracket has the same port and LED footprints as the tall bracket.
2.1.4I
2
C-compatible Interface
A three-pin header on the adapter cards is provided as the I
2
C-compatible interface. See Figure 16,
“Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card,” on page 40 for the loca-
tion on the board.
Figure 9: I
2
C-compatible Connector
2.2Power
All adapter cards receive 12V and 3.3V power from the PCI Express edge connector. All other
required power voltages are generated by on-board switch mode regulators. See “Specifications”
on page 37.
2.3SFP+ Power Level
The adapter cards support power levels according to SFF Committee SFF-8431 Specification for
SFP+ (Small Form-factor Pluggable) Transceiver.
2.4Memory
The adapter cards support multiple memory devices through the PCIe, SPI (Flash) and I
2
C inter-
faces.
18
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
2.4.1System Memory
The adapter cards utilize the PCI Express interface to store and access Ethernet fabric connection
information and packet data on the system memory.
2.4.2SPI
Each of the adapter cards includes one 16MB SPI Flash device (M25PX16-VMN6P device by ST
Microelectronics) accessible via the SPI interface of the ConnectX®-3 EN device.
When a jumper is inserted into the drill holes in the adapter it indicates to the device that the on-
board Flash device should not be used for boot instructions. Contact your Mellanox support repre-
sentative should you need to use this jumper. Table 5 provides information on this jumper. The
jumper location on the board is illustrated in Appendix A.3, “Board Mechanical Drawing and
Dimensions,” on page 39.
Table 5 - Jumper Configuration
DescriptionOption
Card Default
Configuration
Flash present/
not present
connection open – Flash presentconnection open –
Flash present
connection shorted – Flash not present
2.4.3EEPROM
Each board incorporates an EEPROM that is accessible through the I
2
C-compatible interface. The
EEPROM capacity is 4Kb. The EEPROM is used for storing the Vital Product Data (VPD).
The PCI VPD (Vital Product Data) layout for each of the adapter cards complies with the format
defined in the PCI 3.0 Specification, Appendix I.
Table 6 - MCX312A-XCBT VPD Layout
Offset
(Decimal)
0
1
2
3
27
Item
Large Resource Type ID
String Tag (0x02)
Length [7:0] LSB
Length [15:8] MSB
Data
Large Resource Type VPD-R
Tag (0x10)
Value
0x82
0x18
0x0
CX312A - Con-
nectX-3 SFP+
0x90
STR
FormatDescription
Mellanox Technologies
19
Rev 1.7
Offset
(Decimal)
28
29
30
32
33
54
56
57
59
61
62
86
88
89
105
107
108
109
110
111
112
114
115
121
123
124
148
150
151
256
258
259
512
514
515
768
770
771
1024
1026
1027
1280
Adapter Card Interfaces
Item
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Part Number
VPD Keyword
Length
Revision
VPD Keyword
Length
SerialNumber
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Large Resource Type VPD-W
Tag (0x11)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Value
0x4F
0x00
PN
0x15
PN
EC
0x2
RV
SN
0x18
SN
V0
0x10
PCIe Gen3 x8
RV
0x1
0,107
0x91
0x8F
0xF
V1
0x6
N/A
YA
0x18
N/A
RW
0x69
FormatDescription
STR
%STR_SPC
STR
Add in Card Part Number
Engineering Change Level of
the card (rev)
PCB revision
Serial Number
“00..00XXXX..XX”
Misc Information
%STR
STR
%STR_SPC
STR
STR_SPC
STR
%CS0
STR
STR_SPC
STR
STR_SPC
STR
STR_ZERO
EFI Driver version
Asset Tag
“N/A”
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR
STR_ZERO
RW
0xfd
STR
STR_ZERO
RW
0xfd
STR
STR_ZERO
RW
0xfd
STR
STR_ZERO
RWSTR
20
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User Manual
Offset
(Decimal)
1282
1283
1536
1538
1539
1792
1794
1795
2048
2050
2051
2304
2306
2307
2560
2562
2563
2816
2818
2819
3072
3074
3075
3328
3330
3331
3584
3586
3587
3840
3842
3843
4095
Rev 1.7
Item
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Small Resource Type END
Tag (0x11)
Value
0xfd
FormatDescription
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfd
STR_ZERO
RW
0xfc
STR_ZERO
0x78
STR
STR
STR
STR
STR
STR
STR
STR
STR
STR
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Remaining read/write area
Reserved (0x00)
Table 7 - MCX312B-XCBT VPD Layout
Offset
(Decimal)
0
1
2
Item
Large Resource Type ID
String Tag (0x02)
Length [7:0] LSB
Length [15:8] MSB
Value
0x82
0x18
0x0
FormatDescription
Mellanox Technologies
21
Rev 1.7Adapter Card Interfaces
Table 7 - MCX312B-XCBT VPD Layout
Offset
(Decimal)
3
27
28
29
30
32
33
54
56
57
59
61
62
86
88
89
105
107
108
109
110
111
112
114
115
121
123
124
148
150
151
256
258
259
512
514
515
768
770
771
Item
Data
Large Resource Type VPD-R
Tag (0x10)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Part Number
VPD Keyword
Length
Revision
VPD Keyword
Length
SerialNumber
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Large Resource Type VPD-W
Tag (0x11)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
RW
0xfd
STR_ZEROReserved (0x00)
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Value
CX312B - Con-
nectX-3 SFP+
0x90
0x4F
0x00
PN
0x15
PN
EC
0x2
RV
SN
0x18
SN
V0
0x10
PCIe Gen3 x8
RV
0x1
0,107
0x91
0x8F
0xF
V1
0x6
N/A
YA
0x18
N/A
RW
0x69
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
STR_SPC
STR
“N/A”
Remaining read/write area
STR_SPC
STRAsset Tag
STREFI Driver version
%CS0
STR_SPC
STR
%STR_SPC
STR
“00..00XXXX..XX”
Misc Information
%STR
STR
PCB revision
Serial Number
%STR_SPC
STREngineering Change Level of
the card (rev)
STRAdd in Card Part Number
Format
STR
Description
22
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Table 7 - MCX312B-XCBT VPD Layout
Offset
(Decimal)
1024
1026
1027
1280
1282
1283
1536
1538
1539
1792
1794
1795
2048
2050
2051
2304
2306
2307
2560
2562
2563
2816
2818
2819
3072
3074
3075
3328
3330
3331
3584
3586
3587
3840
3842
3843
4095
Item
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Small Resource Type END
Tag (0x11)
0x78
RW
0xfc
STR_ZEROReserved (0x00)
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Value
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Format
STR
Description
Remaining read/write area
Mellanox Technologies
23
Rev 1.7Adapter Card Interfaces
Table 8 - MCX311A-XCAT VPD Layout
Offset
(Decimal)
0
1
2
3
27
28
29
30
32
33
54
56
57
59
61
62
86
88
89
105
107
108
109
110
111
112
114
115
121
123
124
148
150
151
256
258
259
512
Item
Large Resource Type ID
String Tag (0x02)
Length [7:0] LSB
Length [15:8] MSB
Data
Large Resource Type VPD-R
Tag (0x10)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Part Number
VPD Keyword
Length
Revision
VPD Keyword
Length
SerialNumber
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Large Resource Type VPD-W
Tag (0x11)
Length [7:0] LSB
Length [15:8] MSB
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD KeywordRW
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Value
0x82
0x18
0x0
CX311A - Con-
nectX-3 SFP+
0x90
0x4F
0x00
PN
0x15
PN
EC
0x2
RV
SN
0x18
SN
V0
0x10
PCIe Gen3 x4
RV
0x1
0,107
0x91
0x8F
0xF
V1
0x6
N/A
YA
0x18
N/A
RW
0x69
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
STR_SPC
STR
“N/A”
Remaining read/write area
STR_SPC
STRAsset Tag
STREFI Driver version
%CS0
STR_SPC
STR
%STR_SPC
STR
“00..00XXXX..XX”
Misc Information
%STR
STR
PCB revision
Serial Number
%STR_SPC
STREngineering Change Level of
the card (rev)
STRAdd in Card Part Number
STR
FormatDescription
24
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Table 8 - MCX311A-XCAT VPD Layout
Offset
(Decimal)
514
515
768
770
771
1024
1026
1027
1280
1282
1283
1536
1538
1539
1792
1794
1795
2048
2050
2051
2304
2306
2307
2560
2562
2563
2816
2818
2819
3072
3074
3075
3328
3330
3331
3584
3586
3587
3840
3842
3843
4095
Item
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
VPD Keyword
Length
Data
Small Resource Type END
Tag (0x11)
0x78
RW
0xfc
STR_ZEROReserved (0x00)
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
RW
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
Value
0xfd
STR_ZERO
STR
Reserved (0x00)
Remaining read/write area
FormatDescription
Mellanox Technologies
25
Rev 1.7Adapter Card Interfaces
26
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
3
3.1
Ethernet Adapter Card Installation
Hardware Requirements
A system with a PCI Express x4 or x8 slot is required for installing the card.
3.2Installation Instructions
Read all installation instructions before connecting the equipment to the power source.
To change a tall bracket to a short bracket see Replacing a Tall Bracket With a Short
Bracket on page 46.
The adapter cards listed above are standard PCI Express cards, each with a standard x8 edge con-
nector. The cards require a PCI Express x4 or x8. Please consult the host machine documentation
for instructions on how to install a PCI Express card.
If the card is installed in a PCI slot with less lanes than the card requires then the adapter
card will not provide the optimum data transfer.
3.3Identify the Card in Your System
1.
2.
3.
4.
Open Device Manager on the server. Click start => Run, and then enter “”.
Expand System Devices and locate your Mellanox ConnectX-3 adapter card.
Select Properties to display the adapter card properties window.
Click the Details tab and select Device Instance Id
(Windows 2003) or
Hardware Ids (Windows 2008/R2) from the Property pull-down menu.
3.3.1On Windows
Mellanox Technologies
27
Rev 1.7Ethernet Adapter Card Installation
Figure 10: PCI Device
the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the
display example above, notice the sub-string “PCIVEN_15B3&DEV_1003”: VEN is equal
to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 1003 –
this is a valid Mellanox Technologies PCI Device ID.
If the PCI device does not have a Mellanox adapter ID, return to Step 4 to check another
device.
The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi-
tory at /read/PC/15b3.
3.3.2On Linux
Get the device location on the PCI bus by running lspci and locating lines with the string “Mella-
nox Technologies”:
> lspci |grep -i Mellanox
27:00.0 Network controller: Mellanox Technologies MT27500 Fam-
ily [ConnectX-3]
28
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
3.4Cables and Modules
Please refer to “Mellanox Products Approved Cable Lists” at /related-
docs/user_manuals/Mellanox_approved_
3.4.1Inserting the Optical Transceiver Module
To insert the module into the cage:
1.
2.
Open the module’s locking mechanism – see Figure 11 and Figure 12.
Make sure that the male connectors on the module will align with the female connectors
inside of the cage. Also check that there is no dirt or foreign matter in the module or in the
cage.
Figure 11: Module With Locking Mechanism Closed
Figure 12: Module With Locking Mechanism Open
3.
4.
Insert the module into the adapter card module cage.
Close the locking Mechanism.
To remove the module from the cage:
1.
2.
Unlock the locking mechanism by opening the handle.
Pull the module out of the cage.
3.4.2Cable Installation
1.
2.
All cables can be inserted or removed with the unit powered on. To insert a cable, press the
connector into the port receptacle until the connector is firmly seated.
After inserting a cable into a port, the GREEN LED indicator will light when the physical
connection is established (that is, when the unit is powered on and a cable is plugged into the
port with the other end of the connector plugged into a functioning port).
After plugging in a cable, lock the connector using the latching mechanism particular to the
cable vendor. When a logical connection is made the YELLOW LED will come on. When
data is being transferred the yellow led will blink.
3.
Mellanox Technologies
29
Rev 1.7Ethernet Adapter Card Installation
When installing cables make sure that the latches engage.
Always install and remove cables by pushing or pulling the cable and connector in a
straight line with the card.
4.
5.
Care should be taken as not to impede the air exhaust flow through the ventilation holes.
Cable lengths which allow for routing horizontally around to the side of the chassis before
bending upward or downward in the rack should be used
To remove a cable, disengage the locks and slowly pull the connector away from the port
receptacle. Both LED indicators will turn off when the cable is unseated.
Cables, especially long copper cables, can weigh a substantial amount. Make sure that
the weight of the cable is supported on its own and is not hanging from the adapter
card.
30
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
4
4.1
Driver Software and Firmware
Driver Software
4.1.1Linux
For Linux, download and install the latest MLNX_EN driver software package available via the
Mellanox web site at: => Products => Software/Drivers => Ethernet
SW/Drivers => ConnectX®-3 EN 10GigE Linux Driver => Download. Follow the installation
instructions included in the download package (also available from the download page). To ensure
that communication has been established follow the instructions below.
Check the link status
First check the network interface name by running the “ifconfig –a” command
To assign an IP address to the interface run:
> ifconfig eth
where 'x' is the OS assigned interface number.
To check driver and device information run:
> ethtool -i eth
Example:
> ethtool -i eth2
driver:mlx4_en (MT_1020110019_CX-3)
version: 1.5.6.33 (Oct 2011)
firmware-version:2.10.0000
bus-info: 0000:07:00.0
4.1.2Windows
For Windows, download the MLNX EN – Ethernet driver. Download this package from the Mella-
nox web site at:
=> Products => Software/Drivers => Ethernet SW/
Drivers => ConnectX® EN 10GigE Windows Driver => Download. Follow the installation
instructions included in the download package (also available from the download page). To ensure
that communication has been established follow the instructions below.
Mellanox Technologies
31
Rev 1.7Driver Software and Firmware
To display a summary of network adapter software-, firmware- and hardware-related information
such as driver version, firmware version, bus interface, adapter identity, and network port link
information, perform the following steps:
Step 1: Display the Device Manager
Figure 13: Device Manager
Step 2: Right-click a Mellanox ConnectX Ethernet adapter (under “Network adapters list) and left-
click Properties.
32
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Step 3: Select the Information tab from the Properties sheet (example).
To save this information for debug purposes, click Save To File and provide the output
file name.
4.2FlexBoot
FlexBoot enables remote boot over Ethernet, Boot over Ethernet (BoE) or Boot over iSCSI (Bo-
iSCSI). This technology is based on the Preboot Execution Environment (PXE) standard specifica-
tion, and FlexBoot software is based on the open source iPXE project (see ). For
more information go to
=> Products => Software/Drivers => Ethernet
Software/Drivers => FlexBoot => Download.
4.3RDMA over Converged Ethernet
ConnectX®-3 connected to an Ethernet fabric provides all of the basic NIC functionality plus
RDMA over Converged Ethernet (RoCE). RoCE utilizes advances in Data Center Bridging (DCB)
to enable efficient and low cost implementations of RDMA over Ethernet, supporting the entire
breadth of RDMA and low latency features. This includes reliable connected service, datagram
service, RDMA and send/receive semantics, atomic operations, user level multicast, user level I/O
access, kernel bypass, and zero copy. This capability is enabled when using the Mellanox OFED or
WinOF VPI drivers.
Mellanox Technologies
33
Rev 1.7Driver Software and Firmware
ConnectX®-3 EN with RoCE based network management is the same as that for any Ethernet and
DCB-based network management, eliminating the need for IT managers to learn new technolo-
gies.
4.3.1Hardware and Software Requirements
Software:
•Mellanox OFED 1.5.1 or WinOF 2.1.2 or later. To download these packages go to
=> Products => Software/Drivers => InfiniBand & VPI SW/Driver.
Hardware:
•ConnectX®-3 EN adapter card
4.4Updating Adapter Card Firmware
Each card is shipped with the latest version of qualified firmware at the time of manufacturing.
Firmware is updated occasionally, and the most recent firmware can be obtained from:
=> Support => Download Firmware. Check that the firmware on your
card is the latest found on the Mellanox site, if not update to the latest version found on the Mella-
nox web site.
Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firm-
ware Tools (MFT) package. This package is available for download, along with its user manual,
from the Mellanox Firmware Tools page. See => Support => Download
Firmware Tools.
A firmware binaries table lists a binary file per adapter card. The file name of each such binary is
composed by combining the firmware name, the firmware release version, and the card part num-
ber. Please contact Mellanox System Support if you cannot find the firmware binary for your
adapter card.
The following steps describe how to retrieve the PSID (firmware identification) and programmed
firmware version of your adapter card. They also describe how to update the card with the latest
firmware version available.
ve the PSID and firmware version:
l the MFT package. The package is available at => Products
=> Software/Drivers => Ethernet Software/Drivers => Firmware Tools. Make sure to down-
load the package corresponding to your computer’s operating system.
: mst start.
the Mellanox mst device name using the command "mst status". The mst device name will be of
the form: /dev/mst/mt4099_pci_cr0.
the PSID (firmware identification) and programmed firmware version using the command.
34
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
> flint -d /dev/mst/mt4099_pci_cr0 q
Image type: ConnectX
FW Version: 2.9.4000
Device ID: 4099
Chip Revision: 0
Description: Node Port1
Port2 Sys image
GUIDs: 000002c900000200 000002c900000201
000002c900000202 000002c900000203
MACs:
000002c90200 000002c90201
Board ID: (MT_1020110019)
VSD:
PSID: MT_1020110019
e the programmed firmware version with the latest available.
to Mellanox’s web site: /supportdownloader.
See Figure 14.
your card PSID to display the latest firmware.
Figure 14: Support Download Assistant
a newer firmware version exists for your adapter card, update the firmware as follows:
ad the firmware image zip file from the Download Center (see Step 2a above).
the firmware image.
the firmware image. Enter:
Mellanox Technologies
35
Rev 1.7Driver Software and Firmware
> flint -d /dev/mst/mt4099_pci_cr0 -i
the computer.
: mst start.
that the card firmware was updated successfully.
> flint -d /dev/mst/mt4099_pci_cr0 q
Image type: ConnectX
FW Version: 2.9.4100
Device ID: 4099
...
36
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Appendix A: Specifications
A.1MCX311A-XCAT Specifications
Table 9 - MCX311A-XCAT Specifications Table
Size: 2.11in. x 4.18 in. (53.59mm x 106.06 mm)
Physical
Connector: SFP+ 10Gb/s
Ethernet: XAUI/10GBASE-KX4 – 4X 10GigE port, XFI/SFI/10GBASE-KR – 1X
10GigE port, SGMII/1000BASE-KX – 1X 1GigE port
Protocol Support
Data Rate: 1/10Gb/s – Ethernet
PCI Express Gen3: SERDES @ 8.0GT/s, 4 lanes (2.0 and 1.1 compatible)
Voltage: 12V, 3.3V
Typ Power: Passive Cables 3.47W
Max Power: Passive Cables 4.84W
Power and
Environmental
Temperature: Operational 0°C to 55°C
Non-operational 0°C to 70°C
Humidity: 90% relative humidity
a
Air Flow: 200LFM
b
EMC: Refer to the following link:
/related-docs/user_manuals/Regulatory_and_Compliance_
Regulatory
Safety: IEC/EN 60950-1:2006
ETSI EN 300 019-2-2
IEC 60068-2- 64, 29, 32
RoHS: RoHS-R6
Cable Support
Please refer to "Mellanox Products Approved Cable Lists" at:
/related-docs/user_manuals/Mellanox_approved_
both operational and non-operational states
flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet.
Mellanox Technologies
37
Rev 1.7
A.2MCX312[AB]-XCBT Specifications
Table 10 - MCX312A-XCBT Specifications Table
Size: 2.71in. x 5.6in. (68.90 mm x 142.25 mm)
Physical
Connector: SFP+ 10Gb/s
Ethernet: 10GBASE-CX4, 10GBASE-R, and 1000BASE-R
Protocol Support
Data Rate: 1/10Gb/s – Ethernet
PCI Express Gen3: SERDES @ 8.0GT/s, 4 lanes (2.0 and 1.1 compatible)
PCI Express Gen2: SERDES @ 5.0GT/s, 8 lanes (1.1 compatible)
Voltage: 12V, 3.3V
Typ Power: PCIe x4 Gen3
Passive Cables 3.83W
PCIe x8 Gen2
Passive Cables 5.31W
Power and
Environmental
Max Power: PCIe x4 Gen3
Passive Cables 4.55W
PCIe x8 Gen2
Passive Cables 6.20W
Temperature: Operational 0°C to 55°C
Non-operational 0°C to 70°C
Humidity: 90% relative humidity
a
Air Flow: 200LFM
b
EMC: Refer to the following link:
/related-docs/user_manuals/Regulatory_and_Compliance_
Regulatory
Safety: IEC/EN 60950-1:2006
ETSI EN 300 019-2-2
IEC 60068-2- 64, 29, 32
RoHS: RoHS-R6
Cable Support
Please refer to "Mellanox Products Approved Cable Lists" at:
/related-docs/user_manuals/Mellanox_approved_
both operational and non-operational states
flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet.
38
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
A.3Board Mechanical Drawing and Dimensions
All dimensions are in millimeters.
All the mechanical tolerances are +/- 0.1mm
Figure 15: Mechanical Drawing of the Dual-port MCX311A-XCAT Adapter Card
J1 - I
2
C Connector
Flash Jumper
142.24
40.5
49.09
22.61
53.59
45.95
12.75
Mellanox Technologies
39
Rev 1.7
Figure 16: Mechanical Drawing of the Dual-port MCX312A-XCBT Adapter Card
J1 - I
2
C Connector
Flash Jumper
142.25
64.4
50.00
56.15
68.90
45.96
12.75
Figure 17: Mechanical Drawing of the Dual-port MCX312B-XCBT
142.24
40.84
49.1
53.59
45.95
40
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
A.4Regulatory Statements
For regulatory statements for all ConnectX®-3 cards please refer to:
/related-docs/user_manuals/Regulatory_and_Compliance_
Mellanox Technologies
41
Rev 1.7
Appendix B: Interface Connectors Pinout
B.1I
2
C-compatible Connector Pinout
Figure 18: Compatible Connector Plug and Pinout
Table 11 - I2C-compatible Connector Pinout
Connector Pin Number
1
2
3
4
5
Signal Name
SPSDA
SPSCL
GND
NC
NC
B.2PCI Express x8 Connector Pinout
The adapter cards use a standard PCI Express x4 or x8 edge connector and the PCI Express x8
standard pinout according to the PCI Express 3.0 specification.
42
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Figure 19: PCIe Connector Pinout
Mellanox Technologies
43
Rev 1.7
B.3SFP+ Connector Pinout
Figure 20: Rear View of Module With Pin Placement
44
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Table 12 - SFP+ Connector Pinout
Pin
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Symbol Name
VeeT
TX_Fault
TX_Disable
SDA
SCL
MOD_ABS
RS0
RX_LOS
RS1
VeeR
VeeR
RD-
RD+
VeeR
VccR
VccT
VeeT
TD+
TD-
VeeT
Description
Transmitter Ground (Common with Receiver Ground)
a
Transmitter Fault.
b
Transmitter Disable. Laser output disabled on high or open.
c
2-wire Serial Interface Data Line
d
2-wire Serial Interface Clock Line
d
Module Absent. Grounded within the module
d
No connection required
Loss of Signal indication. Logic 0 indicates normal operation.
e
No connection required
Receiver Ground (Common with Transmitter Ground)
a
Receiver Ground (Common with Transmitter Ground)
a
Receiver Inverted DATA out. AC Coupled
Receiver Non-inverted DATA out. AC Coupled
Receiver Ground (Common with Transmitter Ground)
a
Receiver Power Supply
Transmitter Power Supply
Transmitter Ground (Common with Receiver Ground)
a
Transmitter Non-Inverted DATA in. AC Coupled.
Transmitter Inverted DATA in. AC Coupled.
Transmitter Ground (Common with Receiver Ground)
a
t ground is internally isolated from chassis ground.
b.T
FAULT
is an open collector/drain output, which should be pulled up with a 4.7k – 10k Ohms resistor
on the host board if intended for use. Pull up voltage should be between 2.0V to Vcc + 0.3V. A high
output indicates a transmitter fault caused by either the TX bias current or the TX output power exceed-
ing the preset alarm thresholds. A low output indicates normal operation. In the low state, the output is
pulled to <0.8V.
c. Laser output disabled on TDIS >2.0V or open, enabled on TDIS <0.8V
be pulled up with 4.7kΩ – 10kΩ on host board to a voltage between 2.0V and 3.6V. MOD_ABS
pulls line low to indicate module is plugged in.
is open collector output. Should be pulled up with 4.7kΩ – 10kΩ on host board to a voltage
between 2.0V and 3.6V. Logic 0 indicates normal operation; logic 1 indicates loss of signal.
Mellanox Technologies
45
Rev 1.7
Appendix C: Replacing a Tall Bracket With a Short Bracket
This section provides instructions on how to remove the tall bracket of a standard Mellanox Tech-
nologies adapter card and replace it with a short one.
C.1Replacing a Bracket
To replace the bracket you will need the following parts:
•the new bracket of the proper height
•one new square gasket for each of the ports
•the 2 screws saved from the removal of the bracket
•the 2 fiber washers saved from the removal of the bracket
C.2Removing the Existing Bracket from the Adapter Card
Figure 21: Bracket Screws
Screws
1.
2.
Remove the two screws holding the bracket in place.
The bracket comes loose from the card.
Be careful not to put stress on the LEDs.
the two screws and the two fiber washers.
46
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
C.3
1.
Installing the New Bracket
Place the bracket onto the card until the screw holes line up.
Do not force the bracket onto the card. You may have to gently push the LEDs using a
small screwdriver to align the LEDs with the holes in the bracket.
2. Screw on the bracket using the screws and washers saved from the procedure above step 1.
Figure 22: Placing the Bracket on the Card (MCX312A-XCBT)
Mellanox Technologies
47
Rev 1.7
Figure 23: Placing the Bracket on the Card (MCX312B-XCBT)
Figure 24: Placing the Bracket on the Card (MCX311A-XCAT)
3.
4.
Make sure that the LEDs are aligned onto the bracket holes.
Use a torque driver to apply up to 2 lbs-in torque on the screws.
48
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Appendix D: Avertissements de sécurité d’installation
(Warnings in French)
ctions d’installation
Lisez toutes les instructions d’installation avant de brancher le matériel à la source
d’alimentation électrique.
érature excessive
Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante
dépassant le maximum recommandé de 55°C (131°F). Un flux d’air de 200LFM à
cette température ambiante maximale est nécessaire. En outre, pour garantir un bon
écoulement de l’air, laissez au moins 8 cm (3 pouces) d’espace libre autour des ouver-
tures de ventilation.
– dangers électriques
Pendant un orage, il ne faut pas utiliser le matériel et il ne faut pas brancher ou
débrancher les câbles.
ement/débranchement des câbles en cuivre
Les câbles en cuivre sont lourds et ne sont pas flexibles, il faut donc faire très attention
en les branchant et en les débranchant des connecteurs. Consultez le fabricant des
câbles pour connaître les mises en garde et les instructions spéciales.
lation du matériel
Ce matériel ne doit être installé, remplacé ou entretenu que par du personnel formé et
qualifié.
ation du matériel
L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et
réglementations nationales en vigueur.
Mellanox Technologies
49
Rev 1.7
électriques locaux et nationaux
Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux.
8. Exposition au rayonnement grave
Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de
procédures autres que ce qui est spécifié dans les présentes peut engendrer une
exposition au rayonnement grave.
PRODUIT LASER DE CLASSE 1 » et références aux normes laser les plus
récentes CEI 60 825-1:1993 + A1:1997 + A2:2001 et NE 60825-
1:1994+A1:1996+ A2:2001
50
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Appendix E: Sicherheitshinweise (Warnings in German)
lationsanleitungen
Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung
anschließen.
2.Übertemperatur
Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der
maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Es ist ein Luft-
strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich. Außerdem
sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen
einwandfreien Luftstrom zu gewährleisten.
Gewitter - Elektrische Gefahr
Arbeiten Sie während eines Gewitters und Blitzschlag nicht am Gerät, schließen Sie
keine Kabel an oder ab.
ießen/Trennen von -Kupferkabel
Kupferkabel sind schwer und nicht flexible. Deshalb müssen sie vorsichtig an die
Anschlüsse angebracht bzw. davon getrennt werden. Lesen Sie die speziellen Warnun-
gen und Anleitungen des Kabelherstellers.
5. Geräteinstallation
Diese Gerät sollte nur von geschultem und qualifiziertem Personal installiert, ausge-
tauscht oder gewartet werden.
äteentsorgung
Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim-
mungen erfolgen.
ale und nationale elektrische Bestimmungen
t
Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen
Bestimmungen installiert werden.
Mellanox Technologies
51
Rev 1.7
This equipment should be installed in compliance with local and national elec-
trical codes.
enkontak
Achtung – Nutzung von Steuerungen oder Einstellungen oder Ausführung von
Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon-
takt führen..
Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards :
ICE 60 825-1:1993 + A1:1997 + A2:2001 und EN 60825-1:1994+A1:1996+
A2:2001
52
Mellanox Technologies
ConnectX®-3 Ethernet Single and Dual SFP+ Port Adapter Card User ManualRev 1.7
Appendix F: Advertencias de seguridad para la instalación
(Warnings in Spanish)
cciones de instalación
Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de
instalación.
alentamiento
No se debe utilizar el equipo en un área con una temperatura ambiente superior a la
máxima recomendada: 55°C(131°F). Además, para garantizar una circulación de aire
adecuada, se debe dejar como mínimo un espacio de 8 cm (3 pulgadas) alrededor de las
aberturas de ventilación.
hay rayos: peligro de descarga eléctrica
No utilizar el equipo ni conectar o desconectar cables durante períodos de actividad de
rayos.
ón y desconexión del cable Copper
Dado que los cables de cobre son pesados y no son flexibles, su conexión a los conec-
tores y su desconexión se deben efectuar con mucho cuidado. Para ver advertencias o
instrucciones especiales, consultar al fabricante del cable.
ación de equipos
La instalación, el reemplazo y el mantenimiento de este equipo estarán a cargo única-
mente de personal capacitado y competente.
ación de equipos
La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y
reglamentaciones nacionales.
Mellanox Technologies
53
Rev 1.7
7.Códigos eléctricos locales y nacionales
Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales.
ción a niveles de radiación peligrosos
Precaución: el uso de controles o ajustes o la realización de procedimientos distintos de
los que aquí se especifican podrían causar exposición a niveles de radiación peligrosos.
PRODUCTO LÁSER DE CLASE 1 y referencia a las normas de láser más recientes:
IEC 60825-1:2007/03 y EN 60825-1:2007
54
Mellanox Technologies