UCS

While UCS bundles often come with a selection of cables for within the UCS cluster, there are still things outstanding from the typical BOM that go unfulfilled. When we arrive at the client site, we’re often left scrambling. Patch cables (including interconnecting UCS with storage, UCS with upstream networks), power cords, heck even cabinets that fit. Some stuff Cisco sells, some stuff has to be sourced elsewhere.

This is a preliminary list. It is not laid out in a quick “this, and this, and this” list because most everything requires thought; length, what are they uplinking into, etc. I’m going to flesh this out more later.

Cabinet dimension requirements

• optimally, prefer cabinets that are ~42" deep. 38" deep would be very tight.
• UCS chassis is 32" deep.
• Usually see 2-3" gap in front, 7-8" gap in rear (lots of power cables and twinax in rear).
• UCS 5108 chassis spec sheet: http://www.cisco.com/en/US/prod/collateral/ps10265/ps10280/spec_sheet_c17-644224.pdf

Power requirements

• 4 2500 watt power supplies per chassis
• 4 C19 sockets per chassis
• 4 C13 sockets per FI cluster, on FRONT of switch, not port side
• Power calculator http://express.salire.com/Go/Cisco/Cisco-UCS-Power-Calculator.aspx

Power cables

• verify c13-c14 and c19-c20.
• per chassis, assume 4 c19-c20 2m
• per pair of FIs, assume 4 c13-c14 2m
• do we have enough sockets in the cabinet PDUs?

UCS FI cluster (Ethernet) cables

• always order 2 6" cat5 or better. Pick a funky colour! (Infinite Cables?)

UCS mgmt cables

• 2 3m cat5 cables. (Infinite Cables?)
• FI mgmt ports are on FRONT, not port side

Upstream interconnect cables and modules, worst case (ie, ask and plan and tune this list)

• assume 4 5m Cisco twinax
  • SFP-H10GB-CU3M= (Cisco)
• assume 4 5m lc-lc om3 patch (Infinite Cables?)
• assume 4 SR optical modules (for the upstream, 4 are usually included in bundles, need more?)
  • SFP-10G-SR= (Cisco)
• assume 8 copper SFP modules (if the client is uplinking into legacy hardware, many are)
  • GLC-T= (Cisco)
  • assume 8 5m cat6 (Infinite Cables?)

Storage interconnect cables

• assume 4 5m lc-lc om3 for FC (Infinite Cables?)
  • UCS bundles come with ~6 2/4/8Gb FC SFP modules, probably don’t need more.

Netapp

• assume 4 5m twinax for file and iSCSI (Netapp) (5M assuming adjacent cabinet)
  • SFP-H10GB-CU5M= (Cisco)

EMC

• assume 4 5m lc-lc om3 for file (EMC) (Infinite Cables?) (5M assuming adjacent cabinet)
• assume 4 5m lc-lc om3 for iSCSI (EMC) (Infinite Cables?) (5M assuming adjacent cabinet)
• I don’t know if VNX supports Cisco Twinax; we need to test at some point.

Cable management

• assume 32 short velcro straps minimum per UCS domain, 1 chassis. Gobs more is recommended.
• assume 32 long velcro straps minimum per UCS domain, 1 chassis.
  • 4-6 per bundle per IOM per chassis
  • 4-6 per storage fibre bundle per side
  • 4-6 per uplink bundle per side
  • 2-4 per power cord
  • 4-8 per side to organize bundles
  • 2+ RACK-CBLMGT-011 cable management straps (Cisco)

Mounting equipment

• assume 50 cage nuts and matching bolts minimum per UCS domain, 1-4 chassis.
• 6 per chassis
  • 16 per FI cluster
  • RACK-FASTEN-001 Mounting screws (qty 100), M6. (Cisco)
  • RACK-FASTEN-002 Cage nuts (qty 50), M6. (Cisco)

Nice-to-haves as stars align and we can use shorter runs, substitute for some of what’s above:

• 4-8 1m twinax
  • 4-8 0.5m lc-lc om3
  • 2 1' c19-c20 per chassis
  • 2 2' c19-c20 per chassis

IP Addressing

• UCS requires at least 3 mgmt IPs (1 per FI, 1 VIP), and 1 mgmt IP per compute resource, same netblock, preferably consecutive.
  • Best plan; 3 mgmt IPs, 2 mgmt IPs per compute resource, same netblock, consecutive, plan for future compute count needs.

Reserved vLANs and vSANs

• the UCS and the underlying Nexus switching has a number of vLANs and vSANs reserved for internal processes and protocols
• I believe the list is a bit different when you’re in Ethernet and FC switching modes (vs end-host-mode), but this is a good start
  • listing of reserved and used-by-default vLANs:

lab-UCS-A(nxos)# show vlan internal usage
 
VLAN        DESCRIPTION
---------   -------------------------------------------------------
3968-4031   Multicast
4032        Online diagnostics vlan1
4033        Online diagnostics vlan2
4034        Online diagnostics vlan3
4035        Online diagnostics vlan4
4036-4043   Reserved
4094        Reserved
 
lab-UCS-A(nxos)# show vlan
 
VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
<snip>
4044 SAM-vlan-management              active    Eth1/1/33
4047 SAM-vlan-boot                    active
4048 fcoe-vsan-4048                   active
4049 VLAN4049                         active

• vSANs available for configuration:2-4078,4080-4093.
• I strive to match FCoE vSAN/vLAN assignments, so I often use 4080 and 4081 for sideA and side B vSANs and vLANs.

-- Dave Cole