The adventure it has been to make NVME SSDs work on a DELL R630 server.

This is the story about a R630 server having NVME, SAS and SATA disks working together. The server is owner by my company but for this blog post let's say it is my server. I did buy the server (Dell PowerEdge R630 1U 10x 2.5" (SFF)) in January of 2022.

DELL R630 missing a CPU

The failed deployment in a datacenter

I was confident I could swap this new machine with the other one, they where quite identical. But this turned out not to work, when we booted it up in the datacenter (I received it some days ago and could not do testing until then) we had a hard time understanding why it was complaining about the missing SAS cables for the 2 backplane ports.

The machine was configured to have a NVME U2 extender and I had did the mistake to say in the configurator that I wanted not RAID card. Of course because I did not want RAID on this system. But anyway, I had to order a PERC (RAID card) that had the cables included with it and that did solve the error I had that prevented the boot the other day at the datacenter.

The PERC card

PERC card

The NVME expander

NVME expander

Mounting the cables and the PERC/RAID kit

PERC card plugged in

I put in place the kit and plugged it the cables each one according to the labels. (yes it knows and complains if you switch them)

PERC card plugged in BackPlane cables

The cables
PERC card

PERC card cables

NVME expander into the BackPlane

NVME expander card cables

Trying to make NVME disks work

I did buy some NVME drives so I could test my setup. I was innocently thinking that it would work right after I slide then in the caddies but obviously there was more funny things to happen. So I received the disks and did setup up one and then two. None of the disks where showing up, and the NVME card was not showing.

I was not understanding what was wrong because the NVME card had green leds and the disks too. I then took the bet it was because they where NVME gen 4 and the system could only handle gen 3 but that was not it. I also did order a SAS disk to try it.

I was now having a system with: 3 NVMEs not working, one SAS and one SATA disk showing up. After some debug and trying dumb things I tried to put the NVME card on the reizer of CPU 1. And boom ! it did show up in Linux. This was it, I had found something (written in very small on the other reizer was "CPU 2"). But on this reizer the cables where too short. So I was still having 3 non showing NVMEs.

A second CPU

I decided to contact the seller to buy a second CPU and RAM. Got them delivered very quickly (thank you "UPS express" !). Now I did put the CPU 2 and still nothing shows up, not even the CPU 2. After some asking Internet on Reddit people started saying it was probably the motherboard.

New motherboard

After contacting the seller I got it repaired by them and when I got back this server it did boot up and show 2 CPUs, all RAM, SAS, NVME and SATA disks. This was the end of 6 months of searching how to make a R630 work with NVME disks.

Conclusion

A second CPU+RAM and a reizer PERC card + NVME enhancement card where needed for this setup. After that 4 NVME slots where available, SAS and SATA disks work too. All thoes types of disks can be together.

DELL R630 with two CPUs a NVME expander and a PERC card

lspci

This are the new lines in lspci

03:00.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
04:05.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
04:06.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
04:07.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
ff:0e.2 Unassigned class [ffff]: Intel Corporation Device 6ff2 (rev 01)
ff:0e.3 Unassigned class [ffff]: Intel Corporation Device 6ff3 (rev 01)

Full lspci before

00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.7 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
01:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
01:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
09:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe Switch [PS]
0a:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe Switch [PS]
0b:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe-PCI Bridge [PPB]
0c:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2 (rev 01)

Full lspci after

00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.7 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
01:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
01:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57800 1/10 Gigabit Ethernet (rev 10)
02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
04:05.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
04:06.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
04:07.0 PCI bridge: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch (rev ab)
0e:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe Switch [PS]
0f:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe Switch [PS]
10:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe-PCI Bridge [PPB]
11:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2 (rev 01)
ff:0e.2 Unassigned class [ffff]: Intel Corporation Device 6ff2 (rev 01)
ff:0e.3 Unassigned class [ffff]: Intel Corporation Device 6ff3 (rev 01)