The original post: /r/ubuntu by /u/guyinsfc on 2024-11-15 18:37:56.

I recently started managing a fleet of GPU servers. Several of them are Gigabyte G493-ZB3-AAP1s with 2x EPYC 9124s and 8x nvidia GPUs. I’m having some issues with PCI enumeration and nvme devices on these machines that I’m hoping maybe somebody has some ideas about. We’re currently stuck on ubuntu 22.04 due to software requirements.

I went to reimage one of these servers recently, and learned that it had more nvme drives connected to it than I had seen before. It turns out there are a total of 7 nvme drives in this machine, but I only knew of six.

I learned of the seventh because I based my imaging environment on SystemRescue, which is derived from Arch. When I boot from SystemRescue, it sees and enumerates all 7 nvme devices without issue.

Booting from the ubuntu-server 22.04.5 live ISO, sometimes 5 are detected, other times 6 are detected, but never all 7. I have a similar issue with another server of the same make and model, where one nvme device randomly appears and disappears. I don’t seem to have (obvious) issues on servers with 4 or fewer nvme devices.

My imaging scripts attempt to use all available storage with LVM, so when I try to reboot back to ubuntu after deploying the image from the SystemRescue environment, the system fails to boot due to “missing” nvme devices.

The root cause appears to be related to PCI enumeration:

  • PCI resource allocation shows many failures in Ubuntu that don’t appear in SystemRescue.
    • failing to assign memory windows
    • failing to assign BAR spaces
    • many prefetchable memory regions mapped in the high address ranges (0x9000000000 and above) and quite a few “[disabled]” regions
    • regions apparently trying to allocate in overlapping spaces
    • possibly trying to allocate across multiple PCI segments simultaneously
  • SystemRescue successfully enumerates all devices with minimal parameters and no errors.
  • Device ordering/enumeration differs between SystemRescue and Ubuntu.
  • The issue appears during PCI resource allocation, not NVMe driver initialization

I’m currently using the HWE kernel (6.8.0-48-generic) but had the same issues with earlier kernel versions. SystemRescue uses Arch’s linux-lts-6.6.47-1 kernel.

I’ve tried a number of different kernel parameters but so far I haven’t found any that seem to help:

  • iomem=relaxed
  • pci=realloc,assign-buses
  • pci=nommconf
  • memmap=64G$256G
  • pci_mmcfg=relaxed
  • pci_reserve_host_bridge_window=4G
  • pci_hotplug.acpi_force=1
  • rootdelay=N (with various values of N)
  • amd_iommu=interrupted
  • systemd.debug-shell=1 systemd.unit=multi-user.target systemd.log_level=debug fastboot=0 nosplash systemd.show_status=true systemd.log_target=kmsg

Notably, SystemRescue doesn’t use any of these special kernel parameters, and it seems to do just fine.

I’m not an expert on complex NUMA architectures, PCI initialization, etc, but I’m beginning to think that the issue is deep in the bowels of ubuntu’s init process. If anybody has any ideas please let me know!