OS Recommendation / Experience?
I've got a Gigabyte motherboard with 7 R9-280's in it (using USB risers)
My thought was to install ESXi so that I could move GPU's between VMs as needed. ESXi sees them all, but can only assign 4 cards (8 PCI devices) to a VM, so isn't going to be a good platform for me.
I then tried using Ubuntu, but that distro only seems to see the GPU's on one PCI bus (4 cards).
Has anyone had a similar experience? Do other distro's seem to deal better with multiple PCI busses?
If both Ubuntu and ESXi are only seeing four cards, then it sounds like you have another issue going on. Maybe you can provide a bit more details about your setup.
Gigabyte z97x G1 Wifi-bk
F5 Firmware for the motherboard (one version back from newest, plan to update)
7x Sapphire r9 280x's
That's about it ...

ESXi sees all the cards, but it has a limit on how many PCIe resources can be assigned to a VM using VT-d. 8 resources = 4 cards in this case, each card looks like 2 devcies to PCIe.

Just was wondering if I had tripped across some dumb ubuntu + pcie or ubuntu + gigabyte thing
Maybe you're missing some chipset driver on ubuntu?