I don't think that using a vmware image is an easy solution for your problem.
With virtualization you need the virtualization system to support having full access to your hardware (in this case the dedicated graphics cards). In this case it's called something like PCI passthrough.
Even though vmware officially supports PCI passthrough such that your guest operating system can use the GPUs, the list of officially supported graphics cards is very small (e.g. for Nvidia only some Tesla and Grid GPUs are officially supported).
Maybe it works, maybe not. But why would you even think about this additional complication.
My suggestion is that you should fix the problem on it's root and not introduce additional problems: maybe the only way is to sell your current custom design GPU and get a founders edition GPU.
Actually, I'm not totally sure if it is the only possible solution, since I know that there are (unfortunately) many more users that use a "gaming" GPU (not-founders-edition) and besides the problems with heat etc they run hashcat without problem.
Unfortuntely, we do not have a list of cards that work or do not work (because we strongly suggest to buy founders edition/reference design GPUs where the vendor doesn't really matter because they are basically all the same and just work out of the box).
From the past we know that some vendors mess around with the firmware that is used on the GPUs, they basically just ship new hardware with adapted older firmwares, which most of the times work (because tested more carefully) with games, but somehow they do not work very well with OpenCL computation tasks. It could be the case for this specific model you have. I do not know for sure.
You can only figure this out if you get a different card (and I highly recommend that you get a founders edition and not a OEM card anymore).
With virtualization you need the virtualization system to support having full access to your hardware (in this case the dedicated graphics cards). In this case it's called something like PCI passthrough.
Even though vmware officially supports PCI passthrough such that your guest operating system can use the GPUs, the list of officially supported graphics cards is very small (e.g. for Nvidia only some Tesla and Grid GPUs are officially supported).
Maybe it works, maybe not. But why would you even think about this additional complication.
My suggestion is that you should fix the problem on it's root and not introduce additional problems: maybe the only way is to sell your current custom design GPU and get a founders edition GPU.
Actually, I'm not totally sure if it is the only possible solution, since I know that there are (unfortunately) many more users that use a "gaming" GPU (not-founders-edition) and besides the problems with heat etc they run hashcat without problem.
Unfortuntely, we do not have a list of cards that work or do not work (because we strongly suggest to buy founders edition/reference design GPUs where the vendor doesn't really matter because they are basically all the same and just work out of the box).
From the past we know that some vendors mess around with the firmware that is used on the GPUs, they basically just ship new hardware with adapted older firmwares, which most of the times work (because tested more carefully) with games, but somehow they do not work very well with OpenCL computation tasks. It could be the case for this specific model you have. I do not know for sure.
You can only figure this out if you get a different card (and I highly recommend that you get a founders edition and not a OEM card anymore).