Yesterday, 01:31 PM
What stands out to me is that the GPUs are running close to 100% utilization, and once the error starts you get several CUDA errors such as cuEventSynchronize(), cuMemcpyHtoDAsync() and cuMemFree(), followed by NVML errors. The fact that the shorter masks complete successfully also makes me think the workload is triggering the problem only after the GPUs have been under load for some time.
I would first test the cards individually, especially the 1080 Ti cards, and see if the error follows a particular GPU. Also worth trying a slightly lower workload (-w 1) and checking whether the same error occurs.
Since you're mixing Pascal and Turing GPUs and using a relatively new driver/CUDA combination, I'd also test with a different known-good NVIDIA driver rather than changing several things at once.
I would first test the cards individually, especially the 1080 Ti cards, and see if the error follows a particular GPU. Also worth trying a slightly lower workload (-w 1) and checking whether the same error occurs.
Since you're mixing Pascal and Turing GPUs and using a relatively new driver/CUDA combination, I'd also test with a different known-good NVIDIA driver rather than changing several things at once.