Ubuntu 16.04 LTS NVIDIA Setup Guide

[evilmog]

Ok here is my setup guide, I could be completely wrong on a few things so please update me, this will get turned into a wiki but I want discussion first:

1) Install Ubuntu 16.04 LTS
2) dpkg --remove-architecture i386
3) ldconfig && sync && reboot
4) apt-get install build-essential xserver-xorg p7zip xorg-dev libgtk-3-dev
5) download latest nvidia drivers
7) [blacklist nouveau]

as root

echo 'blacklist nouveau' > /etc/modprobe.d/blacklist-nouveau.conf
echo 'blacklist lbm-nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf
echo 'options nouveau modeset=0' >> /etc/modprobe.d/blacklist-nouveau.conf
echo 'alias nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf
echo 'alias lbm-nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf
echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
update-initramfs -u
reboot

8) [stop x as root] /etc/init.d/xdm stop
9) install nvidia drivers
10) sudo nvidia-xconfig -s -a --force-generate --allow-empty-initial-configuration --cool-bits=12 --registry-dwords="PerfLevelSrc=0x2222" --no-sli --connected-monitor="DFP-0"

11) ldconfig && sync && reboot
12) as root make a fan.sh script, edit to taste and make executable

#!/bin/bash
export DISPLAY=:0
xhost +
nvidia-settings -a [gpu:0]/GPUFanControlState=1 -a [fan-0]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:1]/GPUFanControlState=1 -a [fan-1]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:2]/GPUFanControlState=1 -a [fan-2]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:3]/GPUFanControlState=1 -a [fan-3]/GPUTargetFanSpeed=100
#nvidia-settings -a [gpu:4]/GPUFanControlState=1 -a [fan-4]/GPUTargetFanSpeed=100
#nvidia-settings -a [gpu:5]/GPUFanControlState=1 -a [fan-5]/GPUTargetFanSpeed=100
#nvidia-settings -a [gpu:6]/GPUFanControlState=1 -a [fan-6]/GPUTargetFanSpeed=100
#nvidia-settings -a [gpu:7]/GPUFanControlState=1 -a [fan-7]/GPUTargetFanSpeed=100

13) run fan.sh  and tweak system until it runs clean, may need to reexecute the nvidia setup

14) setup ufw to firewall allow only to trusted systems as x is a security hole

15) download hashcat or build

16) install systems management software, puppet etc

[atom]

Some comments:

• Which Ubuntu version is it, I'm using server, and people should do so, too, but it's nowhere written down
  
• The document does in no way show how to install the most important opencl packages: ocl-icd-libopencl1 and clinfo - I think that's a very important part
  
• You don't need to blacklist nouveau manually, the nvidia driver does that automatically while installing it
  
• Why do you do ldconfig all the time, it's not required
  
• apt-get install is missing some important packages i think. We should make it more complete to have less problems later when people are asking 1000's of questions. At least you refer xdm, but it's not part of your apt-get install line
  
• You're missing some very important manual steps to do, for example this:
  

  cat > /etc/X11/xdm/Xsetup << EOF
  export PATH=/bin:/usr/bin:/sbin
  export HOME=/root
  export DISPLAY=:0
  xset -dpms
  xset s off
  xhost +
  EOF

  Which is required to do fan speed control from hashcat via SSH and without logging into X11
  
• Do not use any fan script, also do not recommend it,  hashcat manages the fans automatically and does it very good. I know for you it doesn't matter since your rigs run in a datacenter, but for most people it runs in the pc under their desk and they don't want to have 100% fan speed
  
• No need to mention to install system management software, it's not part of this documention. The same goes for firewall
  
• No need to re-execute the nvidia setup
  
• The nvidia-xconfig section is good, but it lacks the sed command afterwards:
  
  

  sed -i '/Driver/a \ \ \ \ Option         "Interactive" "False"' /etc/X11/xorg.conf

  
  

[kartan]

You should make it clear that this is for people who want to build hashcat. I would like to see a clean doc just for setting up a box without the dev packages just to deploy the binary the other steps should be optional.


All the changes atom proposed should be in there as well.

[evilmog]

Some comments:
[*]You don't need to blacklist nouveau manually, the nvidia driver does that automatically while installing it

I had issues with server not accepting the blacklist even with the driver doing it automatically

[evilmog]

1) Install Ubuntu 16.04 LTS
2) dpkg --remove-architecture i386
3) ldconfig && sync && reboot
4) apt-get install build-essential xserver-xorg p7zip xorg-dev libgtk-3-dev xdm
[there will be more packages for this list]
5) download latest nvidia drivers
7) [blacklist nouveau]

as root

echo 'blacklist nouveau' > /etc/modprobe.d/blacklist-nouveau.conf
echo 'blacklist lbm-nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf
echo 'options nouveau modeset=0' >> /etc/modprobe.d/blacklist-nouveau.conf
echo 'alias nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf
echo 'alias lbm-nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf
echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
update-initramfs -u
reboot

8) [stop x as root] /etc/init.d/xdm stop
9) install nvidia drivers
10) sudo nvidia-xconfig -s -a --force-generate --allow-empty-initial-configuration --cool-bits=12 --registry-dwords="PerfLevelSrc=0x2222" --no-sli --connected-monitor="DFP-0"

11) cat > /etc/X11/xdm/Xsetup << EOF

export PATH=/bin:/usr/bin:/sbin

export HOME=/root

export DISPLAY=:0

xset -dpms

xset s off

xhost +
EOF

sed -i '/Driver/a \ \ \ \ Option         "Interactive" "False"' /etc/X11/xorg.conf

12) reboot
13) as root make a fan.sh script, edit to taste and make executable [only for those in a datacenter]

#!/bin/bash
export DISPLAY=:0
xhost +
nvidia-settings -a [gpu:0]/GPUFanControlState=1 -a [fan-0]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:1]/GPUFanControlState=1 -a [fan-1]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:2]/GPUFanControlState=1 -a [fan-2]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:3]/GPUFanControlState=1 -a [fan-3]/GPUTargetFanSpeed=100
#nvidia-settings -a [gpu:4]/GPUFanControlState=1 -a [fan-4]/GPUTargetFanSpeed=100
#nvidia-settings -a [gpu:5]/GPUFanControlState=1 -a [fan-5]/GPUTargetFanSpeed=100
#nvidia-settings -a [gpu:6]/GPUFanControlState=1 -a [fan-6]/GPUTargetFanSpeed=100
#nvidia-settings -a [gpu:7]/GPUFanControlState=1 -a [fan-7]/GPUTargetFanSpeed=100

14) run fan.sh  and tweak system until it runs clean [only if you followed 13

15) download hashcat or build

[kartan]

i've never seen that issue with the nouveau blacklist

[kartan]

and seperate the dev packages this isn't necessary for 90% of the people who just want the binary

[evilmog]

happened on a net new virgin build this weekend

[devilsadvocate]

evilmog, please update your guide with these specific commands so that the latest Nvidia drivers can be downloaded and installed.

Code:

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update
sudo apt-get install nvidia-384 nvidia-libopencl1-384 nvidia-opencl-icd-384

The different versions of Nvidia drivers that are available can be viewed by executing:

Code:

sudo apt-cache search nvidia-

[philsmd]

We, in general, do not recommend using the package maintainers versions of the proprietary driver, since our experience showed they are either totally out-of-date or the packages are totally messed up.

I think this is why evilmog also just wrote

5) download latest nvidia drivers

because this is normally the best way to install the recommended/supported version of the drivers (see https://hashcat.net/hashcat/).

Of course there could be some exceptions (yeah, maybe even "packages" that work better than the downloaded NVidia drivers) e.g. for unsupported distros and/or older versions of a supported distro, but our experience shows that the download from nvidia.com / support.amd.com just works in most of the cases (and in general even better than the "packages").
I know that some people fear to install .run files directly downloaded from the internet (also for reasons that the "files" do not register correctly with the package manager), but fortunately there are also tools provided by the installer that help you to uninstall the current driver (for NVidia and AMD, like nvidia-uninstall or /usr/share/ati/amd-uninstall.sh respectively).

Some distros are also officially supported by Nvidia/AMD, like ubuntu, redhat etc... so normally our recommendation is to pick a supported distro and install the proprietary driver from nvidia.com/support.amd.com.

On the other hand, the "packages" often are just "wrappers" around the .run file. Therefore, they are neither better nor worse compared to the direct download (but very often, they are several versions behind the official drivers).