Solution to cuda driver failure after kernel update
after a recent kernel update (presumably automatic via unattended-upgrades), cuda functionality on my server ceased functioning. the primary symptom was the following error when running nvidia-smi:
NVIDIA-SMI has failed because it couldn‘t communicate with the NVIDIA driver ...this indicates a mismatch between the installed nvidia driver and the current kernel version. the solution involves using dkms (Dynamic Kernel Module Support) to rebuild the nvidia driver against the new kernel.