Issue
I'm running Ubuntu 18.04 and have recently (about a month) installed CUDA 10.2 with runfile installation after A LOT of trouble with the slightly recommended .deb installation. Everything was fine: nvidia-smi showed GPU stats and I was able to run my parallel code on the GPU. Today I started my machine and the software center suggested some updates...it seemed just ordinary stuff a part from this libnvidia-compute-440 package, but i didn't pay too much attention and installed the all the updates. After that, my CUDA codes didn't work, I tried nvidia-smi and got a
Failed to initialize NVML: Driver/library version mismatch
I rebooted the systems, still got the errors. I spent the afternoon googling possible solutions, and I think I was able to find the core of problem: running dmesg |tail -4 gives
NVRM: API mismatch: the client has the version 440.59, but
NVRM: this kernel module has the version 440.33.01. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
The version of the libnvidia-compute-440 package that I have is indeed 440.59 now, but my drivers version is 440.33.01 as dmesg |grep nvidia (or similar commands that I've tried) shows (see on third line)
[ 16.462737] nvidia-nvlink: Nvlink Core is being initialized, major device number 237
[ 16.463235] nvidia 0000:03:00.0: enabling device (0006 -> 0007)
[ 16.785628] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 440.33.01 Tue Nov 12 23:43:11 UTC 2019
[ 16.916202] [drm] [nvidia-drm] [GPU ID 0x00000300] Loading driver
[ 16.916205] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:03:00.0 on minor 1
In fact, by looking at /var/log/apt/history.log I could see that libnvidia-compute-440 was updated from 440.33.01-0ubuntu1 to 440.59-0ubuntu0.18.04.133.01. All these evidence led me to the conclusion that I could try to go back to the previous version of that library, however apt-get install libnvidia-compute-440=440.33.01 (which I think it's the right syntax) gave me E: Version '440.33.01' for 'libnvidia-compute-440' was not found.
I really hope that a solution which doesn't imply uninstalling CUDA does exist, since installing it took me a weekend and as I said it was quite a PITA for me.
Solution
UPDATE: RESOLVED
I was being very cautious, fearing that I would mess up my CUDA installation. However I took courage and updated my nvidia driver with
sudo apt install nvidia-driver-440
It successfully updated the drivers, which now match the libnvidia-compute version 440.59.
After rebooting everything works fine, just as before.
Answered By - user199710