identifying which pid is occupying the gpu can be found following this thread.
However, the gpu number in the above link is not identical with the gpu number displayed with
In my case, the processes that were occupying the gpu memory did not show up in
nvidia-smi but it showed up with
fuser. Since the numbering of gpu of the two does not match, I had to manually cross-check to find out which pid I needed to kill.