got above error msg when i run gpustat. but nvidia-smi works on my machine here are some details OS:Ubuntu 14.04.5 LTS Python Version: anaconda3.6

Error on calling nvidia-smi. Use --debug flag for details
Traceback (most recent call last):
  File "/usr/local/bin/gpustat", line 417, in print_gpustat                                                      gpu_stats = GPUStatCollection.new_query()
  File "/usr/local/bin/gpustat", line 245, in new_query
    return GPUStatCollection(gpu_list)
  File "/usr/local/bin/gpustat", line 218, in __init__
    self.update_process_information()
  File "/usr/local/bin/gpustat", line 316, in update_process_information
    processes = self.running_processes()
  File "/usr/local/bin/gpustat", line 275, in running_processes
    ','.join(map(str, pid_map.keys()))
  File "/usr/local/bin/gpustat", line 46, in execute_process
    stdout = check_output(command_shell, shell=True).strip()
  File "/home/xiyun/apps/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/xiyun/apps/anaconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ps -o pid,user:16,comm -p1 -p 14471' returned non-zero exit status 1.

how can i fix this ?

Hi,

On Ubuntu 20.04 with Python 3.8.3, I failed to run gpustat --debug, as shown below:

$ gpustat --debug
Error on querying NVIDIA devices. Use --debug flag for details
Traceback (most recent call last):
  File "/home/werner/.pyenv/versions/3.8.3/envs/socks5-haproxy/lib/python3.8/site-packages/pynvml.py", line 644, in _LoadNvmlLibrary
    nvmlLib = CDLL("libnvidia-ml.so.1")
  File "/home/werner/.pyenv/versions/3.8.3/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libnvidia-ml.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/werner/.pyenv/versions/3.8.3/envs/socks5-haproxy/lib/python3.8/site-packages/gpustat/__main__.py", line 19, in print_gpustat
    gpu_stats = GPUStatCollection.new_query()
  File "/home/werner/.pyenv/versions/3.8.3/envs/socks5-haproxy/lib/python3.8/site-packages/gpustat/core.py", line 281, in new_query
    N.nvmlInit()
  File "/home/werner/.pyenv/versions/3.8.3/envs/socks5-haproxy/lib/python3.8/site-packages/pynvml.py", line 608, in nvmlInit
    _LoadNvmlLibrary()
  File "/home/werner/.pyenv/versions/3.8.3/envs/socks5-haproxy/lib/python3.8/site-packages/pynvml.py", line 646, in _LoadNvmlLibrary
    _nvmlCheckReturn(NVML_ERROR_LIBRARY_NOT_FOUND)
  File "/home/werner/.pyenv/versions/3.8.3/envs/socks5-haproxy/lib/python3.8/site-packages/pynvml.py", line 310, in _nvmlCheckReturn
    raise NVMLError(ret)
pynvml.NVMLError_LibraryNotFound: NVML Shared Library Not Found

Fixes #50

I added -f, --show-full-cmd that show the full process info as discussed in #50.

Right now it shows the percent of CPU usage and the percent of system memory in use, but that can be changed.

Let me know what you think.

Example:

server1  Wed Jun 26 15:24:33 2019  418.67
[0] Tesla V100-SXM2-16GB | 34'C,  23 % |  1097 / 16130 MB | user1(1087M)
 └─ 72041 ( 80%,  0.24%): python /mnt/home/user1/git/horovod/examples/keras_mnist.py
[1] Tesla V100-SXM2-16GB | 36'C,  23 % |  1097 / 16130 MB | user1(1087M)
 └─ 72042 ( 80%,  0.24%): python /mnt/home/user1/git/horovod/examples/keras_mnist.py
[2] Tesla V100-SXM2-16GB | 35'C, 100 % |  2130 / 16130 MB | user2(777M) user1(1343M)
 ├─ 95638 (100%,  0.16%): /mnt/home/user2/anaconda3/envs/env/bin/python test_c10d.py
 └─ 72043 (100%,  0.24%): python /mnt/home/user1/git/horovod/examples/keras_mnist.py
[3] Tesla V100-SXM2-16GB | 34'C,  22 % |  1097 / 16130 MB | user1(1087M)
 └─ 72044 ( 40%,  0.24%): python /mnt/home/user1/git/horovod/examples/keras_mnist.py

When I use command watch --color -n1.0 gpustat --color I get a lot of extra ^: https://imgur.com/a/A9Fxc

This problem doesn't occur without watch. I'm on Ubuntu 17.10 with wayland.

I re-install gpustat by pip,it stall has error when I used gpustat:

root$ gpustat --debug
Error on querying NVIDIA devices. Use --debug flag for details
Traceback (most recent call last):
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/gpustat/__main__.py", line 19, in print_gpustat
    gpu_stats = GPUStatCollection.new_query()
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/gpustat/core.py", line 396, in new_query
    gpu_info = get_gpu_info(handle)
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/gpustat/core.py", line 365, in get_gpu_info
    process = get_process_info(nv_process)
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/gpustat/core.py", line 294, in get_process_info
    ps_process = psutil.Process(pid=nv_process.pid)
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/psutil/__init__.py", line 339, in __init__
    self._init(pid)
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/psutil/__init__.py", line 366, in _init
    self.create_time()
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/psutil/__init__.py", line 697, in create_time
    self._create_time = self._proc.create_time()
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/psutil/_pslinux.py", line 1459, in wrapper
    return fun(self, *args, **kwargs)
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/psutil/_pslinux.py", line 1641, in create_time
    values = self._parse_stat_file()
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/psutil/_common.py", line 340, in wrapper
    return fun(self)
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/psutil/_pslinux.py", line 1498, in _parse_stat_file
    with open_binary("%s/%s/stat" % (self._procfs_path, self.pid)) as f:
  File "/home/zhudd/anaconda3/lib/python3.7/site-packages/psutil/_pslinux.py", line 205, in open_binary
    return open(fname, "rb", **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/proc/30094/stat'

so , what's wrong with my gpu,please help me

Since 2021, NVIDIA provides an official python binding pynvml: https://pypi.org/project/nvidia-ml-py/#history which should replace a third-party community fork nvidia-ml-py3 that we have been using.

The main motivations are (1) to use an official library and (2) to add MIG support. See #102 for more details.

Need to test whether:

• The new pynvml API works well on old & recent NVIDIA Drivers; maybe some monkey patching needed (see https://github.com/wookayin/gpustat/issues/102#issuecomment-892833816)
• The new pynvml API works well on Windows (see #90)

/cc @XuehaiPan @Stonesjtu

Important Changes

• The official python bindings nvidia-ml-py needs to be installed, not nvidia-ml-py3. When the legacy one is installed for some reason, an error will occur:

  ImportError: pynvml is missing or an outdated version is installed. 
  

• To fix this error, please uninstall nvidia-ml-py3 and install nvidia-ml-py<=11.495.46 (please follow the instruction in the error message). Or you can [bypass] the validation if you really want.

• Due to compatibility reasons, NVIDIA Driver version needs to be 450.66 or higher.

When I call nvmlDeviceGetGraphicsRunningProcesses, raise below exception.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/gitProject/venv/siren/lib64/python3.6/site-packages/pynvml/nvml.py in _nvmlGetFunctionPointer(name)
    759         try:
--> 760             _nvmlGetFunctionPointer_cache[name] = getattr(nvmlLib, name)
    761             return _nvmlGetFunctionPointer_cache[name]

/usr/lib64/python3.6/ctypes/__init__.py in __getattr__(self, name)
    355             raise AttributeError(name)
--> 356         func = self.__getitem__(name)
    357         setattr(self, name, func)

/usr/lib64/python3.6/ctypes/__init__.py in __getitem__(self, name_or_ordinal)
    360     def __getitem__(self, name_or_ordinal):
--> 361         func = self._FuncPtr((name_or_ordinal, self))
    362         if not isinstance(name_or_ordinal, int):

AttributeError: /lib64/libnvidia-ml.so.1: undefined symbol: nvmlDeviceGetGraphicsRunningProcesses_v2

During handling of the above exception, another exception occurred:

NVMLError_FunctionNotFound                Traceback (most recent call last)
<ipython-input-5-6d9d0902fdc2> in <module>
----> 1 nvmlDeviceGetGraphicsRunningProcesses(handle)

~/gitProject/venv/hstk/lib64/python3.6/site-packages/pynvml/nvml.py in nvmlDeviceGetGraphicsRunningProcesses(handle)
   2179
   2180 def nvmlDeviceGetGraphicsRunningProcesses(handle):
-> 2181     return nvmlDeviceGetGraphicsRunningProcesses_v2(handle)
   2182
   2183 def nvmlDeviceGetAutoBoostedClocksEnabled(handle):

~/gitProject/venv/hstk/lib64/python3.6/site-packages/pynvml/nvml.py in nvmlDeviceGetGraphicsRunningProcesses_v2(handle)
   2147     # first call to get the size
   2148     c_count = c_uint(0)
AttributeError                            Traceback (most recent call last)
~/gitProject/venv/hstk/lib64/python3.6/site-packages/pynvml/nvml.py in _nvmlGetFunctionPointer(name)
    759         try:
--> 760             _nvmlGetFunctionPointer_cache[name] = getattr(nvmlLib, name)
    761             return _nvmlGetFunctionPointer_cache[name]

/usr/lib64/python3.6/ctypes/__init__.py in __getattr__(self, name)
    355             raise AttributeError(name)
--> 356         func = self.__getitem__(name)
    357         setattr(self, name, func)

/usr/lib64/python3.6/ctypes/__init__.py in __getitem__(self, name_or_ordinal)
    360     def __getitem__(self, name_or_ordinal):
--> 361         func = self._FuncPtr((name_or_ordinal, self))
    362         if not isinstance(name_or_ordinal, int):

AttributeError: /lib64/libnvidia-ml.so.1: undefined symbol: nvmlDeviceGetGraphicsRunningProcesses_v2

During handling of the above exception, another exception occurred:

NVMLError_FunctionNotFound                Traceback (most recent call last)
<ipython-input-6-85e61951ad1d> in <module>
----> 1 nvmlDeviceGetGraphicsRunningProcesses_v2(handle)

~/gitProject/venv/hstk/lib64/python3.6/site-packages/pynvml/nvml.py in nvmlDeviceGetGraphicsRunningProcesses_v2(handle)
   2147     # first call to get the size
   2148     c_count = c_uint(0)
-> 2149     fn = _nvmlGetFunctionPointer("nvmlDeviceGetGraphicsRunningProcesses_v2")
   2150     ret = fn(handle, byref(c_count), None)
   2151

~/gitProject/venv/hstk/lib64/python3.6/site-packages/pynvml/nvml.py in _nvmlGetFunctionPointer(name)
    761             return _nvmlGetFunctionPointer_cache[name]
    762         except AttributeError:
--> 763             raise NVMLError(NVML_ERROR_FUNCTION_NOT_FOUND)
    764     finally:
    765         # lock is always freed

NVMLError_FunctionNotFound: Function Not Found

So, I guess may be is the pynvml change something lead to this problem #72

C:>gpustat -cp 'nvidia-smi' is not recognized as an internal or external command, operable program or batch file. Error on calling nvidia-smi

C:>nvidia-smi --query-gpu=index,uuid,name,temperature.gpu,utilization.gpu,memory.used,memory.total --format=csv,noheader,nounits 0, GPU-9d01c9ef-1d73-7774-8b4f-5bee4b3bf644, GeForce GTX 1080 Ti, 28, 65, 9219, 11264 1, GPU-9da3de3f-cdf2-8ca9-504d-fd9bc414a78e, GeForce GTX 1080 Ti, 22, 0, 140, 11264

Any idea what might be the issue? Windows 10, Python 3.7.2, latest nvidia drivers, etc, as of the time of this post.

I tried it using

gpustat -cpFP --watch

I use Debian 11.

Here my result :

When I run something in gpu it sow me the amount of memory used only:

Any hints? I miss something?

Thanks.

Hi!

I'm observing gpustat as a soft dependency for ray-project, and because currently released version (0.6.0) is not installable on Windows I'm facing hard times enabling this dependency in conda-forge.

Knowing the compatibility issue is fixed in master for quite some time, it would be really good if you could make a new release.

References:

• PR to conda-forge feedstock of gpustat: https://github.com/conda-forge/gpustat-feedstock/pull/2
• PR to add ray-project to conda-forge (where gpustat is needed): https://github.com/conda-forge/staged-recipes/pull/11160

This PR only adds encoder and decoder utilization to --json from the cmdline, or to the GPUStat object if gpustat is used as a library.

The information is also exposed to the standard command line output via the -e or --show-codec flag

See issue #79

Describe the bug

Something caused a version mismatch somewhere and I can no longer use gpustat. Nothing at all is printed on stdout or stderr. Running with --debug prints nothing as well. I launched it as python -m pdb -m gpustat and stepped through until noticing an error raised in:

/opt/conda/lib/python3.8/site-packages/pynvml/nvml.py(718)

of type pynvml.nvml.NVMLError_LibRmVersionMismatch.

Screenshots or Program Output

Please provide the output of gpustat --debug and nvidia-smi. Or attach screenshots if applicable.

Environment information:

• OS: Ubuntu 20.04
• NVIDIA Driver version: 510.73.08
• The name(s) of GPU card: Tesla V100-SXM2
• gpustat version: 1.0.0
• pynvml version: 11.495.46

Additional context

Add any other context about the problem here.

Commented in https://github.com/wookayin/gpustat/issues/142#issuecomment-1336066463, add new noexcept funcitons gpu_count and is_available.

Closes #142

Describe the bug

python -m gpustat errors if user is not admin or if nvidia drivers are installed but no NVidia card is available

Screenshots or Program Output

Please provide the output of gpustat --debug and nvidia-smi. Or attach screenshots if applicable.

As a regular user:

>python -m gpustat --debug
Error on querying NVIDIA devices. Use --debug flag for details
Traceback (most recent call last):
  File "d:\temp\ray_venv\lib\site-packages\gpustat\cli.py", line 20, in print_gpustat
    gpu_stats = GPUStatCollection.new_query(debug=debug)
  File "d:\temp\ray_venv\lib\site-packages\gpustat\core.py", line 362, in new_query
    N.nvmlInit()
  File "d:\temp\ray_venv\lib\site-packages\pynvml.py", line 1450, in nvmlInit
    nvmlInitWithFlags(0)
  File "d:\temp\ray_venv\lib\site-packages\pynvml.py", line 1440, in nvmlInitWithFlags
    _nvmlCheckReturn(ret)
  File "d:\temp\ray_venv\lib\site-packages\pynvml.py", line 765, in _nvmlCheckReturn
    raise NVMLError(ret)
pynvml.NVMLError_NoPermission: Insufficient Permissions

As an admin:

Error on querying NVIDIA devices. Use --debug flag for details
Traceback (most recent call last):
  File "d:\temp\ray_venv\lib\site-packages\gpustat\cli.py", line 18, in print_gpustat
    gpu_stats = GPUStatCollection.new_query(debug=debug)
  File "d:\temp\ray_venv\lib\site-packages\gpustat\core.py", line 370, in new_query
    N.nvmlInit()
  File "d:\temp\ray_venv\lib\site-packages\pynvml.py", line 1450, in nvmlInit
    nvmlInitWithFlags(0)
  File "d:\temp\ray_venv\lib\site-packages\pynvml.py", line 1440, in nvmlInitWithFlags
    _nvmlCheckReturn(ret)
  File "d:\temp\ray_venv\lib\site-packages\pynvml.py", line 765, in _nvmlCheckReturn
    raise NVMLError(ret)
pynvml.NVMLError_DriverNotLoaded: Driver Not Loaded

As a regular user

>nvidia-smi
NVIDIA-SMI has failed because you are not:
        a) running as an administrator or
        b) there is not at least one TCC device in the system

As an admin

>nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. This can also be happening if non-NVIDIA GPU is running as primary display, and NVIDIA GPU is in WDDM mode.

Environment information:

• OS: windows10
• NVIDIA Driver version: 11.7 (Edit: changed from 11.3 to 11.7)
• The name(s) of GPU card: None
• gpustat version: 1.1.0
• pynvml version: nvidia-ml-py 11.495.46

Additional context

Add any other context about the problem here.

Hi,

Thanks for the program, very useful. I was thinking a nice additional option could be to show the 'fullness' of the GPU ram with a bar instead of just with the number. This for the use case of quickly identifying (almost) empty GPUs when sharing a bunch of GPUs with labmates (eg see image: some of them are full, some have a lot of space left, a bar could show this more 'directly' I feel). See second image for roughly what I mean:

nvidia-smi lists Max Clocks and current Clocks. It would be nice to be able to see these, and maybe display these in a % so you know how much you're throttled.

example:

    Clocks
        Graphics                          : 2025 MHz
        SM                                : 2025 MHz
        Memory                            : 10251 MHz
        Video                             : 1785 MHz
    Max Clocks
        Graphics                          : 2100 MHz
        SM                                : 2100 MHz
        Memory                            : 10501 MHz
        Video                             : 1950 MHz