[Question] `process.cpu_percent()` got over 100% value about nvitop HOT 4 CLOSED

The full script I am using is:

from nvitop import Device
from nvitop import Device, GpuProcess, NA, colored

devices = Device.all()  # or `Device.cuda.all()` to use CUDA ordinal instead
for device in devices:
    processes = device.processes()  # type: Dict[int, GpuProcess]
    sorted_pids = sorted(processes.keys())

    print(device)
    print(f'  - Fan speed:       {device.fan_speed()}%')
    print(f'  - Temperature:     {device.temperature()}C')
    print(f'  - GPU utilization: {device.gpu_utilization()}%')
    print(f'  - Total memory:    {device.memory_total_human()}')
    print(f'  - Used memory:     {device.memory_used_human()}')
    print(f'  - Free memory:     {device.memory_free_human()}')
    print(f'  - Processes ({len(processes)}): {sorted_pids}')
    if len(processes) > 0:
        processes = GpuProcess.take_snapshots(processes.values(), failsafe=True)
        processes.sort(key=lambda process: (process.username, process.pid))

        print(colored(f'  - Processes ({len(processes)}):', color='blue', attrs=('bold',)))
        fmt = '    {pid:<5}  {username:<8} {cpu:>5}  {host_memory:>8} {time:>8}  {gpu_memory:>8}  {sm:>3}  {command:<}'.format
        print(colored(fmt(pid='PID', username='USERNAME',
                          cpu='CPU%', host_memory='HOST-MEM', time='TIME',
                          gpu_memory='GPU-MEM', sm='SM%',
                          command='COMMAND'),
                      attrs=('bold',)))
        for snapshot in processes:
            print(fmt(pid=snapshot.pid,
                      username=snapshot.username[:7] + ('+' if len(snapshot.username) > 8 else snapshot.username[7:8]),
                      cpu=snapshot.cpu_percent, host_memory=snapshot.host_memory_human,
                      time=snapshot.running_time_human,
                      gpu_memory=(snapshot.gpu_memory_human if snapshot.gpu_memory_human is not NA else 'WDDM:N/A'),
                      sm=snapshot.gpu_sm_utilization,
                      command=snapshot.command))
    print('-' * 120)

from nvitop.

Hi @qpc001

From htop manual page (ref: Why is the CPU usage reported by top in Linux over 100%?):

PERCENT_CPU (CPU%)
     The  percentage of the CPU time that the process is currently using.
     This is the default way to represent CPU usage in Linux. Each process can
     consume up to 100% which means the full capacity of the core it is running
     on. This is sometimes called "Irix mode" e.g. in top(1).

The CPU percent is calculated by:

cpu_percent = (user_time + system_time) / elapsed_time

when you are running a multi-threaded program on a multi-core machine. It can be over 100%. E.g. 3000% on a 32-core machine.

Feel free to ask more questions.

from nvitop.

Hi @qpc001

From htop manual page (ref: Why is the CPU usage reported by top in Linux over 100%?):

PERCENT_CPU (CPU%)
     The  percentage of the CPU time that the process is currently using.
     This is the default way to represent CPU usage in Linux. Each process can
     consume up to 100% which means the full capacity of the core it is running
     on. This is sometimes called "Irix mode" e.g. in top(1).

The CPU percent is calculated by:

cpu_percent = (user_time + system_time) / elapsed_time

when you are running a multi-threaded program on a multi-core machine. It can be over 100%. E.g. 3000% on a 32-core machine.

Feel free to ask more questions.

My point is that it is not possible for the CPU usage of this program to reach over 200%, rather than discussing why the program's CPU usage can exceed 100%.

from nvitop.

My point is that it is not possible for the CPU usage of this program to reach over 200%, rather than discussing why the program's CPU usage can exceed 100%.

@qpc001 I'm sorry that I misunderstood your question. But the answer would be similar.

1. It can get CPU percent over 100%. As I mentioned:

   cpu_percent = (user_time + system_time) / elapsed_time

   It's really depending on your sample interval for elapsed_time. It can get a large value if your elapsed_time is relatively small. If you want to get a more accurate result. You should average over a time interval.

   process.cpu_percent(interval=1.0)  # blocking mode

   This would block your program for 1 second and gather the CPU time information during the sample interval.

   nvitop use process.cpu_percent() in non-blocking mode. That is, it will not block your program but use the CPU time information between two calls. You may get an inaccurate result in the first call because you don't have a previous call to compare with.

   process.cpu_percent()  # non-blocking: the first call, may be inaccurate
   # do something
   process.cpu_percent()  # non-blocking: the second call, returns the average CPU percent between the first and the second calls

2. Here is the code for how psutil.Process calculate the CPU percent:

   https://github.com/giampaolo/psutil/blob/90b35e3ae85651d67a095205adc94aaa0a4ec98f/psutil/__init__.py#L946-L1036

   def cpu_percent(self, interval=None):
       """Return a float representing the current process CPU
       utilization as a percentage.
   
       When *interval* is 0.0 or None (default) compares process times
       to system CPU times elapsed since last call, returning
       immediately (non-blocking). That means that the first time
       this is called it will return a meaningful 0.0 value.
   
       When *interval* is > 0.0 compares process times to system CPU
       times elapsed before and after the interval (blocking).
   
       In this case is recommended for accuracy that this function
       be called with at least 0.1 seconds between calls.
   
       A value > 100.0 can be returned in case of processes running
       multiple threads on different CPU cores.
   
       The returned value is explicitly NOT split evenly between
       all available logical CPUs. This means that a busy loop process
       running on a system with 2 logical CPUs will be reported as
       having 100% CPU utilization instead of 50%.
   
       Examples:
   
         >>> import psutil
         >>> p = psutil.Process(os.getpid())
         >>> # blocking
         >>> p.cpu_percent(interval=1)
         2.0
         >>> # non-blocking (percentage since last call)
         >>> p.cpu_percent(interval=None)
         2.9
         >>>
       """
       blocking = interval is not None and interval > 0.0
       if interval is not None and interval < 0:
           raise ValueError("interval is not positive (got %r)" % interval)
       num_cpus = cpu_count() or 1
   
       def timer():
           return _timer() * num_cpus
   
       if blocking:
           st1 = timer()
           pt1 = self._proc.cpu_times()
           time.sleep(interval)
           st2 = timer()
           pt2 = self._proc.cpu_times()
       else:
           st1 = self._last_sys_cpu_times
           pt1 = self._last_proc_cpu_times
           st2 = timer()
           pt2 = self._proc.cpu_times()
           if st1 is None or pt1 is None:
               self._last_sys_cpu_times = st2
               self._last_proc_cpu_times = pt2
               return 0.0
   
       delta_proc = (pt2.user - pt1.user) + (pt2.system - pt1.system)
       delta_time = st2 - st1
       # reset values for next call in case of interval == None
       self._last_sys_cpu_times = st2
       self._last_proc_cpu_times = pt2
   
       try:
           # This is the utilization split evenly between all CPUs.
           # E.g. a busy loop process on a 2-CPU-cores system at this
           # point is reported as 50% instead of 100%.
           overall_cpus_percent = ((delta_proc / delta_time) * 100)
       except ZeroDivisionError:
           # interval was too low
           return 0.0
       else:
           # Note 1:
           # in order to emulate "top" we multiply the value for the num
           # of CPU cores. This way the busy process will be reported as
           # having 100% (or more) usage.
           #
           # Note 2:
           # taskmgr.exe on Windows differs in that it will show 50%
           # instead.
           #
           # Note 3:
           # a percentage > 100 is legitimate as it can result from a
           # process with multiple threads running on different CPU
           # cores (top does the same), see:
           # http://stackoverflow.com/questions/1032357
           # https://github.com/giampaolo/psutil/issues/474
           single_cpu_percent = overall_cpus_percent * num_cpus
           return round(single_cpu_percent, 1)

   The result also depends on how many CPUs are on your machine. nvitop uses psutil to gather process information about CPU and virtual RAM usage. If you think this would be a bug, please report it to giampaolo/psutil.

Here is the result from my side:

from nvitop.