[Enhancement] Monitoring % of tensor cores about nvitop HOT 4 CLOSED

Hi @johnnynunez, the PyTorch Kineto library calculates the tensor core ratio from the kernel times.

https://github.com/pytorch/kineto/blob/6e81ce05c4d9898194fc5432624242cb47a77050/tb_plugin/torch_tb_profiler/profiler/tensor_cores_parser.py#L16-L45

That needs the users explicitly modify their code to register event callbacks.

with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA,
    ]
) as p:
    code_to_profile()

I don't think there is something we can do in nvitop as a monitor tool rather than a profiler. The profiler needs in-process injection to the user program. nvitop is based on the NVML library and runs in a separate process.

from nvitop.

Hi @johnnynunez, nvitop is built on top of the NVIDIA Management Library (NVML), which is instantly usable after installing the NVIDIA driver. The only APIs from NVML to get GPU utilization rates are:

Per device:

• nvmlDeviceGetUtilizationRates (%GPU + %MEM bandwidth)
• nvmlDeviceGetEncoderUtilization (%ENC)
• nvmlDeviceGetDecoderUtilization (%DEC)

Per process:

• nvmlDeviceGetProcessUtilization (%SM (GPU) + %MEM bandwidth + %ENC + %DEC)

nvitop do provide per process GPU utilization usage in the %SM column. I found this blog:

• NVIDIA Ampere Architecture In-Depth

The GA100 streaming multiprocessor (SM).

said:

A100 GPU streaming multiprocessor

The new streaming multiprocessor (SM) in the NVIDIA Ampere architecture-based A100 Tensor Core GPU significantly increases performance, builds upon features introduced in both the Volta and Turing SM architectures, and adds many new capabilities.

The SM unit is consist of multiple tensor cores. Does this resolve your request?

The NVML can only retrieve the SM (streaming multiprocessor) usage in total rather than fine-grained details for the tensor cores. If you want to profile your program, I think using nvprof is the best practice as NVIDIA documented.

from nvitop.

Closing due to inactivity. Please feel free to ask for a reopening.

from nvitop.

Hi, @XuehaiPan pytorch has the capability to watch tensor cores percentatge. Is it possible to use here?

from nvitop.