GPU Infrastructure Monitoring for MLOps: Essential nvidia-smi Commands & Metrics
When running large-scale AI workloads, memory leaks, thermal throttling, and runaway PyTorch processes can crash inference nodes without warning. System administrators managing AI infrastructure need real-time visibility into GPU VRAM utilization, power draw, and process ownership.
1. Advanced nvidia-smi Query Commands
Instead of running standard nvidia-smi, extract structured CSV data for custom log monitors or automated scripts:
# Stream GPU VRAM usage, temperature, and power consumption every 2 seconds
nvidia-smi --query-gpu=timestamp,name,temperature.gpu,utilization.gpu,utilization.memory,memory.used,memory.free --format=csv -l 2
2. Identifying Specific GPU Process Owners
To identify which Python or Docker container is consuming GPU VRAM on multi-tenant servers:
# List all active GPU compute processes with Process IDs (PIDs)
nvidia-smi pmon -s u -v
Once you locate a runaway process PID, inspect its systemd parent unit or Docker container:
# Inspect process details by PID
ps -up [PID]
# Find associated Docker container ID
cat /proc/[PID]/cgroup | grep docker
3. Setting GPU Power Limits for Stability
To prevent server power supplies from tripping during sudden LLM batch inference spikes, cap the maximum wattage allowed per GPU card:
# Enable persistence mode across reboots
sudo nvidia-smi -pm 1
# Cap maximum GPU power limit to 250 Watts (e.g., for NVIDIA RTX 3090 / A4000)
sudo nvidia-smi -pl 250
Comments