2

I have logged the GPU utilization of my 3 GPUs (GeForce GTX Titan X) for 5 days with nvidia-smi, every 10 seconds. The GPUs were all being utilized: I count 1307 occurrences of 99 % while only 8 occurrences of 100 %. Why nvidia-smi's GPU so rarely reaches 100%?

enter image description here

enter image description here

Another example from https://openai.com/blog/infrastructure-for-deep-learning/:

enter image description here

Community
  • 1
Franck Dernoncourt
  • 4,987
  • I haven't found any solution so far. – Franck Dernoncourt Mar 28 '16 at 18:24
  • So I am still interested if someone has any idea. – Franck Dernoncourt Mar 28 '16 at 18:24
  • It highly depends on what your computations/datasets are, and the load-balancing between the GPUs. Not to mention your hardware which could be a factor. But it seems a little bit curious to want to have exactly 100% of used RAM. – Paradox Apr 03 '19 at 11:55
  • @Paradox 100% GPU utilization, not RAM. – Franck Dernoncourt Apr 04 '19 at 00:33
  • @FrankDernoucourt Yes, you are right, my bad. But still, RAM or GPU utilization, it's the same: having exactly 100% is difficult unless everything is optimized to the very last bit and purely theoretical. When seeing 98 or 99%, it already seems pretty impressive for multiple components on a single machine. That's why your numbers do not seem explicitly showing a problem to me, depending on what's running on your GPUs. – Paradox Apr 04 '19 at 09:20

0 Answers0