For brevity, this error message is generated when there is not enough memory to handle the batch size.
Expanding on Steven’s link (I cannot post comments yet), here are a few tricks to monitor/control memory usage in Tensorflow:
- To monitor memory usage during runs, consider logging run metadata. You can then see the memory usage per node in your graph in Tensorboard. See the Tensorboard information page for more information and an example of this.
- By default, Tensorflow will try to allocate as much GPU memory as possible. You can change this using the GPUConfig options, so that Tensorflow will only allocate as much memory as needed. See the documentation on this. There you also find an option that will allow you to only allocate a certain fraction of your GPU memory (I have found this to be broken sometimes though.).