ExecutorService, standard way to avoid to task queue getting too full

The trick is to use a fixed queue size and:

new ThreadPoolExecutor.CallerRunsPolicy()

I also recommend using Guava’s ListeningExecutorService.
Here is an example consumer/producer queues.

private ListeningExecutorService producerExecutorService = MoreExecutors.listeningDecorator(newFixedThreadPoolWithQueueSize(5, 20));
private ListeningExecutorService consumerExecutorService = MoreExecutors.listeningDecorator(newFixedThreadPoolWithQueueSize(5, 20));

private static ExecutorService newFixedThreadPoolWithQueueSize(int nThreads, int queueSize) {
    return new ThreadPoolExecutor(nThreads, nThreads,
                                  5000L, TimeUnit.MILLISECONDS,
                                  new ArrayBlockingQueue<Runnable>(queueSize, true), new ThreadPoolExecutor.CallerRunsPolicy());
}

Anything better and you might want to consider a MQ like RabbitMQ or ActiveMQ as they have QoS technology.