The containers that are not running are not taking any system resources besides disk space.
It is usually good to clean up after yourself, but if you have a lot of them sitting around it shouldn’t slow down performance at all.
If you do notice a slow down when running docker commands with lots of stopped containers, it might be a bug in docker, and you should submit a bug.