Take a look at https://github.com/wavii/rails-dev-tweaks.
Rails is running all of the to_prepare hooks on every Sprockets asset request in development mode. This includes things like auto-(re)loading your code, and various gems sneak work in there too.
rails-dev-tweaks disables to_prepare & reloading on any asset request (and a few others – read the first part of its README). Speeds up your dev environment by a huge amount for any decently sized project. It’s also configurable to do this for any additional requests you like