Consider adding default-lazy-init="true" to your spring context xml beans tag (or add lazy-init="true" to those specific beans that take a long time starting up).
This will ensure that only those beans are created that called with applicationContext.getBean(class-or-bean-name) or injected via @Autowired / @Inject into your tests. (Some other types of beans like @Scheduled beans will be created nevertheless but you need to check if that’s a problem or not)
(if you use spring Java configuration, add @Lazy to the config files)
Caveat – If there is a bean that is not initialized explicitly with applicationContext.getBean() or injected as a dependency used by the bean obtained by using applicationContext.getBean(), then that bean will NO LONGER be constructed or initialized. Depending upon your application, that can cause things to fail OR not. Maybe you can selectively mark those beans as lazy-init="false"