Actually, this is the precise reason why perfect forwarding was introduced. Rewrite the constructor as
template <typename L, typename O>
LinearClassifier(L && loss, O && optimizer)
: _loss(std::forward<L>(loss))
, _optimizer(std::forward<O>(optimizer))
{}
But it will probably be much simpler to do what Ilya Popov suggests in his answer. To be honest, I usually do it this way, since moves are intended to be cheap and one more move does not change things dramatically.
As Howard Hinnant has told, my method can be SFINAE-unfriendly, since now LinearClassifier accepts any pair of types in constructor. Barry’s answer shows how to deal with it.