Finally I found a solution. It looks like since Android P (KeyStore.PrivateKeyEntry) keyStore.getEntry("alias", null) is not a proper way to get private key.
I was able to get rid of this warning by accessing private/public key this way
KeyStore keyStore = KeyStore.getInstance("AndroidKeyStore");
keyStore.load(null);
PrivateKey privateKey = (PrivateKey) keyStore.getKey("alias", null);
PublicKey publicKey = keyStore.getCertificate("alias").getPublicKey();