Ok problem solved!
The solution is this:
First get the self-signed certificate from the mail server via openssl:
echo | openssl s_client -connect yoursever:port 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > yourcert.pem
Then save the yourcert.pem file into this path /Library/Java/Home/lib/security (on macOSX) and put the cert file into the cacerts like this
keytool -keystore cacerts -importcert -alias youralias -file yourcert.pem
The default keystore password is changeit
You can view the change that you made with this command that shows the Certificate fingerprint.
keytool -list -keystore cacerts
After this you should pass these argument in VM
(for windows and linux type yourpath between ” ” )
-Djavax.net.ssl.trustStore="/Library/Java/Home/lib/security/cacerts"
-Djavax.net.ssl.trustStorePassword="changeit"
For Debug:
-Djava.security.debug=certpath
-Djavax.net.debug=trustmanager