Stumbled upon this issue today, here’s my research:
Debugging SMTP connections in the GitLab GUI is not supported yet. However there is a pending feature request and a command line solution.
Set the desired SMTP settings /etc/gitlab/gitlab.rb and run gitlab-ctl reconfigure (see https://docs.gitlab.com/omnibus/settings/smtp.html).
Start the console running gitlab-rails console -e production.
Show the configured delivery method (should be :smtp) running the command ActionMailer::Base.delivery_method. Show all configured SMTP settings running ActionMailer::Base.smtp_settings.
To send a test mail run
Notify.test_email('youremail@example.com', 'Hello World', 'This is a test message').deliver_now
On the admin page in GitLab, the section »Background jobs« shows information about all jobs. Failing SMTP connections are listed there as well.
Please note, you may need to restart the GitLab instance in order to use the newly configured SMTP settings (on my instance the console was able to send mails, the GUI required a restart). Run gitlab-ctl restart to restart your instance.