What is the equivalent of Eclipse “Custom debug Keystore” in Android studio?

You define a keystore in your build.gradle file. See the Signing Configurations section here: https://developer.android.com/studio/build/index.html

In your case, you want to redefine the keystore for the debug variant:

android {
    signingConfigs {
        debug {
            storeFile file("your.keystore")
        }
    }
}

However, you should really be using two different keystores for debug and release configurations.