Firstly, you have a few terminology problems:
- the X509 standard defines certificates, and RSA and DSA are two of the public key algorithms that can be used in those certificates;
- certificates are used to hold public keys, and never private keys.
- PKCS#12 is a standard for a container which can hold an X509 client certificates and the corresponding private keys, as well as (optionally) the X509 certificates of the CAs that signed the X509 client certificate(s).
So, if you’re examining a PKCS#12 file (typically .p12 extension), then you already know:
- It contains at least one X509 client certificate, which contains a public key; and
- It contains the corresponding private keys.
All you don’t know is whether those certificate & private key are RSA or DSA. You can check this by extracting the certificate(s), and then examine them:
openssl pkcs12 -in mycert.p12 -clcerts -nokeys -out mycert.crt
openssl x509 -in mycert.crt -text
The text output of the openssl x509 command should include a Subject Public Key section, which will include fields that let you see if it’s an RSA or DSA key (along with the key size).