Ansible Using Custom ssh config File

Not fully possible. You can set ssh arguments in the ansible.cfg:

[ssh_connection]
ssh_args = -F ~/.ssh/client_1_config amazon-server-01

Unfortunately it is not possible to define this per group, inventory or anything else specific.