Since version 1.8, Ansible supports symbolic modes. Thus, the following would perform the task you want:
- name: Make my directory tree readable
file:
path: dir
mode: u=rwX,g=rX,o=rX
recurse: yes
Because X (instead of x) only applies to directories or files with at least one x bit set.