Openssl on mac is compiled without zlib support. Alternative method described in this article works on my Yosemite:
dd if=backup.ab bs=1 skip=24 | python -c "import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))" | tar -xvf -
Optionaly, if you just want to convert it into tar archive:
dd if=backup.ab bs=1 skip=24 | python -c "import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))" > backup.tar
It skips first 24 bytes of Android header and then uncompresses zlib data.