The latLng of the required address and the latLng of the panorama location (the position of the streetview car from where images are shot) are not the same. As far as I know, the API does not set the heading in the right direction of the address latlng, you will have to set it yourself.
This is how I did it, I used the StreetViewService to find the nearest panorama shot available and then calculated the Pov heading using the computeHeading method here