The Location that you get from the API will have accuracy in meters. You should also check how old the location is.
https://developer.android.com/reference/android/location/Location.html#getAccuracy()
https://developer.android.com/reference/android/location/Location.html#getTime()
People generally discard the location if accuracy is greater than 50 or 100m.
Why this happens?
It takes some time for the GPS of the device to find satellites and get the signal. Also, the API will try to determine your location based on your network. It kinds of jump between network and GPS, until GPS provides some accurate data.
How to avoid this?
In your location listener, check for accuracy and wait till you have a better accuracy.