Ctrl+- should do the trick which is the standard keyboard shortcut for Navigate Backwards. Or using the navigation buttons in the standard toolbar.
This is explained on MSDN at https://blogs.msdn.microsoft.com/zainnab/2010/03/01/navigate-backward-and-navigate-forward/
On this MSDN page, you can see the following explanations
The editor remembers locations when you move in a single command more
than several lines away from where you are currently working, or if
you edit in a particular location that is not adjacent to the last
place you edited.The goal is to remember interesting locations so that you can recall
where you have been working without remembering so many locations the
feature is not useful (such as every character typed, or every line
entering several new lines of code one right after the other).