It depends on the pager that the man command uses on your system.
If it is less (as in ‘less is more’), then you can go backwards. (With less, b goes back a page, amongst other options for doing that; so does Control-B. The h command gives me several pages full of options that I seldom or never use.)
If the pager is more (as in … oh, there isn’t an inverse jibe), then the answer may well be ‘No, you cannot go backwards’.
On Mac OS X (10.7.4), the man man command cites the environment variables MANPAGER and HTMLPAGER that can be set; the default MANPAGER is less.