This has been edited to include a safe, nonspecific answer.
IN GENERAL: dispose everything, because Dispose is the .NET framework’s way to free up external resources (such as file handles, TCP ports, database connections, etc). Resources are not guaranteed to be released unless you call Dispose(). So beware. This is the general, non-SharePoint answer.
SPECIFICALLY WHEN DEALING WITH SharePoint:
When you close the PowerShell.exe process, the memory is freed. If you need to dispose objects to keep memory pressure down (important in production environments or if you’re looping over all sites/webs), make sure to dispose. If not, you don’t need to worry about disposing.
The reason why we’re so crazy about disposing in the first place is because most SharePoint code runs in long-running processes (either in an ASP.NET worker process or OWSTimer.exe) and failing to dispose can cause difficult-to-troubleshoot, sudden catastrophes (i.e., web server go boom). These catastrophic performance issues/OutOfMemoryExceptions don’t affect me most of the time when working in PowerShell. I run ad-hoc scripts, I waste ~3-50MB of RAM because I fail to dispose my objects, I close the PowerShell window and the memory is freed. Most of the time it’s a nonissue.
I’ve built scripts for working with SharePoint, and most of the time I don’t bother disposing.
Here is a script wherein I dispose SPSite and SPWeb objects
Here is a script in which I don’t bother to dispose an SPSite object