You can also go into your settings and use the following:
“window.reopenFolders”: “none”
which will not reopen the folders you were working on when you closed the editor. The other options are one (the default) and all.
Edit 2017-11-09:
The option is now changed in latest versions.
"window.restoreWindows": "none"
See Mathieu DOMER’s answer.
Edit 2018-09-12:
Another setting related to this is the hotExit setting. This has been discussed in this answer to a related question. To prevent reopening and remembering unsaved files, you can set this to:
"files.hotExit": "off"
But from the test I’ve made, when the window.restoreWindows setting is set to none, this is not needed. I haven’t tested every possible combination, so YMMV.
And to answer a question in the comments, to edit the settings, you have to open the settings file. Some documentation can be found here (at least on the date I am writing this).
Edit 2022-03-16:
If you prefer using a GUI to change the settings, see D’Arcy Rittich’s answer.