If you’re just trying to look around, and see the various directories and files in your deployment, you can enter the site’s “Kudu” dashboard, using the url format http://<yoursitename>.scm.azurewebsites.net
This will give you a web-based dashboard, including a debug console (web-based) where you can explore your various directories (and the directories will show up visually as well).
More info can be found in this post from the Azure Websites team.