Docker is the command-line tool that uses containerization to manage multiple images and containers and volumes and such — a container is basically a lightweight virtual machine. See https://docs.docker.com/ for extensive documentation.
Until recently Docker didn’t run on native Mac or Windows OS, so another tool was created, Docker-Machine, which creates a virtual machine (using yet another tool, e.g. Oracle VirtualBox), runs Docker on that VM, and helps coordinate between the host OS and the Docker VM.
Since Docker isn’t running on your actual host OS, docker-machine needs to deal with IP addresses and ports and volumes and such. And its settings are saved in environment variables, which means you have to run commands like this every time you open a new shell:
eval $(docker-machine env default)
docker-machine ip default
Docker-Compose is essentially a higher-level scripting interface on top of Docker itself, making it easier (ostensibly) to manage launching several containers simultaneously. Its config file (docker-compose.yml) is confusing since some of its settings are passed down to the lower-level docker process, and some are used only at the higher level.
I agree that it’s a mess; my advice is to start with a single Dockerfile and get it running either with docker-machine or with the new beta native Mac/Windows Docker, and ignore docker-compose until you feel more comfortable with the lower-level tools.