Although the other answers are both correct they do not highlight the underlying misunderstanding here enough:
With the env_file option you can specify a file with variables to be injected into the environment in the container.
Using variable substitution in the docker-compose.yml you can access variables in the environment of the docker-compose command, i.e. on the host.
You can set these using the usual mechanisms of your OS/shell, e.g. in bash:
export XUSER=you
docker-compose up
Additionally with docker-compose you can use a .env file in the current directory.
So in your concrete example you should just move env/test.env to .env to add the variables to the environment of docker-compose for variable substitution.
If you also want to add them to the environment in the container you can do it like this:
version: '3'
services:
abc:
build: .
image: xyz:latest
container_name: xyz
# add variables from the docker-compose environment to the container:
environment:
- XUSER=$XUSER
# or even shorter:
- XHOME
# use variable from the docker-compose environment in the config:
user: "${XUSER}"