I believe that you need:
COPY downloads/ /tmp/downloads/
That will copy the contents of the downloads directory into a directory called /tmp/downloads/ in the image.
I believe that you need:
COPY downloads/ /tmp/downloads/
That will copy the contents of the downloads directory into a directory called /tmp/downloads/ in the image.