This isn’t generally possible in a Dockerfile.
Depending on the software, you might be able to do some kind of work-around. For example, you could try something like
RUN echo $(grep $(hostname) /etc/hosts | cut -f1) my.host.name >> /etc/hosts && install-software
By setting the hostname within the same RUN command as you install the software, it’ll happen inside the same layer of the container. Docker will later overwrite the hostname and you’ll have to set it anew when running, but your software might be OK with that.
If you have to do a lot of this, you might try Packer for building containers. It can build Docker containers, but doesn’t use multiple layers. This makes it slower to rebuild, faster to download the built images, and makes it more convenient to do multiple operations on an image before freezing it into a container.