Problem solved – solution for others who may find it helpful.
I’ve found that Clojure has the clojure.lang.PersistentQueue class that does what is needed.
You can create an instance like this:
(def x (atom clojure.lang.PersistentQueue/EMPTY))
As far as I can see, you currently need to use the Java interop to create the instance but as Michal helpfully pointed out you can use peek, pop and conj subsequently.