bp|bd # will do it.
Details: The bp command (“buffer previous”) moves us to a different buffer in the current window (bn would work, too), then bd # (“buffer delete” “alternate file”) deletes the buffer we just moved away from. See :help bp, :help bd, and :help alternate-file.