Sheesh Never mind. I didnt really need the Docker Integration plugin. Seems like that is more for deployment and management of Docker directly through Intellij than for debugging.
To debug my jetty app running inside my docker container, I simply remote debugged:
Run | Edit configurations | + | Remote
The command line args were already OK since I used the default remote debugging options. I only needed to change the Host settings. Here I used the hostname I had set within the docker container