Start with Emacs Lisp (given that you seem to have already started customizing your emacs) – it seems to be more immediately useful for you.
Do keep in mind that there is quite a bit of difference between Emacs Lisp (there is a lot of buffer management stuff, especially in the code you’re likely to see in the wild) and Common Lisp (more similar to a general purpose programming language), so which ever route you take, expect some amount of “wastage”