It seems you just need to add padding to your bytes before decoding. There are many other answers on this question, but I want to point out that (at least in Python 3.x) base64.b64decode will truncate any extra padding, provided there is enough in the first place.
So, something like: b'abc=" works just as well as b"abc==' (as does b'abc=====').
What this means is that you can just add the maximum number of padding characters that you would ever need—which is two (b'==')—and base64 will truncate any unnecessary ones.
This lets you write:
base64.b64decode(s + b'==')
which is simpler than:
base64.b64decode(s + b'=' * (-len(s) % 4))