How do I type hint a filename in a function?

I think what you are looking for is Structural Typing, which is not yet supported. It is proposed in PEP 544.

In the mean time, you could do a half-way effort by annotating with Union[str, bytes, os.PathLike].