Using Jest, how can I check that an argument to a mocked function is a function?

To assert any function, you can you use expect.any(constructor):

So with your example it would be like this:

expect(AP.require).toBeCalledWith('messages', expect.any(Function))