No, you’re mistaken. Brute force attacks are one thing, but the real danger is Rainbow Tables that, from a hash value, gives you the plaintext password.
First you never ever store anything as plaintext. If someone breach your security (or even if an employee has malicious intent) you don’t want to expose users’ password. So you will use a properly salted hash.
Then, with a 5-digit PIN, it’s way too short to protect with hash. There are rainbow tables (or even Google searches) that would allow someone to get the password back if they get the hash.