For pthreads it will result in undefined behaviour. From the man page for pthread_mutex_unlock:
Calling pthread_mutex_unlock() with a mutex that the calling thread does not hold will result in undefined behavior.
Other mutexes will have their own behviour. As others have stated, it’s best to read the manual for whichever mutex you’re using.