You need to add the dynamic library equivalent of -L:
-Wl,-rpath-link,/path/to/lib
This will cause the linker to look for shared libraries in non-standard places, but only for the purpose of verifying the link is correct.
If you want the program to find the library at that location at run-time, then there’s a similar option to do that:
-Wl,-rpath,/path/to/lib
But, if your program runs fine without this then you don’t need it.