Your regex is perfect, and yes, you must escape slashes since JavaScript uses the slashes to indicate regexes.
However, the problem is that JavaScript’s replace method does not perform an in-place replace. That is, it does not actually change the string — it just gives you the result of the replace.
Try this:
myString = '/courses/test/user';
myString = myString.replace(/\/courses\/([^\/]*)\/.*/, "$1");
document.write(myString);
This sets myString to the replaced value.