return function a(){};
Here function ... is an expression. A named function expression to be precise. The a here doesn’t matter much, it just gives the anonymous function a .name, but it’s still just a function expression that you’re returning.
return
function a(){};
This here is equivalent to:
return;
function a(){};
Here function a is a declaration, not an expression. It is hoisted, creating a local name a in the scope, shadowing the outer a. I.e. it is equivalent to:
function b(){
var a = function () {};
a = 10;
return;
}