You will have to look inside the code of fmincg because it is not part of Octave. After some search I found that it’s a function file provided by the Machine Learning class of Coursera as part of the homework. Read the comments and answers on this question for a discussion about the algorithms.