Scikit-learn confusion matrix

scikit learn sorts labels in ascending order, thus 0’s are first column/row and 1’s are the second one

>>> from sklearn.metrics import confusion_matrix as cm
>>> y_test = [1, 0, 0]
>>> y_pred = [1, 0, 0]
>>> cm(y_test, y_pred)
array([[2, 0],
       [0, 1]])
>>> y_pred = [4, 0, 0]
>>> y_test = [4, 0, 0]
>>> cm(y_test, y_pred)
array([[2, 0],
       [0, 1]])
>>> y_test = [-2, 0, 0]
>>> y_pred = [-2, 0, 0]
>>> cm(y_test, y_pred)
array([[1, 0],
       [0, 2]])
>>> 

This is written in the docs:

labels : array, shape = [n_classes], optional
List of labels to index the matrix. This may be used to reorder or select a subset of labels. If none is given, those that appear at least once in y_true or y_pred are used in sorted order.

Thus you can alter this behavior by providing labels to confusion_matrix call

>>> y_test = [1, 0, 0]
>>> y_pred = [1, 0, 0]
>>> cm(y_test, y_pred)
array([[2, 0],
       [0, 1]])
>>> cm(y_test, y_pred, labels=[1, 0])
array([[1, 0],
       [0, 2]])

And actual/predicted are oredered just like in your images – predictions are in columns and actual values in rows

>>> y_test = [5, 5, 5, 0, 0, 0]
>>> y_pred = [5, 0, 0, 0, 0, 0]
>>> cm(y_test, y_pred)
array([[3, 0],
       [2, 1]])

• true: 0, predicted: 0 (value: 3, position [0, 0])
• true: 5, predicted: 0 (value: 2, position [1, 0])
• true: 0, predicted: 5 (value: 0, position [0, 1])
• true: 5, predicted: 5 (value: 1, position [1, 1])