A square bracket creates a vector or matrix, whereas curly brackets creates a cell array.
When working with numbers, I’d say that 99% of the time, you will use square brackets. Cell arrays allow you to store different types of data at each location, e.g. a 10×5 matrix at (1,1), a string array at (1,2), …
x = [1 2 3]; #% matrix with values 1, 2, 3
y = {1, 'a', x}; #% cell array storing a number, a character, and 1x3 matrix
Here is the MATLAB documentation on cell arrays: http://www.mathworks.com/help/matlab/cell-arrays.html