How is this bitwise AND operator masking the lower seven order bits of the number?

The number 0177 is an octal number representing the binary pattern below:

0000000001111111

When you AND it using the bitwise operation &, the result keeps the bits of the original only in the bits that are set to 1 in the “mask”; all other bits become zero. This is because “AND” follows this rule:

X & 0 -> 0 for any value of X
X & 1 -> X for any value of X

For example, if you AND 0177 and 0545454, you get

0000000001111111 -- 0000177
0101010101010101 -- 0545454
----------------    -------
0000000001010101 -- 0000154