It seems to be an early or non-standard form of digraph. The code was probably written using EBCDIC instead of ASCII, and EBCDIC doesn’t have [ ] characters (at least not in all code pages).
I found the manual for SAS/C, a C compiler apparently meant for System/370. On page 2-10 (page 42 of the pdf) you can see they list (| |) as “alternate forms” for [ ].
(Though apparently | is not in all the code pages either; but maybe it was in a code page that was more commonly used? I don’t know.)
C99 also included digraphs (and trigraphs) to solve the same problem, but they used <: :> as the digraphs, and ??( ??) for the trigraphs.