That line is almost certainly a reference to Hunter College High School, the high school of the film's writer Thunder Levin, which is nicknamed "The Brick Prison" because it looks like...a brick prison. The high school was built on the site of the Squadron A Armory and 2 of the towers are still there.