If you’re trying to edit the layer mask, I would try to create a square selection using the Marquee tool and then painting the area.  If you wanted it inversed, then invert your select and paint the exterior.

I believe when you use the Rectangle tool in PS, it defaults to creating a new layer.