Mask not working

Question

In After Effects, I want to use a mask to prevent an area from beein affected by a layer. I created a mask using the pen tool, but instead of masking the shape, the program masks the bounds of the shape. It "draws" a rectangle arround the path and masks everything except the rectangle.
I am a beginner. What am I doing wrong?
Thanks!

Edit:
The mask works for simple layers, but my layer has an beam effect on it.

Answer 1

Masks happen before effects so some act strange. Two main options:

1. Apply the mask to a solid above your beam and then use Track Matte
2. Pre-comp your Beam and mask that instead