Trying to mask smoke from partical system but feathering doesn't work

Question

This is such an straight forward problem, but I just can't seem to figure it out. This is what I have:

As you can see I need to mask the part where the smoke is overlapping with the body to hide it.

This is what I get from using pen tool, or shape tool to create the mask.

As you can see the smoke is still overlapping the body, and I get these hard edges which doesn't follow the pen tool path at all.I also try to adjust the feathering setting under the mask, it only moves the bounding box outward, and doesn't smooth the edges at all.

I am not that familiar with AE, so if someone can help me that would be great.

Answer 1

Pre-comp the smoke layer first, then apply the mask in your main outer comp.