If the camera's static, then it's totally doable. Start by making a clean plate image of the whole area, as I think you have done.
Put that on a layer above (on top of) your original footage. Whenever the problem people wander in to shot, cover them up by putting a masked area of your clean plate over them.
As you mentioned, sometimes this means the desirable foreground subjects get covered up by the clean plate. But there is a solution to this, (even though it is time consuming):
Duplicate your original footage, and put the duplicated track on top of your clean plate, so it's the uppermost layer. Whenever you have some "desirable foreground" subject overlapping your clean plate, animate a masked area of it on top of the plate. Use lots of feathering and careful key framing, and you should be able to get an acceptable result.
Here's a (non-video) tutorial showing how to remove an unwanted object:
Here's the excellent video copilot series, showing how to animate a mask:
Neither show the idea of putting another foreground track over the top, but should get you along the way...