If I have 1080p footage in a 720p project, will the keyer effect (chromakey) lose quality?
To ask the question a different way:
Should I have a project with the same dimensions/resolution as the original footage when I key it, and then export it at my desired resolution later? Or is FCPX doing that in the background anyway?
There isn't a right or wrong answer here I don't think. Both approaches have advantages. Keying at 1080p will give you clearer detail when trying to find edges, but also will have more noise that could throw off a key. Additionally, the downscale after the key will apply a slight blur to the image further disguising the edges to make the keying less obvious without having to do an additional blending on the 720p version. That said, it will take additional processing effort as the image is larger and there is more area to process.
On the flip side, keying at 720p will reduce the amount of processing needed, but will not blend as well and will potentially have softer edges that don't key quite as easily. The noise is reduced as mentioned before though, which makes the fill areas easier. It also matches the native resolution of your final output so it may save some quality on the content being keyed in depending on what the source resolution of that content is.
If both source and key are 1080p, I might consider doing it at 1080p and then downscaling the result, but if one of them is 720p to start with, I'd probably key at 720p and soften the edge a bit unless I ran in to problems.