It would be good to know what OS your machine is running and if it's 32 or 64 bit, and the resources as in type of processor (Dual Core, i3, i7?) and how much DRAM is on the motherboard. I am not an After Effects user (yet) but this error appears to be a resource error, which is common to many other video editors.
It would also be helpful to have you describe what you are trying to render in AE. For instance how long is the piece, how many video tracks, what effects are used, etc.
Without knowing your computer's resources is not a show stopper, here are some items to check.
Are you running anything else when AE is trying to render, if so try your render again with nothing running but AE.
If you have a 32 bit machine you may have a 2GB limit on the RAM, which is going to be an issue. Have you tried this on a 64 bit machine with more memory?
What resources do you have AE using besides the requested render? For instance do you have the preview screen on 'auto' size mode, you might try to make the preview smaller and in a 1/4 or 1/2 mode so AE doesn't have to figure out each display frame size thereby freeing up some cycles.