I'd like to achieve certain approximate bitrate for video encoding. If I use higher resolution like 720p, I'll get more fine-grained image, but more encoding artifacts. If I use smaller resolution like 480p, I'll get less detail and more coarse image, but higher image quality. Naturally, if the resolution is too high or too low, the result will be bad. Is there any research or information based on experience, what's the optimum resolution, given a desired bitrate and codec (in my case libx264)?
No, such a thing does not exist. You can get some very rough guidelines in terms of things like suggested recording profiles for Youtube, Vimeo, or bitrates used by Netflix, Hulu and Amazon Instant Watch, but at the end of the day, the bitrate needed for a given resolution is entirely content dependent as well as dependent on how much time you can spend encoding.
It is a complex field and far too in depth to detail everything here, but in general, the slower you encode, the fewer noticeable artifacts you will have. Additionally, the less motion (or more predictable motion) that your footage has, the less bitrate will be needed to display a high resolution image. Conversely, if you are using fast software encoding options, there will be lots of artifacts or wasted bitrate for the level of quality you get.
Video compression is a complex field and each video needs to be calibrated a bit for best results. Alternately, you can use a quality based encoding which will vary the bitrate as necessary to preserve a given level of quality (lack of artifacts.) This would then be independent of the resolution of the file, it would just impact how much bitrate is consumed. More active videos would consume more bitrate and more static or predictable videos would use less. Additionally, slower encodings would use lower bitrates while faster encodings would use higher bitrate.
You're right, at some point it visually looks better to downscale and have a lower rez but less artifacty video. The bitrate where this happens depends strongly on how compressible your content is.
One way to judge this for x264 might be to look at the rate-factor. For 2-pass, the x264 output includes the rate-factor. For CRF mode, the CRF value you set IS the rate-factor. Just guessing off the top of my head, a rate-factor of maybe 30 or higher might be the point where you would be better off downscaling. That might still be content dependent; e.g. if you have some very detailed but very static backgrounds, you might prefer to have the moving parts of the picture blocked/blurred, but the background high rez and crisp. (IIRC, play around with AQ strength and/or psy-rd to control distribution of bits between long-displayed static parts of the image vs. parts that are changing.)
In the extreme case of using way too high a rez for your bitrate, a lot of the bits are just per-macroblock overhead (headers and so on), with little room for motion vectors and DCT coefficients. Newer codecs like h.265 have bigger macroblocks (coding units in h.265 terminology), and also just higher coding efficiency (better rate-distortion tradeoff), so they push down the point at which you should start downscaling.
Visually, modern codecs will start to blur things themselves when there aren't enough bits to look sharp. So once things start to blur as much as they would if you just downscaled, you might as well downscale.
For more discussion about encoding stuff, look around on forums.doom9.org. Especially http://forum.doom9.org/showthread.php?p=1691228#post1691228, which is the discussion I was thinking of when writing this.