What sorts of fabric/material give good results? Both general tips and specific examples/links are helpful.
Great Resources for tips on lighting and materials:
Underexposure (not lighting enough) is your biggest enemy, followed closely by uneven falloff across the screen. Always keep your talent/target a good distance from the screen.
Is it possible to light it adequately without a full lighting kit?
Not easily, but don't dismay, it always takes a skilled hand to get greenscreen lit well, so in many cases, even with a full lighting kit, results will be sub-par. It's a tough thing to get right, because lighting needs vary a lot depending on your shooting situation. Your best bet for getting good, even light, is to have an on-set monitor setup (such as a computer with OnLocation), and test, test, test, before you start shooting.
What sort of encoding is recommended when filming it? (i.e. is it possible to use a DSLR with h.264?)
Great question. This is showing that you've done enough research already to be aware of the issues. "Possible" isn't the word I'd use to describe your capture options, because it's more of a spectrum from good idea, to difficult to work with, almost entirely because of chroma subsampling
Difficult: DSLR on-camera recording, consumer capture codecs (AVCHD, HDV, all of which probably will be 4:2:0)
Better: Output to an HDMI recorder at 4:2:2
Good idea: Uncompressed S-Log 4:4:4 (from something like an F3)
What software would you recommend for post processing the key? Does some work better with homebrewed screens than others?
Some ordinary options are KeyLight (in Premiere and After Effects) and Primatte (which is a real pleasure to use). Keying tools won't care how you created the screen, just how well it was exposed, and how much color separation there is between subject and background.
Finally, a more-radical-than-it-should-be thing to consider is using a neutral gray background instead of green or blue, and relying on a combination of luma keying and roto. One bad example I've always remembered was about the keying in Forrest Gump. Stu Maschwitz once explained the effects artists had to roto every single frame to take care of green spill and edges. Their lives would've been easier if they'd just gone with a neutral gray.
Green does not equal magic movie effects, it's just one of many options.