They came in as .ogg?? I thought you converted to dvd? Are you talking about then ripping the DVD video with HandBrake or something?
Also, you're doing something wrong, or at least have your file extension -> "open with" settings screwed up if renaming a .ogg to a .mp4 is necessary. Maybe it opens with a different player then, and only the player associated with .mp4 is actually working on your computer? (.ogm is the usual extension for ogg video; Maybe you have .ogg associated with an audio-only player?)
Anyway, you're doing several steps here:
- capture a digital version of the video signal from your VCR
- encode that to MPEG2 dvd
- transcode the lossy MPEG2 to something a video codec that can go in a .ogg container. edit (apparently just remux, not xcode, so no quality loss at this step after all).
The achievable quality of step 1 is fairly limited, since the representation stored on the magnetic VHS tapes is of limited quality.
Because of the limited quality of the source, you're probably not losing much quality from having a lossy intermediate codec. Obviously it would be better to go directly from capture -> final, with only lossless steps in between (e.g. capture card in a computer, capture to a lossless file. Or encode in realtime to the final format.)
EDIT: turns out you're just remuxing the dvd-video into an ogg container, not transcoding. I didn't think ogm could hold mpeg2, but I guess it can, and for some reason you're using some software that defaults to muxing into an ogg container.
You ARE probably losing quality if your final output is going into an ogg container, though. It's probably Theora video, which is obsolete (and has a much worse quality vs. bitrate tradeoff than VP8 or x264).
A player like VLC or mpv will let you see the codec details of your video. Or just use mediainfo.
Unless you're archiving in a lossless format (50x the filesize of visually-indistinguishable lossy), you might want to apply some filtering to clean up the video from the VHS tapes before encoding. Depending on the condition of the tapes, they might have significant chroma noise that a denoise filter could help with. This will require some playing around.
edit: if you're going to stick with a capture device that doesn't give access to the digital video except after it encodes to MPEG2 (and makes a trivial menu to make a complete DVD-VIDEO), xcoding the MPEG2 to lossless would only be useful if you wanted to apply some filtering. Never xcode a lossy codec to lossless. You'll increase the filesize by a LOT. The lossy encode is the most compact representation of that exact decoded bitstream, so just save it instead of a lossless representation of decoding the lossy encode.
So, if you can get deinterlace/denoise filtering to work well enough "on-the-fly" in realtime, then you can use dvd images straight from the Magnavox as your archival masters. In 10 or 20 years, there might be smarter filters that will do a better job, so it's best to keep the source, as long as it's in a state that's usable in realtime.
If you can't get the necessary filtering to run well on-the-fly on at least your library computers, then you should probably deinterlace and maybe even denoise now, and make either lossless h.264, FFv1, or something lossy. mcdeint is slow, but does a good job. (don't just use yadif for master copies.) You'd need to decide whether to archive at 720x480p60, or throw away half the temporal resolution and deinterlace 720x480i60 -> 720x480p30. The decision is somewhat content-dependent, but if you're using a codec with inter-prediction (P and B frames), like h.264, then higher fps means frames are more similar to each other, so it takes much less than twice as much space to compress twice the FPS at the same quality.
http://linux.goeszen.com/digitising-vhs-tapes-on-linux.html has some stuff to say about it.
http://ubuntuforums.org/showthread.php?t=2091899 suggests hqdn3d as a denoise filter. That sounds right; I would have tried that one first over any of ffmpeg's other postprocessing filters, if I was trying to do this myself.
end of edit
You'll want to read up on video codecs to make a decision on which one to use for archival storage. VP8 and VP9 are free from patent royalties. h.264 and h.265 aren't, but patents don't live forever, and h.264 is the current playable-by-everything standard for video. You'll want to check up on the patent royalty situation, but as long as you aren't selling access to h.264 video, you shouldn't owe any fees. The best h.264 encoder in the world is x264, which is free and open source, and very mature and well optimized.
h.265 is next-gen, and x265 is still very slow, but can get better quality at the same bitrates as x264. Google's VP9 encoder is somewhat better than x264 for quality-per-bitrate, but takes at least 10x longer to encode. VP8 is not bad, but I think it's a bit worse quality-per-bitrate (aka rate-distortion) compared to x264. (both with near-maxed CPU-usage settings).
If you want library patrons to play files on their own devices right now, I'd suggest going with h.264 because of the massive compatibility advantage over any other format. If we're only talking about playback on library-owned computers, I might play around with VP9.
For future-proofing, every player for the next 50 years at least will support h.264, I'd guess. I think H.264 for video will be like mp3 for audio: people will still be using it LONG after it's obsolete, because it's the first format that doesn't completely suck, and that has widespread player support for everything from streaming to files to videoconferencing.
edit: keeping the original VHS is of limited value. They will continue to degrade, but digital files won't. If you can afford the space in temp-controlled storage, it won't HURT to keep them, but the sooner you get them capped with high-quality analog equipment the better. I'd keep them for at least a couple years after doing the initial digital conversion, in case you discover a problem or error with the digital setup and need to redo it!