It varies a little from one server to another, but the basic components are an encoder/streaming client on the local client that takes input from the camera and turns it in to a stream that can be sent to the stream server on the VPS.
The job of the stream server (such as Red5) is then to provide a publishing point that relays the stream to viewers. The streaming server can either pull from the client or the client can push to a target location on the stream server in order to provide content to the publishing point.
Each viewer then connects to the published RTMP stream on the streaming server and is provided with a copy of the stream that is being uploaded. Note that every copy has to be sent out separately unless you are on a private network where you can use multicast, so bandwidth needs rise quickly, though it should be possible to do 50 SD or lower quality streams from a single VPS provided it has a reliable 100Mbps connection.
The website simply contains a viewer that hooks up to the RTMP (or similar) stream and is actually operating client side, not server side.
So to quickly recap, client uploads to back end of streaming server, streaming server replicates out to every viewer requesting to watch it and the web server simply gives users a client side viewer that connects to the video stream.
To briefly hit on what each one of the component you mentioned.
- RTMP is real time media protocol, which is a common streaming protocol used for actually relaying the video stream content.
- Red5 is a fairly popular free streaming media server software used for replicating out RTMP and similar media streams to clients.
- Wowza is another streaming media server software, however it is a commercial, paid software product. If you are limited to free software, Wowza is not an option.
- FFMpeg is a popular open source video encoder. It runs on the client and can be configured to publish a stream that can be sent to Red5 or similar streaming media server.
Additionally, you will need an end-point viewer capable of consuming the stream. Software like Windows Media Player or Quicktime should be able to view a stream, but it is often best to include an embedded cross platform player in the website that people are going to for the stream. There are many options for this component with different strengths and weaknesses, but any should work for your purposes.