Here's the thing about modern video—most formats are actually designed presuming a varying frame rate. We like to think of them like digital filmstrips, but that model rarely works when we're streaming over a network or capturing from many kinds of devices. So, every frame has what's called a Presentation Time Stamp, or PTS.
The problem here is that, while you do indeed concatenate in that order, the PTS for each frame never changes. You can actually manually set the PTS of a frame with the setpts filter, which is definitely something to look into as panning video can get much more complicated than this, and it's deliciously flexible; but may require some reading to understand all of the parameters.
The fields you'll need to know are PTS, which is the presentation timestamp before the filter is run; STARTPTS, which is the presentation timestamp of the first frame rendered, and TB, which stands for Time Base and is basically the ratio between a PTS "tick" and a second; so if you know that your first clip is going to be five seconds long, and you want your second clip to start immediately afterward, you will want to add 5.0/TB to your new PTS.
So, I suggest (PTS-STARTPTS)+(5/TB) for your second clip; but this isn't the only way to determine it and feel free to adjust it if the result doesn't feel right to you.
We're going to use a complex filter here, as we're splitting our stream into two separate streams and concatenating them at the end. (Complex streams actually aren't all that bad, but do take a little getting used to.) One of the big differences with a complex filter is that you generally start each command by specifying a stream for it to use, and end each command specifying a new stream to write to, both in square brackets. When we begin with [0:v], that means to take the first input (input number zero from file input.mp4) and grab its video stream. What comes at the end can be used to identify it later when you want to filter it again.
Lastly, it helps to use the -map tag to tell FFmpeg which stream it is writing from. It's critical, as if FFmpeg wasn't aware of which one to use you could end up with literally any stream or combination thereof in your file; and an invocation can take a while to complete.
Your final command:
ffmpeg -i input.mp4 -filter_complex " \
-map [outv] output.avi
Remember that if you want to use this for any other file, you will need to adjust the (5/TB) part to match the end of your first clip.
While I'm a very experienced programmer, I'm relatively new to FFmpeg in-depth; so it's possible that someone else here will have more details for you to consider.