A quick, maybe imperfect solution – use an expression in the volume filter, e.g. for linearly decreasing volume between from 4s to 5s:
volume=1-0.85/(5-4)*(t-4):eval=frame
For t=4, the volume will evaluate to 1, for t=5 it will be 0.15:
for t=4: 1 - 0.85 / (5-4) * (4-4) = 1 - 0.85 / (10-4) * 0 = 1 - 0 = 1
for t=5: 1 - 0.85 / (5-4) * (5-4) = 1 - 0.85 = 0.15
You may use (nested) if functions for evaluating the volume value in different times intervals of your audio stream.
For example, to keep the value 1 for first 4 seconds you may extend the given formula
1-0.85/(5-4)*(t-4)
with the help of the lt (less than) function into
if(lt(t\,4),1,1-0.85/(5-4)*(t-4))
and then keep the value 0.15
if(lt(t\,4),1,if(lt(t\,5),1-0.85/(5-4)*(t-4),0.15))
And so on (always rewrite the last item with a new if function).
Of course, the resulting expression is not the optimal one, but the procedure is pretty straightforward.