Adaptive Frame Rate Video Processing

Media technology, in particular video recording and playback, keeps improving to give users high visual quality for real and virtual content. In recent years, the temporal sampling rate of videos and displays has become one focus of technological development, resulting in increased temporal sampling rates. However, how high should sampling rates be? To answer this question, we determine the minimum sampling rate at which a video should be recorded and presented to make temporal sampling imperceptible to viewers. Through subjective tests, we find that this minimum sampling rate depends on the speed of objects visible in the video, as well as the exposure time of the recording camera. We propose a model to compute the required minimum sampling rate based on these two parameters. In addition, state-of-the-art video codecs employ motion vectors indicating object speed. Therefore, we present a procedure to compute the minimum sampling rate given the encoded video and camera exposure time. Since object speed apparent in a video may be non-constant, the corresponding minimum frame rate is varying. This is why the results of this paper are particularly applicable when used together with adaptive frame rate computer generated graphics or novel video communication solutions that drop insignificant frames. In our experiments, we show that videos played back at the minimum adaptive frame rate achieve an average bit rate reduction of 26% compared to constant frame rate playback, while no difference between the playback strategies can be perceived.