Abstract—Video quality has become more important due to the development of information and communication technology. In this study, we propose a spatio-temporal super-resolution method using a Generative Adversarial Network (GAN) in order to achieve a higher frame rate. In recent years, with the development of machine learning technology such as convolutional neural networks, clearer interpolation frame estimation has been realized. Most of the estimation methods use optimization techniques that minimize the mean squared reconstruction error, and the resulting estimates show a high Peak Signal-to-Noise Ratio (PSNR). However, these Mean Squared Error (MSE)-based methods often lack the high-frequency components of the generated frame, resulting in blurry frames. To address this issue, our study adopts GAN that uses spatiotemporal convolution instead of traditional spatial convolution. We propose a method for video frame rate up-conversion with perceptual loss function, which consists of adversarial loss and mean squared loss. This adversarial loss produces a more natural frame using a discriminator network trained to distinguish between the estimated frame and the original frame. We verified the effectiveness of the proposed method using video data containing complex and large motions such as rotational motion and scaling.

Index Terms—video frame interpolation, machine learning, neural networks, deep learning, spatio-temporal data analysis

Cite: Naomichi Takada and Toshiaki Omori, "Video Frame Rate Up-Conversion via Spatio-Temporal Generative Adversarial Networks," Journal of Image and Graphics, Vol. 9, No. 3, pp. 87-94, September 2021. doi: 10.18178/joig.9.3.87-94

Copyright © 2021 by the authors. This is an open access article distributed under the Creative Commons Attribution License (CC BY-NC-ND 4.0), which permits use, distribution and reproduction in any medium, provided that the article is properly cited, the use is non-commercial and no modifications or adaptations are made.