The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences
Download
Publications Copernicus
Download
Citation
Articles | Volume XLII-2/W16
Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-2/W16, 3–11, 2019
https://doi.org/10.5194/isprs-archives-XLII-2-W16-3-2019
Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLII-2/W16, 3–11, 2019
https://doi.org/10.5194/isprs-archives-XLII-2-W16-3-2019

  17 Sep 2019

17 Sep 2019

FUTUREGAN: ANTICIPATING THE FUTURE FRAMES OF VIDEO SEQUENCES USING SPATIO-TEMPORAL 3D CONVOLUTIONS IN PROGRESSIVELY GROWING GANS

S. Aigner and M. Körner S. Aigner and M. Körner
  • Lehrstuhl für Methodik der Fernerkundung,Technical University of Munich, Munich, Germany

Keywords: Deep Learning, Video Prediction, Generative Adversarial Networks, Generative Modeling

Abstract. We introduce a new encoder-decoder GAN model, FutureGAN, that predicts future frames of a video sequence conditioned on a sequence of past frames. During training, the networks solely receive the raw pixel values as an input, without relying on additional constraints or dataset specific conditions. To capture both the spatial and temporal components of a video sequence, spatio-temporal 3d convolutions are used in all encoder and decoder modules. Further, we utilize concepts of the existing progressively growing GAN (PGGAN) that achieves high-quality results on generating high-resolution single images. The FutureGAN model extends this concept to the complex task of video prediction. We conducted experiments on three different datasets, MovingMNIST, KTH Action, and Cityscapes. Our results show that the model learned representations to transform the information of an input sequence into a plausible future sequence effectively for all three datasets. The main advantage of the FutureGAN framework is that it is applicable to various different datasets without additional changes, whilst achieving stable results that are competitive to the state-of-the-art in video prediction. The code to reproduce the results of this paper is publicly available at https://github.com/TUM-LMF/FutureGAN.