Use pix2pix to generate videos

UCF101-Action Recognition dataset and MPII Cooking Activities dataset

Dependencies:

1) numpy

2) keras

3) matplotlib

4) skimage

5) skvideo

Steps to Generate videos:

1) Use an open source tool inside magenta to generate paired video frames.

Paired frame:

2) Copied paired frames to a folder, and run compress.py to create a npz file as the training data set

3) Run train.py to train the pix2pix model

4) Copy at least one frame as the input, and run predict.py to predict a sequence of frames

5) Run jpgstovideo.py to combine the predicted frames to a mp4 video

Output:

128 X 128 frames random length video

Download predicted video with all original frames

Download predicted video with one original frame

Conclusion:

1) Pix2pix is very good to learn picture to picture mappings

2) It can generate very good videos if training with all original frames

3) It only learn limited length of frame to frame mappings. In this project, it only learns transition from two continuous frames

4) It cannot predict longer videos if just use one origin frame as input.

5) Sometimes, it cannot predict video well if no much difference between the continuous two frames. In this case, this model only can predict the same frames over and over again.