Chris Pollett > Students >
Pratikkumar

    ( Print View)

    [Bio]

    [Blog]

    [CS 297 Proposal]

    [PyTorch Bootcamp-IPYNB]

    [Basics of Sanskrit-PDF]

    [Autoencoders-PDF]

    [Overview of GANs-PDF]

    [DeepFakes and Beyond_A Survey-PDF]

    [DeepPrivacy-PDF]

    [Fake_Video_Detection-PDF]

    [Deliverable 1]

    [Deliverable 2]

    [Deliverable 3]

    [Deliverable 4]

    [CS297_Final_Report-PDF]

    [CS298 Proposal]

    [CS298_Final_Report-PDF]

    [CS298_slides-PDF]

    [CS298_Code-ZIP]

Deliverable 4: Face-swap GAN in PyTorch

Data Collection:

GAN model architecture

  • DeepPrivacy (https://arxiv.org/abs/1909.04538) paper's reference implementation is open-sourced by the authors at https://github.com/hukkelas/DeepPrivacy.
    This code is taken as reference and further modification are made.
  • The generator of the original DeepPivacy GAN is shown in Figure 1. The architecture includes all the layers shown in the picture, in
    an encoder-decoder pair. We have simplified the generator architecture by removing layers marked by orange crosses. The model was trained
    on a high-end machine for approx 50 hours. The discriminator is not changed.
  • The specification of the high-end machine: Intel 28 cores CPU, 64GB dram, Nvidia Titan RTX GPU (24GB vram)
                  Figure 1. The architecture of the Generator of the DeepPrivacy GAN.

Architecutes of the models can be seen in below table.

ModelOriginal Generator architecture New Generator architecture
Generator

NetworkWrapper(
  (network): Generator(
    (to_rgb_new): WSConv2dConv2d-[3, 256, 1, 1]
    (to_rgb_old): WSConv2dConv2d-[3, 256, 1, 1]
    (core_blocks_down): ModuleList(
      (0): UnetDownSamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (core_blocks_up): ModuleList(
      (0): Sequential(
        (0): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 295, 1, 1]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): UnetUpsamplingBlock(
            (model): Sequential(
              (0): Sequential(
                (0): WSConv2dConv2d-[256, 256, 3, 3]
                (1): LeakyReLU(negative_slope=0.2)
                (2): PixelwiseNormalization()
              )
              (1): Sequential(
                (0): WSConv2dConv2d-[256, 256, 3, 3]
                (1): LeakyReLU(negative_slope=0.2)
                (2): PixelwiseNormalization()
              )
            )
          )
        )
        (1): UpSamplingBlock()
      )
    )
    (new_up): Sequential(
      (0): Sequential(
        (0): WSConv2dConv2d-[256, 519, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
        (2): PixelwiseNormalization()
      )
      (1): UnetUpsamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (old_up): Sequential()
    (new_down): Sequential(
      (0): UnetDownSamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (from_rgb_new): Sequential(
      (0): WSConv2dConv2d-[256, 3, 1, 1]
      (1): LeakyReLU(negative_slope=0.2)
      (2): PixelwiseNormalization()
    )
    (from_rgb_old): Sequential(
      (0): AvgPool2d(kernel_size=[2, 2], stride=[2, 2], padding=0)
      (1): Sequential(
        (0): WSConv2dConv2d-[256, 3, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
        (2): PixelwiseNormalization()
      )
    )
    (upsampling): UpSamplingBlock()
    (downsampling): AvgPool2d(kernel_size=2, stride=2, padding=0)
  )
  (forward_block): Generator(
    (to_rgb_new): WSConv2dConv2d-[3, 256, 1, 1]
    (to_rgb_old): WSConv2dConv2d-[3, 256, 1, 1]
    (core_blocks_down): ModuleList(
      (0): UnetDownSamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (core_blocks_up): ModuleList(
      (0): Sequential(
        (0): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 295, 1, 1]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): UnetUpsamplingBlock(
            (model): Sequential(
              (0): Sequential(
                (0): WSConv2dConv2d-[256, 256, 3, 3]
                (1): LeakyReLU(negative_slope=0.2)
                (2): PixelwiseNormalization()
              )
              (1): Sequential(
                (0): WSConv2dConv2d-[256, 256, 3, 3]
                (1): LeakyReLU(negative_slope=0.2)
                (2): PixelwiseNormalization()
              )
            )
          )
        )
        (1): UpSamplingBlock()
      )
    )
    (new_up): Sequential(
      (0): Sequential(
        (0): WSConv2dConv2d-[256, 519, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
        (2): PixelwiseNormalization()
      )
      (1): UnetUpsamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (old_up): Sequential()
    (new_down): Sequential(
      (0): UnetDownSamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (from_rgb_new): Sequential(
      (0): WSConv2dConv2d-[256, 3, 1, 1]
      (1): LeakyReLU(negative_slope=0.2)
      (2): PixelwiseNormalization()
    )
    (from_rgb_old): Sequential(
      (0): AvgPool2d(kernel_size=[2, 2], stride=[2, 2], padding=0)
      (1): Sequential(
        (0): WSConv2dConv2d-[256, 3, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
        (2): PixelwiseNormalization()
      )
    )
    (upsampling): UpSamplingBlock()
    (downsampling): AvgPool2d(kernel_size=2, stride=2, padding=0)
  )
)



NetworkWrapper(
  (network): Generator(
    (to_rgb_new): WSConv2dConv2d-[3, 256, 1, 1]
    (to_rgb_old): WSConv2dConv2d-[3, 256, 1, 1]
    (core_blocks_down): ModuleList(
      (0): UnetDownSamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (core_blocks_up): ModuleList(
      (0): Sequential(
        (0): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 295, 1, 1]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): UnetUpsamplingBlock(
            (model): Sequential(
              (0): Sequential(
                (0): WSConv2dConv2d-[256, 256, 3, 3]
                (1): LeakyReLU(negative_slope=0.2)
                (2): PixelwiseNormalization()
              )
            )
          )
        )
        (1): UpSamplingBlock()
      )
    )
    (new_up): Sequential(
      (0): Sequential(
        (0): WSConv2dConv2d-[256, 519, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
        (2): PixelwiseNormalization()
      )
      (1): UnetUpsamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (old_up): Sequential()
    (new_down): Sequential(
      (0): UnetDownSamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (from_rgb_new): Sequential(
      (0): WSConv2dConv2d-[256, 3, 1, 1]
      (1): LeakyReLU(negative_slope=0.2)
      (2): PixelwiseNormalization()
    )
    (from_rgb_old): Sequential(
      (0): AvgPool2d(kernel_size=[2, 2], stride=[2, 2], padding=0)
      (1): Sequential(
        (0): WSConv2dConv2d-[256, 3, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
        (2): PixelwiseNormalization()
      )
    )
    (upsampling): UpSamplingBlock()
    (downsampling): AvgPool2d(kernel_size=2, stride=2, padding=0)
  )
  (forward_block): Generator(
    (to_rgb_new): WSConv2dConv2d-[3, 256, 1, 1]
    (to_rgb_old): WSConv2dConv2d-[3, 256, 1, 1]
    (core_blocks_down): ModuleList(
      (0): UnetDownSamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (core_blocks_up): ModuleList(
      (0): Sequential(
        (0): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 295, 1, 1]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
          (1): UnetUpsamplingBlock(
            (model): Sequential(
              (0): Sequential(
                (0): WSConv2dConv2d-[256, 256, 3, 3]
                (1): LeakyReLU(negative_slope=0.2)
                (2): PixelwiseNormalization()
              )
            )
          )
        )
        (1): UpSamplingBlock()
      )
    )
    (new_up): Sequential(
      (0): Sequential(
        (0): WSConv2dConv2d-[256, 519, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
        (2): PixelwiseNormalization()
      )
      (1): UnetUpsamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (old_up): Sequential()
    (new_down): Sequential(
      (0): UnetDownSamplingBlock(
        (model): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
            (2): PixelwiseNormalization()
          )
        )
      )
    )
    (from_rgb_new): Sequential(
      (0): WSConv2dConv2d-[256, 3, 1, 1]
      (1): LeakyReLU(negative_slope=0.2)
      (2): PixelwiseNormalization()
    )
    (from_rgb_old): Sequential(
      (0): AvgPool2d(kernel_size=[2, 2], stride=[2, 2], padding=0)
      (1): Sequential(
        (0): WSConv2dConv2d-[256, 3, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
        (2): PixelwiseNormalization()
      )
    )
    (upsampling): UpSamplingBlock()
    (downsampling): AvgPool2d(kernel_size=2, stride=2, padding=0)
  )
)

Discriminator
NetworkWrapper(
  (network): DeepDiscriminator(
    (from_rgb_new): Sequential(
      (0): WSConv2dConv2d-[256, 6, 1, 1]
      (1): LeakyReLU(negative_slope=0.2)
    )
    (from_rgb_old): Sequential(
      (0): AvgPool2d(kernel_size=[2, 2], stride=[2, 2], padding=0)
      (1): Sequential(
        (0): WSConv2dConv2d-[256, 6, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
      )
    )
    (new_block): Sequential(
      (0): Sequential(
        (0): WSConv2dConv2d-[256, 263, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
      )
      (1): ResNetBlock(
        (conv): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
          )
          (1): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
          )
          (2): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
          )
          (3): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
          )
        )
      )
      (2): Sequential(
        (0): WSConv2dConv2d-[256, 256, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
      )
      (3): AvgPool2d(kernel_size=[2, 2], stride=[2, 2], padding=0)
    )
    (core_model): Sequential(
      (0): Sequential(
        (0): Sequential(
          (0): WSConv2dConv2d-[256, 263, 1, 1]
          (1): LeakyReLU(negative_slope=0.2)
        )
        (1): ResNetBlock(
          (conv): Sequential(
            (0): Sequential(
              (0): WSConv2dConv2d-[256, 256, 3, 3]
              (1): LeakyReLU(negative_slope=0.2)
            )
            (1): Sequential(
              (0): WSConv2dConv2d-[256, 256, 3, 3]
              (1): LeakyReLU(negative_slope=0.2)
            )
            (2): Sequential(
              (0): WSConv2dConv2d-[256, 256, 3, 3]
              (1): LeakyReLU(negative_slope=0.2)
            )
          )
        )
        (2): Sequential(
          (0): WSConv2dConv2d-[256, 256, 4, 4]
          (1): LeakyReLU(negative_slope=0.2)
        )
      )
    )
    (output_layer): WSLinear(
      (linear): Linear(in_features=256, out_features=1, bias=False)
    )
  )
  (forward_block): DeepDiscriminator(
    (from_rgb_new): Sequential(
      (0): WSConv2dConv2d-[256, 6, 1, 1]
      (1): LeakyReLU(negative_slope=0.2)
    )
    (from_rgb_old): Sequential(
      (0): AvgPool2d(kernel_size=[2, 2], stride=[2, 2], padding=0)
      (1): Sequential(
        (0): WSConv2dConv2d-[256, 6, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
      )
    )
    (new_block): Sequential(
      (0): Sequential(
        (0): WSConv2dConv2d-[256, 263, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
      )
      (1): ResNetBlock(
        (conv): Sequential(
          (0): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
          )
          (1): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
          )
          (2): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
          )
          (3): Sequential(
            (0): WSConv2dConv2d-[256, 256, 3, 3]
            (1): LeakyReLU(negative_slope=0.2)
          )
        )
      )
      (2): Sequential(
        (0): WSConv2dConv2d-[256, 256, 1, 1]
        (1): LeakyReLU(negative_slope=0.2)
      )
      (3): AvgPool2d(kernel_size=[2, 2], stride=[2, 2], padding=0)
    )
    (core_model): Sequential(
      (0): Sequential(
        (0): Sequential(
          (0): WSConv2dConv2d-[256, 263, 1, 1]
          (1): LeakyReLU(negative_slope=0.2)
        )
        (1): ResNetBlock(
          (conv): Sequential(
            (0): Sequential(
              (0): WSConv2dConv2d-[256, 256, 3, 3]
              (1): LeakyReLU(negative_slope=0.2)
            )
            (1): Sequential(
              (0): WSConv2dConv2d-[256, 256, 3, 3]
              (1): LeakyReLU(negative_slope=0.2)
            )
            (2): Sequential(
              (0): WSConv2dConv2d-[256, 256, 3, 3]
              (1): LeakyReLU(negative_slope=0.2)
            )
          )
        )
        (2): Sequential(
          (0): WSConv2dConv2d-[256, 256, 4, 4]
          (1): LeakyReLU(negative_slope=0.2)
        )
      )
    )
    (output_layer): WSLinear(
      (linear): Linear(in_features=256, out_features=1, bias=False)
    )
  )
)

No Change. Used original discriminator only.

Code

PyTorch implementation can be found at: deep_privacy.tgz. It contains the changes made in deep_privacy python module from the original paper.

Sample output















Conclusion

The modified Deep Privacy model also seems to work up to a certain level. Some of the images in sample output seem to generate more human-like faces with different features.
Training longer might generate even more realistic faces.