[ { "path": ".gitignore", "content": "*/images/*.png\n*/images/*.jpg\n*/*.jpg\n*/*.png\n*.json\n*.h5\n*.hdf5\n.DS_Store\n*/datasets\n\n__pycache__\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2017 Erik Linder-Norén\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "

\n \n

\n\n**This repository has gone stale as I unfortunately do not have the time to maintain it anymore. If you would like to continue the development of it as a collaborator send me an email at eriklindernoren@gmail.com.**\n\n## Keras-GAN\nCollection of Keras implementations of Generative Adversarial Networks (GANs) suggested in research papers. These models are in some cases simplified versions of the ones ultimately described in the papers, but I have chosen to focus on getting the core ideas covered instead of getting every layer configuration right. Contributions and suggestions of GAN varieties to implement are very welcomed.\n\nSee also: [PyTorch-GAN](https://github.com/eriklindernoren/PyTorch-GAN)\n\n## Table of Contents\n * [Installation](#installation)\n * [Implementations](#implementations)\n + [Auxiliary Classifier GAN](#ac-gan)\n + [Adversarial Autoencoder](#adversarial-autoencoder)\n + [Bidirectional GAN](#bigan)\n + [Boundary-Seeking GAN](#bgan)\n + [Conditional GAN](#cgan)\n + [Context-Conditional GAN](#cc-gan)\n + [Context Encoder](#context-encoder)\n + [Coupled GANs](#cogan)\n + [CycleGAN](#cyclegan)\n + [Deep Convolutional GAN](#dcgan)\n + [DiscoGAN](#discogan)\n + [DualGAN](#dualgan)\n + [Generative Adversarial Network](#gan)\n + [InfoGAN](#infogan)\n + [LSGAN](#lsgan)\n + [Pix2Pix](#pix2pix)\n + [PixelDA](#pixelda)\n + [Semi-Supervised GAN](#sgan)\n + [Super-Resolution GAN](#srgan)\n + [Wasserstein GAN](#wgan)\n + [Wasserstein GAN GP](#wgan-gp) \n\n## Installation\n $ git clone https://github.com/eriklindernoren/Keras-GAN\n $ cd Keras-GAN/\n $ sudo pip3 install -r requirements.txt\n\n## Implementations \n### AC-GAN\nImplementation of _Auxiliary Classifier Generative Adversarial Network_.\n\n[Code](acgan/acgan.py)\n\nPaper: https://arxiv.org/abs/1610.09585\n\n#### Example\n```\n$ cd acgan/\n$ python3 acgan.py\n```\n\n

\n \n

\n\n### Adversarial Autoencoder\nImplementation of _Adversarial Autoencoder_.\n\n[Code](aae/aae.py)\n\nPaper: https://arxiv.org/abs/1511.05644\n\n#### Example\n```\n$ cd aae/\n$ python3 aae.py\n```\n\n

\n \n

\n\n### BiGAN\nImplementation of _Bidirectional Generative Adversarial Network_.\n\n[Code](bigan/bigan.py)\n\nPaper: https://arxiv.org/abs/1605.09782\n\n#### Example\n```\n$ cd bigan/\n$ python3 bigan.py\n```\n\n### BGAN\nImplementation of _Boundary-Seeking Generative Adversarial Networks_.\n\n[Code](bgan/bgan.py)\n\nPaper: https://arxiv.org/abs/1702.08431\n\n#### Example\n```\n$ cd bgan/\n$ python3 bgan.py\n```\n\n### CC-GAN\nImplementation of _Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks_.\n\n[Code](ccgan/ccgan.py)\n\nPaper: https://arxiv.org/abs/1611.06430\n\n#### Example\n```\n$ cd ccgan/\n$ python3 ccgan.py\n```\n\n

\n \n

\n\n### CGAN\nImplementation of _Conditional Generative Adversarial Nets_.\n\n[Code](cgan/cgan.py)\n\nPaper:https://arxiv.org/abs/1411.1784\n\n#### Example\n```\n$ cd cgan/\n$ python3 cgan.py\n```\n\n

\n \n

\n\n### Context Encoder\nImplementation of _Context Encoders: Feature Learning by Inpainting_.\n\n[Code](context_encoder/context_encoder.py)\n\nPaper: https://arxiv.org/abs/1604.07379\n\n#### Example\n```\n$ cd context_encoder/\n$ python3 context_encoder.py\n```\n\n

\n \n

\n\n### CoGAN\nImplementation of _Coupled generative adversarial networks_.\n\n[Code](cogan/cogan.py)\n\nPaper: https://arxiv.org/abs/1606.07536\n\n#### Example\n```\n$ cd cogan/\n$ python3 cogan.py\n```\n\n### CycleGAN\nImplementation of _Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks_.\n\n[Code](cyclegan/cyclegan.py)\n\nPaper: https://arxiv.org/abs/1703.10593\n\n

\n \n

\n\n#### Example\n```\n$ cd cyclegan/\n$ bash download_dataset.sh apple2orange\n$ python3 cyclegan.py\n``` \n\n

\n \n

\n\n\n### DCGAN\nImplementation of _Deep Convolutional Generative Adversarial Network_.\n\n[Code](dcgan/dcgan.py)\n\nPaper: https://arxiv.org/abs/1511.06434\n\n#### Example\n```\n$ cd dcgan/\n$ python3 dcgan.py\n```\n\n

\n \n

\n\n### DiscoGAN\nImplementation of _Learning to Discover Cross-Domain Relations with Generative Adversarial Networks_.\n\n[Code](discogan/discogan.py)\n\nPaper: https://arxiv.org/abs/1703.05192\n\n

\n \n

\n\n#### Example\n```\n$ cd discogan/\n$ bash download_dataset.sh edges2shoes\n$ python3 discogan.py\n``` \n\n

\n \n

\n\n### DualGAN\nImplementation of _DualGAN: Unsupervised Dual Learning for Image-to-Image Translation_.\n\n[Code](dualgan/dualgan.py)\n\nPaper: https://arxiv.org/abs/1704.02510\n\n#### Example\n```\n$ cd dualgan/\n$ python3 dualgan.py\n```\n\n### GAN\nImplementation of _Generative Adversarial Network_ with a MLP generator and discriminator.\n\n[Code](gan/gan.py)\n\nPaper: https://arxiv.org/abs/1406.2661\n\n#### Example\n```\n$ cd gan/\n$ python3 gan.py\n```\n\n

\n \n

\n\n### InfoGAN\nImplementation of _InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets_.\n\n[Code](infogan/infogan.py)\n\nPaper: https://arxiv.org/abs/1606.03657\n\n#### Example\n```\n$ cd infogan/\n$ python3 infogan.py\n```\n\n

\n \n

\n\n### LSGAN\nImplementation of _Least Squares Generative Adversarial Networks_.\n\n[Code](lsgan/lsgan.py)\n\nPaper: https://arxiv.org/abs/1611.04076\n\n#### Example\n```\n$ cd lsgan/\n$ python3 lsgan.py\n```\n\n### Pix2Pix\nImplementation of _Image-to-Image Translation with Conditional Adversarial Networks_.\n\n[Code](pix2pix/pix2pix.py)\n\nPaper: https://arxiv.org/abs/1611.07004\n\n

\n \n

\n\n#### Example\n```\n$ cd pix2pix/\n$ bash download_dataset.sh facades\n$ python3 pix2pix.py\n``` \n\n

\n \n

\n\n### PixelDA\nImplementation of _Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks_.\n\n[Code](pixelda/pixelda.py)\n\nPaper: https://arxiv.org/abs/1612.05424\n\n#### MNIST to MNIST-M Classification\nTrains a classifier on MNIST images that are translated to resemble MNIST-M (by performing unsupervised image-to-image domain adaptation). This model is compared to the naive solution of training a classifier on MNIST and evaluating it on MNIST-M. The naive model manages a 55% classification accuracy on MNIST-M while the one trained during domain adaptation gets a 95% classification accuracy.\n\n```\n$ cd pixelda/\n$ python3 pixelda.py\n```\n\n| Method | Accuracy |\n| ------------ |:---------:|\n| Naive | 55% |\n| PixelDA | 95% |\n\n### SGAN\nImplementation of _Semi-Supervised Generative Adversarial Network_.\n\n[Code](sgan/sgan.py)\n\nPaper: https://arxiv.org/abs/1606.01583\n\n#### Example\n```\n$ cd sgan/\n$ python3 sgan.py\n```\n\n

\n \n

\n\n### SRGAN\nImplementation of _Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network_.\n\n[Code](srgan/srgan.py)\n\nPaper: https://arxiv.org/abs/1609.04802\n\n

\n \n

\n\n\n#### Example\n```\n$ cd srgan/\n\n$ python3 srgan.py\n```\n\n

\n \n

\n\n### WGAN\nImplementation of _Wasserstein GAN_ (with DCGAN generator and discriminator).\n\n[Code](wgan/wgan.py)\n\nPaper: https://arxiv.org/abs/1701.07875\n\n#### Example\n```\n$ cd wgan/\n$ python3 wgan.py\n```\n\n

\n \n

\n\n### WGAN GP\nImplementation of _Improved Training of Wasserstein GANs_.\n\n[Code](wgan_gp/wgan_gp.py)\n\nPaper: https://arxiv.org/abs/1704.00028\n\n#### Example\n```\n$ cd wgan_gp/\n$ python3 wgan_gp.py\n```\n\n

\n \n

\n" }, { "path": "aae/aae.py", "content": "from __future__ import print_function, division\r\n\r\nfrom keras.datasets import mnist\r\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, multiply, GaussianNoise\r\nfrom keras.layers import BatchNormalization, Activation, Embedding, ZeroPadding2D\r\nfrom keras.layers import MaxPooling2D, merge\r\nfrom keras.layers.advanced_activations import LeakyReLU\r\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\r\nfrom keras.models import Sequential, Model\r\nfrom keras.optimizers import Adam\r\nfrom keras import losses\r\nfrom keras.utils import to_categorical\r\nimport keras.backend as K\r\n\r\nimport matplotlib.pyplot as plt\r\n\r\nimport numpy as np\r\n\r\nclass AdversarialAutoencoder():\r\n def __init__(self):\r\n self.img_rows = 28\r\n self.img_cols = 28\r\n self.channels = 1\r\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\r\n self.latent_dim = 10\r\n\r\n optimizer = Adam(0.0002, 0.5)\r\n\r\n # Build and compile the discriminator\r\n self.discriminator = self.build_discriminator()\r\n self.discriminator.compile(loss='binary_crossentropy',\r\n optimizer=optimizer,\r\n metrics=['accuracy'])\r\n\r\n # Build the encoder / decoder\r\n self.encoder = self.build_encoder()\r\n self.decoder = self.build_decoder()\r\n\r\n img = Input(shape=self.img_shape)\r\n # The generator takes the image, encodes it and reconstructs it\r\n # from the encoding\r\n encoded_repr = self.encoder(img)\r\n reconstructed_img = self.decoder(encoded_repr)\r\n\r\n # For the adversarial_autoencoder model we will only train the generator\r\n self.discriminator.trainable = False\r\n\r\n # The discriminator determines validity of the encoding\r\n validity = self.discriminator(encoded_repr)\r\n\r\n # The adversarial_autoencoder model (stacked generator and discriminator)\r\n self.adversarial_autoencoder = Model(img, [reconstructed_img, validity])\r\n self.adversarial_autoencoder.compile(loss=['mse', 'binary_crossentropy'],\r\n loss_weights=[0.999, 0.001],\r\n optimizer=optimizer)\r\n\r\n\r\n def build_encoder(self):\r\n # Encoder\r\n\r\n img = Input(shape=self.img_shape)\r\n\r\n h = Flatten()(img)\r\n h = Dense(512)(h)\r\n h = LeakyReLU(alpha=0.2)(h)\r\n h = Dense(512)(h)\r\n h = LeakyReLU(alpha=0.2)(h)\r\n mu = Dense(self.latent_dim)(h)\r\n log_var = Dense(self.latent_dim)(h)\r\n latent_repr = merge([mu, log_var],\r\n mode=lambda p: p[0] + K.random_normal(K.shape(p[0])) * K.exp(p[1] / 2),\r\n output_shape=lambda p: p[0])\r\n\r\n return Model(img, latent_repr)\r\n\r\n def build_decoder(self):\r\n\r\n model = Sequential()\r\n\r\n model.add(Dense(512, input_dim=self.latent_dim))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dense(512))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dense(np.prod(self.img_shape), activation='tanh'))\r\n model.add(Reshape(self.img_shape))\r\n\r\n model.summary()\r\n\r\n z = Input(shape=(self.latent_dim,))\r\n img = model(z)\r\n\r\n return Model(z, img)\r\n\r\n def build_discriminator(self):\r\n\r\n model = Sequential()\r\n\r\n model.add(Dense(512, input_dim=self.latent_dim))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dense(256))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dense(1, activation=\"sigmoid\"))\r\n model.summary()\r\n\r\n encoded_repr = Input(shape=(self.latent_dim, ))\r\n validity = model(encoded_repr)\r\n\r\n return Model(encoded_repr, validity)\r\n\r\n def train(self, epochs, batch_size=128, sample_interval=50):\r\n\r\n # Load the dataset\r\n (X_train, _), (_, _) = mnist.load_data()\r\n\r\n # Rescale -1 to 1\r\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\r\n X_train = np.expand_dims(X_train, axis=3)\r\n\r\n # Adversarial ground truths\r\n valid = np.ones((batch_size, 1))\r\n fake = np.zeros((batch_size, 1))\r\n\r\n for epoch in range(epochs):\r\n\r\n # ---------------------\r\n # Train Discriminator\r\n # ---------------------\r\n\r\n # Select a random batch of images\r\n idx = np.random.randint(0, X_train.shape[0], batch_size)\r\n imgs = X_train[idx]\r\n\r\n latent_fake = self.encoder.predict(imgs)\r\n latent_real = np.random.normal(size=(batch_size, self.latent_dim))\r\n\r\n # Train the discriminator\r\n d_loss_real = self.discriminator.train_on_batch(latent_real, valid)\r\n d_loss_fake = self.discriminator.train_on_batch(latent_fake, fake)\r\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\r\n\r\n # ---------------------\r\n # Train Generator\r\n # ---------------------\r\n\r\n # Train the generator\r\n g_loss = self.adversarial_autoencoder.train_on_batch(imgs, [imgs, valid])\r\n\r\n # Plot the progress\r\n print (\"%d [D loss: %f, acc: %.2f%%] [G loss: %f, mse: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss[0], g_loss[1]))\r\n\r\n # If at save interval => save generated image samples\r\n if epoch % sample_interval == 0:\r\n self.sample_images(epoch)\r\n\r\n def sample_images(self, epoch):\r\n r, c = 5, 5\r\n\r\n z = np.random.normal(size=(r*c, self.latent_dim))\r\n gen_imgs = self.decoder.predict(z)\r\n\r\n gen_imgs = 0.5 * gen_imgs + 0.5\r\n\r\n fig, axs = plt.subplots(r, c)\r\n cnt = 0\r\n for i in range(r):\r\n for j in range(c):\r\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\r\n axs[i,j].axis('off')\r\n cnt += 1\r\n fig.savefig(\"images/mnist_%d.png\" % epoch)\r\n plt.close()\r\n\r\n def save_model(self):\r\n\r\n def save(model, model_name):\r\n model_path = \"saved_model/%s.json\" % model_name\r\n weights_path = \"saved_model/%s_weights.hdf5\" % model_name\r\n options = {\"file_arch\": model_path,\r\n \"file_weight\": weights_path}\r\n json_string = model.to_json()\r\n open(options['file_arch'], 'w').write(json_string)\r\n model.save_weights(options['file_weight'])\r\n\r\n save(self.generator, \"aae_generator\")\r\n save(self.discriminator, \"aae_discriminator\")\r\n\r\n\r\nif __name__ == '__main__':\r\n aae = AdversarialAutoencoder()\r\n aae.train(epochs=20000, batch_size=32, sample_interval=200)\r\n" }, { "path": "aae/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "aae/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "acgan/acgan.py", "content": "from __future__ import print_function, division\r\n\r\nfrom keras.datasets import mnist\r\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, multiply\r\nfrom keras.layers import BatchNormalization, Activation, Embedding, ZeroPadding2D\r\nfrom keras.layers.advanced_activations import LeakyReLU\r\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\r\nfrom keras.models import Sequential, Model\r\nfrom keras.optimizers import Adam\r\n\r\nimport matplotlib.pyplot as plt\r\n\r\nimport numpy as np\r\n\r\nclass ACGAN():\r\n def __init__(self):\r\n # Input shape\r\n self.img_rows = 28\r\n self.img_cols = 28\r\n self.channels = 1\r\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\r\n self.num_classes = 10\r\n self.latent_dim = 100\r\n\r\n optimizer = Adam(0.0002, 0.5)\r\n losses = ['binary_crossentropy', 'sparse_categorical_crossentropy']\r\n\r\n # Build and compile the discriminator\r\n self.discriminator = self.build_discriminator()\r\n self.discriminator.compile(loss=losses,\r\n optimizer=optimizer,\r\n metrics=['accuracy'])\r\n\r\n # Build the generator\r\n self.generator = self.build_generator()\r\n\r\n # The generator takes noise and the target label as input\r\n # and generates the corresponding digit of that label\r\n noise = Input(shape=(self.latent_dim,))\r\n label = Input(shape=(1,))\r\n img = self.generator([noise, label])\r\n\r\n # For the combined model we will only train the generator\r\n self.discriminator.trainable = False\r\n\r\n # The discriminator takes generated image as input and determines validity\r\n # and the label of that image\r\n valid, target_label = self.discriminator(img)\r\n\r\n # The combined model (stacked generator and discriminator)\r\n # Trains the generator to fool the discriminator\r\n self.combined = Model([noise, label], [valid, target_label])\r\n self.combined.compile(loss=losses,\r\n optimizer=optimizer)\r\n\r\n def build_generator(self):\r\n\r\n model = Sequential()\r\n\r\n model.add(Dense(128 * 7 * 7, activation=\"relu\", input_dim=self.latent_dim))\r\n model.add(Reshape((7, 7, 128)))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(UpSampling2D())\r\n model.add(Conv2D(128, kernel_size=3, padding=\"same\"))\r\n model.add(Activation(\"relu\"))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(UpSampling2D())\r\n model.add(Conv2D(64, kernel_size=3, padding=\"same\"))\r\n model.add(Activation(\"relu\"))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(self.channels, kernel_size=3, padding='same'))\r\n model.add(Activation(\"tanh\"))\r\n\r\n model.summary()\r\n\r\n noise = Input(shape=(self.latent_dim,))\r\n label = Input(shape=(1,), dtype='int32')\r\n label_embedding = Flatten()(Embedding(self.num_classes, self.latent_dim)(label))\r\n\r\n model_input = multiply([noise, label_embedding])\r\n img = model(model_input)\r\n\r\n return Model([noise, label], img)\r\n\r\n def build_discriminator(self):\r\n\r\n model = Sequential()\r\n\r\n model.add(Conv2D(16, kernel_size=3, strides=2, input_shape=self.img_shape, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.25))\r\n model.add(Conv2D(32, kernel_size=3, strides=2, padding=\"same\"))\r\n model.add(ZeroPadding2D(padding=((0,1),(0,1))))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.25))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.25))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(128, kernel_size=3, strides=1, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.25))\r\n\r\n model.add(Flatten())\r\n model.summary()\r\n\r\n img = Input(shape=self.img_shape)\r\n\r\n # Extract feature representation\r\n features = model(img)\r\n\r\n # Determine validity and label of the image\r\n validity = Dense(1, activation=\"sigmoid\")(features)\r\n label = Dense(self.num_classes, activation=\"softmax\")(features)\r\n\r\n return Model(img, [validity, label])\r\n\r\n def train(self, epochs, batch_size=128, sample_interval=50):\r\n\r\n # Load the dataset\r\n (X_train, y_train), (_, _) = mnist.load_data()\r\n\r\n # Configure inputs\r\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\r\n X_train = np.expand_dims(X_train, axis=3)\r\n y_train = y_train.reshape(-1, 1)\r\n\r\n # Adversarial ground truths\r\n valid = np.ones((batch_size, 1))\r\n fake = np.zeros((batch_size, 1))\r\n\r\n for epoch in range(epochs):\r\n\r\n # ---------------------\r\n # Train Discriminator\r\n # ---------------------\r\n\r\n # Select a random batch of images\r\n idx = np.random.randint(0, X_train.shape[0], batch_size)\r\n imgs = X_train[idx]\r\n\r\n # Sample noise as generator input\r\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\r\n\r\n # The labels of the digits that the generator tries to create an\r\n # image representation of\r\n sampled_labels = np.random.randint(0, 10, (batch_size, 1))\r\n\r\n # Generate a half batch of new images\r\n gen_imgs = self.generator.predict([noise, sampled_labels])\r\n\r\n # Image labels. 0-9 \r\n img_labels = y_train[idx]\r\n\r\n # Train the discriminator\r\n d_loss_real = self.discriminator.train_on_batch(imgs, [valid, img_labels])\r\n d_loss_fake = self.discriminator.train_on_batch(gen_imgs, [fake, sampled_labels])\r\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\r\n\r\n # ---------------------\r\n # Train Generator\r\n # ---------------------\r\n\r\n # Train the generator\r\n g_loss = self.combined.train_on_batch([noise, sampled_labels], [valid, sampled_labels])\r\n\r\n # Plot the progress\r\n print (\"%d [D loss: %f, acc.: %.2f%%, op_acc: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[3], 100*d_loss[4], g_loss[0]))\r\n\r\n # If at save interval => save generated image samples\r\n if epoch % sample_interval == 0:\r\n self.save_model()\r\n self.sample_images(epoch)\r\n\r\n def sample_images(self, epoch):\r\n r, c = 10, 10\r\n noise = np.random.normal(0, 1, (r * c, self.latent_dim))\r\n sampled_labels = np.array([num for _ in range(r) for num in range(c)])\r\n gen_imgs = self.generator.predict([noise, sampled_labels])\r\n # Rescale images 0 - 1\r\n gen_imgs = 0.5 * gen_imgs + 0.5\r\n\r\n fig, axs = plt.subplots(r, c)\r\n cnt = 0\r\n for i in range(r):\r\n for j in range(c):\r\n axs[i,j].imshow(gen_imgs[cnt,:,:,0], cmap='gray')\r\n axs[i,j].axis('off')\r\n cnt += 1\r\n fig.savefig(\"images/%d.png\" % epoch)\r\n plt.close()\r\n\r\n def save_model(self):\r\n\r\n def save(model, model_name):\r\n model_path = \"saved_model/%s.json\" % model_name\r\n weights_path = \"saved_model/%s_weights.hdf5\" % model_name\r\n options = {\"file_arch\": model_path,\r\n \"file_weight\": weights_path}\r\n json_string = model.to_json()\r\n open(options['file_arch'], 'w').write(json_string)\r\n model.save_weights(options['file_weight'])\r\n\r\n save(self.generator, \"generator\")\r\n save(self.discriminator, \"discriminator\")\r\n\r\n\r\nif __name__ == '__main__':\r\n acgan = ACGAN()\r\n acgan.train(epochs=14000, batch_size=32, sample_interval=200)\r\n" }, { "path": "acgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "acgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "bgan/bgan.py", "content": "from __future__ import print_function, division\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nimport keras.backend as K\n\nimport matplotlib.pyplot as plt\n\nimport sys\n\nimport numpy as np\n\nclass BGAN():\n \"\"\"Reference: https://wiseodd.github.io/techblog/2017/03/07/boundary-seeking-gan/\"\"\"\n def __init__(self):\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.latent_dim = 100\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminator\n self.discriminator = self.build_discriminator()\n self.discriminator.compile(loss='binary_crossentropy',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Build the generator\n self.generator = self.build_generator()\n\n # The generator takes noise as input and generated imgs\n z = Input(shape=(self.latent_dim,))\n img = self.generator(z)\n\n # For the combined model we will only train the generator\n self.discriminator.trainable = False\n\n # The valid takes generated images as input and determines validity\n valid = self.discriminator(img)\n\n # The combined model (stacked generator and discriminator)\n # Trains the generator to fool the discriminator\n self.combined = Model(z, valid)\n self.combined.compile(loss=self.boundary_loss, optimizer=optimizer)\n\n def build_generator(self):\n\n model = Sequential()\n\n model.add(Dense(256, input_dim=self.latent_dim))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(1024))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(np.prod(self.img_shape), activation='tanh'))\n model.add(Reshape(self.img_shape))\n\n model.summary()\n\n noise = Input(shape=(self.latent_dim,))\n img = model(noise)\n\n return Model(noise, img)\n\n def build_discriminator(self):\n\n model = Sequential()\n\n model.add(Flatten(input_shape=self.img_shape))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dense(256))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dense(1, activation='sigmoid'))\n model.summary()\n\n img = Input(shape=self.img_shape)\n validity = model(img)\n\n return Model(img, validity)\n\n def boundary_loss(self, y_true, y_pred):\n \"\"\"\n Boundary seeking loss.\n Reference: https://wiseodd.github.io/techblog/2017/03/07/boundary-seeking-gan/\n \"\"\"\n return 0.5 * K.mean((K.log(y_pred) - K.log(1 - y_pred))**2)\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n # Load the dataset\n (X_train, _), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = X_train / 127.5 - 1.\n X_train = np.expand_dims(X_train, axis=3)\n\n # Adversarial ground truths\n valid = np.ones((batch_size, 1))\n fake = np.zeros((batch_size, 1))\n\n for epoch in range(epochs):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n # Select a random batch of images\n idx = np.random.randint(0, X_train.shape[0], batch_size)\n imgs = X_train[idx]\n\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n\n # Generate a batch of new images\n gen_imgs = self.generator.predict(noise)\n\n # Train the discriminator\n d_loss_real = self.discriminator.train_on_batch(imgs, valid)\n d_loss_fake = self.discriminator.train_on_batch(gen_imgs, fake)\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n\n\n # ---------------------\n # Train Generator\n # ---------------------\n\n g_loss = self.combined.train_on_batch(noise, valid)\n\n # Plot the progress\n print (\"%d [D loss: %f, acc.: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n r, c = 5, 5\n noise = np.random.normal(0, 1, (r * c, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/mnist_%d.png\" % epoch)\n plt.close()\n\n\nif __name__ == '__main__':\n bgan = BGAN()\n bgan.train(epochs=30000, batch_size=32, sample_interval=200)\n" }, { "path": "bgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "bgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "bigan/bigan.py", "content": "from __future__ import print_function, division\r\n\r\nfrom keras.datasets import mnist\r\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, multiply, GaussianNoise\r\nfrom keras.layers import BatchNormalization, Activation, Embedding, ZeroPadding2D\r\nfrom keras.layers import MaxPooling2D, concatenate\r\nfrom keras.layers.advanced_activations import LeakyReLU\r\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\r\nfrom keras.models import Sequential, Model\r\nfrom keras.optimizers import Adam\r\nfrom keras import losses\r\nfrom keras.utils import to_categorical\r\nimport keras.backend as K\r\n\r\nimport matplotlib.pyplot as plt\r\n\r\nimport numpy as np\r\n\r\nclass BIGAN():\r\n def __init__(self):\r\n self.img_rows = 28\r\n self.img_cols = 28\r\n self.channels = 1\r\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\r\n self.latent_dim = 100\r\n\r\n optimizer = Adam(0.0002, 0.5)\r\n\r\n # Build and compile the discriminator\r\n self.discriminator = self.build_discriminator()\r\n self.discriminator.compile(loss=['binary_crossentropy'],\r\n optimizer=optimizer,\r\n metrics=['accuracy'])\r\n\r\n # Build the generator\r\n self.generator = self.build_generator()\r\n\r\n # Build the encoder\r\n self.encoder = self.build_encoder()\r\n\r\n # The part of the bigan that trains the discriminator and encoder\r\n self.discriminator.trainable = False\r\n\r\n # Generate image from sampled noise\r\n z = Input(shape=(self.latent_dim, ))\r\n img_ = self.generator(z)\r\n\r\n # Encode image\r\n img = Input(shape=self.img_shape)\r\n z_ = self.encoder(img)\r\n\r\n # Latent -> img is fake, and img -> latent is valid\r\n fake = self.discriminator([z, img_])\r\n valid = self.discriminator([z_, img])\r\n\r\n # Set up and compile the combined model\r\n # Trains generator to fool the discriminator\r\n self.bigan_generator = Model([z, img], [fake, valid])\r\n self.bigan_generator.compile(loss=['binary_crossentropy', 'binary_crossentropy'],\r\n optimizer=optimizer)\r\n\r\n\r\n def build_encoder(self):\r\n model = Sequential()\r\n\r\n model.add(Flatten(input_shape=self.img_shape))\r\n model.add(Dense(512))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Dense(512))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Dense(self.latent_dim))\r\n\r\n model.summary()\r\n\r\n img = Input(shape=self.img_shape)\r\n z = model(img)\r\n\r\n return Model(img, z)\r\n\r\n def build_generator(self):\r\n model = Sequential()\r\n\r\n model.add(Dense(512, input_dim=self.latent_dim))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Dense(512))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Dense(np.prod(self.img_shape), activation='tanh'))\r\n model.add(Reshape(self.img_shape))\r\n\r\n model.summary()\r\n\r\n z = Input(shape=(self.latent_dim,))\r\n gen_img = model(z)\r\n\r\n return Model(z, gen_img)\r\n\r\n def build_discriminator(self):\r\n\r\n z = Input(shape=(self.latent_dim, ))\r\n img = Input(shape=self.img_shape)\r\n d_in = concatenate([z, Flatten()(img)])\r\n\r\n model = Dense(1024)(d_in)\r\n model = LeakyReLU(alpha=0.2)(model)\r\n model = Dropout(0.5)(model)\r\n model = Dense(1024)(model)\r\n model = LeakyReLU(alpha=0.2)(model)\r\n model = Dropout(0.5)(model)\r\n model = Dense(1024)(model)\r\n model = LeakyReLU(alpha=0.2)(model)\r\n model = Dropout(0.5)(model)\r\n validity = Dense(1, activation=\"sigmoid\")(model)\r\n\r\n return Model([z, img], validity)\r\n\r\n def train(self, epochs, batch_size=128, sample_interval=50):\r\n\r\n # Load the dataset\r\n (X_train, _), (_, _) = mnist.load_data()\r\n\r\n # Rescale -1 to 1\r\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\r\n X_train = np.expand_dims(X_train, axis=3)\r\n\r\n # Adversarial ground truths\r\n valid = np.ones((batch_size, 1))\r\n fake = np.zeros((batch_size, 1))\r\n\r\n for epoch in range(epochs):\r\n\r\n\r\n # ---------------------\r\n # Train Discriminator\r\n # ---------------------\r\n\r\n # Sample noise and generate img\r\n z = np.random.normal(size=(batch_size, self.latent_dim))\r\n imgs_ = self.generator.predict(z)\r\n\r\n # Select a random batch of images and encode\r\n idx = np.random.randint(0, X_train.shape[0], batch_size)\r\n imgs = X_train[idx]\r\n z_ = self.encoder.predict(imgs)\r\n\r\n # Train the discriminator (img -> z is valid, z -> img is fake)\r\n d_loss_real = self.discriminator.train_on_batch([z_, imgs], valid)\r\n d_loss_fake = self.discriminator.train_on_batch([z, imgs_], fake)\r\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\r\n\r\n # ---------------------\r\n # Train Generator\r\n # ---------------------\r\n\r\n # Train the generator (z -> img is valid and img -> z is is invalid)\r\n g_loss = self.bigan_generator.train_on_batch([z, imgs], [valid, fake])\r\n\r\n # Plot the progress\r\n print (\"%d [D loss: %f, acc: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss[0]))\r\n\r\n # If at save interval => save generated image samples\r\n if epoch % sample_interval == 0:\r\n self.sample_interval(epoch)\r\n\r\n def sample_interval(self, epoch):\r\n r, c = 5, 5\r\n z = np.random.normal(size=(25, self.latent_dim))\r\n gen_imgs = self.generator.predict(z)\r\n\r\n gen_imgs = 0.5 * gen_imgs + 0.5\r\n\r\n fig, axs = plt.subplots(r, c)\r\n cnt = 0\r\n for i in range(r):\r\n for j in range(c):\r\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\r\n axs[i,j].axis('off')\r\n cnt += 1\r\n fig.savefig(\"images/mnist_%d.png\" % epoch)\r\n plt.close()\r\n\r\n\r\nif __name__ == '__main__':\r\n bigan = BIGAN()\r\n bigan.train(epochs=40000, batch_size=32, sample_interval=400)\r\n" }, { "path": "bigan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "bigan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "ccgan/ccgan.py", "content": "from __future__ import print_function, division\r\n\r\nfrom keras.datasets import mnist\r\nfrom keras_contrib.layers.normalization.instancenormalization import InstanceNormalization\r\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, multiply, GaussianNoise\r\nfrom keras.layers import BatchNormalization, Activation, Embedding, ZeroPadding2D\r\nfrom keras.layers import Concatenate\r\nfrom keras.layers.advanced_activations import LeakyReLU\r\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\r\nfrom keras.models import Sequential, Model\r\nfrom keras.optimizers import Adam\r\nfrom keras import losses\r\nfrom keras.utils import to_categorical\r\nimport keras.backend as K\r\nimport scipy\r\n\r\nimport matplotlib.pyplot as plt\r\n\r\nimport numpy as np\r\n\r\nclass CCGAN():\r\n def __init__(self):\r\n self.img_rows = 32\r\n self.img_cols = 32\r\n self.channels = 1\r\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\r\n self.mask_height = 10\r\n self.mask_width = 10\r\n self.num_classes = 10\r\n\r\n # Number of filters in first layer of generator and discriminator\r\n self.gf = 32\r\n self.df = 32\r\n\r\n optimizer = Adam(0.0002, 0.5)\r\n\r\n # Build and compile the discriminator\r\n self.discriminator = self.build_discriminator()\r\n self.discriminator.compile(loss=['mse', 'categorical_crossentropy'],\r\n loss_weights=[0.5, 0.5],\r\n optimizer=optimizer,\r\n metrics=['accuracy'])\r\n\r\n # Build the generator\r\n self.generator = self.build_generator()\r\n\r\n # The generator takes noise as input and generates imgs\r\n masked_img = Input(shape=self.img_shape)\r\n gen_img = self.generator(masked_img)\r\n\r\n # For the combined model we will only train the generator\r\n self.discriminator.trainable = False\r\n\r\n # The valid takes generated images as input and determines validity\r\n valid, _ = self.discriminator(gen_img)\r\n\r\n # The combined model (stacked generator and discriminator)\r\n # Trains the generator to fool the discriminator\r\n self.combined = Model(masked_img , valid)\r\n self.combined.compile(loss=['mse'],\r\n optimizer=optimizer)\r\n\r\n\r\n def build_generator(self):\r\n \"\"\"U-Net Generator\"\"\"\r\n\r\n def conv2d(layer_input, filters, f_size=4, bn=True):\r\n \"\"\"Layers used during downsampling\"\"\"\r\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\r\n d = LeakyReLU(alpha=0.2)(d)\r\n if bn:\r\n d = BatchNormalization(momentum=0.8)(d)\r\n return d\r\n\r\n def deconv2d(layer_input, skip_input, filters, f_size=4, dropout_rate=0):\r\n \"\"\"Layers used during upsampling\"\"\"\r\n u = UpSampling2D(size=2)(layer_input)\r\n u = Conv2D(filters, kernel_size=f_size, strides=1, padding='same', activation='relu')(u)\r\n if dropout_rate:\r\n u = Dropout(dropout_rate)(u)\r\n u = BatchNormalization(momentum=0.8)(u)\r\n u = Concatenate()([u, skip_input])\r\n return u\r\n\r\n img = Input(shape=self.img_shape)\r\n\r\n # Downsampling\r\n d1 = conv2d(img, self.gf, bn=False)\r\n d2 = conv2d(d1, self.gf*2)\r\n d3 = conv2d(d2, self.gf*4)\r\n d4 = conv2d(d3, self.gf*8)\r\n\r\n # Upsampling\r\n u1 = deconv2d(d4, d3, self.gf*4)\r\n u2 = deconv2d(u1, d2, self.gf*2)\r\n u3 = deconv2d(u2, d1, self.gf)\r\n\r\n u4 = UpSampling2D(size=2)(u3)\r\n output_img = Conv2D(self.channels, kernel_size=4, strides=1, padding='same', activation='tanh')(u4)\r\n\r\n return Model(img, output_img)\r\n\r\n def build_discriminator(self):\r\n\r\n img = Input(shape=self.img_shape)\r\n\r\n model = Sequential()\r\n model.add(Conv2D(64, kernel_size=4, strides=2, padding='same', input_shape=self.img_shape))\r\n model.add(LeakyReLU(alpha=0.8))\r\n model.add(Conv2D(128, kernel_size=4, strides=2, padding='same'))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(InstanceNormalization())\r\n model.add(Conv2D(256, kernel_size=4, strides=2, padding='same'))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(InstanceNormalization())\r\n\r\n model.summary()\r\n\r\n img = Input(shape=self.img_shape)\r\n features = model(img)\r\n\r\n validity = Conv2D(1, kernel_size=4, strides=1, padding='same')(features)\r\n\r\n label = Flatten()(features)\r\n label = Dense(self.num_classes+1, activation=\"softmax\")(label)\r\n\r\n return Model(img, [validity, label])\r\n\r\n def mask_randomly(self, imgs):\r\n y1 = np.random.randint(0, self.img_rows - self.mask_height, imgs.shape[0])\r\n y2 = y1 + self.mask_height\r\n x1 = np.random.randint(0, self.img_rows - self.mask_width, imgs.shape[0])\r\n x2 = x1 + self.mask_width\r\n\r\n masked_imgs = np.empty_like(imgs)\r\n for i, img in enumerate(imgs):\r\n masked_img = img.copy()\r\n _y1, _y2, _x1, _x2 = y1[i], y2[i], x1[i], x2[i],\r\n masked_img[_y1:_y2, _x1:_x2, :] = 0\r\n masked_imgs[i] = masked_img\r\n\r\n return masked_imgs\r\n\r\n\r\n def train(self, epochs, batch_size=128, sample_interval=50):\r\n\r\n # Load the dataset\r\n (X_train, y_train), (_, _) = mnist.load_data()\r\n\r\n # Rescale MNIST to 32x32\r\n X_train = np.array([scipy.misc.imresize(x, [self.img_rows, self.img_cols]) for x in X_train])\r\n\r\n # Rescale -1 to 1\r\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\r\n X_train = np.expand_dims(X_train, axis=3)\r\n y_train = y_train.reshape(-1, 1)\r\n\r\n # Adversarial ground truths\r\n valid = np.ones((batch_size, 4, 4, 1))\r\n fake = np.zeros((batch_size, 4, 4, 1))\r\n\r\n for epoch in range(epochs):\r\n\r\n # ---------------------\r\n # Train Discriminator\r\n # ---------------------\r\n\r\n # Sample half batch of images\r\n idx = np.random.randint(0, X_train.shape[0], batch_size)\r\n imgs = X_train[idx]\r\n labels = y_train[idx]\r\n\r\n masked_imgs = self.mask_randomly(imgs)\r\n\r\n # Generate a half batch of new images\r\n gen_imgs = self.generator.predict(masked_imgs)\r\n\r\n # One-hot encoding of labels\r\n labels = to_categorical(labels, num_classes=self.num_classes+1)\r\n fake_labels = to_categorical(np.full((batch_size, 1), self.num_classes), num_classes=self.num_classes+1)\r\n\r\n # Train the discriminator\r\n d_loss_real = self.discriminator.train_on_batch(imgs, [valid, labels])\r\n d_loss_fake = self.discriminator.train_on_batch(gen_imgs, [fake, fake_labels])\r\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\r\n\r\n # ---------------------\r\n # Train Generator\r\n # ---------------------\r\n\r\n # Train the generator\r\n g_loss = self.combined.train_on_batch(masked_imgs, valid)\r\n\r\n # Plot the progress\r\n print (\"%d [D loss: %f, op_acc: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[4], g_loss))\r\n\r\n # If at save interval => save generated image samples\r\n if epoch % sample_interval == 0:\r\n # Select a random half batch of images\r\n idx = np.random.randint(0, X_train.shape[0], 6)\r\n imgs = X_train[idx]\r\n self.sample_images(epoch, imgs)\r\n self.save_model()\r\n\r\n def sample_images(self, epoch, imgs):\r\n r, c = 3, 6\r\n\r\n masked_imgs = self.mask_randomly(imgs)\r\n gen_imgs = self.generator.predict(masked_imgs)\r\n\r\n imgs = (imgs + 1.0) * 0.5\r\n masked_imgs = (masked_imgs + 1.0) * 0.5\r\n gen_imgs = (gen_imgs + 1.0) * 0.5\r\n\r\n gen_imgs = np.where(gen_imgs < 0, 0, gen_imgs)\r\n\r\n fig, axs = plt.subplots(r, c)\r\n for i in range(c):\r\n axs[0,i].imshow(imgs[i, :, :, 0], cmap='gray')\r\n axs[0,i].axis('off')\r\n axs[1,i].imshow(masked_imgs[i, :, :, 0], cmap='gray')\r\n axs[1,i].axis('off')\r\n axs[2,i].imshow(gen_imgs[i, :, :, 0], cmap='gray')\r\n axs[2,i].axis('off')\r\n fig.savefig(\"images/%d.png\" % epoch)\r\n plt.close()\r\n\r\n def save_model(self):\r\n\r\n def save(model, model_name):\r\n model_path = \"saved_model/%s.json\" % model_name\r\n weights_path = \"saved_model/%s_weights.hdf5\" % model_name\r\n options = {\"file_arch\": model_path,\r\n \"file_weight\": weights_path}\r\n json_string = model.to_json()\r\n open(options['file_arch'], 'w').write(json_string)\r\n model.save_weights(options['file_weight'])\r\n\r\n save(self.generator, \"ccgan_generator\")\r\n save(self.discriminator, \"ccgan_discriminator\")\r\n\r\n\r\nif __name__ == '__main__':\r\n ccgan = CCGAN()\r\n ccgan.train(epochs=20000, batch_size=32, sample_interval=200)\r\n" }, { "path": "ccgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "ccgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "cgan/cgan.py", "content": "from __future__ import print_function, division\r\n\r\nfrom keras.datasets import mnist\r\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, multiply\r\nfrom keras.layers import BatchNormalization, Activation, Embedding, ZeroPadding2D\r\nfrom keras.layers.advanced_activations import LeakyReLU\r\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\r\nfrom keras.models import Sequential, Model\r\nfrom keras.optimizers import Adam\r\n\r\nimport matplotlib.pyplot as plt\r\n\r\nimport numpy as np\r\n\r\nclass CGAN():\r\n def __init__(self):\r\n # Input shape\r\n self.img_rows = 28\r\n self.img_cols = 28\r\n self.channels = 1\r\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\r\n self.num_classes = 10\r\n self.latent_dim = 100\r\n\r\n optimizer = Adam(0.0002, 0.5)\r\n\r\n # Build and compile the discriminator\r\n self.discriminator = self.build_discriminator()\r\n self.discriminator.compile(loss=['binary_crossentropy'],\r\n optimizer=optimizer,\r\n metrics=['accuracy'])\r\n\r\n # Build the generator\r\n self.generator = self.build_generator()\r\n\r\n # The generator takes noise and the target label as input\r\n # and generates the corresponding digit of that label\r\n noise = Input(shape=(self.latent_dim,))\r\n label = Input(shape=(1,))\r\n img = self.generator([noise, label])\r\n\r\n # For the combined model we will only train the generator\r\n self.discriminator.trainable = False\r\n\r\n # The discriminator takes generated image as input and determines validity\r\n # and the label of that image\r\n valid = self.discriminator([img, label])\r\n\r\n # The combined model (stacked generator and discriminator)\r\n # Trains generator to fool discriminator\r\n self.combined = Model([noise, label], valid)\r\n self.combined.compile(loss=['binary_crossentropy'],\r\n optimizer=optimizer)\r\n\r\n def build_generator(self):\r\n\r\n model = Sequential()\r\n\r\n model.add(Dense(256, input_dim=self.latent_dim))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Dense(512))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Dense(1024))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Dense(np.prod(self.img_shape), activation='tanh'))\r\n model.add(Reshape(self.img_shape))\r\n\r\n model.summary()\r\n\r\n noise = Input(shape=(self.latent_dim,))\r\n label = Input(shape=(1,), dtype='int32')\r\n label_embedding = Flatten()(Embedding(self.num_classes, self.latent_dim)(label))\r\n\r\n model_input = multiply([noise, label_embedding])\r\n img = model(model_input)\r\n\r\n return Model([noise, label], img)\r\n\r\n def build_discriminator(self):\r\n\r\n model = Sequential()\r\n\r\n model.add(Dense(512, input_dim=np.prod(self.img_shape)))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dense(512))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.4))\r\n model.add(Dense(512))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.4))\r\n model.add(Dense(1, activation='sigmoid'))\r\n model.summary()\r\n\r\n img = Input(shape=self.img_shape)\r\n label = Input(shape=(1,), dtype='int32')\r\n\r\n label_embedding = Flatten()(Embedding(self.num_classes, np.prod(self.img_shape))(label))\r\n flat_img = Flatten()(img)\r\n\r\n model_input = multiply([flat_img, label_embedding])\r\n\r\n validity = model(model_input)\r\n\r\n return Model([img, label], validity)\r\n\r\n def train(self, epochs, batch_size=128, sample_interval=50):\r\n\r\n # Load the dataset\r\n (X_train, y_train), (_, _) = mnist.load_data()\r\n\r\n # Configure input\r\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\r\n X_train = np.expand_dims(X_train, axis=3)\r\n y_train = y_train.reshape(-1, 1)\r\n\r\n # Adversarial ground truths\r\n valid = np.ones((batch_size, 1))\r\n fake = np.zeros((batch_size, 1))\r\n\r\n for epoch in range(epochs):\r\n\r\n # ---------------------\r\n # Train Discriminator\r\n # ---------------------\r\n\r\n # Select a random half batch of images\r\n idx = np.random.randint(0, X_train.shape[0], batch_size)\r\n imgs, labels = X_train[idx], y_train[idx]\r\n\r\n # Sample noise as generator input\r\n noise = np.random.normal(0, 1, (batch_size, 100))\r\n\r\n # Generate a half batch of new images\r\n gen_imgs = self.generator.predict([noise, labels])\r\n\r\n # Train the discriminator\r\n d_loss_real = self.discriminator.train_on_batch([imgs, labels], valid)\r\n d_loss_fake = self.discriminator.train_on_batch([gen_imgs, labels], fake)\r\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\r\n\r\n # ---------------------\r\n # Train Generator\r\n # ---------------------\r\n\r\n # Condition on labels\r\n sampled_labels = np.random.randint(0, 10, batch_size).reshape(-1, 1)\r\n\r\n # Train the generator\r\n g_loss = self.combined.train_on_batch([noise, sampled_labels], valid)\r\n\r\n # Plot the progress\r\n print (\"%d [D loss: %f, acc.: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss))\r\n\r\n # If at save interval => save generated image samples\r\n if epoch % sample_interval == 0:\r\n self.sample_images(epoch)\r\n\r\n def sample_images(self, epoch):\r\n r, c = 2, 5\r\n noise = np.random.normal(0, 1, (r * c, 100))\r\n sampled_labels = np.arange(0, 10).reshape(-1, 1)\r\n\r\n gen_imgs = self.generator.predict([noise, sampled_labels])\r\n\r\n # Rescale images 0 - 1\r\n gen_imgs = 0.5 * gen_imgs + 0.5\r\n\r\n fig, axs = plt.subplots(r, c)\r\n cnt = 0\r\n for i in range(r):\r\n for j in range(c):\r\n axs[i,j].imshow(gen_imgs[cnt,:,:,0], cmap='gray')\r\n axs[i,j].set_title(\"Digit: %d\" % sampled_labels[cnt])\r\n axs[i,j].axis('off')\r\n cnt += 1\r\n fig.savefig(\"images/%d.png\" % epoch)\r\n plt.close()\r\n\r\n\r\nif __name__ == '__main__':\r\n cgan = CGAN()\r\n cgan.train(epochs=20000, batch_size=32, sample_interval=200)\r\n" }, { "path": "cgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "cgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "cogan/cogan.py", "content": "from __future__ import print_function, division\nimport scipy\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\n\nimport matplotlib.pyplot as plt\n\nimport sys\n\nimport numpy as np\n\nclass COGAN():\n \"\"\"Reference: https://wiseodd.github.io/techblog/2017/02/18/coupled_gan/\"\"\"\n def __init__(self):\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.latent_dim = 100\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminator\n self.d1, self.d2 = self.build_discriminators()\n self.d1.compile(loss='binary_crossentropy',\n optimizer=optimizer,\n metrics=['accuracy'])\n self.d2.compile(loss='binary_crossentropy',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Build the generator\n self.g1, self.g2 = self.build_generators()\n\n # The generator takes noise as input and generated imgs\n z = Input(shape=(self.latent_dim,))\n img1 = self.g1(z)\n img2 = self.g2(z)\n\n # For the combined model we will only train the generators\n self.d1.trainable = False\n self.d2.trainable = False\n\n # The valid takes generated images as input and determines validity\n valid1 = self.d1(img1)\n valid2 = self.d2(img2)\n\n # The combined model (stacked generators and discriminators)\n # Trains generators to fool discriminators\n self.combined = Model(z, [valid1, valid2])\n self.combined.compile(loss=['binary_crossentropy', 'binary_crossentropy'],\n optimizer=optimizer)\n\n def build_generators(self):\n\n # Shared weights between generators\n model = Sequential()\n model.add(Dense(256, input_dim=self.latent_dim))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n\n noise = Input(shape=(self.latent_dim,))\n feature_repr = model(noise)\n\n # Generator 1\n g1 = Dense(1024)(feature_repr)\n g1 = LeakyReLU(alpha=0.2)(g1)\n g1 = BatchNormalization(momentum=0.8)(g1)\n g1 = Dense(np.prod(self.img_shape), activation='tanh')(g1)\n img1 = Reshape(self.img_shape)(g1)\n\n # Generator 2\n g2 = Dense(1024)(feature_repr)\n g2 = LeakyReLU(alpha=0.2)(g2)\n g2 = BatchNormalization(momentum=0.8)(g2)\n g2 = Dense(np.prod(self.img_shape), activation='tanh')(g2)\n img2 = Reshape(self.img_shape)(g2)\n\n model.summary()\n\n return Model(noise, img1), Model(noise, img2)\n\n def build_discriminators(self):\n\n img1 = Input(shape=self.img_shape)\n img2 = Input(shape=self.img_shape)\n\n # Shared discriminator layers\n model = Sequential()\n model.add(Flatten(input_shape=self.img_shape))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dense(256))\n model.add(LeakyReLU(alpha=0.2))\n\n img1_embedding = model(img1)\n img2_embedding = model(img2)\n\n # Discriminator 1\n validity1 = Dense(1, activation='sigmoid')(img1_embedding)\n # Discriminator 2\n validity2 = Dense(1, activation='sigmoid')(img2_embedding)\n\n return Model(img1, validity1), Model(img2, validity2)\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n # Load the dataset\n (X_train, _), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\n X_train = np.expand_dims(X_train, axis=3)\n\n # Images in domain A and B (rotated)\n X1 = X_train[:int(X_train.shape[0]/2)]\n X2 = X_train[int(X_train.shape[0]/2):]\n X2 = scipy.ndimage.interpolation.rotate(X2, 90, axes=(1, 2))\n\n # Adversarial ground truths\n valid = np.ones((batch_size, 1))\n fake = np.zeros((batch_size, 1))\n\n for epoch in range(epochs):\n\n # ----------------------\n # Train Discriminators\n # ----------------------\n\n # Select a random batch of images\n idx = np.random.randint(0, X1.shape[0], batch_size)\n imgs1 = X1[idx]\n imgs2 = X2[idx]\n\n # Sample noise as generator input\n noise = np.random.normal(0, 1, (batch_size, 100))\n\n # Generate a batch of new images\n gen_imgs1 = self.g1.predict(noise)\n gen_imgs2 = self.g2.predict(noise)\n\n # Train the discriminators\n d1_loss_real = self.d1.train_on_batch(imgs1, valid)\n d2_loss_real = self.d2.train_on_batch(imgs2, valid)\n d1_loss_fake = self.d1.train_on_batch(gen_imgs1, fake)\n d2_loss_fake = self.d2.train_on_batch(gen_imgs2, fake)\n d1_loss = 0.5 * np.add(d1_loss_real, d1_loss_fake)\n d2_loss = 0.5 * np.add(d2_loss_real, d2_loss_fake)\n\n\n # ------------------\n # Train Generators\n # ------------------\n\n g_loss = self.combined.train_on_batch(noise, [valid, valid])\n\n # Plot the progress\n print (\"%d [D1 loss: %f, acc.: %.2f%%] [D2 loss: %f, acc.: %.2f%%] [G loss: %f]\" \\\n % (epoch, d1_loss[0], 100*d1_loss[1], d2_loss[0], 100*d2_loss[1], g_loss[0]))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n r, c = 4, 4\n noise = np.random.normal(0, 1, (r * int(c/2), 100))\n gen_imgs1 = self.g1.predict(noise)\n gen_imgs2 = self.g2.predict(noise)\n\n gen_imgs = np.concatenate([gen_imgs1, gen_imgs2])\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/mnist_%d.png\" % epoch)\n plt.close()\n\n\nif __name__ == '__main__':\n gan = COGAN()\n gan.train(epochs=30000, batch_size=32, sample_interval=200)\n" }, { "path": "cogan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "cogan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "context_encoder/context_encoder.py", "content": "from __future__ import print_function, division\r\n\r\nfrom keras.datasets import cifar10\r\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, multiply, GaussianNoise\r\nfrom keras.layers import BatchNormalization, Activation, Embedding, ZeroPadding2D\r\nfrom keras.layers import MaxPooling2D\r\nfrom keras.layers.advanced_activations import LeakyReLU\r\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\r\nfrom keras.models import Sequential, Model\r\nfrom keras.optimizers import Adam\r\nfrom keras import losses\r\nfrom keras.utils import to_categorical\r\nimport keras.backend as K\r\n\r\nimport matplotlib.pyplot as plt\r\n\r\nimport numpy as np\r\n\r\nclass ContextEncoder():\r\n def __init__(self):\r\n self.img_rows = 32\r\n self.img_cols = 32\r\n self.mask_height = 8\r\n self.mask_width = 8\r\n self.channels = 3\r\n self.num_classes = 2\r\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\r\n self.missing_shape = (self.mask_height, self.mask_width, self.channels)\r\n\r\n optimizer = Adam(0.0002, 0.5)\r\n\r\n # Build and compile the discriminator\r\n self.discriminator = self.build_discriminator()\r\n self.discriminator.compile(loss='binary_crossentropy',\r\n optimizer=optimizer,\r\n metrics=['accuracy'])\r\n\r\n # Build the generator\r\n self.generator = self.build_generator()\r\n\r\n # The generator takes noise as input and generates the missing\r\n # part of the image\r\n masked_img = Input(shape=self.img_shape)\r\n gen_missing = self.generator(masked_img)\r\n\r\n # For the combined model we will only train the generator\r\n self.discriminator.trainable = False\r\n\r\n # The discriminator takes generated images as input and determines\r\n # if it is generated or if it is a real image\r\n valid = self.discriminator(gen_missing)\r\n\r\n # The combined model (stacked generator and discriminator)\r\n # Trains generator to fool discriminator\r\n self.combined = Model(masked_img , [gen_missing, valid])\r\n self.combined.compile(loss=['mse', 'binary_crossentropy'],\r\n loss_weights=[0.999, 0.001],\r\n optimizer=optimizer)\r\n\r\n def build_generator(self):\r\n\r\n\r\n model = Sequential()\r\n\r\n # Encoder\r\n model.add(Conv2D(32, kernel_size=3, strides=2, input_shape=self.img_shape, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(128, kernel_size=3, strides=2, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n\r\n model.add(Conv2D(512, kernel_size=1, strides=2, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.5))\r\n\r\n # Decoder\r\n model.add(UpSampling2D())\r\n model.add(Conv2D(128, kernel_size=3, padding=\"same\"))\r\n model.add(Activation('relu'))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(UpSampling2D())\r\n model.add(Conv2D(64, kernel_size=3, padding=\"same\"))\r\n model.add(Activation('relu'))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(self.channels, kernel_size=3, padding=\"same\"))\r\n model.add(Activation('tanh'))\r\n\r\n model.summary()\r\n\r\n masked_img = Input(shape=self.img_shape)\r\n gen_missing = model(masked_img)\r\n\r\n return Model(masked_img, gen_missing)\r\n\r\n def build_discriminator(self):\r\n\r\n model = Sequential()\r\n\r\n model.add(Conv2D(64, kernel_size=3, strides=2, input_shape=self.missing_shape, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(128, kernel_size=3, strides=2, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(256, kernel_size=3, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Flatten())\r\n model.add(Dense(1, activation='sigmoid'))\r\n model.summary()\r\n\r\n img = Input(shape=self.missing_shape)\r\n validity = model(img)\r\n\r\n return Model(img, validity)\r\n\r\n def mask_randomly(self, imgs):\r\n y1 = np.random.randint(0, self.img_rows - self.mask_height, imgs.shape[0])\r\n y2 = y1 + self.mask_height\r\n x1 = np.random.randint(0, self.img_rows - self.mask_width, imgs.shape[0])\r\n x2 = x1 + self.mask_width\r\n\r\n masked_imgs = np.empty_like(imgs)\r\n missing_parts = np.empty((imgs.shape[0], self.mask_height, self.mask_width, self.channels))\r\n for i, img in enumerate(imgs):\r\n masked_img = img.copy()\r\n _y1, _y2, _x1, _x2 = y1[i], y2[i], x1[i], x2[i]\r\n missing_parts[i] = masked_img[_y1:_y2, _x1:_x2, :].copy()\r\n masked_img[_y1:_y2, _x1:_x2, :] = 0\r\n masked_imgs[i] = masked_img\r\n\r\n return masked_imgs, missing_parts, (y1, y2, x1, x2)\r\n\r\n\r\n\r\n def train(self, epochs, batch_size=128, sample_interval=50):\r\n\r\n # Load the dataset\r\n (X_train, y_train), (_, _) = cifar10.load_data()\r\n\r\n # Extract dogs and cats\r\n X_cats = X_train[(y_train == 3).flatten()]\r\n X_dogs = X_train[(y_train == 5).flatten()]\r\n X_train = np.vstack((X_cats, X_dogs))\r\n\r\n # Rescale -1 to 1\r\n X_train = X_train / 127.5 - 1.\r\n y_train = y_train.reshape(-1, 1)\r\n\r\n # Adversarial ground truths\r\n valid = np.ones((batch_size, 1))\r\n fake = np.zeros((batch_size, 1))\r\n\r\n for epoch in range(epochs):\r\n\r\n # ---------------------\r\n # Train Discriminator\r\n # ---------------------\r\n\r\n # Select a random batch of images\r\n idx = np.random.randint(0, X_train.shape[0], batch_size)\r\n imgs = X_train[idx]\r\n\r\n masked_imgs, missing_parts, _ = self.mask_randomly(imgs)\r\n\r\n # Generate a batch of new images\r\n gen_missing = self.generator.predict(masked_imgs)\r\n\r\n # Train the discriminator\r\n d_loss_real = self.discriminator.train_on_batch(missing_parts, valid)\r\n d_loss_fake = self.discriminator.train_on_batch(gen_missing, fake)\r\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\r\n\r\n # ---------------------\r\n # Train Generator\r\n # ---------------------\r\n\r\n g_loss = self.combined.train_on_batch(masked_imgs, [missing_parts, valid])\r\n\r\n # Plot the progress\r\n print (\"%d [D loss: %f, acc: %.2f%%] [G loss: %f, mse: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss[0], g_loss[1]))\r\n\r\n # If at save interval => save generated image samples\r\n if epoch % sample_interval == 0:\r\n idx = np.random.randint(0, X_train.shape[0], 6)\r\n imgs = X_train[idx]\r\n self.sample_images(epoch, imgs)\r\n\r\n def sample_images(self, epoch, imgs):\r\n r, c = 3, 6\r\n\r\n masked_imgs, missing_parts, (y1, y2, x1, x2) = self.mask_randomly(imgs)\r\n gen_missing = self.generator.predict(masked_imgs)\r\n\r\n imgs = 0.5 * imgs + 0.5\r\n masked_imgs = 0.5 * masked_imgs + 0.5\r\n gen_missing = 0.5 * gen_missing + 0.5\r\n\r\n fig, axs = plt.subplots(r, c)\r\n for i in range(c):\r\n axs[0,i].imshow(imgs[i, :,:])\r\n axs[0,i].axis('off')\r\n axs[1,i].imshow(masked_imgs[i, :,:])\r\n axs[1,i].axis('off')\r\n filled_in = imgs[i].copy()\r\n filled_in[y1[i]:y2[i], x1[i]:x2[i], :] = gen_missing[i]\r\n axs[2,i].imshow(filled_in)\r\n axs[2,i].axis('off')\r\n fig.savefig(\"images/%d.png\" % epoch)\r\n plt.close()\r\n\r\n def save_model(self):\r\n\r\n def save(model, model_name):\r\n model_path = \"saved_model/%s.json\" % model_name\r\n weights_path = \"saved_model/%s_weights.hdf5\" % model_name\r\n options = {\"file_arch\": model_path,\r\n \"file_weight\": weights_path}\r\n json_string = model.to_json()\r\n open(options['file_arch'], 'w').write(json_string)\r\n model.save_weights(options['file_weight'])\r\n\r\n save(self.generator, \"generator\")\r\n save(self.discriminator, \"discriminator\")\r\n\r\n\r\nif __name__ == '__main__':\r\n context_encoder = ContextEncoder()\r\n context_encoder.train(epochs=30000, batch_size=64, sample_interval=50)\r\n" }, { "path": "context_encoder/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "context_encoder/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "cyclegan/cyclegan.py", "content": "from __future__ import print_function, division\nimport scipy\n\nfrom keras.datasets import mnist\nfrom keras_contrib.layers.normalization.instancenormalization import InstanceNormalization\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, Concatenate\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nimport datetime\nimport matplotlib.pyplot as plt\nimport sys\nfrom data_loader import DataLoader\nimport numpy as np\nimport os\n\nclass CycleGAN():\n def __init__(self):\n # Input shape\n self.img_rows = 128\n self.img_cols = 128\n self.channels = 3\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n\n # Configure data loader\n self.dataset_name = 'apple2orange'\n self.data_loader = DataLoader(dataset_name=self.dataset_name,\n img_res=(self.img_rows, self.img_cols))\n\n\n # Calculate output shape of D (PatchGAN)\n patch = int(self.img_rows / 2**4)\n self.disc_patch = (patch, patch, 1)\n\n # Number of filters in the first layer of G and D\n self.gf = 32\n self.df = 64\n\n # Loss weights\n self.lambda_cycle = 10.0 # Cycle-consistency loss\n self.lambda_id = 0.1 * self.lambda_cycle # Identity loss\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminators\n self.d_A = self.build_discriminator()\n self.d_B = self.build_discriminator()\n self.d_A.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n self.d_B.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n #-------------------------\n # Construct Computational\n # Graph of Generators\n #-------------------------\n\n # Build the generators\n self.g_AB = self.build_generator()\n self.g_BA = self.build_generator()\n\n # Input images from both domains\n img_A = Input(shape=self.img_shape)\n img_B = Input(shape=self.img_shape)\n\n # Translate images to the other domain\n fake_B = self.g_AB(img_A)\n fake_A = self.g_BA(img_B)\n # Translate images back to original domain\n reconstr_A = self.g_BA(fake_B)\n reconstr_B = self.g_AB(fake_A)\n # Identity mapping of images\n img_A_id = self.g_BA(img_A)\n img_B_id = self.g_AB(img_B)\n\n # For the combined model we will only train the generators\n self.d_A.trainable = False\n self.d_B.trainable = False\n\n # Discriminators determines validity of translated images\n valid_A = self.d_A(fake_A)\n valid_B = self.d_B(fake_B)\n\n # Combined model trains generators to fool discriminators\n self.combined = Model(inputs=[img_A, img_B],\n outputs=[ valid_A, valid_B,\n reconstr_A, reconstr_B,\n img_A_id, img_B_id ])\n self.combined.compile(loss=['mse', 'mse',\n 'mae', 'mae',\n 'mae', 'mae'],\n loss_weights=[ 1, 1,\n self.lambda_cycle, self.lambda_cycle,\n self.lambda_id, self.lambda_id ],\n optimizer=optimizer)\n\n def build_generator(self):\n \"\"\"U-Net Generator\"\"\"\n\n def conv2d(layer_input, filters, f_size=4):\n \"\"\"Layers used during downsampling\"\"\"\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n d = InstanceNormalization()(d)\n return d\n\n def deconv2d(layer_input, skip_input, filters, f_size=4, dropout_rate=0):\n \"\"\"Layers used during upsampling\"\"\"\n u = UpSampling2D(size=2)(layer_input)\n u = Conv2D(filters, kernel_size=f_size, strides=1, padding='same', activation='relu')(u)\n if dropout_rate:\n u = Dropout(dropout_rate)(u)\n u = InstanceNormalization()(u)\n u = Concatenate()([u, skip_input])\n return u\n\n # Image input\n d0 = Input(shape=self.img_shape)\n\n # Downsampling\n d1 = conv2d(d0, self.gf)\n d2 = conv2d(d1, self.gf*2)\n d3 = conv2d(d2, self.gf*4)\n d4 = conv2d(d3, self.gf*8)\n\n # Upsampling\n u1 = deconv2d(d4, d3, self.gf*4)\n u2 = deconv2d(u1, d2, self.gf*2)\n u3 = deconv2d(u2, d1, self.gf)\n\n u4 = UpSampling2D(size=2)(u3)\n output_img = Conv2D(self.channels, kernel_size=4, strides=1, padding='same', activation='tanh')(u4)\n\n return Model(d0, output_img)\n\n def build_discriminator(self):\n\n def d_layer(layer_input, filters, f_size=4, normalization=True):\n \"\"\"Discriminator layer\"\"\"\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n if normalization:\n d = InstanceNormalization()(d)\n return d\n\n img = Input(shape=self.img_shape)\n\n d1 = d_layer(img, self.df, normalization=False)\n d2 = d_layer(d1, self.df*2)\n d3 = d_layer(d2, self.df*4)\n d4 = d_layer(d3, self.df*8)\n\n validity = Conv2D(1, kernel_size=4, strides=1, padding='same')(d4)\n\n return Model(img, validity)\n\n def train(self, epochs, batch_size=1, sample_interval=50):\n\n start_time = datetime.datetime.now()\n\n # Adversarial loss ground truths\n valid = np.ones((batch_size,) + self.disc_patch)\n fake = np.zeros((batch_size,) + self.disc_patch)\n\n for epoch in range(epochs):\n for batch_i, (imgs_A, imgs_B) in enumerate(self.data_loader.load_batch(batch_size)):\n\n # ----------------------\n # Train Discriminators\n # ----------------------\n\n # Translate images to opposite domain\n fake_B = self.g_AB.predict(imgs_A)\n fake_A = self.g_BA.predict(imgs_B)\n\n # Train the discriminators (original images = real / translated = Fake)\n dA_loss_real = self.d_A.train_on_batch(imgs_A, valid)\n dA_loss_fake = self.d_A.train_on_batch(fake_A, fake)\n dA_loss = 0.5 * np.add(dA_loss_real, dA_loss_fake)\n\n dB_loss_real = self.d_B.train_on_batch(imgs_B, valid)\n dB_loss_fake = self.d_B.train_on_batch(fake_B, fake)\n dB_loss = 0.5 * np.add(dB_loss_real, dB_loss_fake)\n\n # Total disciminator loss\n d_loss = 0.5 * np.add(dA_loss, dB_loss)\n\n\n # ------------------\n # Train Generators\n # ------------------\n\n # Train the generators\n g_loss = self.combined.train_on_batch([imgs_A, imgs_B],\n [valid, valid,\n imgs_A, imgs_B,\n imgs_A, imgs_B])\n\n elapsed_time = datetime.datetime.now() - start_time\n\n # Plot the progress\n print (\"[Epoch %d/%d] [Batch %d/%d] [D loss: %f, acc: %3d%%] [G loss: %05f, adv: %05f, recon: %05f, id: %05f] time: %s \" \\\n % ( epoch, epochs,\n batch_i, self.data_loader.n_batches,\n d_loss[0], 100*d_loss[1],\n g_loss[0],\n np.mean(g_loss[1:3]),\n np.mean(g_loss[3:5]),\n np.mean(g_loss[5:6]),\n elapsed_time))\n\n # If at save interval => save generated image samples\n if batch_i % sample_interval == 0:\n self.sample_images(epoch, batch_i)\n\n def sample_images(self, epoch, batch_i):\n os.makedirs('images/%s' % self.dataset_name, exist_ok=True)\n r, c = 2, 3\n\n imgs_A = self.data_loader.load_data(domain=\"A\", batch_size=1, is_testing=True)\n imgs_B = self.data_loader.load_data(domain=\"B\", batch_size=1, is_testing=True)\n\n # Demo (for GIF)\n #imgs_A = self.data_loader.load_img('datasets/apple2orange/testA/n07740461_1541.jpg')\n #imgs_B = self.data_loader.load_img('datasets/apple2orange/testB/n07749192_4241.jpg')\n\n # Translate images to the other domain\n fake_B = self.g_AB.predict(imgs_A)\n fake_A = self.g_BA.predict(imgs_B)\n # Translate back to original domain\n reconstr_A = self.g_BA.predict(fake_B)\n reconstr_B = self.g_AB.predict(fake_A)\n\n gen_imgs = np.concatenate([imgs_A, fake_B, reconstr_A, imgs_B, fake_A, reconstr_B])\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n titles = ['Original', 'Translated', 'Reconstructed']\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt])\n axs[i, j].set_title(titles[j])\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/%s/%d_%d.png\" % (self.dataset_name, epoch, batch_i))\n plt.close()\n\n\nif __name__ == '__main__':\n gan = CycleGAN()\n gan.train(epochs=200, batch_size=1, sample_interval=200)\n" }, { "path": "cyclegan/data_loader.py", "content": "import scipy\nfrom glob import glob\nimport numpy as np\n\nclass DataLoader():\n def __init__(self, dataset_name, img_res=(128, 128)):\n self.dataset_name = dataset_name\n self.img_res = img_res\n\n def load_data(self, domain, batch_size=1, is_testing=False):\n data_type = \"train%s\" % domain if not is_testing else \"test%s\" % domain\n path = glob('./datasets/%s/%s/*' % (self.dataset_name, data_type))\n\n batch_images = np.random.choice(path, size=batch_size)\n\n imgs = []\n for img_path in batch_images:\n img = self.imread(img_path)\n if not is_testing:\n img = scipy.misc.imresize(img, self.img_res)\n\n if np.random.random() > 0.5:\n img = np.fliplr(img)\n else:\n img = scipy.misc.imresize(img, self.img_res)\n imgs.append(img)\n\n imgs = np.array(imgs)/127.5 - 1.\n\n return imgs\n\n def load_batch(self, batch_size=1, is_testing=False):\n data_type = \"train\" if not is_testing else \"val\"\n path_A = glob('./datasets/%s/%sA/*' % (self.dataset_name, data_type))\n path_B = glob('./datasets/%s/%sB/*' % (self.dataset_name, data_type))\n\n self.n_batches = int(min(len(path_A), len(path_B)) / batch_size)\n total_samples = self.n_batches * batch_size\n\n # Sample n_batches * batch_size from each path list so that model sees all\n # samples from both domains\n path_A = np.random.choice(path_A, total_samples, replace=False)\n path_B = np.random.choice(path_B, total_samples, replace=False)\n\n for i in range(self.n_batches-1):\n batch_A = path_A[i*batch_size:(i+1)*batch_size]\n batch_B = path_B[i*batch_size:(i+1)*batch_size]\n imgs_A, imgs_B = [], []\n for img_A, img_B in zip(batch_A, batch_B):\n img_A = self.imread(img_A)\n img_B = self.imread(img_B)\n\n img_A = scipy.misc.imresize(img_A, self.img_res)\n img_B = scipy.misc.imresize(img_B, self.img_res)\n\n if not is_testing and np.random.random() > 0.5:\n img_A = np.fliplr(img_A)\n img_B = np.fliplr(img_B)\n\n imgs_A.append(img_A)\n imgs_B.append(img_B)\n\n imgs_A = np.array(imgs_A)/127.5 - 1.\n imgs_B = np.array(imgs_B)/127.5 - 1.\n\n yield imgs_A, imgs_B\n\n def load_img(self, path):\n img = self.imread(path)\n img = scipy.misc.imresize(img, self.img_res)\n img = img/127.5 - 1.\n return img[np.newaxis, :, :, :]\n\n def imread(self, path):\n return scipy.misc.imread(path, mode='RGB').astype(np.float)\n" }, { "path": "cyclegan/download_dataset.sh", "content": "mkdir datasets\nFILE=$1\n\nif [[ $FILE != \"ae_photos\" && $FILE != \"apple2orange\" && $FILE != \"summer2winter_yosemite\" && $FILE != \"horse2zebra\" && $FILE != \"monet2photo\" && $FILE != \"cezanne2photo\" && $FILE != \"ukiyoe2photo\" && $FILE != \"vangogh2photo\" && $FILE != \"maps\" && $FILE != \"cityscapes\" && $FILE != \"facades\" && $FILE != \"iphone2dslr_flower\" && $FILE != \"ae_photos\" ]]; then\n echo \"Available datasets are: apple2orange, summer2winter_yosemite, horse2zebra, monet2photo, cezanne2photo, ukiyoe2photo, vangogh2photo, maps, cityscapes, facades, iphone2dslr_flower, ae_photos\"\n exit 1\nfi\n\nURL=https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/$FILE.zip\nZIP_FILE=./datasets/$FILE.zip\nTARGET_DIR=./datasets/$FILE/\nwget -N $URL -O $ZIP_FILE\nmkdir $TARGET_DIR\nunzip $ZIP_FILE -d ./datasets/\nrm $ZIP_FILE\n" }, { "path": "cyclegan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "cyclegan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "dcgan/dcgan.py", "content": "from __future__ import print_function, division\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\n\nimport matplotlib.pyplot as plt\n\nimport sys\n\nimport numpy as np\n\nclass DCGAN():\n def __init__(self):\n # Input shape\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.latent_dim = 100\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminator\n self.discriminator = self.build_discriminator()\n self.discriminator.compile(loss='binary_crossentropy',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Build the generator\n self.generator = self.build_generator()\n\n # The generator takes noise as input and generates imgs\n z = Input(shape=(self.latent_dim,))\n img = self.generator(z)\n\n # For the combined model we will only train the generator\n self.discriminator.trainable = False\n\n # The discriminator takes generated images as input and determines validity\n valid = self.discriminator(img)\n\n # The combined model (stacked generator and discriminator)\n # Trains the generator to fool the discriminator\n self.combined = Model(z, valid)\n self.combined.compile(loss='binary_crossentropy', optimizer=optimizer)\n\n def build_generator(self):\n\n model = Sequential()\n\n model.add(Dense(128 * 7 * 7, activation=\"relu\", input_dim=self.latent_dim))\n model.add(Reshape((7, 7, 128)))\n model.add(UpSampling2D())\n model.add(Conv2D(128, kernel_size=3, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Activation(\"relu\"))\n model.add(UpSampling2D())\n model.add(Conv2D(64, kernel_size=3, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Activation(\"relu\"))\n model.add(Conv2D(self.channels, kernel_size=3, padding=\"same\"))\n model.add(Activation(\"tanh\"))\n\n model.summary()\n\n noise = Input(shape=(self.latent_dim,))\n img = model(noise)\n\n return Model(noise, img)\n\n def build_discriminator(self):\n\n model = Sequential()\n\n model.add(Conv2D(32, kernel_size=3, strides=2, input_shape=self.img_shape, padding=\"same\"))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\n model.add(ZeroPadding2D(padding=((0,1),(0,1))))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(128, kernel_size=3, strides=2, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(256, kernel_size=3, strides=1, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Flatten())\n model.add(Dense(1, activation='sigmoid'))\n\n model.summary()\n\n img = Input(shape=self.img_shape)\n validity = model(img)\n\n return Model(img, validity)\n\n def train(self, epochs, batch_size=128, save_interval=50):\n\n # Load the dataset\n (X_train, _), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = X_train / 127.5 - 1.\n X_train = np.expand_dims(X_train, axis=3)\n\n # Adversarial ground truths\n valid = np.ones((batch_size, 1))\n fake = np.zeros((batch_size, 1))\n\n for epoch in range(epochs):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n # Select a random half of images\n idx = np.random.randint(0, X_train.shape[0], batch_size)\n imgs = X_train[idx]\n\n # Sample noise and generate a batch of new images\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n\n # Train the discriminator (real classified as ones and generated as zeros)\n d_loss_real = self.discriminator.train_on_batch(imgs, valid)\n d_loss_fake = self.discriminator.train_on_batch(gen_imgs, fake)\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n\n # ---------------------\n # Train Generator\n # ---------------------\n\n # Train the generator (wants discriminator to mistake images as real)\n g_loss = self.combined.train_on_batch(noise, valid)\n\n # Plot the progress\n print (\"%d [D loss: %f, acc.: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss))\n\n # If at save interval => save generated image samples\n if epoch % save_interval == 0:\n self.save_imgs(epoch)\n\n def save_imgs(self, epoch):\n r, c = 5, 5\n noise = np.random.normal(0, 1, (r * c, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/mnist_%d.png\" % epoch)\n plt.close()\n\n\nif __name__ == '__main__':\n dcgan = DCGAN()\n dcgan.train(epochs=4000, batch_size=32, save_interval=50)\n" }, { "path": "dcgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "dcgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "discogan/data_loader.py", "content": "import scipy\nfrom glob import glob\nimport numpy as np\n\nclass DataLoader():\n def __init__(self, dataset_name, img_res=(128, 128)):\n self.dataset_name = dataset_name\n self.img_res = img_res\n\n def load_data(self, batch_size=1, is_testing=False):\n data_type = \"train\" if not is_testing else \"val\"\n path = glob('./datasets/%s/%s/*' % (self.dataset_name, data_type))\n\n batch = np.random.choice(path, size=batch_size)\n\n imgs_A, imgs_B = [], []\n for img in batch:\n img = self.imread(img)\n h, w, _ = img.shape\n half_w = int(w/2)\n img_A = img[:, :half_w, :]\n img_B = img[:, half_w:, :]\n\n img_A = scipy.misc.imresize(img_A, self.img_res)\n img_B = scipy.misc.imresize(img_B, self.img_res)\n\n if not is_testing and np.random.random() > 0.5:\n img_A = np.fliplr(img_A)\n img_B = np.fliplr(img_B)\n\n imgs_A.append(img_A)\n imgs_B.append(img_B)\n\n imgs_A = np.array(imgs_A)/127.5 - 1.\n imgs_B = np.array(imgs_B)/127.5 - 1.\n\n return imgs_A, imgs_B\n\n def load_batch(self, batch_size=1, is_testing=False):\n data_type = \"train\" if not is_testing else \"val\"\n path = glob('./datasets/%s/%s/*' % (self.dataset_name, data_type))\n\n self.n_batches = int(len(path) / batch_size)\n\n for i in range(self.n_batches-1):\n batch = path[i*batch_size:(i+1)*batch_size]\n imgs_A, imgs_B = [], []\n for img in batch:\n img = self.imread(img)\n h, w, _ = img.shape\n half_w = int(w/2)\n img_A = img[:, :half_w, :]\n img_B = img[:, half_w:, :]\n\n img_A = scipy.misc.imresize(img_A, self.img_res)\n img_B = scipy.misc.imresize(img_B, self.img_res)\n\n if not is_testing and np.random.random() > 0.5:\n img_A = np.fliplr(img_A)\n img_B = np.fliplr(img_B)\n\n imgs_A.append(img_A)\n imgs_B.append(img_B)\n\n imgs_A = np.array(imgs_A)/127.5 - 1.\n imgs_B = np.array(imgs_B)/127.5 - 1.\n\n yield imgs_A, imgs_B\n\n def load_img(self, path):\n img = self.imread(path)\n img = scipy.misc.imresize(img, self.img_res)\n img = img/127.5 - 1.\n return img[np.newaxis, :, :, :]\n\n def imread(self, path):\n return scipy.misc.imread(path, mode='RGB').astype(np.float)\n" }, { "path": "discogan/discogan.py", "content": "from __future__ import print_function, division\nimport scipy\n\nfrom keras.datasets import mnist\nfrom keras_contrib.layers.normalization.instancenormalization import InstanceNormalization\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, Concatenate\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nimport datetime\nimport matplotlib.pyplot as plt\nimport sys\nfrom data_loader import DataLoader\nimport numpy as np\nimport os\n\nclass DiscoGAN():\n def __init__(self):\n # Input shape\n self.img_rows = 128\n self.img_cols = 128\n self.channels = 3\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n\n # Configure data loader\n self.dataset_name = 'edges2shoes'\n self.data_loader = DataLoader(dataset_name=self.dataset_name,\n img_res=(self.img_rows, self.img_cols))\n\n\n # Calculate output shape of D (PatchGAN)\n patch = int(self.img_rows / 2**4)\n self.disc_patch = (patch, patch, 1)\n\n # Number of filters in the first layer of G and D\n self.gf = 64\n self.df = 64\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminators\n self.d_A = self.build_discriminator()\n self.d_B = self.build_discriminator()\n self.d_A.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n self.d_B.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n #-------------------------\n # Construct Computational\n # Graph of Generators\n #-------------------------\n\n # Build the generators\n self.g_AB = self.build_generator()\n self.g_BA = self.build_generator()\n\n # Input images from both domains\n img_A = Input(shape=self.img_shape)\n img_B = Input(shape=self.img_shape)\n\n # Translate images to the other domain\n fake_B = self.g_AB(img_A)\n fake_A = self.g_BA(img_B)\n # Translate images back to original domain\n reconstr_A = self.g_BA(fake_B)\n reconstr_B = self.g_AB(fake_A)\n\n # For the combined model we will only train the generators\n self.d_A.trainable = False\n self.d_B.trainable = False\n\n # Discriminators determines validity of translated images\n valid_A = self.d_A(fake_A)\n valid_B = self.d_B(fake_B)\n\n # Objectives\n # + Adversarial: Fool domain discriminators\n # + Translation: Minimize MAE between e.g. fake B and true B\n # + Cycle-consistency: Minimize MAE between reconstructed images and original\n self.combined = Model(inputs=[img_A, img_B],\n outputs=[ valid_A, valid_B,\n fake_B, fake_A,\n reconstr_A, reconstr_B ])\n self.combined.compile(loss=['mse', 'mse',\n 'mae', 'mae',\n 'mae', 'mae'],\n optimizer=optimizer)\n\n def build_generator(self):\n \"\"\"U-Net Generator\"\"\"\n\n def conv2d(layer_input, filters, f_size=4, normalize=True):\n \"\"\"Layers used during downsampling\"\"\"\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n if normalize:\n d = InstanceNormalization()(d)\n return d\n\n def deconv2d(layer_input, skip_input, filters, f_size=4, dropout_rate=0):\n \"\"\"Layers used during upsampling\"\"\"\n u = UpSampling2D(size=2)(layer_input)\n u = Conv2D(filters, kernel_size=f_size, strides=1, padding='same', activation='relu')(u)\n if dropout_rate:\n u = Dropout(dropout_rate)(u)\n u = InstanceNormalization()(u)\n u = Concatenate()([u, skip_input])\n return u\n\n # Image input\n d0 = Input(shape=self.img_shape)\n\n # Downsampling\n d1 = conv2d(d0, self.gf, normalize=False)\n d2 = conv2d(d1, self.gf*2)\n d3 = conv2d(d2, self.gf*4)\n d4 = conv2d(d3, self.gf*8)\n d5 = conv2d(d4, self.gf*8)\n d6 = conv2d(d5, self.gf*8)\n d7 = conv2d(d6, self.gf*8)\n\n # Upsampling\n u1 = deconv2d(d7, d6, self.gf*8)\n u2 = deconv2d(u1, d5, self.gf*8)\n u3 = deconv2d(u2, d4, self.gf*8)\n u4 = deconv2d(u3, d3, self.gf*4)\n u5 = deconv2d(u4, d2, self.gf*2)\n u6 = deconv2d(u5, d1, self.gf)\n\n u7 = UpSampling2D(size=2)(u6)\n output_img = Conv2D(self.channels, kernel_size=4, strides=1,\n padding='same', activation='tanh')(u7)\n\n return Model(d0, output_img)\n\n def build_discriminator(self):\n\n def d_layer(layer_input, filters, f_size=4, normalization=True):\n \"\"\"Discriminator layer\"\"\"\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n if normalization:\n d = InstanceNormalization()(d)\n return d\n\n img = Input(shape=self.img_shape)\n\n d1 = d_layer(img, self.df, normalization=False)\n d2 = d_layer(d1, self.df*2)\n d3 = d_layer(d2, self.df*4)\n d4 = d_layer(d3, self.df*8)\n\n validity = Conv2D(1, kernel_size=4, strides=1, padding='same')(d4)\n\n return Model(img, validity)\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n start_time = datetime.datetime.now()\n\n # Adversarial loss ground truths\n valid = np.ones((batch_size,) + self.disc_patch)\n fake = np.zeros((batch_size,) + self.disc_patch)\n\n for epoch in range(epochs):\n\n for batch_i, (imgs_A, imgs_B) in enumerate(self.data_loader.load_batch(batch_size)):\n\n # ----------------------\n # Train Discriminators\n # ----------------------\n\n # Translate images to opposite domain\n fake_B = self.g_AB.predict(imgs_A)\n fake_A = self.g_BA.predict(imgs_B)\n\n # Train the discriminators (original images = real / translated = Fake)\n dA_loss_real = self.d_A.train_on_batch(imgs_A, valid)\n dA_loss_fake = self.d_A.train_on_batch(fake_A, fake)\n dA_loss = 0.5 * np.add(dA_loss_real, dA_loss_fake)\n\n dB_loss_real = self.d_B.train_on_batch(imgs_B, valid)\n dB_loss_fake = self.d_B.train_on_batch(fake_B, fake)\n dB_loss = 0.5 * np.add(dB_loss_real, dB_loss_fake)\n\n # Total disciminator loss\n d_loss = 0.5 * np.add(dA_loss, dB_loss)\n\n # ------------------\n # Train Generators\n # ------------------\n\n # Train the generators\n g_loss = self.combined.train_on_batch([imgs_A, imgs_B], [valid, valid, \\\n imgs_B, imgs_A, \\\n imgs_A, imgs_B])\n\n elapsed_time = datetime.datetime.now() - start_time\n # Plot the progress\n print (\"[%d] [%d/%d] time: %s, [d_loss: %f, g_loss: %f]\" % (epoch, batch_i,\n self.data_loader.n_batches,\n elapsed_time,\n d_loss[0], g_loss[0]))\n\n # If at save interval => save generated image samples\n if batch_i % sample_interval == 0:\n self.sample_images(epoch, batch_i)\n\n def sample_images(self, epoch, batch_i):\n os.makedirs('images/%s' % self.dataset_name, exist_ok=True)\n r, c = 2, 3\n\n imgs_A, imgs_B = self.data_loader.load_data(batch_size=1, is_testing=True)\n\n # Translate images to the other domain\n fake_B = self.g_AB.predict(imgs_A)\n fake_A = self.g_BA.predict(imgs_B)\n # Translate back to original domain\n reconstr_A = self.g_BA.predict(fake_B)\n reconstr_B = self.g_AB.predict(fake_A)\n\n gen_imgs = np.concatenate([imgs_A, fake_B, reconstr_A, imgs_B, fake_A, reconstr_B])\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n titles = ['Original', 'Translated', 'Reconstructed']\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt])\n axs[i, j].set_title(titles[j])\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/%s/%d_%d.png\" % (self.dataset_name, epoch, batch_i))\n plt.close()\n\n\nif __name__ == '__main__':\n gan = DiscoGAN()\n gan.train(epochs=20, batch_size=1, sample_interval=200)\n" }, { "path": "discogan/download_dataset.sh", "content": "mkdir datasets\nFILE=$1\nURL=https://people.eecs.berkeley.edu/~tinghuiz/projects/pix2pix/datasets/$FILE.tar.gz\nTAR_FILE=./datasets/$FILE.tar.gz\nTARGET_DIR=./datasets/$FILE/\nwget -N $URL -O $TAR_FILE\nmkdir $TARGET_DIR\ntar -zxvf $TAR_FILE -C ./datasets/\nrm $TAR_FILE\n" }, { "path": "discogan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "discogan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "dualgan/dualgan.py", "content": "from __future__ import print_function, division\nimport scipy\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, Concatenate\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import RMSprop, Adam\nfrom keras.utils import to_categorical\nimport keras.backend as K\n\nimport matplotlib.pyplot as plt\n\nimport sys\n\nimport numpy as np\n\nclass DUALGAN():\n def __init__(self):\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_dim = self.img_rows*self.img_cols\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminators\n self.D_A = self.build_discriminator()\n self.D_A.compile(loss=self.wasserstein_loss,\n optimizer=optimizer,\n metrics=['accuracy'])\n self.D_B = self.build_discriminator()\n self.D_B.compile(loss=self.wasserstein_loss,\n optimizer=optimizer,\n metrics=['accuracy'])\n\n #-------------------------\n # Construct Computational\n # Graph of Generators\n #-------------------------\n\n # Build the generators\n self.G_AB = self.build_generator()\n self.G_BA = self.build_generator()\n\n # For the combined model we will only train the generators\n self.D_A.trainable = False\n self.D_B.trainable = False\n\n # The generator takes images from their respective domains as inputs\n imgs_A = Input(shape=(self.img_dim,))\n imgs_B = Input(shape=(self.img_dim,))\n\n # Generators translates the images to the opposite domain\n fake_B = self.G_AB(imgs_A)\n fake_A = self.G_BA(imgs_B)\n\n # The discriminators determines validity of translated images\n valid_A = self.D_A(fake_A)\n valid_B = self.D_B(fake_B)\n\n # Generators translate the images back to their original domain\n recov_A = self.G_BA(fake_B)\n recov_B = self.G_AB(fake_A)\n\n # The combined model (stacked generators and discriminators)\n self.combined = Model(inputs=[imgs_A, imgs_B], outputs=[valid_A, valid_B, recov_A, recov_B])\n self.combined.compile(loss=[self.wasserstein_loss, self.wasserstein_loss, 'mae', 'mae'],\n optimizer=optimizer,\n loss_weights=[1, 1, 100, 100])\n\n def build_generator(self):\n\n X = Input(shape=(self.img_dim,))\n\n model = Sequential()\n model.add(Dense(256, input_dim=self.img_dim))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dropout(0.4))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dropout(0.4))\n model.add(Dense(1024))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dropout(0.4))\n model.add(Dense(self.img_dim, activation='tanh'))\n\n X_translated = model(X)\n\n return Model(X, X_translated)\n\n def build_discriminator(self):\n\n img = Input(shape=(self.img_dim,))\n\n model = Sequential()\n model.add(Dense(512, input_dim=self.img_dim))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dense(256))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(1))\n\n validity = model(img)\n\n return Model(img, validity)\n\n def sample_generator_input(self, X, batch_size):\n # Sample random batch of images from X\n idx = np.random.randint(0, X.shape[0], batch_size)\n return X[idx]\n\n def wasserstein_loss(self, y_true, y_pred):\n return K.mean(y_true * y_pred)\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n # Load the dataset\n (X_train, _), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\n\n # Domain A and B (rotated)\n X_A = X_train[:int(X_train.shape[0]/2)]\n X_B = scipy.ndimage.interpolation.rotate(X_train[int(X_train.shape[0]/2):], 90, axes=(1, 2))\n\n X_A = X_A.reshape(X_A.shape[0], self.img_dim)\n X_B = X_B.reshape(X_B.shape[0], self.img_dim)\n\n clip_value = 0.01\n n_critic = 4\n\n # Adversarial ground truths\n valid = -np.ones((batch_size, 1))\n fake = np.ones((batch_size, 1))\n\n for epoch in range(epochs):\n\n # Train the discriminator for n_critic iterations\n for _ in range(n_critic):\n\n # ----------------------\n # Train Discriminators\n # ----------------------\n\n # Sample generator inputs\n imgs_A = self.sample_generator_input(X_A, batch_size)\n imgs_B = self.sample_generator_input(X_B, batch_size)\n\n # Translate images to their opposite domain\n fake_B = self.G_AB.predict(imgs_A)\n fake_A = self.G_BA.predict(imgs_B)\n\n # Train the discriminators\n D_A_loss_real = self.D_A.train_on_batch(imgs_A, valid)\n D_A_loss_fake = self.D_A.train_on_batch(fake_A, fake)\n\n D_B_loss_real = self.D_B.train_on_batch(imgs_B, valid)\n D_B_loss_fake = self.D_B.train_on_batch(fake_B, fake)\n\n D_A_loss = 0.5 * np.add(D_A_loss_real, D_A_loss_fake)\n D_B_loss = 0.5 * np.add(D_B_loss_real, D_B_loss_fake)\n\n # Clip discriminator weights\n for d in [self.D_A, self.D_B]:\n for l in d.layers:\n weights = l.get_weights()\n weights = [np.clip(w, -clip_value, clip_value) for w in weights]\n l.set_weights(weights)\n\n # ------------------\n # Train Generators\n # ------------------\n\n # Train the generators\n g_loss = self.combined.train_on_batch([imgs_A, imgs_B], [valid, valid, imgs_A, imgs_B])\n\n # Plot the progress\n print (\"%d [D1 loss: %f] [D2 loss: %f] [G loss: %f]\" \\\n % (epoch, D_A_loss[0], D_B_loss[0], g_loss[0]))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.save_imgs(epoch, X_A, X_B)\n\n def save_imgs(self, epoch, X_A, X_B):\n r, c = 4, 4\n\n # Sample generator inputs\n imgs_A = self.sample_generator_input(X_A, c)\n imgs_B = self.sample_generator_input(X_B, c)\n\n # Images translated to their opposite domain\n fake_B = self.G_AB.predict(imgs_A)\n fake_A = self.G_BA.predict(imgs_B)\n\n gen_imgs = np.concatenate([imgs_A, fake_B, imgs_B, fake_A])\n gen_imgs = gen_imgs.reshape((r, c, self.img_rows, self.img_cols, 1))\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[i, j, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/mnist_%d.png\" % epoch)\n plt.close()\n\n\nif __name__ == '__main__':\n gan = DUALGAN()\n gan.train(epochs=30000, batch_size=32, sample_interval=200)\n" }, { "path": "dualgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "dualgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "gan/gan.py", "content": "from __future__ import print_function, division\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\n\nimport matplotlib.pyplot as plt\n\nimport sys\n\nimport numpy as np\n\nclass GAN():\n def __init__(self):\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.latent_dim = 100\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminator\n self.discriminator = self.build_discriminator()\n self.discriminator.compile(loss='binary_crossentropy',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Build the generator\n self.generator = self.build_generator()\n\n # The generator takes noise as input and generates imgs\n z = Input(shape=(self.latent_dim,))\n img = self.generator(z)\n\n # For the combined model we will only train the generator\n self.discriminator.trainable = False\n\n # The discriminator takes generated images as input and determines validity\n validity = self.discriminator(img)\n\n # The combined model (stacked generator and discriminator)\n # Trains the generator to fool the discriminator\n self.combined = Model(z, validity)\n self.combined.compile(loss='binary_crossentropy', optimizer=optimizer)\n\n\n def build_generator(self):\n\n model = Sequential()\n\n model.add(Dense(256, input_dim=self.latent_dim))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(1024))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(np.prod(self.img_shape), activation='tanh'))\n model.add(Reshape(self.img_shape))\n\n model.summary()\n\n noise = Input(shape=(self.latent_dim,))\n img = model(noise)\n\n return Model(noise, img)\n\n def build_discriminator(self):\n\n model = Sequential()\n\n model.add(Flatten(input_shape=self.img_shape))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dense(256))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dense(1, activation='sigmoid'))\n model.summary()\n\n img = Input(shape=self.img_shape)\n validity = model(img)\n\n return Model(img, validity)\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n # Load the dataset\n (X_train, _), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = X_train / 127.5 - 1.\n X_train = np.expand_dims(X_train, axis=3)\n\n # Adversarial ground truths\n valid = np.ones((batch_size, 1))\n fake = np.zeros((batch_size, 1))\n\n for epoch in range(epochs):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n # Select a random batch of images\n idx = np.random.randint(0, X_train.shape[0], batch_size)\n imgs = X_train[idx]\n\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n\n # Generate a batch of new images\n gen_imgs = self.generator.predict(noise)\n\n # Train the discriminator\n d_loss_real = self.discriminator.train_on_batch(imgs, valid)\n d_loss_fake = self.discriminator.train_on_batch(gen_imgs, fake)\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n\n # ---------------------\n # Train Generator\n # ---------------------\n\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n\n # Train the generator (to have the discriminator label samples as valid)\n g_loss = self.combined.train_on_batch(noise, valid)\n\n # Plot the progress\n print (\"%d [D loss: %f, acc.: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n r, c = 5, 5\n noise = np.random.normal(0, 1, (r * c, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/%d.png\" % epoch)\n plt.close()\n\n\nif __name__ == '__main__':\n gan = GAN()\n gan.train(epochs=30000, batch_size=32, sample_interval=200)\n" }, { "path": "gan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "gan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "infogan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "infogan/infogan.py", "content": "from __future__ import print_function, division\r\n\r\nfrom keras.datasets import mnist\r\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, multiply, concatenate\r\nfrom keras.layers import BatchNormalization, Activation, Embedding, ZeroPadding2D, Lambda\r\nfrom keras.layers.advanced_activations import LeakyReLU\r\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\r\nfrom keras.models import Sequential, Model\r\nfrom keras.optimizers import Adam\r\nfrom keras.utils import to_categorical\r\nimport keras.backend as K\r\n\r\nimport matplotlib.pyplot as plt\r\n\r\nimport numpy as np\r\n\r\nclass INFOGAN():\r\n def __init__(self):\r\n self.img_rows = 28\r\n self.img_cols = 28\r\n self.channels = 1\r\n self.num_classes = 10\r\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\r\n self.latent_dim = 72\r\n\r\n\r\n optimizer = Adam(0.0002, 0.5)\r\n losses = ['binary_crossentropy', self.mutual_info_loss]\r\n\r\n # Build and the discriminator and recognition network\r\n self.discriminator, self.auxilliary = self.build_disk_and_q_net()\r\n\r\n self.discriminator.compile(loss=['binary_crossentropy'],\r\n optimizer=optimizer,\r\n metrics=['accuracy'])\r\n\r\n # Build and compile the recognition network Q\r\n self.auxilliary.compile(loss=[self.mutual_info_loss],\r\n optimizer=optimizer,\r\n metrics=['accuracy'])\r\n\r\n # Build the generator\r\n self.generator = self.build_generator()\r\n\r\n # The generator takes noise and the target label as input\r\n # and generates the corresponding digit of that label\r\n gen_input = Input(shape=(self.latent_dim,))\r\n img = self.generator(gen_input)\r\n\r\n # For the combined model we will only train the generator\r\n self.discriminator.trainable = False\r\n\r\n # The discriminator takes generated image as input and determines validity\r\n valid = self.discriminator(img)\r\n # The recognition network produces the label\r\n target_label = self.auxilliary(img)\r\n\r\n # The combined model (stacked generator and discriminator)\r\n self.combined = Model(gen_input, [valid, target_label])\r\n self.combined.compile(loss=losses,\r\n optimizer=optimizer)\r\n\r\n\r\n def build_generator(self):\r\n\r\n model = Sequential()\r\n\r\n model.add(Dense(128 * 7 * 7, activation=\"relu\", input_dim=self.latent_dim))\r\n model.add(Reshape((7, 7, 128)))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(UpSampling2D())\r\n model.add(Conv2D(128, kernel_size=3, padding=\"same\"))\r\n model.add(Activation(\"relu\"))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(UpSampling2D())\r\n model.add(Conv2D(64, kernel_size=3, padding=\"same\"))\r\n model.add(Activation(\"relu\"))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(self.channels, kernel_size=3, padding='same'))\r\n model.add(Activation(\"tanh\"))\r\n\r\n gen_input = Input(shape=(self.latent_dim,))\r\n img = model(gen_input)\r\n\r\n model.summary()\r\n\r\n return Model(gen_input, img)\r\n\r\n\r\n def build_disk_and_q_net(self):\r\n\r\n img = Input(shape=self.img_shape)\r\n\r\n # Shared layers between discriminator and recognition network\r\n model = Sequential()\r\n model.add(Conv2D(64, kernel_size=3, strides=2, input_shape=self.img_shape, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.25))\r\n model.add(Conv2D(128, kernel_size=3, strides=2, padding=\"same\"))\r\n model.add(ZeroPadding2D(padding=((0,1),(0,1))))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.25))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(256, kernel_size=3, strides=2, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.25))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Conv2D(512, kernel_size=3, strides=2, padding=\"same\"))\r\n model.add(LeakyReLU(alpha=0.2))\r\n model.add(Dropout(0.25))\r\n model.add(BatchNormalization(momentum=0.8))\r\n model.add(Flatten())\r\n\r\n img_embedding = model(img)\r\n\r\n # Discriminator\r\n validity = Dense(1, activation='sigmoid')(img_embedding)\r\n\r\n # Recognition\r\n q_net = Dense(128, activation='relu')(img_embedding)\r\n label = Dense(self.num_classes, activation='softmax')(q_net)\r\n\r\n # Return discriminator and recognition network\r\n return Model(img, validity), Model(img, label)\r\n\r\n\r\n def mutual_info_loss(self, c, c_given_x):\r\n \"\"\"The mutual information metric we aim to minimize\"\"\"\r\n eps = 1e-8\r\n conditional_entropy = K.mean(- K.sum(K.log(c_given_x + eps) * c, axis=1))\r\n entropy = K.mean(- K.sum(K.log(c + eps) * c, axis=1))\r\n\r\n return conditional_entropy + entropy\r\n\r\n def sample_generator_input(self, batch_size):\r\n # Generator inputs\r\n sampled_noise = np.random.normal(0, 1, (batch_size, 62))\r\n sampled_labels = np.random.randint(0, self.num_classes, batch_size).reshape(-1, 1)\r\n sampled_labels = to_categorical(sampled_labels, num_classes=self.num_classes)\r\n\r\n return sampled_noise, sampled_labels\r\n\r\n def train(self, epochs, batch_size=128, sample_interval=50):\r\n\r\n # Load the dataset\r\n (X_train, y_train), (_, _) = mnist.load_data()\r\n\r\n # Rescale -1 to 1\r\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\r\n X_train = np.expand_dims(X_train, axis=3)\r\n y_train = y_train.reshape(-1, 1)\r\n\r\n # Adversarial ground truths\r\n valid = np.ones((batch_size, 1))\r\n fake = np.zeros((batch_size, 1))\r\n\r\n for epoch in range(epochs):\r\n\r\n # ---------------------\r\n # Train Discriminator\r\n # ---------------------\r\n\r\n # Select a random half batch of images\r\n idx = np.random.randint(0, X_train.shape[0], batch_size)\r\n imgs = X_train[idx]\r\n\r\n # Sample noise and categorical labels\r\n sampled_noise, sampled_labels = self.sample_generator_input(batch_size)\r\n gen_input = np.concatenate((sampled_noise, sampled_labels), axis=1)\r\n\r\n # Generate a half batch of new images\r\n gen_imgs = self.generator.predict(gen_input)\r\n\r\n # Train on real and generated data\r\n d_loss_real = self.discriminator.train_on_batch(imgs, valid)\r\n d_loss_fake = self.discriminator.train_on_batch(gen_imgs, fake)\r\n\r\n # Avg. loss\r\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\r\n\r\n # ---------------------\r\n # Train Generator and Q-network\r\n # ---------------------\r\n\r\n g_loss = self.combined.train_on_batch(gen_input, [valid, sampled_labels])\r\n\r\n # Plot the progress\r\n print (\"%d [D loss: %.2f, acc.: %.2f%%] [Q loss: %.2f] [G loss: %.2f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss[1], g_loss[2]))\r\n\r\n # If at save interval => save generated image samples\r\n if epoch % sample_interval == 0:\r\n self.sample_images(epoch)\r\n\r\n def sample_images(self, epoch):\r\n r, c = 10, 10\r\n\r\n fig, axs = plt.subplots(r, c)\r\n for i in range(c):\r\n sampled_noise, _ = self.sample_generator_input(c)\r\n label = to_categorical(np.full(fill_value=i, shape=(r,1)), num_classes=self.num_classes)\r\n gen_input = np.concatenate((sampled_noise, label), axis=1)\r\n gen_imgs = self.generator.predict(gen_input)\r\n gen_imgs = 0.5 * gen_imgs + 0.5\r\n for j in range(r):\r\n axs[j,i].imshow(gen_imgs[j,:,:,0], cmap='gray')\r\n axs[j,i].axis('off')\r\n fig.savefig(\"images/%d.png\" % epoch)\r\n plt.close()\r\n\r\n def save_model(self):\r\n\r\n def save(model, model_name):\r\n model_path = \"saved_model/%s.json\" % model_name\r\n weights_path = \"saved_model/%s_weights.hdf5\" % model_name\r\n options = {\"file_arch\": model_path,\r\n \"file_weight\": weights_path}\r\n json_string = model.to_json()\r\n open(options['file_arch'], 'w').write(json_string)\r\n model.save_weights(options['file_weight'])\r\n\r\n save(self.generator, \"generator\")\r\n save(self.discriminator, \"discriminator\")\r\n\r\n\r\nif __name__ == '__main__':\r\n infogan = INFOGAN()\r\n infogan.train(epochs=50000, batch_size=128, sample_interval=50)\r\n" }, { "path": "infogan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "lsgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "lsgan/lsgan.py", "content": "from __future__ import print_function, division\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\n\nimport matplotlib.pyplot as plt\n\nimport sys\n\nimport numpy as np\n\nclass LSGAN():\n def __init__(self):\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.latent_dim = 100\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminator\n self.discriminator = self.build_discriminator()\n self.discriminator.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Build the generator\n self.generator = self.build_generator()\n\n # The generator takes noise as input and generated imgs\n z = Input(shape=(self.latent_dim,))\n img = self.generator(z)\n\n # For the combined model we will only train the generator\n self.discriminator.trainable = False\n\n # The valid takes generated images as input and determines validity\n valid = self.discriminator(img)\n\n # The combined model (stacked generator and discriminator)\n # Trains generator to fool discriminator\n self.combined = Model(z, valid)\n # (!!!) Optimize w.r.t. MSE loss instead of crossentropy\n self.combined.compile(loss='mse', optimizer=optimizer)\n\n def build_generator(self):\n\n model = Sequential()\n\n model.add(Dense(256, input_dim=self.latent_dim))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(1024))\n model.add(LeakyReLU(alpha=0.2))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Dense(np.prod(self.img_shape), activation='tanh'))\n model.add(Reshape(self.img_shape))\n\n model.summary()\n\n noise = Input(shape=(self.latent_dim,))\n img = model(noise)\n\n return Model(noise, img)\n\n def build_discriminator(self):\n\n model = Sequential()\n\n model.add(Flatten(input_shape=self.img_shape))\n model.add(Dense(512))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dense(256))\n model.add(LeakyReLU(alpha=0.2))\n # (!!!) No softmax\n model.add(Dense(1))\n model.summary()\n\n img = Input(shape=self.img_shape)\n validity = model(img)\n\n return Model(img, validity)\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n # Load the dataset\n (X_train, _), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\n X_train = np.expand_dims(X_train, axis=3)\n\n # Adversarial ground truths\n valid = np.ones((batch_size, 1))\n fake = np.zeros((batch_size, 1))\n\n for epoch in range(epochs):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n # Select a random batch of images\n idx = np.random.randint(0, X_train.shape[0], batch_size)\n imgs = X_train[idx]\n\n # Sample noise as generator input\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n\n # Generate a batch of new images\n gen_imgs = self.generator.predict(noise)\n\n # Train the discriminator\n d_loss_real = self.discriminator.train_on_batch(imgs, valid)\n d_loss_fake = self.discriminator.train_on_batch(gen_imgs, fake)\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n\n\n # ---------------------\n # Train Generator\n # ---------------------\n\n g_loss = self.combined.train_on_batch(noise, valid)\n\n # Plot the progress\n print (\"%d [D loss: %f, acc.: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n r, c = 5, 5\n noise = np.random.normal(0, 1, (r * c, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/mnist_%d.png\" % epoch)\n plt.close()\n\n\nif __name__ == '__main__':\n gan = LSGAN()\n gan.train(epochs=30000, batch_size=32, sample_interval=200)\n" }, { "path": "lsgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "pix2pix/data_loader.py", "content": "import scipy\nfrom glob import glob\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nclass DataLoader():\n def __init__(self, dataset_name, img_res=(128, 128)):\n self.dataset_name = dataset_name\n self.img_res = img_res\n\n def load_data(self, batch_size=1, is_testing=False):\n data_type = \"train\" if not is_testing else \"test\"\n path = glob('./datasets/%s/%s/*' % (self.dataset_name, data_type))\n\n batch_images = np.random.choice(path, size=batch_size)\n\n imgs_A = []\n imgs_B = []\n for img_path in batch_images:\n img = self.imread(img_path)\n\n h, w, _ = img.shape\n _w = int(w/2)\n img_A, img_B = img[:, :_w, :], img[:, _w:, :]\n\n img_A = scipy.misc.imresize(img_A, self.img_res)\n img_B = scipy.misc.imresize(img_B, self.img_res)\n\n # If training => do random flip\n if not is_testing and np.random.random() < 0.5:\n img_A = np.fliplr(img_A)\n img_B = np.fliplr(img_B)\n\n imgs_A.append(img_A)\n imgs_B.append(img_B)\n\n imgs_A = np.array(imgs_A)/127.5 - 1.\n imgs_B = np.array(imgs_B)/127.5 - 1.\n\n return imgs_A, imgs_B\n\n def load_batch(self, batch_size=1, is_testing=False):\n data_type = \"train\" if not is_testing else \"val\"\n path = glob('./datasets/%s/%s/*' % (self.dataset_name, data_type))\n\n self.n_batches = int(len(path) / batch_size)\n\n for i in range(self.n_batches-1):\n batch = path[i*batch_size:(i+1)*batch_size]\n imgs_A, imgs_B = [], []\n for img in batch:\n img = self.imread(img)\n h, w, _ = img.shape\n half_w = int(w/2)\n img_A = img[:, :half_w, :]\n img_B = img[:, half_w:, :]\n\n img_A = scipy.misc.imresize(img_A, self.img_res)\n img_B = scipy.misc.imresize(img_B, self.img_res)\n\n if not is_testing and np.random.random() > 0.5:\n img_A = np.fliplr(img_A)\n img_B = np.fliplr(img_B)\n\n imgs_A.append(img_A)\n imgs_B.append(img_B)\n\n imgs_A = np.array(imgs_A)/127.5 - 1.\n imgs_B = np.array(imgs_B)/127.5 - 1.\n\n yield imgs_A, imgs_B\n\n\n def imread(self, path):\n return scipy.misc.imread(path, mode='RGB').astype(np.float)\n" }, { "path": "pix2pix/download_dataset.sh", "content": "mkdir datasets\nFILE=$1\nURL=https://people.eecs.berkeley.edu/~tinghuiz/projects/pix2pix/datasets/$FILE.tar.gz\nTAR_FILE=./datasets/$FILE.tar.gz\nTARGET_DIR=./datasets/$FILE/\nwget -N $URL -O $TAR_FILE\nmkdir $TARGET_DIR\ntar -zxvf $TAR_FILE -C ./datasets/\nrm $TAR_FILE\n" }, { "path": "pix2pix/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "pix2pix/pix2pix.py", "content": "from __future__ import print_function, division\nimport scipy\n\nfrom keras.datasets import mnist\nfrom keras_contrib.layers.normalization.instancenormalization import InstanceNormalization\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, Concatenate\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nimport datetime\nimport matplotlib.pyplot as plt\nimport sys\nfrom data_loader import DataLoader\nimport numpy as np\nimport os\n\nclass Pix2Pix():\n def __init__(self):\n # Input shape\n self.img_rows = 256\n self.img_cols = 256\n self.channels = 3\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n\n # Configure data loader\n self.dataset_name = 'facades'\n self.data_loader = DataLoader(dataset_name=self.dataset_name,\n img_res=(self.img_rows, self.img_cols))\n\n\n # Calculate output shape of D (PatchGAN)\n patch = int(self.img_rows / 2**4)\n self.disc_patch = (patch, patch, 1)\n\n # Number of filters in the first layer of G and D\n self.gf = 64\n self.df = 64\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminator\n self.discriminator = self.build_discriminator()\n self.discriminator.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n #-------------------------\n # Construct Computational\n # Graph of Generator\n #-------------------------\n\n # Build the generator\n self.generator = self.build_generator()\n\n # Input images and their conditioning images\n img_A = Input(shape=self.img_shape)\n img_B = Input(shape=self.img_shape)\n\n # By conditioning on B generate a fake version of A\n fake_A = self.generator(img_B)\n\n # For the combined model we will only train the generator\n self.discriminator.trainable = False\n\n # Discriminators determines validity of translated images / condition pairs\n valid = self.discriminator([fake_A, img_B])\n\n self.combined = Model(inputs=[img_A, img_B], outputs=[valid, fake_A])\n self.combined.compile(loss=['mse', 'mae'],\n loss_weights=[1, 100],\n optimizer=optimizer)\n\n def build_generator(self):\n \"\"\"U-Net Generator\"\"\"\n\n def conv2d(layer_input, filters, f_size=4, bn=True):\n \"\"\"Layers used during downsampling\"\"\"\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n if bn:\n d = BatchNormalization(momentum=0.8)(d)\n return d\n\n def deconv2d(layer_input, skip_input, filters, f_size=4, dropout_rate=0):\n \"\"\"Layers used during upsampling\"\"\"\n u = UpSampling2D(size=2)(layer_input)\n u = Conv2D(filters, kernel_size=f_size, strides=1, padding='same', activation='relu')(u)\n if dropout_rate:\n u = Dropout(dropout_rate)(u)\n u = BatchNormalization(momentum=0.8)(u)\n u = Concatenate()([u, skip_input])\n return u\n\n # Image input\n d0 = Input(shape=self.img_shape)\n\n # Downsampling\n d1 = conv2d(d0, self.gf, bn=False)\n d2 = conv2d(d1, self.gf*2)\n d3 = conv2d(d2, self.gf*4)\n d4 = conv2d(d3, self.gf*8)\n d5 = conv2d(d4, self.gf*8)\n d6 = conv2d(d5, self.gf*8)\n d7 = conv2d(d6, self.gf*8)\n\n # Upsampling\n u1 = deconv2d(d7, d6, self.gf*8)\n u2 = deconv2d(u1, d5, self.gf*8)\n u3 = deconv2d(u2, d4, self.gf*8)\n u4 = deconv2d(u3, d3, self.gf*4)\n u5 = deconv2d(u4, d2, self.gf*2)\n u6 = deconv2d(u5, d1, self.gf)\n\n u7 = UpSampling2D(size=2)(u6)\n output_img = Conv2D(self.channels, kernel_size=4, strides=1, padding='same', activation='tanh')(u7)\n\n return Model(d0, output_img)\n\n def build_discriminator(self):\n\n def d_layer(layer_input, filters, f_size=4, bn=True):\n \"\"\"Discriminator layer\"\"\"\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n if bn:\n d = BatchNormalization(momentum=0.8)(d)\n return d\n\n img_A = Input(shape=self.img_shape)\n img_B = Input(shape=self.img_shape)\n\n # Concatenate image and conditioning image by channels to produce input\n combined_imgs = Concatenate(axis=-1)([img_A, img_B])\n\n d1 = d_layer(combined_imgs, self.df, bn=False)\n d2 = d_layer(d1, self.df*2)\n d3 = d_layer(d2, self.df*4)\n d4 = d_layer(d3, self.df*8)\n\n validity = Conv2D(1, kernel_size=4, strides=1, padding='same')(d4)\n\n return Model([img_A, img_B], validity)\n\n def train(self, epochs, batch_size=1, sample_interval=50):\n\n start_time = datetime.datetime.now()\n\n # Adversarial loss ground truths\n valid = np.ones((batch_size,) + self.disc_patch)\n fake = np.zeros((batch_size,) + self.disc_patch)\n\n for epoch in range(epochs):\n for batch_i, (imgs_A, imgs_B) in enumerate(self.data_loader.load_batch(batch_size)):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n # Condition on B and generate a translated version\n fake_A = self.generator.predict(imgs_B)\n\n # Train the discriminators (original images = real / generated = Fake)\n d_loss_real = self.discriminator.train_on_batch([imgs_A, imgs_B], valid)\n d_loss_fake = self.discriminator.train_on_batch([fake_A, imgs_B], fake)\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n\n # -----------------\n # Train Generator\n # -----------------\n\n # Train the generators\n g_loss = self.combined.train_on_batch([imgs_A, imgs_B], [valid, imgs_A])\n\n elapsed_time = datetime.datetime.now() - start_time\n # Plot the progress\n print (\"[Epoch %d/%d] [Batch %d/%d] [D loss: %f, acc: %3d%%] [G loss: %f] time: %s\" % (epoch, epochs,\n batch_i, self.data_loader.n_batches,\n d_loss[0], 100*d_loss[1],\n g_loss[0],\n elapsed_time))\n\n # If at save interval => save generated image samples\n if batch_i % sample_interval == 0:\n self.sample_images(epoch, batch_i)\n\n def sample_images(self, epoch, batch_i):\n os.makedirs('images/%s' % self.dataset_name, exist_ok=True)\n r, c = 3, 3\n\n imgs_A, imgs_B = self.data_loader.load_data(batch_size=3, is_testing=True)\n fake_A = self.generator.predict(imgs_B)\n\n gen_imgs = np.concatenate([imgs_B, fake_A, imgs_A])\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n titles = ['Condition', 'Generated', 'Original']\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt])\n axs[i, j].set_title(titles[i])\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/%s/%d_%d.png\" % (self.dataset_name, epoch, batch_i))\n plt.close()\n\n\nif __name__ == '__main__':\n gan = Pix2Pix()\n gan.train(epochs=200, batch_size=1, sample_interval=200)\n" }, { "path": "pix2pix/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "pixelda/data_loader.py", "content": "import scipy\nfrom glob import glob\nimport numpy as np\nfrom keras.datasets import mnist\nfrom skimage.transform import resize as imresize\nimport pickle\nimport os\nimport urllib\nimport gzip\n\nclass DataLoader():\n \"\"\"Loads images from MNIST (domain A) and MNIST-M (domain B)\"\"\"\n def __init__(self, img_res=(128, 128)):\n self.img_res = img_res\n\n self.mnistm_url = 'https://github.com/VanushVaswani/keras_mnistm/releases/download/1.0/keras_mnistm.pkl.gz'\n\n self.setup_mnist(img_res)\n self.setup_mnistm(img_res)\n\n def normalize(self, images):\n return images.astype(np.float32) / 127.5 - 1.\n\n def setup_mnist(self, img_res):\n\n print (\"Setting up MNIST...\")\n\n if not os.path.exists('datasets/mnist_x.npy'):\n # Load the dataset\n (mnist_X, mnist_y), (_, _) = mnist.load_data()\n\n # Normalize and rescale images\n mnist_X = self.normalize(mnist_X)\n mnist_X = np.array([imresize(x, img_res) for x in mnist_X])\n mnist_X = np.expand_dims(mnist_X, axis=-1)\n mnist_X = np.repeat(mnist_X, 3, axis=-1)\n\n self.mnist_X, self.mnist_y = mnist_X, mnist_y\n\n # Save formatted images\n np.save('datasets/mnist_x.npy', self.mnist_X)\n np.save('datasets/mnist_y.npy', self.mnist_y)\n else:\n self.mnist_X = np.load('datasets/mnist_x.npy')\n self.mnist_y = np.load('datasets/mnist_y.npy')\n\n print (\"+ Done.\")\n\n def setup_mnistm(self, img_res):\n\n print (\"Setting up MNIST-M...\")\n\n if not os.path.exists('datasets/mnistm_x.npy'):\n\n # Download the MNIST-M pkl file\n filepath = 'datasets/keras_mnistm.pkl.gz'\n if not os.path.exists(filepath.replace('.gz', '')):\n print('+ Downloading ' + self.mnistm_url)\n data = urllib.request.urlopen(self.mnistm_url)\n with open(filepath, 'wb') as f:\n f.write(data.read())\n with open(filepath.replace('.gz', ''), 'wb') as out_f, \\\n gzip.GzipFile(filepath) as zip_f:\n out_f.write(zip_f.read())\n os.unlink(filepath)\n\n # load MNIST-M images from pkl file\n with open('datasets/keras_mnistm.pkl', \"rb\") as f:\n data = pickle.load(f, encoding='bytes')\n\n # Normalize and rescale images\n mnistm_X = np.array(data[b'train'])\n mnistm_X = self.normalize(mnistm_X)\n mnistm_X = np.array([imresize(x, img_res) for x in mnistm_X])\n\n self.mnistm_X, self.mnistm_y = mnistm_X, self.mnist_y.copy()\n\n # Save formatted images\n np.save('datasets/mnistm_x.npy', self.mnistm_X)\n np.save('datasets/mnistm_y.npy', self.mnistm_y)\n else:\n self.mnistm_X = np.load('datasets/mnistm_x.npy')\n self.mnistm_y = np.load('datasets/mnistm_y.npy')\n\n print (\"+ Done.\")\n\n\n def load_data(self, domain, batch_size=1):\n\n X = self.mnist_X if domain == 'A' else self.mnistm_X\n y = self.mnist_y if domain == 'A' else self.mnistm_y\n\n idx = np.random.choice(list(range(len(X))), size=batch_size)\n\n return X[idx], y[idx]\n" }, { "path": "pixelda/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "pixelda/pixelda.py", "content": "from __future__ import print_function, division\nimport scipy\n\nfrom keras.datasets import mnist\nfrom keras_contrib.layers.normalization.instancenormalization import InstanceNormalization\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, Concatenate\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D, Add\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nfrom keras.utils import to_categorical\nimport datetime\nimport matplotlib.pyplot as plt\nimport sys\nfrom data_loader import DataLoader\nimport numpy as np\nimport os\n\nclass PixelDA():\n def __init__(self):\n # Input shape\n self.img_rows = 32\n self.img_cols = 32\n self.channels = 3\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.num_classes = 10\n\n # Configure MNIST and MNIST-M data loader\n self.data_loader = DataLoader(img_res=(self.img_rows, self.img_cols))\n\n # Loss weights\n lambda_adv = 10\n lambda_clf = 1\n\n # Calculate output shape of D (PatchGAN)\n patch = int(self.img_rows / 2**4)\n self.disc_patch = (patch, patch, 1)\n\n # Number of residual blocks in the generator\n self.residual_blocks = 6\n\n optimizer = Adam(0.0002, 0.5)\n\n # Number of filters in first layer of discriminator and classifier\n self.df = 64\n self.cf = 64\n\n # Build and compile the discriminators\n self.discriminator = self.build_discriminator()\n self.discriminator.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Build the generator\n self.generator = self.build_generator()\n\n # Build the task (classification) network\n self.clf = self.build_classifier()\n\n # Input images from both domains\n img_A = Input(shape=self.img_shape)\n img_B = Input(shape=self.img_shape)\n\n # Translate images from domain A to domain B\n fake_B = self.generator(img_A)\n\n # Classify the translated image\n class_pred = self.clf(fake_B)\n\n # For the combined model we will only train the generator and classifier\n self.discriminator.trainable = False\n\n # Discriminator determines validity of translated images\n valid = self.discriminator(fake_B)\n\n self.combined = Model(img_A, [valid, class_pred])\n self.combined.compile(loss=['mse', 'categorical_crossentropy'],\n loss_weights=[lambda_adv, lambda_clf],\n optimizer=optimizer,\n metrics=['accuracy'])\n\n def build_generator(self):\n \"\"\"Resnet Generator\"\"\"\n\n def residual_block(layer_input):\n \"\"\"Residual block described in paper\"\"\"\n d = Conv2D(64, kernel_size=3, strides=1, padding='same')(layer_input)\n d = BatchNormalization(momentum=0.8)(d)\n d = Activation('relu')(d)\n d = Conv2D(64, kernel_size=3, strides=1, padding='same')(d)\n d = BatchNormalization(momentum=0.8)(d)\n d = Add()([d, layer_input])\n return d\n\n # Image input\n img = Input(shape=self.img_shape)\n\n l1 = Conv2D(64, kernel_size=3, padding='same', activation='relu')(img)\n\n # Propogate signal through residual blocks\n r = residual_block(l1)\n for _ in range(self.residual_blocks - 1):\n r = residual_block(r)\n\n output_img = Conv2D(self.channels, kernel_size=3, padding='same', activation='tanh')(r)\n\n return Model(img, output_img)\n\n\n def build_discriminator(self):\n\n def d_layer(layer_input, filters, f_size=4, normalization=True):\n \"\"\"Discriminator layer\"\"\"\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n if normalization:\n d = InstanceNormalization()(d)\n return d\n\n img = Input(shape=self.img_shape)\n\n d1 = d_layer(img, self.df, normalization=False)\n d2 = d_layer(d1, self.df*2)\n d3 = d_layer(d2, self.df*4)\n d4 = d_layer(d3, self.df*8)\n\n validity = Conv2D(1, kernel_size=4, strides=1, padding='same')(d4)\n\n return Model(img, validity)\n\n def build_classifier(self):\n\n def clf_layer(layer_input, filters, f_size=4, normalization=True):\n \"\"\"Classifier layer\"\"\"\n d = Conv2D(filters, kernel_size=f_size, strides=2, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n if normalization:\n d = InstanceNormalization()(d)\n return d\n\n img = Input(shape=self.img_shape)\n\n c1 = clf_layer(img, self.cf, normalization=False)\n c2 = clf_layer(c1, self.cf*2)\n c3 = clf_layer(c2, self.cf*4)\n c4 = clf_layer(c3, self.cf*8)\n c5 = clf_layer(c4, self.cf*8)\n\n class_pred = Dense(self.num_classes, activation='softmax')(Flatten()(c5))\n\n return Model(img, class_pred)\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n half_batch = int(batch_size / 2)\n\n # Classification accuracy on 100 last batches of domain B\n test_accs = []\n\n # Adversarial ground truths\n valid = np.ones((batch_size, *self.disc_patch))\n fake = np.zeros((batch_size, *self.disc_patch))\n\n for epoch in range(epochs):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n imgs_A, labels_A = self.data_loader.load_data(domain=\"A\", batch_size=batch_size)\n imgs_B, labels_B = self.data_loader.load_data(domain=\"B\", batch_size=batch_size)\n\n # Translate images from domain A to domain B\n fake_B = self.generator.predict(imgs_A)\n\n # Train the discriminators (original images = real / translated = Fake)\n d_loss_real = self.discriminator.train_on_batch(imgs_B, valid)\n d_loss_fake = self.discriminator.train_on_batch(fake_B, fake)\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n\n\n # --------------------------------\n # Train Generator and Classifier\n # --------------------------------\n\n # One-hot encoding of labels\n labels_A = to_categorical(labels_A, num_classes=self.num_classes)\n\n # Train the generator and classifier\n g_loss = self.combined.train_on_batch(imgs_A, [valid, labels_A])\n\n #-----------------------\n # Evaluation (domain B)\n #-----------------------\n\n pred_B = self.clf.predict(imgs_B)\n test_acc = np.mean(np.argmax(pred_B, axis=1) == labels_B)\n\n # Add accuracy to list of last 100 accuracy measurements\n test_accs.append(test_acc)\n if len(test_accs) > 100:\n test_accs.pop(0)\n\n\n # Plot the progress\n print ( \"%d : [D - loss: %.5f, acc: %3d%%], [G - loss: %.5f], [clf - loss: %.5f, acc: %3d%%, test_acc: %3d%% (%3d%%)]\" % \\\n (epoch, d_loss[0], 100*float(d_loss[1]),\n g_loss[1], g_loss[2], 100*float(g_loss[-1]),\n 100*float(test_acc), 100*float(np.mean(test_accs))))\n\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n r, c = 2, 5\n\n imgs_A, _ = self.data_loader.load_data(domain=\"A\", batch_size=5)\n\n # Translate images to the other domain\n fake_B = self.generator.predict(imgs_A)\n\n gen_imgs = np.concatenate([imgs_A, fake_B])\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n #titles = ['Original', 'Translated']\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt])\n #axs[i, j].set_title(titles[i])\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/%d.png\" % (epoch))\n plt.close()\n\n\nif __name__ == '__main__':\n gan = PixelDA()\n gan.train(epochs=30000, batch_size=32, sample_interval=500)\n" }, { "path": "pixelda/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "pixelda/test.py", "content": "from __future__ import print_function, division\nimport scipy\n\nimport datetime\nimport matplotlib.pyplot as plt\nimport sys\nfrom data_loader import DataLoader\nimport numpy as np\nimport os\n\n\n# Configure MNIST and MNIST-M data loader\ndata_loader = DataLoader(img_res=(32, 32))\n\nmnist, _ = data_loader.load_data(domain=\"A\", batch_size=25)\nmnistm, _ = data_loader.load_data(domain=\"B\", batch_size=25)\n\nr, c = 5, 5\n\nfor img_i, imgs in enumerate([mnist, mnistm]):\n\n #titles = ['Original', 'Translated']\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(imgs[cnt])\n #axs[i, j].set_title(titles[i])\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"%d.png\" % (img_i))\n plt.close()\n" }, { "path": "requirements.txt", "content": "keras\ngit+https://www.github.com/keras-team/keras-contrib.git\nmatplotlib\nnumpy\nscipy\npillow\n#urllib\n#skimage\nscikit-image\n#gzip\n#pickle\n" }, { "path": "sgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "sgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "sgan/sgan.py", "content": "from __future__ import print_function, division\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, multiply, GaussianNoise\nfrom keras.layers import BatchNormalization, Activation, Embedding, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nfrom keras import losses\nfrom keras.utils import to_categorical\nimport keras.backend as K\n\nimport matplotlib.pyplot as plt\n\nimport numpy as np\n\nclass SGAN:\n def __init__(self):\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.num_classes = 10\n self.latent_dim = 100\n\n optimizer = Adam(0.0002, 0.5)\n\n # Build and compile the discriminator\n self.discriminator = self.build_discriminator()\n self.discriminator.compile(\n loss=['binary_crossentropy', 'categorical_crossentropy'],\n loss_weights=[0.5, 0.5],\n optimizer=optimizer,\n metrics=['accuracy']\n )\n\n # Build the generator\n self.generator = self.build_generator()\n\n # The generator takes noise as input and generates imgs\n noise = Input(shape=(100,))\n img = self.generator(noise)\n\n # For the combined model we will only train the generator\n self.discriminator.trainable = False\n\n # The valid takes generated images as input and determines validity\n valid, _ = self.discriminator(img)\n\n # The combined model (stacked generator and discriminator)\n # Trains generator to fool discriminator\n self.combined = Model(noise, valid)\n self.combined.compile(loss=['binary_crossentropy'], optimizer=optimizer)\n\n def build_generator(self):\n\n model = Sequential()\n\n model.add(Dense(128 * 7 * 7, activation=\"relu\", input_dim=self.latent_dim))\n model.add(Reshape((7, 7, 128)))\n model.add(BatchNormalization(momentum=0.8))\n model.add(UpSampling2D())\n model.add(Conv2D(128, kernel_size=3, padding=\"same\"))\n model.add(Activation(\"relu\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(UpSampling2D())\n model.add(Conv2D(64, kernel_size=3, padding=\"same\"))\n model.add(Activation(\"relu\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Conv2D(1, kernel_size=3, padding=\"same\"))\n model.add(Activation(\"tanh\"))\n\n model.summary()\n\n noise = Input(shape=(self.latent_dim,))\n img = model(noise)\n\n return Model(noise, img)\n\n def build_discriminator(self):\n\n model = Sequential()\n\n model.add(Conv2D(32, kernel_size=3, strides=2, input_shape=self.img_shape, padding=\"same\"))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\n model.add(ZeroPadding2D(padding=((0,1),(0,1))))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Conv2D(128, kernel_size=3, strides=2, padding=\"same\"))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Conv2D(256, kernel_size=3, strides=1, padding=\"same\"))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Flatten())\n\n model.summary()\n\n img = Input(shape=self.img_shape)\n\n features = model(img)\n valid = Dense(1, activation=\"sigmoid\")(features)\n label = Dense(self.num_classes+1, activation=\"softmax\")(features)\n\n return Model(img, [valid, label])\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n # Load the dataset\n (X_train, y_train), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\n X_train = np.expand_dims(X_train, axis=3)\n y_train = y_train.reshape(-1, 1)\n\n # Class weights:\n # To balance the difference in occurences of digit class labels.\n # 50% of labels that the discriminator trains on are 'fake'.\n # Weight = 1 / frequency\n half_batch = batch_size // 2\n cw1 = {0: 1, 1: 1}\n cw2 = {i: self.num_classes / half_batch for i in range(self.num_classes)}\n cw2[self.num_classes] = 1 / half_batch\n\n # Adversarial ground truths\n valid = np.ones((batch_size, 1))\n fake = np.zeros((batch_size, 1))\n\n for epoch in range(epochs):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n # Select a random batch of images\n idx = np.random.randint(0, X_train.shape[0], batch_size)\n imgs = X_train[idx]\n\n # Sample noise and generate a batch of new images\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n\n # One-hot encoding of labels\n labels = to_categorical(y_train[idx], num_classes=self.num_classes+1)\n fake_labels = to_categorical(np.full((batch_size, 1), self.num_classes), num_classes=self.num_classes+1)\n\n # Train the discriminator\n d_loss_real = self.discriminator.train_on_batch(imgs, [valid, labels], class_weight=[cw1, cw2])\n d_loss_fake = self.discriminator.train_on_batch(gen_imgs, [fake, fake_labels], class_weight=[cw1, cw2])\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n\n\n # ---------------------\n # Train Generator\n # ---------------------\n\n g_loss = self.combined.train_on_batch(noise, valid, class_weight=[cw1, cw2])\n\n # Plot the progress\n print (\"%d [D loss: %f, acc: %.2f%%, op_acc: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[3], 100*d_loss[4], g_loss))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n r, c = 5, 5\n noise = np.random.normal(0, 1, (r * c, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/mnist_%d.png\" % epoch)\n plt.close()\n\n def save_model(self):\n\n def save(model, model_name):\n model_path = \"saved_model/%s.json\" % model_name\n weights_path = \"saved_model/%s_weights.hdf5\" % model_name\n options = {\"file_arch\": model_path,\n \"file_weight\": weights_path}\n json_string = model.to_json()\n open(options['file_arch'], 'w').write(json_string)\n model.save_weights(options['file_weight'])\n\n save(self.generator, \"mnist_sgan_generator\")\n save(self.discriminator, \"mnist_sgan_discriminator\")\n save(self.combined, \"mnist_sgan_adversarial\")\n\n\nif __name__ == '__main__':\n sgan = SGAN()\n sgan.train(epochs=20000, batch_size=32, sample_interval=50)\n\n" }, { "path": "srgan/data_loader.py", "content": "import scipy\nfrom glob import glob\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nclass DataLoader():\n def __init__(self, dataset_name, img_res=(128, 128)):\n self.dataset_name = dataset_name\n self.img_res = img_res\n\n def load_data(self, batch_size=1, is_testing=False):\n data_type = \"train\" if not is_testing else \"test\"\n \n path = glob('./datasets/%s/*' % (self.dataset_name))\n\n batch_images = np.random.choice(path, size=batch_size)\n\n imgs_hr = []\n imgs_lr = []\n for img_path in batch_images:\n img = self.imread(img_path)\n\n h, w = self.img_res\n low_h, low_w = int(h / 4), int(w / 4)\n\n img_hr = scipy.misc.imresize(img, self.img_res)\n img_lr = scipy.misc.imresize(img, (low_h, low_w))\n\n # If training => do random flip\n if not is_testing and np.random.random() < 0.5:\n img_hr = np.fliplr(img_hr)\n img_lr = np.fliplr(img_lr)\n\n imgs_hr.append(img_hr)\n imgs_lr.append(img_lr)\n\n imgs_hr = np.array(imgs_hr) / 127.5 - 1.\n imgs_lr = np.array(imgs_lr) / 127.5 - 1.\n\n return imgs_hr, imgs_lr\n\n\n def imread(self, path):\n return scipy.misc.imread(path, mode='RGB').astype(np.float)\n" }, { "path": "srgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "srgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "srgan/srgan.py", "content": "\"\"\"\nSuper-resolution of CelebA using Generative Adversarial Networks.\n\nThe dataset can be downloaded from: https://www.dropbox.com/sh/8oqt9vytwxb3s4r/AADIKlz8PR9zr6Y20qbkunrba/Img/img_align_celeba.zip?dl=0\n\nInstrustion on running the script:\n1. Download the dataset from the provided link\n2. Save the folder 'img_align_celeba' to 'datasets/'\n4. Run the sript using command 'python srgan.py'\n\"\"\"\n\nfrom __future__ import print_function, division\nimport scipy\n\nfrom keras.datasets import mnist\nfrom keras_contrib.layers.normalization.instancenormalization import InstanceNormalization\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, Concatenate\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D, Add\nfrom keras.layers.advanced_activations import PReLU, LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.applications import VGG19\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nimport datetime\nimport matplotlib.pyplot as plt\nimport sys\nfrom data_loader import DataLoader\nimport numpy as np\nimport os\n\nimport keras.backend as K\n\nclass SRGAN():\n def __init__(self):\n # Input shape\n self.channels = 3\n self.lr_height = 64 # Low resolution height\n self.lr_width = 64 # Low resolution width\n self.lr_shape = (self.lr_height, self.lr_width, self.channels)\n self.hr_height = self.lr_height*4 # High resolution height\n self.hr_width = self.lr_width*4 # High resolution width\n self.hr_shape = (self.hr_height, self.hr_width, self.channels)\n\n # Number of residual blocks in the generator\n self.n_residual_blocks = 16\n\n optimizer = Adam(0.0002, 0.5)\n\n # We use a pre-trained VGG19 model to extract image features from the high resolution\n # and the generated high resolution images and minimize the mse between them\n self.vgg = self.build_vgg()\n self.vgg.trainable = False\n self.vgg.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Configure data loader\n self.dataset_name = 'img_align_celeba'\n self.data_loader = DataLoader(dataset_name=self.dataset_name,\n img_res=(self.hr_height, self.hr_width))\n\n # Calculate output shape of D (PatchGAN)\n patch = int(self.hr_height / 2**4)\n self.disc_patch = (patch, patch, 1)\n\n # Number of filters in the first layer of G and D\n self.gf = 64\n self.df = 64\n\n # Build and compile the discriminator\n self.discriminator = self.build_discriminator()\n self.discriminator.compile(loss='mse',\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Build the generator\n self.generator = self.build_generator()\n\n # High res. and low res. images\n img_hr = Input(shape=self.hr_shape)\n img_lr = Input(shape=self.lr_shape)\n\n # Generate high res. version from low res.\n fake_hr = self.generator(img_lr)\n\n # Extract image features of the generated img\n fake_features = self.vgg(fake_hr)\n\n # For the combined model we will only train the generator\n self.discriminator.trainable = False\n\n # Discriminator determines validity of generated high res. images\n validity = self.discriminator(fake_hr)\n\n self.combined = Model([img_lr, img_hr], [validity, fake_features])\n self.combined.compile(loss=['binary_crossentropy', 'mse'],\n loss_weights=[1e-3, 1],\n optimizer=optimizer)\n\n\n def build_vgg(self):\n \"\"\"\n Builds a pre-trained VGG19 model that outputs image features extracted at the\n third block of the model\n \"\"\"\n vgg = VGG19(weights=\"imagenet\")\n # Set outputs to outputs of last conv. layer in block 3\n # See architecture at: https://github.com/keras-team/keras/blob/master/keras/applications/vgg19.py\n vgg.outputs = [vgg.layers[9].output]\n\n img = Input(shape=self.hr_shape)\n\n # Extract image features\n img_features = vgg(img)\n\n return Model(img, img_features)\n\n def build_generator(self):\n\n def residual_block(layer_input, filters):\n \"\"\"Residual block described in paper\"\"\"\n d = Conv2D(filters, kernel_size=3, strides=1, padding='same')(layer_input)\n d = Activation('relu')(d)\n d = BatchNormalization(momentum=0.8)(d)\n d = Conv2D(filters, kernel_size=3, strides=1, padding='same')(d)\n d = BatchNormalization(momentum=0.8)(d)\n d = Add()([d, layer_input])\n return d\n\n def deconv2d(layer_input):\n \"\"\"Layers used during upsampling\"\"\"\n u = UpSampling2D(size=2)(layer_input)\n u = Conv2D(256, kernel_size=3, strides=1, padding='same')(u)\n u = Activation('relu')(u)\n return u\n\n # Low resolution image input\n img_lr = Input(shape=self.lr_shape)\n\n # Pre-residual block\n c1 = Conv2D(64, kernel_size=9, strides=1, padding='same')(img_lr)\n c1 = Activation('relu')(c1)\n\n # Propogate through residual blocks\n r = residual_block(c1, self.gf)\n for _ in range(self.n_residual_blocks - 1):\n r = residual_block(r, self.gf)\n\n # Post-residual block\n c2 = Conv2D(64, kernel_size=3, strides=1, padding='same')(r)\n c2 = BatchNormalization(momentum=0.8)(c2)\n c2 = Add()([c2, c1])\n\n # Upsampling\n u1 = deconv2d(c2)\n u2 = deconv2d(u1)\n\n # Generate high resolution output\n gen_hr = Conv2D(self.channels, kernel_size=9, strides=1, padding='same', activation='tanh')(u2)\n\n return Model(img_lr, gen_hr)\n\n def build_discriminator(self):\n\n def d_block(layer_input, filters, strides=1, bn=True):\n \"\"\"Discriminator layer\"\"\"\n d = Conv2D(filters, kernel_size=3, strides=strides, padding='same')(layer_input)\n d = LeakyReLU(alpha=0.2)(d)\n if bn:\n d = BatchNormalization(momentum=0.8)(d)\n return d\n\n # Input img\n d0 = Input(shape=self.hr_shape)\n\n d1 = d_block(d0, self.df, bn=False)\n d2 = d_block(d1, self.df, strides=2)\n d3 = d_block(d2, self.df*2)\n d4 = d_block(d3, self.df*2, strides=2)\n d5 = d_block(d4, self.df*4)\n d6 = d_block(d5, self.df*4, strides=2)\n d7 = d_block(d6, self.df*8)\n d8 = d_block(d7, self.df*8, strides=2)\n\n d9 = Dense(self.df*16)(d8)\n d10 = LeakyReLU(alpha=0.2)(d9)\n validity = Dense(1, activation='sigmoid')(d10)\n\n return Model(d0, validity)\n\n def train(self, epochs, batch_size=1, sample_interval=50):\n\n start_time = datetime.datetime.now()\n\n for epoch in range(epochs):\n\n # ----------------------\n # Train Discriminator\n # ----------------------\n\n # Sample images and their conditioning counterparts\n imgs_hr, imgs_lr = self.data_loader.load_data(batch_size)\n\n # From low res. image generate high res. version\n fake_hr = self.generator.predict(imgs_lr)\n\n valid = np.ones((batch_size,) + self.disc_patch)\n fake = np.zeros((batch_size,) + self.disc_patch)\n\n # Train the discriminators (original images = real / generated = Fake)\n d_loss_real = self.discriminator.train_on_batch(imgs_hr, valid)\n d_loss_fake = self.discriminator.train_on_batch(fake_hr, fake)\n d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n\n # ------------------\n # Train Generator\n # ------------------\n\n # Sample images and their conditioning counterparts\n imgs_hr, imgs_lr = self.data_loader.load_data(batch_size)\n\n # The generators want the discriminators to label the generated images as real\n valid = np.ones((batch_size,) + self.disc_patch)\n\n # Extract ground truth image features using pre-trained VGG19 model\n image_features = self.vgg.predict(imgs_hr)\n\n # Train the generators\n g_loss = self.combined.train_on_batch([imgs_lr, imgs_hr], [valid, image_features])\n\n elapsed_time = datetime.datetime.now() - start_time\n # Plot the progress\n print (\"%d time: %s\" % (epoch, elapsed_time))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n os.makedirs('images/%s' % self.dataset_name, exist_ok=True)\n r, c = 2, 2\n\n imgs_hr, imgs_lr = self.data_loader.load_data(batch_size=2, is_testing=True)\n fake_hr = self.generator.predict(imgs_lr)\n\n # Rescale images 0 - 1\n imgs_lr = 0.5 * imgs_lr + 0.5\n fake_hr = 0.5 * fake_hr + 0.5\n imgs_hr = 0.5 * imgs_hr + 0.5\n\n # Save generated images and the high resolution originals\n titles = ['Generated', 'Original']\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for row in range(r):\n for col, image in enumerate([fake_hr, imgs_hr]):\n axs[row, col].imshow(image[row])\n axs[row, col].set_title(titles[col])\n axs[row, col].axis('off')\n cnt += 1\n fig.savefig(\"images/%s/%d.png\" % (self.dataset_name, epoch))\n plt.close()\n\n # Save low resolution images for comparison\n for i in range(r):\n fig = plt.figure()\n plt.imshow(imgs_lr[i])\n fig.savefig('images/%s/%d_lowres%d.png' % (self.dataset_name, epoch, i))\n plt.close()\n\nif __name__ == '__main__':\n gan = SRGAN()\n gan.train(epochs=30000, batch_size=1, sample_interval=50)\n" }, { "path": "wgan/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "wgan/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "wgan/wgan.py", "content": "from __future__ import print_function, division\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import RMSprop\n\nimport keras.backend as K\n\nimport matplotlib.pyplot as plt\n\nimport sys\n\nimport numpy as np\n\nclass WGAN():\n def __init__(self):\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.latent_dim = 100\n\n # Following parameter and optimizer set as recommended in paper\n self.n_critic = 5\n self.clip_value = 0.01\n optimizer = RMSprop(lr=0.00005)\n\n # Build and compile the critic\n self.critic = self.build_critic()\n self.critic.compile(loss=self.wasserstein_loss,\n optimizer=optimizer,\n metrics=['accuracy'])\n\n # Build the generator\n self.generator = self.build_generator()\n\n # The generator takes noise as input and generated imgs\n z = Input(shape=(self.latent_dim,))\n img = self.generator(z)\n\n # For the combined model we will only train the generator\n self.critic.trainable = False\n\n # The critic takes generated images as input and determines validity\n valid = self.critic(img)\n\n # The combined model (stacked generator and critic)\n self.combined = Model(z, valid)\n self.combined.compile(loss=self.wasserstein_loss,\n optimizer=optimizer,\n metrics=['accuracy'])\n\n def wasserstein_loss(self, y_true, y_pred):\n return K.mean(y_true * y_pred)\n\n def build_generator(self):\n\n model = Sequential()\n\n model.add(Dense(128 * 7 * 7, activation=\"relu\", input_dim=self.latent_dim))\n model.add(Reshape((7, 7, 128)))\n model.add(UpSampling2D())\n model.add(Conv2D(128, kernel_size=4, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Activation(\"relu\"))\n model.add(UpSampling2D())\n model.add(Conv2D(64, kernel_size=4, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Activation(\"relu\"))\n model.add(Conv2D(self.channels, kernel_size=4, padding=\"same\"))\n model.add(Activation(\"tanh\"))\n\n model.summary()\n\n noise = Input(shape=(self.latent_dim,))\n img = model(noise)\n\n return Model(noise, img)\n\n def build_critic(self):\n\n model = Sequential()\n\n model.add(Conv2D(16, kernel_size=3, strides=2, input_shape=self.img_shape, padding=\"same\"))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(32, kernel_size=3, strides=2, padding=\"same\"))\n model.add(ZeroPadding2D(padding=((0,1),(0,1))))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(128, kernel_size=3, strides=1, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Flatten())\n model.add(Dense(1))\n\n model.summary()\n\n img = Input(shape=self.img_shape)\n validity = model(img)\n\n return Model(img, validity)\n\n def train(self, epochs, batch_size=128, sample_interval=50):\n\n # Load the dataset\n (X_train, _), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\n X_train = np.expand_dims(X_train, axis=3)\n\n # Adversarial ground truths\n valid = -np.ones((batch_size, 1))\n fake = np.ones((batch_size, 1))\n\n for epoch in range(epochs):\n\n for _ in range(self.n_critic):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n # Select a random batch of images\n idx = np.random.randint(0, X_train.shape[0], batch_size)\n imgs = X_train[idx]\n \n # Sample noise as generator input\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n\n # Generate a batch of new images\n gen_imgs = self.generator.predict(noise)\n\n # Train the critic\n d_loss_real = self.critic.train_on_batch(imgs, valid)\n d_loss_fake = self.critic.train_on_batch(gen_imgs, fake)\n d_loss = 0.5 * np.add(d_loss_fake, d_loss_real)\n\n # Clip critic weights\n for l in self.critic.layers:\n weights = l.get_weights()\n weights = [np.clip(w, -self.clip_value, self.clip_value) for w in weights]\n l.set_weights(weights)\n\n\n # ---------------------\n # Train Generator\n # ---------------------\n\n g_loss = self.combined.train_on_batch(noise, valid)\n\n # Plot the progress\n print (\"%d [D loss: %f] [G loss: %f]\" % (epoch, 1 - d_loss[0], 1 - g_loss[0]))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n r, c = 5, 5\n noise = np.random.normal(0, 1, (r * c, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/mnist_%d.png\" % epoch)\n plt.close()\n\n\nif __name__ == '__main__':\n wgan = WGAN()\n wgan.train(epochs=4000, batch_size=32, sample_interval=50)\n" }, { "path": "wgan_gp/images/.gitignore", "content": "*\n!.gitignore" }, { "path": "wgan_gp/saved_model/.gitignore", "content": "*\n!.gitignore" }, { "path": "wgan_gp/wgan_gp.py", "content": "\n# Large amount of credit goes to:\n# https://github.com/keras-team/keras-contrib/blob/master/examples/improved_wgan.py\n# which I've used as a reference for this implementation\n\nfrom __future__ import print_function, division\n\nfrom keras.datasets import mnist\nfrom keras.layers.merge import _Merge\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import RMSprop\nfrom functools import partial\n\nimport keras.backend as K\n\nimport matplotlib.pyplot as plt\n\nimport sys\n\nimport numpy as np\n\nclass RandomWeightedAverage(_Merge):\n \"\"\"Provides a (random) weighted average between real and generated image samples\"\"\"\n def _merge_function(self, inputs):\n alpha = K.random_uniform((32, 1, 1, 1))\n return (alpha * inputs[0]) + ((1 - alpha) * inputs[1])\n\nclass WGANGP():\n def __init__(self):\n self.img_rows = 28\n self.img_cols = 28\n self.channels = 1\n self.img_shape = (self.img_rows, self.img_cols, self.channels)\n self.latent_dim = 100\n\n # Following parameter and optimizer set as recommended in paper\n self.n_critic = 5\n optimizer = RMSprop(lr=0.00005)\n\n # Build the generator and critic\n self.generator = self.build_generator()\n self.critic = self.build_critic()\n\n #-------------------------------\n # Construct Computational Graph\n # for the Critic\n #-------------------------------\n\n # Freeze generator's layers while training critic\n self.generator.trainable = False\n\n # Image input (real sample)\n real_img = Input(shape=self.img_shape)\n\n # Noise input\n z_disc = Input(shape=(self.latent_dim,))\n # Generate image based of noise (fake sample)\n fake_img = self.generator(z_disc)\n\n # Discriminator determines validity of the real and fake images\n fake = self.critic(fake_img)\n valid = self.critic(real_img)\n\n # Construct weighted average between real and fake images\n interpolated_img = RandomWeightedAverage()([real_img, fake_img])\n # Determine validity of weighted sample\n validity_interpolated = self.critic(interpolated_img)\n\n # Use Python partial to provide loss function with additional\n # 'averaged_samples' argument\n partial_gp_loss = partial(self.gradient_penalty_loss,\n averaged_samples=interpolated_img)\n partial_gp_loss.__name__ = 'gradient_penalty' # Keras requires function names\n\n self.critic_model = Model(inputs=[real_img, z_disc],\n outputs=[valid, fake, validity_interpolated])\n self.critic_model.compile(loss=[self.wasserstein_loss,\n self.wasserstein_loss,\n partial_gp_loss],\n optimizer=optimizer,\n loss_weights=[1, 1, 10])\n #-------------------------------\n # Construct Computational Graph\n # for Generator\n #-------------------------------\n\n # For the generator we freeze the critic's layers\n self.critic.trainable = False\n self.generator.trainable = True\n\n # Sampled noise for input to generator\n z_gen = Input(shape=(self.latent_dim,))\n # Generate images based of noise\n img = self.generator(z_gen)\n # Discriminator determines validity\n valid = self.critic(img)\n # Defines generator model\n self.generator_model = Model(z_gen, valid)\n self.generator_model.compile(loss=self.wasserstein_loss, optimizer=optimizer)\n\n\n def gradient_penalty_loss(self, y_true, y_pred, averaged_samples):\n \"\"\"\n Computes gradient penalty based on prediction and weighted real / fake samples\n \"\"\"\n gradients = K.gradients(y_pred, averaged_samples)[0]\n # compute the euclidean norm by squaring ...\n gradients_sqr = K.square(gradients)\n # ... summing over the rows ...\n gradients_sqr_sum = K.sum(gradients_sqr,\n axis=np.arange(1, len(gradients_sqr.shape)))\n # ... and sqrt\n gradient_l2_norm = K.sqrt(gradients_sqr_sum)\n # compute lambda * (1 - ||grad||)^2 still for each single sample\n gradient_penalty = K.square(1 - gradient_l2_norm)\n # return the mean as loss over all the batch samples\n return K.mean(gradient_penalty)\n\n\n def wasserstein_loss(self, y_true, y_pred):\n return K.mean(y_true * y_pred)\n\n def build_generator(self):\n\n model = Sequential()\n\n model.add(Dense(128 * 7 * 7, activation=\"relu\", input_dim=self.latent_dim))\n model.add(Reshape((7, 7, 128)))\n model.add(UpSampling2D())\n model.add(Conv2D(128, kernel_size=4, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Activation(\"relu\"))\n model.add(UpSampling2D())\n model.add(Conv2D(64, kernel_size=4, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(Activation(\"relu\"))\n model.add(Conv2D(self.channels, kernel_size=4, padding=\"same\"))\n model.add(Activation(\"tanh\"))\n\n model.summary()\n\n noise = Input(shape=(self.latent_dim,))\n img = model(noise)\n\n return Model(noise, img)\n\n def build_critic(self):\n\n model = Sequential()\n\n model.add(Conv2D(16, kernel_size=3, strides=2, input_shape=self.img_shape, padding=\"same\"))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(32, kernel_size=3, strides=2, padding=\"same\"))\n model.add(ZeroPadding2D(padding=((0,1),(0,1))))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Conv2D(128, kernel_size=3, strides=1, padding=\"same\"))\n model.add(BatchNormalization(momentum=0.8))\n model.add(LeakyReLU(alpha=0.2))\n model.add(Dropout(0.25))\n model.add(Flatten())\n model.add(Dense(1))\n\n model.summary()\n\n img = Input(shape=self.img_shape)\n validity = model(img)\n\n return Model(img, validity)\n\n def train(self, epochs, batch_size, sample_interval=50):\n\n # Load the dataset\n (X_train, _), (_, _) = mnist.load_data()\n\n # Rescale -1 to 1\n X_train = (X_train.astype(np.float32) - 127.5) / 127.5\n X_train = np.expand_dims(X_train, axis=3)\n\n # Adversarial ground truths\n valid = -np.ones((batch_size, 1))\n fake = np.ones((batch_size, 1))\n dummy = np.zeros((batch_size, 1)) # Dummy gt for gradient penalty\n for epoch in range(epochs):\n\n for _ in range(self.n_critic):\n\n # ---------------------\n # Train Discriminator\n # ---------------------\n\n # Select a random batch of images\n idx = np.random.randint(0, X_train.shape[0], batch_size)\n imgs = X_train[idx]\n # Sample generator input\n noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n # Train the critic\n d_loss = self.critic_model.train_on_batch([imgs, noise],\n [valid, fake, dummy])\n\n # ---------------------\n # Train Generator\n # ---------------------\n\n g_loss = self.generator_model.train_on_batch(noise, valid)\n\n # Plot the progress\n print (\"%d [D loss: %f] [G loss: %f]\" % (epoch, d_loss[0], g_loss))\n\n # If at save interval => save generated image samples\n if epoch % sample_interval == 0:\n self.sample_images(epoch)\n\n def sample_images(self, epoch):\n r, c = 5, 5\n noise = np.random.normal(0, 1, (r * c, self.latent_dim))\n gen_imgs = self.generator.predict(noise)\n\n # Rescale images 0 - 1\n gen_imgs = 0.5 * gen_imgs + 0.5\n\n fig, axs = plt.subplots(r, c)\n cnt = 0\n for i in range(r):\n for j in range(c):\n axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n axs[i,j].axis('off')\n cnt += 1\n fig.savefig(\"images/mnist_%d.png\" % epoch)\n plt.close()\n\n\nif __name__ == '__main__':\n wgan = WGANGP()\n wgan.train(epochs=30000, batch_size=32, sample_interval=100)\n" } ]