| { |
| "cells": [ |
| { |
| "cell_type": "markdown", |
| "id": "93a5a75f", |
| "metadata": {}, |
| "source": [ |
| "<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n", |
| "<!--- or more contributor license agreements. See the NOTICE file -->\n", |
| "<!--- distributed with this work for additional information -->\n", |
| "<!--- regarding copyright ownership. The ASF licenses this file -->\n", |
| "<!--- to you under the Apache License, Version 2.0 (the -->\n", |
| "<!--- \"License\"); you may not use this file except in compliance -->\n", |
| "<!--- with the License. You may obtain a copy of the License at -->\n", |
| "\n", |
| "<!--- http://www.apache.org/licenses/LICENSE-2.0 -->\n", |
| "\n", |
| "<!--- Unless required by applicable law or agreed to in writing, -->\n", |
| "<!--- software distributed under the License is distributed on an -->\n", |
| "<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n", |
| "<!--- KIND, either express or implied. See the License for the -->\n", |
| "<!--- specific language governing permissions and limitations -->\n", |
| "<!--- under the License. -->\n", |
| "\n", |
| "\n", |
| "# Image similarity search with InfoGAN\n", |
| "\n", |
| "This notebook shows how to implement an InfoGAN based on Gluon. InfoGAN is an extension of GANs, where the generator input is split in 2 parts: random noise and a latent code (see [InfoGAN Paper](https://arxiv.org/pdf/1606.03657.pdf)). \n", |
| "The codes are made meaningful by maximizing the mutual information between code and generator output. InfoGAN learns a disentangled representation in a completely unsupervised manner. It can be used for many applications such as image similarity search. This notebook uses the DCGAN example from the [Straight Dope Book](https://gluon.mxnet.io/chapter14_generative-adversarial-networks/dcgan.html) and extends it to create an InfoGAN." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "0e1b5574", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "from __future__ import print_function\n", |
| "from datetime import datetime\n", |
| "import logging\n", |
| "import multiprocessing\n", |
| "import os\n", |
| "import sys\n", |
| "import tarfile\n", |
| "import time\n", |
| "\n", |
| "import numpy as np\n", |
| "from matplotlib import pyplot as plt\n", |
| "from mxboard import SummaryWriter\n", |
| "import mxnet as mx\n", |
| "from mxnet import gluon\n", |
| "from mxnet import ndarray as nd\n", |
| "from mxnet.gluon import nn, utils\n", |
| "from mxnet import autograd\n", |
| "\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "581776ca", |
| "metadata": {}, |
| "source": [ |
| "The latent code vector can contain several variables, which can be categorical and/or continuous. We set `n_continuous` to 2 and `n_categories` to 10." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "bcf2419d", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "batch_size = 64\n", |
| "z_dim = 100\n", |
| "n_continuous = 2\n", |
| "n_categories = 10\n", |
| "ctx = mx.gpu() if mx.context.num_gpus() else mx.cpu()\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "0e802492", |
| "metadata": {}, |
| "source": [ |
| "Some functions to load and normalize images." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "394a26b4", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "lfw_url = 'http://vis-www.cs.umass.edu/lfw/lfw-deepfunneled.tgz'\n", |
| "data_path = 'lfw_dataset'\n", |
| "if not os.path.exists(data_path):\n", |
| " os.makedirs(data_path)\n", |
| " data_file = utils.download(lfw_url)\n", |
| " with tarfile.open(data_file) as tar:\n", |
| " tar.extractall(path=data_path)\n", |
| "\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "75577236", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "def transform(data, width=64, height=64):\n", |
| " data = mx.image.imresize(data, width, height)\n", |
| " data = nd.transpose(data, (2,0,1))\n", |
| " data = data.astype(np.float32)/127.5 - 1\n", |
| " if data.shape[0] == 1:\n", |
| " data = nd.tile(data, (3, 1, 1))\n", |
| " return data.reshape((1,) + data.shape)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "9eb40f10", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "def get_files(data_dir):\n", |
| " images = []\n", |
| " filenames = []\n", |
| " for path, _, fnames in os.walk(data_dir):\n", |
| " for fname in fnames:\n", |
| " if not fname.endswith('.jpg'):\n", |
| " continue\n", |
| " img = os.path.join(path, fname)\n", |
| " img_arr = mx.image.imread(img)\n", |
| " img_arr = transform(img_arr)\n", |
| " images.append(img_arr)\n", |
| " filenames.append(path + \"/\" + fname)\n", |
| " return images, filenames \n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "babf6ee5", |
| "metadata": {}, |
| "source": [ |
| "Load the dataset `lfw_dataset` which contains images of celebrities." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "93f9284a", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "data_dir = 'lfw_dataset'\n", |
| "images, filenames = get_files(data_dir)\n", |
| "split = int(len(images)*0.8)\n", |
| "test_images = images[split:]\n", |
| "test_filenames = filenames[split:]\n", |
| "train_images = images[:split]\n", |
| "train_filenames = filenames[:split]\n", |
| "\n", |
| "train_data = gluon.data.ArrayDataset(nd.concatenate(train_images))\n", |
| "train_dataloader = gluon.data.DataLoader(train_data, batch_size=batch_size, shuffle=True, last_batch='rollover', num_workers=multiprocessing.cpu_count()-1)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "f3bb86df", |
| "metadata": {}, |
| "source": [ |
| "## Generator\n", |
| "Define the Generator model. Architecture is taken from the DCGAN implementation in [Straight Dope Book](https://gluon.mxnet.io/chapter14_generative-adversarial-networks/dcgan.html). The Generator consist of 4 layers where each layer involves a strided convolution, batch normalization, and rectified nonlinearity. It takes as input random noise and the latent code and produces an `(64,64,3)` output image." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "208ede77", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "class Generator(gluon.HybridBlock):\n", |
| " def __init__(self, **kwargs):\n", |
| " super(Generator, self).__init__(**kwargs)\n", |
| " with self.name_scope():\n", |
| " self.prev = nn.HybridSequential()\n", |
| " self.prev.add(nn.Dense(1024, use_bias=False), nn.BatchNorm(), nn.Activation(activation='relu'))\n", |
| " self.G = nn.HybridSequential()\n", |
| " \n", |
| " self.G.add(nn.Conv2DTranspose(64 * 8, 4, 1, 0, use_bias=False))\n", |
| " self.G.add(nn.BatchNorm())\n", |
| " self.G.add(nn.Activation('relu'))\n", |
| " self.G.add(nn.Conv2DTranspose(64 * 4, 4, 2, 1, use_bias=False))\n", |
| " self.G.add(nn.BatchNorm())\n", |
| " self.G.add(nn.Activation('relu'))\n", |
| " self.G.add(nn.Conv2DTranspose(64 * 2, 4, 2, 1, use_bias=False))\n", |
| " self.G.add(nn.BatchNorm())\n", |
| " self.G.add(nn.Activation('relu'))\n", |
| " self.G.add(nn.Conv2DTranspose(64, 4, 2, 1, use_bias=False))\n", |
| " self.G.add(nn.BatchNorm())\n", |
| " self.G.add(nn.Activation('relu'))\n", |
| " self.G.add(nn.Conv2DTranspose(3, 4, 2, 1, use_bias=False))\n", |
| " self.G.add(nn.Activation('tanh'))\n", |
| "\n", |
| " def hybrid_forward(self, F, x):\n", |
| " x = self.prev(x)\n", |
| " x = F.reshape(x, (0, -1, 1, 1))\n", |
| " return self.G(x)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "84cd3c64", |
| "metadata": {}, |
| "source": [ |
| "## Discriminator\n", |
| "Define the Discriminator and Q model. The Q model shares many layers with the Discriminator. Its task is to estimate the code `c` for a given fake image. It is used to maximize the lower bound to the mutual information." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "a0e8031b", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "class Discriminator(gluon.HybridBlock):\n", |
| " def __init__(self, **kwargs):\n", |
| " super(Discriminator, self).__init__(**kwargs)\n", |
| " with self.name_scope():\n", |
| " self.D = nn.HybridSequential()\n", |
| " self.D.add(nn.Conv2D(64, 4, 2, 1, use_bias=False))\n", |
| " self.D.add(nn.LeakyReLU(0.2))\n", |
| " self.D.add(nn.Conv2D(64 * 2, 4, 2, 1, use_bias=False))\n", |
| " self.D.add(nn.BatchNorm())\n", |
| " self.D.add(nn.LeakyReLU(0.2))\n", |
| " self.D.add(nn.Conv2D(64 * 4, 4, 2, 1, use_bias=False))\n", |
| " self.D.add(nn.BatchNorm())\n", |
| " self.D.add(nn.LeakyReLU(0.2))\n", |
| " self.D.add(nn.Conv2D(64 * 8, 4, 2, 1, use_bias=False))\n", |
| " self.D.add(nn.BatchNorm())\n", |
| " self.D.add(nn.LeakyReLU(0.2))\n", |
| "\n", |
| " self.D.add(nn.Dense(1024, use_bias=False), nn.BatchNorm(), nn.Activation(activation='relu'))\n", |
| " \n", |
| " self.prob = nn.Dense(1)\n", |
| " self.feat = nn.HybridSequential()\n", |
| " self.feat.add(nn.Dense(128, use_bias=False), nn.BatchNorm(), nn.Activation(activation='relu'))\n", |
| " self.category_prob = nn.Dense(n_categories)\n", |
| " self.continuous_mean = nn.Dense(n_continuous)\n", |
| " self.Q = nn.HybridSequential()\n", |
| " self.Q.add(self.feat, self.category_prob, self.continuous_mean)\n", |
| "\n", |
| " def hybrid_forward(self, F, x):\n", |
| " x = self.D(x)\n", |
| " prob = self.prob(x)\n", |
| " feat = self.feat(x)\n", |
| " category_prob = self.category_prob(feat)\n", |
| " continuous_mean = self.continuous_mean(feat)\n", |
| " \n", |
| " return prob, category_prob, continuous_mean\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "cbe2e8a3", |
| "metadata": {}, |
| "source": [ |
| "The InfoGAN has the following layout.\n", |
| "<img src=\"https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/info_gan/InfoGAN.png\" style=\"width:800px;height:250px;\">\n", |
| "\n", |
| "Discriminator and Generator are the same as in the DCGAN example. On top of the Disciminator is the Q model, which is estimating the code `c` for given fake images. The Generator's input is random noise and the latent code `c`. \n", |
| "\n", |
| "## Training Loop\n", |
| "Initialize Generator and Discriminator and define correspoing trainer function." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "fdc35237", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "generator = Generator()\n", |
| "generator.hybridize()\n", |
| "generator.initialize(mx.init.Normal(0.002), ctx=ctx)\n", |
| "\n", |
| "discriminator = Discriminator()\n", |
| "discriminator.hybridize()\n", |
| "discriminator.initialize(mx.init.Normal(0.002), ctx=ctx)\n", |
| "\n", |
| "lr = 0.0001\n", |
| "beta = 0.5\n", |
| "\n", |
| "g_trainer = gluon.Trainer(generator.collect_params(), 'adam', {'learning_rate': lr, 'beta1': beta})\n", |
| "d_trainer = gluon.Trainer(discriminator.collect_params(), 'adam', {'learning_rate': lr, 'beta1': beta})\n", |
| "q_trainer = gluon.Trainer(discriminator.Q.collect_params(), 'adam', {'learning_rate': lr, 'beta1': beta})\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "837d9445", |
| "metadata": {}, |
| "source": [ |
| "Create vectors with real (=1) and fake labels (=0)." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "46b427ff", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "real_label = nd.ones((batch_size,), ctx=ctx)\n", |
| "fake_label = nd.zeros((batch_size,),ctx=ctx)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "1d082163", |
| "metadata": {}, |
| "source": [ |
| "Load a pretrained model." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "825a4255", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "if os.path.isfile('infogan_d_latest.params') and os.path.isfile('infogan_g_latest.params'):\n", |
| " discriminator.load_parameters('infogan_d_latest.params', ctx=ctx, allow_missing=True, ignore_extra=True)\n", |
| " generator.load_parameters('infogan_g_latest.params', ctx=ctx, allow_missing=True, ignore_extra=True)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "06fc4b8f", |
| "metadata": {}, |
| "source": [ |
| "There are 2 differences between InfoGAN and DCGAN: the extra latent code and the Q network to estimate the code.\n", |
| "The latent code is part of the Generator input and it contains mutliple variables (continuous, categorical) that can represent different distributions. In order to make sure that the Generator uses the latent code, mutual information is introduced into the GAN loss term. Mutual information measures how much X is known given Y or vice versa. It is defined as:\n", |
| "\n", |
| " \n", |
| "\n", |
| "The InfoGAN loss is:\n", |
| "\n", |
| "\n", |
| "\n", |
| "where `V(D,G)` is the GAN loss and the mutual information `I(c, G(z, c))` goes in as regularization. The goal is to reach high mutual information, in order to learn meaningful codes for the data. \n", |
| "\n", |
| "\n", |
| "Define the loss functions. `SoftmaxCrossEntropyLoss` for the categorical code, `L2Loss` for the continious code and `SigmoidBinaryCrossEntropyLoss` for the normal GAN loss." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "b501ad86", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "loss1 = gluon.loss.SigmoidBinaryCrossEntropyLoss()\n", |
| "loss2 = gluon.loss.L2Loss()\n", |
| "loss3 = gluon.loss.SoftmaxCrossEntropyLoss()\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "3990081e", |
| "metadata": {}, |
| "source": [ |
| "This function samples `c`, `z`, and concatenates them to create the generator input." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "ed84e872", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "def create_generator_input():\n", |
| " \n", |
| " #create random noise\n", |
| " z = nd.random_normal(0, 1, shape=(batch_size, z_dim), ctx=ctx)\n", |
| " label = nd.array(np.random.randint(n_categories, size=batch_size)).as_in_context(ctx)\n", |
| " c1 = nd.one_hot(label, depth=n_categories).as_in_context(ctx)\n", |
| " c2 = nd.random.uniform(-1, 1, shape=(batch_size, n_continuous)).as_in_context(ctx)\n", |
| "\n", |
| " # concatenate random noise with c which will be the input of the generator\n", |
| " return nd.concat(z, c1, c2, dim=1), label, c2\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "272ed77c", |
| "metadata": {}, |
| "source": [ |
| "Define the training loop. \n", |
| "1. The discriminator receives `real_data` and `loss1` measures how many real images have been identified as real\n", |
| "2. The discriminator receives `fake_image` from the Generator and `loss1` measures how many fake images have been identified as fake\n", |
| "3. Update Discriminator. Currently, it is updated every second iteration in order to avoid that the Discriminator becomes too strong. You may want to change that.\n", |
| "4. The updated discriminator receives `fake_image` and `loss1` measures how many fake images have been been identified as real, `loss2` measures the difference between the sampled continuous latent code `c` and the output of the Q model and `loss3` measures the difference between the sampled categorical latent code `c` and the output of the Q model.\n", |
| "4. Update Generator and Q" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "385fe23b", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "with SummaryWriter(logdir='./logs/') as sw:\n", |
| " \n", |
| " epochs = 1\n", |
| " counter = 0\n", |
| " for epoch in range(epochs):\n", |
| " print(\"Epoch\", epoch)\n", |
| " starttime = time.time()\n", |
| " \n", |
| " d_error_epoch = nd.zeros((1,), ctx=ctx)\n", |
| " g_error_epoch = nd.zeros((1,), ctx=ctx)\n", |
| " \n", |
| " for idx, data in enumerate(train_dataloader):\n", |
| " \n", |
| " #get real data and generator input\n", |
| " real_data = data.as_in_context(ctx) \n", |
| " g_input, label, c2 = create_generator_input()\n", |
| "\n", |
| " \n", |
| " #Update discriminator: Input real data and fake data\n", |
| " with autograd.record():\n", |
| " output_real,_,_ = discriminator(real_data)\n", |
| " d_error_real = loss1(output_real, real_label)\n", |
| " \n", |
| " # create fake image and input it to discriminator\n", |
| " fake_image = generator(g_input)\n", |
| " output_fake,_,_ = discriminator(fake_image.detach())\n", |
| " d_error_fake = loss1(output_fake, fake_label)\n", |
| " \n", |
| " # total discriminator error\n", |
| " d_error = d_error_real + d_error_fake\n", |
| "\n", |
| " d_error_epoch += d_error.mean()\n", |
| " \n", |
| " #Update D every second iteration\n", |
| " if (counter+1) % 2 == 0:\n", |
| " d_error.backward()\n", |
| " d_trainer.step(batch_size)\n", |
| "\n", |
| " #Update generator: Input random noise and latent code vector\n", |
| " with autograd.record():\n", |
| " fake_image = generator(g_input)\n", |
| " output_fake, category_prob, continuous_mean = discriminator(fake_image)\n", |
| " g_error = loss1(output_fake, real_label) + loss3(category_prob, label) + loss2(c2, continuous_mean)\n", |
| "\n", |
| " g_error.backward()\n", |
| " g_error_epoch += g_error.mean()\n", |
| " \n", |
| " g_trainer.step(batch_size)\n", |
| " q_trainer.step(batch_size)\n", |
| "\n", |
| " # logging\n", |
| " if idx % 10 == 0:\n", |
| " count = idx + 1\n", |
| " logging.info('speed: {} samples/s'.format(batch_size / (time.time() - starttime)))\n", |
| " logging.info('discriminator loss = %f, generator loss = %f at iter %d epoch %d'\n", |
| " %(d_error_epoch.asscalar()/count,g_error_epoch.asscalar()/count, count, epoch))\n", |
| "\n", |
| " g_input,_,_ = create_generator_input()\n", |
| " \n", |
| " # create some fake image for logging in MXBoard\n", |
| " fake_image = generator(g_input)\n", |
| "\n", |
| " sw.add_scalar(tag='Loss_D', value={'test':d_error_epoch.asscalar()/count}, global_step=counter)\n", |
| " sw.add_scalar(tag='Loss_G', value={'test':d_error_epoch.asscalar()/count}, global_step=counter)\n", |
| " sw.add_image(tag='data_image', image=((fake_image[0]+ 1.0) * 127.5).astype(np.uint8) , global_step=counter)\n", |
| " sw.flush()\n", |
| " \n", |
| " discriminator.save_parameters(\"infogan_d_latest.params\")\n", |
| " generator.save_parameters(\"infogan_g_latest.params\")\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "2f9ea76d", |
| "metadata": {}, |
| "source": [ |
| "## Image similarity\n", |
| "Once the InfoGAN is trained, we can use the Discriminator to do an image similarity search. The idea is that the network learned meaningful features from the images based on the mutual information e.g. pose of people in an image.\n", |
| "\n", |
| "Load the trained discriminator and retrieve one of its last layers." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "31dd1ab8", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "discriminator = Discriminator()\n", |
| "discriminator.load_parameters(\"infogan_d_latest.params\", ctx=ctx, ignore_extra=True)\n", |
| "\n", |
| "discriminator = discriminator.D[:11]\n", |
| "print (discriminator)\n", |
| "\n", |
| "discriminator.hybridize()\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "df49da66", |
| "metadata": {}, |
| "source": [ |
| "Nearest neighbor function, which takes a matrix of features and an input feature vector. It returns the 3 closest features." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "35d8a722", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "def get_knn(features, input_vector, k=3):\n", |
| " dist = (nd.square(features - input_vector).sum(axis=1))/features.shape[0]\n", |
| " indices = dist.asnumpy().argsort()[:k]\n", |
| " return [(index, dist[index].asscalar()) for index in indices]\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "5f61a202", |
| "metadata": {}, |
| "source": [ |
| "A helper function to visualize image data." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "397b97fb", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "def visualize(img_array):\n", |
| " plt.imshow(((img_array.asnumpy().transpose(1, 2, 0) + 1.0) * 127.5).astype(np.uint8))\n", |
| " plt.axis('off')\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "8c6f1aba", |
| "metadata": {}, |
| "source": [ |
| "Take some images from the test data, obtain its feature vector from `discriminator.D[:11]` and plot images of the corresponding closest vectors in the feature space." |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "0b21fe69", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "feature_size = 8192 \n", |
| "\n", |
| "features = nd.zeros((len(test_images), feature_size), ctx=ctx)\n", |
| "\n", |
| "for idx, image in enumerate(test_images):\n", |
| " \n", |
| " feature = discriminator(nd.array(image, ctx=ctx))\n", |
| " feature = feature.reshape(feature_size,)\n", |
| " features[idx,:] = feature.copyto(ctx)\n", |
| "\n", |
| "\n", |
| "for image in test_images[:100]:\n", |
| "\n", |
| " feature = discriminator(mx.nd.array(image, ctx=ctx))\n", |
| " feature = feature.reshape((feature_size,))\n", |
| " image = image.reshape((3,64,64))\n", |
| "\n", |
| "\n", |
| " indices = get_knn(features, feature, k=10)\n", |
| " fig = plt.figure(figsize=(15,12))\n", |
| " plt.subplot(1,10,1)\n", |
| "\n", |
| " visualize(image)\n", |
| " for i in range(2,9): \n", |
| " if indices[i-1][1] < 1.5:\n", |
| " plt.subplot(1,10,i)\n", |
| " sim = test_images[indices[i-1][0]].reshape(3,64,64)\n", |
| " visualize(sim)\n", |
| " plt.show()\n", |
| " plt.clf()\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "73b1061e", |
| "metadata": {}, |
| "source": [ |
| "<!--notebook-skip-line--> \n", |
| "\n", |
| "## How the Generator learns\n", |
| "We trained the Generator for a couple of epochs and stored a couple of fake images per epoch. Check the video.\n", |
| " \n", |
| " \n", |
| "\n", |
| "The following function computes the TSNE on the feature matrix and stores the result in a json-file. This file can be loaded with [TSNEViewer](https://ml4a.github.io/guides/ImageTSNEViewer/)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "40b25aaa", |
| "metadata": {}, |
| "source": [ |
| "```python\n", |
| "import json\n", |
| "\n", |
| "from sklearn.manifold import TSNE\n", |
| "from scipy.spatial import distance\n", |
| "\n", |
| "tsne = TSNE(n_components=2, learning_rate=150, perplexity=30, verbose=2).fit_transform(features.asnumpy())\n", |
| "\n", |
| "# save data to json\n", |
| "data = []\n", |
| "counter = 0\n", |
| "for i,f in enumerate(test_filenames):\n", |
| " \n", |
| " point = [float((tsne[i,k] - np.min(tsne[:,k]))/(np.max(tsne[:,k]) - np.min(tsne[:,k]))) for k in range(2) ]\n", |
| " data.append({\"path\": os.path.abspath(os.path.join(os.getcwd(),f)), \"point\": point})\n", |
| " \n", |
| "with open(\"imagetsne.json\", 'w') as outfile:\n", |
| " json.dump(data, outfile)\n", |
| "```\n" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "328be51c", |
| "metadata": {}, |
| "source": [ |
| "Load the file with TSNEViewer. You can now inspect whether similiar looking images are grouped nearby or not. \n", |
| "\n", |
| "<img src=\"https://raw.githubusercontent.com/NRauschmayr/web-data/master/mxnet/doc/tutorials/info_gan/tsne.png\" style=\"width:800px;height:600px;\">\n", |
| "\n", |
| "<!-- INSERT SOURCE DOWNLOAD BUTTONS -->" |
| ] |
| } |
| ], |
| "metadata": { |
| "language_info": { |
| "name": "python" |
| } |
| }, |
| "nbformat": 4, |
| "nbformat_minor": 5 |
| } |
