Repository: awjuliani/Pix2Pix-Film
Branch: master
Commit: a19c52904990
Files: 4
Total size: 20.1 KB
Directory structure:
gitextract_8rvhh8xk/
├── LICENSE
├── Pix2Pix.ipynb
├── README.md
└── helper.py
================================================
FILE CONTENTS
================================================
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2017 Arthur Juliani
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: Pix2Pix.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Pix2Pix in Tensorflow"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This tutorials walks through an implementation of Pix2Pix as described in [Image-to-Image Translation with Conditional Adversarial Networks](https://arxiv.org/abs/1611.07004).\n",
"\n",
"This specific implementation is designed to 'remaster' black-and-white frames from films with 4:3 aspect ratio into full-color and 16:9 aspect ratio frames. \n",
"\n",
"To learn more about the Pix2Pix framework, and the images that can be generated using this framework, see my [Medium post](https://medium.com/p/f4d551fa0503).\n",
"\n",
"This notebook requires the additional helper.py file, which can be obtained [here]()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#Import the libraries we will need.\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import tensorflow.contrib.slim as slim\n",
"import os\n",
"import scipy.misc\n",
"import scipy\n",
"from PIL import Image\n",
"from glob import glob\n",
"import os\n",
"from helper import *\n",
"%matplotlib inline\n",
"\n",
"#Size of image frames\n",
"height = 144\n",
"width = 256"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Defining the Adversarial Networks"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generator Network"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def generator(c):\n",
" with tf.variable_scope('generator'):\n",
" #Encoder\n",
" enc0 = slim.conv2d(c,64,[3,3],padding=\"SAME\",\n",
" biases_initializer=None,activation_fn=lrelu,\n",
" weights_initializer=initializer)\n",
" enc0 = tf.space_to_depth(enc0,2)\n",
" \n",
" enc1 = slim.conv2d(enc0,128,[3,3],padding=\"SAME\",\n",
" activation_fn=lrelu,normalizer_fn=slim.batch_norm,\n",
" weights_initializer=initializer)\n",
" enc1 = tf.space_to_depth(enc1,2)\n",
"\n",
" enc2 = slim.conv2d(enc1,128,[3,3],padding=\"SAME\",\n",
" normalizer_fn=slim.batch_norm,activation_fn=lrelu,\n",
" weights_initializer=initializer)\n",
" enc2 = tf.space_to_depth(enc2,2)\n",
"\n",
" enc3 = slim.conv2d(enc2,256,[3,3],padding=\"SAME\",\n",
" normalizer_fn=slim.batch_norm,activation_fn=lrelu,\n",
" weights_initializer=initializer)\n",
" enc3 = tf.space_to_depth(enc3,2)\n",
" \n",
" #Decoder\n",
" gen0 = slim.conv2d(\n",
" enc3,num_outputs=256,kernel_size=[3,3],\n",
" padding=\"SAME\",normalizer_fn=slim.batch_norm,\n",
" activation_fn=tf.nn.elu, weights_initializer=initializer)\n",
" gen0 = tf.depth_to_space(gen0,2)\n",
"\n",
" gen1 = slim.conv2d(\n",
" tf.concat([gen0,enc2],3),num_outputs=256,kernel_size=[3,3],\n",
" padding=\"SAME\",normalizer_fn=slim.batch_norm,\n",
" activation_fn=tf.nn.elu,weights_initializer=initializer)\n",
" gen1 = tf.depth_to_space(gen1,2)\n",
"\n",
" gen2 = slim.conv2d(\n",
" tf.concat([gen1,enc1],3),num_outputs=128,kernel_size=[3,3],\n",
" padding=\"SAME\",normalizer_fn=slim.batch_norm,\n",
" activation_fn=tf.nn.elu,weights_initializer=initializer)\n",
" gen2 = tf.depth_to_space(gen2,2)\n",
"\n",
" gen3 = slim.conv2d(\n",
" tf.concat([gen2,enc0],3),num_outputs=128,kernel_size=[3,3],\n",
" padding=\"SAME\",normalizer_fn=slim.batch_norm,\n",
" activation_fn=tf.nn.elu, weights_initializer=initializer)\n",
" gen3 = tf.depth_to_space(gen3,2)\n",
" \n",
" g_out = slim.conv2d(\n",
" gen3,num_outputs=3,kernel_size=[1,1],padding=\"SAME\",\n",
" biases_initializer=None,activation_fn=tf.nn.tanh,\n",
" weights_initializer=initializer)\n",
" return g_out"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Discriminator Network"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def discriminator(bottom, reuse=False):\n",
" with tf.variable_scope('discriminator'):\n",
" filters = [32,64,128,128]\n",
" \n",
" #Programatically define layers\n",
" for i in range(len(filters)):\n",
" if i == 0:\n",
" layer = slim.conv2d(bottom,filters[i],[3,3],padding=\"SAME\",scope='d'+str(i),\n",
" biases_initializer=None,activation_fn=lrelu,stride=[2,2],\n",
" reuse=reuse,weights_initializer=initializer)\n",
" else:\n",
" layer = slim.conv2d(bottom,filters[i],[3,3],padding=\"SAME\",scope='d'+str(i),\n",
" normalizer_fn=slim.batch_norm,activation_fn=lrelu,stride=[2,2],\n",
" reuse=reuse,weights_initializer=initializer)\n",
" bottom = layer\n",
"\n",
" dis_full = slim.fully_connected(slim.flatten(bottom),1024,activation_fn=lrelu,scope='dl',\n",
" reuse=reuse, weights_initializer=initializer)\n",
"\n",
" d_out = slim.fully_connected(dis_full,1,activation_fn=tf.nn.sigmoid,scope='do',\n",
" reuse=reuse, weights_initializer=initializer)\n",
" return d_out"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Connecting them together"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"tf.reset_default_graph()\n",
"\n",
"#This initializaer is used to initialize all the weights of the network.\n",
"initializer = tf.truncated_normal_initializer(stddev=0.02)\n",
"\n",
"#\n",
"condition_in = tf.placeholder(shape=[None,height,width,3],dtype=tf.float32)\n",
"real_in = tf.placeholder(shape=[None,height,width,3],dtype=tf.float32) #Real images\n",
"\n",
"Gx = generator(condition_in) #Generates images from random z vectors\n",
"Dx = discriminator(real_in) #Produces probabilities for real images\n",
"Dg = discriminator(Gx,reuse=True) #Produces probabilities for generator images\n",
"\n",
"#These functions together define the optimization objective of the GAN.\n",
"d_loss = -tf.reduce_mean(tf.log(Dx) + tf.log(1.-Dg)) #This optimizes the discriminator.\n",
"#For generator we use traditional GAN objective as well as L1 loss\n",
"g_loss = -tf.reduce_mean(tf.log(Dg)) + 100*tf.reduce_mean(tf.abs(Gx - real_in)) #This optimizes the generator.\n",
"\n",
"#The below code is responsible for applying gradient descent to update the GAN.\n",
"trainerD = tf.train.AdamOptimizer(learning_rate=0.0002,beta1=0.5)\n",
"trainerG = tf.train.AdamOptimizer(learning_rate=0.002,beta1=0.5)\n",
"d_grads = trainerD.compute_gradients(d_loss,slim.get_variables(scope='discriminator'))\n",
"g_grads = trainerG.compute_gradients(g_loss, slim.get_variables(scope='generator'))\n",
"\n",
"update_D = trainerD.apply_gradients(d_grads)\n",
"update_G = trainerG.apply_gradients(g_grads)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Training the network\n",
"Now that we have fully defined our network, it is time to train it!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"batch_size = 4 #Size of image batch to apply at each iteration.\n",
"iterations = 500000 #Total number of iterations to use.\n",
"subset_size = 5000 #How many images to load at a time, will vary depending on available resources\n",
"frame_directory = './frames' #Directory where training images are located\n",
"sample_directory = './samples' #Directory to save sample images from generator in.\n",
"model_directory = './model' #Directory to save trained model to.\n",
"sample_frequency = 200 #How often to generate sample gif of translated images.\n",
"save_frequency = 5000 #How often to save model.\n",
"load_model = False #Whether to load the model or begin training from scratch."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [],
"source": [
"subset = 0\n",
"dataS = sorted(glob(os.path.join(frame_directory, \"*.png\")))\n",
"total_subsets = len(dataS)/subset_size\n",
"init = tf.global_variables_initializer()\n",
"saver = tf.train.Saver()\n",
"with tf.Session() as sess: \n",
" sess.run(init)\n",
" if load_model == True: \n",
" ckpt = tf.train.get_checkpoint_state(model_directory)\n",
" saver.restore(sess,ckpt.model_checkpoint_path)\n",
"\n",
" imagesY,imagesX = loadImages(dataS[0:subset_size],False, np.random.randint(0,2)) #Load a subset of images\n",
" print \"Loaded subset \" + str(subset)\n",
" draw = range(len(imagesX))\n",
" for i in range(iterations):\n",
" if i % (subset_size/batch_size) != 0 or i == 0:\n",
" batch_index = np.random.choice(draw,size=batch_size,replace=False)\n",
" else:\n",
" subset = np.random.randint(0,total_subsets+1)\n",
" imagesY,imagesX = loadImages(dataS[subset*subset_size:(subset+1)*subset_size],False, np.random.randint(0,2))\n",
" print \"Loaded subset \" + str(subset)\n",
" draw = range(len(imagesX))\n",
" batch_index = np.random.choice(draw,size=batch_size,replace=False)\n",
" \n",
" ys = (np.reshape(imagesY[batch_index],[batch_size,height,width,3]) - 0.5) * 2.0 #Transform to be between -1 and 1\n",
" xs = (np.reshape(imagesX[batch_index],[batch_size,height,width,3]) - 0.5) * 2.0\n",
" _,dLoss = sess.run([update_D,d_loss],feed_dict={real_in:ys,condition_in:xs}) #Update the discriminator\n",
" _,gLoss = sess.run([update_G,g_loss],feed_dict={real_in:ys,condition_in:xs}) #Update the generator\n",
" if i % sample_frequency == 0:\n",
" print \"Gen Loss: \" + str(gLoss) + \" Disc Loss: \" + str(dLoss)\n",
" start_point = np.random.randint(0,len(imagesX)-32)\n",
" xs = (np.reshape(imagesX[start_point:start_point+32],[32,height,width,3]) - 0.5) * 2.0\n",
" ys = (np.reshape(imagesY[start_point:start_point+32],[32,height,width,3]) - 0.5) * 2.0\n",
" sample_G = sess.run(Gx,feed_dict={condition_in:xs}) #Use new z to get sample images from generator.\n",
" allS = np.concatenate([xs,sample_G,ys],axis=1)\n",
" if not os.path.exists(sample_directory):\n",
" os.makedirs(sample_directory)\n",
" #Save sample generator images for viewing training progress.\n",
" make_gif(allS,'./'+sample_directory+'/a_vid'+str(i)+'.gif',\n",
" duration=len(allS)*0.2,true_image=False)\n",
" if i % save_frequency == 0 and i != 0:\n",
" if not os.path.exists(model_directory):\n",
" os.makedirs(model_directory)\n",
" saver.save(sess,model_directory+'/model-'+str(i)+'.cptk')\n",
" print \"Saved Model\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using a trained network\n",
"Once we have a trained model saved, we may want to use it to generate new images, and explore the representation it has learned."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"test_directory = './test_frames' #Directory to load test frames from\n",
"subset_size = 5000\n",
"batch_size = 60 # Size of image batch to apply at each iteration. Will depend of available resources.\n",
"sample_directory = './test_samples' #Directory to save sample images from generator in.\n",
"model_directory = './model' #Directory to save trained model to.\n",
"load_model = True #Whether to load a saved model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"dataS = sorted(glob(os.path.join(test_directory, \"*.png\")))\n",
"subset = 0\n",
"total_subsets = len(dataS)/subset_size\n",
"iterations = subset_size / batch_size #Total number of iterations to use.\n",
"\n",
"\n",
"if not os.path.exists(sample_directory):\n",
" os.makedirs(sample_directory)\n",
"\n",
"init = tf.global_variables_initializer()\n",
"saver = tf.train.Saver()\n",
"with tf.Session() as sess: \n",
" sess.run(init)\n",
" if load_model == True: \n",
" ckpt = tf.train.get_checkpoint_state(model_directory)\n",
" saver.restore(sess,ckpt.model_checkpoint_path)\n",
" for s in range(total_subsets):\n",
" generated_frames = []\n",
" _,imagesX = loadImages(dataS[s*subset_size:s*subset_size+subset_size],False, False) #Load a subset of images\n",
" for i in range(iterations):\n",
" start_point = i * batch_size\n",
" xs = (np.reshape(imagesX[start_point:start_point+batch_size],[batch_size,height,width,3]) - 0.5) * 2.0\n",
" sample_G = sess.run(Gx,feed_dict={condition_in:xs}) #Use new z to get sample images from generator. \n",
" #allS = np.concatenate([xs,sample_G],axis=2)\n",
" generated_frames.append(sample_G)\n",
" generated_frames = np.vstack(generated_frames)\n",
" for i in range(len(generated_frames)):\n",
" im = Image.fromarray(((generated_frames[i]/2.0 + 0.5) * 256).astype('uint8'))\n",
" im.save('./'+sample_directory+'/frame'+str(s*subset_size + i)+'.png') \n",
" #make_gif(generated_frames,'./'+sample_directory+'/a_vid'+str(i)+'.gif',\n",
" # duration=len(generated_frames)/10.0,true_image=False)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
================================================
FILE: README.md
================================================
# Pix2Pix-Film
An implementation of [Pix2Pix](https://arxiv.org/abs/1611.07004) in Tensorflow for use with colorizing and increasing the field of view in frames from classic films. For more information, see my [Medium Post](https://medium.com/p/f4d551fa0503) on the project.
Pretrained model available [here](https://drive.google.com/open?id=0B8x0IeJAaBccNFVQMkQ0QW15TjQ). It was trained using Alfred Hitchcock films, so it generalizes best to similar movies.
To generate frames, use `ffmpeg` to resize and create `.png` frames from video.
* To resize video: `ffmpeg -strict -2 -i input.mp4 -vf scale=256:144 output.mp4 -strict -2`
* To generate frames: ` ffmpeg -i output.mp4 -vf fps=10 ./frames/out%6d.png`
In order to 'remaster' frames, run Pix2Pix iPython notebook.
To convert 'remastered' frames back into a video, use:
`ffmpeg -framerate 10 -i frame%01d.png -c:v libx264 -r 30 -pix_fmt yuv420p out.mp4`
================================================
FILE: helper.py
================================================
import numpy as np
import random
import tensorflow as tf
import matplotlib.pyplot as plt
import scipy.misc
import os
import csv
import itertools
import tensorflow.contrib.slim as slim
from PIL import Image
#Generates gifs
def make_gif(images, fname, duration=2, true_image=False):
import moviepy.editor as mpy
def make_frame(t):
try:
x = images[int(len(images)/duration*t)]
except:
x = images[-1]
if true_image:
return x.astype(np.uint8)
else:
return ((x+1)/2*255).astype(np.uint8)
clip = mpy.VideoClip(make_frame, duration=duration)
clip.write_gif(fname, fps = len(images) / duration,verbose=False)
#Function loads images from list of files. bw_bool is True when the source images are originally greyscale and 4:3
#Flip determines whether images should be flipped
def loadImages(data,bw_bool,flip):
images = []
images_bw = []
if bw_bool == False:
for myFile in data:
img = Image.open(myFile)
bw = np.max(img,2)
bw = np.stack([bw,bw,bw],2)
bw[:,:40,:] = 0
bw[:,-40:,:] = 0
if flip == False:
images.append(np.array(img))
images_bw.append(bw)
else:
img_flip = np.fliplr(img)
images.append(img_flip)
bw_flip = np.fliplr(bw)
images_bw.append(bw_flip)
images = np.array(images)
images = images.astype('float32')
images = images / 256
images_bw = np.array(images_bw)
images_bw = images_bw.astype('float32')
images_bw = images_bw / 256
return images,images_bw
else:
for myFile in data:
img = Image.open(myFile)
bw = img.resize((196,144))
bw = np.max(bw,2)
bw = np.stack([bw,bw,bw],2)
bw_w = np.zeros([144,256,3])
bw_w[:,30:-30,:] = bw
bw_w[:,:40,:] = 0
bw_w[:,-40:,:] = 0
images.append(bw_w)
images = np.array(images)
images = images.astype('float32')
images = images / 256
return images
#This function performns a leaky relu activation, which is needed for the discriminator network.
def lrelu(x, leak=0.2, name="lrelu"):
with tf.variable_scope(name):
f1 = 0.5 * (1 + leak)
f2 = 0.5 * (1 - leak)
return f1 * x + f2 * abs(x)
#The below functions are taken from carpdem20's implementation https://github.com/carpedm20/DCGAN-tensorflow
#They allow for saving sample images from the generator to follow progress
def save_images(images, size, image_path):
return imsave(inverse_transform(images), size, image_path)
def imsave(images, size, path):
return scipy.misc.imsave(path, merge(images, size))
def inverse_transform(images):
return (images+1.)/2.
def merge(images, size):
h, w = images.shape[1], images.shape[2]
img = np.zeros((h * size[0], w * size[1],3))
for idx, image in enumerate(images):
i = idx % size[1]
j = idx / size[1]
img[j*h:j*h+h, i*w:i*w+w,:] = image
return img
gitextract_8rvhh8xk/ ├── LICENSE ├── Pix2Pix.ipynb ├── README.md └── helper.py
SYMBOL INDEX (7 symbols across 1 files) FILE: helper.py function make_gif (line 14) | def make_gif(images, fname, duration=2, true_image=False): function loadImages (line 33) | def loadImages(data,bw_bool,flip): function lrelu (line 75) | def lrelu(x, leak=0.2, name="lrelu"): function save_images (line 83) | def save_images(images, size, image_path): function imsave (line 86) | def imsave(images, size, path): function inverse_transform (line 89) | def inverse_transform(images): function merge (line 92) | def merge(images, size):
Condensed preview — 4 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (22K chars).
[
{
"path": "LICENSE",
"chars": 1071,
"preview": "MIT License\n\nCopyright (c) 2017 Arthur Juliani\n\nPermission is hereby granted, free of charge, to any person obtaining a "
},
{
"path": "Pix2Pix.ipynb",
"chars": 15459,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Pix2Pix in Tensorflow\"\n ]\n },\n"
},
{
"path": "README.md",
"chars": 943,
"preview": "# Pix2Pix-Film\n\n\nAn implementation of [Pix2Pix](https://arxiv.org/abs/1611.07004) in Tensorflow"
},
{
"path": "helper.py",
"chars": 3129,
"preview": "import numpy as np\nimport random\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nimport scipy.misc\nimport os\nimp"
}
]
About this extraction
This page contains the full source code of the awjuliani/Pix2Pix-Film GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 4 files (20.1 KB), approximately 5.8k tokens, and a symbol index with 7 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.