Repository: jensleitloff/CNN-Sentinel Branch: master Commit: 094b8d3ff897 Files: 17 Total size: 87.0 KB Directory structure: gitextract_jvujk87r/ ├── .gitignore ├── LICENSE ├── README.md ├── bibliography.bib ├── py/ │ ├── 01_split_data_to_train_and_validation.py │ ├── 02_train_rgb_finetuning.py │ ├── 03_train_rgb_from_scratch.py │ ├── 04_train_ms_finetuning.py │ ├── 04_train_ms_finetuning_alternative.py │ ├── 05_train_ms_from_scratch.py │ ├── 06_classify_image.py │ ├── Image_functions.ipynb │ ├── Train_from_Scratch.ipynb │ ├── Transfer_learning.ipynb │ ├── image_functions.py │ └── statistics.py └── requirements.txt ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ *.DS_Store .vscode/ data/PyCon py/.ipynb_checkpoints/ __* py/logs/ ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) 2018 Jens Leitloff & Felix Riese Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3268451.svg)](https://doi.org/10.5281/zenodo.3268451) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![Codacy Badge](https://api.codacy.com/project/badge/Grade/a6b8568aab8c4c319a4f58d84cccf7c0)](https://www.codacy.com/manual/jensleitloff/CNN-Sentinel?utm_source=github.com&utm_medium=referral&utm_content=jensleitloff/CNN-Sentinel&utm_campaign=Badge_Grade) # Analyzing Sentinel-2 satellite data in Python with TensorFlow.Keras Overview about state-of-the-art land-use classification from satellite data with CNNs based on an open dataset ## Outline * [Scripts you will find here](#scripts-you-will-find-here) * [Requirements (what we used):](#requirements--what-we-used--) * [Setup Environment](#setup-environment) * [Our talks about this topic](#our-talks-about-this-topic) * [Resources](#resources) * [How to get Sentinel-2 data](#how-to-get-sentinel-2-data) * [Citation](#citation) ## Scripts you will find here * `01_split_data_to_train_and_validation.py`: split complete dataset into train and validation * `02_train_rgb_finetuning.py`: train VGG16 or DenseNet201 using RGB data with pre-trained weights on ImageNet * `03_train_rgb_from_scratch.py`: train VGG16 or DenseNet201 from scratch using RGB data * `04_train_ms_finetuning.py`: train VGG16 or DenseNet201 using multispectral data with pre-trained weights on ImageNet * `04_train_ms_finetuning_alternative.py`: an alternative way to train VGG16 or DenseNet201 using multispectral data with pre-trained weights on ImageNet * `05_train_ms_from_scratch.py`: train VGG16 or DenseNet201 from scratch using multispectral data * `06_classify_image.py`: a simple implementation to classify images with trained models * `image_functions.py`: functions for image normalization and a simple generator for training data augmentation * `statistics.py`: a simple implementation to calculate normalization parameters (i.e. mean and std of training data) Additionally you will find the following notebooks: * `Image_functions.ipynb`: notebook of `image_functions.py` * `Train_from_Scratch.ipynb`: notebook of `05_train_ms_from_scratch.py` * `Transfer_learning.ipynb`: notebook of `02_train_rgb_finetuning.py` ## Requirements (what we used) We have defined the requirements in [requirements.txt](requirements.txt). We used: * python 3.6.x * tensorflow 2.2 * scikit-image (0.14.1) * gdal (2.2.4) for `06_classify_image.py` ## Frequently asked questions (FAQs) * **How can I interpret the classification results?** - Please have a look at our answers [#3](https://github.com/jensleitloff/CNN-Sentinel/issues/3), [#4](https://github.com/jensleitloff/CNN-Sentinel/issues/4), and [#6](https://github.com/jensleitloff/CNN-Sentinel/issues/6). * **Is there a paper I can cite for this repository?** - Please have a look at [Citation](#citation) ## Setup environment Append conda-forge to your Anaconda channels: ```bash conda config --append channels conda-forge ``` Create new environment: ```bash conda create -n pycon scikit-image gdal tqdm conda activate pycon pip install tensorflow-gpu pip install keras ``` (or use tensorflow version of keras, i.e. `from tensorflow import keras`) See also: * [Keras](https://keras.io/) ## Our talks about this topic ### Podcast episode @ TechTiefen * **Title:** "Fernerkundung mit multispektralen Satellitenbildern" * **Episode:** [Episode 18](https://techtiefen.de/18-fernerkundung-mit-multispektralen-satellitenbildern/) * **Podcast:** [TechTiefen](https://techtiefen.de) by Nico Kreiling * **Language:** German (Deutsch) * **Date:** July 2019
Abstract Jens Leitloff und Felix Riese berichten in Folge 18 von ihrer Forschung am “Institut für Photogrammetrie und Fernerkundung” des Karlsruher Instituts für Technologie. Mit der Bestrebung Nachhaltigkeit zu stärken erforschen die beiden etwa Verfahren, um Wasserqualität anhand von Satellitenaufnahmen zu bewerten oder die Nutzung landwirtschaftlicher Flächen zu kartografieren. Hierfür kommen unterschiedlichste Verfahren zum Einsatz wie Radaraufnahmen oder multispektrale Bilderdaten, die mehr als die drei von Menschen wahrnehmbaren Farbkanäle erfassen. Außerdem geht es um Drohnen, Satelliten und zahlreiche ML-Verfahren wie Transfer- und Aktive Learning. Persönliche Erfahrungen von Jens und Felix im Umgang mit unterschiedlichen Datenmengen runden eine thematisch Breite und anschauliche Folge ab.
### M3 Minds mastering machines 2019 @ Mannheim * **Title:** "Satellite Computer Vision mit Keras und Tensorflow - Best practices und beispiele aus der Forschung" * **Slides:** [Slides](slides/M3-2019_RieseLeitloff_SatelliteCV.pdf) * **Language:** German (Deutsch) * **Date:** 15 - 16 May 2019 * **DOI:** [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4056744.svg)](https://doi.org/10.5281/zenodo.4056744) * **URL:** [m3-konferenz.de](https://m3-konferenz.de/2019/)
Abstract > Im Forschungsfeld des Maschinellen Lernens werden zunehmend leicht zugängliche Framework wie Keras, Tensorflow oder Pytorch verwendet. Hierdurch ist ein Austausch und die Wiederverwendung bestehender (trainierter) neuronaler Netze möglich. > > Wir am Institut für Photogrammetrie und Fernerkundung (IPF) des Karlsruher Institut für Technologie (KIT) beschäftigen uns unter anderem mit der Analyse von optischen Satellitendaten. Satellitenprogramme wie Sentinel-2 von Copernicus liefern wöchentliche, weltweite und dabei frei zugängliche multispektrale Bilder, die eine Vielzahl neuartiger Anwendungen ermöglichen. Wir nehmen das zum Anlass, eine interaktive Einführung in die Auswertung dieser Satellitendaten mit Learnings aus unserer täglichen Forschung zu geben. Wir sprechen unter anderem über die folgenden Themen: > > * Einfacher Umgang mit georeferenzierten Bilddaten > * Einführung in Learning-From-Scratch und Transfer Learning mit Keras > * Anpassung von fertigen Netzen an neue Eingangsdaten (RGB → multispektral) > * Anschauliche Interpretation von Klassifikationsergebnissen > * Best Practices aus unserer Forschung, die die Arbeit mit Neuronalen Netzen wesentlich vereinfachen und beschleunigen > * Code und Daten für die ersten Schritte mit CNNs mit Keras in Python, welche in einem GitHub Repository zur Verfügung gestellt werden
### PyCon.DE 2018 @ Karlsruhe * **Title:** "Satellite data is for everyone: insights into modern remote sensing research with open data and Python" * **Slides:** [Slides](slides/PyCon2018_LeitloffRiese_SatelliteData.pdf) * **Video:** [youtube.com/watch?v=tKRoMcBeWjQ](https://www.youtube.com/watch?v=tKRoMcBeWjQ) * **Language:** English * **Date:** 24 - 28 October 2018 * **DOI:** [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4056516.svg)](https://doi.org/10.5281/zenodo.4056516) * **URL:** [de.pycon.org](https://de.pycon.org)
Abstract > The largest earth observation programme Copernicus (http://copernicus.eu) makes it possible to perform terrestrial observations providing data for all kinds of purposes. One important objective is to monitor the land-use and land-cover changes with the Sentinel-2 satellite mission. These satellites measure the sun reflectance on the earth surface with multispectral cameras (13 channels between 440 nm to 2190 nm). Machine learning techniques like convolutional neural networks (CNN) are able to learn the link between the satellite image (spectrum) and the ground truth (land use class). In this talk, we give an overview about the state-of-the-art land-use classification with CNNs based on an open dataset. > > We use different out-of-box CNNs for the Keras deep learning library (https://keras.io/). All networks are either included in Keras itself or are available from Github repositories. We show the process of transfer learning for the RGB datasets. Furthermore, the minimal changes required to apply commonly used CNNs to multispectral data are demonstrated. Thus, the interested audience will be able to perform their own classification of remote sensing data within a very short time. Results of different network structures are visually compared. Especially the differences of transfer learning and learning from scratch are demonstrated. This also includes the amount of necessary training epochs, progress of training and validation error and visual comparison of the results of the trained networks. Finally, we give a quick overview about the current research topics including recurrent neural networks for spatio-temporal land-use classification and further applications of multi- and hyperspectral data, e.g. for the estimation of water parameters and soil characteristics.
## Resources **This talk:** * EuroSAT Data (Sentinel-2, [Link](http://madm.dfki.de/downloads)) **Platforms for datasets:** * HyperLabelMe: a Web Platform for Benchmarking Remote Sensing Image Classifiers ([Link](http://hyperlabelme.uv.es/)) * GRSS Data and Algorithm Standard Evaluation (DASE) website ([Link](http://dase.ticinumaerospace.com/)) **Datasets:** * ISPRS 2D labeling challenge ([Link](http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html)) * UC Merced Land Use Dataset ([Link](http://weegee.vision.ucmerced.edu/datasets/landuse.html)) * AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification ([Link](https://captain-whu.github.io/AID/)) * NWPU-RESISC45 (RGB, [Link](http://www.escience.cn/people/JunweiHan/NWPU-RESISC45.html)) * Zurich Summer Dataset (RGB, [Link](https://sites.google.com/site/michelevolpiresearch/data/zurich-dataset)) * **Note**: Many German state authorities offer free geodata (high resolution images, land use/cover vector data, ...) over their geoportals. You can find an overview of all geoportals here ([geoportals](https://www.geoportal.nrw/geoportale_bundeslaender_nachbarstaaten)) **Image Segmentation Resources:** * More than 100 combinations for image segmentation routines with Keras and pretrained weights for endcoding phase ([Segmentation Models](https://github.com/qubvel/segmentation_models)) * Another source for image segmentation with Keras including pretrained weights ([Keras-FCN](https://github.com/aurora95/Keras-FCN)) * Great link collection of image segmantation networks and datasets ([Link](https://github.com/mrgloom/awesome-semantic-segmentation)) * Free land use vector data of NRW ([BasisDLM](https://www.bezreg-koeln.nrw.de/brk_internet/geobasis/landschaftsmodelle/basis_dlm/index.html) or [openNRW](https://open.nrw/en/node/154)) **Other:** * DeepHyperX - Deep learning for Hyperspectral imagery: [gitlab.inria.fr/naudeber/DeepHyperX/](https://gitlab.inria.fr/naudeber/DeepHyperX/) ## How to get Sentinel-2 data 1. Register at Copernicus [Open Access Hub](https://scihub.copernicus.eu/dhus/#/home) or [EarthExplorer](https://earthexplorer.usgs.gov/) 2. Find your region 3. Choose tile(s) (→ area) and date * Less tiles makes things easier * Less clouds in the image are better * Consider multiple dates for classes like “annual crop” 4. Download L1C data 5. Decide of you want to apply L2A atmospheric corrections * Your CNN might be able to do this by itself * If you want to correct, use [Sen2Cor](http://step.esa.int/main/third-party-plugins-2/sen2cor/) 6. Have fun with the data ## Citation Jens Leitloff and Felix M. Riese, "Examples for CNN training and classification on Sentinel-2 data", Zenodo, [10.5281/zenodo.3268451](http://doi.org/10.5281/zenodo.3268451), 2018. ```tex @misc{leitloff2018examples, author = {Leitloff, Jens and Riese, Felix~M.}, title = {{Examples for CNN training and classification on Sentinel-2 data}}, year = {2018}, DOI = {10.5281/zenodo.3268451}, publisher = {Zenodo}, howpublished = {\href{http://doi.org/10.5281/zenodo.3268451}{http://doi.org/10.5281/zenodo.3268451}} } ``` ================================================ FILE: bibliography.bib ================================================ @misc{leitloff2018examples, author = {Leitloff, Jens and Riese, Felix~M.}, title = {{Examples for CNN training and classification on Sentinel-2 data}}, year = {2018}, DOI = {10.5281/zenodo.3268451}, publisher = {Zenodo}, howpublished = {\href{http://doi.org/10.5281/zenodo.3268451}{http://doi.org/10.5281/zenodo.3268451}} } @inproceedings{leitloff2018satellite, author = {Leitloff, Jens and Riese, Felix~M.}, title = {{Satellite data is for everyone: insights into modern remote sensing research with open data and Python}}, year = {2018}, booktitle = {PyCon.DE 2018}, address = {Karlsruhe, Germany}, DOI = {10.5281/zenodo.4056516}, publisher = {Zenodo}, howpublished = {\href{http://doi.org/10.5281/zenodo.4056516}{http://doi.org/10.5281/zenodo.4056516}} } @inproceedings{leitloff2019satellite, author = {Leitloff, Jens and Riese, Felix~M.}, title = {{Satellite Computer Vision mit Keras und Tensorflow - Best practices und beispiele aus der Forschung}}, year = {2019}, booktitle = {Minds Mastering Machines (M3)}, address = {Mannheim, Germany}, DOI = {10.5281/zenodo.4056744}, publisher = {Zenodo}, howpublished = {\href{http://doi.org/10.5281/zenodo.4056744}{http://doi.org/10.5281/zenodo.4056744}} } ================================================ FILE: py/01_split_data_to_train_and_validation.py ================================================ # -*- coding: utf-8 -*- """ Code for the PyCon.DE 2018 talk by Jens Leitloff and Felix M. Riese. PyCon 2018 talk: Satellite data is for everyone: insights into modern remote sensing research with open data and Python. License: MIT """ import os import random import shutil random.seed(42) # variables # root path to folders "AnnualCrop, Forest ..." in home ("~") path_to_all_images = "~/Documents/Data/EuroSAT/AllBands" path_to_all_images = r'C:\Users\Jens\Downloads\EuroSAT_RGB' # path to new created folders "train" and "validation" with subfolders # "AnnualCrop, Forest ..." in home ("~") path_to_split_datasets = "~/Documents/Data/PyCon/AllBands" path_to_split_datasets = r'C:\Users\Jens\Downloads\PyCon\RGB' # percentage of validation data (between 0 an 1) percentage_validation = 0.3 # !!! If "True", complete "path_to_split_datasets" tree will be deleted !!! delete_old_path_to_split_datasets = True # contruct path path_to_home = os.path.expanduser("~") path_to_all_images = path_to_all_images.replace("~", path_to_home) path_to_split_datasets = path_to_split_datasets.replace("~", path_to_home) # create directories if necessary if delete_old_path_to_split_datasets and os.path.isdir(path_to_split_datasets): shutil.rmtree(path_to_split_datasets) path_to_train = os.path.join(path_to_split_datasets, "train") path_to_validation = os.path.join(path_to_split_datasets, "validation") if not os.path.isdir(path_to_train): os.makedirs(path_to_train) if not os.path.isdir(path_to_validation): os.makedirs(path_to_validation) # copy files sub_dirs = [sub_dir for sub_dir in os.listdir(path_to_all_images) if os.path.isdir(os.path.join(path_to_all_images, sub_dir))] for sub_dir in sub_dirs: # list and shuffle images in class directories current_dir = os.path.join(path_to_all_images, sub_dir) files = os.listdir(current_dir) random.shuffle(files) # split files into train and validation set split_idx = int(len(files)*percentage_validation) files_for_validation = files[:split_idx] files_for_train = files[split_idx:] # copy files to path_to_split_datasets if not os.path.isdir(os.path.join(path_to_train, sub_dir)): os.makedirs(os.path.join(path_to_train, sub_dir)) if not os.path.isdir(os.path.join(path_to_validation, sub_dir)): os.makedirs(os.path.join(path_to_validation, sub_dir)) for file in files_for_train: shutil.copy2(os.path.join(current_dir, file), os.path.join(path_to_train, sub_dir)) for file in files_for_validation: shutil.copy2(os.path.join(current_dir, file), os.path.join(path_to_validation, sub_dir)) ================================================ FILE: py/02_train_rgb_finetuning.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Code for the PyCon.DE 2018 talk by Jens Leitloff and Felix M. Riese. PyCon 2018 talk: Satellite data is for everyone: insights into modern remote sensing research with open data and Python. License: MIT """ import os from tensorflow.keras.applications.densenet import DenseNet201 as DenseNet from tensorflow.keras.applications.vgg16 import VGG16 as VGG from tensorflow.keras.callbacks import (EarlyStopping, ModelCheckpoint, TensorBoard) from tensorflow.keras.layers import Dense, GlobalAveragePooling2D from tensorflow.keras.models import Model from tensorflow.keras.optimizers import SGD from tensorflow.keras.preprocessing.image import ImageDataGenerator from image_functions import preprocessing_image_rgb # variables path_to_split_datasets = "~/Documents/Data/PyCon/RGB" use_vgg = True batch_size = 64 # contruct path path_to_home = os.path.expanduser("~") path_to_split_datasets = path_to_split_datasets.replace("~", path_to_home) path_to_train = os.path.join(path_to_split_datasets, "train") path_to_validation = os.path.join(path_to_split_datasets, "validation") # get number of classes sub_dirs = [sub_dir for sub_dir in os.listdir(path_to_train) if os.path.isdir(os.path.join(path_to_train, sub_dir))] num_classes = len(sub_dirs) # parameters for CNN if use_vgg: base_model = VGG(include_top=False, weights='imagenet', input_shape=(64, 64, 3)) else: base_model = DenseNet(include_top=False, weights='imagenet', input_shape=(64, 64, 3)) # add a global spatial average pooling layer top_model = base_model.output top_model = GlobalAveragePooling2D()(top_model) # or just flatten the layers # top_model = Flatten()(top_model) # let's add a fully-connected layer if use_vgg: # only in VGG19 a fully connected nn is added for classfication # DenseNet tends to overfitting if using additionally dense layers top_model = Dense(2048, activation='relu')(top_model) top_model = Dense(2048, activation='relu')(top_model) # and a logistic layer predictions = Dense(num_classes, activation='softmax')(top_model) # this is the model we will train model = Model(inputs=base_model.input, outputs=predictions) # print network structure model.summary() # defining ImageDataGenerators # ... initialization for training train_datagen = ImageDataGenerator( fill_mode="reflect", rotation_range=45, horizontal_flip=True, vertical_flip=True, preprocessing_function=preprocessing_image_rgb) # ... initialization for validation test_datagen = ImageDataGenerator( preprocessing_function=preprocessing_image_rgb) # ... definition for training train_generator = train_datagen.flow_from_directory(path_to_train, target_size=(64, 64), batch_size=batch_size, class_mode='categorical') # just for information class_indices = train_generator.class_indices print(class_indices) # ... definition for validation validation_generator = test_datagen.flow_from_directory( path_to_validation, target_size=(64, 64), batch_size=batch_size, class_mode='categorical') # first: train only the top layers (which were randomly initialized) # i.e. freeze all convolutional layers for layer in base_model.layers: layer.trainable = False # compile the model (should be done *after* setting layers to non-trainable) model.compile(optimizer='adadelta', loss='categorical_crossentropy', metrics=['categorical_accuracy']) # generate callback to save best model w.r.t val_categorical_accuracy if use_vgg: file_name = "vgg" else: file_name = "dense" checkpointer = ModelCheckpoint("../data/models/" + file_name + "_rgb_transfer_init." + "{epoch:02d}-{val_categorical_accuracy:.3f}." + "hdf5", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=10, mode='max', restore_best_weights=True) tensorboard = TensorBoard(log_dir='./logs', write_graph=True, write_images=True, update_freq='epoch') history = model.fit( train_generator, steps_per_epoch=1000, epochs=10000, callbacks=[checkpointer, earlystopper, tensorboard], validation_data=validation_generator, validation_steps=500) initial_epoch = len(history.history['loss'])+1 # at this point, the top layers are well trained and we can start fine-tuning # convolutional layers. We will freeze the bottom N layers # and train the remaining top layers. # let's visualize layer names and layer indices to see how many layers # we should freeze: names = [] for i, layer in enumerate(model.layers): names.append([i, layer.name, layer.trainable]) print(names) if use_vgg: # we will freaze the first convolutional block and train all # remaining blocks, including top layers. for layer in model.layers[:4]: layer.trainable = False for layer in model.layers[4:]: layer.trainable = True else: for layer in model.layers[:7]: layer.trainable = False for layer in model.layers[7:]: layer.trainable = True # we need to recompile the model for these modifications to take effect # we use SGD with a low learning rate model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metrics=['categorical_accuracy']) # generate callback to save best model w.r.t val_categorical_accuracy if use_vgg: file_name = "vgg" else: file_name = "dense" checkpointer = ModelCheckpoint("../data/models/" + file_name + "_rgb_transfer_final." + "{epoch:02d}-{val_categorical_accuracy:.3f}" + ".hdf5", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=50, mode='max') model.fit( train_generator, steps_per_epoch=1000, epochs=10000, callbacks=[checkpointer, earlystopper, tensorboard], validation_data=validation_generator, validation_steps=500, initial_epoch=initial_epoch) ================================================ FILE: py/03_train_rgb_from_scratch.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Code for the PyCon.DE 2018 talk by Jens Leitloff and Felix M. Riese. PyCon 2018 talk: Satellite data is for everyone: insights into modern remote sensing research with open data and Python. License: MIT """ import os from tensorflow.keras.applications.densenet import DenseNet201 as DenseNet from tensorflow.keras.applications.vgg16 import VGG16 as VGG from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint from tensorflow.keras.layers import Dense, GlobalAveragePooling2D from tensorflow.keras.models import Model from tensorflow.keras.preprocessing.image import ImageDataGenerator from image_functions import preprocessing_image_rgb # variables path_to_split_datasets = "~/Documents/Data/PyCon/RGB" use_vgg = False batch_size = 64 # contruct path path_to_home = os.path.expanduser("~") path_to_split_datasets = path_to_split_datasets.replace("~", path_to_home) path_to_train = os.path.join(path_to_split_datasets, "train") path_to_validation = os.path.join(path_to_split_datasets, "validation") # get number of classes sub_dirs = [sub_dir for sub_dir in os.listdir(path_to_train) if os.path.isdir(os.path.join(path_to_train, sub_dir))] num_classes = len(sub_dirs) # parameters for CNN if use_vgg: base_model = VGG(include_top=False, weights=None, input_shape=(64, 64, 3)) else: base_model = DenseNet(include_top=False, weights=None, input_shape=(64, 64, 3)) # add a global spatial average pooling layer top_model = base_model.output top_model = GlobalAveragePooling2D()(top_model) # or just flatten the layers # top_model = Flatten()(top_model) # let's add a fully-connected layer if use_vgg: # only in VGG19 a fully connected nn is added for classfication # DenseNet tends to overfitting if using additionally dense layers top_model = Dense(2048, activation='relu')(top_model) top_model = Dense(2048, activation='relu')(top_model) # and a logistic layer predictions = Dense(num_classes, activation='softmax')(top_model) # this is the model we will train model = Model(inputs=base_model.input, outputs=predictions) # print network structure model.summary() # defining ImageDataGenerators # ... initialization for training train_datagen = ImageDataGenerator( fill_mode="reflect", rotation_range=45, horizontal_flip=True, vertical_flip=True, preprocessing_function=preprocessing_image_rgb) # ... initialization for validation test_datagen = ImageDataGenerator( preprocessing_function=preprocessing_image_rgb) # ... definition for training train_generator = train_datagen.flow_from_directory(path_to_train, target_size=(64, 64), batch_size=batch_size, class_mode='categorical') print(train_generator.class_indices) # ... definition for validation validation_generator = test_datagen.flow_from_directory( path_to_validation, target_size=(64, 64), batch_size=batch_size, class_mode='categorical') # compile the model (should be done *after* setting layers to non-trainable) model.compile(optimizer='adadelta', loss='categorical_crossentropy', metrics=['categorical_accuracy']) # generate callback to save best model w.r.t val_categorical_accuracy if use_vgg: file_name = "vgg" else: file_name = "dense" checkpointer = ModelCheckpoint("../data/models/" + file_name + "_rgb_from_scratch." + "{epoch:02d}-{val_categorical_accuracy:.3f}" + ".hdf5", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=50, mode='max') model.fit( train_generator, steps_per_epoch=1000, epochs=10000, callbacks=[checkpointer, earlystopper], validation_data=validation_generator, validation_steps=500) ================================================ FILE: py/04_train_ms_finetuning.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Code for the PyCon.DE 2018 talk by Jens Leitloff and Felix M. Riese. PyCon 2018 talk: Satellite data is for everyone: insights into modern remote sensing research with open data and Python. License: MIT """ import os from glob import glob from tensorflow.keras.applications.densenet import DenseNet201 as DenseNet from tensorflow.keras.applications.vgg16 import VGG16 as VGG from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint from tensorflow.keras.layers import (Conv2D, Dense, GlobalAveragePooling2D, Input) from tensorflow.keras.models import Model from tensorflow.keras.optimizers import SGD from image_functions import simple_image_generator # variables path_to_split_datasets = "~/Documents/Data/PyCon/AllBands" use_vgg = False batch_size = 64 class_indices = {'AnnualCrop': 0, 'Forest': 1, 'HerbaceousVegetation': 2, 'Highway': 3, 'Industrial': 4, 'Pasture': 5, 'PermanentCrop': 6, 'Residential': 7, 'River': 8, 'SeaLake': 9} num_classes = len(class_indices) # contruct path path_to_home = os.path.expanduser("~") path_to_split_datasets = path_to_split_datasets.replace("~", path_to_home) path_to_train = os.path.join(path_to_split_datasets, "train") path_to_validation = os.path.join(path_to_split_datasets, "validation") # parameters for CNN input_tensor = Input(shape=(64, 64, 13)) # introduce a additional layer to get from 13 to 3 input channels input_tensor = Conv2D(3, (1, 1))(input_tensor) if use_vgg: base_model_imagenet = VGG(include_top=False, weights='imagenet', input_shape=(64, 64, 3)) base_model = VGG(include_top=False, weights=None, input_tensor=input_tensor) for i, layer in enumerate(base_model_imagenet.layers): # we must skip input layer, which has no weights if i == 0: continue base_model.layers[i+1].set_weights(layer.get_weights()) else: base_model_imagenet = DenseNet(include_top=False, weights='imagenet', input_shape=(64, 64, 3)) base_model = DenseNet(include_top=False, weights=None, input_tensor=input_tensor) for i, layer in enumerate(base_model_imagenet.layers): # we must skip input layer, which has no weights if i == 0: continue base_model.layers[i+1].set_weights(layer.get_weights()) # add a global spatial average pooling layer top_model = base_model.output top_model = GlobalAveragePooling2D()(top_model) # or just flatten the layers # top_model = Flatten()(top_model) # let's add a fully-connected layer if use_vgg: # only in VGG19 a fully connected nn is added for classfication # DenseNet tends to overfitting if using additionally dense layers top_model = Dense(2048, activation='relu')(top_model) top_model = Dense(2048, activation='relu')(top_model) # and a logistic layer predictions = Dense(num_classes, activation='softmax')(top_model) # this is the model we will train model = Model(inputs=base_model.input, outputs=predictions) # print network structure model.summary() # defining ImageDataGenerators # ... initialization for training training_files = glob(path_to_train + "/**/*.tif") train_generator = simple_image_generator(training_files, class_indices, batch_size=batch_size, rotation_range=45, horizontal_flip=True, vertical_flip=True) # ... initialization for validation validation_files = glob(path_to_validation + "/**/*.tif") validation_generator = simple_image_generator(validation_files, class_indices, batch_size=batch_size) # first: train only the top layers (which were randomly initialized) # i.e. freeze all convolutional layers for layer in base_model.layers: layer.trainable = False # set convolution block for reducing 13 to 3 layers trainable for layer in model.layers[:2]: layer.trainable = True # compile the model (should be done *after* setting layers to non-trainable) model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy']) # generate callback to save best model w.r.t val_categorical_accuracy if use_vgg: file_name = "vgg" else: file_name = "dense" checkpointer = ModelCheckpoint("../data/models/" + file_name + "_ms_transfer_init." + "{epoch:02d}-{val_categorical_accuracy:.3f}." + "hdf5", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=10, mode='max', restore_best_weights=True) history = model.fit( train_generator, steps_per_epoch=1000, epochs=10000, callbacks=[checkpointer, earlystopper], validation_data=validation_generator, validation_steps=500) initial_epoch = len(history.history['loss'])+1 # at this point, the top layers are well trained and we can start fine-tuning # convolutional layers. We will freeze the bottom N layers # and train the remaining top layers. # let's visualize layer names and layer indices to see how many layers # we should freeze: names = [] for i, layer in enumerate(model.layers): names.append([i, layer.name, layer.trainable]) print(names) if use_vgg: # we will freaze the first convolutional block and train all # remaining blocks, including top layers. for layer in model.layers[:2]: layer.trainable = True for layer in model.layers[2:5]: layer.trainable = False for layer in model.layers[5:]: layer.trainable = True else: for layer in model.layers[:2]: layer.trainable = True for layer in model.layers[2:8]: layer.trainable = False for layer in model.layers[8:]: layer.trainable = True # we need to recompile the model for these modifications to take effect # we use SGD with a low learning rate model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metrics=['categorical_accuracy']) # generate callback to save best model w.r.t val_categorical_accuracy if use_vgg: file_name = "vgg" else: file_name = "dense" checkpointer = ModelCheckpoint("../data/models/" + file_name + "_ms_transfer_final." + "{epoch:02d}-{val_categorical_accuracy:.3f}" + ".hdf5", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=10, mode='max') model.fit( train_generator, steps_per_epoch=1000, epochs=10000, callbacks=[checkpointer, earlystopper], validation_data=validation_generator, validation_steps=500, initial_epoch=initial_epoch) ================================================ FILE: py/04_train_ms_finetuning_alternative.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Code for the PyCon.DE 2018 talk by Jens Leitloff and Felix M. Riese. PyCon 2018 talk: Satellite data is for everyone: insights into modern remote sensing research with open data and Python. License: MIT """ import os from glob import glob # from tensorflow.keras.applications.vgg19 import VGG19 as VGG # from tensorflow.keras.applications.densenet import DenseNet121 as DenseNet from tensorflow.keras.applications.densenet import DenseNet201 as DenseNet from tensorflow.keras.applications.vgg16 import VGG16 as VGG from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint from tensorflow.keras.layers import Dense, GlobalAveragePooling2D from tensorflow.keras.models import Model from tensorflow.keras.optimizers import SGD from image_functions import simple_image_generator # variables path_to_split_datasets = "~/Dokumente/Data/PyCon/AllBands" use_vgg = False batch_size = 64 class_indices = {'AnnualCrop': 0, 'Forest': 1, 'HerbaceousVegetation': 2, 'Highway': 3, 'Industrial': 4, 'Pasture': 5, 'PermanentCrop': 6, 'Residential': 7, 'River': 8, 'SeaLake': 9} num_classes = len(class_indices) # contruct path path_to_home = os.path.expanduser("~") path_to_split_datasets = path_to_split_datasets.replace("~", path_to_home) path_to_train = os.path.join(path_to_split_datasets, "train") path_to_validation = os.path.join(path_to_split_datasets, "validation") # parameters for CNN if use_vgg: base_model_imagenet = VGG(include_top=False, weights='imagenet', input_shape=(64, 64, 3)) base_model = VGG(include_top=False, weights=None, input_shape=(64, 64, 13)) for i, layer in enumerate(base_model_imagenet.layers): # we must skip input layer and first convolutional layer if i < 2: continue base_model.layers[i].set_weights(layer.get_weights()) else: base_model_imagenet = DenseNet(include_top=False, weights='imagenet', input_shape=(64, 64, 3)) base_model = DenseNet(include_top=False, weights=None, input_shape=(64, 64, 13)) for i, layer in enumerate(base_model_imagenet.layers): # we must skip input layer, zeropadding and first convolutional layer if i < 3: continue base_model.layers[i].set_weights(layer.get_weights()) # add a global spatial average pooling layer top_model = base_model.output top_model = GlobalAveragePooling2D()(top_model) # or just flatten the layers # top_model = Flatten()(top_model) # let's add a fully-connected layer if use_vgg: # only in VGG19 a fully connected nn is added for classfication # DenseNet tends to overfitting if using additionally dense layers top_model = Dense(2048, activation='relu')(top_model) top_model = Dense(2048, activation='relu')(top_model) # and a logistic layer predictions = Dense(num_classes, activation='softmax')(top_model) # this is the model we will train model = Model(inputs=base_model.input, outputs=predictions) # print network structure model.summary() # defining ImageDataGenerators # ... initialization for training training_files = glob(path_to_train + "/**/*.tif") train_generator = simple_image_generator(training_files, class_indices, batch_size=batch_size, rotation_range=45, horizontal_flip=True, vertical_flip=True) # ... initialization for validation validation_files = glob(path_to_validation + "/**/*.tif") validation_generator = simple_image_generator(validation_files, class_indices, batch_size=batch_size) # first: train only the top layers (which were randomly initialized) # i.e. freeze all convolutional layers for layer in base_model.layers: layer.trainable = False # set first convolution block for reducing 13 to 3 layers trainable for layer in model.layers[:3]: layer.trainable = True # compile the model (should be done *after* setting layers to non-trainable) model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy']) # generate callback to save best model w.r.t val_categorical_accuracy if use_vgg: file_name = "vgg" else: file_name = "dense" checkpointer = ModelCheckpoint("../data/models/" + file_name + "_ms_transfer_alternative_init." + "{epoch:02d}-{val_categorical_accuracy:.3f}." + "hdf5", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=10, mode='max', restore_best_weights=True) history = model.fit( train_generator, steps_per_epoch=1000, epochs=10000, callbacks=[checkpointer, earlystopper], validation_data=validation_generator, validation_steps=500) initial_epoch = len(history.history['loss'])+1 # at this point, the top layers are well trained and we can start fine-tuning # convolutional layers. We will freeze the bottom N layers # and train the remaining top layers. # let's visualize layer names and layer indices to see how many layers # we should freeze: names = [] for i, layer in enumerate(model.layers): names.append([i, layer.name, layer.trainable]) print(names) if use_vgg: # we will freaze the first convolutional block and train all # remaining blocks, including top layers. for layer in model.layers[:2]: layer.trainable = True for layer in model.layers[2:5]: layer.trainable = False for layer in model.layers[5:]: layer.trainable = True else: for layer in model.layers[:3]: layer.trainable = True for layer in model.layers[3:7]: layer.trainable = False for layer in model.layers[7:]: layer.trainable = True # we need to recompile the model for these modifications to take effect # we use SGD with a low learning rate model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metrics=['categorical_accuracy']) # generate callback to save best model w.r.t val_categorical_accuracy if use_vgg: file_name = "vgg" else: file_name = "dense" checkpointer = ModelCheckpoint("../data/models/" + file_name + "_ms_transfer_alternative_final." + "{epoch:02d}-{val_categorical_accuracy:.3f}" + ".hdf5", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=10, mode='max') model.fit( train_generator, steps_per_epoch=1000, epochs=10000, callbacks=[checkpointer, earlystopper], validation_data=validation_generator, validation_steps=500, initial_epoch=initial_epoch) ================================================ FILE: py/05_train_ms_from_scratch.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Code for the PyCon.DE 2018 talk by Jens Leitloff and Felix M. Riese. PyCon 2018 talk: Satellite data is for everyone: insights into modern remote sensing research with open data and Python. License: MIT """ import os from glob import glob from tensorflow.keras.applications.densenet import DenseNet201 as DenseNet from tensorflow.keras.applications.vgg16 import VGG16 as VGG from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint from tensorflow.keras.layers import Dense, GlobalAveragePooling2D from tensorflow.keras.models import Model from image_functions import simple_image_generator # variables path_to_split_datasets = "~/Documents/Data/PyCon/AllBands" use_vgg = False batch_size = 64 class_indices = {'AnnualCrop': 0, 'Forest': 1, 'HerbaceousVegetation': 2, 'Highway': 3, 'Industrial': 4, 'Pasture': 5, 'PermanentCrop': 6, 'Residential': 7, 'River': 8, 'SeaLake': 9} num_classes = len(class_indices) # contruct path path_to_home = os.path.expanduser("~") path_to_split_datasets = path_to_split_datasets.replace("~", path_to_home) path_to_train = os.path.join(path_to_split_datasets, "train") path_to_validation = os.path.join(path_to_split_datasets, "validation") # parameters for CNN if use_vgg: base_model = VGG(include_top=False, weights=None, input_shape=(64, 64, 13)) else: base_model = DenseNet(include_top=False, weights=None, input_shape=(64, 64, 13)) # add a global spatial average pooling layer top_model = base_model.output top_model = GlobalAveragePooling2D()(top_model) # or just flatten the layers # top_model = Flatten()(top_model) # let's add a fully-connected layer if use_vgg: # only in VGG19 a fully connected nn is added for classfication # DenseNet tends to overfitting if using additionally dense layers top_model = Dense(2048, activation='relu')(top_model) top_model = Dense(2048, activation='relu')(top_model) # and a logistic layer predictions = Dense(num_classes, activation='softmax')(top_model) # this is the model we will train model = Model(inputs=base_model.input, outputs=predictions) # print network structure model.summary() # defining ImageDataGenerators # ... initialization for training training_files = glob(path_to_train + "/**/*.tif") train_generator = simple_image_generator(training_files, class_indices, batch_size=batch_size, rotation_range=45, horizontal_flip=True, vertical_flip=True) # ... initialization for validation validation_files = glob(path_to_validation + "/**/*.tif") validation_generator = simple_image_generator(validation_files, class_indices, batch_size=batch_size) # compile the model model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy']) # generate callback to save best model w.r.t val_categorical_accuracy if use_vgg: file_name = "vgg" else: file_name = "dense" checkpointer = ModelCheckpoint("../data/models/" + file_name + "_ms_from_scratch." + "{epoch:02d}-{val_categorical_accuracy:.3f}." + "hdf5", monitor='val_categorical_accuracy', verbose=1, save_best_only=True, mode='max') earlystopper = EarlyStopping(monitor='val_categorical_accuracy', patience=50, mode='max', restore_best_weights=True) model.fit( train_generator, steps_per_epoch=1000, epochs=10000, callbacks=[checkpointer, earlystopper], validation_data=validation_generator, validation_steps=500) ================================================ FILE: py/06_classify_image.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ Code for the PyCon.DE 2018 talk by Jens Leitloff and Felix M. Riese. PyCon 2018 talk: Satellite data is for everyone: insights into modern remote sensing research with open data and Python. License: MIT """ import gdal import numpy as np from skimage.io import imread from skimage.util import pad from tensorflow.keras.models import load_model from tqdm import tqdm # input files path_to_image = "../data/karlsruhe/2018_zugeschnitten.tif" path_to_model = "../data/models/vgg/vgg_ms_transfer_alternative_final.27-0.985.hdf5" # output files path_to_label_image = "../data/karlsruhe/2018_zugeschnitten_10m_vgg_ms_label.tif" path_to_prob_image = "../data/karlsruhe/2018_zugeschnitten_10m_vgg_ms_prob.tif" # read image and model image = np.array(imread(path_to_image), dtype=float) _, num_cols_unpadded, _ = image.shape model = load_model(path_to_model) # get input shape of model _, input_rows, input_cols, input_channels = model.layers[0].input_shape _, output_classes = model.layers[-1].output_shape in_rows_half = int(input_rows/2) in_cols_half = int(input_cols/2) # import correct preprocessing if input_channels is 3: from image_functions import preprocessing_image_rgb as preprocessing_image else: from image_functions import preprocessing_image_ms as preprocessing_image # pad image image = pad(image, ((input_rows, input_rows), (input_cols, input_cols), (0, 0)), 'symmetric') # don't forget to preprocess image = preprocessing_image(image) num_rows, num_cols, _ = image.shape # sliding window over image image_classified_prob = np.zeros((num_rows, num_cols, output_classes)) row_images = np.zeros((num_cols_unpadded, input_rows, input_cols, input_channels)) for row in tqdm(range(input_rows, num_rows-input_rows), desc="Processing..."): # get all images along one row for idx, col in enumerate(range(input_cols, num_cols-input_cols)): # cut smal image patch row_images[idx, ...] = image[row-in_rows_half:row+in_rows_half, col-in_cols_half:col+in_cols_half, :] # classify images row_classified = model.predict(row_images, batch_size=1024, verbose=0) # put them to final image image_classified_prob[row, input_cols:num_cols-input_cols, : ] = row_classified # crop padded final image image_classified_prob = image_classified_prob[input_rows:num_rows-input_rows, input_cols:num_cols-input_cols, :] image_classified_label = np.argmax(image_classified_prob, axis=-1) image_classified_prob = np.sort(image_classified_prob, axis=-1)[..., -1] # write image as Geotiff for correct georeferencing # read geotransformation image = gdal.Open(path_to_image, gdal.GA_ReadOnly) geotransform = image.GetGeoTransform() # create image driver driver = gdal.GetDriverByName('GTiff') # create destination for label file file = driver.Create(path_to_label_image, image_classified_label.shape[1], image_classified_label.shape[0], 1, gdal.GDT_Byte, ['TFW=YES', 'NUM_THREADS=1']) file.SetGeoTransform(geotransform) file.SetProjection(image.GetProjection()) # write label file file.GetRasterBand(1).WriteArray(image_classified_label) file = None # create destination for probability file file = driver.Create(path_to_prob_image, image_classified_prob.shape[1], image_classified_prob.shape[0], 1, gdal.GDT_Float32, ['TFW=YES', 'NUM_THREADS=1']) file.SetGeoTransform(geotransform) file.SetProjection(image.GetProjection()) # write label file file.GetRasterBand(1).WriteArray(image_classified_prob) file = None image = None ================================================ FILE: py/Image_functions.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Notebook for image_functions.py" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "from random import choice, sample\n", "\n", "import numpy as np\n", "from skimage.io import imread\n", "from skimage.transform import rotate\n", "from tensorflow.keras.utils import to_categorical" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preprocessing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### RGB data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def preprocessing_image_rgb(x):\n", " # define mean and std values\n", " mean = [87.845, 96.965, 103.947]\n", " std = [23.657, 16.474, 13.793]\n", " # loop over image channels\n", " for idx, mean_value in enumerate(mean):\n", " x[..., idx] -= mean_value\n", " x[..., idx] /= std[idx]\n", " return x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### MS data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def preprocessing_image_ms(x):\n", " # define mean and std values\n", " mean = [1353.036, 1116.468, 1041.475, 945.344, 1198.498, 2004.878,\n", " 2376.699, 2303.738, 732.957, 12.092, 1818.820, 1116.271, 2602.579]\n", " std = [65.479, 154.008, 187.997, 278.508, 228.122, 356.598, 456.035,\n", " 531.570, 98.947, 1.188, 378.993, 303.851, 503.181]\n", " # loop over image channels\n", " for idx, mean_value in enumerate(mean):\n", " x[..., idx] -= mean_value\n", " x[..., idx] /= std[idx]\n", " return x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Image Generator for MS data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Get label from file name" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def categorical_label_from_full_file_name(files, class_indices):\n", " # file basename without extension\n", " base_name = [os.path.splitext(os.path.basename(i))[0] for i in files]\n", " # class label from filename\n", " base_name = [i.split(\"_\")[0] for i in base_name]\n", " # label to indices\n", " image_class = [class_indices[i] for i in base_name]\n", " # class indices to one-hot-label\n", " return to_categorical(image_class, num_classes=len(class_indices))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Generate images" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def simple_image_generator(files, class_indices, batch_size=32,\n", " rotation_range=0, horizontal_flip=False,\n", " vertical_flip=False):\n", "\n", " while True:\n", " # select batch_size number of samples without replacement\n", " batch_files = sample(files, batch_size)\n", " # get one_hot_label\n", " batch_Y = categorical_label_from_full_file_name(batch_files,\n", " class_indices)\n", " # array for images\n", " batch_X = []\n", " # loop over images of the current batch\n", " for idx, input_path in enumerate(batch_files):\n", " image = np.array(imread(input_path), dtype=float)\n", " image = preprocessing_image_ms(image)\n", " # process image\n", " if horizontal_flip:\n", " # randomly flip image up/down\n", " if choice([True, False]):\n", " image = np.flipud(image)\n", " if vertical_flip:\n", " # randomly flip image left/right\n", " if choice([True, False]):\n", " image = np.fliplr(image)\n", " # rotate image by random angle between\n", " # -rotation_range <= angle < rotation_range\n", " if rotation_range is not 0:\n", " angle = np.random.uniform(low=-abs(rotation_range),\n", " high=abs(rotation_range))\n", " image = rotate(image, angle, mode='reflect',\n", " order=1, preserve_range=True)\n", " # put all together\n", " batch_X += [image]\n", " # convert lists to np.array\n", " X = np.array(batch_X)\n", " Y = np.array(batch_Y)\n", "\n", " yield(X, Y)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: py/Train_from_Scratch.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Notebook for 05_train_ms_from_scratch.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Import libaries" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "from glob import glob\n", "\n", "from tensorflow.keras.applications.vgg16 import VGG16 as VGG\n", "from tensorflow.keras.applications.densenet import DenseNet201 as DenseNet\n", "from tensorflow.keras.layers import GlobalAveragePooling2D, Dense\n", "from tensorflow.keras.models import Model\n", "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard\n", "\n", "from image_functions import simple_image_generator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### define path to training and validation data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# variables\n", "path_to_split_datasets = \"~/Documents/Data/PyCon/AllBands\"\n", "use_vgg = False\n", "batch_size = 64\n", "\n", "# contruct path\n", "path_to_home = os.path.expanduser(\"~\")\n", "path_to_split_datasets = path_to_split_datasets.replace(\"~\", path_to_home)\n", "path_to_train = os.path.join(path_to_split_datasets, \"train\")\n", "path_to_validation = os.path.join(path_to_split_datasets, \"validation\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![tree](images_for_notebook/tree_files.png \"file_tree\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### define classes" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "class_indices = {'AnnualCrop': 0, 'Forest': 1, 'HerbaceousVegetation': 2,\n", " 'Highway': 3, 'Industrial': 4, 'Pasture': 5,\n", " 'PermanentCrop': 6, 'Residential': 7, 'River': 8,\n", " 'SeaLake': 9}\n", "num_classes = len(class_indices)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training from scratch" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![vgg16](images_for_notebook/vgg16.png \"Original VGG\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Initialize network model without top layers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![vgg16_no_top](images_for_notebook/vgg16_no_top.png \"VGG no top\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# parameters for CNN\n", "if use_vgg:\n", " base_model = VGG(include_top=False,\n", " weights=None,\n", " input_shape=(64, 64, 13))\n", "else:\n", " base_model = DenseNet(include_top=False,\n", " weights=None,\n", " input_shape=(64, 64, 13))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. define new top layers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![vgg16_sentinel_rgb](images_for_notebook/vgg16_sentinel_rgb.png \"VGG RGB Sentinel\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# add a global spatial average pooling layer\n", "top_model = base_model.output\n", "top_model = GlobalAveragePooling2D()(top_model)\n", "# or just flatten the layers\n", "# top_model = Flatten()(top_model)\n", "# let's add a fully-connected layer\n", "if use_vgg:\n", " # only in VGG19 a fully connected nn is added for classfication\n", " # DenseNet tends to overfitting if using additionally dense layers\n", " top_model = Dense(2048, activation='relu')(top_model)\n", " top_model = Dense(2048, activation='relu')(top_model)\n", "# and a logistic layer\n", "predictions = Dense(num_classes, activation='softmax')(top_model)\n", "# this is the model we will train\n", "model = Model(inputs=base_model.input, outputs=predictions)\n", "# print network structure\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. define data augmentation" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# defining ImageDataGenerators\n", "# ... initialization for training\n", "training_files = glob(path_to_train + \"/**/*.tif\")\n", "train_generator = simple_image_generator(training_files, class_indices,\n", " batch_size=batch_size,\n", " rotation_range=45,\n", " horizontal_flip=True,\n", " vertical_flip=True)\n", "\n", "# ... initialization for validation\n", "validation_files = glob(path_to_validation + \"/**/*.tif\")\n", "validation_generator = simple_image_generator(validation_files, class_indices,\n", " batch_size=batch_size)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. define callbacks" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# generate callback to save best model w.r.t val_categorical_accuracy\n", "if use_vgg:\n", " file_name = \"vgg\"\n", "else:\n", " file_name = \"dense\"\n", "checkpointer = ModelCheckpoint(\"../data/models/\" + file_name +\n", " \"_ms_from_scratch.\" +\n", " \"{epoch:02d}-{val_categorical_accuracy:.3f}.\" +\n", " \"hdf5\",\n", " monitor='val_categorical_accuracy',\n", " verbose=1,\n", " save_best_only=True,\n", " mode='max')\n", "earlystopper = EarlyStopping(monitor='val_categorical_accuracy',\n", " patience=50,\n", " mode='max',\n", " restore_best_weights=True)\n", "\n", "tensorboard = TensorBoard(log_dir='./logs', write_graph=True,\n", " write_images=True, update_freq='epoch')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![tensorflow](images_for_notebook/tensorflow.png \"VGG RGB Sentinel\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8. fit model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# compile the model\n", "model.compile(optimizer='adam', loss='categorical_crossentropy',\n", " metrics=['categorical_accuracy'])\n", "\n", "model.fit(\n", " train_generator,\n", " steps_per_epoch=100,\n", " epochs=5,\n", " callbacks=[checkpointer, earlystopper, tensorboard],\n", " validation_data=validation_generator,\n", " validation_steps=500)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: py/Transfer_learning.ipynb ================================================ { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Note for 02_train_rgb_finetuning.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Import libaries" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] } ], "source": [ "import os\n", "\n", "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n", "from tensorflow.keras.applications.vgg16 import VGG16 as VGG\n", "from tensorflow.keras.applications.densenet import DenseNet201 as DenseNet\n", "from tensorflow.keras.optimizers import SGD\n", "from tensorflow.keras.layers import GlobalAveragePooling2D, Dense\n", "from tensorflow.keras.models import Model\n", "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, TensorBoard\n", "\n", "from image_functions import preprocessing_image_rgb" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### define path to training and validation data" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# variables\n", "path_to_split_datasets = \"~/Documents/Data/PyCon/RGB\"\n", "use_vgg = True\n", "batch_size = 64\n", "\n", "# contruct path\n", "path_to_home = os.path.expanduser(\"~\")\n", "path_to_split_datasets = path_to_split_datasets.replace(\"~\", path_to_home)\n", "path_to_train = os.path.join(path_to_split_datasets, \"train\")\n", "path_to_validation = os.path.join(path_to_split_datasets, \"validation\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![tree](images_for_notebook/tree_files.png \"file_tree\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### determine number of classes from data" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# get number of classes\n", "sub_dirs = [sub_dir for sub_dir in os.listdir(path_to_train)\n", " if os.path.isdir(os.path.join(path_to_train, sub_dir))]\n", "num_classes = len(sub_dirs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Transfer-learning " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![vgg16](images_for_notebook/vgg16.png \"Original VGG\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Pretrained network model without top layers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![vgg16_no_top](images_for_notebook/vgg16_no_top.png \"VGG no top\")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# parameters for CNN\n", "if use_vgg:\n", " base_model = VGG(include_top=False,\n", " weights='imagenet',\n", " input_shape=(64, 64, 3))\n", "else:\n", " base_model = DenseNet(include_top=False,\n", " weights='imagenet',\n", " input_shape=(64, 64, 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. define new top layers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![vgg16_sentinel_rgb](images_for_notebook/vgg16_sentinel_rgb.png \"VGG RGB Sentinel\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# add a global spatial average pooling layer\n", "top_model = base_model.output\n", "top_model = GlobalAveragePooling2D()(top_model)\n", "# or just flatten the layers\n", "# top_model = Flatten()(top_model)\n", "# let's add a fully-connected layer\n", "if use_vgg:\n", " # only in VGG19 a fully connected nn is added for classfication\n", " # DenseNet tends to overfitting if using additionally dense layers\n", " top_model = Dense(2048, activation='relu')(top_model)\n", " top_model = Dense(2048, activation='relu')(top_model)\n", "# and a logistic layer\n", "predictions = Dense(num_classes, activation='softmax')(top_model)\n", "\n", "# this is the model we will train\n", "model = Model(inputs=base_model.input, outputs=predictions)\n", "\n", "# print network structure\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. define data augmentation" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found 18900 images belonging to 10 classes.\n", "{'AnnualCrop': 0, 'Forest': 1, 'HerbaceousVegetation': 2, 'Highway': 3, 'Industrial': 4, 'Pasture': 5, 'PermanentCrop': 6, 'Residential': 7, 'River': 8, 'SeaLake': 9}\n", "Found 8100 images belonging to 10 classes.\n" ] } ], "source": [ "# defining ImageDataGenerators\n", "# ... initialization for training\n", "train_datagen = ImageDataGenerator(fill_mode=\"reflect\",\n", " rotation_range=45,\n", " horizontal_flip=True,\n", " vertical_flip=True,\n", " preprocessing_function=preprocessing_image_rgb)\n", "# ... initialization for validation\n", "test_datagen = ImageDataGenerator(preprocessing_function=preprocessing_image_rgb)\n", "# ... definition for training\n", "train_generator = train_datagen.flow_from_directory(path_to_train,\n", " target_size=(64, 64),\n", " batch_size=batch_size,\n", " class_mode='categorical')\n", "# just for information\n", "class_indices = train_generator.class_indices\n", "print(class_indices)\n", "\n", "# ... definition for validation\n", "validation_generator = test_datagen.flow_from_directory(path_to_validation,\n", " target_size=(64, 64),\n", " batch_size=batch_size,\n", " class_mode='categorical')\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. define callbacks" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "# generate callback to save best model w.r.t val_categorical_accuracy\n", "if use_vgg:\n", " file_name = \"vgg\"\n", "else:\n", " file_name = \"dense\"\n", "\n", "checkpointer = ModelCheckpoint(\"../data/models/\" + file_name +\n", " \"_rgb_transfer_init.\" +\n", " \"{epoch:02d}-{val_categorical_accuracy:.3f}.\" +\n", " \"hdf5\",\n", " monitor='val_categorical_accuracy',\n", " verbose=1,\n", " save_best_only=True,\n", " mode='max')\n", "\n", "earlystopper = EarlyStopping(monitor='val_categorical_accuracy',\n", " patience=10,\n", " mode='max',\n", " restore_best_weights=True)\n", "\n", "tensorboard = TensorBoard(log_dir='./logs', write_graph=True, \n", " write_images=True, update_freq='epoch')\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![tensorflow](images_for_notebook/tensorflow.png \"VGG RGB Sentinel\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5. set base layers non trainable " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![vgg16_rgb_init](images_for_notebook/vgg16_rgb_init.png \"VGG RGB Sentinel\")" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "# first: train only the top layers (which were randomly initialized)\n", "# i.e. freeze all convolutional layers\n", "for layer in base_model.layers:\n", " layer.trainable = False\n", "\n", "# compile the model (should be done *after* setting layers to non-trainable)\n", "model.compile(optimizer='adadelta', loss='categorical_crossentropy',\n", " metrics=['categorical_accuracy'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6. fit model (train new top layers)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/5\n", "100/100 [==============================] - 18s 182ms/step - loss: 0.6681 - categorical_accuracy: 0.7841 - val_loss: 0.3120 - val_categorical_accuracy: 0.8931\n", "\n", "Epoch 00001: val_categorical_accuracy improved from -inf to 0.89309, saving model to ../data/models/vgg_rgb_transfer_init.01-0.893.hdf5\n", "Epoch 2/5\n", "100/100 [==============================] - 17s 174ms/step - loss: 0.3477 - categorical_accuracy: 0.8808 - val_loss: 0.2733 - val_categorical_accuracy: 0.9039\n", "\n", "Epoch 00002: val_categorical_accuracy improved from 0.89309 to 0.90388, saving model to ../data/models/vgg_rgb_transfer_init.02-0.904.hdf5\n", "Epoch 3/5\n", "100/100 [==============================] - 17s 172ms/step - loss: 0.3098 - categorical_accuracy: 0.8961 - val_loss: 0.2515 - val_categorical_accuracy: 0.9126\n", "\n", "Epoch 00003: val_categorical_accuracy improved from 0.90388 to 0.91263, saving model to ../data/models/vgg_rgb_transfer_init.03-0.913.hdf5\n", "Epoch 4/5\n", "100/100 [==============================] - 17s 174ms/step - loss: 0.2719 - categorical_accuracy: 0.9106 - val_loss: 0.2330 - val_categorical_accuracy: 0.9200\n", "\n", "Epoch 00004: val_categorical_accuracy improved from 0.91263 to 0.92003, saving model to ../data/models/vgg_rgb_transfer_init.04-0.920.hdf5\n", "Epoch 5/5\n", "100/100 [==============================] - 17s 172ms/step - loss: 0.2601 - categorical_accuracy: 0.9136 - val_loss: 0.3209 - val_categorical_accuracy: 0.8893\n", "\n", "Epoch 00005: val_categorical_accuracy did not improve from 0.92003\n" ] } ], "source": [ "history = model.fit(train_generator,\n", " steps_per_epoch=100,\n", " epochs=5,\n", " callbacks=[checkpointer, earlystopper,\n", " tensorboard],\n", " validation_data=validation_generator,\n", " validation_steps=500)\n", "initial_epoch = len(history.history['loss'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 7. set (some) base layers trainable" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![vgg16_rgb_finetune](images_for_notebook/vgg16_rgb_finetune.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# at this point, the top layers are well trained and we can start fine-tuning\n", "# convolutional layers. We will freeze the bottom N layers\n", "# and train the remaining top layers.\n", "\n", "# let's visualize layer names and layer indices to see how many layers\n", "# we should freeze:\n", "names = []\n", "for i, layer in enumerate(model.layers):\n", " print([i, layer.name, layer.trainable])\n", "\n", "if use_vgg:\n", " # we will freaze the first convolutional block and train all\n", " # remaining blocks, including top layers.\n", " for layer in model.layers[:4]:\n", " layer.trainable = False\n", " for layer in model.layers[4:]:\n", " layer.trainable = True\n", "else:\n", " for layer in model.layers[:7]:\n", " layer.trainable = False\n", " for layer in model.layers[7:]:\n", " layer.trainable = True\n", "\n", "# we need to recompile the model for these modifications to take effect\n", "# we use SGD with a low learning rate\n", "model.compile(optimizer=SGD(lr=0.0001, momentum=0.9),\n", " loss='categorical_crossentropy',\n", " metrics=['categorical_accuracy'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8. fit model (fine-tune base and top layers)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 6/10\n", "100/100 [==============================] - 18s 181ms/step - loss: 0.1926 - categorical_accuracy: 0.9299 - val_loss: 0.1883 - val_categorical_accuracy: 0.9356\n", "\n", "Epoch 00006: val_categorical_accuracy improved from -inf to 0.93564, saving model to ../data/models/vgg_rgb_transfer_final.06-0.936.hdf5\n", "Epoch 7/10\n", "100/100 [==============================] - 18s 178ms/step - loss: 0.1697 - categorical_accuracy: 0.9409 - val_loss: 0.1771 - val_categorical_accuracy: 0.9399\n", "\n", "Epoch 00007: val_categorical_accuracy improved from 0.93564 to 0.93985, saving model to ../data/models/vgg_rgb_transfer_final.07-0.940.hdf5\n", "Epoch 8/10\n", "100/100 [==============================] - 18s 176ms/step - loss: 0.1614 - categorical_accuracy: 0.9436 - val_loss: 0.1674 - val_categorical_accuracy: 0.9430\n", "\n", "Epoch 00008: val_categorical_accuracy improved from 0.93985 to 0.94299, saving model to ../data/models/vgg_rgb_transfer_final.08-0.943.hdf5\n", "Epoch 9/10\n", "100/100 [==============================] - 17s 174ms/step - loss: 0.1489 - categorical_accuracy: 0.9515 - val_loss: 0.1619 - val_categorical_accuracy: 0.9466\n", "\n", "Epoch 00009: val_categorical_accuracy improved from 0.94299 to 0.94663, saving model to ../data/models/vgg_rgb_transfer_final.09-0.947.hdf5\n", "Epoch 10/10\n", "100/100 [==============================] - 17s 174ms/step - loss: 0.1279 - categorical_accuracy: 0.9544 - val_loss: 0.1482 - val_categorical_accuracy: 0.9483\n", "\n", "Epoch 00010: val_categorical_accuracy improved from 0.94663 to 0.94829, saving model to ../data/models/vgg_rgb_transfer_final.10-0.948.hdf5\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# generate callback to save best model w.r.t val_categorical_accuracy\n", "if use_vgg:\n", " file_name = \"vgg\"\n", "else:\n", " file_name = \"dense\"\n", "checkpointer = ModelCheckpoint(\"../data/models/\" + file_name +\n", " \"_rgb_transfer_final.\" +\n", " \"{epoch:02d}-{val_categorical_accuracy:.3f}\" +\n", " \".hdf5\",\n", " monitor='val_categorical_accuracy',\n", " verbose=1,\n", " save_best_only=True,\n", " mode='max')\n", "earlystopper = EarlyStopping(monitor='val_categorical_accuracy',\n", " patience=50,\n", " mode='max')\n", "model.fit(train_generator,\n", " steps_per_epoch=100,\n", " epochs=initial_epoch+5,\n", " callbacks=[checkpointer, earlystopper, tensorboard],\n", " validation_data=validation_generator,\n", " validation_steps=500,\n", " initial_epoch=initial_epoch)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 2 } ================================================ FILE: py/image_functions.py ================================================ # -*- coding: utf-8 -*- import os from random import choice, sample import numpy as np from skimage.io import imread from skimage.transform import rotate from tensorflow.keras.utils import to_categorical def preprocessing_image_rgb(x): # define mean and std values mean = [87.845, 96.965, 103.947] std = [23.657, 16.474, 13.793] # loop over image channels for idx, mean_value in enumerate(mean): x[..., idx] -= mean_value x[..., idx] /= std[idx] return x def preprocessing_image_ms(x): # define mean and std values mean = [1353.036, 1116.468, 1041.475, 945.344, 1198.498, 2004.878, 2376.699, 2303.738, 732.957, 12.092, 1818.820, 1116.271, 2602.579] std = [65.479, 154.008, 187.997, 278.508, 228.122, 356.598, 456.035, 531.570, 98.947, 1.188, 378.993, 303.851, 503.181] # loop over image channels for idx, mean_value in enumerate(mean): x[..., idx] -= mean_value x[..., idx] /= std[idx] return x def categorical_label_from_full_file_name(files, class_indices): # file basename without extension base_name = [os.path.splitext(os.path.basename(i))[0] for i in files] # class label from filename base_name = [i.split("_")[0] for i in base_name] # label to indices image_class = [class_indices[i] for i in base_name] # class indices to one-hot-label return to_categorical(image_class, num_classes=len(class_indices)) def simple_image_generator(files, class_indices, batch_size=32, rotation_range=0, horizontal_flip=False, vertical_flip=False): while True: # select batch_size number of samples without replacement batch_files = sample(files, batch_size) # get one_hot_label batch_Y = categorical_label_from_full_file_name(batch_files, class_indices) # array for images batch_X = [] # loop over images of the current batch for idx, input_path in enumerate(batch_files): image = np.array(imread(input_path), dtype=float) image = preprocessing_image_ms(image) # process image if horizontal_flip: # randomly flip image up/down if choice([True, False]): image = np.flipud(image) if vertical_flip: # randomly flip image left/right if choice([True, False]): image = np.fliplr(image) # rotate image by random angle between # -rotation_range <= angle < rotation_range if rotation_range is not 0: angle = np.random.uniform(low=-abs(rotation_range), high=abs(rotation_range)) image = rotate(image, angle, mode='reflect', order=1, preserve_range=True) # put all together batch_X += [image] # convert lists to np.array X = np.array(batch_X) Y = np.array(batch_Y) yield(X, Y) ================================================ FILE: py/statistics.py ================================================ # -*- coding: utf-8 -*- """Some statistics about the EuroSAT dataset.""" import glob import os import numpy as np from osgeo import gdal def getMeanStd(path, n_bands=3, n_max=-1): """Get mean and standard deviation from images. Parameters ---------- path : str Path to training images n_bands : int Number of spectral bands (3 for RGB, 13 for Sentinel-2) n_max : int Maximum number of iterations (-1 = all) Return ------ """ if not os.path.isdir(path): print("Error: Directory does not exist.") return 0 mean_array = [[] for _ in range(n_bands)] std_array = [[] for _ in range(n_bands)] # iterate over the images i = 0 for tif in glob.glob(path+"*/*.*"): if (i < n_max) or (n_max == -1): ds = gdal.Open(tif) for band in range(n_bands): mean_array[band].append( np.mean(ds.GetRasterBand(band+1).ReadAsArray())) std_array[band].append( np.std(ds.GetRasterBand(band+1).ReadAsArray())) i+=1 else: break # results res_mean = [np.mean(mean_array[band]) for band in range(n_bands)] res_std = [np.mean(std_array[band]) for band in range(n_bands)] # print results table print("Band | Mean | Std") print("-"*28) for band in range(n_bands): print("{band:4d} | {mean:8.3f} | {std:8.3f}".format( band=band, mean=res_mean[band], std=res_std[band])) return res_mean, res_std if __name__ == "__main__": getMeanStd(path="data/PyCon/RGB/train/", n_bands=3) ================================================ FILE: requirements.txt ================================================ tensorflow>=2.1.0 skimage>=0.14.1 gdal==2.2.4 tqdm>=4.0.0 numpy>=1.17.4