Repository: azavea/raster-vision-examples
Branch: master
Commit: d6957a5de6d4
Files: 30
Total size: 161.9 KB
Directory structure:
gitextract_o_v_juae/
├── .dockerignore
├── .gitignore
├── Dockerfile
├── LICENSE
├── README.md
├── docker/
│ ├── build
│ └── run
├── examples/
│ ├── __init__.py
│ ├── cowc/
│ │ ├── __init__.py
│ │ └── object_detection.py
│ ├── potsdam/
│ │ ├── __init__.py
│ │ └── semantic_segmentation.py
│ ├── spacenet/
│ │ ├── __init__.py
│ │ ├── rio/
│ │ │ ├── __init__.py
│ │ │ ├── chip_classification.py
│ │ │ └── semantic_segmentation.py
│ │ └── vegas/
│ │ ├── __init__.py
│ │ ├── all.py
│ │ ├── hyperparameters.py
│ │ └── simple_segmentation.py
│ ├── test.py
│ ├── utils.py
│ └── xview/
│ ├── __init__.py
│ └── object_detection.py
├── notebooks/
│ ├── spacenet/
│ │ └── spacenet_rio_chip_class_data_prep.ipynb
│ └── xview/
│ └── xview-vehicles-data-prep.ipynb
├── qgis/
│ ├── spacenet_viz.py
│ └── style.qml
└── scripts/
├── console
└── debug
================================================
FILE CONTENTS
================================================
================================================
FILE: .dockerignore
================================================
data/
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg
# Jupyter
.ipynb_checkpoints
./data/
data/
.vscode/*
================================================
FILE: Dockerfile
================================================
ARG BASE_IMAGE
FROM $BASE_IMAGE
COPY ./examples /opt/src/examples
CMD ["bash"]
================================================
FILE: LICENSE
================================================
This software is licensed under the Apache 2 license, quoted below.
Copyright 2017 Azavea [http://www.azavea.com]
Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at
[http://www.apache.org/licenses/LICENSE-2.0]
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.
================================================
FILE: README.md
================================================
# Raster Vision Examples (for RV < 0.12)
This repository contains examples of using Raster Vision on open datasets.
## ⚠️ For RV >= 0.12, the examples have moved into the [main repo](https://docs.rastervision.io/en/0.12/examples.html).
Note: The `master` branch of this examples repo should be used in conjunction with the `master` branch (or `latest` Docker image tag) of [Raster Vision](https://github.com/azavea/raster-vision)
which contains the latest changes. For versions of this examples repo that correspond to stable, released versions of Raster Vision, see:
* [0.11](https://github.com/azavea/raster-vision-examples/tree/0.11)
* [0.10](https://github.com/azavea/raster-vision-examples/tree/0.10)
* [0.9](https://github.com/azavea/raster-vision-examples/tree/0.9)
Table of Contents:
- [Setup and Requirements](#setup-and-requirements)
- [How to Run an Example](#how-to-run-an-example)
- [SpaceNet Rio Building Chip Classification (demonstrates data prep)](#spacenet-rio-building-chip-classification)
- [Visualization using QGIS](#visualization-using-qgis)
- [Other examples](#other-examples)
- [SpaceNet Rio Building Semantic Segmentation](#spacenet-rio-building-semantic-segmentation)
- [SpaceNet Vegas](#spacenet-vegas)
- [Simple Segmentation (easy to run and understand)](#spacenet-vegas-simple-semantic-segmentation)
- [Roads and Buildings: All Tasks (demonstrates multiple tasks, line string, class filter, vector tiles, polygon output)](#spacenet-vegas-all-tasks)
- [Hyperparameter Search (demonstrates branching experiment structure)](#spacenet-vegas-hyperparameter-search)
- [ISPRS Potsdam Semantic Segmentation (demonstrates RGB input/output, multiple classes)](#isprs-potsdam-semantic-segmentation)
- [COWC Potsdam Car Object Detection](#cowc-potsdam-car-object-detection)
- [xView Vehicle Object Detection (demonstrates data prep)](#xview-vehicle-object-detection)
- [Model Zoo](#model-zoo)
## Setup and Requirements
### ⚠️ PyTorch vs. Tensorflow Backends
We have recently added a set of PyTorch-based backends to Raster Vision. The existing Tensorflow-based backends are still there, but we do not plan on maintaining them, so we suggest starting to use the PyTorch ones. The examples in this repo default to using the PyTorch backends. However, for three of the examples there is a `use_tf` option which allows running it using a Tensorflow backend: [examples.cowc.object_detection](examples/potsdam/object_detection.py), [examples.potsdam.semantic_segmentation](examples/potsdam/semantic_segmentation.py), and [examples.spacenet.rio.chip_classification](examples/spacenet/rio/chip_classification.py).
### Docker
You'll need `docker` (preferably version 18 or above) installed. After cloning this repo, to build the Docker images, run the following command:
```shell
> docker/build
```
This will pull down the latest `raster-vision:pytorch-latest`, `raster-vision:tf-cpu-latest`, and `raster-vision:tf-gpu-latest` Docker images and add some of this repo's code to them. If you only want the Tensorflow images, use the `--tf` flag, and similar for `--pytorch`. Before running the container, set an environment variable to a local directory in which to store data.
```shell
> export RASTER_VISION_DATA_DIR="/path/to/data"
```
To run a Bash console in the Docker container, invoke:
```shell
> docker/run
```
This will mount the following local directories to directories inside the container:
* `$RASTER_VISION_DATA_DIR -> /opt/data/`
* `examples/ -> /opt/src/examples/`
This script also has options for forwarding AWS credentials, running Jupyter notebooks, and switching between different images, which can be seen below.
**Remember to use the correct image for the backend you are using!**
```
> ./docker/run --help
Usage: run <options> <command>
Run a console in the raster-vision-examples-cpu Docker image locally.
Environment variables:
RASTER_VISION_DATA_DIR (directory for storing data; mounted to /opt/data)
AWS_PROFILE (optional AWS profile)
RASTER_VISION_REPO (optional path to main RV repo; mounted to /opt/src)
Options:
--aws forwards AWS credentials (sets AWS_PROFILE env var and mounts ~/.aws to /root/.aws)
--tensorboard maps port 6006
--gpu use the NVIDIA runtime and GPU image
--name sets the name of the running container
--jupyter forwards port 8888, mounts ./notebooks to /opt/notebooks, and runs Jupyter
--debug maps port 3007 on localhost to 3000 inside container
--tf-gpu use raster-vision-examples-tf-gpu image and nvidia runtime
--tf-cpu use raster-vision-examples-tf-cpu image
--pytorch-gpu use raster-vision-examples-pytorch image and nvidia runtime
Note: raster-vision-examples-pytorch image is used by default
All arguments after above options are passed to 'docker run'.
```
### Debug Mode
For debugging, it can be helpful to use a local copy of the Raster Vision source code rather than the version baked into the default Docker image. To do this, you can set the `RASTER_VISION_REPO` environment variable to the location of the main repo on your local filesystem. If this is set, `docker/build` will set the base image to `raster-vision-{cpu,gpu}`, and `docker/run` will mount `$RASTER_VISION_REPO/rastervision` to `/opt/src/rastervision` inside the container. You can then set breakpoints in your local copy of Raster Vision in order to debug experiments running inside the container.
## How to Run an Example
There is a common structure across all of the examples which represents a best practice for defining experiments. Running an example involves the following steps.
* Acquire raw dataset.
* (Optional) Get processed dataset which is derived from the raw dataset, either using a Jupyter notebook, or by downloading the processed dataset.
* (Optional) Do an abbreviated test run of the experiment on a small subset of data locally.
* Run full experiment on GPU.
* Inspect output
* (Optional) Make predictions on new imagery
Each of the [experiments](examples/) has several arguments that can be set on the command line:
* The input data for each experiment is divided into two directories: the raw data which is publicly distributed, and the processed data which is derived from the raw data. These two directories are set using the `raw_uri` and `processed_uri` arguments.
* The output generated by the experiment is stored in the directory set by the `root_uri` argument.
* The `raw_uri`, `processed_uri`, and `root_uri` can each be local or remote (on S3), and don't need to agree on whether they are local or remote.
* Experiments have a `test` argument which runs an abbreviated experiment for testing/debugging purposes.
In the next section, we describe in detail how to run one of the examples, SpaceNet Rio Chip Classification. For other examples, we only note example-specific details.
### SpaceNet Rio Building Chip Classification
This example performs chip classification to detect buildings on the Rio AOI of the [SpaceNet](https://spacenetchallenge.github.io/) dataset.
#### Step 1: Acquire Raw Dataset
The dataset is stored on AWS S3 at `s3://spacenet-dataset`. You will need an AWS account to access this dataset, but it will not be charged for accessing it. (To forward you AWS credentials into the container, use `docker/run --aws`).
Optional: to run this example with the data stored locally, first copy the data using something like the following inside the container.
```
aws s3 sync s3://spacenet-dataset/AOIs/AOI_1_Rio/ /opt/data/spacenet-dataset/AOIs/AOI_1_Rio/
```
#### Step 2: Run the Jupyter Notebook
You'll need to do some data preprocessing, which we can do in the Jupyter notebook supplied.
```shell
> docker/run --jupyter [--aws]
```
The `--aws` option is only needed if pulling data from S3. In Jupyter inside the browser, navigate to the [spacenet/spacenet_rio_chip_classification_data_prep.ipynb](notebooks/spacenet/spacenet_rio_chip_class_data_prep.ipynb) notebook. Set the URIs in the first cell and then run the rest of the notebook. Set the `processed_uri` to a local or S3 URI depending on where you want to run the experiment.
#### Step 3: Do a test run locally
The experiment we want to run is in [examples/spacenet/rio/chip_classification.py](examples/spacenet/rio/chip_classification.py). To run this, first get to the Docker console using:
```shell
> docker/run [--aws] [--gpu] [--tensorboard]
```
The `--aws` option is only needed if running experiments on AWS or using data stored on S3. The `--gpu` option should only be used if running on a local GPU.
The `--tensorboard` option should be used if running locally and you would like to view Tensorboard. The test run can be executed using something like:
```shell
export RAW_URI="s3://spacenet-dataset/"
export PROCESSED_URI="/opt/data/examples/spacenet/rio/processed-data"
export ROOT_URI="/opt/data/examples/spacenet/rio/local-output"
rastervision run local -e examples.spacenet.rio.chip_classification \
-a raw_uri $RAW_URI -a processed_uri $PROCESSED_URI -a root_uri $ROOT_URI \
-a test True --splits 2
```
The sample above assumes that the raw data is on S3, and the processed data and output are stored locally. The `raw_uri` directory is assumed to contain an `AOI_1_Rio` subdirectory. This runs two parallel jobs for the `chip` and `predict` commands via `--splits 2`. See `rastervision --help` and `rastervision run --help` for more usage information.
Note that when running with `-a test True`, some crops of the test scenes are created and stored in `processed_uri/crops/`. All of the examples that use big image files use this trick to make the experiment run faster in test mode.
After running this, the main thing to check is that it didn't crash, and that the debug chips look correct. The debug chips can be found in the debug zip files in `$ROOT_URI/chip/spacenet-rio-chip-classification/`.
#### Step 4: Run full experiment
To run the full experiment on GPUs using AWS Batch, use something like the following. Note that all the URIs are on S3 since remote instances will not have access to your local file system.
```shell
export RAW_URI="s3://spacenet-dataset/"
export PROCESSED_URI="s3://mybucket/examples/spacenet/rio/processed-data"
export ROOT_URI="s3://mybucket/examples/spacenet/rio/remote-output"
rastervision run aws_batch -e examples.spacenet.rio.chip_classification \
-a raw_uri $RAW_URI -a processed_uri $PROCESSED_URI -a root_uri $ROOT_URI \
-a test False --splits 8
```
For instructions on setting up AWS Batch resources and configuring Raster Vision to use them, see [AWS Batch Setup](https://docs.rastervision.io/en/latest/setup.html#aws-batch-setup). To monitor the training process using Tensorboard, visit `<public dns>:6006` for the EC2 instance running the training job.
If you would like to run on a local GPU, replace `aws_batch` with `local`, and use local URIs. To monitor the training process using Tensorboard, visit `localhost:6006`, assuming you used `docker/run --tensorboard`.
#### Step 5: Inspect results
After everything completes, which should take about 1.5 hours if you're running on AWS using a `p3.2xlarge` instance for training and 8 splits, you should be able to find the predictions over the validation scenes in `$root_uri/predict/spacenet-rio-chip-classification/`. The evaluation metrics can be found in `$root_uri/eval/spacenet-rio-chip-classification/eval.json`. This is an example of the scores from a run, which show an F1 score of `0.96` for detecting chips with buildings.
```javascript
[
{
"gt_count": 1460.0,
"count_error": 0.0,
"f1": 0.962031922725018,
"class_name": "building",
"recall": 0.9527397260273971,
"precision": 0.9716098420590342,
"class_id": 1
},
{
"gt_count": 2314.0,
"count_error": 0.0,
"f1": 0.9763865660344931,
"class_name": "no_building",
"recall": 0.9822817631806394,
"precision": 0.9706292067263268,
"class_id": 2
},
{
"gt_count": 3774.0,
"count_error": 0.0,
"f1": 0.970833365390128,
"class_name": "average",
"recall": 0.9708532061473236,
"precision": 0.9710085728062825,
"class_id": -1
}
]
```
#### Step 6: Predict on new imagery
After running an experiment, a *predict package* is saved into `$root_uri/bundle/spacenet-rio-chip-classification/`. This can be used to make predictions on new images. See the [Model Zoo](#model-zoo) section for more details.
## Visualization using QGIS
To visualize a Raster Vision experiment, you can use [QGIS](https://qgis.org/en/site/) to display the imagery, ground truth, and predictions associated with each scene. Although it's possible to just drag and drop files into QGIS, it's often more convenient to write a script to do this. Here is an example of a [script](qgis/spacenet_viz.py) to visualize the results for [SpaceNet Vegas Semantic Segmentation](#spacenet-vegas).
## Other Examples
### SpaceNet Rio Building Semantic Segmentation
This [experiment](examples/spacenet/rio/semantic_segmentation.py) trains a semantic segmentation model to find buildings using the [SpaceNet Rio](https://spacenetchallenge.github.io/AOI_Lists/AOI_1_Rio.html) dataset. A prerequisite is running the [Rio Chip Classification](#spacenet-rio-building-chip-classification) Jupyter notebook, and all other details are the same as in that example.
Below are sample predictions and eval metrics.
<details><summary>Eval Metrics</summary>
```json
"overall": [
{
"recall": 0.6933642097495366,
"precision": 0.7181072275154092,
"class_name": "Building",
"gt_count": 11480607,
"count_error": 119679.64457523893,
"f1": 0.7023217656506746,
"class_id": 1
},
{
"recall": 0.978149141560173,
"precision": 0.9763586125303796,
"class_name": "Background",
"gt_count": 147757124,
"count_error": 31820.188126279452,
"f1": 0.9771849696422493,
"class_id": 2
},
{
"recall": 0.9576169230896666,
"precision": 0.9577393905661922,
"class_name": "average",
"gt_count": 159237731,
"count_error": 38154.615804881076,
"f1": 0.9573680807430468,
"class_id": null
}
]
```
</details>
### SpaceNet Vegas
This is a collection of examples using the [SpaceNet Vegas](https://spacenetchallenge.github.io/AOI_Lists/AOI_2_Vegas.html) dataset.
#### SpaceNet Vegas Simple Semantic Segmentation
This [experiment](examples/spacenet/vegas/simple_segmentation.py) is a simple example of doing semantic segmentation: the code is simple, there is no need to pre-process any data, and you don't need permission to use the data.
Arguments:
* `raw_uri` should be set to the root of the SpaceNet data repository, which is at `s3://spacenet-dataset`, or a local copy of it. A copy only needs to contain the `SpaceNet_Buildings_Dataset_Round2/spacenetV2_Train/AOI_2_Vegas` subdirectory.
* `processed_uri` should not be set because there is no processed data in this example.
Below are sample predictions and eval metrics.
<details><summary>Eval Metrics</summary>
```json
[
{
"class_id": 1,
"precision": 0.9166443308607926,
"recall": 0.7788752910479124,
"gt_count": 62924777,
"count_error": 31524.39656560088,
"class_name": "Building",
"f1": 0.8387483150445183
},
{
"class_id": 2,
"precision": 0.9480938442744736,
"recall": 0.9648479452702291,
"gt_count": 262400223,
"count_error": 29476.379317139523,
"class_name": "Background",
"f1": 0.9527945047747147
},
{
"class_id": null,
"precision": 0.942010839223173,
"recall": 0.9288768769691843,
"gt_count": 325325000,
"count_error": 29872.509429032507,
"class_name": "average",
"f1": 0.930735545099091
}
]
```
</details>
#### SpaceNet Vegas Roads and Buildings: All Tasks <a name="spacenet-vegas-all-tasks"></a>
This [experiment](examples/spacenet/vegas/all.py) can be configured to do any of the three tasks on either roads or buildings. It is an example of how to structure experiment code to support a variety of options. It also demonstrates how to utilize line strings as labels for roads using buffering, and generating polygon output for semantic segmentation on buildings.
Arguments:
* `raw_uri` should be set to the root of the SpaceNet data repository, which is at `s3://spacenet-dataset`, or a local copy of it. For buildings, a copy only needs to contain the `SpaceNet_Buildings_Dataset_Round2/spacenetV2_Train/AOI_2_Vegas` subdirectory.
For roads, `SpaceNet_Roads_Competition/Train/AOI_2_Vegas_Roads_Train`.
* `processed_uri` should not be set because there is no processed data in this example.
* `task_type` can be set to `chip_classification`, `object_detection`, or `semantic_segmentation`.
* `target` can be `buildings` or `roads`
Note that for semantic segmentation on buildings, polygon output in the form of GeoJSON files will be saved to the `predict` directory alongside the GeoTIFF files. In addition, a vector evaluation file using SpaceNet metrics will be saved to the `eval` directory. Running semantic segmentation on roads trains a Mobilenet for 100k steps which takes about 6hrs on a P3 instance.
Sample predictions and eval metrics can be seen below.
<details><summary>Eval Metrics</summary>
```json
[
{
"count_error": 131320.3497452814,
"precision": 0.79827727905979,
"f1": 0.7733719736453241,
"class_name": "Road",
"class_id": 1,
"recall": 0.7574370618553649,
"gt_count": 47364639
},
{
"count_error": 213788.03361026093,
"precision": 0.9557015578601281,
"f1": 0.909516065847437,
"class_name": "Background",
"class_id": 2,
"recall": 0.8988113906793058,
"gt_count": 283875361
},
{
"count_error": 201995.82229692052,
"precision": 0.9331911601569118,
"f1": 0.8900485625895702,
"class_name": "average",
"class_id": null,
"recall": 0.8785960059171598,
"gt_count": 331240000
}
]
```
</details>
##### Variant: Use vector tiles to get labels
It is possible to use vector tiles as a source of labels, either in `z/x/y` or `.mbtiles` format. To use vector tiles instead of GeoJSON, run the experiment with the following argument: `-a vector_tile_options "<uri>,<zoom>,<id_field>"`. See the [vector tile docs](https://docs.rastervision.io/en/latest/api.html#rv-vector-tile-source) for a description of these arguments.
To run this example using OSM tiles in `.mbtiles` format, first create an extract around Las Vegas using:
```shell
cd /opt/data
wget https://s3.amazonaws.com/mapbox/osm-qa-tiles-production/latest.country/united_states_of_america.mbtiles.gz
# unzipping takes a few minutes
gunzip united_states_of_america.mbtiles.gz
npm install tilelive-copy
tilelive-copy \
--minzoom=0 --maxzoom=14 \
--bounds="-115.43472290039062,35.98689628443789,-114.91836547851562,36.361586786517776" \ united_states_of_america.mbtiles vegas.mbtiles
```
Using the entire USA file would be very slow. Then run the roads example using something like `-a vector_tile_options "/opt/data/vegas.mbtiles,12,@id"`.
If you are not using OSM tiles, you might need to change the `class_id_to_filter` values in the experiment configuration. Each `class_id_to_filter` is a mapping from `class_id` to a Mapbox GL filter which is to used to assign class ids to features based on their `properties` field.
#### SpaceNet Vegas Hyperparameter Search
This [experiment set](examples/spacenet/vegas/hyperparameters.py) runs several related experiments in which the base learning rate varies over them. These experiments are all related, in that they all work over the same dataset (SpaceNet Vegas buildings), and in fact the `analyze` and `chip` stages are shared between all of the experiments.
That sharing of early stages is achieving by making sure that the `chip_key` and `analyze_key` are [the same for all of the experiments](examples/spacenet/vegas/hyperparameters.py#L80-L81) so that Raster Vision can detect the redundancy.
Arguments:
* `raw_uri` should be set to the root of the SpaceNet data repository, which is at `s3://spacenet-dataset`, or a local copy of it. A copy only needs to contain the `SpaceNet_Buildings_Dataset_Round2/spacenetV2_Train/AOI_2_Vegas` subdirectory.
* `processed_uri` should not be set because there is no processed data in this example.
* `learning_rates` is a comma-delimited list of learning rates to use for the experiments. Example: `-a learning_rates '0.0001,0.001,0.002,0.003,0.004,0.005,0.10'`
The number of steps is 10,000 for all experiments. Because this is for demonstration purposes only, the training dataset has been reduced to only 128 scenes.
The F1 scores for buildings as a function of base learning rate are shown below.
| Base Learning Rate | Building F1 Score |
| ------------------- | ------------------ |
| 0.0001 | 0.7337961752864327 |
| 0.001 | 0.7616993477580662 |
| 0.002 | 0.7889177881341606 |
| 0.003 | 0.7864549469541627 |
| 0.004 | 0.4194065664072375 |
| 0.005 | 0.5070458576486434 |
| 0.1 | 0.5046626369613472 |
Disclaimer: We are not claiming that the numbers above are useful or interesting, the sole intent here is demonstrate how to vary hyperparameters using Raster Vision.
### ISPRS Potsdam Semantic Segmentation
This [experiment](examples/potsdam/semantic_segmentation.py) performs semantic segmentation on the [ISPRS Potsdam dataset](http://www2.isprs.org/commissions/comm3/wg4/2d-sem-label-potsdam.html). The dataset consists of 5cm aerial imagery over Potsdam, Germany, segmented into six classes including building, tree, low vegetation, impervious, car, and clutter. For more info see our [blog post](https://www.azavea.com/blog/2017/05/30/deep-learning-on-aerial-imagery/).
Data:
* The dataset can only be downloaded after filling in this [request form](http://www2.isprs.org/commissions/comm3/wg4/data-request-form2.html). After your request is granted, follow the link to 'POTSDAM 2D LABELING' and download and unzip `4_Ortho_RGBIR.zip`, and `5_Labels_for_participants.zip` into a directory, and then upload to S3 if desired.
Arguments:
* `raw_uri` should contain `4_Ortho_RGBIR` and `5_Labels_for_participants` subdirectories.
* `processed_uri` should be set to a directory which will be used to store test crops.
Below are sample predictions and eval metrics.
<details><summary>Eval Metrics</summary>
```javascript
[
{
"precision": 0.9003686311706696,
"recall": 0.8951149482868683,
"f1": 0.8973353554371246,
"count_error": 129486.40233074076,
"gt_count": 1746655.0,
"conf_mat": [
0.0,
1563457.0,
7796.0,
5679.0,
10811.0,
126943.0,
31969.0
],
"class_id": 1,
"class_name": "Car"
},
{
"precision": 0.9630047813515502,
"recall": 0.9427071079228886,
"f1": 0.9525027991356272,
"count_error": 1000118.8466519706,
"gt_count": 28166583.0,
"conf_mat": [
0.0,
6976.0,
26552838.0,
743241.0,
71031.0,
556772.0,
235725.0
],
"class_id": 2,
"class_name": "Building"
},
{
"precision": 0.8466609755403327,
"recall": 0.8983221897241067,
"f1": 0.8715991836041085,
"count_error": 3027173.8852443425,
"gt_count": 30140893.0,
"conf_mat": [
0.0,
4306.0,
257258.0,
27076233.0,
1405095.0,
1110647.0,
287354.0
],
"class_id": 3,
"class_name": "Low Vegetation"
},
{
"precision": 0.883517319858661,
"recall": 0.8089167109558072,
"f1": 0.8439042868078945,
"count_error": 1882745.6869677808,
"gt_count": 16928529.0,
"conf_mat": [
0.0,
34522.0,
157012.0,
2484523.0,
13693770.0,
485790.0,
72912.0
],
"class_id": 4,
"class_name": "Tree"
},
{
"precision": 0.9123212945945467,
"recall": 0.9110533473255575,
"f1": 0.9115789047144218,
"count_error": 1785561.1048684688,
"gt_count": 29352493.0,
"conf_mat": [
0.0,
99015.0,
451628.0,
1307686.0,
262292.0,
26741687.0,
490185.0
],
"class_id": 5,
"class_name": "Impervious"
},
{
"precision": 0.42014399072332975,
"recall": 0.47418711749488085,
"f1": 0.44406088467218563,
"count_error": 787395.6814824425,
"gt_count": 1664847.0,
"conf_mat": [
0.0,
28642.0,
157364.0,
340012.0,
59034.0,
290346.0,
789449.0
],
"class_id": 6,
"class_name": "Clutter"
},
{
"precision": 0.8949197573420392,
"recall": 0.8927540185185187,
"f1": 0.8930493260224918,
"count_error": 1900291.674768574,
"gt_count": 108000000.0,
"conf_mat": [
[
0.0,
0.0,
0.0,
0.0,
0.0,
0.0,
0.0
],
[
0.0,
1563457.0,
7796.0,
5679.0,
10811.0,
126943.0,
31969.0
],
[
0.0,
6976.0,
26552838.0,
743241.0,
71031.0,
556772.0,
235725.0
],
[
0.0,
4306.0,
257258.0,
27076233.0,
1405095.0,
1110647.0,
287354.0
],
[
0.0,
34522.0,
157012.0,
2484523.0,
13693770.0,
485790.0,
72912.0
],
[
0.0,
99015.0,
451628.0,
1307686.0,
262292.0,
26741687.0,
490185.0
],
[
0.0,
28642.0,
157364.0,
340012.0,
59034.0,
290346.0,
789449.0
]
],
"class_id": null,
"class_name": "average"
}
]
```
</details>
### COWC Potsdam Car Object Detection
This [experiment](examples/cowc/object_detection.py) performs object detection on cars with the [Cars Overhead With Context](https://gdo152.llnl.gov/cowc/) dataset over Potsdam, Germany.
Data:
* The imagery can only be downloaded after filling in this [request form](http://www2.isprs.org/commissions/comm3/wg4/data-request-form2.html). After your request is granted, follow the link to 'POTSDAM 2D LABELING' and download and unzip `4_Ortho_RGBIR.zip` into a directory, and then upload to S3 if desired. (This step uses the same imagery as [ISPRS Potsdam Semantic Segmentation](#isprs-potsdam-semantic-segmentation))
* Download the [processed labels](https://github.com/azavea/raster-vision-data/releases/download/v0.0.1/cowc-potsdam-labels.zip) and unzip. These files were generated from the [COWC car detection dataset](https://gdo152.llnl.gov/cowc/) using scripts in [cowc.data](cowc/data/). TODO: make a Jupyter notebook showing how to process the raw labels from scratch.
Arguments:
* `raw_uri` should point to the imagery directory created above, and should contain the `4_Ortho_RGBIR` subdirectory.
* `processed_uri` should point to the labels directory created above. It should contain the `labels/all` subdirectory.
Below are sample predictions and eval metrics.
<details><summary>Eval Metrics</summary>
```javascript
{
"precision": 0.9390652367984924,
"recall": 0.9524752475247524,
"f1": 0.945173902480464,
"count_error": 0.015841584158415842,
"gt_count": 505.0,
"class_id": 1,
"class_name": "vehicle"
},
{
"precision": 0.9390652367984924,
"recall": 0.9524752475247524,
"f1": 0.945173902480464,
"count_error": 0.015841584158415842,
"gt_count": 505.0,
"class_id": null,
"class_name": "average"
}
```
</details>
### xView Vehicle Object Detection
This [experiment](examples/xview/object_detection.py) performs object detection to find vehicles using the [DIUx xView Detection Challenge](http://xviewdataset.org/) dataset.
Data:
* Sign up for an account for the [DIUx xView Detection Challenge](http://xviewdataset.org/). Navigate to the [downloads page](https://challenge.xviewdataset.org/download-links) and download the zipped training images and labels. Unzip both of these files and place their contents in a directory, and upload to S3 if desired.
* Run the [xview-vehicles-data-prep.ipynb](notebooks/xview/xview-vehicles-data-prep.ipynb) Jupyter notebook, pointing the `raw_uri` to the directory created above.
Arguments:
* The `raw_uri` should point to the directory created above, and contain a labels GeoJSON file named `xView_train.geojson`, and a directory named `train_images`.
* The `processed_uri` should point to the processed data generated by the notebook.
Below are sample predictions and eval metrics.
<details><summary>Eval Metrics</summary>
```javascript
{
"class_name": "vehicle",
"precision": 0.4789625193065175,
"class_id": 1,
"f1": 0.4036499117825103,
"recall": 0.3597840599059615,
"count_error": -0.2613920009287745,
"gt_count": 17227
},
{
"class_name": "average",
"precision": 0.4789625193065175,
"class_id": null,
"f1": 0.4036499117825103,
"recall": 0.3597840599059615,
"count_error": -0.2613920009287745,
"gt_count": 17227
}
```
</details>
## Model Zoo
Using the Model Zoo, you can download prediction packages which contain pre-trained models and configuration, and then run them on sample test images that the model wasn't trained on.
```shell
> rastervision predict <predict_package> <infile> <outfile>
```
Note that the input file is assumed to have the same channel order and statistics as the images the model was trained on. See `rastervision predict --help` to see options for manually overriding these. It shouldn't take more than a minute on a CPU to make predictions for each sample. For some of the examples, there are also model files that can be used for fine-tuning on another dataset.
**Disclaimer**: These models are provided for testing and demonstration purposes and aren't particularly accurate. As is usually the case for deep learning models, the accuracy drops greatly when used on input that is outside the training distribution. In other words, a model trained in one city probably won't work well in another city (unless they are very similar) or at a different imagery resolution.
### PyTorch Models
For the PyTorch models, the prediction package (when unzipped) contains a `model` file which can be used for fine-tuning.
| Dataset | Task | Model | Prediction Package | Sample Image |
| --- | --- | --- | --- | --- |
| SpaceNet Rio Buildings | Chip Classification | Resnet50 | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/rio-cc-pytorch/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/rio-cc/013022223130_sample.tif) |
| SpaceNet Vegas Buildings | Semantic Segmentation | DeeplabV3/Resnet50 | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-building-seg-pytorch/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-building-seg/1929.tif) |
| SpaceNet Vegas Roads | Semantic Segmentation | DeeplabV3/Resnet50 | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-road-seg-pytorch/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-road-seg/524.tif) |
| ISPRS Potsdam | Semantic Segmentation | DeeplabV3/Resnet50 | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/potsdam-seg-pytorch/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/potsdam-seg/3_12_sample.tif) |
| COWC Potsdam (Cars) | Object Detection | Faster-RCNN/Resnet18 | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/cowc-od-pytorch/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/cowc-od/3_10_sample.tif) |
### Tensorflow Models
| Dataset | Task | Model | Prediction Package | Sample Image | Model (for fine-tuning) |
| --- | --- | --- | --- | --- | --- |
| SpaceNet Rio Buildings | Chip Classification | Resnet50 | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/rio-cc/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/rio-cc/013022223130_sample.tif) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/rio-cc/model-weights.hdf5) |
| SpaceNet Vegas Buildings | Semantic Segmentation | Mobilenet | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-building-seg/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-building-seg/1929.tif) | n/a |
| SpaceNet Vegas Roads | Semantic Segmentation | Mobilenet | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-road-seg/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-road-seg/524.tif) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/vegas-road-seg/roads-mobilenet.tar.gz) |
| ISPRS Potsdam | Semantic Segmentation | Mobilenet | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/potsdam-seg/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/potsdam-seg/3_12_sample.tif) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/potsdam-seg/model.tar.gz) |
| COWC Potsdam (Cars) | Object Detection | Mobilenet | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/cowc-od/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/cowc-od/3_10_sample.tif) | n/a |
| xView Vehicle | Object Detection | Mobilenet | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/xview-vehicle-od/predict_package.zip) | [link](https://s3.amazonaws.com/azavea-research-public-data/raster-vision/examples/model-zoo/xview-vehicle-od/1677.tif) | n/a |
================================================
FILE: docker/build
================================================
#!/bin/bash
set -e
if [[ -n "${RASTER_VISION_DEBUG}" ]]; then
set -x
fi
function usage() {
echo -n \
"Usage: $(basename "$0") [--cpu|--gpu]
Build Docker images.
If the type is not specified, it will build both cpu and gpu Docker images.
"
}
if [ "${BASH_SOURCE[0]}" = "${0}" ]
then
if [ "${1:-}" = "--help" ]
then
usage
exit
fi
if [ $# -eq 0 -o "${1:-}" = "--tf" ]
then
if [ -z "$RASTER_VISION_REPO" ]
then
BASE_IMAGE="quay.io/azavea/raster-vision:tf-cpu-0.11"
else
BASE_IMAGE="raster-vision-tf-cpu"
fi
docker build --build-arg BASE_IMAGE="$BASE_IMAGE" -t raster-vision-examples-tf-cpu -f Dockerfile .
if [ -z "$RASTER_VISION_REPO" ]
then
BASE_IMAGE="quay.io/azavea/raster-vision:tf-gpu-0.11"
else
BASE_IMAGE="raster-vision-tf-gpu"
fi
docker build --build-arg BASE_IMAGE="$BASE_IMAGE" -t raster-vision-examples-tf-gpu -f Dockerfile .
fi
if [ $# -eq 0 -o "${1:-}" = "--pytorch" ]
then
if [ -z "$RASTER_VISION_REPO" ]
then
BASE_IMAGE="quay.io/azavea/raster-vision:pytorch-0.11"
else
BASE_IMAGE="raster-vision-pytorch"
fi
docker build --build-arg BASE_IMAGE="$BASE_IMAGE" -t raster-vision-examples-pytorch -f Dockerfile .
fi
fi
================================================
FILE: docker/run
================================================
#!/bin/bash
# This is the same as the main repo `run` script except that it:
# * mounts the examples directory
# * uses the -examples Docker images
# * mounts the main RV repo only if it's pointed to by RASTER_VISION_REPO
# * removes the --docs option
# * uses $REPO_ROOT/notebooks as the notebooks dir
set -e
if [[ -n "${RASTER_VISION_DEBUG}" ]]; then
set -x
fi
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done
DOCKER_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
REPO_ROOT="$( cd -P "$( dirname "$DOCKER_DIR" )" && pwd )"
function usage() {
echo -n \
"Usage: $(basename "$0") <options> <command>
Run a console in the raster-vision-examples-cpu Docker image locally.
Environment variables:
RASTER_VISION_DATA_DIR (directory for storing data; mounted to /opt/data)
AWS_PROFILE (optional AWS profile)
RASTER_VISION_REPO (optional path to main RV repo; mounted to /opt/src)
Options:
--aws forwards AWS credentials (sets AWS_PROFILE env var and mounts ~/.aws to /root/.aws)
--tensorboard maps port 6006
--gpu use the NVIDIA runtime and GPU image
--name sets the name of the running container
--jupyter forwards port 8888, mounts ./notebooks to /opt/notebooks, and runs Jupyter
--debug maps port 3007 on localhost to 3000 inside container
--tf-gpu use raster-vision-examples-tf-gpu image and nvidia runtime
--tf-cpu use raster-vision-examples-tf-cpu image
--pytorch-gpu use raster-vision-examples-pytorch image and nvidia runtime
Note: raster-vision-examples-pytorch image is used by default
All arguments after above options are passed to 'docker run'.
"
}
IMAGE="raster-vision-pytorch"
RASTER_VISION_DATA_DIR="${RASTER_VISION_DATA_DIR:-${REPO_ROOT}/data}"
# Parse options using scheme in
# https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
--help)
usage
exit 0
shift
;;
--aws)
AWS="-e AWS_PROFILE=${AWS_PROFILE:-default} -v ${HOME}/.aws:/root/.aws:ro"
shift # past argument
;;
--tensorboard)
TENSORBOARD="-p 6006:6006"
shift # past argument
;;
--tf-gpu)
IMAGE="raster-vision-examples-tf-gpu"
RUNTIME="--runtime=nvidia"
shift # past argument
;;
--tf-cpu)
IMAGE="raster-vision-examples-tf-cpu"
shift # past argument
;;
--pytorch-gpu)
IMAGE="raster-vision-examples-pytorch"
RUNTIME="--runtime=nvidia"
shift # past argument
;;
--name)
shift
NAME="--name $1"
shift
;;
--jupyter)
JUPYTER="-v ${REPO_ROOT}/notebooks:/opt/notebooks -p 8888:8888"
CMD=(jupyter notebook --ip 0.0.0.0 --port 8888 --no-browser --allow-root --notebook-dir=/opt/notebooks)
shift
;;
--debug)
DEBUG="-p 3007:3000"
shift
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters
if [ -z "${CMD}" ]
then
CMD=(${@:1})
fi
if [ -z "$RASTER_VISION_REPO" ]
then
RV_SRC_FLAGS=""
else
RV_SRC_FLAGS="-v ${RASTER_VISION_REPO}/rastervision:/opt/src/rastervision"
fi
if [ "${BASH_SOURCE[0]}" = "${0}" ]
then
docker run ${RUNTIME} ${NAME} --rm -it \
-v "${HOME}"/.rastervision:/root/.rastervision \
-v "${REPO_ROOT}"/examples:/opt/src/examples \
-v ${RASTER_VISION_DATA_DIR}:/opt/data \
${TENSORBOARD} ${AWS} ${JUPYTER} ${DOCS} ${DEBUG} \
${RV_SRC_FLAGS} \
${IMAGE} "${CMD[@]}"
fi
================================================
FILE: examples/__init__.py
================================================
================================================
FILE: examples/cowc/__init__.py
================================================
================================================
FILE: examples/cowc/object_detection.py
================================================
import os
from os.path import join
import rastervision as rv
from examples.utils import str_to_bool, save_image_crop
class CowcObjectDetectionExperiments(rv.ExperimentSet):
def exp_main(self, raw_uri, processed_uri, root_uri, test=False, use_tf=False):
"""Object detection on COWC (Cars Overhead with Context) Potsdam dataset
Args:
raw_uri: (str) directory of raw data
processed_uri: (str) directory of processed data
root_uri: (str) root directory for experiment output
test: (bool) if True, run a very small experiment as a test and generate
debug output
use_tf: (bool) if True, use Tensorflow-based backend
"""
test = str_to_bool(test)
exp_id = 'cowc-object-detection'
num_steps = 100000
batch_size = 8
debug = False
train_scene_ids = ['2_10', '2_11', '2_12', '2_14', '3_11',
'3_13', '4_10', '5_10', '6_7', '6_9']
val_scene_ids = ['2_13', '6_8', '3_10']
if test:
exp_id += '-test'
num_steps = 1
batch_size = 2
debug = True
train_scene_ids = train_scene_ids[0:1]
val_scene_ids = val_scene_ids[0:1]
task = rv.TaskConfig.builder(rv.OBJECT_DETECTION) \
.with_chip_size(300) \
.with_classes({'vehicle': (1, 'red')}) \
.with_chip_options(neg_ratio=5.0,
ioa_thresh=0.9) \
.with_predict_options(merge_thresh=0.5,
score_thresh=0.9) \
.build()
if use_tf:
backend = rv.BackendConfig.builder(rv.TF_OBJECT_DETECTION) \
.with_task(task) \
.with_model_defaults(rv.SSD_MOBILENET_V1_COCO) \
.with_debug(debug) \
.with_batch_size(batch_size) \
.with_num_steps(num_steps) \
.build()
else:
batch_size = 16
num_epochs = 10
if test:
batch_size = 2
num_epochs = 2
backend = rv.BackendConfig.builder(rv.PYTORCH_OBJECT_DETECTION) \
.with_task(task) \
.with_train_options(
lr=1e-4,
one_cycle=True,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet18',
debug=debug) \
.build()
def make_scene(id):
raster_uri = join(
raw_uri, '4_Ortho_RGBIR/top_potsdam_{}_RGBIR.tif'.format(id))
label_uri = join(
processed_uri, 'labels', 'all', 'top_potsdam_{}_RGBIR.json'.format(id))
if test:
crop_uri = join(
processed_uri, 'crops', os.path.basename(raster_uri))
save_image_crop(raster_uri, crop_uri, label_uri=label_uri,
size=1000, min_features=5)
raster_uri = crop_uri
return rv.SceneConfig.builder() \
.with_id(id) \
.with_task(task) \
.with_raster_source(raster_uri, channel_order=[0, 1, 2]) \
.with_label_source(label_uri) \
.build()
train_scenes = [make_scene(id) for id in train_scene_ids]
val_scenes = [make_scene(id) for id in val_scene_ids]
dataset = rv.DatasetConfig.builder() \
.with_train_scenes(train_scenes) \
.with_validation_scenes(val_scenes) \
.build()
experiment = rv.ExperimentConfig.builder() \
.with_id(exp_id) \
.with_root_uri(root_uri) \
.with_task(task) \
.with_backend(backend) \
.with_dataset(dataset) \
.build()
return experiment
if __name__ == '__main__':
rv.main()
================================================
FILE: examples/potsdam/__init__.py
================================================
================================================
FILE: examples/potsdam/semantic_segmentation.py
================================================
import os
from os.path import join
import rastervision as rv
from examples.utils import str_to_bool, save_image_crop
class PotsdamSemanticSegmentation(rv.ExperimentSet):
def exp_main(self, raw_uri, processed_uri, root_uri, test=False, use_tf=False):
"""Run an experiment on the ISPRS Potsdam dataset.
Uses Tensorflow Deeplab backend with Mobilenet architecture. Should get to
F1 score of ~0.86 (including clutter class) after 6 hours of training on a P3
instance.
Args:
raw_uri: (str) directory of raw data
root_uri: (str) root directory for experiment output
test: (bool) if True, run a very small experiment as a test and generate
debug output
use_tf: (bool) if True, use Tensorflow Deeplab backend.
"""
test = str_to_bool(test)
use_tf = str_to_bool(use_tf)
exp_id = 'potsdam-seg'
train_ids = ['2-10', '2-11', '3-10', '3-11', '4-10', '4-11', '4-12', '5-10',
'5-11', '5-12', '6-10', '6-11', '6-7', '6-9', '7-10', '7-11',
'7-12', '7-7', '7-8', '7-9']
val_ids = ['2-12', '3-12', '6-12']
# infrared, red, green
channel_order = [3, 0, 1]
debug = False
if test:
debug = True
train_ids = train_ids[0:1]
val_ids = val_ids[0:1]
exp_id += '-test'
classes = {
'Car': (1, '#ffff00'),
'Building': (2, '#0000ff'),
'Low Vegetation': (3, '#00ffff'),
'Tree': (4, '#00ff00'),
'Impervious': (5, "#ffffff"),
'Clutter': (6, "#ff0000")
}
task = rv.TaskConfig.builder(rv.SEMANTIC_SEGMENTATION) \
.with_chip_size(300) \
.with_classes(classes) \
.with_chip_options(window_method='sliding',
stride=300, debug_chip_probability=1.0) \
.build()
if use_tf:
batch_size = 8
num_steps = 100000
if test:
num_steps = 1
batch_size = 2
model_type = rv.MOBILENET_V2
backend = rv.BackendConfig.builder(rv.TF_DEEPLAB) \
.with_task(task) \
.with_model_defaults(model_type) \
.with_train_options(sync_interval=600) \
.with_num_steps(num_steps) \
.with_batch_size(batch_size) \
.with_debug(debug) \
.build()
else:
batch_size = 8
num_epochs = 10
if test:
batch_size = 2
num_epochs = 1
backend = rv.BackendConfig.builder(rv.PYTORCH_SEMANTIC_SEGMENTATION) \
.with_task(task) \
.with_train_options(
lr=1e-4,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet50',
debug=debug) \
.build()
def make_scene(id):
id = id.replace('-', '_')
raster_uri = '{}/4_Ortho_RGBIR/top_potsdam_{}_RGBIR.tif'.format(
raw_uri, id)
label_uri = '{}/5_Labels_for_participants/top_potsdam_{}_label.tif'.format(
raw_uri, id)
if test:
crop_uri = join(
processed_uri, 'crops', os.path.basename(raster_uri))
save_image_crop(raster_uri, crop_uri, size=600)
raster_uri = crop_uri
# Using with_rgb_class_map because label TIFFs have classes encoded as RGB colors.
label_source = rv.LabelSourceConfig.builder(rv.SEMANTIC_SEGMENTATION) \
.with_rgb_class_map(task.class_map) \
.with_raster_source(label_uri) \
.build()
# URI will be injected by scene config.
# Using with_rgb(True) because we want prediction TIFFs to be in RGB format.
label_store = rv.LabelStoreConfig.builder(rv.SEMANTIC_SEGMENTATION_RASTER) \
.with_rgb(True) \
.build()
scene = rv.SceneConfig.builder() \
.with_task(task) \
.with_id(id) \
.with_raster_source(raster_uri,
channel_order=channel_order) \
.with_label_source(label_source) \
.with_label_store(label_store) \
.build()
return scene
train_scenes = [make_scene(id) for id in train_ids]
val_scenes = [make_scene(id) for id in val_ids]
dataset = rv.DatasetConfig.builder() \
.with_train_scenes(train_scenes) \
.with_validation_scenes(val_scenes) \
.build()
experiment = rv.ExperimentConfig.builder() \
.with_id(exp_id) \
.with_task(task) \
.with_backend(backend) \
.with_dataset(dataset) \
.with_root_uri(root_uri) \
.build()
return experiment
if __name__ == '__main__':
rv.main()
================================================
FILE: examples/spacenet/__init__.py
================================================
================================================
FILE: examples/spacenet/rio/__init__.py
================================================
================================================
FILE: examples/spacenet/rio/chip_classification.py
================================================
import os
from os.path import join
import rastervision as rv
from examples.utils import get_scene_info, str_to_bool, save_image_crop
aoi_path = 'AOIs/AOI_1_Rio/srcData/buildingLabels/Rio_OUTLINE_Public_AOI.geojson'
class ChipClassificationExperiments(rv.ExperimentSet):
def exp_main(self, raw_uri, processed_uri, root_uri, test=False, use_tf=False):
"""Chip classification experiment on Spacenet Rio dataset.
Run the data prep notebook before running this experiment. Note all URIs can be
local or remote.
Args:
raw_uri: (str) directory of raw data
processed_uri: (str) directory of processed data
root_uri: (str) root directory for experiment output
test: (bool) if True, run a very small experiment as a test and generate
debug output
use_tf: (bool) if True, use Tensorflow Deeplab backend
"""
test = str_to_bool(test)
use_tf = str_to_bool(use_tf)
exp_id = 'spacenet-rio-chip-classification'
debug = False
train_scene_info = get_scene_info(join(processed_uri, 'train-scenes.csv'))
val_scene_info = get_scene_info(join(processed_uri, 'val-scenes.csv'))
if test:
exp_id += '-test'
debug = True
train_scene_info = train_scene_info[0:1]
val_scene_info = val_scene_info[0:1]
task = rv.TaskConfig.builder(rv.CHIP_CLASSIFICATION) \
.with_chip_size(200) \
.with_classes({
'building': (1, 'red'),
'no_building': (2, 'black')
}) \
.build()
if use_tf:
num_epochs = 20
batch_size = 32
if test:
num_epochs = 1
batch_size = 1
backend = rv.BackendConfig.builder(rv.KERAS_CLASSIFICATION) \
.with_task(task) \
.with_model_defaults(rv.RESNET50_IMAGENET) \
.with_debug(debug) \
.with_batch_size(batch_size) \
.with_num_epochs(num_epochs) \
.with_config({
'trainer': {
'options': {
'saveBest': True,
'lrSchedule': [
{
'epoch': 0,
'lr': 0.0005
},
{
'epoch': 10,
'lr': 0.0001
},
{
'epoch': 15,
'lr': 0.00001
}
]
}
}
}, set_missing_keys=True) \
.build()
else:
num_epochs = 20
batch_size = 32
if test:
num_epochs = 1
batch_size = 2
backend = rv.BackendConfig.builder(rv.PYTORCH_CHIP_CLASSIFICATION) \
.with_task(task) \
.with_train_options(
lr=1e-4,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet50',
debug=debug) \
.build()
def make_scene(scene_info):
(raster_uri, label_uri) = scene_info
raster_uri = join(raw_uri, raster_uri)
label_uri = join(processed_uri, label_uri)
aoi_uri = join(raw_uri, aoi_path)
if test:
crop_uri = join(
processed_uri, 'crops', os.path.basename(raster_uri))
save_image_crop(raster_uri, crop_uri, label_uri=label_uri,
size=600, min_features=20)
raster_uri = crop_uri
id = os.path.splitext(os.path.basename(raster_uri))[0]
label_source = rv.LabelSourceConfig.builder(rv.CHIP_CLASSIFICATION) \
.with_uri(label_uri) \
.with_ioa_thresh(0.5) \
.with_use_intersection_over_cell(False) \
.with_pick_min_class_id(True) \
.with_background_class_id(2) \
.with_infer_cells(True) \
.build()
return rv.SceneConfig.builder() \
.with_task(task) \
.with_id(id) \
.with_raster_source(raster_uri) \
.with_label_source(label_source) \
.with_aoi_uri(aoi_uri) \
.build()
train_scenes = [make_scene(info) for info in train_scene_info]
val_scenes = [make_scene(info) for info in val_scene_info]
dataset = rv.DatasetConfig.builder() \
.with_train_scenes(train_scenes) \
.with_validation_scenes(val_scenes) \
.build()
experiment = rv.ExperimentConfig.builder() \
.with_id(exp_id) \
.with_root_uri(root_uri) \
.with_task(task) \
.with_backend(backend) \
.with_dataset(dataset) \
.build()
return experiment
if __name__ == '__main__':
rv.main()
================================================
FILE: examples/spacenet/rio/semantic_segmentation.py
================================================
import os
from os.path import join
import rastervision as rv
from examples.utils import get_scene_info, str_to_bool, save_image_crop
aoi_path = 'AOIs/AOI_1_Rio/srcData/buildingLabels/Rio_OUTLINE_Public_AOI.geojson'
class SemanticSegmentationExperiments(rv.ExperimentSet):
def exp_main(self, raw_uri, processed_uri, root_uri, test=False):
"""Semantic segmentation experiment on Spacenet Rio dataset.
Run the data prep notebook before running this experiment. Note all URIs can be
local or remote.
Args:
raw_uri: (str) directory of raw data
processed_uri: (str) directory of processed data
root_uri: (str) root directory for experiment output
test: (bool) if True, run a very small experiment as a test and generate
debug output
"""
test = str_to_bool(test)
exp_id = 'spacenet-rio-semseg'
debug = False
batch_size = 8
num_epochs = 20
train_scene_info = get_scene_info(join(processed_uri, 'train-scenes.csv'))
val_scene_info = get_scene_info(join(processed_uri, 'val-scenes.csv'))
if test:
exp_id += '-test'
debug = True
num_epochs = 1
batch_size = 2
train_scene_info = train_scene_info[0:1]
val_scene_info = val_scene_info[0:1]
class_map = {
'Building': (1, 'orange'),
'Background': (2, 'black')
}
task = rv.TaskConfig.builder(rv.SEMANTIC_SEGMENTATION) \
.with_chip_size(300) \
.with_classes(class_map) \
.with_chip_options(
stride=300,
window_method='sliding',
debug_chip_probability=1.0) \
.build()
backend = rv.BackendConfig.builder(rv.PYTORCH_SEMANTIC_SEGMENTATION) \
.with_task(task) \
.with_train_options(
lr=1e-4,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet50',
debug=debug) \
.build()
def make_scene(scene_info):
(raster_uri, label_uri) = scene_info
raster_uri = join(raw_uri, raster_uri)
label_uri = join(processed_uri, label_uri)
if test:
crop_uri = join(
processed_uri, 'crops', os.path.basename(raster_uri))
save_image_crop(raster_uri, crop_uri, label_uri=label_uri,
size=600)
raster_uri = crop_uri
aoi_uri = join(raw_uri, aoi_path)
id = os.path.splitext(os.path.basename(raster_uri))[0]
background_class_id = 2
label_raster_source = rv.RasterSourceConfig.builder(rv.RASTERIZED_SOURCE) \
.with_vector_source(label_uri) \
.with_rasterizer_options(background_class_id) \
.build()
label_source = rv.LabelSourceConfig.builder(rv.SEMANTIC_SEGMENTATION) \
.with_raster_source(label_raster_source) \
.build()
return rv.SceneConfig.builder() \
.with_task(task) \
.with_id(id) \
.with_raster_source(raster_uri) \
.with_label_source(label_source) \
.with_aoi_uri(aoi_uri) \
.build()
train_scenes = [make_scene(info) for info in train_scene_info]
val_scenes = [make_scene(info) for info in val_scene_info]
dataset = rv.DatasetConfig.builder() \
.with_train_scenes(train_scenes) \
.with_validation_scenes(val_scenes) \
.build()
experiment = rv.ExperimentConfig.builder() \
.with_id(exp_id) \
.with_root_uri(root_uri) \
.with_task(task) \
.with_backend(backend) \
.with_dataset(dataset) \
.build()
return experiment
if __name__ == '__main__':
rv.main()
================================================
FILE: examples/spacenet/vegas/__init__.py
================================================
================================================
FILE: examples/spacenet/vegas/all.py
================================================
import re
import random
import os
from abc import abstractmethod
import rastervision as rv
from rastervision.utils.files import list_paths
from examples.utils import str_to_bool
BUILDINGS = 'buildings'
ROADS = 'roads'
class SpacenetConfig(object):
def __init__(self, raw_uri):
self.raw_uri = raw_uri
@staticmethod
def create(raw_uri, target):
if target.lower() == BUILDINGS:
return VegasBuildings(raw_uri)
elif target.lower() == ROADS:
return VegasRoads(raw_uri)
else:
raise ValueError('{} is not a valid target.'.format(target))
def get_raster_source_uri(self, id):
return os.path.join(
self.raw_uri, self.base_dir, self.raster_dir,
'{}{}.tif'.format(self.raster_fn_prefix, id))
def get_geojson_uri(self, id):
return os.path.join(
self.raw_uri, self.base_dir, self.label_dir,
'{}{}.geojson'.format(self.label_fn_prefix, id))
def get_scene_ids(self):
label_dir = os.path.join(self.raw_uri, self.base_dir, self.label_dir)
label_paths = list_paths(label_dir, ext='.geojson')
label_re = re.compile(r'.*{}(\d+)\.geojson'.format(self.label_fn_prefix))
scene_ids = [
label_re.match(label_path).group(1)
for label_path in label_paths]
return scene_ids
@abstractmethod
def get_class_map(self):
pass
@abstractmethod
def get_class_id_to_filter(self):
pass
class VegasRoads(SpacenetConfig):
def __init__(self, raw_uri):
self.base_dir = 'spacenet/SN3_roads/train/AOI_2_Vegas/'
self.raster_dir = 'PS-RGB/'
self.label_dir = 'geojson_roads/'
self.raster_fn_prefix = 'SN3_roads_train_AOI_2_Vegas_PS-RGB_img'
self.label_fn_prefix = 'SN3_roads_train_AOI_2_Vegas_geojson_roads_img'
super().__init__(raw_uri)
def get_class_map(self):
# First class should be background when using GeoJSONRasterSource
return {
'Road': (1, 'orange'),
'Background': (2, 'black')
}
def get_class_id_to_filter(self):
return {1: ['has', 'highway']}
class VegasBuildings(SpacenetConfig):
def __init__(self, raw_uri):
self.base_dir = 'spacenet/SN2_buildings/train/AOI_2_Vegas'
self.raster_dir = 'PS-RGB'
self.label_dir = 'geojson_buildings'
self.raster_fn_prefix = 'SN2_buildings_train_AOI_2_Vegas_PS-RGB_img'
self.label_fn_prefix = 'SN2_buildings_train_AOI_2_Vegas_geojson_buildings_img'
super().__init__(raw_uri)
def get_class_map(self):
# First class should be background when using GeoJSONRasterSource
return {
'Building': (1, 'orange'),
'Background': (2, 'black')
}
def get_class_id_to_filter(self):
return {1: ['has', 'building']}
def build_scene(task, spacenet_config, id, channel_order=None, vector_tile_options=None):
# Need to use stats_transformer because imagery is uint16.
raster_source = rv.RasterSourceConfig.builder(rv.RASTERIO_SOURCE) \
.with_uri(spacenet_config.get_raster_source_uri(id)) \
.with_channel_order(channel_order) \
.with_stats_transformer() \
.build()
label_store = None
# Set a line buffer to convert line strings to polygons.
if vector_tile_options is None:
label_uri = spacenet_config.get_geojson_uri(id)
vector_source = rv.VectorSourceConfig.builder(rv.GEOJSON_SOURCE) \
.with_uri(label_uri) \
.with_buffers(line_bufs={1: 15}) \
.build()
else:
options = vector_tile_options
class_id_to_filter = spacenet_config.get_class_id_to_filter()
vector_source = rv.VectorSourceConfig.builder(rv.VECTOR_TILE_SOURCE) \
.with_class_inference(class_id_to_filter=class_id_to_filter,
default_class_id=None) \
.with_uri(options.uri) \
.with_zoom(options.zoom) \
.with_id_field(options.id_field) \
.with_buffers(line_bufs={1: 15}) \
.build()
if task.task_type == rv.SEMANTIC_SEGMENTATION:
background_class_id = 2
label_raster_source = rv.RasterSourceConfig.builder(rv.RASTERIZED_SOURCE) \
.with_vector_source(vector_source) \
.with_rasterizer_options(background_class_id) \
.build()
label_source = rv.LabelSourceConfig.builder(rv.SEMANTIC_SEGMENTATION) \
.with_raster_source(label_raster_source) \
.build()
# Generate polygon output for segmented buildings.
if isinstance(spacenet_config, VegasBuildings):
vector_output = {'mode': 'polygons', 'class_id': 1, 'denoise': 3}
label_store = rv.LabelStoreConfig.builder(rv.SEMANTIC_SEGMENTATION_RASTER) \
.with_vector_output([vector_output]) \
.build()
elif task.task_type == rv.CHIP_CLASSIFICATION:
label_source = rv.LabelSourceConfig.builder(rv.CHIP_CLASSIFICATION) \
.with_vector_source(vector_source) \
.with_ioa_thresh(0.01) \
.with_use_intersection_over_cell(True) \
.with_pick_min_class_id(True) \
.with_background_class_id(2) \
.with_infer_cells(True) \
.build()
elif task.task_type == rv.OBJECT_DETECTION:
label_source = rv.LabelSourceConfig.builder(rv.OBJECT_DETECTION) \
.with_vector_source(vector_source) \
.build()
scene = rv.SceneConfig.builder() \
.with_task(task) \
.with_id(id) \
.with_raster_source(raster_source) \
.with_label_source(label_source) \
.with_label_store(label_store) \
.build()
return scene
def build_dataset(task, spacenet_config, test, vector_tile_options=None):
scene_ids = spacenet_config.get_scene_ids()
if len(scene_ids) == 0:
raise ValueError('No scenes found. Something is configured incorrectly.')
random.seed(5678)
scene_ids = sorted(scene_ids)
random.shuffle(scene_ids)
# Workaround to handle scene 1000 missing on S3.
if '1000' in scene_ids:
scene_ids.remove('1000')
split_ratio = 0.8
num_train_ids = round(len(scene_ids) * split_ratio)
train_ids = scene_ids[0:num_train_ids]
val_ids = scene_ids[num_train_ids:]
num_train_scenes = len(train_ids)
num_val_scenes = len(val_ids)
if test:
num_train_scenes = 16
num_val_scenes = 4
train_ids = train_ids[0:num_train_scenes]
val_ids = val_ids[0:num_val_scenes]
channel_order = [0, 1, 2]
train_scenes = [build_scene(task, spacenet_config, id, channel_order,
vector_tile_options=vector_tile_options)
for id in train_ids]
val_scenes = [build_scene(task, spacenet_config, id, channel_order,
vector_tile_options=vector_tile_options)
for id in val_ids]
dataset = rv.DatasetConfig.builder() \
.with_train_scenes(train_scenes) \
.with_validation_scenes(val_scenes) \
.build()
return dataset
def build_task(task_type, class_map):
if task_type == rv.SEMANTIC_SEGMENTATION:
task = rv.TaskConfig.builder(rv.SEMANTIC_SEGMENTATION) \
.with_chip_size(300) \
.with_classes(class_map) \
.with_chip_options(
chips_per_scene=9,
debug_chip_probability=0.25,
negative_survival_probability=1.0,
target_classes=[1],
target_count_threshold=1000) \
.build()
elif task_type == rv.CHIP_CLASSIFICATION:
task = rv.TaskConfig.builder(rv.CHIP_CLASSIFICATION) \
.with_chip_size(200) \
.with_classes(class_map) \
.build()
elif task_type == rv.OBJECT_DETECTION:
task = rv.TaskConfig.builder(rv.OBJECT_DETECTION) \
.with_chip_size(300) \
.with_classes(class_map) \
.with_chip_options(neg_ratio=1.0,
ioa_thresh=0.8) \
.with_predict_options(merge_thresh=0.1,
score_thresh=0.5) \
.build()
return task
def build_backend(task, test):
debug = False
if test:
debug = True
if task.task_type == rv.SEMANTIC_SEGMENTATION:
batch_size = 8
num_epochs = 2
if test:
batch_size = 2
num_epochs = 1
backend = rv.BackendConfig.builder(rv.PYTORCH_SEMANTIC_SEGMENTATION) \
.with_task(task) \
.with_train_options(
lr=1e-4,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet50',
debug=debug) \
.build()
elif task.task_type == rv.CHIP_CLASSIFICATION:
num_epochs = 2
batch_size = 32
if test:
num_epochs = 1
batch_size = 2
backend = rv.BackendConfig.builder(rv.PYTORCH_CHIP_CLASSIFICATION) \
.with_task(task) \
.with_train_options(
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet18',
debug=debug) \
.build()
elif task.task_type == rv.OBJECT_DETECTION:
batch_size = 16
num_epochs = 2
if test:
batch_size = 1
num_epochs = 2
backend = rv.BackendConfig.builder(rv.PYTORCH_OBJECT_DETECTION) \
.with_task(task) \
.with_train_options(
lr=1e-4,
one_cycle=True,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet18',
debug=debug) \
.build()
return backend
def str_to_bool(x):
if type(x) == str:
if x.lower() == 'true':
return True
elif x.lower() == 'false':
return False
else:
raise ValueError('{} is expected to be true or false'.format(x))
return x
def validate_options(task_type, target, vector_tile_options=None):
if task_type not in [rv.SEMANTIC_SEGMENTATION, rv.CHIP_CLASSIFICATION,
rv.OBJECT_DETECTION]:
raise ValueError('{} is not a valid task_type'.format(task_type))
if target not in [ROADS, BUILDINGS]:
raise ValueError('{} is not a valid target'.format(target))
if target == ROADS:
if task_type in [rv.OBJECT_DETECTION]:
raise ValueError('{} is not valid task_type for target="roads"'.format(
task_type))
if vector_tile_options is not None:
if len(vector_tile_options.split(',')) != 3:
raise ValueError(
'vector_tile_options needs to have 3 comma-delimited values')
class VectorTileOptions():
def __init__(self, uri, zoom, id_field):
self.uri = uri
self.zoom = int(zoom)
self.id_field = id_field
@staticmethod
def build(config_str):
if config_str is None:
return None
else:
uri, zoom, id_field = config_str.split(',')
return VectorTileOptions(uri, zoom, id_field)
class SpacenetVegas(rv.ExperimentSet):
def exp_main(self, raw_uri, root_uri, test=False,
target=BUILDINGS, task_type=rv.SEMANTIC_SEGMENTATION,
vector_tile_options=None):
"""Run an experiment on the Spacenet Vegas road or building dataset.
This is an example of how to do all three tasks on the same dataset.
Args:
raw_uri: (str) directory of raw data (the root of the Spacenet dataset)
root_uri: (str) root directory for experiment output
test: (bool) if True, run a very small experiment as a test and generate
debug output
target: (str) 'buildings' or 'roads'
task_type: (str) 'semantic_segmentation', 'object_detection', or
'chip_classification'
vector_tile_options: (str or None) space delimited list of uri, zoom, and
id_field. See VectorTileVectorSourceConfigBuilder.with_uri, .with_zoom
and .with_id_field methods for more details.
"""
test = str_to_bool(test)
exp_id = '{}-{}'.format(target, task_type.lower())
task_type = task_type.upper()
spacenet_config = SpacenetConfig.create(raw_uri, target)
validate_options(task_type, target, vector_tile_options)
vector_tile_options = VectorTileOptions.build(vector_tile_options)
task = build_task(task_type, spacenet_config.get_class_map())
backend = build_backend(task, test)
analyzer = rv.AnalyzerConfig.builder(rv.STATS_ANALYZER) \
.build()
dataset = build_dataset(task, spacenet_config, test,
vector_tile_options=vector_tile_options)
# Need to use stats_analyzer because imagery is uint16.
experiment = rv.ExperimentConfig.builder() \
.with_id(exp_id) \
.with_task(task) \
.with_backend(backend) \
.with_analyzer(analyzer) \
.with_dataset(dataset) \
.with_root_uri(root_uri) \
.build()
return experiment
if __name__ == '__main__':
rv.main()
================================================
FILE: examples/spacenet/vegas/hyperparameters.py
================================================
import os
import rastervision as rv
from examples.spacenet.vegas.all import (
SpacenetConfig, build_dataset, build_task, validate_options, BUILDINGS)
from examples.utils import str_to_bool
def build_backend(task, test, learning_rate):
debug = False
batch_size = 8
num_epochs = 10
if test:
debug = True
batch_size = 2
num_epochs = 1
backend = rv.BackendConfig.builder(rv.PYTORCH_SEMANTIC_SEGMENTATION) \
.with_task(task) \
.with_train_options(
lr=learning_rate,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet50',
debug=debug) \
.build()
return backend
class HyperParameterSearch(rv.ExperimentSet):
def exp_main(self, raw_uri, root_uri, test=False, learning_rates='0.001'):
"""Run a hyper-parameter search experiment on Spacenet Vegas.
Generates an experiment for each learning rate using a TF Deeplab semantic
segmentation backend on the Spacenet Vegas Buildings dataset.
Args:
raw_uri: (str) directory of raw data (the root of the Spacenet dataset)
root_uri: (str) root directory for experiment output
test: (bool) if True, run a very small experiment as a test and
generate debug output
learning_rates: (str) comma-delimited list of learning rates to use
"""
test = str_to_bool(test)
target = BUILDINGS
task_type = rv.SEMANTIC_SEGMENTATION
learning_rates = learning_rates.split(',')
task_type = task_type.upper()
spacenet_config = SpacenetConfig.create(raw_uri, target)
ac_key = '{}_{}'.format(target, task_type.lower())
validate_options(task_type, target)
task = build_task(task_type, spacenet_config.get_class_map())
analyzer = rv.AnalyzerConfig.builder(rv.STATS_ANALYZER) \
.build()
dataset = build_dataset(task, spacenet_config, test)
# Reduce number of scenes
dataset.train_scenes = dataset.train_scenes[0:2**7]
exps = []
for learning_rate in learning_rates:
backend = build_backend(task, test, learning_rate)
exp_id = '{}_{}_rate={}'.format(target, task_type.lower(),
learning_rate)
# Need to use stats_analyzer because imagery is uint16.
# Set the analyze and chip key to share analyze and chip output
# between the experiments.
experiment = rv.ExperimentConfig.builder() \
.with_id(exp_id) \
.with_task(task) \
.with_backend(backend) \
.with_analyzer(analyzer) \
.with_dataset(dataset) \
.with_root_uri(root_uri) \
.with_analyze_key(ac_key) \
.with_chip_key(ac_key)
exps.append(experiment.build())
return exps
if __name__ == '__main__':
rv.main()
================================================
FILE: examples/spacenet/vegas/simple_segmentation.py
================================================
import re
import random
import os
from os.path import join
import rastervision as rv
from rastervision.utils.files import list_paths
from examples.utils import str_to_bool
# Check out the docs for more information about the Raster Vision API:
# https://docs.rastervision.io/en/0.9/index.html#documentation
# Raster Vision ExperimentSets run similarly to the unittest.TestSuite.
# When you pass this script to the Raster Vision command line tool, it
# will reflexively run any ExperimentSet method that starts with 'exp_'.
# Return ExperimentConfigs from these methods in order to kick off their
# corresponding experiments.
class SpacenetVegasSimpleSegmentation(rv.ExperimentSet):
def exp_main(self, raw_uri, root_uri, test=False):
"""Run an experiment on the Spacenet Vegas building dataset.
This is a simple example of how to do semantic segmentation on data that
doesn't require any pre-processing or special permission to access.
Args:
raw_uri: (str) directory of raw data (the root of the Spacenet dataset)
root_uri: (str) root directory for experiment output
test: (bool) if True, run a very small experiment as a test and generate
debug output
"""
# Specify the location of the raw data
base_uri = join(
raw_uri, 'spacenet/SN2_buildings/train/AOI_2_Vegas')
# The images and labels are in two separate directories within the base_uri
raster_uri = join(base_uri, 'PS-RGB')
label_uri = join(base_uri, 'geojson_buildings')
# The tiff (raster) and geojson (label) files have have a naming convention of
# '[prefix]_[image id].geojson.' The prefix indicates the type of data and the
# image id indicates which scene each is associated with.
raster_fn_prefix = 'SN2_buildings_train_AOI_2_Vegas_PS-RGB_img'
label_fn_prefix = 'SN2_buildings_train_AOI_2_Vegas_geojson_buildings_img'
# Find all of the image ids that have associated images and labels. Collect
# these values to use as our scene ids.
label_paths = list_paths(label_uri, ext='.geojson')
label_re = re.compile(r'.*{}(\d+)\.geojson'.format(label_fn_prefix))
scene_ids = [
label_re.match(label_path).group(1)
for label_path in label_paths]
# Set some trainin parameters:
# The exp_id will be the label associated with this experiment, it will be used
# to name the experiment config json.
exp_id = 'spacenet-simple-seg'
# Number of times to go through the entire dataset during training.
num_epochs = 2
# Number of images in each batch
batch_size = 8
# Specify whether or not to make debug chips (a zipped sample of png chips
# that you can examine to help debug the chipping process)
debug = False
# This experiment includes an option to run a small test experiment before
# running the whole thing. You can set this using the 'test' parameter. If
# this parameter is set to True it will run a tiny test example with a new
# experiment id. This will be small enough to run locally. It is recommended
# to run a test example locally before submitting the whole experiment to AWs
# Batch.
test = str_to_bool(test)
if test:
exp_id += '-test'
num_epochs = 1
batch_size = 2
debug = True
scene_ids = scene_ids[0:10]
# Split the data into training and validation sets:
# Randomize the order of all scene ids
random.seed(5678)
scene_ids = sorted(scene_ids)
random.shuffle(scene_ids)
# Workaround to handle scene 1000 missing on S3.
if '1000' in scene_ids:
scene_ids.remove('1000')
# Figure out how many scenes make up 80% of the whole set
num_train_ids = round(len(scene_ids) * 0.8)
# Split the scene ids into training and validation lists
train_ids = scene_ids[0:num_train_ids]
val_ids = scene_ids[num_train_ids:]
# The TaskConfigBuilder constructs a child class of TaskConfig that
# corresponds to the type of computer vision task you are taking on.
# This experiment includes a semantic segmentation task but Raster
# Vision also has backends for object detection and chip classification.
# Before building the task config you can also set parameters using
# 'with_' methods. In the example below we set the chip size, the
# pixel class names and colors, and addiitonal chip options.
task = rv.TaskConfig.builder(rv.SEMANTIC_SEGMENTATION) \
.with_chip_size(300) \
.with_classes({
'Building': (1, 'orange'),
'Background': (2, 'black')
}) \
.with_chip_options(
chips_per_scene=9,
debug_chip_probability=0.25,
negative_survival_probability=1.0,
target_classes=[1],
target_count_threshold=1000) \
.build()
# Next we will create a backend that is built on top of a third-party
# deep learning library. In this case we will construct the
# BackendConfig for the pytorch semantic segmentation backend.
backend = rv.BackendConfig.builder(rv.PYTORCH_SEMANTIC_SEGMENTATION) \
.with_task(task) \
.with_train_options(
lr=1e-4,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet50',
debug=debug) \
.build()
# We will use this function to create a list of scenes that we will pass
# to the DataSetConfig builder.
def make_scene(id):
"""Make a SceneConfig object for each image/label pair
Args:
id (str): The id that corresponds to both the .tiff image source
and .geojson label source for a given scene
Returns:
rv.data.SceneConfig: a SceneConfig object which is composed of
images, labels and optionally AOIs
"""
# Find the uri for the image associated with this is
train_image_uri = os.path.join(raster_uri,
'{}{}.tif'.format(raster_fn_prefix, id))
# Construct a raster source from an image uri that can be handled by Rasterio.
# We also specify the order of image channels by their indices and add a
# stats transformer which normalizes pixel values into uint8.
raster_source = rv.RasterSourceConfig.builder(rv.RASTERIO_SOURCE) \
.with_uri(train_image_uri) \
.with_channel_order([0, 1, 2]) \
.with_stats_transformer() \
.build()
# Next create a label source config to pair with the raster source:
# define the geojson label source uri
vector_source = os.path.join(
label_uri, '{}{}.geojson'.format(label_fn_prefix, id))
# Since this is a semantic segmentation experiment and the labels
# are distributed in a vector-based GeoJSON format, we need to rasterize
# the labels. We create aRasterSourceConfigBuilder using
# `rv.RASTERIZED_SOURCE`
# indicating that it will come from a vector source. We then specify the uri
# of the vector source and (in the 'with_rasterizer_options' method) the id
# of the pixel class we would like to use as background.
label_raster_source = rv.RasterSourceConfig.builder(rv.RASTERIZED_SOURCE) \
.with_vector_source(vector_source) \
.with_rasterizer_options(2) \
.build()
# Create a semantic segmentation label source from rasterized source config
# that we built in the previous line.
label_source = rv.LabelSourceConfig.builder(rv.SEMANTIC_SEGMENTATION) \
.with_raster_source(label_raster_source) \
.build()
# Finally we can build a scene config object using the scene id and the
# configs we just defined
scene = rv.SceneConfig.builder() \
.with_task(task) \
.with_id(id) \
.with_raster_source(raster_source) \
.with_label_source(label_source) \
.build()
return scene
# Create lists of train and test scene configs
train_scenes = [make_scene(id) for id in train_ids]
val_scenes = [make_scene(id) for id in val_ids]
# Construct a DataSet config using the lists of train and
# validation scenes
dataset = rv.DatasetConfig.builder() \
.with_train_scenes(train_scenes) \
.with_validation_scenes(val_scenes) \
.build()
# We will need to convert this imagery from uint16 to uint8
# in order to use it. We specified that this conversion should take place
# when we built the train raster source but that process will require
# dataset-level statistics. To get these stats we need to create an
# analyzer.
analyzer = rv.AnalyzerConfig.builder(rv.STATS_ANALYZER) \
.build()
# We use the previously-constructed configs to create the constituent
# parts of the experiment. We also give the builder strings that define
# the experiment id and and root uri. The root uri indicates where all
# of the output will be written.
experiment = rv.ExperimentConfig.builder() \
.with_id(exp_id) \
.with_task(task) \
.with_backend(backend) \
.with_analyzer(analyzer) \
.with_dataset(dataset) \
.with_root_uri(root_uri) \
.build()
# Return one or more experiment configs to run the experiment(s)
return experiment
if __name__ == '__main__':
rv.main()
================================================
FILE: examples/test.py
================================================
import subprocess
from os.path import join, basename, dirname
import json
import pprint
import glob
import os
import click
from rastervision.utils.files import (
list_paths, download_or_copy, file_exists, make_dir, file_to_json)
cfg = [
{
'key': 'cowc-object-detection-tf',
'module': 'examples.cowc.object_detection',
'local': {
'raw_uri': '/opt/data/raw-data/isprs-potsdam/',
'processed_uri': '/opt/data/examples/cowc-potsdam/processed-data',
},
'remote': {
'raw_uri': 's3://raster-vision-raw-data/isprs-potsdam',
'processed_uri': 's3://raster-vision-lf-dev/examples/cowc-potsdam/processed-data',
},
'extra_args': [['use_tf', 'True']],
'rv_profile': 'tf',
},
{
'key': 'cowc-object-detection-pytorch',
'module': 'examples.cowc.object_detection',
'local': {
'raw_uri': '/opt/data/raw-data/isprs-potsdam/',
'processed_uri': '/opt/data/examples/cowc-potsdam/processed-data',
},
'remote': {
'raw_uri': 's3://raster-vision-raw-data/isprs-potsdam',
'processed_uri': 's3://raster-vision-lf-dev/examples/cowc-potsdam/processed-data',
},
'rv_profile': 'pytorch',
},
{
'key': 'potsdam-semantic-segmentation-pytorch',
'module': 'examples.potsdam.semantic_segmentation',
'local': {
'raw_uri': '/opt/data/raw-data/isprs-potsdam/',
'processed_uri': '/opt/data/examples/potsdam/processed-data',
},
'remote': {
'raw_uri': 's3://raster-vision-raw-data/isprs-potsdam',
'processed_uri': 's3://raster-vision-lf-dev/examples/potsdam/processed-data',
},
'rv_profile': 'pytorch',
},
{
'key': 'potsdam-semantic-segmentation-tf',
'module': 'examples.potsdam.semantic_segmentation',
'local': {
'raw_uri': '/opt/data/raw-data/isprs-potsdam/',
'processed_uri': '/opt/data/examples/potsdam/processed-data',
},
'remote': {
'raw_uri': 's3://raster-vision-raw-data/isprs-potsdam',
'processed_uri': 's3://raster-vision-lf-dev/examples/potsdam/processed-data',
},
'extra_args': [['use_tf', 'True']],
'rv_profile': 'tf',
},
{
'key': 'spacenet-rio-chip-classification-pytorch',
'module': 'examples.spacenet.rio.chip_classification',
'local': {
'raw_uri': '/opt/data/raw-data/spacenet-dataset',
'processed_uri': '/opt/data/examples/spacenet/rio/processed-data',
},
'remote': {
'raw_uri': 's3://spacenet-dataset/',
'processed_uri': 's3://raster-vision-lf-dev/examples/spacenet/rio/processed-data',
},
'rv_profile': 'pytorch',
},
{
'key': 'spacenet-rio-chip-classification-tf',
'module': 'examples.spacenet.rio.chip_classification',
'local': {
'raw_uri': '/opt/data/raw-data/spacenet-dataset',
'processed_uri': '/opt/data/examples/spacenet/rio/processed-data',
},
'remote': {
'raw_uri': 's3://spacenet-dataset/',
'processed_uri': 's3://raster-vision-lf-dev/examples/spacenet/rio/processed-data',
},
'extra_args': [['use_tf', 'True']],
'rv_profile': 'tf',
},
{
'key': 'spacenet-vegas-buildings-semantic-segmentation-pytorch',
'module': 'examples.spacenet.vegas.all',
'local': {
'raw_uri': '/opt/data/raw-data/spacenet-dataset',
'processed_uri': '/opt/data/examples/spacenet/vegas/processed-data',
},
'remote': {
'raw_uri': 's3://spacenet-dataset/',
'processed_uri': 's3://raster-vision-lf-dev/examples/spacenet/vegas/processed-data',
},
'extra_args': [['target', 'buildings'],
['task_type', 'semantic_segmentation']],
'rv_profile': 'pytorch',
},
{
'key': 'spacenet-vegas-roads-semantic-segmentation-pytorch',
'module': 'examples.spacenet.vegas.all',
'local': {
'raw_uri': '/opt/data/raw-data/spacenet-dataset',
'processed_uri': '/opt/data/examples/spacenet/vegas/processed-data',
},
'remote': {
'raw_uri': 's3://spacenet-dataset/',
'processed_uri': 's3://raster-vision-lf-dev/examples/spacenet/vegas/processed-data',
},
'extra_args': [['target', 'roads'],
['task_type', 'semantic_segmentation']],
'rv_profile': 'pytorch',
},
{
'key': 'xview-object-detection-tf',
'module': 'examples.xview.object_detection',
'local': {
'raw_uri': 's3://raster-vision-xview-example/raw-data',
'processed_uri': '/opt/data/examples/xview/processed-data',
},
'remote': {
'raw_uri': 's3://raster-vision-xview-example/raw-data',
'processed_uri': 's3://raster-vision-lf-dev/examples/xview/processed-data',
},
'rv_profile': 'tf',
},
]
def run_experiment(exp_cfg, root_uri, test=True, remote=False, commands=None):
uris = exp_cfg['remote'] if remote else exp_cfg['local']
cmd = ['rastervision']
rv_profile = exp_cfg.get('rv_profile')
if rv_profile is not None:
cmd += ['-p', rv_profile]
cmd += ['run', 'aws_batch' if remote else 'inprocess', '-e', exp_cfg['module']]
cmd += ['-a', 'raw_uri', uris['raw_uri']]
if 'processed_uri' in uris:
cmd += ['-a', 'processed_uri', uris['processed_uri']]
cmd += ['-a', 'root_uri', join(root_uri, exp_cfg['key'])]
cmd += ['-a', 'test', 'True' if test else 'False']
extra_args = exp_cfg.get('extra_args')
if extra_args:
for k, v in extra_args:
cmd += ['-a', str(k), str(v)]
cmd += ['--splits', '2', '-x']
if commands is not None:
cmd += ['-r'] + commands
print('running command:')
print(' '.join(cmd))
proc = subprocess.run(cmd)
if proc.returncode != 0:
print('failure!')
print(' '.join(cmd))
exit()
def collect_experiment(key, root_uri, output_dir, get_pred_package=False):
print('\nCollecting experiment {}...\n'.format(key))
if root_uri.startswith('s3://'):
predict_package_uris = list_paths(join(root_uri, key, 'bundle'), ext='predict_package.zip')
eval_json_uris = list_paths(join(root_uri, key, 'eval'), ext='eval.json')
else:
predict_package_uris = glob.glob(join(root_uri, key, 'bundle', '*', 'predict_package.zip'))
eval_json_uris = glob.glob(join(root_uri, key, 'eval', '*', 'eval.json'))
if len(predict_package_uris) > 1 or len(eval_json_uris) > 1:
print('Cannot collect from key with multiple experiments!!!')
return
if len(predict_package_uris) == 0 or len(eval_json_uris) == 0:
print('Missing output!!!')
return
predict_package_uri = predict_package_uris[0]
eval_json_uri = eval_json_uris[0]
make_dir(join(output_dir, key))
if get_pred_package:
download_or_copy(predict_package_uri, join(output_dir, key))
download_or_copy(eval_json_uri, join(output_dir, key))
eval_json = file_to_json(join(output_dir, key, 'eval.json'))
pprint.pprint(eval_json['overall'], indent=4)
def validate_keys(keys):
exp_keys = [exp_cfg['key'] for exp_cfg in cfg]
invalid_keys = set(keys).difference(exp_keys)
if invalid_keys:
raise ValueError('{} are invalid keys'.format(', '.join(invalid_keys)))
@click.group()
def test():
pass
@test.command()
@click.argument('root_uri')
@click.argument('keys', nargs=-1)
@click.option('--test', is_flag=True)
@click.option('--remote', is_flag=True)
@click.option('--commands')
def run(root_uri, keys, test, remote, commands):
run_all = len(keys) == 0
validate_keys(keys)
if commands is not None:
commands = commands.split(' ')
for exp_cfg in cfg:
if run_all or exp_cfg['key'] in keys:
run_experiment(exp_cfg, root_uri, test=test, remote=remote,
commands=commands)
@test.command()
@click.argument('root_uri')
@click.argument('output_dir')
@click.argument('keys', nargs=-1)
@click.option('--get-pred-package', is_flag=True)
def collect(root_uri, output_dir, keys, get_pred_package):
run_all = len(keys) == 0
validate_keys(keys)
for exp_cfg in cfg:
key = exp_cfg['key']
if run_all or key in keys:
collect_experiment(key, root_uri, output_dir, get_pred_package)
@test.command()
@click.argument('root_uri')
def collect_eval_dir(root_uri):
eval_json_uris = list_paths(join(root_uri, 'eval'), ext='eval.json')
for eval_json_uri in eval_json_uris:
eval_json = file_to_json(eval_json_uri)
print(basename(dirname(eval_json_uri)))
print(eval_json['overall'][-1]['f1'])
print()
if __name__ == '__main__':
test()
================================================
FILE: examples/utils.py
================================================
import csv
from io import StringIO
import tempfile
import os
import rasterio
from shapely.strtree import STRtree
from shapely.geometry import shape
from rastervision.core import Box
from rastervision.data import RasterioCRSTransformer, GeoJSONVectorSource
from rastervision.utils.files import (
file_to_str, file_exists, get_local_path, upload_or_copy, make_dir,
file_to_json)
from rastervision.filesystem import S3FileSystem
def str_to_bool(x):
if type(x) == str:
if x.lower() == 'true':
return True
elif x.lower() == 'false':
return False
else:
raise ValueError('{} is expected to be true or false'.format(x))
return x
def get_scene_info(csv_uri):
csv_str = file_to_str(csv_uri)
reader = csv.reader(StringIO(csv_str), delimiter=',')
return list(reader)
def save_image_crop(image_uri, crop_uri, label_uri=None, size=600,
min_features=10):
"""Save a crop of an image to use for testing.
If label_uri is set, the crop needs to cover >= min_features.
Args:
image_uri: URI of original image
crop_uri: URI of cropped image to save
label_uri: optional URI of GeoJSON file
size: height and width of crop
Raises:
ValueError if cannot find a crop satisfying min_features constraint.
"""
if not file_exists(crop_uri):
print('Saving test crop to {}...'.format(crop_uri))
old_environ = os.environ.copy()
try:
request_payer = S3FileSystem.get_request_payer()
if request_payer == 'requester':
os.environ['AWS_REQUEST_PAYER'] = request_payer
im_dataset = rasterio.open(image_uri)
h, w = im_dataset.height, im_dataset.width
extent = Box(0, 0, h, w)
windows = extent.get_windows(size, size)
if label_uri is not None:
crs_transformer = RasterioCRSTransformer.from_dataset(im_dataset)
vs = GeoJSONVectorSource(label_uri, crs_transformer)
geojson = vs.get_geojson()
geoms = []
for f in geojson['features']:
g = shape(f['geometry'])
geoms.append(g)
tree = STRtree(geoms)
for w in windows:
use_window = True
if label_uri is not None:
w_polys = tree.query(w.to_shapely())
use_window = len(w_polys) >= min_features
if use_window:
w = w.rasterio_format()
im = im_dataset.read(window=w)
with tempfile.TemporaryDirectory() as tmp_dir:
crop_path = get_local_path(crop_uri, tmp_dir)
make_dir(crop_path, use_dirname=True)
meta = im_dataset.meta
meta['width'], meta['height'] = size, size
meta['transform'] = rasterio.windows.transform(
w, im_dataset.transform)
with rasterio.open(crop_path, 'w', **meta) as dst:
dst.colorinterp = im_dataset.colorinterp
dst.write(im)
upload_or_copy(crop_path, crop_uri)
break
if not use_window:
raise ValueError('Could not find a good crop.')
finally:
os.environ.clear()
os.environ.update(old_environ)
================================================
FILE: examples/xview/__init__.py
================================================
================================================
FILE: examples/xview/object_detection.py
================================================
import os
from os.path import join
import rastervision as rv
from examples.utils import get_scene_info, str_to_bool, save_image_crop
class ObjectDetectionExperiments(rv.ExperimentSet):
def exp_xview(self, raw_uri, processed_uri, root_uri, test=False):
"""Object detection experiment on xView data.
Run the data prep notebook before running this experiment. Note all URIs can be
local or remote.
Args:
raw_uri: (str) directory of raw data
processed_uri: (str) directory of processed data
root_uri: (str) root directory for experiment output
test: (bool) if True, run a very small experiment as a test and generate
debug output
"""
test = str_to_bool(test)
exp_id = 'xview-vehicles'
batch_size = 16
num_epochs = 20
debug = False
train_scene_info = get_scene_info(join(processed_uri, 'train-scenes.csv'))
val_scene_info = get_scene_info(join(processed_uri, 'val-scenes.csv'))
if test:
exp_id += '-test'
batch_size = 2
num_epochs = 2
debug = True
train_scene_info = train_scene_info[0:1]
val_scene_info = val_scene_info[0:1]
task = rv.TaskConfig.builder(rv.OBJECT_DETECTION) \
.with_chip_size(300) \
.with_classes({'vehicle': (1, 'red')}) \
.with_chip_options(neg_ratio=1.0,
ioa_thresh=0.8) \
.with_predict_options(merge_thresh=0.1,
score_thresh=0.5) \
.build()
backend = rv.BackendConfig.builder(rv.PYTORCH_OBJECT_DETECTION) \
.with_task(task) \
.with_train_options(
lr=1e-4,
one_cycle=True,
batch_size=batch_size,
num_epochs=num_epochs,
model_arch='resnet18',
debug=debug) \
.build()
def make_scene(scene_info):
(raster_uri, label_uri) = scene_info
raster_uri = join(raw_uri, raster_uri)
label_uri = join(processed_uri, label_uri)
if test:
crop_uri = join(
processed_uri, 'crops', os.path.basename(raster_uri))
save_image_crop(raster_uri, crop_uri, size=600, min_features=5)
raster_uri = crop_uri
id = os.path.splitext(os.path.basename(raster_uri))[0]
label_source = rv.LabelSourceConfig.builder(rv.OBJECT_DETECTION) \
.with_uri(label_uri) \
.build()
return rv.SceneConfig.builder() \
.with_task(task) \
.with_id(id) \
.with_raster_source(raster_uri) \
.with_label_source(label_source) \
.build()
train_scenes = [make_scene(info) for info in train_scene_info]
val_scenes = [make_scene(info) for info in val_scene_info]
dataset = rv.DatasetConfig.builder() \
.with_train_scenes(train_scenes) \
.with_validation_scenes(val_scenes) \
.build()
experiment = rv.ExperimentConfig.builder() \
.with_id(exp_id) \
.with_root_uri(root_uri) \
.with_task(task) \
.with_backend(backend) \
.with_dataset(dataset) \
.build()
return experiment
if __name__ == '__main__':
rv.main()
================================================
FILE: notebooks/spacenet/spacenet_rio_chip_class_data_prep.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# SpaceNet Rio Chip Classification Data Prep\n",
"\n",
"This notebook prepares data for training a chip classification model on the Rio SpaceNet dataset.\n",
"\n",
"* Set `raw_uri` to the local or S3 directory containing the raw dataset.\n",
"* Set `processed_uri` to a local or S3 directory (you can write to), which will store the processed data generated by this notebook.\n",
"\n",
"This is all you will need to do in order to run this notebook."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"raw_uri = 's3://spacenet-dataset/'\n",
"# processed_uri = 's3://raster-vision-lf-dev/examples/spacenet/rio/processed-data/'\n",
"processed_uri = '/opt/data/examples/spacenet/rio/processed-data'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The steps we'll take to make the data are as follows:\n",
"\n",
"- Get the building labels and AOI (most likely from the SpaceNet AWS public dataset bucket)\n",
"- Use the AOI and the image bounds to determine which images can be used for training and validation\n",
"- Split the building labels by image, save a label GeoJSON file per image\n",
"- Split the labeled images into a training and validation set, using the percentage of the AOI each covers, aiming at an 80%/20% split.\n",
"\n",
"This process will save the split label files, and `train_scenes.csv` and `val_scenes.csv` files that are used by the experiment at `experiments.spacenet.chip_classification`"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"from os.path import join\n",
"import json\n",
"import tempfile\n",
"\n",
"import boto3\n",
"import botocore\n",
"import rasterio\n",
"from shapely.geometry import (Polygon, shape)\n",
"\n",
"import rastervision as rv\n",
"from rastervision.utils.files import (\n",
" download_if_needed, list_paths, file_to_json, json_to_file, \n",
" get_local_path, make_dir, sync_to_dir, str_to_file)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get the label and AOI data from AWS's public dataset of Space Net"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"label_uri = join(raw_uri, 'AOIs/AOI_1_Rio/srcData/buildingLabels/Rio_Buildings_Public_AOI_v2.geojson')\n",
"aoi_uri = join(raw_uri, 'AOIs/AOI_1_Rio/srcData/buildingLabels/Rio_OUTLINE_Public_AOI.geojson')\n",
"\n",
"label_json = file_to_json(label_uri)\n",
"aoi_json = file_to_json(aoi_uri)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Use the AOI to determine what images are inside the training set\n",
"\n",
"Here we compare the AOI to the image extents to deteremine which images we can use for training and validation. We're using `rasterio`'s ability to read the metadata from raster data on S3 without downloading the whole image"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"image/svg+xml": [
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"100.0\" height=\"100.0\" viewBox=\"-43.7856105430399 -23.02091800403995 0.31122581507989366 0.20602391507999585\" preserveAspectRatio=\"xMinYMin meet\"><g transform=\"matrix(1,0,0,-1,0,-45.8358120929999)\"><path fill-rule=\"evenodd\" fill=\"#66cc99\" stroke=\"#555555\" stroke-width=\"0.006224516301597874\" opacity=\"0.6\" d=\"M -43.596484619999956,-22.95362395099994 L -43.59650719599995,-22.95109540999988 L -43.59647891599991,-22.95037898599992 L -43.59637522299988,-22.94957772099997 L -43.596292080999945,-22.949084805999973 L -43.59576155699989,-22.948589650999963 L -43.594048344999976,-22.947675937999918 L -43.5926777759999,-22.946933546999958 L -43.59119299199989,-22.945505869999977 L -43.59119299199989,-22.94373555099986 L -43.59104681399998,-22.942907205999973 L -43.591821169999946,-22.94207944599998 L -43.593477274999884,-22.941508375999945 L -43.59399123799989,-22.93990937799998 L -43.593705702999955,-22.93836748699988 L -43.592563561999896,-22.937739308999937 L -43.587443556999915,-22.93625720199998 L -43.58426263599989,-22.93533640999982 L -43.58382618099995,-22.938907407999977 L -43.58097082799992,-22.93916698599986 L -43.58017132899994,-22.940194912999914 L -43.57954315099994,-22.941279946999884 L -43.57897208099996,-22.94236498099997 L -43.578229688999954,-22.94350712299996 L -43.577236024999934,-22.94419240799982 L -43.5765735839999,-22.94419240799982 L -43.57520301399984,-22.94333580199998 L -43.57291873199995,-22.94173680399996 L -43.56817723499984,-22.94168792199997 L -43.56659287699995,-22.937666091999915 L -43.564559444999816,-22.937718907999965 L -43.56197144099991,-22.937349193999978 L -43.56162278499994,-22.937271136999925 L -43.56098335499996,-22.935797668999953 L -43.55995542799997,-22.934883955999965 L -43.55652900399997,-22.934883955999965 L -43.556354161999934,-22.936938351999913 L -43.555976778999934,-22.937085111999977 L -43.55489404199989,-22.937639683999976 L -43.55270642799991,-22.93805244099991 L -43.54884249199995,-22.934188504999952 L -43.545651629999895,-22.931588542999975 L -43.543997109999964,-22.929106761999947 L -43.54379066999991,-22.92869388199989 L -43.54391789299996,-22.92715872499997 L -43.5452351859999,-22.926571155999966 L -43.54659707099995,-22.926329530999965 L -43.55563784499998,-22.92603408099984 L -43.55296679299994,-22.91672586699997 L -43.55569693499996,-22.91628422599996 L -43.556352880999896,-22.916571201999886 L -43.556795432999934,-22.91867859199982 L -43.55729718899994,-22.923352844999954 L -43.55758767899994,-22.92604648199989 L -43.563027768999916,-22.926284154999905 L -43.56303233199992,-22.926243343999943 L -43.56424254699988,-22.92794787199989 L -43.56551014099995,-22.93029820299995 L -43.56717385799993,-22.930166161999978 L -43.56976186199995,-22.93003412099995 L -43.57005657899998,-22.92991725099995 L -43.57012048599995,-22.929972748999944 L -43.573147159999905,-22.931743067999946 L -43.57451772999997,-22.931857281999896 L -43.57680201199997,-22.931628852999836 L -43.57817258199998,-22.930486711999833 L -43.57908629499997,-22.92774557299998 L -43.57897208099996,-22.925232861999973 L -43.57600251299982,-22.923234114999957 L -43.57047885999992,-22.922819840999978 L -43.57018439299998,-22.92258700699989 L -43.566940651999914,-22.921002853999937 L -43.566510410999854,-22.919120546999977 L -43.56804675199987,-22.91870691699995 L -43.568874011999924,-22.918056926999895 L -43.57029217299993,-22.915397874999883 L -43.57123761299994,-22.913920624999946 L -43.57194669399996,-22.912797913999952 L -43.570552470999985,-22.91207291799992 L -43.57051274,-22.911069714999883 L -43.57043540699988,-22.90988593399993 L -43.57026193599984,-22.90784509399998 L -43.57060791799995,-22.907803904999923 L -43.571893959999954,-22.90771451699993 L -43.574063412999976,-22.907792305999976 L -43.57419211499996,-22.91143884299987 L -43.57330576399988,-22.915693325999825 L -43.576555715999916,-22.917406935999963 L -43.578623866999976,-22.917288755999948 L -43.57988680899996,-22.917288755999948 L -43.579969907999896,-22.917779794999888 L -43.58001154799996,-22.91798204999992 L -43.58006369199995,-22.91877842199989 L -43.58005749899996,-22.918795143999944 L -43.58014515699983,-22.919814576999954 L -43.58151927799992,-22.921188697999924 L -43.58601012099996,-22.922725038999943 L -43.5889055319999,-22.92319775899989 L -43.59183355299996,-22.92145220799989 L -43.59471066799989,-22.922972968999943 L -43.59586470699992,-22.922794509999846 L -43.596994950999886,-22.921842724999976 L -43.59733997299992,-22.920617302999915 L -43.59711566499993,-22.92024345699997 L -43.598624881999854,-22.919118241999968 L -43.599065081999925,-22.919023063999816 L -43.599374411999975,-22.918178354999895 L -43.59944579599994,-22.917892819999906 L -43.60115900799991,-22.91428793599988 L -43.59825606499987,-22.91390722299991 L -43.59742808499993,-22.913833498999963 L -43.5972923839999,-22.912646107999933 L -43.59541083999994,-22.912704385999973 L -43.594829640999876,-22.908767586999943 L -43.59228294299993,-22.9085695099999 L -43.58556126799988,-22.907117296999957 L -43.58506859299996,-22.906369326999823 L -43.58520472199996,-22.90500803899988 L -43.584383807999984,-22.900261013999966 L -43.58436001299998,-22.899463893999894 L -43.58545456499991,-22.899214050999944 L -43.58532240499994,-22.898589963999882 L -43.58717789799988,-22.89816826099991 L -43.587170695999816,-22.897642554999948 L -43.587279547999856,-22.89739700499996 L -43.58754646099993,-22.896794899999975 L -43.58760797699995,-22.8966561299999 L -43.58775074499994,-22.896406286999877 L -43.587738847999844,-22.895906599999932 L -43.58810766399989,-22.894788253999934 L -43.588500274999944,-22.89466928 L -43.588678734999974,-22.894419436999897 L -43.588976757999944,-22.89349314799989 L -43.59013221599997,-22.89330057199993 L -43.59026107599988,-22.893979236999883 L -43.5904157409999,-22.89438374499997 L -43.59267622899989,-22.89619213599991 L -43.59419908399997,-22.89748894199994 L -43.59396923899993,-22.895575814999916 L -43.594336390999956,-22.894916004999857 L -43.595523614999934,-22.89548453499998 L -43.596587235999834,-22.89560271499994 L -43.59721517899993,-22.89576951199996 L -43.5978753519999,-22.90052275499994 L -43.60173007799983,-22.901474538999935 L -43.60182260499994,-22.901388741999938 L -43.60375261999991,-22.90283083199995 L -43.60596551899988,-22.90103433899992 L -43.60801369799998,-22.901306909999903 L -43.6080160599999,-22.90131028499991 L -43.608344979999856,-22.903544669999974 L -43.60876138599991,-22.9061739739999 L -43.60936814799993,-22.906733147999887 L -43.61111359499995,-22.907611807999956 L -43.61162863599992,-22.91195606499997 L -43.6119974529999,-22.912122626999974 L -43.614243700999964,-22.912675441999966 L -43.61397240499997,-22.91371686599996 L -43.61400128499997,-22.91484894499996 L -43.61311579899996,-22.91485900699996 L -43.61325856699989,-22.91635806799991 L -43.612449549999894,-22.91640565699987 L -43.61186658199989,-22.916703088999952 L -43.61148586899992,-22.916857753999977 L -43.61118843599991,-22.917000521999967 L -43.61104566799992,-22.917214673999922 L -43.61106946299992,-22.91871373399988 L -43.613996199999974,-22.918915987999924 L -43.61586407699997,-22.919142036999972 L -43.61594717799994,-22.91910549199997 L -43.61820784599996,-22.919308598999976 L -43.62006306599994,-22.919470626999896 L -43.62093232899986,-22.919546544999946 L -43.620949073999896,-22.91985034399994 L -43.62101560999997,-22.92105750299993 L -43.62126545299998,-22.922758817999977 L -43.62138442599996,-22.923520244999963 L -43.62193170199993,-22.92360352599991 L -43.62226482699998,-22.923270401999957 L -43.622574156999974,-22.9231871209999 L -43.62293107599993,-22.923258503999932 L -43.62316595199991,-22.923282801999903 L -43.62325230299996,-22.923448860999883 L -43.62515587199988,-22.92426977499997 L -43.62567935299995,-22.9243292619999 L -43.626227556999936,-22.92249488999994 L -43.626500266999926,-22.92239000099994 L -43.62666682999998,-22.92239000099994 L -43.6277375869999,-22.92262794699991 L -43.628903522999906,-22.922937276999846 L -43.62939131299987,-22.923068147999913 L -43.62981961599996,-22.92267553699986 L -43.630592939999985,-22.923008660999983 L -43.63080709199994,-22.922937276999846 L -43.631294880999974,-22.922651741999914 L -43.63168749199997,-22.922330514999828 L -43.632151486999874,-22.92224723399994 L -43.632579789999966,-22.921985492999966 L -43.63340070399994,-22.922128259999965 L -43.63440007799994,-22.922021184999892 L -43.633832060999964,-22.918568004999884 L -43.63457853799997,-22.91766677099986 L -43.63547083499992,-22.917619181999896 L -43.635875343999885,-22.917155186999878 L -43.63726120299998,-22.91724481999995 L -43.640729443999874,-22.919225317999917 L -43.642133325999964,-22.91966551799993 L -43.64217384099993,-22.919671305999884 L -43.642704396999875,-22.92004623199989 L -43.64400120299996,-22.920664891999934 L -43.64522662599995,-22.92105750299993 L -43.646210629999985,-22.921340895999833 L -43.647070707999944,-22.921949800999926 L -43.64856976799996,-22.923377477999964 L -43.649830882999936,-22.92340127199998 L -43.650449541999876,-22.922473281999828 L -43.6500450339999,-22.921521497999947 L -43.64936688799992,-22.921140783999874 L -43.64930207399988,-22.920725976999847 L -43.64949775799994,-22.919356187999938 L -43.64917653099991,-22.917631078999932 L -43.64859356299996,-22.917107597999973 L -43.64673758299989,-22.91621529999992 L -43.6453099069999,-22.915727509999897 L -43.64458417099996,-22.915727509999897 L -43.64427484099997,-22.915905969999983 L -43.644036894999886,-22.916369964999944 L -43.643894126999896,-22.916857753999977 L -43.64414397099989,-22.918071278999946 L -43.643822743999976,-22.91829732799988 L -43.643300987999964,-22.918354040999873 L -43.64187158599998,-22.91796420399993 L -43.642085736999945,-22.915085055999953 L -43.64174071499997,-22.915001774999894 L -43.64120533599993,-22.914930390999928 L -43.64124102799997,-22.914728136999884 L -43.6414670769999,-22.914276038999958 L -43.64174071499997,-22.913835838999944 L -43.64144328299989,-22.91338374099996 L -43.640503857999875,-22.912914028999978 L -43.63706390599992,-22.90907524199997 L -43.622527430999924,-22.90218668899996 L -43.622409578999964,-22.90151171799988 L -43.61945507699994,-22.897493595999947 L -43.61685511599984,-22.89442091499984 L -43.61502332499998,-22.892175493999957 L -43.613841523999895,-22.889220991999935 L -43.61345562999992,-22.88836478999997 L -43.614314244999946,-22.88632558099988 L -43.61596876599987,-22.885557409999933 L -43.61839145699986,-22.88508468999987 L -43.61998688799997,-22.885734680999917 L -43.62376864899994,-22.886443760999896 L -43.62492186899982,-22.886379692999924 L -43.63991881599998,-22.89650421399989 L -43.64011666999983,-22.896348635999914 L -43.64955563399991,-22.904856122999945 L -43.65162576499995,-22.905284425999923 L -43.65469526899983,-22.90578411199988 L -43.65662263299987,-22.908496697999965 L -43.66026320799989,-22.909496071999968 L -43.66526007599998,-22.91006714199989 L -43.66821189399997,-22.911051081999858 L -43.66940033799989,-22.912779727999975 L -43.66954310599988,-22.91534954599996 L -43.66818681299986,-22.92006087799996 L -43.666341460999945,-22.922247962999904 L -43.662560998999936,-22.92216395199989 L -43.66106193899992,-22.922092568999858 L -43.65972943999998,-22.92230671999988 L -43.6582065849999,-22.922639844999935 L -43.654708777999986,-22.923758190999934 L -43.6524958789999,-22.924495823999905 L -43.65152029999996,-22.925518991999922 L -43.651139585999886,-22.92713702599997 L -43.65173445099998,-22.927303587999972 L -43.650996818999886,-22.929920994999918 L -43.650958309999965,-22.93012453999995 L -43.650842153999974,-22.930123248999962 L -43.648819611999954,-22.93002807099998 L -43.64729675699988,-22.930147043999966 L -43.645494274999976,-22.935454350999976 L -43.645467047999944,-22.935534518999873 L -43.64810416199998,-22.93758560899994 L -43.64998537899993,-22.937870641999893 L -43.651815352999904,-22.939610159999916 L -43.65241497699998,-22.939705338999886 L -43.65306219099989,-22.939391249999858 L -43.65311929799998,-22.939267517999895 L -43.65425192099991,-22.93911523299994 L -43.65449938499995,-22.939191374999837 L -43.65505141999989,-22.939400767999928 L -43.655641525999954,-22.93972437499997 L -43.656307774999846,-22.93998135599992 L -43.656802702999926,-22.94030496299996 L -43.657069202999935,-22.94041917699991 L -43.65723100599985,-22.93983858899992 L -43.65803050499983,-22.93998135599992 L -43.658281908999925,-22.93930570799995 L -43.65909650399993,-22.939733891999936 L -43.6598103419999,-22.940266891999897 L -43.66073654899998,-22.940872255999977 L -43.66065767599997,-22.941700423999976 L -43.65889662899997,-22.943141280999896 L -43.65736425599994,-22.944445224999868 L -43.65693595299996,-22.945282795999958 L -43.65691520799993,-22.945499928999936 L -43.656001232999984,-22.94606967899989 L -43.65468022399989,-22.94598711599997 L -43.65365229699995,-22.94595856199993 L -43.65287183399994,-22.945587366999973 L -43.65206281699989,-22.9452066529999 L -43.65122524699996,-22.94533038499992 L -43.650587550999944,-22.94557784899996 L -43.649797569999976,-22.94572061599996 L -43.64835085799996,-22.94615843699995 L -43.64726582399993,-22.946834203999913 L -43.64679944899996,-22.947757434999914 L -43.646694752999906,-22.948909093999816 L -43.647227751999935,-22.94981328899985 L -43.647341965999885,-22.952211785999964 L -43.64726582399993,-22.953277784999955 L -43.64714209199997,-22.954419925999957 L -43.64716206599985,-22.954418058999977 L -43.64687559199996,-22.955714352999962 L -43.647303894999936,-22.957570331999932 L -43.6479796619999,-22.95863633099998 L -43.64905517799997,-22.958988490999957 L -43.6500450339999,-22.959350168999947 L -43.651063442999885,-22.959807025999908 L -43.65141197899993,-22.960083449999956 L -43.651092551999966,-22.960738990999857 L -43.649654802999976,-22.962348289999966 L -43.64932167799992,-22.962462503999973 L -43.64887433899986,-22.962414914999954 L -43.64848410799988,-22.962586235999936 L -43.64812242999989,-22.963252484999884 L -43.64791303699997,-22.963623680999945 L -43.64769412699991,-22.96419475199997 L -43.647551358999976,-22.96485148299996 L -43.64803676899993,-22.96535592899994 L -43.649093249999964,-22.96568905299995 L -43.64967383799984,-22.96607928499992 L -43.650587550999944,-22.966136391999953 L -43.651891495999905,-22.96588892799997 L -43.65269099499989,-22.96558435699984 L -43.65409626899998,-22.965420811999877 L -43.657251575999965,-22.967721266999888 L -43.657992752999974,-22.97141519699983 L -43.65813316899988,-22.975013363999892 L -43.65804954099997,-22.97577796799993 L -43.65801146999996,-22.976567948999843 L -43.65801146999996,-22.977034322999884 L -43.65780207699993,-22.977186608999943 L -43.65769738099988,-22.977548286999934 L -43.65710727399983,-22.978490552999915 L -43.65684077499998,-22.979290051999897 L -43.65674336399991,-22.97987451399996 L -43.65550827699997,-22.98119362099993 L -43.655089490999956,-22.981707584999924 L -43.65444227799992,-22.982278654999902 L -43.65352856499993,-22.982830689999957 L -43.65319544099998,-22.983163814999955 L -43.65306219099989,-22.983554045999938 L -43.65304315499998,-22.98390620699996 L -43.65302411899995,-22.98421077799992 L -43.65346193999994,-22.98501027599997 L -43.65419507099983,-22.985812927999973 L -43.65369869699998,-22.986090789999935 L -43.65379149499984,-22.98633349499994 L -43.65346312999992,-22.98646555499994 L -43.653106210999965,-22.986718966999945 L -43.65333674999994,-22.98693215399993 L -43.653438145999985,-22.987025917999972 L -43.65354522099989,-22.98718653199984 L -43.65305267299988,-22.98746492799995 L -43.65290335699996,-22.987246316999972 L -43.65240664899983,-22.98651909299997 L -43.65187840899989,-22.98672253699982 L -43.65176062599994,-22.986954533999892 L -43.651614288999895,-22.98700807199998 L -43.6514929359999,-22.986890288999916 L -43.65124666199995,-22.986747520999927 L -43.65067650299994,-22.986119667999958 L -43.65032343099995,-22.985134007999932 L -43.649990306999825,-22.98461647599993 L -43.64968097699989,-22.984485604999918 L -43.64936569799994,-22.984455861999948 L -43.64899688199995,-22.984664064999947 L -43.648158416999934,-22.985129123999968 L -43.64778735399989,-22.985334936999948 L -43.647622946999945,-22.985426125999936 L -43.646504395999955,-22.98480088399998 L -43.64588750099989,-22.984396253999876 L -43.64536528999997,-22.984120302999884 L -43.6444580599999,-22.98364089699993 L -43.643928629999834,-22.984438015999956 L -43.6434948939999,-22.98495759599996 L -43.642145223999876,-22.98421077799992 L -43.64184101099994,-22.984078511999883 L -43.64188229299998,-22.983438641999953 L -43.64178711499994,-22.98259988199993 L -43.64171003499996,-22.982439298999964 L -43.64217056499996,-22.98155113499996 L -43.64234188599988,-22.981137108999974 L -43.642463238999824,-22.9806802519999 L -43.642227671999876,-22.98023053399993 L -43.641992104999986,-22.979916444999958 L -43.64008742399989,-22.979545119999955 L -43.640336,-22.978274616999897 L -43.640393106999966,-22.974973812999906 L -43.640393106999966,-22.974148859999957 L -43.640393106999966,-22.971821518999946 L -43.64037883099991,-22.970450948999883 L -43.64046449099993,-22.970208243999878 L -43.64063581299996,-22.969708557999923 L -43.64049304499997,-22.969123209999964 L -43.63983631399998,-22.968580692999865 L -43.63867989599987,-22.968166666999878 L -43.637537753999936,-22.967981068999904 L -43.6357103279999,-22.96799534599984 L -43.6337972419999,-22.968166666999878 L -43.633326107999835,-22.96893761199982 L -43.63322617099993,-22.970251074999965 L -43.63354025999996,-22.971193340999946 L -43.63298811199991,-22.972355420999975 L -43.631259866999926,-22.97311512899995 L -43.62978546999989,-22.97302076699998 L -43.62950972599998,-22.97298525899987 L -43.62693011699997,-22.971650197999907 L -43.62604495799991,-22.970879251999975 L -43.625659484999915,-22.97049378 L -43.62544533299996,-22.969822771999873 L -43.62548816399993,-22.969351637999978 L -43.625745145999986,-22.96870918399992 L -43.62595929699995,-22.968266603999894 L -43.626173448999964,-22.967638426999883 L -43.62641615399991,-22.966838927999902 L -43.62604495799991,-22.965996597999947 L -43.6240319339999,-22.96558257199996 L -43.62297545299998,-22.966225026999894 L -43.62216167699995,-22.96740999799988 L -43.621719097999915,-22.968323710999982 L -43.621547775999886,-22.968837674999975 L -43.62119962799994,-22.969533971999965 L -43.61862103899995,-22.970836421999877 L -43.618310300999894,-22.971118585999875 L -43.61765973699988,-22.970967291999955 L -43.61614283099988,-22.970699602999957 L -43.614447464999955,-22.970699602999957 L -43.61391208599997,-22.970283196999958 L -43.61316850399987,-22.969985764999933 L -43.61096750299993,-22.968439114999967 L -43.61007780799997,-22.967976473999954 L -43.609785522999914,-22.96800226399995 L -43.60949547999991,-22.967825714999947 L -43.60914238299995,-22.96767438799992 L -43.60891539199997,-22.967447396999887 L -43.6085496849999,-22.96582062999994 L -43.60814614599991,-22.963563332999968 L -43.60798221899995,-22.96246573699989 L -43.60781423999998,-22.961840481999957 L -43.59753398099991,-22.96334016299994 L -43.598272046999966,-22.965631470999938 L -43.59899809799998,-22.969095771999946 L -43.601262321999855,-22.974163322999914 L -43.60263938899993,-22.976648255999976 L -43.6078444599999,-22.979354892999936 L -43.61473464799997,-22.983770236999874 L -43.61936262599994,-22.987425041999927 L -43.617321594999964,-22.9932479819999 L -43.616468867999856,-22.994817 L -43.617287485999896,-22.994987544999958 L -43.617287485999896,-22.99536274499991 L -43.617071461999956,-22.99577205299994 L -43.616866806999894,-22.996010816999956 L -43.61657119499995,-22.996454234999874 L -43.61582079599998,-22.997727639999823 L -43.61527505099991,-22.99859173599998 L -43.61486574199989,-22.999262547999876 L -43.61449054199994,-22.999990207999872 L -43.62095566499994,-23.002536403999954 L -43.62310822099994,-23.00788068099996 L -43.630898913999886,-23.009391121999954 L -43.642902940999875,-23.006370240999843 L -43.651448853999966,-23.00366734799985 L -43.65653665299993,-23.000209233999897 L -43.66450626699998,-22.98890505399993 L -43.67126642099993,-22.983136388999924 L -43.67354338499996,-22.97990206399993 L -43.67403500299997,-22.977987343999928 L -43.67455249499994,-22.977159356999834 L -43.67830431099998,-22.97612437299989 L -43.681745631999945,-22.97620199699992 L -43.68195262899985,-22.97664186499992 L -43.68200437799993,-22.977469851999956 L -43.68180068399988,-22.980248824999933 L -43.681714048999936,-22.980415906999895 L -43.68097046699995,-22.985472261999917 L -43.686026821999974,-22.992015778999928 L -43.69316520499996,-22.99305679299988 L -43.704467644999966,-22.98576969399994 L -43.707888119999836,-22.97922617599994 L -43.7069958219999,-22.976103133999914 L -43.70298048199993,-22.972980090999897 L -43.701247734999924,-22.972368532999837 L -43.70075042699989,-22.970582743999955 L -43.70153969899991,-22.970069717999877 L -43.70386841299995,-22.970147340999972 L -43.70632649899994,-22.96986272099997 L -43.70743910699997,-22.969086482999955 L -43.70762022899987,-22.96794800099991 L -43.70793072399988,-22.96580040899994 L -43.708655212999986,-22.963808064999967 L -43.7090494389999,-22.962664808999932 L -43.734657055999946,-22.943831693999982 L -43.754882474999874,-22.932083104999947 L -43.76648234699991,-22.92747289899995 L -43.76899116599992,-22.92289799399998 L -43.773322232999874,-22.92326503299995 L -43.774083660999906,-22.92140905399998 L -43.772564480999904,-22.91638194999996 L -43.774066878999975,-22.913642281999955 L -43.77376944699995,-22.903678288999913 L -43.76514390099993,-22.888955373999977 L -43.74729794299992,-22.87676063699996 L -43.73659036899994,-22.874976040999968 L -43.728559687999905,-22.879586246999963 L -43.729005836999875,-22.88999638899986 L -43.73435962399992,-22.895201459999953 L -43.735951142999966,-22.896960505999914 L -43.73554851299997,-22.89888888799993 L -43.73553639199997,-22.898946941999952 L -43.735450088999926,-22.89892282799991 L -43.73230032499998,-22.89804274699992 L -43.72982568599997,-22.89756685499998 L -43.72735104599991,-22.897281318999887 L -43.72459087199991,-22.89751926499997 L -43.72221141099993,-22.89732890899984 L -43.720600698999874,-22.89732890899984 L -43.71935605699986,-22.894473554999934 L -43.718832575999954,-22.892998288999877 L -43.71611998999998,-22.890904363999937 L -43.703889560999926,-22.881434108999827 L -43.698559567999894,-22.875961348999965 L -43.693515110999954,-22.872677691999968 L -43.6905645789999,-22.87258251399993 L -43.68923208099994,-22.874724028999935 L -43.68794717199995,-22.87796009599998 L -43.68780104399997,-22.87918757099993 L -43.68558067299983,-22.882114423999894 L -43.68528324099998,-22.889104090999922 L -43.68559833399996,-22.893279078999967 L -43.68417679099991,-22.894434082999965 L -43.68189250899985,-22.896789749999982 L -43.679608225999914,-22.897646355999882 L -43.67796639799997,-22.89671836599996 L -43.67746671099985,-22.894077163999896 L -43.679037154999946,-22.891435962999935 L -43.67910853899997,-22.888651992999826 L -43.67746671099985,-22.886796013999913 L -43.67325506499998,-22.88622494299989 L -43.66954310599988,-22.88615355899998 L -43.66675913699993,-22.882727134999982 L -43.66483177299989,-22.880085933999908 L -43.66147673299997,-22.878515488999938 L -43.657907541999975,-22.87930071099987 L -43.656265713999915,-22.880228700999908 L -43.65363918199995,-22.87982901099997 L -43.655919626999946,-22.876389651999887 L -43.662153814999954,-22.862636366999936 L -43.65549132399997,-22.860875565999947 L -43.649447492999855,-22.86206529599997 L -43.645735533999925,-22.86354056199997 L -43.64447713399994,-22.865003579999893 L -43.63927636199992,-22.86816483399997 L -43.63565463699996,-22.87356867799997 L -43.625423169999976,-22.870843992999937 L -43.61927780699983,-22.872025793999967 L -43.61254154399995,-22.875866645999963 L -43.609409771999935,-22.878052976999925 L -43.60296194799997,-22.88276693199998 L -43.60201364999995,-22.881474876999903 L -43.6046234799999,-22.879116596999893 L -43.60482082999994,-22.87852454699987 L -43.59381000399998,-22.86824403199995 L -43.592346547999966,-22.867572036999945 L -43.590848206999965,-22.866884022999955 L -43.590699665999864,-22.865309480999883 L -43.58965865099998,-22.858617246999927 L -43.58787405599992,-22.854899338999815 L -43.58177668699989,-22.850586565999947 L -43.57734898699994,-22.849701025999934 L -43.57473241699989,-22.848121654999886 L -43.566309125999965,-22.846408442999916 L -43.563096852999934,-22.845194917999947 L -43.56109810599986,-22.843338937999874 L -43.55895659099997,-22.838342069999953 L -43.556458156999895,-22.83191752499988 L -43.553103116999864,-22.826992040999926 L -43.5484631679999,-22.826420970999948 L -43.54475120899991,-22.829204939999954 L -43.54260969399991,-22.83684300999994 L -43.54125340099995,-22.845052149999958 L -43.54025402799988,-22.84619429099996 L -43.538312375999965,-22.84666172599998 L -43.53686435999998,-22.846571224999877 L -43.53314645299986,-22.846125075999907 L -43.52987469399994,-22.845827643999883 L -43.52779266499988,-22.84731480699992 L -43.52794138199994,-22.84909940299997 L -43.52883367899989,-22.851032714999974 L -43.52957726099993,-22.852668593999965 L -43.52987469399994,-22.854007040999875 L -43.53046955899998,-22.85608906899995 L -43.532105437999974,-22.8572788 L -43.53567463,-22.8572788 L -43.53872296099985,-22.857173684999964 L -43.54132584999991,-22.859658260999936 L -43.54385402699995,-22.860699274999888 L -43.546530920999885,-22.861442856999872 L -43.547125785999924,-22.86382231799996 L -43.54697706999997,-22.865606912999965 L -43.54742321899994,-22.869176104999895 L -43.54816679999982,-22.872001714999897 L -43.55203342399989,-22.873786310999947 L -43.55768464399995,-22.87452989199994 L -43.557974938999905,-22.87452989199994 L -43.55825571499997,-22.87523183299993 L -43.56353811799994,-22.875731519999874 L -43.56882052199995,-22.875874287999977 L -43.571818641999926,-22.875160448999964 L -43.57581613699995,-22.87416107599995 L -43.5786001059999,-22.873375853999903 L -43.58001668099996,-22.873131616999956 L -43.5839419699999,-22.873975764999898 L -43.58562591799989,-22.87500077699997 L -43.58596396099989,-22.879817887999934 L -43.58439266099998,-22.879884751999953 L -43.580171306999944,-22.88018627699995 L -43.57399003799992,-22.881693903999917 L -43.57234316899991,-22.88220500099993 L -43.561034537999944,-22.882242949999977 L -43.56048322799995,-22.881775648999906 L -43.55906506699995,-22.88029839799998 L -43.55758781599985,-22.87923477699991 L -43.55339242399998,-22.87870296699998 L -43.54854704199994,-22.87769843699982 L -43.54523799999987,-22.876752995999937 L -43.54257894899996,-22.874921204999964 L -43.527629170999944,-22.874389394999923 L -43.5266246409999,-22.88130292799997 L -43.53271091399995,-22.883548348999966 L -43.54163350799996,-22.887566470999957 L -43.545379387999844,-22.88849192399988 L -43.54609894799995,-22.888779747999934 L -43.5505575819999,-22.890730400999928 L -43.55203335399989,-22.895070904999955 L -43.5523288039999,-22.897966316999884 L -43.55221062399983,-22.89914811699998 L -43.55102882299991,-22.89956174699995 L -43.54842886199998,-22.899502656999914 L -43.546478890999936,-22.899502656999914 L -43.54562314799995,-22.899632314999906 L -43.54228349899995,-22.898379946999853 L -43.5330063639999,-22.895779985999923 L -43.532533643999955,-22.896961785999906 L -43.53466088499994,-22.898498126999925 L -43.53525178499996,-22.900802637999846 L -43.53495633499995,-22.903402598999946 L -43.534069984999974,-22.90541166 L -43.53347908399991,-22.90665255099998 L -43.53353817399994,-22.907597990999875 L -43.534188164999875,-22.909902502999955 L -43.533892713999876,-22.912325193999948 L -43.533774533999974,-22.9132706339999 L -43.532297283999924,-22.91433425499997 L -43.52993368299997,-22.91634331599994 L -43.52793205699987,-22.91667085499995 L -43.527146912999854,-22.915100566999968 L -43.52798371199998,-22.91007977299995 L -43.52544283999998,-22.90434803999989 L -43.52307923899991,-22.900270827999975 L -43.51653947899996,-22.898517413999855 L -43.51448446499995,-22.895520517999955 L -43.51112942499998,-22.891879942999935 L -43.5069891629999,-22.888096599999983 L -43.50420519299996,-22.885669549999875 L -43.49806618399987,-22.88545539799992 L -43.49385453799994,-22.887668296999948 L -43.490356729999974,-22.889238740999872 L -43.48635923599994,-22.899303860999908 L -43.48798194299985,-22.90464308999998 L -43.48591161,-22.90606165099996 L -43.48608888099989,-22.909252511999966 L -43.49744270599996,-22.941724452999892 L -43.499763054999846,-22.943257782999922 L -43.50980826499983,-22.949895845999947 L -43.518931721999934,-22.96517998799993 L -43.524763735999954,-22.979039598999975 L -43.53211627099995,-22.982180487999983 L -43.536898986999915,-22.98539275999991 L -43.54567919899995,-22.984678921999944 L -43.55181820799993,-22.985107224999922 L -43.55481632899995,-22.987605658999883 L -43.55595847,-22.99217422399994 L -43.55795721699997,-22.99617171899996 L -43.562668549999955,-22.99674278899994 L -43.56780818599992,-22.994815425999832 L -43.570592154999986,-22.989533021999932 L -43.57344750899989,-22.98253740699994 L -43.57523210399995,-22.98068142799991 L -43.57701669999989,-22.979039598999975 L -43.57801607399989,-22.975399023999955 L -43.57908683099993,-22.969331398999884 L -43.57594594299991,-22.965262519999953 L -43.569898585999965,-22.95996522299987 L -43.568933667999886,-22.954807899999935 L -43.5690693,-22.954483561999893 L -43.570877690999964,-22.953436598999872 L -43.57292402699994,-22.953151063999883 L -43.575779379999915,-22.953674544999956 L -43.57806366299991,-22.953626955999823 L -43.58134731899992,-22.95310347399993 L -43.583166517999985,-22.952992094999956 L -43.587538139999936,-22.954528785999912 L -43.5920495979999,-22.955499605999933 L -43.59467652299992,-22.955042749999905 L -43.59644684199992,-22.953786393999962 L -43.596484619999956,-22.95362395099994 z\" /></g></svg>"
],
"text/plain": [
"<shapely.geometry.polygon.Polygon at 0x7f4011303b38>"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"aoi = shape(aoi_json['features'][0]['geometry'])\n",
"aoi"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"images_uri = join(raw_uri, 'AOIs/AOI_1_Rio/srcData/mosaic_3band')\n",
"image_paths = list_paths(images_uri)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"['s3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223103.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223112.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223113.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223121.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223123.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223130.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223131.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223132.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223133.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223301.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223310.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223311.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232002.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232003.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232020.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232021.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232022.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232023.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232200.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232201.tif']"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"image_paths"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"def bounds_to_shape(bounds):\n",
" return Polygon([[bounds.left, bounds.bottom],\n",
" [bounds.left, bounds.top],\n",
" [bounds.right, bounds.top],\n",
" [bounds.right, bounds.bottom],\n",
" [bounds.left, bounds.bottom]])\n",
"\n",
"image_to_extents = {}\n",
"for img in image_paths:\n",
" with rasterio.open(img, 'r') as ds:\n",
" image_to_extents[img] = bounds_to_shape(ds.bounds)\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"intersecting_images = []\n",
"for img in image_to_extents:\n",
" if image_to_extents[img].intersects(aoi):\n",
" intersecting_images.append(img)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['s3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223123.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223130.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223131.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223132.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022223133.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232020.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232021.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232022.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232023.tif',\n",
" 's3://spacenet-dataset/AOIs/AOI_1_Rio/srcData/mosaic_3band/013022232200.tif']"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"intersecting_images"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Match labels to images\n",
"\n",
"Find the labels that intersect with the image's bounding box, which will be saved into a labels geojson that matches the image name."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# Add a class_id and class_name to the properties of each feature\n",
"for feature in label_json['features']:\n",
" feature['properties']['class_id'] = 1\n",
" feature['properties']['class_name'] = 'building'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"image_to_features = {}\n",
"for img in intersecting_images:\n",
" image_to_features[img] = []\n",
" bbox = image_to_extents[img]\n",
" for feature in label_json['features']:\n",
" if shape(feature['geometry']).intersects(bbox):\n",
" image_to_features[img].append(feature)\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"processed_labels_uri = join(processed_uri, 'labels')\n",
"\n",
"for img in image_to_features:\n",
" fc = {}\n",
" fc['type'] = 'FeatureCollection'\n",
" fc['crs'] = label_json['crs']\n",
" fc['features'] = image_to_features[img]\n",
" img_id = os.path.splitext(os.path.basename(img))[0]\n",
" label_path = join(processed_labels_uri, '{}.geojson'.format(img_id))\n",
" json_to_file(fc, label_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Split into train and validation\n",
"\n",
"Split up training and validation data. There's an odd shaped AOI and not that many images, so we'll split the train and validation roughly based on how much area each scene covers of the AOI. \n",
"\n",
"Create a CSV that our experiments will use to load the training and validation data.\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Split training and validation\n",
"ratio = 0.8\n",
"aoi_area = aoi.area\n",
"images_to_area = {}\n",
"for img in intersecting_images:\n",
" area = image_to_extents[img].intersection(aoi).area\n",
" images_to_area[img] = area / aoi_area\n",
"\n",
"train_imgs = []\n",
"val_imgs = []\n",
"train_area_covered = 0\n",
"for img in sorted(intersecting_images, reverse=True, key=lambda img: images_to_area[img]):\n",
" if train_area_covered < ratio:\n",
" train_imgs.append(img)\n",
" train_area_covered += images_to_area[img]\n",
" else:\n",
" val_imgs.append(img)\n",
"print(\"{} training images with {}% area.\".format(len(train_imgs), train_area_covered))\n",
"print(\"{} validation images with {} area.\".format(len(val_imgs), 1 - train_area_covered))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def save_split_csv(imgs, path):\n",
" csv_rows = []\n",
" for img in imgs:\n",
" img_id = os.path.splitext(os.path.basename(img))[0]\n",
" img_path = join('AOIs', 'AOI_1_Rio', 'srcData/mosaic_3band', '{}.tif'.format(img_id))\n",
" label_path = join('labels','{}.geojson'.format(img_id))\n",
" csv_rows.append('\"{}\",\"{}\"'.format(img_path, label_path))\n",
" str_to_file('\\n'.join(csv_rows), path)\n",
" \n",
"save_split_csv(train_imgs, join(processed_uri, 'train-scenes.csv'))\n",
"save_split_csv(val_imgs, join(processed_uri, 'val-scenes.csv'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: notebooks/xview/xview-vehicles-data-prep.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# xView Vehicle Object Detection Data Prep\n",
"\n",
"This notebook prepares data for training an object detection model on the xView dataset.\n",
"\n",
"\n",
"* Download the training images and labels from the xView competition site, unzip them, and put the contents of each zipfile in a local or S3 directory.\n",
"* Set `raw_uri` to this directory containing the raw dataset.\n",
"* Set `processed_uri` to a local or S3 directory (you can write to), which will store the processed data generated by this notebook.\n",
"\n",
"This is all you will need to do in order to run this notebook."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# raw_uri = '/opt/data/examples/xview/raw-data'\n",
"raw_uri = 's3://raster-vision-xview-example/raw-data'\n",
"processed_uri = '/opt/data/examples/xview/processed-data'\n",
"# processed_uri = 's3://raster-vision-xview-example/processed-data'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The steps we'll take to prepare the data are as follows:\n",
"\n",
"- Filter out all of the non-vehicle bounding boxes from the labels. Combine all vehicle types into one class. \n",
"- Subset the entire xView dataset to only include the images that are most densely populated with vehicles.\n",
"- Split the selected images randomly into 80%/20% training and validation sets\n",
"- Split the vehicle labels by image, and save off a label GeoJSON file per image\n",
"\n",
"\n",
"This process will save the split labels, and `train_scenes.csv` and `val_scenes.csv` files that are used by the experiment at `examples.xview.object_detection` to `processed_uri`."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import os\n",
"from os.path import join\n",
"import json\n",
"import random\n",
"from collections import defaultdict\n",
"\n",
"from rastervision.utils.files import (\n",
" download_if_needed, list_paths, file_to_json, json_to_file, \n",
" get_local_path, make_dir, sync_to_dir, str_to_file)\n",
"\n",
"random.seed(12345)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Filter out non-vehicle labels\n",
"\n",
"The xView dataset includes labels for a number of different types of objects. We are only interested in building a detector for objects that can be categorized as vehicles (e.g. 'small car', 'passenger vehicle', 'bus'). We have pre-determined the ids that map to vehicle labels and will use them to extract all the vehicles from the whole xView label set. In this section we also assign a class name of 'vehicle' to all of the resulting labels."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"label_uri = join(raw_uri, 'xView_train.geojson')\n",
"label_js = file_to_json(label_uri)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"vehicle_type_ids = [17, 18, 19, 20, 21, 23, 24, 25, 26, 27, 28, 29, 32, \n",
" 53, 54, 55, 56, 57, 59, 60, 61, 62, 63, 64, 65, 66]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"vehicle_features = []\n",
"for f in label_js['features']:\n",
" if f['properties']['type_id'] in vehicle_type_ids:\n",
" f['properties']['class_name'] = 'vehicle'\n",
" vehicle_features.append(f)\n",
"label_js['features'] = vehicle_features"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Subset images with the most vehicles\n",
"\n",
"In this section we determine which images contain the most vehicles and are therefore the best candidates for this experiment."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"image_to_vehicle_counts = defaultdict(int)\n",
"for f in label_js['features']:\n",
" image_id = f['properties']['image_id']\n",
" image_to_vehicle_counts[image_id] += 1"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"# Use top 10% of images by vehicle count.\n",
"experiment_image_count = round(len(image_to_vehicle_counts.keys()) * 0.1)\n",
"sorted_images_and_counts = sorted(image_to_vehicle_counts.items(), key=lambda x: x[1])\n",
"selected_images_and_counts = sorted_images_and_counts[-experiment_image_count:]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Split into train and validation\n",
"\n",
"Split up training and validation data. Use 80% of images in the training set and 20% in the validation set."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"ratio = 0.8\n",
"training_sample_size = round(ratio * experiment_image_count)\n",
"train_sample = random.sample(range(experiment_image_count), training_sample_size)\n",
"\n",
"train_images = []\n",
"val_images = []"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"for i in range(training_sample_size):\n",
" img = selected_images_and_counts[i][0]\n",
" img_path = join('train_images', img)\n",
" if i in train_sample:\n",
" train_images.append(img_path)\n",
" else:\n",
" val_images.append(img_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Divide labels up by image\n",
"\n",
"Using one vehicle label geojson for all of the training and validation images can become unwieldy. Instead, we will divide the labels up so that each image has a unique geojson associated with it. We will save each of these geojsons to the base directory you provided at the outset.\n",
"\n",
"Then, we will create CSVs that our experiments will use to load the training and validation data."
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"def subset_labels(images):\n",
" for i in images:\n",
" img_fn = os.path.basename(i)\n",
" img_id = os.path.splitext(img_fn)[0]\n",
" tiff_features = []\n",
" for l in label_js['features']:\n",
" image_id = l['properties']['image_id']\n",
" if image_id == img_fn:\n",
" tiff_features.append(l)\n",
"\n",
" tiff_geojson = {}\n",
" for key in label_js:\n",
" if not key == 'features':\n",
" tiff_geojson[key] = label_js[key]\n",
" tiff_geojson['features'] = tiff_features\n",
" \n",
" json_to_file(tiff_geojson, join(processed_uri, 'labels', '{}.geojson'.format(img_id)))"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"subset_labels(train_images)\n",
"subset_labels(val_images)"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [],
"source": [
"def create_csv(images, path):\n",
" csv_rows = []\n",
" for img in images:\n",
" img_id = os.path.splitext(os.path.basename(img))[0]\n",
" img_path = join('train_images', '{}.tif'.format(img_id))\n",
" labels_path = join('labels','{}.geojson'.format(img_id))\n",
" csv_rows.append('\"{}\",\"{}\"'.format(img_path, labels_path))\n",
" str_to_file('\\n'.join(csv_rows), path)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"create_csv(train_images, join(processed_uri, 'train-scenes.csv'))\n",
"create_csv(val_images, join(processed_uri, 'val-scenes.csv'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.5.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: qgis/spacenet_viz.py
================================================
"""Visualize imagery and predictions in QGIS.
This script should be run in the QGIS (3.0+) Python console after
downloading the data locally, and adjusting constants.
"""
from os.path import join, splitext, basename
import glob
from rastervision.utils.files import download_if_needed, file_to_json
def clear_layers():
layer_ids = QgsProject.instance().mapLayers().keys()
for layer_id in layer_ids:
QgsProject.instance().removeMapLayer(layer_id)
style_uri = '/Users/lfishgold/projects/raster-vision-examples/qgis/style.qml'
predict_dir = '/Users/lfishgold/raster-vision-data/examples/spacenet/vegas/buildings-local-output/predict/buildings-semantic_segmentation'
predict_paths = glob.glob(join(predict_dir, '*.tif'))
image_dir = '/Users/lfishgold/raster-vision-data/raw-data/spacenet-dataset/SpaceNet_Buildings_Dataset_Round2/spacenetV2_Train/AOI_2_Vegas/RGB-PanSharpen/'
gt_dir = '/Users/lfishgold/raster-vision-data/raw-data/spacenet-dataset/SpaceNet_Buildings_Dataset_Round2/spacenetV2_Train/AOI_2_Vegas/geojson/buildings/'
clear_layers()
max_scenes = min(5, len(predict_paths))
for s in range(max_scenes):
predict_path = predict_paths[s]
scene_id = splitext(basename(predict_path))[0]
image_path = join(image_dir, 'RGB-PanSharpen_AOI_2_Vegas_img{}.tif'.format(scene_id))
gt_path = join(gt_dir, 'buildings_AOI_2_Vegas_img{}.geojson'.format(scene_id))
l = iface.addRasterLayer(image_path, scene_id + '-image')
l = iface.addVectorLayer(gt_path, scene_id + '-gt', 'ogr')
l = iface.addRasterLayer(predict_path, scene_id + '-predict')
l.loadNamedStyle(style_uri)
iface.zoomToActiveLayer()
================================================
FILE: qgis/style.qml
================================================
<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>
<qgis minScale="1e+8" version="3.0.2-Girona" maxScale="0" hasScaleBasedVisibilityFlag="0">
<pipe>
<rasterrenderer type="paletted" alphaBand="-1" band="1" opacity="1">
<rasterTransparency/>
<minMaxOrigin>
<limits>None</limits>
<extent>WholeRaster</extent>
<statAccuracy>Estimated</statAccuracy>
<cumulativeCutLower>0.02</cumulativeCutLower>
<cumulativeCutUpper>0.98</cumulativeCutUpper>
<stdDevFactor>2</stdDevFactor>
</minMaxOrigin>
<colorPalette>
<paletteEntry color="#ca3c3c" value="0" label="0" alpha="255"/>
<paletteEntry color="#ff7c01" value="1" label="Building" alpha="127"/>
<paletteEntry color="#2e31da" value="2" label="Background" alpha="0"/>
</colorPalette>
<colorramp type="randomcolors" name="[source]"/>
</rasterrenderer>
<brightnesscontrast contrast="0" brightness="0"/>
<huesaturation grayscaleMode="0" colorizeBlue="128" saturation="0" colorizeGreen="128" colorizeRed="255" colorizeStrength="100" colorizeOn="0"/>
<rasterresampler maxOversampling="2"/>
</pipe>
<blendMode>0</blendMode>
</qgis>
================================================
FILE: scripts/console
================================================
#!/bin/bash
set -e
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done
SCRIPTS_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
REPO_ROOT="$( cd -P "$( dirname "$SCRIPTS_DIR" )" && pwd )"
function usage() {
echo -n \
"Usage: $(basename "$0")
Run a console in the raster-vision docker image locally.
--name sets the name of the running container
--gpu use the NVIDIA runtime and GPU image
All arguments after above options are passed to 'docker run'.
"
}
IMAGE="raster-vision-examples-cpu"
# Parse options using scheme in
# https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
--gpu)
IMAGE="raster-vision-examples-gpu"
RUNTIME="--runtime=nvidia"
shift # past argument
;;
--name)
shift
NAME="--name $1"
shift
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters
DATA_DIR="${RASTER_VISION_DATA_DIR:-${REPO_ROOT}/data}"
if [ -z "$RASTER_VISION_REPO" ]
then
SRC_FLAGS=""
else
SRC_FLAGS="-v ${RASTER_VISION_REPO}/rastervision:/opt/src/rastervision"
fi
if [ "${BASH_SOURCE[0]}" = "${0}" ]
then
AWS="-e AWS_PROFILE=${AWS_PROFILE:-default} -v ${HOME}/.aws:/root/.aws:ro"
TENSORBOARD="-p 6006:6006"
RV_CONFIG="-v ${HOME}/.rastervision:/root/.rastervision:ro"
docker run ${RUNTIME} ${NAME} --rm -it ${TENSORBOARD} ${AWS} ${RV_CONFIG} \
-v "$REPO_ROOT"/spacenet:/opt/src/spacenet \
-v "$REPO_ROOT"/potsdam:/opt/src/potsdam \
-v "$REPO_ROOT"/xview:/opt/src/xview \
-v "$REPO_ROOT"/cowc:/opt/src/cowc \
-v "$REPO_ROOT"/other:/opt/src/other \
-v "$REPO_ROOT"/notebooks:/opt/notebooks \
${SRC_FLAGS} -v "$DATA_DIR":/opt/data \
${IMAGE} "${@:1}"
fi
================================================
FILE: scripts/debug
================================================
#!/bin/bash
set -e
if [[ -n "${RASTER_VISION_DEBUG}" ]]; then
set -x
fi
function usage() {
echo -n \
"Usage: $(basename "$0") <python module + args>
Run a Python module using VS code debugger.
Example:
./scripts/debug rastervision run inprocess -e examples.potsdam
"
}
python -m ptvsd --host 0.0.0.0 --port 3000 --wait -m ${@:1}
gitextract_o_v_juae/
├── .dockerignore
├── .gitignore
├── Dockerfile
├── LICENSE
├── README.md
├── docker/
│ ├── build
│ └── run
├── examples/
│ ├── __init__.py
│ ├── cowc/
│ │ ├── __init__.py
│ │ └── object_detection.py
│ ├── potsdam/
│ │ ├── __init__.py
│ │ └── semantic_segmentation.py
│ ├── spacenet/
│ │ ├── __init__.py
│ │ ├── rio/
│ │ │ ├── __init__.py
│ │ │ ├── chip_classification.py
│ │ │ └── semantic_segmentation.py
│ │ └── vegas/
│ │ ├── __init__.py
│ │ ├── all.py
│ │ ├── hyperparameters.py
│ │ └── simple_segmentation.py
│ ├── test.py
│ ├── utils.py
│ └── xview/
│ ├── __init__.py
│ └── object_detection.py
├── notebooks/
│ ├── spacenet/
│ │ └── spacenet_rio_chip_class_data_prep.ipynb
│ └── xview/
│ └── xview-vehicles-data-prep.ipynb
├── qgis/
│ ├── spacenet_viz.py
│ └── style.qml
└── scripts/
├── console
└── debug
SYMBOL INDEX (53 symbols across 11 files)
FILE: examples/cowc/object_detection.py
class CowcObjectDetectionExperiments (line 8) | class CowcObjectDetectionExperiments(rv.ExperimentSet):
method exp_main (line 9) | def exp_main(self, raw_uri, processed_uri, root_uri, test=False, use_t...
FILE: examples/potsdam/semantic_segmentation.py
class PotsdamSemanticSegmentation (line 8) | class PotsdamSemanticSegmentation(rv.ExperimentSet):
method exp_main (line 9) | def exp_main(self, raw_uri, processed_uri, root_uri, test=False, use_t...
FILE: examples/spacenet/rio/chip_classification.py
class ChipClassificationExperiments (line 10) | class ChipClassificationExperiments(rv.ExperimentSet):
method exp_main (line 11) | def exp_main(self, raw_uri, processed_uri, root_uri, test=False, use_t...
FILE: examples/spacenet/rio/semantic_segmentation.py
class SemanticSegmentationExperiments (line 10) | class SemanticSegmentationExperiments(rv.ExperimentSet):
method exp_main (line 11) | def exp_main(self, raw_uri, processed_uri, root_uri, test=False):
FILE: examples/spacenet/vegas/all.py
class SpacenetConfig (line 14) | class SpacenetConfig(object):
method __init__ (line 15) | def __init__(self, raw_uri):
method create (line 19) | def create(raw_uri, target):
method get_raster_source_uri (line 27) | def get_raster_source_uri(self, id):
method get_geojson_uri (line 32) | def get_geojson_uri(self, id):
method get_scene_ids (line 37) | def get_scene_ids(self):
method get_class_map (line 47) | def get_class_map(self):
method get_class_id_to_filter (line 51) | def get_class_id_to_filter(self):
class VegasRoads (line 55) | class VegasRoads(SpacenetConfig):
method __init__ (line 56) | def __init__(self, raw_uri):
method get_class_map (line 64) | def get_class_map(self):
method get_class_id_to_filter (line 71) | def get_class_id_to_filter(self):
class VegasBuildings (line 75) | class VegasBuildings(SpacenetConfig):
method __init__ (line 76) | def __init__(self, raw_uri):
method get_class_map (line 84) | def get_class_map(self):
method get_class_id_to_filter (line 91) | def get_class_id_to_filter(self):
function build_scene (line 95) | def build_scene(task, spacenet_config, id, channel_order=None, vector_ti...
function build_dataset (line 166) | def build_dataset(task, spacenet_config, test, vector_tile_options=None):
function build_task (line 203) | def build_task(task_type, class_map):
function build_backend (line 233) | def build_backend(task, test):
function str_to_bool (line 290) | def str_to_bool(x):
function validate_options (line 301) | def validate_options(task_type, target, vector_tile_options=None):
class VectorTileOptions (line 320) | class VectorTileOptions():
method __init__ (line 321) | def __init__(self, uri, zoom, id_field):
method build (line 327) | def build(config_str):
class SpacenetVegas (line 335) | class SpacenetVegas(rv.ExperimentSet):
method exp_main (line 336) | def exp_main(self, raw_uri, root_uri, test=False,
FILE: examples/spacenet/vegas/hyperparameters.py
function build_backend (line 9) | def build_backend(task, test, learning_rate):
class HyperParameterSearch (line 32) | class HyperParameterSearch(rv.ExperimentSet):
method exp_main (line 33) | def exp_main(self, raw_uri, root_uri, test=False, learning_rates='0.00...
FILE: examples/spacenet/vegas/simple_segmentation.py
class SpacenetVegasSimpleSegmentation (line 18) | class SpacenetVegasSimpleSegmentation(rv.ExperimentSet):
method exp_main (line 19) | def exp_main(self, raw_uri, root_uri, test=False):
FILE: examples/test.py
function run_experiment (line 142) | def run_experiment(exp_cfg, root_uri, test=True, remote=False, commands=...
function collect_experiment (line 171) | def collect_experiment(key, root_uri, output_dir, get_pred_package=False):
function validate_keys (line 201) | def validate_keys(keys):
function test (line 209) | def test():
function run (line 219) | def run(root_uri, keys, test, remote, commands):
function collect (line 236) | def collect(root_uri, output_dir, keys, get_pred_package):
function collect_eval_dir (line 248) | def collect_eval_dir(root_uri):
FILE: examples/utils.py
function str_to_bool (line 18) | def str_to_bool(x):
function get_scene_info (line 29) | def get_scene_info(csv_uri):
function save_image_crop (line 35) | def save_image_crop(image_uri, crop_uri, label_uri=None, size=600,
FILE: examples/xview/object_detection.py
class ObjectDetectionExperiments (line 8) | class ObjectDetectionExperiments(rv.ExperimentSet):
method exp_xview (line 9) | def exp_xview(self, raw_uri, processed_uri, root_uri, test=False):
FILE: qgis/spacenet_viz.py
function clear_layers (line 10) | def clear_layers():
Condensed preview — 30 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (173K chars).
[
{
"path": ".dockerignore",
"chars": 5,
"preview": "data/"
},
{
"path": ".gitignore",
"chars": 307,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packagi"
},
{
"path": "Dockerfile",
"chars": 81,
"preview": "ARG BASE_IMAGE\nFROM $BASE_IMAGE\n\nCOPY ./examples /opt/src/examples\n\nCMD [\"bash\"]\n"
},
{
"path": "LICENSE",
"chars": 642,
"preview": "This software is licensed under the Apache 2 license, quoted below.\n\nCopyright 2017 Azavea [http://www.azavea.com]\n\nLice"
},
{
"path": "README.md",
"chars": 37021,
"preview": "# Raster Vision Examples (for RV < 0.12)\n\nThis repository contains examples of using Raster Vision on open datasets. \n\n#"
},
{
"path": "docker/build",
"chars": 1394,
"preview": "#!/bin/bash\n\nset -e\n\nif [[ -n \"${RASTER_VISION_DEBUG}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n echo -n \\\n "
},
{
"path": "docker/run",
"chars": 3752,
"preview": "#!/bin/bash\n\n# This is the same as the main repo `run` script except that it:\n# * mounts the examples directory\n# * uses"
},
{
"path": "examples/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/cowc/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/cowc/object_detection.py",
"chars": 4557,
"preview": "import os\nfrom os.path import join\n\nimport rastervision as rv\nfrom examples.utils import str_to_bool, save_image_crop\n\n\n"
},
{
"path": "examples/potsdam/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/potsdam/semantic_segmentation.py",
"chars": 5610,
"preview": "import os\nfrom os.path import join\n\nimport rastervision as rv\nfrom examples.utils import str_to_bool, save_image_crop\n\n\n"
},
{
"path": "examples/spacenet/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/spacenet/rio/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/spacenet/rio/chip_classification.py",
"chars": 6018,
"preview": "import os\nfrom os.path import join\n\nimport rastervision as rv\nfrom examples.utils import get_scene_info, str_to_bool, sa"
},
{
"path": "examples/spacenet/rio/semantic_segmentation.py",
"chars": 4509,
"preview": "import os\nfrom os.path import join\n\nimport rastervision as rv\nfrom examples.utils import get_scene_info, str_to_bool, sa"
},
{
"path": "examples/spacenet/vegas/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/spacenet/vegas/all.py",
"chars": 14592,
"preview": "import re\nimport random\nimport os\nfrom abc import abstractmethod\n\nimport rastervision as rv\nfrom rastervision.utils.file"
},
{
"path": "examples/spacenet/vegas/hyperparameters.py",
"chars": 3322,
"preview": "import os\n\nimport rastervision as rv\nfrom examples.spacenet.vegas.all import (\n SpacenetConfig, build_dataset, build_"
},
{
"path": "examples/spacenet/vegas/simple_segmentation.py",
"chars": 10584,
"preview": "import re\nimport random\nimport os\nfrom os.path import join\n\nimport rastervision as rv\nfrom rastervision.utils.files impo"
},
{
"path": "examples/test.py",
"chars": 9058,
"preview": "import subprocess\nfrom os.path import join, basename, dirname\nimport json\nimport pprint\nimport glob\nimport os\n\nimport cl"
},
{
"path": "examples/utils.py",
"chars": 3549,
"preview": "import csv\nfrom io import StringIO\nimport tempfile\nimport os\n\nimport rasterio\nfrom shapely.strtree import STRtree\nfrom s"
},
{
"path": "examples/xview/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/xview/object_detection.py",
"chars": 4014,
"preview": "import os\nfrom os.path import join\n\nimport rastervision as rv\nfrom examples.utils import get_scene_info, str_to_bool, sa"
},
{
"path": "notebooks/spacenet/spacenet_rio_chip_class_data_prep.ipynb",
"chars": 42721,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# SpaceNet Rio Chip Classification "
},
{
"path": "notebooks/xview/xview-vehicles-data-prep.ipynb",
"chars": 8857,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# xView Vehicle Object Detection Da"
},
{
"path": "qgis/spacenet_viz.py",
"chars": 1638,
"preview": "\"\"\"Visualize imagery and predictions in QGIS.\nThis script should be run in the QGIS (3.0+) Python console after\ndownload"
},
{
"path": "qgis/style.qml",
"chars": 1205,
"preview": "<!DOCTYPE qgis PUBLIC 'http://mrcc.com/qgis.dtd' 'SYSTEM'>\n<qgis minScale=\"1e+8\" version=\"3.0.2-Girona\" maxScale=\"0\" has"
},
{
"path": "scripts/console",
"chars": 2043,
"preview": "#!/bin/bash\n\nset -e\n\nSOURCE=\"${BASH_SOURCE[0]}\"\nwhile [ -h \"$SOURCE\" ] ; do SOURCE=\"$(readlink \"$SOURCE\")\"; done\nSCRIPTS"
},
{
"path": "scripts/debug",
"chars": 351,
"preview": "#!/bin/bash\n\nset -e\n\nif [[ -n \"${RASTER_VISION_DEBUG}\" ]]; then\n set -x\nfi\n\nfunction usage() {\n echo -n \\\n "
}
]
About this extraction
This page contains the full source code of the azavea/raster-vision-examples GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 30 files (161.9 KB), approximately 46.9k tokens, and a symbol index with 53 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.