## About
| Continuous Integration | License | Chat |
|:------------------:|:-------:|:----:|
| | [](https://opensource.org/licenses/Apache-2.0) | [](https://gitter.im/locationtech/geowave?utm_source=badge&utm_medium=badge&utm_content=badge) |
GeoWave is an open source set of software that:
* Capabilities
* Adds multi-dimensional indexing capability to key/value stores (currently [Apache Accumulo](https://accumulo.apache.org), [Apache HBase](https://hbase.apache.org), [Apache Cassandra](http://cassandra.apache.org/), [Amazon DynamoDB](https://aws.amazon.com/dynamodb/), [Cloud Bigtable](https://cloud.google.com/bigtable/), [Redis](https://redis.io/), [RocksDB](https://rocksdb.org/), and [Apache Kudu](https://kudu.apache.org/), as well as direct FileSystem support)
* Adds support for geographic objects and geospatial operators to these stores
* Provides Map-Reduce input and output formats for distributed processing and analysis of geospatial data
* Geospatial software plugins
* [GeoServer](http://geoserver.org/) plugin to allow geospatial data in various key/value stores to be shared and visualized via OGC standard services
Basically, GeoWave is working to bridge geospatial software with modern key/value stores and distributed compute systems.
## The Docs
* [GeoWave](https://locationtech.github.io/geowave/latest/index.html) - Latest snapshot documentation homepage
* [GeoWave Overview](https://locationtech.github.io/geowave/latest/overview.html) - Overview of GeoWave's capabilities
* [Installation Guide](https://locationtech.github.io/geowave/latest/installation-guide.html) - Installation instructions for standalone installers and from RPMs
* [Quickstart Guide](https://locationtech.github.io/geowave/latest/quickstart.html) - A quick demo of GeoWave features using the command-line interface
* [User Guide](https://locationtech.github.io/geowave/latest/userguide.html) - A guide for using GeoWave through the command-line interface and GeoServer plugin
* [Developer Guide](https://locationtech.github.io/geowave/latest/devguide.html) - A guide for developing applications that utilize GeoWave
* [Command-Line Interface](https://locationtech.github.io/geowave/latest/commands.html) - Full documentation for the GeoWave CLI
* [Changelog](https://locationtech.github.io/geowave/latest/changelog.html) - Changes and features for each of our [GitHub releases](https://github.com/locationtech/geowave/releases)
* The underlying principles employed in GeoWave are outlined in past academic publications to include largely the background theory in [Advances in Spatial and Temporal Databases 2017](https://link.springer.com/chapter/10.1007/978-3-319-64367-0_6) and a derivative, more applied paper in [FOSS4G Conference Proceedings 2017](http://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1027&context=foss4g).
## The Software
* We have [multi-platform standalone installers](https://locationtech.github.io/geowave/latest/installation-guide.html#standalone-installers) for the GeoWave's command-line tools to help get started
* This is often the quickest and easiest way to get started using GeoWave on your own machine
* We have a [RPM repository](https://locationtech.github.io/geowave/latest/downloads.html)
* This contains various packages including puppet modules, best used for distributed environments.
* See the [Installation Guide](https://locationtech.github.io/geowave/latest/installation-guide.html#installation-from-rpm) for more info.
* Maven artifacts are available on Maven Central
* And you can always [build from source](https://locationtech.github.io/geowave/latest/devguide.html#development-setup)
## Community
* Community support is available on [chat](https://gitter.im/locationtech/geowave) and on [our mailing list](mailto:geowave-dev@eclipse.org).
## Getting Started
### Programmatic Access
You can use Maven to reference pre-built GeoWave artifacts with the following pom.xml snippet (replacing `${keyvalue-datastore}` with your data store of choice and `${geowave.version}` with the GeoWave version you'd like to use):
```
org.locationtech.geowavegeowave-datastore-${keyvalue-datastore}${geowave.version}org.locationtech.geowavegeowave-adapter-vector${geowave.version}org.locationtech.geowavegeowave-adapter-raster${geowave.version}
```
Use the libraries available in the `api` package to leverage GeoWave's capabilities (where `` might be `AccumuloRequiredOptions` or `HBaseRequiredOptions` and simple examples of creating the data type and index can be found in `SimpleIngest` within the `examples` directory):
```java
DataStore store = DataStoreFactory.createDataStore();
store.addType(, );
try(Writer writer = store.createWriter()){
//write data
writer.writer();
}
//this just queries everything
try(CloseableIterator it = store.query(QueryBuilder.newBuilder().build())){
while(it.hasNext()){
//retrieve results matching query criteria and do something
it.next();
}
}
```
See the [Developer Guide](https://locationtech.github.io/geowave/latest/devguide.html#programmatic-api-examples) for more detailed programmatic API examples.
### Command-line Access
Alternatively, you can always use the GeoWave command-line to access the same capabilities:
```bash
# Add a new RocksDB data store called myStore in the current directory
geowave store add -t rocksdb myStore
# Add a spatial index called spatialIdx to myStore
geowave index add -t spatial myStore spatialIdx
# Ingest a shapefile with states into myStore in the spatialIdx index
geowave ingest localToGW -f geotools-vector states.shp myStore spatialIdx
# Query all the data in the states type from myStore
geowave vector query "SELECT * FROM myStore.states"
```
See the [CLI documentation](https://locationtech.github.io/geowave/latest/commands.html) for a full list of commands and their options.
## Some GeoWave rendered eye candy
See [Example Screenshots](https://locationtech.github.io/geowave/latest/overview.html#example-screenshots) in the GeoWave Overview for more information.
## Supported versions of core libraries
We work to maintain a N and N-1 tested and supported version pace for the following core libraries.
| GeoServer | GeoTools | Accumulo | HBase | Hadoop | Java |
|:---------:|:--------:|:--------:|:-----:|:------:|:----:|
| 2.19.x | 25.x | [1.9.x,2.0.x] | 2.4.x | [2.10.x,3.1.x] | Java8 |
* [Apache Maven](http://maven.apache.org/) 3.x or greater is required for building
## Origin
GeoWave was originally developed at the National Geospatial-Intelligence Agency (NGA) in collaboration with [RadiantBlue Technologies](http://www.radiantblue.com/) (now [Maxar Technologies](https://www.maxar.com/)) and [Booz Allen Hamilton](http://www.boozallen.com/). The software use, modification, and distribution rights are stipulated within the [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0.html) license.
## Contributing
All pull request contributions to this project will be released under the Apache 2.0 or compatible license. Contributions are welcome and guidelines are provided [here](https://locationtech.github.io/geowave/latest/devguide.html#how-to-contribute).
Did I mention our [documentation!](https://locationtech.github.io/geowave/latest/index.html)
================================================
FILE: analytics/api/pom.xml
================================================
4.0.0geowave-analytic-parentorg.locationtech.geowave../2.0.2-SNAPSHOTgeowave-analytic-apiGeoWave Analytics APIorg.locationtech.geowavegeowave-adapter-vector${project.version}jdk.toolsjdk.toolsmockito-allorg.mockitojavax.servlet*org.geoservergs-wmsorg.geoservergs-mainorg.locationtech.geowavegeowave-adapter-raster${project.version}jdk.toolsjdk.toolsmockito-allorg.mockitojavax.servlet*org.geoservergs-wmsorg.geoservergs-main
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/AdapterWithObjectWritable.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.ObjectWritable;
import org.apache.hadoop.io.Writable;
import org.locationtech.geowave.core.index.ByteArray;
import org.locationtech.geowave.mapreduce.HadoopWritableSerializationTool;
public class AdapterWithObjectWritable implements Writable {
private ObjectWritable objectWritable;
private Short internalAdapterId = null;
private ByteArray dataId;
public void setObject(final ObjectWritable data) {
objectWritable = data;
}
public ObjectWritable getObjectWritable() {
return objectWritable;
}
protected void setObjectWritable(final ObjectWritable objectWritable) {
this.objectWritable = objectWritable;
}
public Short getInternalAdapterId() {
return internalAdapterId;
}
public void setInternalAdapterId(final short internalAdapterId) {
this.internalAdapterId = internalAdapterId;
}
public ByteArray getDataId() {
return dataId;
}
public void setDataId(final ByteArray dataId) {
this.dataId = dataId;
}
@Override
public void readFields(final DataInput input) throws IOException {
internalAdapterId = input.readShort();
final int dataIdLength = input.readUnsignedShort();
if (dataIdLength > 0) {
final byte[] dataIdBinary = new byte[dataIdLength];
input.readFully(dataIdBinary);
dataId = new ByteArray(dataIdBinary);
}
if (objectWritable == null) {
objectWritable = new ObjectWritable();
}
objectWritable.readFields(input);
}
@Override
public void write(final DataOutput output) throws IOException {
output.writeShort(internalAdapterId);
if (dataId != null) {
final byte[] dataIdBinary = dataId.getBytes();
output.writeShort((short) dataIdBinary.length);
output.write(dataIdBinary);
} else {
output.writeShort(0);
}
objectWritable.write(output);
}
public static void fillWritableWithAdapter(
final HadoopWritableSerializationTool serializationTool,
final AdapterWithObjectWritable writableToFill,
final short internalAdapterId,
final ByteArray dataId,
final Object entry) {
writableToFill.setInternalAdapterId(internalAdapterId);
writableToFill.setDataId(dataId);
writableToFill.setObject(serializationTool.toWritable(internalAdapterId, entry));
}
public static Object fromWritableWithAdapter(
final HadoopWritableSerializationTool serializationTool,
final AdapterWithObjectWritable writableToExtract) {
final short internalAdapterId = writableToExtract.getInternalAdapterId();
final Object innerObj = writableToExtract.objectWritable.get();
return (innerObj instanceof Writable)
? serializationTool.getHadoopWritableSerializerForAdapter(internalAdapterId).fromWritable(
(Writable) innerObj)
: innerObj;
}
@Override
public String toString() {
return "AdapterWithObjectWritable [ internalAdapterId="
+ internalAdapterId
+ ", dataId="
+ dataId.getString()
+ "]";
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/AnalyticFeature.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.util.List;
import org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.feature.simple.SimpleFeatureTypeBuilder;
import org.geotools.feature.type.BasicFeatureTypes;
import org.geotools.referencing.CRS;
import org.locationtech.geowave.adapter.vector.FeatureDataAdapter;
import org.locationtech.geowave.core.geotime.index.SpatialDimensionalityTypeProvider;
import org.locationtech.geowave.core.geotime.index.SpatialOptions;
import org.locationtech.jts.geom.Geometry;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.feature.type.AttributeDescriptor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* A set of utilities to describe and create a simple feature for use within the set of analytics.
*/
public class AnalyticFeature {
static final Logger LOGGER = LoggerFactory.getLogger(AnalyticFeature.class);
public static SimpleFeature createGeometryFeature(
final SimpleFeatureType featureType,
final String batchId,
final String dataId,
final String name,
final String groupID,
final double weight,
final Geometry geometry,
final String[] extraDimensionNames,
final double[] extraDimensions,
final int zoomLevel,
final int iteration,
final long count) {
if (extraDimensionNames.length != extraDimensions.length) {
LOGGER.error(
"The number of extraDimension names does not equal the number of extraDimensions");
throw new IllegalArgumentException(
"The number of extraDimension names does not equal the number of extraDimensions");
}
final List descriptors = featureType.getAttributeDescriptors();
final Object[] defaults = new Object[descriptors.size()];
int p = 0;
for (final AttributeDescriptor descriptor : descriptors) {
defaults[p++] = descriptor.getDefaultValue();
}
final SimpleFeature newFeature = SimpleFeatureBuilder.build(featureType, defaults, dataId);
newFeature.setAttribute(ClusterFeatureAttribute.NAME.attrName(), name);
newFeature.setAttribute(ClusterFeatureAttribute.GROUP_ID.attrName(), groupID);
newFeature.setAttribute(ClusterFeatureAttribute.ITERATION.attrName(), iteration);
newFeature.setAttribute(ClusterFeatureAttribute.WEIGHT.attrName(), weight);
newFeature.setAttribute(ClusterFeatureAttribute.BATCH_ID.attrName(), batchId);
newFeature.setAttribute(ClusterFeatureAttribute.COUNT.attrName(), count);
newFeature.setAttribute(ClusterFeatureAttribute.GEOMETRY.attrName(), geometry);
newFeature.setAttribute(ClusterFeatureAttribute.ZOOM_LEVEL.attrName(), zoomLevel);
int i = 0;
for (final String dimName : extraDimensionNames) {
newFeature.setAttribute(dimName, new Double(extraDimensions[i++]));
}
return newFeature;
}
public static FeatureDataAdapter createFeatureAdapter(
final String centroidDataTypeId,
final String[] extraNumericDimensions,
final String namespaceURI,
final String SRID,
final ClusterFeatureAttribute[] attributes,
final Class geometryClass) {
try {
final SimpleFeatureTypeBuilder builder = new SimpleFeatureTypeBuilder();
builder.setName(centroidDataTypeId);
builder.setNamespaceURI(
namespaceURI == null ? BasicFeatureTypes.DEFAULT_NAMESPACE : namespaceURI);
builder.setSRS(SRID);
builder.setCRS(CRS.decode(SRID, true));
for (final ClusterFeatureAttribute attrVal : attributes) {
builder.add(
attrVal.name,
attrVal.equals(ClusterFeatureAttribute.GEOMETRY) ? geometryClass : attrVal.type);
}
for (final String extraDim : extraNumericDimensions) {
builder.add(extraDim, Double.class);
}
final FeatureDataAdapter adapter = new FeatureDataAdapter(builder.buildFeatureType());
return adapter;
} catch (final Exception e) {
LOGGER.warn("Schema Creation Error. Hint: Check the SRID.", e);
}
return null;
}
public static FeatureDataAdapter createGeometryFeatureAdapter(
final String centroidDataTypeId,
final String[] extraNumericDimensions,
final String namespaceURI,
final String SRID) {
return createFeatureAdapter(
centroidDataTypeId,
extraNumericDimensions,
namespaceURI,
SRID,
ClusterFeatureAttribute.values(),
Geometry.class);
}
public static enum ClusterFeatureAttribute {
NAME("name", String.class),
GROUP_ID("groupID", String.class),
ITERATION("iteration", Integer.class),
GEOMETRY("geometry", Geometry.class),
WEIGHT("weight", Double.class),
COUNT("count", Long.class),
ZOOM_LEVEL("level", Integer.class),
BATCH_ID("batchID", String.class);
private final String name;
private final Class type;
ClusterFeatureAttribute(final String name, final Class type) {
this.name = name;
this.type = type;
}
public String attrName() {
return name;
}
public Class getType() {
return type;
}
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/AnalyticItemWrapper.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import org.locationtech.jts.geom.Geometry;
/**
* Wrap an object used to by analytical processes. This class provides generic wrapper to specific
* functions associated with analytic processes such as managing centroids.
*
* @param
*/
public interface AnalyticItemWrapper {
public String getID();
public T getWrappedItem();
public long getAssociationCount();
public void resetAssociatonCount();
public void incrementAssociationCount(long increment);
public int getIterationID();
public String getName();
public String[] getExtraDimensions();
public double[] getDimensionValues();
public Geometry getGeometry();
public double getCost();
public void setCost(double cost);
public String getGroupID();
public void setGroupID(String groupID);
public void setZoomLevel(int level);
public int getZoomLevel();
public void setBatchID(String batchID);
public String getBatchID();
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/AnalyticItemWrapperFactory.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.IOException;
import org.apache.hadoop.mapreduce.JobContext;
import org.locationtech.jts.geom.Coordinate;
import org.slf4j.Logger;
/**
* Create an analytic item wrapper for the provided item.
*
* @param the type of the item to wrap
*/
public interface AnalyticItemWrapperFactory {
/**
* Wrap the item.
*/
public AnalyticItemWrapper create(T item);
/**
* Creates a new item based on the old item with new coordinates and dimension values
*/
public AnalyticItemWrapper createNextItem(
final T feature,
final String groupID,
final Coordinate coordinate,
final String[] extraNames,
final double[] extraValues);
public void initialize(final JobContext context, Class scope, Logger logger)
throws IOException;
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/AnalyticPersistableRegistry.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import org.locationtech.geowave.analytic.clustering.DistortionGroupManagement.BatchIdFilter;
import org.locationtech.geowave.analytic.clustering.DistortionGroupManagement.BatchIdQuery;
import org.locationtech.geowave.analytic.clustering.DistortionGroupManagement.DistortionDataAdapter;
import org.locationtech.geowave.analytic.store.PersistableStore;
import org.locationtech.geowave.core.index.persist.InternalPersistableRegistry;
import org.locationtech.geowave.core.index.persist.PersistableRegistrySpi;
public class AnalyticPersistableRegistry implements
PersistableRegistrySpi,
InternalPersistableRegistry {
@Override
public PersistableIdAndConstructor[] getSupportedPersistables() {
return new PersistableIdAndConstructor[] {
new PersistableIdAndConstructor((short) 700, GeoObjectDimensionValues::new),
new PersistableIdAndConstructor((short) 701, BatchIdFilter::new),
new PersistableIdAndConstructor((short) 702, DistortionDataAdapter::new),
new PersistableIdAndConstructor((short) 703, PersistableStore::new),
new PersistableIdAndConstructor((short) 704, BatchIdQuery::new)};
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/GeoObjectDimensionValues.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.nio.ByteBuffer;
import org.locationtech.geowave.core.index.VarintUtils;
import org.locationtech.geowave.core.index.persist.Persistable;
/**
* Extracted numeric dimension values associated with an item or a sum of dimension values from
* multiple items.
*/
public class GeoObjectDimensionValues implements Persistable {
public double x = 0.0;
public double y = 0.0;
public double z = 0.0;
public double[] values = new double[0];
public double distance = 0.0;
public long count = 0;
public GeoObjectDimensionValues(final int extraValuesCount) {
values = new double[extraValuesCount];
}
public GeoObjectDimensionValues() {}
public GeoObjectDimensionValues(
final double x,
final double y,
final double z,
final double[] extraDimensions,
final double distance) {
super();
this.x = x;
this.y = y;
this.z = z;
values = extraDimensions;
this.distance = distance;
count = 1;
}
public void add(final GeoObjectDimensionValues association) {
x += association.x;
y += association.y;
z += association.z;
for (int i = 0; i < values.length; i++) {
values[i] += association.values[i];
}
distance += association.distance;
count += association.count;
}
public void set(
final double x,
final double y,
final double z,
final double[] extraDimensions,
final double distance) {
this.x = x;
this.y = y;
this.z = z;
values = extraDimensions;
this.distance = distance;
count = 1;
}
public long getCount() {
return count;
}
public void setCount(final long count) {
this.count = count;
}
public double getX() {
return x;
}
public void setX(final double x) {
this.x = x;
}
public double getY() {
return y;
}
public void setY(final double y) {
this.y = y;
}
public double getZ() {
return z;
}
public void setZ(final double z) {
this.z = z;
}
public double getDistance() {
return distance;
}
public void setDistance(final double distance) {
this.distance = distance;
}
@Override
public byte[] toBinary() {
final ByteBuffer b =
ByteBuffer.allocate(
((4 + values.length) * 8)
+ VarintUtils.unsignedIntByteLength(values.length)
+ VarintUtils.unsignedLongByteLength(count));
VarintUtils.writeUnsignedLong(count, b);
b.putDouble(x);
b.putDouble(y);
b.putDouble(z);
b.putDouble(distance);
VarintUtils.writeUnsignedInt(values.length, b);
for (final double value : values) {
b.putDouble(value);
}
return b.array();
}
@Override
public void fromBinary(final byte[] bytes) {
final ByteBuffer b = ByteBuffer.wrap(bytes);
count = VarintUtils.readUnsignedLong(b);
x = b.getDouble();
y = b.getDouble();
z = b.getDouble();
distance = b.getDouble();
int i = VarintUtils.readUnsignedInt(b);
values = new double[i];
for (; i > 0; i--) {
values[i - 1] = b.getDouble();
}
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/GeometryCalculations.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.util.LinkedList;
import java.util.List;
import javax.measure.Unit;
import javax.measure.quantity.Length;
import org.geotools.geometry.jts.ReferencedEnvelope;
import org.geotools.referencing.GeodeticCalculator;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.GeometryFactory;
import org.locationtech.jts.geom.PrecisionModel;
import org.opengis.geometry.DirectPosition;
import org.opengis.referencing.crs.CoordinateReferenceSystem;
import org.opengis.referencing.operation.TransformException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import tech.units.indriya.unit.Units;
public class GeometryCalculations {
private static final Logger LOGGER = LoggerFactory.getLogger(GeometryCalculations.class);
final GeometryFactory factory;
final CoordinateReferenceSystem crs;
final double xMin, yMin, xMax, yMax;
public GeometryCalculations(final CoordinateReferenceSystem crs) {
factory = new GeometryFactory(new PrecisionModel(), 4326);
this.crs = crs;
xMin = crs.getCoordinateSystem().getAxis(0).getMinimumValue();
xMax = crs.getCoordinateSystem().getAxis(0).getMaximumValue();
yMin = crs.getCoordinateSystem().getAxis(1).getMinimumValue();
yMax = crs.getCoordinateSystem().getAxis(1).getMaximumValue();
}
/**
* Build geometries with the provided coordinate at the center. The width of the geometry is twice
* the distance provided. More than one geometry is return when passing the date line.
*
* @param distances [x,y] = [longitude, latitude]
* @param unit
* @param coordinate
* @return the geometries that were built
*/
public List buildSurroundingGeometries(
final double[] distances,
final Unit unit,
final Coordinate coordinate) {
final List geos = new LinkedList<>();
final GeodeticCalculator geoCalc = new GeodeticCalculator();
geoCalc.setStartingGeographicPoint(coordinate.x, coordinate.y);
try {
geoCalc.setDirection(0, unit.getConverterTo(Units.METRE).convert(distances[1]));
final DirectPosition north = geoCalc.getDestinationPosition();
geoCalc.setDirection(90, unit.getConverterTo(Units.METRE).convert(distances[0]));
final DirectPosition east = geoCalc.getDestinationPosition();
geoCalc.setStartingGeographicPoint(coordinate.x, coordinate.y);
geoCalc.setDirection(-90, unit.getConverterTo(Units.METRE).convert(distances[0]));
final DirectPosition west = geoCalc.getDestinationPosition();
geoCalc.setDirection(180, unit.getConverterTo(Units.METRE).convert(distances[1]));
final DirectPosition south = geoCalc.getDestinationPosition();
final double x1 = west.getOrdinate(0);
final double x2 = east.getOrdinate(0);
final double y1 = north.getOrdinate(1);
final double y2 = south.getOrdinate(1);
handleBoundaries(geos, coordinate, x1, x2, y1, y2);
return geos;
} catch (final TransformException ex) {
LOGGER.error("Unable to build geometry", ex);
}
return null;
}
private void handleBoundaries(
final List geos,
final Coordinate coordinate,
final double x1,
final double x2,
final double y1,
final double y2) {
if (Math.signum(x1) > Math.signum(coordinate.x)) {
ReferencedEnvelope bounds =
new ReferencedEnvelope(x1, xMax, Math.max(y1, yMin), Math.min(y2, yMax), crs);
geos.add(factory.toGeometry(bounds));
bounds = new ReferencedEnvelope(xMin, x2, Math.max(y1, yMin), Math.min(y2, yMax), crs);
geos.add(factory.toGeometry(bounds));
} else if (Math.signum(x2) < Math.signum(coordinate.x)) {
ReferencedEnvelope bounds =
new ReferencedEnvelope(xMin, x2, Math.max(y1, yMin), Math.min(y2, yMax), crs);
geos.add(factory.toGeometry(bounds));
bounds = new ReferencedEnvelope(x1, xMax, Math.max(y1, yMin), Math.min(y2, yMax), crs);
geos.add(factory.toGeometry(bounds));
} else {
final ReferencedEnvelope bounds =
new ReferencedEnvelope(x1, x2, Math.max(y1, yMin), Math.min(y2, yMax), crs);
geos.add(factory.toGeometry(bounds));
}
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/GeometryDataSetGenerator.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import java.util.UUID;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.commons.math3.geometry.euclidean.twod.Vector2D;
import org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.feature.simple.SimpleFeatureTypeBuilder;
/** Generate clusters of geometries. */
import org.geotools.referencing.CRS;
import org.locationtech.geowave.adapter.vector.FeatureDataAdapter;
import org.locationtech.geowave.analytic.distance.CoordinateCircleDistanceFn;
import org.locationtech.geowave.analytic.distance.DistanceFn;
import org.locationtech.geowave.core.geotime.index.SpatialDimensionalityTypeProvider;
import org.locationtech.geowave.core.geotime.index.SpatialOptions;
import org.locationtech.geowave.core.store.api.DataStore;
import org.locationtech.geowave.core.store.api.Index;
import org.locationtech.geowave.core.store.api.Writer;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.GeometryFactory;
import org.locationtech.jts.geom.LineString;
import org.locationtech.jts.geom.Point;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.type.AttributeDescriptor;
import org.opengis.feature.type.GeometryType;
import org.opengis.referencing.FactoryException;
import org.opengis.referencing.cs.CoordinateSystem;
import org.opengis.referencing.cs.CoordinateSystemAxis;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** Generate clusters of geometries. */
public class GeometryDataSetGenerator {
static final Logger LOGGER = LoggerFactory.getLogger(GeometryDataSetGenerator.class);
private final Random rand = new Random();
private final GeometryFactory geoFactory = new GeometryFactory();
private final DistanceFn distanceFunction;
private final SimpleFeatureBuilder builder;
// coordinate system boundaries
private SimpleFeature minFeature;
private double[] minAxis;
private double[] maxAxis;
private CoordinateSystem coordSystem;
private boolean includePolygons = true;
public GeometryDataSetGenerator(
final DistanceFn distanceFunction,
final SimpleFeatureBuilder builder) {
super();
this.distanceFunction = distanceFunction;
this.builder = builder;
init();
}
public boolean isIncludePolygons() {
return includePolygons;
}
public void setIncludePolygons(final boolean includePolygons) {
this.includePolygons = includePolygons;
}
public SimpleFeature getCorner() {
return minFeature;
}
public Geometry getBoundingRegion() {
final int[] adder = {1, 2, -1, 2};
int num = 0;
int addCnt = 0;
final int dims = coordSystem.getDimension();
final int coords = (int) Math.pow(dims, 2);
final Coordinate[] coordinates = new Coordinate[coords + 1];
for (int i = 0; i < coords; i++) {
coordinates[i] = new Coordinate();
for (int j = 0; j < dims; j++) {
final boolean isMin = ((num >> j) % 2) == 0;
coordinates[i].setOrdinate(j, isMin ? minAxis[j] : maxAxis[j]);
}
num += adder[addCnt];
addCnt = (addCnt + 1) % 4;
}
coordinates[coords] = coordinates[0];
return geoFactory.createPolygon(coordinates);
}
/**
* Calculate the range for the given bounds
*
* @param factor
* @param minAxis
* @param maxAxis
* @return
*/
private double[] createRange(
final double factor,
final double[] minAxis,
final double[] maxAxis) {
final double[] range = new double[minAxis.length];
for (int i = 0; i < minAxis.length; i++) {
range[i] = (maxAxis[i] - minAxis[i]) * factor;
}
return range;
}
/**
* Pick a random grid cell and supply the boundary. The grid is determined by the parameter,which
* provides a percentage of distance over the total range for each cell.
*
* @param minCenterDistanceFactor
* @return
*/
private Pair gridCellBounds(
final double minCenterDistanceFactor,
final double[] minAxis,
final double[] maxAxis) {
final double[] range = createRange(1.0, minAxis, maxAxis);
final double[] min = new double[range.length];
final double[] max = new double[range.length];
for (int i = 0; i < range.length; i++) {
// HP Fortify "Insecure Randomness" false positive
// This random number is not used for any purpose
// related to security or cryptography
min[i] =
Math.max(
minAxis[i]
+ (minCenterDistanceFactor
* (rand.nextInt(Integer.MAX_VALUE) % (range[i] / minCenterDistanceFactor))),
minAxis[i]);
max[i] = Math.min(min[i] + (minCenterDistanceFactor * range[i]), maxAxis[i]);
}
return Pair.of(min, max);
}
public void writeToGeoWave(final DataStore dataStore, final List featureData)
throws IOException {
final Index index =
SpatialDimensionalityTypeProvider.createIndexFromOptions(new SpatialOptions());
final FeatureDataAdapter adapter = new FeatureDataAdapter(featureData.get(0).getFeatureType());
final SimpleFeatureBuilder featureBuilder =
new SimpleFeatureBuilder(featureData.get(0).getFeatureType());
LOGGER.info(
"Writing " + featureData.size() + " records to " + adapter.getFeatureType().getTypeName());
dataStore.addType(adapter, index);
try (Writer writer = dataStore.createWriter(adapter.getTypeName())) {
for (final SimpleFeature feature : featureData) {
writer.write(feature);
featureBuilder.reset();
}
}
}
public List generatePointSet(
final double minCenterDistanceFactor,
final double outlierFactor,
final int numberOfCenters,
final int minSetSize) {
return this.generatePointSet(
minCenterDistanceFactor,
outlierFactor,
numberOfCenters,
minSetSize,
minAxis,
maxAxis);
}
public List generatePointSet(
final LineString line,
final double distanceFactor,
final int points) {
final List pointSet = new ArrayList<>();
for (final Point point : CurvedDensityDataGeneratorTool.generatePoints(
line,
distanceFactor,
points)) {
pointSet.add(createFeatureWithGeometry(point));
}
return pointSet;
}
public List generatePointSet(
final double minCenterDistanceFactor,
final double outlierFactor,
final int numberOfCenters,
final int minSetSize,
final double[] minAxis,
final double[] maxAxis) {
final List pointSet = new ArrayList<>();
final List minForCenter = new ArrayList<>();
final List maxForCenter = new ArrayList<>();
final double[] range = createRange(minCenterDistanceFactor, minAxis, maxAxis);
if (numberOfCenters >= minSetSize) {
LOGGER.error("The number of centers passed much be less than the minimum set size");
throw new IllegalArgumentException(
"The number of centers passed much be less than the minimum set size");
}
final double minDistance = computeMinDistance(minCenterDistanceFactor, minAxis, maxAxis);
/** Pick the initial centers which have minimum distance from each other. */
while (pointSet.size() < numberOfCenters) {
final Pair axis =
gridCellBounds(minCenterDistanceFactor, minAxis, maxAxis);
final SimpleFeature nextFeature = createNewFeature(axis.getLeft(), axis.getRight());
if (isFarEnough(nextFeature, pointSet, minDistance)) {
pointSet.add(nextFeature);
}
}
/**
* Calculate the boundaries around each center point to place additional points, thus creating
* clusters
*/
for (final SimpleFeature center : pointSet) {
final double[] centerMinAxis = new double[coordSystem.getDimension()];
final double[] centerMaxAxis = new double[coordSystem.getDimension()];
final Geometry geo = (Geometry) center.getDefaultGeometry();
final Coordinate centerCoord = geo.getCentroid().getCoordinate();
for (int i = 0; i < centerMinAxis.length; i++) {
centerMinAxis[i] = centerCoord.getOrdinate(i) - (range[i] / 2.0);
centerMaxAxis[i] = centerCoord.getOrdinate(i) + (range[i] / 2.0);
}
minForCenter.add(centerMinAxis);
maxForCenter.add(centerMaxAxis);
}
/*
* Pick a random center point and add a new geometry with the bounding range around that point.
*/
final int clusterdItemsCount = (int) Math.ceil((minSetSize) * (1.0 - outlierFactor));
while (pointSet.size() < clusterdItemsCount) {
// HP Fortify "Insecure Randomness" false positive
// This random number is not used for any purpose
// related to security or cryptography
final int centerPos = rand.nextInt(Integer.MAX_VALUE) % minForCenter.size();
pointSet.add(createNewFeature(minForCenter.get(centerPos), maxForCenter.get(centerPos)));
}
/** Add random points as potential outliers (no guarantees) */
while (pointSet.size() < minSetSize) {
pointSet.add(createNewFeature(minAxis, maxAxis));
}
return pointSet;
}
public List addRandomNoisePoints(
final List pointSet,
final int minSetSize,
final double[] minAxis,
final double[] maxAxis) {
while (pointSet.size() < minSetSize) {
pointSet.add(createNewFeature(minAxis, maxAxis));
}
return pointSet;
}
private void init() {
coordSystem = builder.getFeatureType().getCoordinateReferenceSystem().getCoordinateSystem();
minAxis = new double[coordSystem.getDimension()];
maxAxis = new double[coordSystem.getDimension()];
for (int i = 0; i < coordSystem.getDimension(); i++) {
final CoordinateSystemAxis axis = coordSystem.getAxis(i);
minAxis[i] = axis.getMinimumValue();
maxAxis[i] = axis.getMaximumValue();
}
final int dims = coordSystem.getDimension();
final Coordinate coordinate = new Coordinate();
for (int i = 0; i < dims; i++) {
coordinate.setOrdinate(i, minAxis[i]);
}
minFeature = createFeatureWithGeometry(geoFactory.createPoint(coordinate));
}
private boolean isFarEnough(
final SimpleFeature feature,
final List set,
final double minDistance) {
for (final SimpleFeature setItem : set) {
if (distanceFunction.measure(feature, setItem) < minDistance) {
return false;
}
}
return true;
}
/**
* Find the distance maximum distance of the entire space and multiply that by the distance factor
* to determine a minimum distance each initial center point occurs from each other.
*
* @param minCenterDistanceFactor
* @return
*/
private double computeMinDistance(
final double minCenterDistanceFactor,
final double[] minAxis,
final double[] maxAxis) {
assert minCenterDistanceFactor < 0.75;
final int dims = coordSystem.getDimension();
Coordinate coordinate = new Coordinate();
for (int i = 0; i < dims; i++) {
coordinate.setOrdinate(i, minAxis[i]);
}
final SimpleFeature minFeature = createFeatureWithGeometry(geoFactory.createPoint(coordinate));
coordinate = new Coordinate();
for (int i = 0; i < dims; i++) {
coordinate.setOrdinate(i, maxAxis[i]);
}
final SimpleFeature maxFeature = createFeatureWithGeometry(geoFactory.createPoint(coordinate));
return minCenterDistanceFactor * distanceFunction.measure(minFeature, maxFeature);
}
private SimpleFeature createNewFeature(final double[] minAxis, final double[] maxAxis) {
final int dims = coordSystem.getDimension();
// HP Fortify "Insecure Randomness" false positive
// This random number is not used for any purpose
// related to security or cryptography
final int shapeSize = includePolygons ? (rand.nextInt(Integer.MAX_VALUE) % 5) + 1 : 1;
final Coordinate[] shape = new Coordinate[shapeSize > 2 ? shapeSize + 1 : shapeSize];
final double[] constrainedMaxAxis = Arrays.copyOf(maxAxis, maxAxis.length);
final double[] constrainedMinAxis = Arrays.copyOf(minAxis, minAxis.length);
for (int s = 0; s < shapeSize; s++) {
final Coordinate coordinate = new Coordinate();
for (int i = 0; i < dims; i++) {
// HP Fortify "Insecure Randomness" false positive
// This random number is not used for any purpose
// related to security or cryptography
coordinate.setOrdinate(
i,
constrainedMinAxis[i]
+ (rand.nextDouble() * (constrainedMaxAxis[i] - constrainedMinAxis[i])));
}
shape[s] = coordinate;
if (s == 0) {
constrain(coordinate, constrainedMaxAxis, constrainedMinAxis);
}
}
if (shapeSize > 2) {
shape[shapeSize] = shape[0];
return createFeatureWithGeometry(geoFactory.createLinearRing(shape).convexHull());
} else if (shapeSize == 2) {
return createFeatureWithGeometry(geoFactory.createLineString(shape));
} else {
return createFeatureWithGeometry(geoFactory.createPoint(shape[0]));
}
}
public GeometryFactory getFactory() {
return geoFactory;
}
/**
* Change the constrain min and max to center around the coordinate to keep the polygons tight.
*
* @param coordinate
* @param constrainedMaxAxis
* @param constrainedMinAxis
*/
private void constrain(
final Coordinate coordinate,
final double[] constrainedMaxAxis,
final double[] constrainedMinAxis) {
for (int i = 0; i < constrainedMaxAxis.length; i++) {
final double range = (constrainedMaxAxis[i] - constrainedMinAxis[i]) * 0.001;
constrainedMaxAxis[i] = Math.min(coordinate.getOrdinate(i) + range, constrainedMaxAxis[i]);
constrainedMinAxis[i] = Math.max(coordinate.getOrdinate(i) - range, constrainedMinAxis[i]);
}
}
private SimpleFeature createFeatureWithGeometry(final Geometry geometry) {
final Object[] values = new Object[builder.getFeatureType().getAttributeCount()];
for (int i = 0; i < values.length; i++) {
final AttributeDescriptor desc = builder.getFeatureType().getDescriptor(i);
if (desc.getType() instanceof GeometryType) {
values[i] = geometry;
} else {
final Class binding = desc.getType().getBinding();
if (String.class.isAssignableFrom(binding)) {
values[i] = UUID.randomUUID().toString();
}
}
}
return builder.buildFeature(UUID.randomUUID().toString(), values);
}
// public static void main(
// final String[] args )
// throws Exception {
// final Options allOptions = new Options();
// DataStoreCommandLineOptions.applyOptions(allOptions);
// final Option typeNameOption = new Option(
// "typename",
// true,
// "a name for the feature type (required)");
// typeNameOption.setRequired(true);
// allOptions.addOption(typeNameOption);
// CommandLine commandLine = new BasicParser().parse(
// allOptions,
// args);
//
// final CommandLineResult dataStoreOption =
// DataStoreCommandLineOptions.parseOptions(
// allOptions,
// commandLine);
// if (dataStoreOption.isCommandLineChange()) {
// commandLine = dataStoreOption.getCommandLine();
// }
// else {
// throw new ParseException(
// "Unable to parse data store from command line");
// }
// final DataStore dataStore = dataStoreOption.getResult().createStore();
// final String typeName = commandLine.getOptionValue("typename");
// final GeometryDataSetGenerator dataGenerator = new
// GeometryDataSetGenerator(
// new FeatureCentroidDistanceFn(),
// getBuilder(typeName));
// dataGenerator.writeToGeoWave(
// dataStore,
// dataGenerator.generatePointSet(
// 0.2,
// 0.2,
// 5,
// 5000,
// new double[] {
// -100,
// -45
// },
// new double[] {
// -90,
// -35
// }));
// dataGenerator.writeToGeoWave(
// dataStore,
// dataGenerator.generatePointSet(
// 0.2,
// 0.2,
// 7,
// 5000,
// new double[] {
// 0,
// 0
// },
// new double[] {
// 10,
// 10
// }));
// dataGenerator.writeToGeoWave(
// dataStore,
// dataGenerator.addRandomNoisePoints(
// dataGenerator.generatePointSet(
// 0.2,
// 0.2,
// 6,
// 5000,
// new double[] {
// 65,
// 35
// },
// new double[] {
// 75,
// 45
// }),
// 6000,
// new double[] {
// -90,
// -90
// },
// new double[] {
// 90,
// 90
// }));
// }
private static SimpleFeatureBuilder getBuilder(final String name) throws FactoryException {
final SimpleFeatureTypeBuilder typeBuilder = new SimpleFeatureTypeBuilder();
typeBuilder.setName(name);
typeBuilder.setCRS(CRS.decode("EPSG:4326", true)); // <- Coordinate
// reference
// add attributes in order
typeBuilder.add("geom", Geometry.class);
typeBuilder.add("name", String.class);
typeBuilder.add("count", Long.class);
// build the type
return new SimpleFeatureBuilder(typeBuilder.buildFeatureType());
}
public static class CurvedDensityDataGeneratorTool {
private static final CoordinateCircleDistanceFn DISTANCE_FN = new CoordinateCircleDistanceFn();
private CurvedDensityDataGeneratorTool() {}
public static final List generatePoints(
final LineString line,
final double distanceFactor,
final int points) {
final List results = new ArrayList<>();
Coordinate lastCoor = null;
double distanceTotal = 0.0;
final double[] distancesBetweenCoords = new double[line.getCoordinates().length - 1];
int i = 0;
for (final Coordinate coor : line.getCoordinates()) {
if (lastCoor != null) {
distancesBetweenCoords[i] = Math.abs(DISTANCE_FN.measure(lastCoor, coor));
distanceTotal += distancesBetweenCoords[i++];
}
lastCoor = coor;
}
lastCoor = null;
i = 0;
for (final Coordinate coor : line.getCoordinates()) {
if (lastCoor != null) {
results.addAll(
generatePoints(
line.getFactory(),
toVec(coor),
toVec(lastCoor),
distanceFactor,
(int) ((points) * (distancesBetweenCoords[i++] / distanceTotal))));
}
lastCoor = coor;
}
return results;
}
private static final List generatePoints(
final GeometryFactory factory,
final Vector2D coordinateOne,
final Vector2D coordinateTwo,
final double distanceFactor,
final int points) {
final List results = new ArrayList<>();
final Random rand = new Random();
final Vector2D originVec = coordinateTwo.subtract(coordinateOne);
for (int i = 0; i < points; i++) {
// HP Fortify "Insecure Randomness" false positive
// This random number is not used for any purpose
// related to security or cryptography
final double factor = rand.nextDouble();
final Vector2D projectionPoint = originVec.scalarMultiply(factor);
final double direction = rand.nextGaussian() * distanceFactor;
final Vector2D orthogonal = new Vector2D(originVec.getY(), -originVec.getX());
results.add(
factory.createPoint(
toCoordinate(
orthogonal.scalarMultiply(direction).add(projectionPoint).add(coordinateOne))));
}
return results;
}
public static Coordinate toCoordinate(final Vector2D vec) {
return new Coordinate(vec.getX(), vec.getY());
}
public static Vector2D toVec(final Coordinate coor) {
return new Vector2D(coor.x, coor.y);
}
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/GeometryHullTool.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Set;
import java.util.TreeSet;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.commons.math.util.MathUtils;
import org.apache.commons.math3.geometry.Vector;
import org.apache.commons.math3.geometry.euclidean.twod.Euclidean2D;
import org.apache.commons.math3.geometry.euclidean.twod.Vector2D;
import org.locationtech.geowave.analytic.clustering.NeighborData;
import org.locationtech.geowave.analytic.distance.DistanceFn;
import org.locationtech.geowave.core.index.FloatCompareUtils;
import org.locationtech.jts.algorithm.CGAlgorithms;
import org.locationtech.jts.algorithm.ConvexHull;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.GeometryFactory;
import org.locationtech.jts.geom.Polygon;
import org.locationtech.jts.operation.union.UnaryUnionOp;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** Set of algorithms to mere hulls and increase the gradient of convexity over hulls. */
public class GeometryHullTool {
protected static final Logger LOGGER = LoggerFactory.getLogger(GeometryHullTool.class);
DistanceFn distanceFnForCoordinate;
double concaveThreshold = 1.8;
public void connect(final List geometries) {}
public DistanceFn getDistanceFnForCoordinate() {
return distanceFnForCoordinate;
}
public void setDistanceFnForCoordinate(final DistanceFn distanceFnForCoordinate) {
this.distanceFnForCoordinate = distanceFnForCoordinate;
}
protected double getConcaveThreshold() {
return concaveThreshold;
}
/*
* Set the threshold for the concave algorithm
*/
protected void setConcaveThreshold(final double concaveThreshold) {
this.concaveThreshold = concaveThreshold;
}
protected static class Edge implements Comparable {
Coordinate start;
Coordinate end;
double distance;
Edge next, last;
private TreeSet> points = null;
public Edge(final Coordinate start, final Coordinate end, final double distance) {
super();
this.start = start;
this.end = end;
this.distance = distance;
}
public TreeSet> getPoints() {
if (points == null) {
points = new TreeSet<>();
}
return points;
}
@Override
public int compareTo(final Edge edge) {
return (distance - edge.distance) > 0 ? 1 : -1;
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = (prime * result) + ((end == null) ? 0 : end.hashCode());
result = (prime * result) + ((start == null) ? 0 : start.hashCode());
return result;
}
public void connectLast(final Edge last) {
this.last = last;
last.next = this;
}
@Override
public boolean equals(final Object obj) {
if (this == obj) {
return true;
}
if (obj == null) {
return false;
}
if (getClass() != obj.getClass()) {
return false;
}
final Edge other = (Edge) obj;
if (end == null) {
if (other.end != null) {
return false;
}
} else if (!end.equals(other.end)) {
return false;
}
if (start == null) {
if (other.start != null) {
return false;
}
} else if (!start.equals(other.start)) {
return false;
}
return true;
}
@Override
public String toString() {
return "Edge [start=" + start + ", end=" + end + ", distance=" + distance + "]";
}
}
private Edge createEdgeWithSideEffects(
final Coordinate start,
final Coordinate end,
final Set innerPoints,
final TreeSet edges) {
final Edge newEdge = new Edge(start, end, distanceFnForCoordinate.measure(start, end));
innerPoints.remove(newEdge.start);
innerPoints.remove(newEdge.end);
edges.add(newEdge);
return newEdge;
}
/*
* Generate a concave hull, if possible, given a geometry and a set of additional points.
*
* @param fast expedite processing allowing for some outliers.
*/
public Geometry createHullFromGeometry(
final Geometry clusterGeometry,
final Collection additionalPoints,
final boolean fast) {
if (additionalPoints.isEmpty()) {
return clusterGeometry;
}
final Set batchCoords = new HashSet<>();
if (clusterGeometry != null) {
for (final Coordinate coordinate : clusterGeometry.getCoordinates()) {
batchCoords.add(coordinate);
}
}
for (final Coordinate coordinate : additionalPoints) {
batchCoords.add(coordinate);
}
final GeometryFactory factory =
clusterGeometry == null ? new GeometryFactory() : clusterGeometry.getFactory();
final Coordinate[] actualCoords = batchCoords.toArray(new Coordinate[batchCoords.size()]);
if (batchCoords.size() == 2) {
return factory.createLineString(actualCoords);
}
final ConvexHull convexHull = new ConvexHull(actualCoords, factory);
final Geometry convexHullGeo = convexHull.getConvexHull();
try {
// does this shape benefit from concave hulling?
// it cannot be a line string
if ((batchCoords.size() > 5) && (convexHullGeo.getArea() > 0.0)) {
final Geometry concaveHull =
fast ? concaveHull(convexHullGeo, batchCoords)
: concaveHullParkOhMethod(convexHullGeo, batchCoords);
if (fast && !concaveHull.isSimple()) {
LOGGER.warn("Produced non simple hull", concaveHull.toText());
return concaveHullParkOhMethod(convexHullGeo, batchCoords);
}
return concaveHull;
} else {
return convexHullGeo;
}
} catch (final Exception ex) {
/*
* Geometry[] points = new Geometry[actualCoords.length + 1]; for (int i = 0; i <
* actualCoords.length; i++) points[i] = hull.getFactory().createPoint( actualCoords[i]);
* points[points.length - 1] = hull; try { ShapefileTool.writeShape( "test_perf_xh", new File(
* "./targettest_perf_xh"), points); } catch (IOException e) { e.printStackTrace(); }
*/
LOGGER.error("Failed to compute hull", ex);
return convexHullGeo;
}
}
/**
* Gift unwrapping (e.g. dig) concept, taking a convex hull and a set of inner points, add inner
* points to the hull without violating hull invariants--all points must reside on the hull or
* inside the hull. Based on: Jin-Seo Park and Se-Jong Oh. "A New Concave Algorithm and
* Concaveness Measure for n-dimensional Datasets" . Department of Nanobiomedical Science. Dankook
* University". 2010.
*
*
Per the paper, N = concaveThreshold
*/
public Geometry concaveHullParkOhMethod(
final Geometry geometry,
final Collection providedInnerPoints) {
final Set innerPoints = new HashSet<>(providedInnerPoints);
final TreeSet edges = new TreeSet<>();
final Coordinate[] geoCoordinateList = geometry.getCoordinates();
final int s = geoCoordinateList.length - 1;
final Edge firstEdge =
createEdgeWithSideEffects(geoCoordinateList[0], geoCoordinateList[1], innerPoints, edges);
Edge lastEdge = firstEdge;
for (int i = 1; i < s; i++) {
final Edge newEdge =
createEdgeWithSideEffects(
geoCoordinateList[i],
geoCoordinateList[i + 1],
innerPoints,
edges);
newEdge.connectLast(lastEdge);
lastEdge = newEdge;
}
firstEdge.connectLast(lastEdge);
while (!edges.isEmpty() && !innerPoints.isEmpty()) {
final Edge edge = edges.pollLast();
lastEdge = edge;
double score = Double.MAX_VALUE;
Coordinate selectedCandidate = null;
for (final Coordinate candidate : innerPoints) {
final double dist = calcDistance(edge.start, edge.end, candidate);
// on the hull
if (MathUtils.equals(dist, 0.0, 0.000000001)) {
score = 0.0;
selectedCandidate = candidate;
break;
}
if ((dist > 0) && (dist < score)) {
score = dist;
selectedCandidate = candidate;
}
}
if (selectedCandidate == null) {
continue;
}
// if one a line segment of the hull, then remove candidate
if (FloatCompareUtils.checkDoublesEqual(score, 0.0)) {
innerPoints.remove(selectedCandidate);
edges.add(edge);
continue;
}
// Park and Oh look only at the neighbor edges
// but this fails in some cases.
if (isCandidateCloserToAnotherEdge(score, edge, edges, selectedCandidate)) {
continue;
}
innerPoints.remove(selectedCandidate);
final double eh = edge.distance;
final double startToCandidate =
distanceFnForCoordinate.measure(edge.start, selectedCandidate);
final double endToCandidate = distanceFnForCoordinate.measure(edge.end, selectedCandidate);
final double min = Math.min(startToCandidate, endToCandidate);
// protected against duplicates
if ((eh / min) > concaveThreshold) {
final Edge newEdge1 = new Edge(edge.start, selectedCandidate, startToCandidate);
final Edge newEdge2 = new Edge(selectedCandidate, edge.end, endToCandidate);
// need to replace this with something more intelligent. This
// occurs in cases of sharp angles. An angular approach may also
// work
// look for an angle to flip in the reverse direction.
if (!intersectAnotherEdge(newEdge1, edge)
&& !intersectAnotherEdge(newEdge2, edge)
&& !intersectAnotherEdge(newEdge1, edge.last)
&& !intersectAnotherEdge(newEdge2, edge.next)) {
edges.add(newEdge2);
edges.add(newEdge1);
newEdge1.connectLast(edge.last);
newEdge2.connectLast(newEdge1);
edge.next.connectLast(newEdge2);
lastEdge = newEdge1;
}
}
}
return geometry.getFactory().createPolygon(reassemble(lastEdge));
}
/**
* Gift unwrapping (e.g. dig) concept, taking a convex hull and a set of inner points, add inner
* points to the hull without violating hull invariants--all points must reside on the hull or
* inside the hull. Based on: Jin-Seo Park and Se-Jong Oh. "A New Concave Algorithm and
* Concaveness Measure for n-dimensional Datasets" . Department of Nanobiomedical Science. Dankook
* University". 2010.
*
*
Per the paper, N = concaveThreshold.
*
*
This algorithm evaluates remarkably faster than Park and Oh, but the quality of the result
* is marginally less. If it is acceptable to have some small number of points fall outside of the
* hull and speed is critical, use this method. The measure of error is difficult to calculate
* since it is not directly calculated based on the number of inner points. Rather, the measure is
* based on some number of points in proximity the optimal concave hull.
*/
public Geometry concaveHull(
final Geometry geometry,
final Collection providedInnerPoints) {
final Set innerPoints =
(providedInnerPoints instanceof Set) ? (Set) providedInnerPoints
: new HashSet<>(providedInnerPoints);
final TreeSet edges = new TreeSet<>();
final Coordinate[] geoCoordinateList = geometry.getCoordinates();
final int s = geoCoordinateList.length - 1;
final Edge firstEdge =
createEdgeWithSideEffects(geoCoordinateList[0], geoCoordinateList[1], innerPoints, edges);
Edge lastEdge = firstEdge;
for (int i = 1; i < s; i++) {
final Edge newEdge =
createEdgeWithSideEffects(
geoCoordinateList[i],
geoCoordinateList[i + 1],
innerPoints,
edges);
newEdge.connectLast(lastEdge);
lastEdge = newEdge;
}
firstEdge.connectLast(lastEdge);
for (final Coordinate candidate : innerPoints) {
double min = Double.MAX_VALUE;
Edge bestEdge = null;
for (final Edge edge : edges) {
final double dist = calcDistance(edge.start, edge.end, candidate);
if ((dist > 0) && (dist < min)) {
min = dist;
bestEdge = edge;
}
}
if (bestEdge != null) {
bestEdge.getPoints().add(new NeighborData<>(candidate, null, min));
}
}
while (!edges.isEmpty()) {
final Edge edge = edges.pollLast();
lastEdge = edge;
NeighborData candidate = edge.getPoints().pollFirst();
while (candidate != null) {
if (!MathUtils.equals(candidate.getDistance(), 0.0, 0.000000001)) {
final Coordinate selectedCandidate = candidate.getElement();
final double eh = edge.distance;
final double startToCandidate =
distanceFnForCoordinate.measure(edge.start, selectedCandidate);
final double endToCandidate =
distanceFnForCoordinate.measure(edge.end, selectedCandidate);
final double min = Math.min(startToCandidate, endToCandidate);
// protected against duplicates
if ((eh / min) > concaveThreshold) {
final Edge newEdge1 = new Edge(edge.start, selectedCandidate, startToCandidate);
final Edge newEdge2 = new Edge(selectedCandidate, edge.end, endToCandidate);
edges.add(newEdge2);
edges.add(newEdge1);
newEdge1.connectLast(edge.last);
newEdge2.connectLast(newEdge1);
edge.next.connectLast(newEdge2);
lastEdge = newEdge1;
for (final NeighborData otherPoint : edge.getPoints()) {
final double[] distProfile1 =
calcDistanceSegment(newEdge1.start, newEdge1.end, otherPoint.getElement());
final double[] distProfile2 =
calcDistanceSegment(newEdge2.start, newEdge2.end, otherPoint.getElement());
if ((distProfile1[0] >= 0.0) && (distProfile1[0] <= 1.0)) {
if ((distProfile1[0] < 0.0)
|| (distProfile1[0] > 1.0)
|| (distProfile2[1] > distProfile1[1])) {
otherPoint.setDistance(distProfile1[1]);
newEdge1.getPoints().add(otherPoint);
} else {
otherPoint.setDistance(distProfile2[1]);
newEdge2.getPoints().add(otherPoint);
}
} else if ((distProfile2[0] >= 0.0) && (distProfile2[0] <= 1.0)) {
otherPoint.setDistance(distProfile2[1]);
newEdge2.getPoints().add(otherPoint);
}
}
edge.getPoints().clear(); // forces this loop to end
}
}
candidate = edge.getPoints().pollFirst();
}
}
return geometry.getFactory().createPolygon(reassemble(lastEdge));
}
public static boolean intersectAnotherEdge(final Edge newEdge, final Edge edgeToReplace) {
Edge nextEdge = edgeToReplace.next.next;
final Edge stopEdge = edgeToReplace.last;
while (nextEdge != stopEdge) {
if (edgesIntersect(newEdge, nextEdge)) {
return true;
}
nextEdge = nextEdge.next;
}
return false;
}
public static boolean edgesIntersect(final Edge e1, final Edge e2) {
return CGAlgorithms.distanceLineLine(e1.start, e1.end, e2.start, e2.end) <= 0.0;
}
private static boolean isCandidateCloserToAnotherEdge(
final double distanceToBeat,
final Edge selectedEdgeToBeat,
final Collection edges,
final Coordinate selectedCandidate) {
for (final Edge edge : edges) {
if (selectedEdgeToBeat.equals(edge)) {
continue;
}
final double dist = calcDistance(edge.start, edge.end, selectedCandidate);
if ((dist >= 0.0) && (dist < distanceToBeat)) {
return true;
}
}
return false;
}
private static Coordinate[] reassemble(final Edge lastEdge) {
final List coordinates = new ArrayList<>();
coordinates.add(lastEdge.start);
Edge nextEdge = lastEdge.next;
while (nextEdge != lastEdge) {
coordinates.add(nextEdge.start);
nextEdge = nextEdge.next;
}
coordinates.add(lastEdge.start);
return coordinates.toArray(new Coordinate[coordinates.size()]);
}
protected boolean isInside(final Coordinate coor, final Coordinate[] hullCoordinates) {
double maxAngle = 0;
for (int i = 1; i < hullCoordinates.length; i++) {
final Coordinate hullCoordinate = hullCoordinates[i];
maxAngle = Math.max(calcAngle(hullCoordinates[0], coor, hullCoordinate), maxAngle);
}
// return 360 == Math.abs(maxAngle);
return ((Math.abs(maxAngle) >= 359.999) && (Math.abs(maxAngle) <= 360.0001));
}
/**
* Forms create edges between two shapes maintaining convexity.
*
*
Does not currently work if the shapes intersect
*/
public Geometry connect(final Geometry shape1, final Geometry shape2) {
try {
if ((shape1 instanceof Polygon)
&& (shape2 instanceof Polygon)
&& !shape1.intersects(shape2)) {
return connect(shape1, shape2, getClosestPoints(shape1, shape2, distanceFnForCoordinate));
}
return UnaryUnionOp.union(Arrays.asList(shape1, shape2));
} catch (final Exception ex) {
LOGGER.warn("Exception caught in connect method", ex);
}
return createHullFromGeometry(shape1, Arrays.asList(shape2.getCoordinates()), false);
}
protected Geometry connect(
final Geometry shape1,
final Geometry shape2,
final Pair closestCoordinates) {
Coordinate[] leftCoords = shape1.getCoordinates(), rightCoords = shape2.getCoordinates();
int startLeft, startRight;
if ((leftCoords[closestCoordinates.getLeft()].x < rightCoords[closestCoordinates.getRight()].x)) {
startLeft = closestCoordinates.getLeft();
startRight = closestCoordinates.getRight();
} else {
leftCoords = shape2.getCoordinates();
rightCoords = shape1.getCoordinates();
startLeft = closestCoordinates.getRight();
startRight = closestCoordinates.getLeft();
}
final HashSet visitedSet = new HashSet<>();
visitedSet.add(leftCoords[startLeft]);
visitedSet.add(rightCoords[startRight]);
final boolean leftClockwise = clockwise(leftCoords);
final boolean rightClockwise = clockwise(rightCoords);
final Pair upperCoords =
walk(visitedSet, leftCoords, rightCoords, startLeft, startRight, new DirectionFactory() {
@Override
public Direction createLeftFootDirection(final int start, final int max) {
return leftClockwise ? new IncreaseDirection(start, max, true)
: new DecreaseDirection(start, max, true);
}
@Override
public Direction createRightFootDirection(final int start, final int max) {
return rightClockwise ? new DecreaseDirection(start, max, false)
: new IncreaseDirection(start, max, false);
}
});
final Pair lowerCoords =
walk(visitedSet, leftCoords, rightCoords, startLeft, startRight, new DirectionFactory() {
@Override
public Direction createLeftFootDirection(final int start, final int max) {
return leftClockwise ? new DecreaseDirection(start, max, false)
: new IncreaseDirection(start, max, false);
}
@Override
public Direction createRightFootDirection(final int start, final int max) {
return rightClockwise ? new IncreaseDirection(start, max, true)
: new DecreaseDirection(start, max, true);
}
});
final List newCoordinateSet = new ArrayList<>();
final Direction leftSet =
leftClockwise
? new IncreaseDirection(
upperCoords.getLeft(),
lowerCoords.getLeft() + 1,
leftCoords.length)
: new DecreaseDirection(
upperCoords.getLeft(),
lowerCoords.getLeft() - 1,
leftCoords.length);
newCoordinateSet.add(leftCoords[upperCoords.getLeft()]);
while (leftSet.hasNext()) {
newCoordinateSet.add(leftCoords[leftSet.next()]);
}
final Direction rightSet =
rightClockwise
? new IncreaseDirection(
lowerCoords.getRight(),
upperCoords.getRight() + 1,
rightCoords.length)
: new DecreaseDirection(
lowerCoords.getRight(),
upperCoords.getRight() - 1,
rightCoords.length);
newCoordinateSet.add(rightCoords[lowerCoords.getRight()]);
while (rightSet.hasNext()) {
newCoordinateSet.add(rightCoords[rightSet.next()]);
}
newCoordinateSet.add(leftCoords[upperCoords.getLeft()]);
return shape1.getFactory().createPolygon(
newCoordinateSet.toArray(new Coordinate[newCoordinateSet.size()]));
}
private Pair walk(
final Set visited,
final Coordinate[] shape1Coords,
final Coordinate[] shape2Coords,
final int start1,
final int start2,
final DirectionFactory factory) {
final int upPos =
takeBiggestStep(
visited,
shape2Coords[start2],
shape1Coords,
factory.createLeftFootDirection(start1, shape1Coords.length));
// even if the left foot was stationary, try to move the right foot
final int downPos =
takeBiggestStep(
visited,
shape1Coords[upPos],
shape2Coords,
factory.createRightFootDirection(start2, shape2Coords.length));
// if the right step moved, then see if another l/r step can be taken
if (downPos != start2) {
return walk(visited, shape1Coords, shape2Coords, upPos, downPos, factory);
}
return Pair.of(upPos, start2);
}
/**
* Determine if the polygon is defined clockwise
*/
public static boolean clockwise(final Coordinate[] set) {
double sum = 0.0;
for (int i = 1; i < set.length; i++) {
sum += (set[i].x - set[i - 1].x) / (set[i].y + set[i - 1].y);
}
return sum > 0.0;
}
public static double calcSmallestAngle(
final Coordinate one,
final Coordinate vertex,
final Coordinate two) {
final double angle = Math.abs(calcAngle(one, vertex, two));
return (angle > 180.0) ? angle - 180.0 : angle;
}
/**
* Calculate the angle between two points and a given vertex
*/
public static double calcAngle(
final Coordinate one,
final Coordinate vertex,
final Coordinate two) {
final double p1x = one.x - vertex.x;
final double p1y = one.y - vertex.y;
final double p2x = two.x - vertex.x;
final double p2y = two.y - vertex.y;
final double angle1 = Math.toDegrees(Math.atan2(p1y, p1x));
final double angle2 = Math.toDegrees(Math.atan2(p2y, p2x));
return angle2 - angle1;
}
/**
* Calculate the distance between two points and a given vertex
*
* @return array if doubles double[0] = length of the projection from start on the line containing
* the segment(start to end) double[1] = distance to the segment double[2] = distance to
* the line containing the segment(start to end)
*/
public static double[] calcDistanceSegment(
final Coordinate start,
final Coordinate end,
final Coordinate point) {
final Vector vOne = new Vector2D(start.x, start.y);
final Vector vTwo = new Vector2D(end.x, end.y);
final Vector vVertex = new Vector2D(point.x, point.y);
final Vector E1 = vTwo.subtract(vOne);
final Vector E2 = vVertex.subtract(vOne);
final double distOneTwo = E2.dotProduct(E1);
final double lengthVOneSq = E1.getNormSq();
final double projectionLength = distOneTwo / lengthVOneSq;
final Vector projection = E1.scalarMultiply(projectionLength).add(vOne);
final double o =
((projectionLength < 0.0) ? vOne.distance(vVertex)
: ((projectionLength > 1.0) ? vTwo.distance(vVertex) : vVertex.distance(projection)));
return new double[] {projectionLength, o, vVertex.distance(projection)};
}
public static double calcDistance(
final Coordinate start,
final Coordinate end,
final Coordinate point) {
final double[] p = calcDistanceSegment(start, end, point);
return ((p[0] < 0.0) || (p[0] > 1.0)) ? -1 : p[1];
}
public static Pair getClosestPoints(
final Geometry shape1,
final Geometry shape2,
final DistanceFn distanceFnForCoordinate) {
int bestShape1Position = 0;
int bestShape2Position = 0;
double minDist = Double.MAX_VALUE;
int pos1 = 0, pos2 = 0;
for (final Coordinate coord1 : shape1.getCoordinates()) {
pos2 = 0;
for (final Coordinate coord2 : shape2.getCoordinates()) {
final double dist = (distanceFnForCoordinate.measure(coord1, coord2));
if (dist < minDist) {
bestShape1Position = pos1;
bestShape2Position = pos2;
minDist = dist;
}
pos2++;
}
pos1++;
}
return Pair.of(bestShape1Position, bestShape2Position);
}
private int takeBiggestStep(
final Set visited,
final Coordinate station,
final Coordinate[] shapeCoords,
final Direction legIncrement) {
double angle = 0.0;
final Coordinate startPoint = shapeCoords[legIncrement.getStart()];
int last = legIncrement.getStart();
Coordinate lastCoordinate = shapeCoords[last];
while (legIncrement.hasNext()) {
final int pos = legIncrement.next();
// skip over duplicate (a ring or polygon has one duplicate)
if (shapeCoords[pos].equals(lastCoordinate)) {
continue;
}
lastCoordinate = shapeCoords[pos];
if (visited.contains(lastCoordinate)) {
break;
}
double currentAngle =
legIncrement.angleChange(calcAngle(startPoint, station, lastCoordinate));
currentAngle = currentAngle < -180 ? currentAngle + 360 : currentAngle;
if ((currentAngle >= angle) && (currentAngle < 180.0)) {
angle = currentAngle;
last = pos;
visited.add(shapeCoords[pos]);
} else {
return last;
}
}
return last;
}
private interface DirectionFactory {
Direction createLeftFootDirection(int start, int max);
Direction createRightFootDirection(int start, int max);
}
private interface Direction extends Iterator {
public int getStart();
public double angleChange(double angle);
}
private class IncreaseDirection implements Direction {
final int max;
final int start;
final int stop;
int current = 0;
final boolean angleIsNegative;
@Override
public int getStart() {
return start;
}
public IncreaseDirection(final int start, final int max, final boolean angleIsNegative) {
super();
this.max = max;
current = getNext(start);
stop = start;
this.start = start;
this.angleIsNegative = angleIsNegative;
}
public IncreaseDirection(final int start, final int stop, final int max) {
super();
this.max = max;
current = getNext(start);
this.stop = stop;
this.start = start;
angleIsNegative = true;
}
@Override
public Integer next() {
final int n = current;
current = getNext(current);
return n;
}
@Override
public boolean hasNext() {
return current != stop;
}
protected int getNext(final int n) {
return (n + 1) % max;
}
@Override
public void remove() {}
@Override
public double angleChange(final double angle) {
return angleIsNegative ? -angle : angle;
}
}
private class DecreaseDirection extends IncreaseDirection implements Direction {
public DecreaseDirection(final int start, final int max, final boolean angleIsNegative) {
super(start, max, angleIsNegative);
}
public DecreaseDirection(final int start, final int stop, final int max) {
super(start, stop, max);
}
@Override
protected int getNext(final int n) {
return (n == 0) ? max - 1 : n - 1;
}
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/IndependentJobRunner.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.util.Collection;
import org.locationtech.geowave.analytic.param.ParameterEnum;
public interface IndependentJobRunner {
public int run(PropertyManagement properties) throws Exception;
public Collection> getParameters();
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/Projection.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.JobContext;
import org.locationtech.jts.geom.Geometry;
/**
* Project a n-dimensional item into a two-dimensional polygon for convex hull construction.
*
* @param
*/
public interface Projection {
public Geometry getProjection(T anItem);
public void initialize(JobContext context, Class scope) throws IOException;
public void setup(
PropertyManagement runTimeProperties,
Class scope,
Configuration configuration);
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/PropertyManagement.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.Serializable;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.ParseException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.locationtech.geowave.analytic.param.ParameterEnum;
import org.locationtech.geowave.core.index.ByteArrayUtils;
import org.locationtech.geowave.core.index.numeric.NumericRange;
import org.locationtech.geowave.core.index.persist.Persistable;
import org.locationtech.geowave.core.index.persist.PersistenceUtils;
import org.locationtech.geowave.core.store.api.Query;
import org.locationtech.geowave.core.store.api.QueryBuilder;
import org.locationtech.geowave.core.store.query.constraints.QueryConstraints;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Manage properties used by the Map Reduce environment that are provided through the API (e.g.
* command). Allow these arguments to be placed an 'args' list for 'main' executables (e.g.
* ToolRunner).
*
*
The class supports some basic conversions.
*
*
Non-serializable objects: {@link Persistable} instances are converted to and from byte
* formats. {@link QueryConstraints} is a special case, supporting WKT String. {@link Path} are
* converted to a from string representation of the their URI.
*
*
NOTE: ConfigutationWrapper implementation is scopeless.
*
*
EXPECTED FUTURE WORK: I am bit unsatisfied with the duality of the parameters base class. In
* one case, in is treated a description for a class value and, in the other case, it is treated as
* a description for the type of a property value. The former is really a descriptor of a Class of
* type class. Generics do not help due to erasure. The impact of this inconsistency is the
* inability to validate on 'store'. Instead, validation occurs on 'gets'. The ultimate goal is to
* uniformly provide feedback to parameters from command line arguments and property files on
* submission to the manager rather than on extraction from the manager.
*/
public class PropertyManagement implements Serializable {
/** */
private static final long serialVersionUID = -4186468044516636362L;
static final Logger LOGGER = LoggerFactory.getLogger(PropertyManagement.class);
private final Map, Serializable> localProperties = new HashMap<>();
private final List> converters = new ArrayList<>();
private PropertyManagement nestProperties = null;
public PropertyManagement() {
converters.add(new QueryConverter());
converters.add(new PathConverter());
converters.add(new PersistableConverter());
converters.add(new DoubleConverter());
converters.add(new IntegerConverter());
converters.add(new ByteConverter());
}
public PropertyManagement(
final PropertyConverter[] converters,
final ParameterEnum[] names,
final Object[] values) {
this.converters.add(new QueryConverter());
this.converters.add(new PathConverter());
this.converters.add(new PersistableConverter());
this.converters.add(new DoubleConverter());
this.converters.add(new IntegerConverter());
this.converters.add(new ByteConverter());
for (final PropertyConverter converter : converters) {
addConverter(converter);
}
storeAll(names, values);
}
public PropertyManagement(final ParameterEnum[] names, final Object[] values) {
converters.add(new QueryConverter());
converters.add(new PathConverter());
converters.add(new PersistableConverter());
converters.add(new DoubleConverter());
converters.add(new IntegerConverter());
converters.add(new ByteConverter());
storeAll(names, values);
}
public PropertyManagement(final PropertyManagement pm) {
nestProperties = pm;
converters.addAll(pm.converters);
}
public Serializable get(final ParameterEnum propertyName) {
return getPropertyValue(propertyName);
}
public synchronized void store(
final ParameterEnum property,
final T value,
final PropertyConverter converter) {
Serializable convertedValue;
try {
convertedValue = converter.convert(value);
} catch (final Exception e) {
throw new IllegalArgumentException(
String.format(
"Cannot store %s with value %s. Expected type = %s; Error message = %s",
property.self().toString(),
value.toString(),
property.getHelper().getBaseClass().toString(),
e.getLocalizedMessage()),
e);
}
localProperties.put(property, convertedValue);
addConverter(converter);
}
public synchronized void store(final ParameterEnum property, final Object value) {
if (value != null) {
Serializable convertedValue;
try {
convertedValue = convertIfNecessary(property, value);
} catch (final Exception e) {
throw new IllegalArgumentException(
String.format(
"Cannot store %s with value %s:%s",
property.self().toString(),
value.toString(),
e.getLocalizedMessage()));
}
localProperties.put(property, convertedValue);
}
}
/** Does not work for non-serializable data (e.g. Path or Persistable) */
public synchronized Serializable storeIfEmpty(
final ParameterEnum propertyEnum,
final Serializable value) {
if (!containsPropertyValue(propertyEnum) && (value != null)) {
LOGGER.info("Setting parameter : {} to {}", propertyEnum.toString(), value.toString());
store(propertyEnum, value);
return value;
}
return getPropertyValue(propertyEnum);
}
public synchronized void copy(
final ParameterEnum propertyNameFrom,
final ParameterEnum propertyNameTo) {
if (containsPropertyValue(propertyNameFrom)) {
localProperties.put(propertyNameTo, getPropertyValue(propertyNameFrom));
}
}
public synchronized void storeAll(final ParameterEnum[] names, final Object[] values) {
if (values.length != names.length) {
LOGGER.error(
"The number of values must equal the number of names passed to the store method");
throw new IllegalArgumentException(
"The number of values must equal the number of names passed to the store method");
}
int i = 0;
for (final Object value : values) {
store(names[i++], value);
}
}
public void setConfig(
final ParameterEnum[] parameters,
final Configuration config,
final Class scope) {
for (final ParameterEnum param : parameters) {
Object value;
try {
value = getProperty(param);
param.getHelper().setValue(config, scope, value);
} catch (final Exception e) {
LOGGER.error("Property " + param.self().toString() + " is not available", e);
throw new IllegalArgumentException(
"Property " + param.self().toString() + " is not available",
e);
}
}
}
@SuppressWarnings("unchecked")
public T getClassInstance(
final ParameterEnum property,
final Class iface,
final Class defaultClass) throws InstantiationException {
final Object o = getPropertyValue(property);
try {
final Class clazz =
o == null ? defaultClass
: (o instanceof Class) ? (Class) o : Class.forName(o.toString());
if (!property.getHelper().getBaseClass().isAssignableFrom(clazz)) {
LOGGER.error(
"Class for property "
+ property.self().toString()
+ " does not implement "
+ property.getHelper().getBaseClass().toString());
}
return (T) clazz.newInstance();
} catch (final ClassNotFoundException e) {
LOGGER.error("Class for property " + property.self().toString() + " is not found", e);
throw new InstantiationException(property.self().toString());
} catch (final InstantiationException e) {
LOGGER.error("Class for property " + property.self().toString() + " is not instiatable", e);
throw new InstantiationException(property.self().toString());
} catch (final IllegalAccessException e) {
LOGGER.error("Class for property " + property.self().toString() + " is not accessible", e);
throw new InstantiationException(property.self().toString());
}
}
public synchronized boolean hasProperty(final ParameterEnum property) {
return containsPropertyValue(property);
}
public String getPropertyAsString(final ParameterEnum property) {
return getPropertyAsString(property, null);
}
/**
* Returns the value as, without conversion from the properties. Throws an exception if a
* conversion is required to a specific type
*/
public Object getProperty(final ParameterEnum property) throws Exception {
final Serializable value = getPropertyValue(property);
if (!Serializable.class.isAssignableFrom(property.getHelper().getBaseClass())) {
for (final PropertyConverter converter : converters) {
if (converter.baseClass().isAssignableFrom(property.getHelper().getBaseClass())) {
return this.validate(property, converter.convert(value));
}
}
}
return this.validate(property, value);
}
/**
* Returns the value after conversion. Throws an exception if a conversion fails.
*/
public T getProperty(final ParameterEnum property, final PropertyConverter converter)
throws Exception {
final Serializable value = getPropertyValue(property);
return converter.convert(value);
}
public byte[] getPropertyAsBytes(final ParameterEnum property) {
final Object val = getPropertyValue(property);
if (val != null) {
if (val instanceof byte[]) {
return (byte[]) val;
}
return ByteArrayUtils.byteArrayFromString(val.toString());
}
return null;
}
public String getPropertyAsString(final ParameterEnum property, final String defaultValue) {
// not using containsKey to avoid synchronization
final Object value = getPropertyValue(property);
return (String) validate(property, value == null ? defaultValue : value.toString());
}
public Boolean getPropertyAsBoolean(final ParameterEnum property, final Boolean defaultValue) {
final Object val = getPropertyValue(property);
if (val != null) {
return Boolean.valueOf(val.toString());
}
LOGGER.warn("Using default value for parameter : " + property.self().toString());
return defaultValue;
}
public Integer getPropertyAsInt(final ParameterEnum property, final int defaultValue) {
final Object val = getPropertyValue(property);
if (val != null) {
if (val instanceof Integer) {
return (Integer) val;
}
return (Integer) validate(property, Integer.parseInt(val.toString()));
}
LOGGER.warn("Using default value for parameter : " + property.self().toString());
return defaultValue;
}
public Double getPropertyAsDouble(final ParameterEnum property, final double defaultValue) {
final Object val = getPropertyValue(property);
if (val != null) {
if (val instanceof Double) {
return (Double) val;
}
return Double.parseDouble(val.toString());
}
LOGGER.warn("Using default value for parameter : " + property.self().toString());
return defaultValue;
}
public NumericRange getPropertyAsRange(
final ParameterEnum property,
final NumericRange defaultValue) {
final Object val = getPropertyValue(property);
if (val != null) {
if (val instanceof NumericRange) {
return (NumericRange) val;
}
final String p = val.toString();
final String[] parts = p.split(",");
try {
if (parts.length == 2) {
return new NumericRange(
Double.parseDouble(parts[0].trim()),
Double.parseDouble(parts[1].trim()));
} else {
return new NumericRange(0, Double.parseDouble(p));
}
} catch (final Exception ex) {
LOGGER.error("Invalid range parameter " + property.self().toString(), ex);
return defaultValue;
}
}
LOGGER.warn("Using default value for parameter : " + property.self().toString());
return defaultValue;
}
public Class getPropertyAsClass(final ParameterEnum property) {
final Object val = getPropertyValue(property);
if (val != null) {
if (val instanceof Class) {
return validate((Class) val, property.getHelper().getBaseClass());
}
try {
return validate(
(Class) Class.forName(val.toString()),
property.getHelper().getBaseClass());
} catch (final ClassNotFoundException e) {
LOGGER.error("Class not found for property " + property, e);
} catch (final java.lang.IllegalArgumentException ex) {
LOGGER.error("Invalid class for property" + property, ex);
throw new IllegalArgumentException("Invalid class for property" + property);
}
}
return null;
}
public Class getPropertyAsClass(final ParameterEnum property, final Class iface)
throws ClassNotFoundException {
final Object val = getPropertyValue(property);
if (val != null) {
if (val instanceof Class) {
return validate((Class) val, property.getHelper().getBaseClass());
}
try {
return validate(
(Class) Class.forName(val.toString()),
property.getHelper().getBaseClass());
} catch (final ClassNotFoundException e) {
LOGGER.error("Class not found for property " + property.self().toString());
throw e;
} catch (final java.lang.IllegalArgumentException ex) {
LOGGER.error("Invalid class for property" + property.self().toString(), ex);
throw new IllegalArgumentException("Invalid class for property" + property);
}
} else {
LOGGER.error("Value not found for property " + property.self().toString());
}
throw new ClassNotFoundException("Value not found for property " + property.self().toString());
}
public Class getPropertyAsClass(
final ParameterEnum property,
final Class iface,
final Class defaultClass) {
final Object val = getPropertyValue(property);
if (val != null) {
if (val instanceof Class) {
return validate((Class) val, property.getHelper().getBaseClass());
}
try {
return validate(
(Class) Class.forName(val.toString()),
property.getHelper().getBaseClass());
} catch (final ClassNotFoundException e) {
LOGGER.error("Class not found for property " + property, e);
} catch (final java.lang.IllegalArgumentException ex) {
LOGGER.error("Invalid class for property" + property, ex);
throw new IllegalArgumentException("Invalid class for property" + property);
}
}
LOGGER.warn("Using default class for parameter : " + property.self().toString());
return defaultClass;
}
private Class validate(final Class classToValidate, final Class iface)
throws IllegalArgumentException {
if (!iface.isAssignableFrom(classToValidate)) {
throw new IllegalArgumentException(classToValidate + "is an invalid subclass of " + iface);
}
return classToValidate;
}
public Query getPropertyAsQuery(final ParameterEnum property) throws Exception {
final Serializable val = getPropertyValue(property);
if (val != null) {
return (Query) validate(property, new QueryConverter().convert(val));
}
return null;
}
public Path getPropertyAsPath(final ParameterEnum property) throws Exception {
final Serializable val = getPropertyValue(property);
if (val != null) {
return (Path) validate(property, new PathConverter().convert(val));
}
return null;
}
public Persistable getPropertyAsPersistable(final ParameterEnum property) throws Exception {
final Serializable val = getPropertyValue(property);
if (val != null) {
return (Persistable) validate(property, new PersistableConverter().convert(val));
}
return null;
}
public void setJobConfiguration(final Configuration configuration, final Class scope) {
for (final ParameterEnum param : localProperties.keySet()) {
param.getHelper().setValue(configuration, scope, param.getHelper().getValue(this));
}
if ((nestProperties != null) && !nestProperties.localProperties.isEmpty()) {
nestProperties.setJobConfiguration(configuration, scope);
}
}
public void dump() {
LOGGER.info("Properties : ");
for (final Map.Entry, Serializable> prop : localProperties.entrySet()) {
LOGGER.info("{} = {}", prop.getKey(), prop.getValue());
}
nestProperties.dump();
}
/**
* Add to the set of converters used to take a String representation of a value and convert it
* into another serializable form.
*
*
This is done if the preferred internal representation does not match that of a string. For
* example, a query is maintained as bytes even though it can be provided as a query
*
* @param converter
*/
public synchronized void addConverter(final PropertyConverter converter) {
converters.add(converter);
}
private static byte[] toBytes(final Persistable persistableObject)
throws UnsupportedEncodingException {
return PersistenceUtils.toBinary(persistableObject);
}
private static Persistable fromBytes(final byte[] data) throws InstantiationException,
IllegalAccessException, ClassNotFoundException, UnsupportedEncodingException {
return PersistenceUtils.fromBinary(data);
}
private Object validate(final ParameterEnum propertyName, final Object value) {
if (value != null) {
if (value instanceof Class) {
if (((Class) value).isAssignableFrom(propertyName.getHelper().getBaseClass())) {
throw new IllegalArgumentException(
String.format(
"%s does not accept class %s",
propertyName.self().toString(),
((Class) value).getName()));
}
} else if (!propertyName.getHelper().getBaseClass().isInstance(value)) {
throw new IllegalArgumentException(
String.format(
"%s does not accept type %s",
propertyName.self().toString(),
value.getClass().getName()));
}
}
return value;
}
@SuppressWarnings("unchecked")
private Serializable convertIfNecessary(final ParameterEnum property, final Object value)
throws Exception {
if (!(value instanceof Serializable)) {
for (@SuppressWarnings("rawtypes")
final PropertyConverter converter : converters) {
if (converter.baseClass().isAssignableFrom(property.getHelper().getBaseClass())) {
return converter.convert(value);
}
}
}
if (!property.getHelper().getBaseClass().isInstance(value) && (value instanceof String)) {
for (@SuppressWarnings("rawtypes")
final PropertyConverter converter : converters) {
if (converter.baseClass().isAssignableFrom(property.getHelper().getBaseClass())) {
return converter.convert(converter.convert(value.toString()));
}
}
}
return (Serializable) value;
}
public interface PropertyConverter extends Serializable {
public Serializable convert(T ob) throws Exception;
public T convert(Serializable ob) throws Exception;
public Class baseClass();
}
public interface PropertyGroup extends Serializable {
public T convert(CommandLine commandLine) throws ParseException;
public ParameterEnum getParameter();
}
public static class QueryConverter implements PropertyConverter {
/** */
private static final long serialVersionUID = 1L;
@Override
public Serializable convert(final Query ob) {
try {
return toBytes(ob);
} catch (final UnsupportedEncodingException e) {
throw new IllegalArgumentException(
String.format(
"Cannot convert %s to a Query: %s",
ob.toString(),
e.getLocalizedMessage()));
}
}
@Override
public Query convert(final Serializable ob) throws Exception {
if (ob instanceof byte[]) {
return (Query) PropertyManagement.fromBytes((byte[]) ob);
} else if (ob instanceof Query) {
return (Query) ob;
}
return QueryBuilder.newBuilder().build();
}
@Override
public Class baseClass() {
return Query.class;
}
}
public static class PathConverter implements PropertyConverter {
/** */
private static final long serialVersionUID = 1L;
@Override
public Serializable convert(final Path ob) {
return ob.toUri().toString();
}
@Override
public Path convert(final Serializable ob) throws Exception {
return new Path(ob.toString());
}
@Override
public Class baseClass() {
return Path.class;
}
}
public static class ByteConverter implements PropertyConverter {
private static final long serialVersionUID = 1L;
@Override
public Serializable convert(final byte[] ob) {
return ByteArrayUtils.byteArrayToString(ob);
}
@Override
public byte[] convert(final Serializable ob) throws Exception {
return ByteArrayUtils.byteArrayFromString(ob.toString());
}
@Override
public Class baseClass() {
return byte[].class;
}
}
public static class IntegerConverter implements PropertyConverter {
private static final long serialVersionUID = 1L;
@Override
public Serializable convert(final Integer ob) {
return ob;
}
@Override
public Integer convert(final Serializable ob) throws Exception {
return Integer.parseInt(ob.toString());
}
@Override
public Class baseClass() {
return Integer.class;
}
}
public static class DoubleConverter implements PropertyConverter {
/** */
private static final long serialVersionUID = 1L;
@Override
public Serializable convert(final Double ob) {
return ob;
}
@Override
public Double convert(final Serializable ob) throws Exception {
return Double.parseDouble(ob.toString());
}
@Override
public Class baseClass() {
return Double.class;
}
}
public static class PersistableConverter implements PropertyConverter {
/** */
private static final long serialVersionUID = 1L;
@Override
public Serializable convert(final Persistable ob) {
try {
return toBytes(ob);
} catch (final UnsupportedEncodingException e) {
throw new IllegalArgumentException(
String.format(
"Cannot convert %s to a Persistable: %s",
ob.toString(),
e.getLocalizedMessage()));
}
}
@Override
public Persistable convert(final Serializable ob) throws Exception {
if (ob instanceof byte[]) {
return fromBytes((byte[]) ob);
}
throw new IllegalArgumentException(
String.format("Cannot convert %s to Persistable", ob.toString()));
}
@Override
public Class baseClass() {
return Persistable.class;
}
}
private boolean containsPropertyValue(final ParameterEnum property) {
return ((nestProperties != null) && nestProperties.containsPropertyValue(property))
|| localProperties.containsKey(property);
}
private Serializable getPropertyValue(final ParameterEnum property) {
final Serializable val = localProperties != null ? localProperties.get(property) : null;
if (val == null) {
return nestProperties != null ? nestProperties.getPropertyValue(property) : null;
}
return val;
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/ScopedJobConfiguration.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import org.apache.hadoop.conf.Configuration;
import org.locationtech.geowave.core.index.ByteArrayUtils;
import org.locationtech.geowave.mapreduce.GeoWaveConfiguratorBase;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class ScopedJobConfiguration {
protected static final Logger LOGGER = LoggerFactory.getLogger(ScopedJobConfiguration.class);
private final Configuration jobConfiguration;
private final Class scope;
private Logger logger = LOGGER;
public ScopedJobConfiguration(final Configuration jobConfiguration, final Class scope) {
super();
this.jobConfiguration = jobConfiguration;
this.scope = scope;
}
public ScopedJobConfiguration(
final Configuration jobConfiguration,
final Class scope,
final Logger logger) {
super();
this.jobConfiguration = jobConfiguration;
this.scope = scope;
this.logger = logger;
}
public int getInt(final Enum property, final int defaultValue) {
final String propName = GeoWaveConfiguratorBase.enumToConfKey(scope, property);
if (jobConfiguration.getRaw(propName) == null) {
logger.warn("Using default for property " + propName);
}
final int v = jobConfiguration.getInt(propName, defaultValue);
return v;
}
public String getString(final Enum property, final String defaultValue) {
final String propName = GeoWaveConfiguratorBase.enumToConfKey(scope, property);
if (jobConfiguration.getRaw(propName) == null) {
logger.warn("Using default for property " + propName);
}
return jobConfiguration.get(propName, defaultValue);
}
public T getInstance(
final Enum property,
final Class iface,
final Class defaultValue) throws InstantiationException, IllegalAccessException {
try {
final String propName = GeoWaveConfiguratorBase.enumToConfKey(scope, property);
if (jobConfiguration.getRaw(propName) == null) {
if (defaultValue == null) {
return null;
}
logger.warn("Using default for property " + propName);
}
return jobConfiguration.getClass(
GeoWaveConfiguratorBase.enumToConfKey(scope, property),
defaultValue,
iface).newInstance();
} catch (final Exception ex) {
logger.error("Cannot instantiate " + GeoWaveConfiguratorBase.enumToConfKey(scope, property));
throw ex;
}
}
public double getDouble(final Enum property, final double defaultValue) {
final String propName = GeoWaveConfiguratorBase.enumToConfKey(scope, property);
if (jobConfiguration.getRaw(propName) == null) {
logger.warn("Using default for property " + propName);
}
return jobConfiguration.getDouble(propName, defaultValue);
}
public byte[] getBytes(final Enum property) {
final String propName = GeoWaveConfiguratorBase.enumToConfKey(scope, property);
final String data = jobConfiguration.getRaw(propName);
if (data == null) {
logger.error(propName + " not found ");
}
return ByteArrayUtils.byteArrayFromString(data);
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/SerializableAdapterStore.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.IOException;
import java.io.Serializable;
import org.locationtech.geowave.core.store.adapter.TransientAdapterStore;
import org.locationtech.geowave.core.store.api.DataTypeAdapter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Support for adapter stores that are Serializable. Rather than for an adapter store to serialize
* its state, wrap an adapter store. If the adapter store is not serializable, then log a warning
* message upon serialization.
*/
public class SerializableAdapterStore implements TransientAdapterStore, Serializable {
/** */
private static final long serialVersionUID = 1L;
static final Logger LOGGER = LoggerFactory.getLogger(SerializableAdapterStore.class);
transient TransientAdapterStore adapterStore;
public SerializableAdapterStore() {}
public SerializableAdapterStore(final TransientAdapterStore adapterStore) {
super();
this.adapterStore = adapterStore;
}
private TransientAdapterStore getAdapterStore() {
if (adapterStore == null) {
throw new IllegalStateException("AdapterStore has not been initialized");
}
return adapterStore;
}
@Override
public void addAdapter(final DataTypeAdapter adapter) {
getAdapterStore().addAdapter(adapter);
}
@Override
public DataTypeAdapter getAdapter(final String typeName) {
return getAdapterStore().getAdapter(typeName);
}
@Override
public boolean adapterExists(final String typeName) {
return getAdapterStore().adapterExists(typeName);
}
@Override
public DataTypeAdapter[] getAdapters() {
return getAdapterStore().getAdapters();
}
@Override
public void removeAll() {
getAdapterStore().removeAll();
}
private void writeObject(final java.io.ObjectOutputStream out) throws IOException {
if (adapterStore instanceof Serializable) {
out.writeBoolean(true);
out.writeObject(adapterStore);
} else {
out.writeBoolean(false);
}
}
private void readObject(final java.io.ObjectInputStream in)
throws IOException, ClassNotFoundException {
if (in.readBoolean()) {
adapterStore = (TransientAdapterStore) in.readObject();
} else {
LOGGER.warn("Unable to initialized AdapterStore; the store is not serializable");
}
}
@Override
public void removeAdapter(final String typeName) {
getAdapterStore().removeAdapter(typeName);
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/ShapefileTool.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.File;
import java.io.IOException;
import java.io.Serializable;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.io.FileUtils;
import org.geotools.data.DefaultTransaction;
import org.geotools.data.FeatureWriter;
import org.geotools.data.Transaction;
import org.geotools.data.shapefile.ShapefileDataStore;
import org.geotools.data.shapefile.ShapefileDataStoreFactory;
import org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.feature.simple.SimpleFeatureTypeBuilder;
import org.geotools.referencing.crs.DefaultGeographicCRS;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.Point;
import org.locationtech.jts.geom.Polygon;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.feature.type.AttributeDescriptor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class ShapefileTool {
private static final Logger LOGGER = LoggerFactory.getLogger(ShapefileTool.class);
private static SimpleFeatureType createFeatureType(final String typeName, final boolean isPoint) {
final SimpleFeatureTypeBuilder builder = new SimpleFeatureTypeBuilder();
builder.setName(typeName);
builder.setCRS(DefaultGeographicCRS.WGS84); // <- Coordinate reference
// system
// add attributes in order
builder.add("the_geom", isPoint ? Point.class : Polygon.class);
builder.length(15).add("Name", String.class); // <- 15 chars width for name field
// build the type
return builder.buildFeatureType();
}
@edu.umd.cs.findbugs.annotations.SuppressFBWarnings(
value = "RV_RETURN_VALUE_IGNORED_BAD_PRACTICE",
justification = "Directories may alreadybe there")
public static void writeShape(final String typeName, final File dir, final Geometry[] shapes)
throws IOException {
FileUtils.deleteDirectory(dir);
dir.mkdirs();
final SimpleFeatureBuilder featureBuilder =
new SimpleFeatureBuilder(createFeatureType(typeName, shapes[0] instanceof Point));
final ShapefileDataStoreFactory dataStoreFactory = new ShapefileDataStoreFactory();
final Map params = new HashMap<>();
params.put("url", new File(dir.getAbsolutePath() + "/" + typeName + ".shp").toURI().toURL());
params.put("create spatial index", Boolean.TRUE);
final ShapefileDataStore newDataStore =
(ShapefileDataStore) dataStoreFactory.createNewDataStore(params);
newDataStore.createSchema(createFeatureType(typeName, shapes[0] instanceof Point));
final Transaction transaction = new DefaultTransaction("create");
try (final FeatureWriter writer =
newDataStore.getFeatureWriterAppend(typeName, transaction)) {
final int i = 1;
for (final Geometry shape : shapes) {
featureBuilder.add(shape);
featureBuilder.add(Integer.valueOf(i));
final SimpleFeature feature = featureBuilder.buildFeature(null);
final SimpleFeature copy = writer.next();
for (final AttributeDescriptor attrD : feature.getFeatureType().getAttributeDescriptors()) {
// the null case should only happen for geometry
if (copy.getFeatureType().getDescriptor(attrD.getName()) != null) {
copy.setAttribute(attrD.getName(), feature.getAttribute(attrD.getName()));
}
}
// shape files force geometry name to be 'the_geom'. So isolate
// this change
copy.setDefaultGeometry(feature.getDefaultGeometry());
writer.write();
}
} catch (final IOException e) {
LOGGER.warn("Problem with the FeatureWritter", e);
transaction.rollback();
} finally {
transaction.commit();
transaction.close();
}
}
@edu.umd.cs.findbugs.annotations.SuppressFBWarnings(
value = "RV_RETURN_VALUE_IGNORED_BAD_PRACTICE",
justification = "Directories may alreadybe there")
public static void writeShape(final File dir, final List shapes)
throws IOException {
FileUtils.deleteDirectory(dir);
dir.mkdirs();
final ShapefileDataStoreFactory dataStoreFactory = new ShapefileDataStoreFactory();
final String typeName = shapes.get(0).getType().getTypeName();
final Map params = new HashMap<>();
params.put("url", new File(dir.getAbsolutePath() + "/" + typeName + ".shp").toURI().toURL());
params.put("create spatial index", Boolean.TRUE);
final ShapefileDataStore newDataStore =
(ShapefileDataStore) dataStoreFactory.createNewDataStore(params);
newDataStore.createSchema(shapes.get(0).getFeatureType());
final Transaction transaction = new DefaultTransaction("create");
try (final FeatureWriter writer =
newDataStore.getFeatureWriterAppend(typeName, transaction)) {
for (final SimpleFeature shape : shapes) {
final SimpleFeature copy = writer.next();
for (final AttributeDescriptor attrD : copy.getFeatureType().getAttributeDescriptors()) {
// the null case should only happen for geometry
if (copy.getFeatureType().getDescriptor(attrD.getName()) != null) {
copy.setAttribute(attrD.getName(), shape.getAttribute(attrD.getName()));
}
}
// shape files force geometry name to be 'the_geom'. So isolate
// this change
copy.setDefaultGeometry(shape.getDefaultGeometry());
writer.write();
}
} catch (final IOException e) {
LOGGER.warn("Problem with the FeatureWritter", e);
transaction.rollback();
} finally {
transaction.commit();
transaction.close();
}
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/SimpleFeatureItemWrapperFactory.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.IOException;
import java.util.UUID;
import org.apache.hadoop.mapreduce.JobContext;
import org.locationtech.geowave.analytic.AnalyticFeature.ClusterFeatureAttribute;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.Geometry;
import org.opengis.feature.simple.SimpleFeature;
import org.slf4j.Logger;
public class SimpleFeatureItemWrapperFactory implements AnalyticItemWrapperFactory {
@Override
public AnalyticItemWrapper create(final SimpleFeature item) {
return new SimpleFeatureAnalyticItemWrapper(item);
}
@Override
public void initialize(final JobContext context, final Class scope, final Logger logger)
throws IOException {}
public static class SimpleFeatureAnalyticItemWrapper implements
AnalyticItemWrapper {
final SimpleFeature item;
public SimpleFeatureAnalyticItemWrapper(final SimpleFeature item) {
this.item = item;
}
@Override
public String getID() {
return item.getID();
}
@Override
public SimpleFeature getWrappedItem() {
return item;
}
@Override
public long getAssociationCount() {
final Long countO = (Long) item.getAttribute(ClusterFeatureAttribute.COUNT.attrName());
return (countO != null) ? countO.longValue() : 0;
}
@Override
public int getIterationID() {
return ((Integer) item.getAttribute(ClusterFeatureAttribute.ITERATION.attrName())).intValue();
}
@Override
public String getGroupID() {
return getAttribute(item, ClusterFeatureAttribute.GROUP_ID.attrName());
}
@Override
public void setGroupID(final String groupID) {
item.setAttribute(ClusterFeatureAttribute.GROUP_ID.attrName(), groupID);
}
@Override
public void resetAssociatonCount() {
item.setAttribute(ClusterFeatureAttribute.COUNT.attrName(), 0);
}
@Override
public void incrementAssociationCount(final long increment) {
item.setAttribute(
ClusterFeatureAttribute.COUNT.attrName(),
getAssociationCount() + increment);
}
@Override
public String toString() {
return "SimpleFeatureCentroid [item="
+ item.getID()
+ ", + group="
+ getGroupID()
+ ", + count="
+ getAssociationCount()
+ ", cost="
+ getCost()
+ "]";
}
@Override
public double getCost() {
final Double costO = (Double) item.getAttribute(ClusterFeatureAttribute.WEIGHT.attrName());
return (costO != null) ? costO.doubleValue() : 0.0;
}
@Override
public void setCost(final double cost) {
// GENERIC GEOMETRY HAS A DISTANCE, NOT A COST
item.setAttribute(ClusterFeatureAttribute.WEIGHT.attrName(), cost);
}
@Override
public String getName() {
return item.getAttribute(ClusterFeatureAttribute.NAME.attrName()).toString();
}
@Override
public String[] getExtraDimensions() {
return new String[0];
}
@Override
public double[] getDimensionValues() {
return new double[0];
}
@Override
public Geometry getGeometry() {
return (Geometry) item.getAttribute(ClusterFeatureAttribute.GEOMETRY.attrName());
}
@Override
public void setZoomLevel(final int level) {
item.setAttribute(ClusterFeatureAttribute.ZOOM_LEVEL.attrName(), Integer.valueOf(level));
}
@Override
public int getZoomLevel() {
return getIntAttribute(item, ClusterFeatureAttribute.ZOOM_LEVEL.attrName(), 1);
}
@Override
public void setBatchID(final String batchID) {
item.setAttribute(ClusterFeatureAttribute.BATCH_ID.attrName(), batchID);
}
@Override
public String getBatchID() {
return item.getAttribute(ClusterFeatureAttribute.BATCH_ID.attrName()).toString();
}
}
private static String getAttribute(final SimpleFeature feature, final String name) {
final Object att = feature.getAttribute(name);
return att == null ? null : att.toString();
}
private static int getIntAttribute(
final SimpleFeature feature,
final String name,
final int defaultValue) {
final Object att = feature.getAttribute(name);
return att == null ? defaultValue
: (att instanceof Number ? ((Number) att).intValue() : Integer.parseInt(att.toString()));
}
/*
* @see org.locationtech.geowave.analytics.tools.CentroidFactory#createNextCentroid
* (java.lang.Object, org.locationtech.jts.geom.Coordinate, java.lang.String[], double[])
*/
@Override
public AnalyticItemWrapper createNextItem(
final SimpleFeature feature,
final String groupID,
final Coordinate coordinate,
final String[] extraNames,
final double[] extraValues) {
final Geometry geometry =
(Geometry) feature.getAttribute(ClusterFeatureAttribute.GEOMETRY.attrName());
return new SimpleFeatureAnalyticItemWrapper(
AnalyticFeature.createGeometryFeature(
feature.getFeatureType(),
feature.getAttribute(ClusterFeatureAttribute.BATCH_ID.attrName()).toString(),
UUID.randomUUID().toString(),
getAttribute(feature, ClusterFeatureAttribute.NAME.attrName()),
groupID,
((Double) feature.getAttribute(
ClusterFeatureAttribute.WEIGHT.attrName())).doubleValue(),
geometry.getFactory().createPoint(coordinate),
extraNames,
extraValues,
((Integer) feature.getAttribute(
ClusterFeatureAttribute.ZOOM_LEVEL.attrName())).intValue(),
((Integer) feature.getAttribute(
ClusterFeatureAttribute.ITERATION.attrName())).intValue() + 1,
((Long) feature.getAttribute(ClusterFeatureAttribute.COUNT.attrName())).longValue()));
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/SimpleFeatureProjection.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.JobContext;
import org.locationtech.jts.geom.Geometry;
import org.opengis.feature.simple.SimpleFeature;
/** Assumes two-dimensional simple feature without time dimensions. */
public class SimpleFeatureProjection implements Projection {
@Override
public Geometry getProjection(final SimpleFeature anItem) {
return (Geometry) anItem.getDefaultGeometry();
}
@Override
public void initialize(final JobContext context, final Class scope) throws IOException {}
@Override
public void setup(
final PropertyManagement runTimeProperties,
final Class scope,
final Configuration configuration) {}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/clustering/CentroidItemWrapperFactory.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic.clustering;
import java.io.IOException;
import org.apache.hadoop.mapreduce.JobContext;
import org.locationtech.geowave.analytic.AnalyticItemWrapper;
import org.locationtech.geowave.analytic.AnalyticItemWrapperFactory;
import org.locationtech.geowave.analytic.kmeans.AssociationNotification;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.Geometry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Determine the group ID for an item dynamically.
*
* @param
*/
public class CentroidItemWrapperFactory implements AnalyticItemWrapperFactory {
static final Logger LOGGER = LoggerFactory.getLogger(CentroidItemWrapperFactory.class);
private AnalyticItemWrapperFactory itemFactory;
private NestedGroupCentroidAssignment nestedGroupCentroidAssignment;
@Override
public AnalyticItemWrapper create(final T item) {
return new CentroidItemWrapper(item);
}
@Override
public void initialize(final JobContext context, final Class scope, final Logger logger)
throws IOException {
try {
nestedGroupCentroidAssignment = new NestedGroupCentroidAssignment<>(context, scope, logger);
} catch (InstantiationException | IllegalAccessException e) {
throw new IOException("Failed to instantiate", e);
}
itemFactory.initialize(context, scope, logger);
}
public AnalyticItemWrapperFactory getItemFactory() {
return itemFactory;
}
public void setItemFactory(final AnalyticItemWrapperFactory itemFactory) {
this.itemFactory = itemFactory;
}
public class CentroidItemWrapper implements AnalyticItemWrapper {
final AnalyticItemWrapper wrappedItem;
AnalyticItemWrapper centroidItem;
public CentroidItemWrapper(final T item) {
wrappedItem = itemFactory.create(item);
try {
nestedGroupCentroidAssignment.findCentroidForLevel(
wrappedItem,
new AssociationNotification() {
@Override
public void notify(final CentroidPairing pairing) {
centroidItem = pairing.getCentroid();
}
});
} catch (final IOException e) {
LOGGER.error("Cannot resolve paired centroid for " + wrappedItem.getID(), e);
centroidItem = wrappedItem;
}
}
@Override
public String getID() {
return centroidItem.getID();
}
@Override
public T getWrappedItem() {
return centroidItem.getWrappedItem();
}
@Override
public long getAssociationCount() {
return centroidItem.getAssociationCount();
}
@Override
public int getIterationID() {
return centroidItem.getIterationID();
}
// this is not a mistake...the group id is the centroid itself
@Override
public String getGroupID() {
return centroidItem.getID();
}
@Override
public void setGroupID(final String groupID) {}
@Override
public void resetAssociatonCount() {}
@Override
public void incrementAssociationCount(final long increment) {}
@Override
public double getCost() {
return centroidItem.getCost();
}
@Override
public void setCost(final double cost) {}
@Override
public String getName() {
return centroidItem.getName();
}
@Override
public String[] getExtraDimensions() {
return new String[0];
}
@Override
public double[] getDimensionValues() {
return new double[0];
}
@Override
public Geometry getGeometry() {
return centroidItem.getGeometry();
}
@Override
public void setZoomLevel(final int level) {}
@Override
public int getZoomLevel() {
return centroidItem.getZoomLevel();
}
@Override
public void setBatchID(final String batchID) {}
@Override
public String getBatchID() {
return centroidItem.getBatchID();
}
}
/*
* @see org.locationtech.geowave.analytics.tools.CentroidFactory#createNextCentroid
* (java.lang.Object, org.locationtech.jts.geom.Coordinate, java.lang.String[], double[])
*/
@Override
public AnalyticItemWrapper createNextItem(
final T feature,
final String groupID,
final Coordinate coordinate,
final String[] extraNames,
final double[] extraValues) {
return this.itemFactory.createNextItem(feature, groupID, coordinate, extraNames, extraValues);
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/clustering/CentroidManager.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic.clustering;
import java.io.IOException;
import java.util.List;
import org.locationtech.geowave.analytic.AnalyticItemWrapper;
import org.locationtech.geowave.analytic.clustering.exception.MatchingCentroidNotFoundException;
import org.locationtech.jts.geom.Coordinate;
/**
* Manage centroids created per batch and per group of analytic processes. There can be multiple
* groups per batch. A group is loosely interpreted as a set of item geometries under analysis. The
* sets can be defined by shared characteristics.
*
* @param The type of item that is used to represent a centroid.
*/
public interface CentroidManager {
/**
* Creates a new centroid based on the old centroid with new coordinates and dimension values
*/
public AnalyticItemWrapper createNextCentroid(
final T feature,
final String groupID,
final Coordinate coordinate,
final String[] extraNames,
final double[] extraValues);
public AnalyticItemWrapper getCentroidById(final String id, final String groupID)
throws IOException, MatchingCentroidNotFoundException;
public void delete(final String[] dataIds) throws IOException;
public List getAllCentroidGroups() throws IOException;
public List> getCentroidsForGroup(final String groupID) throws IOException;
public List> getCentroidsForGroup(
final String batchID,
final String groupID) throws IOException;
public int processForAllGroups(CentroidProcessingFn fn) throws IOException;
public static interface CentroidProcessingFn {
public int processGroup(final String groupID, final List> centroids);
}
public AnalyticItemWrapper getCentroid(final String id);
public void clear();
public String getDataTypeName();
public String getIndexName();
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/clustering/CentroidManagerGeoWave.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic.clustering;
import java.io.IOException;
import java.io.Serializable;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Calendar;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.collections.map.LRUMap;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.JobContext;
import org.geotools.data.DefaultTransaction;
import org.geotools.data.FeatureWriter;
import org.geotools.data.Transaction;
import org.geotools.data.shapefile.ShapefileDataStore;
import org.geotools.data.shapefile.ShapefileDataStoreFactory;
import org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.feature.simple.SimpleFeatureTypeBuilder;
import org.geotools.feature.type.BasicFeatureTypes;
import org.geotools.filter.FilterFactoryImpl;
import org.locationtech.geowave.adapter.vector.FeatureDataAdapter;
import org.locationtech.geowave.analytic.AnalyticFeature;
import org.locationtech.geowave.analytic.AnalyticFeature.ClusterFeatureAttribute;
import org.locationtech.geowave.analytic.AnalyticItemWrapper;
import org.locationtech.geowave.analytic.AnalyticItemWrapperFactory;
import org.locationtech.geowave.analytic.PropertyManagement;
import org.locationtech.geowave.analytic.ScopedJobConfiguration;
import org.locationtech.geowave.analytic.clustering.exception.MatchingCentroidNotFoundException;
import org.locationtech.geowave.analytic.param.CentroidParameters;
import org.locationtech.geowave.analytic.param.GlobalParameters;
import org.locationtech.geowave.analytic.param.ParameterEnum;
import org.locationtech.geowave.analytic.param.StoreParameters;
import org.locationtech.geowave.analytic.store.PersistableStore;
import org.locationtech.geowave.core.geotime.index.SpatialDimensionalityTypeProvider;
import org.locationtech.geowave.core.geotime.index.SpatialOptions;
import org.locationtech.geowave.core.geotime.store.GeotoolsFeatureDataAdapter;
import org.locationtech.geowave.core.geotime.store.query.api.VectorQueryBuilder;
import org.locationtech.geowave.core.index.StringUtils;
import org.locationtech.geowave.core.store.CloseableIterator;
import org.locationtech.geowave.core.store.adapter.PersistentAdapterStore;
import org.locationtech.geowave.core.store.api.DataStore;
import org.locationtech.geowave.core.store.api.Index;
import org.locationtech.geowave.core.store.api.QueryBuilder;
import org.locationtech.geowave.core.store.api.Writer;
import org.locationtech.geowave.core.store.index.IndexStore;
import org.locationtech.geowave.mapreduce.GeoWaveConfiguratorBase;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.Point;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.feature.type.AttributeDescriptor;
import org.opengis.feature.type.GeometryType;
import org.opengis.filter.Filter;
import org.opengis.filter.expression.Expression;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Manages the population of centroids by group id and batch id.
*
* @param The item type used to represent a centroid.
*/
public class CentroidManagerGeoWave implements CentroidManager {
static final Logger LOGGER = LoggerFactory.getLogger(CentroidManagerGeoWave.class);
private static final ParameterEnum[] MY_PARAMS =
new ParameterEnum[] {
StoreParameters.StoreParam.INPUT_STORE,
GlobalParameters.Global.BATCH_ID,
CentroidParameters.Centroid.DATA_TYPE_ID,
CentroidParameters.Centroid.DATA_NAMESPACE_URI,
CentroidParameters.Centroid.INDEX_NAME,
CentroidParameters.Centroid.WRAPPER_FACTORY_CLASS,
CentroidParameters.Centroid.ZOOM_LEVEL};
private String batchId;
private int level = 0;
private AnalyticItemWrapperFactory centroidFactory;
private GeotoolsFeatureDataAdapter adapter;
private String centroidDataTypeId;
private DataStore dataStore;
private IndexStore indexStore;
private Index index;
public CentroidManagerGeoWave(
final DataStore dataStore,
final IndexStore indexStore,
final PersistentAdapterStore adapterStore,
final AnalyticItemWrapperFactory centroidFactory,
final String centroidDataTypeId,
final short centroidInternalAdapterId,
final String indexName,
final String batchId,
final int level) {
this.centroidFactory = centroidFactory;
this.level = level;
this.batchId = batchId;
this.dataStore = dataStore;
this.indexStore = indexStore;
this.centroidDataTypeId = centroidDataTypeId;
index = indexStore.getIndex(indexName);
adapter =
(GeotoolsFeatureDataAdapter) adapterStore.getAdapter(
centroidInternalAdapterId).getAdapter();
}
public CentroidManagerGeoWave(final PropertyManagement properties) throws IOException {
final Class scope = CentroidManagerGeoWave.class;
final Configuration configuration = new Configuration();
properties.setJobConfiguration(configuration, scope);
init(Job.getInstance(configuration), scope, LOGGER);
}
public CentroidManagerGeoWave(final JobContext context, final Class scope) throws IOException {
this(context, scope, LOGGER);
}
public CentroidManagerGeoWave(final JobContext context, final Class scope, final Logger logger)
throws IOException {
init(context, scope, logger);
}
private void init(final JobContext context, final Class scope, final Logger logger)
throws IOException {
final ScopedJobConfiguration scopedJob =
new ScopedJobConfiguration(context.getConfiguration(), scope, logger);
try {
centroidFactory =
(AnalyticItemWrapperFactory) CentroidParameters.Centroid.WRAPPER_FACTORY_CLASS.getHelper().getValue(
context,
scope,
CentroidItemWrapperFactory.class);
centroidFactory.initialize(context, scope, logger);
} catch (final Exception e1) {
LOGGER.error(
"Cannot instantiate "
+ GeoWaveConfiguratorBase.enumToConfKey(
this.getClass(),
CentroidParameters.Centroid.WRAPPER_FACTORY_CLASS));
throw new IOException(e1);
}
this.level = scopedJob.getInt(CentroidParameters.Centroid.ZOOM_LEVEL, 1);
centroidDataTypeId = scopedJob.getString(CentroidParameters.Centroid.DATA_TYPE_ID, "centroid");
batchId =
scopedJob.getString(
GlobalParameters.Global.BATCH_ID,
Long.toString(Calendar.getInstance().getTime().getTime()));
final String indexName =
scopedJob.getString(
CentroidParameters.Centroid.INDEX_NAME,
SpatialDimensionalityTypeProvider.createIndexFromOptions(
new SpatialOptions()).getName());
final PersistableStore store =
(PersistableStore) StoreParameters.StoreParam.INPUT_STORE.getHelper().getValue(
context,
scope,
null);
dataStore = store.getDataStoreOptions().createDataStore();
indexStore = store.getDataStoreOptions().createIndexStore();
index = indexStore.getIndex(indexName);
final PersistentAdapterStore adapterStore = store.getDataStoreOptions().createAdapterStore();
adapter =
(GeotoolsFeatureDataAdapter) adapterStore.getAdapter(
store.getDataStoreOptions().createInternalAdapterStore().getAdapterId(
centroidDataTypeId)).getAdapter();
}
/**
* Creates a new centroid based on the old centroid with new coordinates and dimension values
*/
@Override
public AnalyticItemWrapper createNextCentroid(
final T feature,
final String groupID,
final Coordinate coordinate,
final String[] extraNames,
final double[] extraValues) {
return centroidFactory.createNextItem(feature, groupID, coordinate, extraNames, extraValues);
}
private final int capacity = 100;
private final LRUMap groupToCentroid = new LRUMap(capacity);
@Override
public void clear() {
groupToCentroid.clear();
}
@Override
public void delete(final String[] dataIds) throws IOException {
for (final String dataId : dataIds) {
if (dataId != null) {
final QueryBuilder bldr =
QueryBuilder.newBuilder().addTypeName(centroidDataTypeId).indexName(index.getName());
dataStore.delete(
bldr.constraints(
bldr.constraintsFactory().dataIds(StringUtils.stringToBinary(dataId))).build());
}
}
}
@Override
public List getAllCentroidGroups() throws IOException {
final List groups = new ArrayList<>();
final CloseableIterator it = getRawCentroids(this.batchId, null);
while (it.hasNext()) {
final AnalyticItemWrapper item = centroidFactory.create(it.next());
final String groupID = item.getGroupID();
int pos = groups.indexOf(groupID);
if (pos < 0) {
pos = groups.size();
groups.add(groupID);
}
// cache the first set
if (pos < capacity) {
getCentroidsForGroup(groupID);
}
}
it.close();
return groups;
}
@Override
public List> getCentroidsForGroup(final String groupID)
throws IOException {
return getCentroidsForGroup(this.batchId, groupID);
}
@Override
public List> getCentroidsForGroup(
final String batchID,
final String groupID) throws IOException {
final String lookupGroup = (groupID == null) ? "##" : groupID;
final Pair gid = Pair.of(batchID, lookupGroup);
@SuppressWarnings("unchecked")
List> centroids =
(List>) groupToCentroid.get(gid);
if (centroids == null) {
centroids = groupID == null ? loadCentroids(batchID, null) : loadCentroids(batchID, groupID);
groupToCentroid.put(gid, centroids);
}
return centroids;
}
@Override
public AnalyticItemWrapper getCentroidById(final String id, final String groupID)
throws IOException, MatchingCentroidNotFoundException {
for (final AnalyticItemWrapper centroid : this.getCentroidsForGroup(groupID)) {
if (centroid.getID().equals(id)) {
return centroid;
}
}
throw new MatchingCentroidNotFoundException(id);
}
private List> loadCentroids(final String batchID, final String groupID)
throws IOException {
final List> centroids = new ArrayList<>();
try {
CloseableIterator it = null;
try {
it = this.getRawCentroids(batchID, groupID);
while (it.hasNext()) {
centroids.add(centroidFactory.create(it.next()));
}
return centroids;
} finally {
if (it != null) {
it.close();
}
}
} catch (final IOException e) {
LOGGER.error("Cannot load centroids");
throw new IOException(e);
}
}
@Override
@SuppressWarnings("unchecked")
public AnalyticItemWrapper getCentroid(final String dataId) {
final QueryBuilder bldr =
(QueryBuilder) QueryBuilder.newBuilder().addTypeName(centroidDataTypeId).indexName(
index.getName());
try (CloseableIterator it =
dataStore.query(
bldr.constraints(
bldr.constraintsFactory().dataIds(StringUtils.stringToBinary(dataId))).build())) {
if (it.hasNext()) {
return centroidFactory.create(it.next());
}
}
return null;
}
@SuppressWarnings("unchecked")
protected CloseableIterator getRawCentroids(final String batchId, final String groupID)
throws IOException {
final FilterFactoryImpl factory = new FilterFactoryImpl();
final Expression expB1 = factory.property(ClusterFeatureAttribute.BATCH_ID.attrName());
final Expression expB2 = factory.literal(batchId);
final Filter batchIdFilter = factory.equal(expB1, expB2, false);
Filter finalFilter = batchIdFilter;
if (groupID != null) {
final Expression exp1 = factory.property(ClusterFeatureAttribute.GROUP_ID.attrName());
final Expression exp2 = factory.literal(groupID);
// ignore levels for group IDS
finalFilter = factory.and(factory.equal(exp1, exp2, false), batchIdFilter);
} else if (level > 0) {
final Expression exp1 = factory.property(ClusterFeatureAttribute.ZOOM_LEVEL.attrName());
final Expression exp2 = factory.literal(level);
finalFilter = factory.and(factory.equal(exp1, exp2, false), batchIdFilter);
}
final VectorQueryBuilder bldr =
VectorQueryBuilder.newBuilder().addTypeName(adapter.getTypeName()).indexName(
index.getName());
return (CloseableIterator) dataStore.query(
bldr.constraints(bldr.constraintsFactory().filterConstraints(finalFilter)).build());
}
@SuppressWarnings("unchecked")
public void transferBatch(final String fromBatchId, final String groupID) throws IOException {
int count = 0;
try (final CloseableIterator it = getRawCentroids(fromBatchId, groupID)) {
dataStore.addType(adapter, index);
try (final Writer indexWriter = dataStore.createWriter(adapter.getTypeName())) {
while (it.hasNext()) {
final AnalyticItemWrapper item = centroidFactory.create(it.next());
item.setBatchID(this.batchId);
count++;
indexWriter.write(item.getWrappedItem());
}
// indexWriter.close();
}
}
LOGGER.info("Transfer " + count + " centroids");
}
@Override
public int processForAllGroups(final CentroidProcessingFn fn) throws IOException {
List centroidGroups;
try {
centroidGroups = getAllCentroidGroups();
} catch (final IOException e) {
throw new IOException(e);
}
int status = 0;
for (final String groupID : centroidGroups) {
status = fn.processGroup(groupID, getCentroidsForGroup(groupID));
if (status != 0) {
break;
}
}
return status;
}
public static Collection> getParameters() {
return Arrays.asList(MY_PARAMS);
}
public static void setParameters(
final Configuration config,
final Class scope,
final PropertyManagement runTimeProperties) {
runTimeProperties.setConfig(MY_PARAMS, config, scope);
}
@Override
public String getIndexName() {
return index.getName();
}
public String getBatchId() {
return this.batchId;
}
private ToSimpleFeatureConverter getFeatureConverter(
final List> items,
final Class shapeClass) {
return (adapter instanceof FeatureDataAdapter)
? new SimpleFeatureConverter((FeatureDataAdapter) adapter, shapeClass)
: new NonSimpleFeatureConverter(
items.isEmpty() ? new String[0] : items.get(0).getExtraDimensions(),
shapeClass);
}
private interface ToSimpleFeatureConverter {
SimpleFeatureType getFeatureType();
SimpleFeature toSimpleFeature(AnalyticItemWrapper item);
}
private static SimpleFeatureType createFeatureType(
final SimpleFeatureType featureType,
final Class shapeClass) {
try {
final SimpleFeatureTypeBuilder builder = new SimpleFeatureTypeBuilder();
builder.setName(featureType.getName().getLocalPart());
builder.setNamespaceURI(featureType.getName().getNamespaceURI());
builder.setCRS(featureType.getCoordinateReferenceSystem());
for (final AttributeDescriptor attr : featureType.getAttributeDescriptors()) {
if (attr.getType() instanceof GeometryType) {
builder.add(attr.getLocalName(), shapeClass);
} else {
builder.add(attr.getLocalName(), attr.getType().getBinding());
}
}
return builder.buildFeatureType();
} catch (final Exception e) {
LOGGER.warn("Schema Creation Error. Hint: Check the SRID.", e);
}
return null;
}
private static Geometry convert(
final Geometry value,
final Class shapeClass) {
if (shapeClass.isInstance(value)) {
return value;
}
if (shapeClass.isAssignableFrom(Point.class)) {
return value.getCentroid();
}
final Geometry hull = value.convexHull();
if (shapeClass.isInstance(hull)) {
return hull;
}
return null;
}
private class SimpleFeatureConverter implements ToSimpleFeatureConverter {
final SimpleFeatureType type;
final Object[] defaults;
final Class shapeClass;
public SimpleFeatureConverter(
final FeatureDataAdapter adapter,
final Class shapeClass) {
type = createFeatureType(adapter.getFeatureType(), shapeClass);
int p = 0;
this.shapeClass = shapeClass;
final List descriptors =
adapter.getFeatureType().getAttributeDescriptors();
defaults = new Object[descriptors.size()];
for (final AttributeDescriptor descriptor : descriptors) {
defaults[p++] = descriptor.getDefaultValue();
}
}
@Override
public SimpleFeatureType getFeatureType() {
return type;
}
@Override
public SimpleFeature toSimpleFeature(final AnalyticItemWrapper item) {
final SimpleFeature newFeature = SimpleFeatureBuilder.build(type, defaults, item.getID());
int i = 0;
for (final Object value : ((SimpleFeature) item.getWrappedItem()).getAttributes()) {
if (value instanceof Geometry) {
final Geometry newValue = convert((Geometry) value, shapeClass);
if (newValue == null) {
return null;
}
newFeature.setAttribute(i++, newValue);
} else {
newFeature.setAttribute(i++, value);
}
}
return newFeature;
}
}
private class NonSimpleFeatureConverter implements ToSimpleFeatureConverter {
final SimpleFeatureType featureType;
final Object[] defaults;
final Class shapeClass;
public NonSimpleFeatureConverter(
final String[] extraDimensionNames,
final Class shapeClass) {
featureType =
AnalyticFeature.createFeatureAdapter(
centroidDataTypeId,
extraDimensionNames,
BasicFeatureTypes.DEFAULT_NAMESPACE,
ClusteringUtils.CLUSTERING_CRS,
ClusterFeatureAttribute.values(),
shapeClass).getFeatureType();
this.shapeClass = shapeClass;
final List descriptors = featureType.getAttributeDescriptors();
defaults = new Object[descriptors.size()];
int p = 0;
for (final AttributeDescriptor descriptor : descriptors) {
defaults[p++] = descriptor.getDefaultValue();
}
}
@Override
public SimpleFeatureType getFeatureType() {
return featureType;
}
@Override
public SimpleFeature toSimpleFeature(final AnalyticItemWrapper item) {
final Geometry value = item.getGeometry();
final Geometry newValue = convert(value, shapeClass);
if (newValue == null) {
return null;
}
return AnalyticFeature.createGeometryFeature(
featureType,
item.getBatchID(),
item.getID(),
item.getName(),
item.getGroupID(),
item.getCost(),
newValue,
item.getExtraDimensions(),
item.getDimensionValues(),
item.getZoomLevel(),
item.getIterationID(),
item.getAssociationCount());
}
}
public void toShapeFile(final String parentDir, final Class shapeClass)
throws IOException {
// File shp = new File(parentDir + "/" + this.batchId + ".shp");
// File shx = new File(parentDir + "/" + this.batchId + ".shx");
final ShapefileDataStoreFactory dataStoreFactory = new ShapefileDataStoreFactory();
final Map params = new HashMap<>();
try {
params.put("url", new URL("file://" + parentDir + "/" + this.batchId + ".shp"));
} catch (final MalformedURLException e) {
LOGGER.error("Error creating URL", e);
}
params.put("create spatial index", Boolean.TRUE);
final List> centroids = loadCentroids(batchId, null);
final ToSimpleFeatureConverter converter = getFeatureConverter(centroids, shapeClass);
final ShapefileDataStore newDataStore =
(ShapefileDataStore) dataStoreFactory.createNewDataStore(params);
newDataStore.createSchema(converter.getFeatureType());
final Transaction transaction = new DefaultTransaction("create");
final String typeName = newDataStore.getTypeNames()[0];
try (final FeatureWriter writer =
newDataStore.getFeatureWriterAppend(typeName, transaction)) {
for (final AnalyticItemWrapper item : centroids) {
final SimpleFeature copy = writer.next();
final SimpleFeature newFeature = converter.toSimpleFeature(item);
for (final AttributeDescriptor attrD : newFeature.getFeatureType().getAttributeDescriptors()) {
// the null case should only happen for geometry
if (copy.getFeatureType().getDescriptor(attrD.getName()) != null) {
copy.setAttribute(attrD.getName(), newFeature.getAttribute(attrD.getName()));
}
}
// shape files force geometry name to be 'the_geom'. So isolate
// this change
copy.setDefaultGeometry(newFeature.getDefaultGeometry());
writer.write();
}
} catch (final IOException e) {
LOGGER.warn("Problem with the FeatureWritter", e);
transaction.rollback();
} finally {
transaction.commit();
transaction.close();
}
}
@Override
public String getDataTypeName() {
return this.centroidDataTypeId;
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/clustering/CentroidPairing.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic.clustering;
import org.locationtech.geowave.analytic.AnalyticItemWrapper;
public class CentroidPairing {
private AnalyticItemWrapper centroid;
private AnalyticItemWrapper pairedItem;
private double distance;
public CentroidPairing() {}
public CentroidPairing(
final AnalyticItemWrapper centroid,
final AnalyticItemWrapper pairedItem,
final double distance) {
super();
this.centroid = centroid;
this.pairedItem = pairedItem;
this.distance = distance;
}
public AnalyticItemWrapper getCentroid() {
return centroid;
}
public void setCentroid(final AnalyticItemWrapper centroid) {
this.centroid = centroid;
}
public AnalyticItemWrapper getPairedItem() {
return pairedItem;
}
public void setPairedItem(final AnalyticItemWrapper pairedItem) {
this.pairedItem = pairedItem;
}
public double getDistance() {
return distance;
}
public void setDistance(final double distance) {
this.distance = distance;
}
@Override
public int hashCode() {
final int prime = 31;
int result = 1;
result = (prime * result) + ((centroid == null) ? 0 : centroid.hashCode());
long temp;
temp = Double.doubleToLongBits(distance);
result = (prime * result) + (int) (temp ^ (temp >>> 32));
result = (prime * result) + ((pairedItem == null) ? 0 : pairedItem.hashCode());
return result;
}
@Override
public boolean equals(final Object obj) {
if (this == obj) {
return true;
}
if (obj == null) {
return false;
}
if (getClass() != obj.getClass()) {
return false;
}
final CentroidPairing other = (CentroidPairing) obj;
if (centroid == null) {
if (other.centroid != null) {
return false;
}
} else if (!centroid.equals(other.centroid)) {
return false;
}
if (Double.doubleToLongBits(distance) != Double.doubleToLongBits(other.distance)) {
return false;
}
if (pairedItem == null) {
if (other.pairedItem != null) {
return false;
}
} else if (!pairedItem.equals(other.pairedItem)) {
return false;
}
return true;
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/clustering/ClusteringUtils.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic.clustering;
import java.io.IOException;
import java.util.LinkedList;
import java.util.List;
import org.geotools.feature.type.BasicFeatureTypes;
import org.locationtech.geowave.adapter.vector.FeatureDataAdapter;
import org.locationtech.geowave.analytic.AnalyticFeature;
import org.locationtech.geowave.analytic.PropertyManagement;
import org.locationtech.geowave.analytic.extract.DimensionExtractor;
import org.locationtech.geowave.analytic.param.CentroidParameters;
import org.locationtech.geowave.analytic.param.CommonParameters;
import org.locationtech.geowave.analytic.param.StoreParameters;
import org.locationtech.geowave.analytic.store.PersistableStore;
import org.locationtech.geowave.core.geotime.index.SpatialDimensionalityTypeProvider;
import org.locationtech.geowave.core.geotime.index.SpatialOptions;
import org.locationtech.geowave.core.geotime.store.query.ExplicitSpatialQuery;
import org.locationtech.geowave.core.index.ByteArray;
import org.locationtech.geowave.core.index.QueryRanges;
import org.locationtech.geowave.core.store.adapter.AdapterStore;
import org.locationtech.geowave.core.store.api.DataTypeAdapter;
import org.locationtech.geowave.core.store.api.Index;
import org.locationtech.geowave.core.store.index.IndexStore;
import org.locationtech.geowave.core.store.util.DataStoreUtils;
import org.locationtech.jts.geom.Polygon;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class ClusteringUtils {
public static final String CLUSTERING_CRS = "EPSG:4326";
static final Logger LOGGER = LoggerFactory.getLogger(ClusteringUtils.class);
private static DataTypeAdapter createAdapter(
final String sampleDataTypeId,
final String sampleDataNamespaceURI,
final AdapterStore adapterStore,
final String[] dimensionNames) {
final FeatureDataAdapter adapter =
AnalyticFeature.createGeometryFeatureAdapter(
sampleDataTypeId,
dimensionNames,
sampleDataNamespaceURI,
CLUSTERING_CRS);
final ByteArray dbId = new ByteArray(sampleDataTypeId);
if (!adapterStore.adapterExists(dbId)) {
adapterStore.addAdapter(adapter);
return adapter;
} else {
return adapterStore.getAdapter(dbId);
}
}
public static DataTypeAdapter[] getAdapters(final PropertyManagement propertyManagement)
throws IOException {
final PersistableStore store =
(PersistableStore) StoreParameters.StoreParam.INPUT_STORE.getHelper().getValue(
propertyManagement);
final AdapterStore adapterStore = store.getDataStoreOptions().createAdapterStore();
return adapterStore.getAdapters();
}
public static Index[] getIndices(final PropertyManagement propertyManagement) {
final PersistableStore store =
(PersistableStore) StoreParameters.StoreParam.INPUT_STORE.getHelper().getValue(
propertyManagement);
final IndexStore indexStore = store.getDataStoreOptions().createIndexStore();
try (final org.locationtech.geowave.core.store.CloseableIterator it =
indexStore.getIndices()) {
final List indices = new LinkedList<>();
while (it.hasNext()) {
indices.add(it.next());
}
final Index[] result = new Index[indices.size()];
indices.toArray(result);
return result;
}
}
/*
* Method takes in a polygon and generates the corresponding ranges in a GeoWave spatial index
*/
protected static QueryRanges getGeoWaveRangesForQuery(final Polygon polygon) {
final Index index =
SpatialDimensionalityTypeProvider.createIndexFromOptions(new SpatialOptions());
final QueryRanges ranges =
DataStoreUtils.constraintsToQueryRanges(
new ExplicitSpatialQuery(polygon).getIndexConstraints(index),
index,
null,
-1);
return ranges;
}
public static Index createIndex(final PropertyManagement propertyManagement) {
final PersistableStore store =
(PersistableStore) StoreParameters.StoreParam.INPUT_STORE.getHelper().getValue(
propertyManagement);
final IndexStore indexStore = store.getDataStoreOptions().createIndexStore();
return indexStore.getIndex(
propertyManagement.getPropertyAsString(CentroidParameters.Centroid.INDEX_NAME));
}
public static DataTypeAdapter createAdapter(final PropertyManagement propertyManagement)
throws ClassNotFoundException, InstantiationException, IllegalAccessException {
final Class dimensionExtractorClass =
propertyManagement.getPropertyAsClass(
CommonParameters.Common.DIMENSION_EXTRACT_CLASS,
DimensionExtractor.class);
return ClusteringUtils.createAdapter(
propertyManagement.getPropertyAsString(CentroidParameters.Centroid.DATA_TYPE_ID),
propertyManagement.getPropertyAsString(
CentroidParameters.Centroid.DATA_NAMESPACE_URI,
BasicFeatureTypes.DEFAULT_NAMESPACE),
((PersistableStore) StoreParameters.StoreParam.INPUT_STORE.getHelper().getValue(
propertyManagement)).getDataStoreOptions().createAdapterStore(),
dimensionExtractorClass.newInstance().getDimensionNames());
}
}
================================================
FILE: analytics/api/src/main/java/org/locationtech/geowave/analytic/clustering/DistortionGroupManagement.java
================================================
/**
* Copyright (c) 2013-2022 Contributors to the Eclipse Foundation
*
*
See the NOTICE file distributed with this work for additional information regarding copyright
* ownership. All rights reserved. This program and the accompanying materials are made available
* under the terms of the Apache License, Version 2.0 which accompanies this distribution and is
* available at http://www.apache.org/licenses/LICENSE-2.0.txt
*/
package org.locationtech.geowave.analytic.clustering;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.hadoop.io.Writable;
import org.locationtech.geowave.analytic.AnalyticItemWrapperFactory;
import org.locationtech.geowave.core.index.StringUtils;
import org.locationtech.geowave.core.index.numeric.MultiDimensionalNumericData;
import org.locationtech.geowave.core.store.CloseableIterator;
import org.locationtech.geowave.core.store.adapter.FieldDescriptor;
import org.locationtech.geowave.core.store.adapter.FieldDescriptorBuilder;
import org.locationtech.geowave.core.store.adapter.InternalAdapterStore;
import org.locationtech.geowave.core.store.adapter.PersistentAdapterStore;
import org.locationtech.geowave.core.store.api.DataStore;
import org.locationtech.geowave.core.store.api.DataTypeAdapter;
import org.locationtech.geowave.core.store.api.RowBuilder;
import org.locationtech.geowave.core.store.api.Index;
import org.locationtech.geowave.core.store.api.QueryBuilder;
import org.locationtech.geowave.core.store.cli.store.DataStorePluginOptions;
import org.locationtech.geowave.core.store.data.IndexedPersistenceEncoding;
import org.locationtech.geowave.core.store.data.field.FieldReader;
import org.locationtech.geowave.core.store.data.field.FieldUtils;
import org.locationtech.geowave.core.store.index.CommonIndexModel;
import org.locationtech.geowave.core.store.index.IndexStore;
import org.locationtech.geowave.core.store.index.NullIndex;
import org.locationtech.geowave.core.store.query.constraints.QueryConstraints;
import org.locationtech.geowave.core.store.query.filter.QueryFilter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Find the max change in distortion between some k and k-1, picking the value k associated with
* that change.
*
*
In a multi-group setting, each group may have a different optimal k. Thus, the optimal batch
* may be different for each group. Each batch is associated with a different value k.
*
*
Choose the appropriate batch for each group. Then change the batch identifier for group
* centroids to a final provided single batch identifier ( parent batch ).
*/
public class DistortionGroupManagement {
static final Logger LOGGER = LoggerFactory.getLogger(DistortionGroupManagement.class);
public static final Index DISTORTIONS_INDEX = new NullIndex("DISTORTIONS");
public static final String[] DISTORTIONS_INDEX_ARRAY = new String[] {DISTORTIONS_INDEX.getName()};
final DataStore dataStore;
final IndexStore indexStore;
final PersistentAdapterStore adapterStore;
final InternalAdapterStore internalAdapterStore;
public DistortionGroupManagement(final DataStorePluginOptions dataStoreOptions) {
dataStore = dataStoreOptions.createDataStore();
indexStore = dataStoreOptions.createIndexStore();
adapterStore = dataStoreOptions.createAdapterStore();
internalAdapterStore = dataStoreOptions.createInternalAdapterStore();
final DistortionDataAdapter adapter = new DistortionDataAdapter();
dataStore.addType(adapter, DISTORTIONS_INDEX);
}
public static class BatchIdFilter implements QueryFilter {
String batchId;
public BatchIdFilter() {}
public BatchIdFilter(final String batchId) {
super();
this.batchId = batchId;
}
@Override
public boolean accept(
final CommonIndexModel indexModel,
final IndexedPersistenceEncoding persistenceEncoding) {
return new DistortionEntry(persistenceEncoding.getDataId(), 0.0).batchId.equals(batchId);
}
@Override
public byte[] toBinary() {
return StringUtils.stringToBinary(batchId);
}
@Override
public void fromBinary(final byte[] bytes) {
batchId = StringUtils.stringFromBinary(bytes);
}
}
public static class BatchIdQuery implements QueryConstraints {
String batchId;
public BatchIdQuery() {}
public BatchIdQuery(final String batchId) {
super();
this.batchId = batchId;
}
@Override
public List createFilters(final Index index) {
return Collections.singletonList(new BatchIdFilter(batchId));
}
@Override
public List getIndexConstraints(final Index index) {
return Collections.emptyList();
}
@Override
public byte[] toBinary() {
return StringUtils.stringToBinary(batchId);
}
@Override
public void fromBinary(final byte[] bytes) {
batchId = StringUtils.stringFromBinary(bytes);
}
}
public int retainBestGroups(
final AnalyticItemWrapperFactory itemWrapperFactory,
final String dataTypeId,
final String indexId,
final String batchId,
final int level) {
try {
final Map groupDistortions = new HashMap<>();
// row id is group id
// colQual is cluster count
try (CloseableIterator it =
(CloseableIterator) dataStore.query(
QueryBuilder.newBuilder().addTypeName(
DistortionDataAdapter.ADAPTER_TYPE_NAME).indexName(
DISTORTIONS_INDEX.getName()).constraints(
new BatchIdQuery(batchId)).build())) {
while (it.hasNext()) {
final DistortionEntry entry = it.next();
final String groupID = entry.getGroupId();
final Integer clusterCount = entry.getClusterCount();
final Double distortion = entry.getDistortionValue();
DistortionGroup grp = groupDistortions.get(groupID);
if (grp == null) {
grp = new DistortionGroup(groupID);
groupDistortions.put(groupID, grp);
}
grp.addPair(clusterCount, distortion);
}
}
final CentroidManagerGeoWave centroidManager =
new CentroidManagerGeoWave<>(
dataStore,
indexStore,
adapterStore,
itemWrapperFactory,
dataTypeId,
internalAdapterStore.getAdapterId(dataTypeId),
indexId,
batchId,
level);
for (final DistortionGroup grp : groupDistortions.values()) {
final int optimalK = grp.bestCount();
final String kbatchId = batchId + "_" + optimalK;
centroidManager.transferBatch(kbatchId, grp.getGroupID());
}
} catch (final RuntimeException ex) {
throw ex;
} catch (final Exception ex) {
LOGGER.error("Cannot determine groups for batch", ex);
return 1;
}
return 0;
}
public static class DistortionEntry implements Writable {
private String groupId;
private String batchId;
private Integer clusterCount;
private Double distortionValue;
public DistortionEntry() {}
public DistortionEntry(
final String groupId,
final String batchId,
final Integer clusterCount,
final Double distortionValue) {
this.groupId = groupId;
this.batchId = batchId;
this.clusterCount = clusterCount;
this.distortionValue = distortionValue;
}
private DistortionEntry(final byte[] dataId, final Double distortionValue) {
final String dataIdStr = StringUtils.stringFromBinary(dataId);
final String[] split = dataIdStr.split("/");
batchId = split[0];
groupId = split[1];
clusterCount = Integer.parseInt(split[2]);
this.distortionValue = distortionValue;
}
public String getGroupId() {
return groupId;
}
public Integer getClusterCount() {
return clusterCount;
}
public Double getDistortionValue() {
return distortionValue;
}
private byte[] getDataId() {
return StringUtils.stringToBinary(batchId + "/" + groupId + "/" + clusterCount);
}
@Override
public void write(final DataOutput out) throws IOException {
out.writeUTF(groupId);
out.writeUTF(batchId);
out.writeInt(clusterCount);
out.writeDouble(distortionValue);
}
@Override
public void readFields(final DataInput in) throws IOException {
groupId = in.readUTF();
batchId = in.readUTF();
clusterCount = in.readInt();
distortionValue = in.readDouble();
}
}
private static class DistortionGroup {
final String groupID;
final List> clusterCountToDistortion = new ArrayList<>();
public DistortionGroup(final String groupID) {
this.groupID = groupID;
}
public void addPair(final Integer count, final Double distortion) {
clusterCountToDistortion.add(Pair.of(count, distortion));
}
public String getGroupID() {
return groupID;
}
public int bestCount() {
Collections.sort(clusterCountToDistortion, new Comparator>() {
@Override
public int compare(final Pair arg0, final Pair arg1) {
return arg0.getKey().compareTo(arg1.getKey());
}
});
double maxJump = -1.0;
Integer jumpIdx = -1;
Double oldD = 0.0; // base case !?
for (final Pair pair : clusterCountToDistortion) {
final Double jump = pair.getValue() - oldD;
if (jump > maxJump) {
maxJump = jump;
jumpIdx = pair.getKey();
}
oldD = pair.getValue();
}
return jumpIdx;
}
}
public static class DistortionDataAdapter implements DataTypeAdapter {
public static final String ADAPTER_TYPE_NAME = "distortion";
private static final String DISTORTION_FIELD_NAME = "distortion";
private static final FieldDescriptor DESC =
new FieldDescriptorBuilder<>(Double.class).fieldName(DISTORTION_FIELD_NAME).build();
private static final FieldDescriptor[] DESC_ARRAY = new FieldDescriptor[] {DESC};
public DistortionDataAdapter() {
super();
}
@Override
public String getTypeName() {
return ADAPTER_TYPE_NAME;
}
@Override
public byte[] getDataId(final DistortionEntry entry) {
return entry.getDataId();
}
@Override
public FieldReader
| [](https://opensource.org/licenses/Apache-2.0) | [](https://gitter.im/locationtech/geowave?utm_source=badge&utm_medium=badge&utm_content=badge) |
GeoWave is an open source set of software that:
* Capabilities
* Adds multi-dimensional indexing capability to key/value stores (currently [Apache Accumulo](https://accumulo.apache.org), [Apache HBase](https://hbase.apache.org), [Apache Cassandra](http://cassandra.apache.org/), [Amazon DynamoDB](https://aws.amazon.com/dynamodb/), [Cloud Bigtable](https://cloud.google.com/bigtable/), [Redis](https://redis.io/), [RocksDB](https://rocksdb.org/), and [Apache Kudu](https://kudu.apache.org/), as well as direct FileSystem support)
* Adds support for geographic objects and geospatial operators to these stores
* Provides Map-Reduce input and output formats for distributed processing and analysis of geospatial data
* Geospatial software plugins
* [GeoServer](http://geoserver.org/) plugin to allow geospatial data in various key/value stores to be shared and visualized via OGC standard services
Basically, GeoWave is working to bridge geospatial software with modern key/value stores and distributed compute systems.
## The Docs
* [GeoWave](https://locationtech.github.io/geowave/latest/index.html) - Latest snapshot documentation homepage
* [GeoWave Overview](https://locationtech.github.io/geowave/latest/overview.html) - Overview of GeoWave's capabilities
* [Installation Guide](https://locationtech.github.io/geowave/latest/installation-guide.html) - Installation instructions for standalone installers and from RPMs
* [Quickstart Guide](https://locationtech.github.io/geowave/latest/quickstart.html) - A quick demo of GeoWave features using the command-line interface
* [User Guide](https://locationtech.github.io/geowave/latest/userguide.html) - A guide for using GeoWave through the command-line interface and GeoServer plugin
* [Developer Guide](https://locationtech.github.io/geowave/latest/devguide.html) - A guide for developing applications that utilize GeoWave
* [Command-Line Interface](https://locationtech.github.io/geowave/latest/commands.html) - Full documentation for the GeoWave CLI
* [Changelog](https://locationtech.github.io/geowave/latest/changelog.html) - Changes and features for each of our [GitHub releases](https://github.com/locationtech/geowave/releases)
* The underlying principles employed in GeoWave are outlined in past academic publications to include largely the background theory in [Advances in Spatial and Temporal Databases 2017](https://link.springer.com/chapter/10.1007/978-3-319-64367-0_6) and a derivative, more applied paper in [FOSS4G Conference Proceedings 2017](http://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1027&context=foss4g).
## The Software
* We have [multi-platform standalone installers](https://locationtech.github.io/geowave/latest/installation-guide.html#standalone-installers) for the GeoWave's command-line tools to help get started
* This is often the quickest and easiest way to get started using GeoWave on your own machine
* We have a [RPM repository](https://locationtech.github.io/geowave/latest/downloads.html)
* This contains various packages including puppet modules, best used for distributed environments.
* See the [Installation Guide](https://locationtech.github.io/geowave/latest/installation-guide.html#installation-from-rpm) for more info.
* Maven artifacts are available on Maven Central
* And you can always [build from source](https://locationtech.github.io/geowave/latest/devguide.html#development-setup)
## Community
* Community support is available on [chat](https://gitter.im/locationtech/geowave) and on [our mailing list](mailto:geowave-dev@eclipse.org).
## Getting Started
### Programmatic Access
You can use Maven to reference pre-built GeoWave artifacts with the following pom.xml snippet (replacing `${keyvalue-datastore}` with your data store of choice and `${geowave.version}` with the GeoWave version you'd like to use):
```
