Showing preview only (482K chars total). Download the full file or copy to clipboard to get everything.
Repository: awslabs/amazon-kinesis-aggregators
Branch: master
Commit: a9e2f60bfdb3
Files: 100
Total size: 29.7 MB
Directory structure:
gitextract_tmef7pg6/
├── .gitignore
├── LICENSE.txt
├── NOTICE.txt
├── README.md
├── assembly.xml
├── dist/
│ ├── AmazonKinesisAggregators.jar-complete.jar
│ ├── AmazonKinesisAggregators.war
│ ├── amazon-kinesis-aggregators-.9.2.8.jar
│ ├── amazon-kinesis-aggregators-.9.2.9-sources.jar
│ └── amazon-kinesis-aggregators-.9.2.9.jar
├── pom.xml
├── sample/
│ ├── bin/
│ │ └── run-producer.sh
│ ├── java/
│ │ ├── model/
│ │ │ ├── SensorReading.java
│ │ │ └── SensorState.java
│ │ └── producer/
│ │ └── SensorReadingProducer.java
│ └── resources/
│ ├── BySegment-CSV.json
│ ├── BySegment-Json.json
│ └── BySegment-Regex.json
└── src/
├── .gitkeep
├── log4j.properties
└── main/
├── WebContent/
│ ├── .ebextensions/
│ │ └── as.config
│ ├── META-INF/
│ │ └── MANIFEST.MF
│ ├── WEB-INF/
│ │ └── web.xml
│ ├── index.html
│ └── styles/
│ └── styles.css
└── java/
└── com/
└── amazonaws/
└── services/
└── kinesis/
├── aggregators/
│ ├── AggregateData.java
│ ├── AggregatorGroup.java
│ ├── AggregatorType.java
│ ├── AggregatorsConstants.java
│ ├── EnvironmentType.java
│ ├── IStreamAggregator.java
│ ├── InputEvent.java
│ ├── InventoryModel.java
│ ├── InventoryStatus.java
│ ├── LabelSet.java
│ ├── StreamAggregator.java
│ ├── StreamAggregatorUtils.java
│ ├── TableKeyStructure.java
│ ├── TimeHorizon.java
│ ├── annotations/
│ │ ├── Aggregate.java
│ │ ├── AnnotationProcessor.java
│ │ ├── DateValue.java
│ │ ├── Label.java
│ │ └── Summary.java
│ ├── app/
│ │ ├── AbstractQueryServlet.java
│ │ ├── AggregatorsBeanstalkApp.java
│ │ ├── DateQueryServlet.java
│ │ ├── FetchConfigurationServlet.java
│ │ ├── ListAggregateKeysServlet.java
│ │ ├── QueryByLabelServlet.java
│ │ ├── ShowConfigFileServlet.java
│ │ └── ShowConfigurationServlet.java
│ ├── cache/
│ │ ├── AggregateCache.java
│ │ ├── UpdateKey.java
│ │ └── UpdateValue.java
│ ├── cli/
│ │ └── AggregatorsCli.java
│ ├── configuration/
│ │ ├── ConfigFileUtils.java
│ │ ├── DataExtractor.java
│ │ ├── ExternalConfigurationModel.java
│ │ └── json.schema
│ ├── consumer/
│ │ └── AggregatorConsumer.java
│ ├── datastore/
│ │ ├── AggregateAttributeModification.java
│ │ ├── DevNullDataStore.java
│ │ ├── DynamoDataStore.java
│ │ ├── DynamoQueryEngine.java
│ │ ├── DynamoUtils.java
│ │ └── IDataStore.java
│ ├── exception/
│ │ ├── ClassNotAnnotatedException.java
│ │ ├── InvalidConfigurationException.java
│ │ ├── SerializationException.java
│ │ └── UnsupportedCalculationException.java
│ ├── factory/
│ │ ├── CSVAggregatorFactory.java
│ │ ├── ExternallyConfiguredAggregatorFactory.java
│ │ ├── JsonAggregatorFactory.java
│ │ ├── ObjectAggregatorFactory.java
│ │ └── RegexAggregatorFactory.java
│ ├── idempotency/
│ │ ├── DefaultIdempotencyCheck.java
│ │ └── IIdempotencyCheck.java
│ ├── metrics/
│ │ ├── CloudWatchMetricsEmitter.java
│ │ ├── IMetricsEmitter.java
│ │ └── MetricsEmitterThrottledException.java
│ ├── processor/
│ │ ├── AggregatorProcessor.java
│ │ └── AggregatorProcessorFactory.java
│ └── summary/
│ ├── SummaryCalculation.java
│ ├── SummaryConfiguration.java
│ └── SummaryElement.java
└── io/
├── AbstractDataExtractor.java
├── CsvDataExtractor.java
├── IDataExtractor.java
├── JsonDataExtractor.java
├── ObjectExtractor.java
├── RegexDataExtractor.java
├── StringDataExtractor.java
└── serializer/
├── CsvSerializer.java
├── IKinesisSerializer.java
├── JavaSerializationSerializer.java
├── JsonSerializer.java
├── RegexSerializer.java
├── SerializationUtils.java
└── StringSerializer.java
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
/bin/
/target/
.classpath
.project
.settings/
================================================
FILE: LICENSE.txt
================================================
http://www.apache.org/licenses/LICENSE-2.0.html
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
You must give any other recipients of the Work or Derivative Works a copy of this License; and
You must cause any modified files to carry prominent notices stating that You changed the files; and
You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
Note: Other license terms may apply to certain, identified software files contained within or distributed with the accompanying software if such terms are included in the directory containing the accompanying software. Such other license terms will then apply in lieu of the terms of the software license above.
END OF TERMS AND CONDITIONS
================================================
FILE: NOTICE.txt
================================================
amazon-kinesis-aggregators
Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at
http://aws.amazon.com/apache2.0/
or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
================================================
FILE: README.md
================================================
# Amazon Kinesis Aggregators
----
*This project is now deprecated, and only updates for security vulnerabilities in dependencies will be made. We advise the use of Apache Flink on Amazon Kinesis Analytics instead.*
----
Amazon Kinesis Aggregators is a Java framework that enables the automatic creation of real-time aggregated time series data from Amazon Kinesis streams.
You can use this data to answer questions such as ‘how many times per second has ‘x’ occurred’ or ‘what was the breakdown by hour over the day of the streamed data containing ‘y'. Using this framework, you simply describe the format of the data on your stream (CSV, JSON, and so on), the granularity of times series that you require (seconds, minutes, hours, and so on), and how the data elements that are streamed should be grouped; the framework handles all the time series calculations and data persistence. You then simply consume the time series aggregates in your application using Amazon DynamoDB, or interact with the time series using Amazon CloudWatch or the Web Query API.
You can also analyze the data using Hive on Amazon Elastic MapReduce, or bulk import it to Amazon Redshift. The process runs as a standalone Amazon Kinesis-enabled application which only requires configuration, or can be integrated into existing Amazon Kinesis applications.
The data is stored in a time series based on how you aggregate it. A dataset aggregating Telecoms Call Data Records in DynamoDB might look like this:
The corresponding data in CloudWatch would look like this:
## Building Aggregators
Amazon Kinesis Aggregators is built using Apache Maven. To build, simply run Maven from the amazon-kinesis-aggregators directory. The target directory contains the following build artifacts:
* **amazon-kinesis-aggregators-.9.2.7.4.jar** - Includes no compiled dependencies
* **AmazonKinesisAggregators.jar-complete.jar** - Includes all required dependencies
* **AmazonKinesisAggregator.war** - The web application archive file
## Running Aggregators
Amazon Kinesis Aggregators ships with several deployment options, which should enable you to run with minimal operational overhead while also accommodating advanced deployment use cases. You can run Amazon Kinesis Aggregators as:
* A fully-managed Elastic Beanstalk application. All you need to do is deploy the KinesisAggregators.war file, and provide a configuration file that is accessible using HTTP.
* A managed Java client, running through the host orchestration of your choice. For example, you can deploy this managed Java client as part of an Amazon EC2 fleet that uses Auto Scaling.
* As part of an existing Amazon Kinesis-enabled application. This enables an existing application to 'sideload' aggregator processing, as an augmentation to an already-established application.
### Running Amazon Kinesis Aggregators Using Elastic Beanstalk
Amazon Kinesis Aggregators compiles a web application archive (WAR) file, which enables easy deployment on Java application servers, such as Apache Tomcat, using Elastic Beanstalk (http://aws.amazon.com/elasticbeanstalk). Amazon Kinesis Aggregators also includes configuration options that instruct Elastic Beanstalk to scale the application on CPU load, which is typically the bottleneck for applications as they scale up. This is the recommended deployment method.
To deploy Amazon Kinesis Aggregators as an Elastic Beanstalk application, start by creating a new Elastic Beanstalk web server application with the pre-configured Tomcat stack. When prompted by the AWS Management Console, upload the KinesisAggregators.war file from your local build. Select an instance type that is suitable for the type of aggregation that you are running (specifically, the higher the granularity of label items and the more fine-grained the TimeHorizon value, the larger the instance type you require). After deployment, click the URL for the application environment; the following message is displayed:
```OK - Kinesis Aggregators Managed Application hosted in Elastic Beanstalk Online ```
Furthermore, if you request a log snapshot from the Elastic Beanstalk console, you see a log line indicating the following:
```No Aggregators Configuration File found in Beanstalk Configuration config-file-url. Application is Idle```
This indicates that the application is deployed but not configured. To configure the application, add these Elastic Beanstalk configuration parameters as required:
* **stream-name** - The name of the stream.
* **application-name** - The name of the Amazon Kinesis application.
* **failures-tolerated** - The number of worker exceptions allowed before the worker terminates.
* **position-in-stream** - The position in the stream to start consuming data from. The possible values are 'LATEST' and 'TRIM_HORIZON'.
* **max-records** - The maximum number of records to consume from a stream in a single cycle. You can set this value if your stream processing (in addition to aggregation) is slow.
* **region** - The region to use for the stream, the DynamoDB lease tables, and the CloudWatch and aggregate data stores. Amazon Kinesis Aggregators does not currently support cross-region deployment.
* **environment** - The name of the environment. This ensures that all DynamoDB tables are prefixed with the environment, enabling you to keep data sets separate for test and production (for example).
* **config-file-url** - The URL for the configuration file.
This is typically done by adding `-D` flags to the JVM command line options. Then, choose 'Save' and Elastic Beanstalk applies the changes to the environment. Wait a minute or so, and then snapshot logs to confirm that Amazon Kinesis Aggregators is running.
### Running the Managed Java Client Application
This is a great option if you have data in Amazon Kinesis, but don’t want to use Elastic Beanstalk. You can start the application from a server using the following command:
```java -cp AmazonKinesisAggregators.jar-complete.jar -Dconfig-file-path=<configuration> -Dstream-name=<stream name> -Dapplication-name=<application name> -Dregion=<region name - us-east-1, eu-west-1, etc> com.amazonaws.services.kinesis.aggregators.consumer.AggregatorConsumer```
In addition to the configuration items outlined in the Elastic Beanstalk section, use the following configuration item:
* **config-file-path** - The path to the configuration file.
We recommend that you run your servers in an Auto Scaling group to ensure fault tolerance if the host fails.
### Configuration
You can use the configuration file to create one or more aggregations against the same stream. It is a JSON file that creates a set of aggregator objects managed by the framework. Create one aggregator for each distinct label that you want to aggregate on. Each aggregator can then have its own properties of time granularity, aggregator type, and so on.
The core structure of the configuration file is an array of aggregator objects. For example, the following configuration creates two aggregators:
```[{aggregatorDef1}, {aggregatorDef2}]```
Note that aggregatorDef*N* is an aggregator configuration. An aggregator configuration must include the following attributes:
* **namespace** (String) - Enables you to create separate time series data stores. This namespace is used with the application name and environment to create the underlying data tables for the time series, as well as the namespace for custom CloudWatch metrics. Use something that's meaningful based upon the label and time granularity.
* **labelItems** (array<String>) - Includes a list of the elements of the data stream to aggregate on. The data stored in the time series is aggregated by the unique values from the stream for these attributes, and by time. For instance, to aggregate data for searches made against a car website, you might have a label item set of ["Make","Model","Year"]. If you are using CSV data, then this same configuration might be positional based on the fields in the line, such as [0,3,5].
* **labelAttributeAlias** (String) - Enables you to name the target database attribute for the label. This is particularly useful when you are using CSV or regex-extracted data, and would otherwise end up with a label attribute named the same as the label attribute index.
* **type** (enum) - The type of aggregation to run. The available types are 'COUNT' and 'SUM'. Counting aggregators simply counts the instances of unique values in Label Items by time. Using the previous example, it would generate a count of searches by configured time period for each unique combination of Make, Model, and Year. Building on this 'SUM' type, aggregators also calculate summaries of other numeric values on the stream. For more information, see the configuration option **'summaryItems**'.
* **timeHorizons** (array<enum>) - Because the data is captured as a time series, you must tell the aggregator which definition of time you require. To have the data on the stream aggregated by minute, specify 'MINUTE'. To put data into buckets of 5 minutes duration, specify MINUTES_GROUPED(5). You can specify multiple timeHorizon values, and the aggregator automatically maintains the time series data at that granularity. A common configuration might be ["SECOND","HOUR","FOREVER"], which gives per-second aggregates, a rollup by Hour, and a simple data set to view everything that ever occurred in a single value. The possible values are:
* SECOND
* MINUTE
* MINUTES\_GROUPED(int minutePeriod) - Groups data into time buckets using a minute period. For 4 buckets per hour, use '15', or use '5' for buckets of aggregation that are 5 minutes long.
* HOUR
* DAY
* MONTH
* YEAR
* FOREVER - Rolls up everything that occurred in a single value '*'.
* **dataExtractor** (enum) - Tells the aggregator how to parse and extract the Label Items from your stream. Currently, the following data formats are supported for external configurations using the configuration file:
* **CSV** - Character-separated UTF-8 data. The default delimiter is a comma. To override the delimiter, set the configuration option 'delimiter' to the character value to use as the field terminator. Also, note that all data extractors support multi-value events. This means that you can have many CSV 'lines' within a single event, which are extracted with a line terminator of "\n". To override the line terminator on any data extractor that is text-based, set the configuration option 'lineTerminator' to the character to use as the line terminator. When this data extractor is used, indicate the Label Items using zero-index position values of the fields.
* **JSON** - UTF-8 encoded JSON data. This data can either reside in a JSON array on the event (for example [{object1},{object2},{object3}]) or can be a single object per 'line' (for example {object1}\n {object2}). To control the object delimiter, use the configuration option 'lineTerminator'.
* **REGEX** - UTF-8 encoded strings of arbitrary data. With this configuration option, you must include the 'regularExpression' configuration option. This data extraction method also uses zero-indexed positional values for Label Items.
* **OBJECT** - Serialized objects using Jackson JSON binary data. With this configuration option, you must include the 'class' configuration option. Using this data extraction method, an event can include only 1 serialized object.
* **dateItem** (String) - The attribute or field index that defines when the event occurred. This is used to generate the aggregate for the correct time period for the event. This can be formatted as a long value of epoch seconds, or a String value. If you provide a String value in the event, you must also set the configuration option 'dateFormat'. If it is omitted, then the timestamp of the event is set to the timestamp of the server instance when it processes the item.
* **dateAttributeAlias** (String) - Similar to labelAttributeAlias, this enables you to set the name of the date attribute in the aggregated data table.
* **dateFormat** (String) - The date format of the dateItem, using date format strings as specified at ```http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html```.
You can also include the following options in the configuration:
* **summaryItems** (array<String>) - If the aggregator 'type' is SUM, then the aggregator automatically performs a time series aggregation on the summary items configured. These summary items must be numeric values that you want aggregated for the indicated time period and values of Label Items. For example, if your stream includes call data records, you might want to sum the duration of all calls made by mobile network by hour. Along with configuration of the Label Items and time, you would include a summary item of 'callDuration'. As with the configuration of the Label Items, summary items are zero-index positional values for CSV and regex data extractions, attribute names for JSON, and method names for OBJECT. For more information, see the Summary Items Mini-Language section.
* **filterRegex** (String) - Filters the stream data using a filtering regular expression. If provided, only the data that conforms to the regular expression is passed in for subsequent parsing. Note this step is applied on the raw underlying stream data as String values, and is not available for object serialization.
* **failOnDataExtraction** (boolean) - By default, the aggregator fails if it can't understand the data on the stream, to ensure that all events are properly accounted for. If you have a data stream that contains internally inconsistent data, and you want to perform a simple aggregation whenever you can successfully parse the data stream, set this value to 'true'. Alternatively, consider writing a **filterRegex** expression that extracts only the data that fits the configuration of Label, Date, and Summary Items.
* **tableName** (String) - Sets the name of the underlying time series data table in the data store.
* **environment** (String) - Runs an aggregator with a specified environment type. This enables you to separate the underlying data stores used for the time series data into production and test, for example.
* **readIOPS** (int) - Use with the default DynamoDB IDataStore to set up the number of read I/O operations per second (IOPS) you want on the time series data store.
* **writeIOPS** (int) - Use with the default DynamoDB IDataStore to set up how many write IOPS you want on the time series data store.
* **IDataStore** (String) - Configures alternative backing data stores other than DynamoDB. If you have written your own data store implementation, specify the full class name, including the package, to have this data store used. You can also specify the internal alternate data store 'com.amazonaws.services.kinesis.aggregators.datastore.DevNullDataStore', which does NOT store the time series data, and is useful only to consume the time series from CloudWatch.
* **emitMetrics** (boolean) - Emits the time series aggregated data as a custom CloudWatch domain of metrics. Set this value to 'true' to create a custom CloudWatch metric for the application name and namespace of the aggregator, with dimensions on the label and summary items.
### Summary Items Mini-Language
You can configure summary items and the type of summary using a miniature specification language, and navigate complex document structures in JSON data. You can apply the following type of summary transformations:
* **SUM** - Applies the default summary if you do not specify a summary type. This sums up all values seen for label and time values.
* **MIN** - Calculates the minimum value observed for the time period and label values.
* **MAX** - Calculates the maximum value observed for the time period and label values.
* **FIRST** - Stores the first observed value for the time period and label values.
* **LAST** - Is equal to the latest value for the time period and label values.
Summary items can have aliases applied, as in SQL, to control the name of the generated attribute in the data store you write to. You simply add the name of the item you require to the definition of the summary item, including functions.
You can also navigate an entity structure in a JSON-formatted stream data using dot notation; for example, given the following object, you can access the calculated duration using a summary item of 'timeValues.durations.calculated':
```
{
"name": "Object To Be Aggregated",
"timeValues": {
"durations": {
"calculated": 60,
"recorded": 58
},
"endTime": "01/01/1970 01:00:00",
"startTime": "01/01/1970 00:00:00"
}
}
```
These concepts can be combined into a mini-specification:
Example 1 - Calculate the min, max, and sum of value 7 in a CSV stream, giving them friendly names - ```["min(7) min-purchase-price","max(7) max-purchase-price","sum(7) total-sales]"```
Example 2 - Calculate the sum and maximum value of the calculated duration in the JSON stream - ```["sum(timeValues.durations.calculated)","max(timeValues.durations.calculated)"]```
### Sample Configurations
* **JSON** - http://amazon-kinesis-aggregators.s3.amazonaws.com/sample/json-aggregator.json
* **CSV** - http://amazon-kinesis-aggregators.s3.amazonaws.com/sample/csv-aggregator.json
* **Regular Expression** - http://amazon-kinesis-aggregators.s3.amazonaws.com/sample/regex-aggregator.json
* **Object Serialized Data** - http://amazon-kinesis-aggregators.s3.amazonaws.com/sample/object-aggregator.json
### Aggregator Data Structure
The data structure for aggregated data is arranged as a hash/range table in DynamoDB on the Label attributes and Date attribute at the configured granularity of time. Every table also includes the following:
* **eventCount** - The number of events consumed during the period.
* **lastWriteSeq** - The last sequence value from the Amazon Kinesis stream that generated an update to the time period and aggregate label.
* **lastWriteTime** - The time on the consumer application when the update was made to the aggregate data.
* **scatterPrefix** - A random number between 0 and 99 used to ensure that there are no write bottlenecks on global secondary indexes for the time period and last write sequence.
Of course, the table also includes any summary values that were added to the aggregator configuration. The format of these summary attributes in DynamoDB follow the pattern <attribute>-<summary type>, or use the alias provided.
* For JSON streams, the attribute is the attribute name configured.
* For object-serialized streams, the attribute is the summary method converted to a user-friendly name. For example 'getComputedValue' is written to the data store as 'computedValue'.
* For CSV and String data parsed using regular expressions, the attribute value is the position in the stream, indexed from 0.
* The summary type is one of the following values: MIN, MAX, SUM, FIRST, or LAST.
#### Indexes
All aggregator data stores have global secondary indexes (logically) on the date value and on lastWriteSeq. To ensure adequate write performance, these indexes are structured as hash/range on the scatterPrefix (a random number between 0 and 99) and the value is indexed.
#### Web-based Query API
The Amazon Kinesis Aggregators web application also provides several query API operations, which return data in the JSON format. When deployed, you can make an HTTP request to a variety of endpoints to retrieve different types of data. Currently, there is no security offered for the Web API operations, so you must ensure that they are only accessible from within your VPC using security group rules or similar. Do NOT make these endpoints publicly accessible.
##### Viewing the Running Configuration
You can view the configuration of your aggregators at the URL ```<web application>/configuration```, which returns an object such as:
```
{
"application-name": "EnergyRealTimeDataConsumer",
"config-file-url": "s3://mybucket/kinesis/sensor-consumer-regex.json",
"environment": null,
"failures-tolerated": null,
"max-records": "2500",
"position-in-stream": "LATEST",
"region": "eu-west-1",
"stream-name": "EnergyPipelineSensors",
"version": ".9.2.7.4"
}
```
##### Date-based Queries
Use the Date query to find data that has been aggregated on the basis of the stream timestamp value. For example, use this interface to periodically retrieve all new data that has been processed, or to pull data for specific time ranges for comparative analysis. The URL is:
```
<web application>/dateQuery?params
```
Parameters:
* **namespace** - The namespace for the aggregator configuration.
* **operator** - The condition to query for, from the DynamoDB ComparisonOperator enum: EQ, GT, GE, and so on. Note that BETWEEN is not yet supported.
* **granularity** – The granularity of time required, from the TimeHorizon enum: SECOND, MINUTE, HOUR, and so on.
* **date–value** – The date value to query relative to, in yyyy-MM-dd+hh:mm:ss format (for example, 2014–09–01+18:00:00).
This returns all data from the aggregated table for the date period specified.
You can also use the internal Java API:
```
public List<Map<String, AttributeValue>> queryByDate(Date dateValue, TimeHorizon h,
ComparisonOperator comp, int threads) throws Exception
```
This method queries by the Date, TimeHorizon, and ComparisonOperator values you select. For example, to find all hourly aggregates after 3pm, use:
```
dateValue=Date('2014-01-01 15:00:00'), TimeHorizon.HOUR, ComparisonOperator.GT
```
The Threads parameter is the number of threads used to do the query. This is due to the index being organized on hash/range of scatterPrefix/DateValue.
##### Query for Label/Date Values
To query the application to find the unique set of labels and date values that have been aggregated, use the following URL:
```<web application>/keyQuery?params```
Parameters:
* **namespace** - The namespace for the aggregator configuration.
* **scope** - Use 'HashKey' to get just the unique aggregate label values or 'HashAndRangeKey' to get both the label and date values.
This returns a unique list of all keys from the aggregated table.
You can also use the internal Java API:
```
public Map<String, AttributeValue> queryValue(String label, Date dateValue, TimeHorizon h)
throws Exception
```
This method takes the label you are interested in, as well as a date for the date value. If you have multiple TimeHorizon values configured on the aggregator, it generates the correct dateValue to query the underlying table with. You are likely to use this interface to query across aggregator data stores looking for related time-based values.
## Integrating Aggregators into Existing Java Applications
In addition to running aggregators as stand-alone Amazon Kinesis applications, you can integrate them into existing Amazon Kinesis applications. You can:
* Run the managed consumer from an existing control environment
* Inject a set of aggregators into a managed IRecordProcessorFactory
* Use an existing IRecordProcessor to send data to one or more aggregators
### Managed IRecordProcessorFactory
To build your Amazon Kinesis worker and configure it explicitly, you can still use aggregators to create IRecordProcessorFactory. In this case, simply create a new instance of com.amazonaws.services.kinesis.aggregators.processor.AggregatorProcessorFactory with the configured aggregators.
### Integration with Existing IRecordProcessors
If you have an existing worker application and you simply want to add the aggregation capability, you can directly integrate with one or more aggregators. To do this, simply construct the aggregators using a configuration file, or using a pure Java configuration. Then, to inject new data into the aggregator, simply call:
```void aggregate(List<record> records)```
This causes the time series calculations to be done based upon the configuration of the aggregators. Then, when your worker normally calls checkpoint(), also call:
```void checkpoint()```
This flushes the in-memory time series state to the backing data store. You must ensure that the aggregators are initialized correctly against the shard for the worker by calling this method in the existing KCL Application IRecordProcessor initialize() method:
```void initialize(String shardId)```
You must also ensure that if the shutdown() method is invoked on your Amazon Kinesis application, you call:
```void shutdown(boolean flushState)```
If the shutdown reason specified in the shutdown method for IRecordProcessor is ShutdownReason.ZOMBIE, set flushState to 'false' to allow the data to be re-aggregated by another worker. However, if the value is ShutdownReason.TERMINATE, you should flush the aggregator state on termination.
### Configuring Aggregators in Existing Applications
There are a variety of ways to configure aggregators when you are integrating into existing applications. You might use a factory to create one or more aggregators from a simple set of arguments, or you can configure each aggregator directly and manage it as part of an aggregator group.
#### Aggregator Factories
There are a variety of aggregator factories available in the com.amazonaws.services.kinesis.aggregators.factory package, which generally map to the configuration types found in the configuration file. In fact, you can use configuration files to configure aggregators from Java using the following:
```
ExternallyConfiguredAggregatorFactory.buildFromConfig(
String streamName,
String applicationName,
KinesisClientLibConfiguration config,
String configFile)
```
You can also take advantage of aggregators that are specific to the type of data to be aggregated:
##### JSON Data
```
JsonAggregatorFactory.newInstance(String streamName
, String appName
, KinesisClientLibConfiguration config
, String namespace
, TimeHorizon timeHorizon
, AggregatorType aggregatorType
, List<string> labelAttributes
, String dateAttribute
, String dateFormat
, List<string> summaryAttributes)
```
##### CSV Data
```
CsvAggregatorFactory.newInstance(String streamName
, String appName
, KinesisClientLibConfiguration config
, String namespace
, TimeHorizon timeHorizon
, AggregatorType aggregatorType
, String delimiter
, List<integer> labelIndicies
, int dateIndex
, String dateFormat
, List<object> summaryIndicies)
```
##### String Data parsed with Regular Expressions
```
RegexAggregatorFactory.newInstance(String streamName
, String appName
, KinesisClientLibConfiguration config
, String namespace
, List<timehorizon> timeHorizons
, AggregatorType aggregatorType
, String regularExpression
, List<integer> labelIndicies
, int dateIndex
, String dateFormat
, List<object> summaryIndicies)
```
##### Object Serialized Data
You can generate aggregators for object-serialized data using annotations:
```
ObjectAggregatorFactory.newInstance(String streamName
, String appName
, KinesisClientLibConfiguration config
, Class clazz)
```
Note that 'clazz' is a class that has been configured using annotations found in the com.amazonaws.services.kinesis.aggregators.annotations package. This factory method throws an error if the class is not annotated.
Alternatively, you can configure the aggregator directly:
```
ObjectAggregatorFactory.newInstance(String streamName
, String appName
, KinesisClientLibConfiguration config
, String namespace
, List<TimeHorizon> timeHorizons
, AggregatorType aggregatorType
, Class clazz
, List<String> labelMethods
, String dateMethod
, List<String> summaryMethods)
```
#### Direct Configuration
If you want even more control over the configuration of a given set of aggregators, then you can configure them directly. To effectively do this, you must understand how aggregators work. Aggregators are built around several subsystems that their factory methods configure automatically. When you build aggregators directly, you must construct an aggregator from its constituent subsystems. For more information, see the 'Extending Aggregators' section of this document.
To configure an aggregator directly, you must configure two of the subsystems: the aggregator and the IDataExtractor that extracts the data from the stream.
##### IDataExtractor
When you create an aggregator directly, you must specify the IDataExtractor to get data out of the stream for aggregation. There are IDataExtractors in the com.amazonaws.services.kinesis.aggregators.io package. Each of these map to the supported data formats, and provide relevant configuration options, including label, Date, and summary items. IDataExtractors use fluent builders for all optional configurations. For example, creating a JsonDataExtractor looks like this:
```
new JsonDataExtractor(labelAttributes)
.withDateValueAttribute(dateAttribute)
.withSummaryAttributes(summaryAttributes)
.withDateFormat(dateFormat);
```
##### Aggregator
You then create the aggregator with the options that are specific to it, including KinesisClientLibConfiguration, required TimeHorizon values, and options for emitting metrics. For example, using the example JsonDataExtractor, you might configure the aggregator as follows:
```
return new StreamAggregator(streamName, appName, namespace, config, dataExtractor)
.withTimeHorizon(timeHorizons)
.withAggregatorType(aggregatorType)
.withCloudWatchMetrics(true);
```
## Extending Aggregators
You might want to extend aggregators for a variety of reasons. The use cases that we know of today that will require extension include supporting data on a stream that is compressed, encrypted, and uses an object serialization format other than Jackson/JSON, or implementing large objects. We designed aggregators with extensibility in mind. You can extend the framework at the following integration points.
### Data Format & Handling
The ability to support CSV, JSON, arbitrary string data and object serialization is provided by the IDataExtractor and IKinesisSerializer interfaces, residing at com.amazonaws.services.kinesis.aggregators.io and io.Serializer.
#### IKinesisSerializer
This interface interoperates between the internal data format used by IDataExtractors, and byte arrays are used on the stream. You implement IKinesisSerializer to support compressed stream data or if your data is encrypted, for example. The implementation would conform to the following interface, which is identical to the Amazon Kinesis Connector ITransformer class:
```
/**
* Transforms data from a Record (byte array) to the data
* model class (T) for processing in the application and from the data model
* class to the output type (U) for the emitter.
*
* @param <T> the data type stored in the record
*/
public interface IKinesisSerializer<T, U> {
/**
* Transform the record into an object of its original class.
*
* @param record raw record from the stream
* @return data using its original class
* @throws IOException if it could not convert the record to a T
*/
public T toClass(InputEvent event) throws IOException;
/**
* Transform the record from its original class to a byte array.
*
* @param record data as its original class
* @return a data byte array
*/
public U fromClass(T record) throws IOException;
}
```
#### IDataExtractor
IDataExtractors take the deserialized data and extract the relevant Label, Date, and Summary items. They also typically do any filtering that is exposed by the IDataExtractor. Implement a new IDataExtractor if the type of data returned by a custom IKinesisSerializer implementation is not compatible with the existing IDataExtractors in the io package. This new IDataExtractor would conform to:
```
/**
* Enables pluggable data extractors for different types of
* stream data. Aggregators use IDataExtractor to interoperate between the
* stream data format and the internal format required for aggregation.
* IDataExtractors likely use IKinesisSerializers to read and write to and from
* the stream
*/
public interface IDataExtractor {
/**
* Gets the name of the label value to be extracted.
*
* @return
*/
public String getAggregateLabelName();
/**
* Gets the name of the date value to be extracted.
*
* @return
*/
public String getDateValueName();
/**
* Extracts one or more aggregatable items from a Amazon Kinesis record.
*
* @param event The Amazon Kinesis record from which we want to extract data.
* @return A list of ExtractedData elements that have been resolved from
* the input data.
* @throws SerializationException
*/
public List<AggregateData> getData(InputEvent event) throws SerialisationException;
/**
* Sets the type of aggregator that contains this IDataExtractor. Used to
* boost efficiency in that the extractor will not extract summary items for
* COUNT-based aggregator integration.
*
* @param type
*/
public void setAggregatorType(AggregatorType type);
/**
* Validates that the extractor is well formed.
*
* @throws Exception
*/
public void validate() throws Exception;
/**
* Gets the summary configuration that is driving data extraction against the
* data stream.
*
* @return
*/
public SummaryConfiguration getSummaryConfig();
public IDataExtractor copy() throws Exception;
}
```
Also note that an IDataExtractor returns multiple aggregatable objects from the stream. If you had a requirement to support M:N Kinesis Events to Aggregatable Events, an IDataExtractor could do the job using local state.
Note that the IDataExtractor is STATEFUL for the life of an aggregator running on a shard, and contains the configuration of the data that is to be extracted. Because a new IDataExtractor is generated when a new aggregator is initialized on a shard, you must ensure that it is thread-safe and implement the copy() interface correctly to ensure that multiple instances can operate within a single JVM.
### Data Store
The Amazon Kinesis Aggregators framework backs its data onto DynamoDB, and takes advantage of powerful DynamoDB features such as hash/range keys, atomic increment, and conditional updates. It also implements a defensive flush mechanism, which means that at any provisioned I/O rate, the aggregator can flush its state to DynamoDB without timing out.
To extend aggregators with support for an alternate backing store, such as a relational database or Redis, implement com.amazonaws.services.kinesis.aggregators.datastore.IDataStore. This implementation must meet the following service levels:
* Flushes all internal state to the data store in 5 minutes or less (this is due to the Amazon Kinesis worker timeout)
* Supports a composite primary key for all label values and date value
* Performs an atomic, transactional increment operation
* Conditionally updates a discrete value in the table
The implementation of a new IDataStore must conform to the following:
```
/**
* Enables the in-memory cached aggregates
* to be saved to a persistent store
*/
public interface IDataStore {
/**
* Writes a set of Update key/value pairs to the backing store
*
* @param data The input dataset to be updated
* @return A data structure that maps a set of
* AggregateAttributeModifications to the values that were
* affected on the underlying data store, by UpdateKey
* @throws Exception
*/
public Map<UpdateKey, Map<String, AggregateAttributeModification>> write(
Map<UpdateKey, UpdateValue> data) throws Exception;
/**
* Method called on creation of the IDataStore
*
* @throws Exception
*/
public void initialize() throws Exception;
/**
* Method that is periodically invoked to allow the IDataStore to
* refresh tolerated limits for how often write() should be called
*
* @return
* @throws Exception
*/
public long refreshForceCheckpointThresholds() throws Exception;
/**
* Sets the region for the IDataStore
*
* @param region
*/
public void setRegion(Region region);
}
```
### Metrics Service
By default, Amazon Kinesis Aggregators integrates with CloudWatch for the purpose of metrics dashboards and alerts. However, you might want to push metrics to platforms such as Ganglia or New Relic. In these cases, you would provide an implementation of the com.amazonaws.services.kinesis.aggregators.metrics.IMetricsEmitter. This implementation would conform to the following:
```
/**
* Provides classes that can write to metrics services.
* Receives the output of the IDataStore modifications, and applies the data to
* the metrics service.
*/
public interface IMetricsEmitter {
/**
* Emits a new set of metrics to the metrics service
*
* @param metricData Input Data to be intrumented
* @throws Exception
*/
public void emit(Map<UpdateKey, Map<String, AggregateAttributeModification>> metricData)
throws Exception;
/**
* Sets the region of the metrics service
*
* @param region
*/
public void setRegion(Region region);
}
```
----
Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Amazon Software License (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at
http://aws.amazon.com/asl/
================================================
FILE: assembly.xml
================================================
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>complete</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<outputDirectory>/</outputDirectory>
<unpack>true</unpack>
<scope>runtime</scope>
</dependencySet>
</dependencySets>
</assembly>
================================================
FILE: dist/AmazonKinesisAggregators.jar-complete.jar
================================================
[File too large to display: 14.6 MB]
================================================
FILE: dist/AmazonKinesisAggregators.war
================================================
[File too large to display: 14.7 MB]
================================================
FILE: pom.xml
================================================
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-kinesis-aggregators</artifactId>
<version>.9.2.9</version>
<properties>
<sdk-version>1.11.745</sdk-version>
</properties>
<build>
<defaultGoal>clean source:jar install assembly:assembly war:war</defaultGoal>
<sourceDirectory>src/main/java</sourceDirectory>
<resources>
<resource>
<directory>src/main/java</directory>
<includes>
<include>**/*.properties</include>
</includes>
</resource>
<resource>
<directory>sample/java</directory>
</resource>
<resource>
<directory>sample/resources</directory>
<includes>
<include>**/*.properties</include>
</includes>
</resource>
</resources>
<testSourceDirectory>tst</testSourceDirectory>
<testResources>
<testResource>
<directory>tst</directory>
<filtering>false</filtering>
<includes>
<include>**/*.java</include>
<include>**/*.properties</include>
<include>**/*.json</include>
</includes>
</testResource>
</testResources>
<pluginManagement>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-war-plugin</artifactId>
<version>2.3</version>
<configuration>
<warSourceDirectory>src/main/WebContent</warSourceDirectory>
<failOnMissingWebXml>true</failOnMissingWebXml>
<warName>AmazonKinesisAggregators</warName>
</configuration>
</plugin>
<!-- delete this plugin to remove the AWS sample application from the
aggregator core build -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.8</version>
<executions>
<execution>
<id>add-extra-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>sample</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.1</version>
<configuration>
<finalName>AmazonKinesisAggregators.jar</finalName>
<descriptors>
<descriptor>assembly.xml</descriptor>
</descriptors>
</configuration>
</plugin>
<plugin>
<groupId>com.mycila.maven-license-plugin</groupId>
<artifactId>maven-license-plugin</artifactId>
<version>1.8.0</version>
<configuration>
<header>com/amazonaws/services/kinesis/aggregators/license.txt</header>
<properties>
<owner>Ian Meyers</owner>
<year>2014</year>
<email>meyersi@amazon.co.uk</email>
</properties>
<excludes>
<exclude>**/README</exclude>
<exclude>**/license.txt</exclude>
<exclude>src/test/resources/**</exclude>
<exclude>src/main/resources/**</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<!-- <parallel>classes</parallel> <threadCount>5</threadCount> -->
<reuseForks>false</reuseForks>
<forkCount>1</forkCount>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
<configuration>
<show>private</show>
<nohelp>true</nohelp>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>${sdk-version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>${sdk-version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-kinesis</artifactId>
<version>${sdk-version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-cloudwatch</artifactId>
<version>${sdk-version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-cloudwatchmetrics</artifactId>
<version>${sdk-version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>1.7.0</version>
<exclusions>
<exclusion>
<artifactId>aws-java-sdk</artifactId>
<groupId>com.amazonaws</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<version>20040616</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.2</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>[2.7,)</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>[2.9.10.1,)</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>[2.9.10.1,)</version>
</dependency>
</dependencies>
<organization>
<name>Amazon Web Services UK Ltd</name>
</organization>
</project>
================================================
FILE: sample/bin/run-producer.sh
================================================
#!/bin/bash
#
# Amazon Kinesis Aggregators
#
# Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Amazon Software License (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
# http://aws.amazon.com/asl/
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.
#
# Number of messages for this producer to create
num_messages=$1
# Format should be one of 'json','csv', or 'string'
format=$2
# Stream to write messages to
stream=$3
# AWS Region Name to use, such as 'us-east-1' or 'eu-west=1'. US East is Default
region=$4
java -cp ../../target/AmazonKinesisAggregators.jar-complete.jar producer.SensorReadingProducer $num_messages $format $stream $region
================================================
FILE: sample/java/model/SensorReading.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package model;
import com.fasterxml.jackson.databind.ObjectMapper;
public class SensorReading {
private static ObjectMapper mapper = new ObjectMapper();
public enum OutputFormat {
json, csv, string;
}
private OutputFormat outputAs = OutputFormat.json;
private String id;
private long captureTs;
private String segment;
private double lat;
private double lng;
private double pressure;
private double temperature;
private double flowRate;
private double corrosionIndex;
private double segmentIncline;
private SensorReading() {
}
public SensorReading(String id, String segment, long captureTs, double lat,
double lng, double pressure, double temperature, double flowRate,
double corrosionIndex, double segmentIncline) {
this.id = id;
this.segment = segment;
this.captureTs = captureTs;
this.lat = lat;
this.lng = lng;
this.pressure = pressure;
this.temperature = temperature;
this.flowRate = flowRate;
this.corrosionIndex = corrosionIndex;
this.segmentIncline = segmentIncline;
}
public String getId() {
return this.id;
}
public String getSegment() {
return this.segment;
}
public long getCaptureTs() {
return this.captureTs;
}
public double getLat() {
return this.lat;
}
public double getLng() {
return this.lng;
}
public double getPressure() {
return this.pressure;
}
public double getTemp() {
return this.temperature;
}
public double getFlowRate() {
return this.flowRate;
}
public double getCorrosionIndex() {
return this.corrosionIndex;
}
public double getSegmentIncline() {
return this.segmentIncline;
}
public SensorReading withOutputFormat(OutputFormat format) {
this.outputAs = format;
return this;
}
public String asJson() throws Exception {
return mapper.writeValueAsString(this);
}
public String asString() throws Exception {
return String.format("%s (%s) ts-%s %sx%s %s at %s T:%s c:%10f deg%10f",
this.id, this.segment, this.captureTs, this.lat, this.lng,
this.pressure, this.flowRate, this.temperature,
this.corrosionIndex, this.segmentIncline);
}
public String asCSV() throws Exception {
return String.format("%s|%s|%s|%s|%s|%s|%s|%s|%10f|%10f", this.id,
this.segment, this.captureTs, this.lat, this.lng,
this.pressure, this.temperature, this.flowRate,
this.corrosionIndex, this.segmentIncline);
}
@Override
public String toString() {
try {
switch (this.outputAs) {
case string:
return this.asString();
case csv:
return this.asCSV();
default:
return this.asJson();
}
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
}
================================================
FILE: sample/java/model/SensorState.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package model;
public class SensorState {
private String segment;
private double lat;
private double lng;
private double pressure;
private double flowRate;
private double temp;
private double corrosion;
private double incline;
public SensorState(String segment, double lat, double lng, double pressure,
double flowRate, double temp, double corrosion, double incline) {
this.segment = segment;
this.lat = lat;
this.lng = lng;
this.pressure = pressure;
this.flowRate = flowRate;
this.temp = temp;
this.corrosion = corrosion;
this.incline = incline;
}
public String getSegment() {
return segment;
}
public double getLat() {
return lat;
}
public double getLng() {
return lng;
}
public double getPressure() {
return pressure;
}
public double getFlowRate() {
return flowRate;
}
public double getTemp() {
return temp;
}
public double getCorrosion() {
return corrosion;
}
public double getIncline() {
return incline;
}
}
================================================
FILE: sample/java/producer/SensorReadingProducer.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package producer;
import java.nio.ByteBuffer;
import java.security.SecureRandom;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
import model.SensorReading;
import model.SensorReading.OutputFormat;
import model.SensorState;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.kinesis.AmazonKinesis;
import com.amazonaws.services.kinesis.AmazonKinesisClient;
import com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException;
import com.amazonaws.services.kinesis.model.PutRecordRequest;
public class SensorReadingProducer {
private final Random rand = new SecureRandom();
private final int ID_SPACE_SIZE = 1000;
private final int NUM_SEGMENTS = 40;
private final double PRESSURE_BASE = 108D;
private final double PRESSURE_VOLATILITY = 3D;
private final double FLOW_BASE = 1D;
private final double FLOW_VOLATILITY = 1D;
private final double TEMP_BASE = 16D;
private final double TEMP_VOLATILITY = .2D;
private final double INCLINE_BASE = 0D;
private final double CORROSION_BASE = .0000234D;
private final int BATCH_SIZE = 20;
private final int BACKOFF = 5;
private int sensorsGenerated = 0;
private Map<String, SensorState> sensorCache = new HashMap<>();
final double londonLat = 51.50722D;
final double londonLng = -0.12750D;
final double aberdeenLat = 57.1436900D;
final double aberdeenLng = -2.0981400D;
final double lineATan = Math.atan((aberdeenLng - londonLng)
/ (aberdeenLat - londonLat));
final double lineLength = Math.sqrt((Math.pow(aberdeenLng - londonLng, 2))
+ (Math.pow(aberdeenLat - londonLat, 2)));
final double lineIncrement = lineLength / NUM_SEGMENTS;
public SensorReadingProducer() {
}
public double[] getLinePoint() {
// random distance
double dist = Math.random() * lineLength;
// calculate which segment the point is in
double seg = Math.floor(dist / lineIncrement) + 1;
// derive lat/lng
double lat = (dist / Math.cos(lineATan)) + londonLat;
double lng = (dist * Math.sin(lineATan)) + londonLng;
return new double[] { lat, lng, seg };
}
public SensorReading nextSensorReading(final OutputFormat format) {
return nextSensorReading(format, rand.nextInt(ID_SPACE_SIZE));
}
public SensorReading nextSensorReading(final OutputFormat format,
int position) {
String id = Integer.toHexString(position);
SensorState sensorState = sensorCache.get(id);
if (sensorState == null) {
sensorsGenerated++;
System.out.println(String.format("Generating Sensor %s",
sensorsGenerated));
double[] location = getLinePoint();
String segment = Integer.toHexString(new Double(location[2])
.intValue());
double pressure = PRESSURE_BASE
+ (PRESSURE_VOLATILITY * rand.nextDouble());
double flow = FLOW_BASE + (FLOW_VOLATILITY * rand.nextDouble());
double temp = TEMP_BASE + (TEMP_VOLATILITY * rand.nextDouble());
double corrosion = CORROSION_BASE + (rand.nextDouble() / 1000);
double incline = INCLINE_BASE + (rand.nextDouble() / 1_000_000);
sensorState = new SensorState(segment, location[0], location[1],
pressure, flow, temp, corrosion, incline);
sensorCache.put(id, sensorState);
}
double pressure = sensorState.getPressure()
+ (PRESSURE_VOLATILITY * rand.nextDouble());
double temp = sensorState.getTemp()
+ (TEMP_VOLATILITY * rand.nextDouble());
double flow = sensorState.getFlowRate()
+ (FLOW_VOLATILITY * rand.nextDouble());
double corrosion = sensorState.getCorrosion()
+ (rand.nextDouble() / 1000);
double incline = sensorState.getIncline()
+ (rand.nextDouble() / 1_000_000);
SensorReading reading = new SensorReading(id, sensorState.getSegment(),
System.currentTimeMillis(), sensorState.getLat(),
sensorState.getLng(), pressure, temp, flow, corrosion, incline);
reading.withOutputFormat(format);
return reading;
}
private void run(final int events, final OutputFormat format,
final String streamName, final String region) throws Exception {
AmazonKinesis kinesisClient = new AmazonKinesisClient(
new DefaultAWSCredentialsProviderChain());
kinesisClient.setRegion(Region.getRegion(Regions.fromName(region)));
int count = 0;
SensorReading r = null;
do {
r = nextSensorReading(format);
try {
PutRecordRequest req = new PutRecordRequest()
.withPartitionKey("" + rand.nextLong())
.withStreamName(streamName)
.withData(ByteBuffer.wrap(r.toString().getBytes()));
kinesisClient.putRecord(req);
} catch (ProvisionedThroughputExceededException e) {
Thread.sleep(BACKOFF);
}
System.out.println(r);
count++;
} while (count < events);
}
public static void main(String[] args) throws Exception {
Integer i = Integer.parseInt(args[0]);
OutputFormat format = OutputFormat.valueOf(args[1]);
String streamName = args[2];
String region = args[3];
new SensorReadingProducer().run(i, format, streamName, region);
}
}
================================================
FILE: sample/resources/BySegment-CSV.json
================================================
[{"dataExtractor": "CSV",
"dateFormat": "",
"dateItem": 2,
"dateAttributeAlias":"sensorTS",
"labelItems": [
1
],
"labelAttributeAlias":"segment",
"namespace": "BySegment-CSV",
"delimiter": "|",
"summaryItems": [
"max(5) max-pressure",
"max(6) max-flow",
"max(9) max-corrosion",
"min(3) lat",
"min(4) lng"
],
"timeHorizons": [
"SECOND"
],
"type": "SUM",
"emitMetrics":"true",
"writeIOPS":100,
"readIOPS":25
}
]
================================================
FILE: sample/resources/BySegment-Json.json
================================================
[{"dataExtractor": "JSON",
"dateFormat": "",
"dateAttribute": "captureTs",
"labelItems": [
"segment"
],
"namespace": "BySegment-Json",
"summaryItems": [
"max(pressure)",
"max(flowRate)",
"max(corrosionIndex)",
"min(lat) lat",
"min(lng) lng"
],
"timeHorizons": [
"SECOND"
],
"type": "SUM",
"emitMetrics":"true",
"writeIOPS":100,
"readIOPS":25
}
]
================================================
FILE: sample/resources/BySegment-Regex.json
================================================
[{"dataExtractor": "REGEX",
"dateFormat": "",
"dateItem": 2,
"dateAttributeAlias":"sensorTS",
"labelItems": [
1
],
"labelAttributeAlias":"segment",
"namespace": "BySegment-Regex",
"regularExpression": "^(.*) \\((.*)\\) ts-(\\d+) (\\d+\\.\\d+)x(\\-\\d+\\.\\d+) (\\d+.\\d+) at (\\d+.\\d+) T:(\\d+\\.\\d+) c:\\ +(\\d+\\.\\d+) deg\\ +(\\d+\\.\\d+)$",
"summaryItems": [
"max(5) max-pressure",
"max(6) max-flow",
"max(9) max-corrosion",
"min(3) lat",
"min(4) lng"
],
"timeHorizons": [
"SECOND"
],
"type": "SUM",
"emitMetrics":"true",
"writeIOPS":100,
"readIOPS":25
}
]
================================================
FILE: src/.gitkeep
================================================
Feel free to delete this file as soon as actual Java code is added to this
directory.
================================================
FILE: src/log4j.properties
================================================
# Root logger option
log4j.rootLogger=INFO, stdout
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
================================================
FILE: src/main/WebContent/.ebextensions/as.config
================================================
option_settings:
- namespace: aws:autoscaling:asg
option_name: MinSize
value: 2
- namespace: aws:autoscaling:trigger
option_name: MeasureName
value: CPUUtilization
- namespace: aws:autoscaling:trigger
option_name: LowerThreshold
value: 40
- namespace: aws:autoscaling:trigger
option_name: UpperThreshold
value: 90
================================================
FILE: src/main/WebContent/META-INF/MANIFEST.MF
================================================
Manifest-Version: 1.0
Class-Path:
================================================
FILE: src/main/WebContent/WEB-INF/web.xml
================================================
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://java.sun.com/xml/ns/javaee"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
id="WebApp_ID" version="3.0">
<display-name>KinesisAggregatorsBeanstalkApplication</display-name>
<welcome-file-list>
<welcome-file>index.html</welcome-file>
</welcome-file-list>
<listener>
<listener-class>com.amazonaws.services.kinesis.aggregators.app.AggregatorsBeanstalkApp</listener-class>
</listener>
<servlet>
<servlet-name>DateQuery</servlet-name>
<servlet-class>com.amazonaws.services.kinesis.aggregators.app.DateQueryServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>DateQuery</servlet-name>
<url-pattern>/dateQuery</url-pattern>
</servlet-mapping>
<servlet>
<servlet-name>KeyQuery</servlet-name>
<servlet-class>com.amazonaws.services.kinesis.aggregators.app.ListAggregateKeysServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>KeyQuery</servlet-name>
<url-pattern>/keyQuery</url-pattern>
</servlet-mapping>
<servlet>
<servlet-name>LabelQuery</servlet-name>
<servlet-class>com.amazonaws.services.kinesis.aggregators.app.QueryByLabelServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>LabelQuery</servlet-name>
<url-pattern>/labelQuery</url-pattern>
</servlet-mapping>
<servlet>
<servlet-name>ConfigParams</servlet-name>
<servlet-class>com.amazonaws.services.kinesis.aggregators.app.FetchConfigurationServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ConfigParams</servlet-name>
<url-pattern>/configParams</url-pattern>
</servlet-mapping>
<servlet>
<servlet-name>Configuration</servlet-name>
<servlet-class>com.amazonaws.services.kinesis.aggregators.app.ShowConfigurationServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>Configuration</servlet-name>
<url-pattern>/configuration</url-pattern>
</servlet-mapping>
<servlet>
<servlet-name>ConfigFile</servlet-name>
<servlet-class>com.amazonaws.services.kinesis.aggregators.app.ShowConfigFileServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ConfigFile</servlet-name>
<url-pattern>/configFile</url-pattern>
</servlet-mapping>
</web-app>
================================================
FILE: src/main/WebContent/index.html
================================================
<!--
Amazon Kinesis Aggregators
Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
Licensed under the Amazon Software License (the "License").
You may not use this file except in compliance with the License.
A copy of the License is located at
http://aws.amazon.com/asl/
or in the "license" file accompanying this file. This file is distributed
on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied. See the License for the specific language governing
permissions and limitations under the License.
-->
<html>
<head>
<title>Kinesis Aggregators Managed Application in Elastic Beanstalk</title>
</head>
<body>
OK - Kinesis Aggregators Managed Application hosted in Elastic Beanstalk Online
</body>
</html>
================================================
FILE: src/main/WebContent/styles/styles.css
================================================
/*************************************
GENERAL
*************************************/
body {
margin: 0;
padding: 0;
font: 12px/1.4em "Lucida Grande", Verdana, sans-serif;
color: #333;
overflow-y: scroll;
text-rendering: optimizeLegibility;
background-color: #d5e9ed;
}
h2 {
font-size: 1.3em;
line-height: 1.5em;
font-weight: bold;
margin: 20px 0 0 0;
padding: 0;
border-bottom: 3px solid #eee;
/* icon setup */
padding: 0.2em 1em 0.2em 30px;
background-position: 0 50%;
background-repeat: no-repeat;
}
/*************************************
SECTIONS
*************************************/
div#content {
margin: 30px auto;
padding: 0 30px 15px 30px;
background-color: #fff;
width: 940px;
/* box-shadow */
-moz-box-shadow: 0 5px 10px #aaa;
-webkit-box-shadow: 0 5px 10px #aaa;
box-shadow: 0 5px 10px #aaa;
/* bottom corners */
-webkit-border-bottom-right-radius: 7px;
-webkit-border-bottom-left-radius: 7px;
-moz-border-radius-bottomright: 7px;
-moz-border-radius-bottomleft: 7px;
border-bottom-right-radius: 7px;
border-bottom-left-radius: 7px;
}
/*div#content div.section {}*/
div#content div.section ul {
margin: 0;
padding: 1em 0 0 2em;
overflow: hidden;
}
div#content div.section ul li {
list-style-type: square;
white-space: nowrap;
line-height: 1.5em;
}
/* Section titles */
div#content div.section.s3 h2 {
background-image: url(../images/drive.png);
}
div#content div.section.ec2 h2 {
background-image: url(../images/server.png);
}
div#content div.section.sdb h2 {
background-image: url(../images/database.png);
}
/*************************************
CONTAINERS
*************************************/
.container {
zoom: 1;
}
.container:after {
content: ".";
display: block;
height: 0;
clear: both;
visibility: hidden;
}
/*************************************
GRIDS
*************************************/
.grid { float: left; margin-right: 20px; }
.gridlast { margin-right: 0; }
.grid5 { width: 300px; }
.grid15 { width: 940px; }
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/AggregateData.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.util.Date;
import java.util.Map;
public class AggregateData {
private String uniqueId;
private LabelSet labels;
private Date date;
private Map<String, Double> summaries;
public AggregateData(String uniqueId, LabelSet labels, Date date, Map<String, Double> summaries) {
this.uniqueId = uniqueId;
this.labels = labels;
this.date = date;
this.summaries = summaries;
}
public String getUniqueId() {
return this.uniqueId;
}
public String getLabel() {
return this.labels.valuesAsString();
}
public LabelSet getLabels() {
return this.labels;
}
public Date getDate() {
return this.date;
}
public Map<String, Double> getSummaries() {
return this.summaries;
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorGroup.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.util.ArrayList;
import java.util.List;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.services.kinesis.model.Record;
/**
* Class which provides a simple automation around a number of aggregators.
* Register any number of aggregators with the container, and then call all of
* the registered aggregators aggregate and checkpoint methods through this
* simple proxy
*/
public class AggregatorGroup implements IStreamAggregator {
List<StreamAggregator> aggregators = new ArrayList<>();
public AggregatorGroup() {
}
public AggregatorGroup(AggregatorGroup template) throws Exception {
// create a new aggregator group from all of the aggregators this one
// encapsulates, by instantiating new aggregators with their copy
// constructors
for (StreamAggregator agg : template.aggregators) {
this.registerAggregator(new StreamAggregator(agg));
}
}
public void registerAggregator(StreamAggregator agg) {
this.aggregators.add(agg);
}
public List<StreamAggregator> getAggregators() {
return this.aggregators;
}
/**
* {@inheritDoc}
*/
@Override
public void aggregate(List<Record> records) throws Exception {
for (IStreamAggregator agg : aggregators) {
agg.aggregate(records);
}
}
public void aggregateEvents(List<InputEvent> events) throws Exception {
for (IStreamAggregator agg : aggregators) {
agg.aggregateEvents(events);
}
}
/**
* {@inheritDoc}
*/
@Override
public void checkpoint() throws Exception {
for (IStreamAggregator agg : aggregators) {
agg.checkpoint();
}
}
@Override
public void initialize(String shardId) throws Exception {
for (IStreamAggregator agg : aggregators) {
agg.initialize(shardId);
}
}
/**
* {@inheritDoc}
*/
@Override
public void shutdown(boolean flushState) throws Exception {
for (IStreamAggregator agg : aggregators) {
agg.shutdown(flushState);
}
}
/**
* N/A - use getTableNames()
*/
@Override
public String getTableName() {
return null;
}
public List<String> getTableNames() {
List<String> out = new ArrayList<>(this.aggregators.size());
for (IStreamAggregator i : this.aggregators) {
out.add(i.getTableName());
}
return out;
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorType.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
/**
* Types of Aggregators supported by the Kinesis Aggregator Framework.
*/
public enum AggregatorType {
/**
* Count Aggregators maintain only an Event Count observed for the indicated
* {@link com.amazonaws.services.kinesis.aggregators.TimeHorizon}
*/
COUNT,
/**
* Sum Aggregators maintain an Event Count, plus a set of summary values for
* data indicated on the stream as being a summary value. Summary Values can
* be any of
* {@link com.amazonaws.services.kinesis.aggregators.SummaryCalculation}
*/
SUM;
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorsConstants.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
public class AggregatorsConstants {
public static final String CONFIG_URL_PARAM = "config-file-url";
public static final String CONFIG_PATH_PARAM = "config-file-path";
public static final String STREAM_NAME_PARAM = "stream-name";
public static final String NAMESPACE_PARAM = "namespace";
public static final String APP_NAME_PARAM = "application-name";
public static final String REGION_PARAM = "region";
public static final String STREAM_POSITION_PARAM = "position-in-stream";
public static final String MAX_RECORDS_PARAM = "max-records";
public static final String ENVIRONMENT_PARAM = "environment";
public static final String FAILURES_TOLERATED_PARAM = "failures-tolerated";
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/EnvironmentType.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
public enum EnvironmentType {
DEV, TEST, INT, PERF, PROD;
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/IStreamAggregator.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.util.List;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.services.kinesis.model.Record;
/**
* Stream aggregators give end users the ability to dynamically aggregate
* Kinesis data in Dynamo DB. All a consumer must do is create an Aggregator and
* then call aggregate within the processRecords method of an IRecordProcessor.
* Please note all writes made within the aggregate context are durable.
*
* @author meyersi
*/
public interface IStreamAggregator {
/**
* Aggregate a set of records received from the Kinesis Client Library.
*
* @param records The set of Records received from a processRecords
* invocation
* @throws Exception
*/
public void aggregate(List<Record> records) throws Exception;
public void aggregateEvents(List<InputEvent> events) throws Exception;
/**
* Commit all aggregated data to the backing store.
*/
public void checkpoint() throws Exception;
/**
* Initialise the Aggregator on a shard. Should be called by
* IRecordProcessor.initialize().
*/
public void initialize(String shardId) throws Exception;
/**
* Terminate an Aggregator running, which will mark the process as offline
* in the {@link InventoryModel} table.
*/
public void shutdown(boolean flushState) throws Exception;
/** Get the underlying data store name for the aggregator. */
public String getTableName();
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/InputEvent.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import com.amazonaws.services.kinesis.model.Record;
public class InputEvent {
private String sequenceNumber;
private String partitionKey;
private byte[] data;
public InputEvent(Record record) {
this.sequenceNumber = record.getSequenceNumber();
this.partitionKey = record.getPartitionKey();
this.data = record.getData().array();
}
public InputEvent withSequence(String sequence) {
this.sequenceNumber = sequence;
return this;
}
public InputEvent withPartitionKey(String partitionKey) {
this.partitionKey = partitionKey;
return this;
}
public String getSequenceNumber() {
return sequenceNumber;
}
public String getPartitionKey() {
return partitionKey;
}
public byte[] getData() {
return data;
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/InventoryModel.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;
import com.amazonaws.services.dynamodbv2.model.AttributeAction;
import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
import com.amazonaws.services.dynamodbv2.model.AttributeValueUpdate;
import com.amazonaws.services.dynamodbv2.model.DeleteItemRequest;
import com.amazonaws.services.dynamodbv2.model.GetItemResult;
import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
import com.amazonaws.services.dynamodbv2.model.KeyType;
import com.amazonaws.services.dynamodbv2.model.UpdateItemRequest;
import com.amazonaws.services.kinesis.aggregators.datastore.DynamoUtils;
/**
* Class used to provide configuration and setup for the Worker Inventory table
* in Dynamo DB.
*/
@SuppressWarnings("serial")
public final class InventoryModel {
private boolean online = false;
/**
* Name of the Table in Dynamo DB.
*/
public static final String TABLE_NAME = "KinesisAggregatorWorkerState";
/**
* Column name used to store the Kinesis Stream Name for an Aggregator.
*/
public static final String AGGREGATOR = "aggregator";
/**
* Column name used to store the Shard ID for an Aggregator.
*/
public static final String SHARD_ID = "shardId";
/**
* Column name used to store the last time an Aggregator updated.
*/
public static final String LAST_WRITE_TIME = "lastWriteTime";
/**
* Column name used to store the lowest sequence value updated in the last
* flush of an Aggregator.
*/
public static final String LAST_LOW_SEQ = "lastLowSeq";
/**
* Column name used to store the highest sequence value updated in the last
* flush of an Aggregator.
*/
public static final String LAST_HIGH_SEQ = "lastHighSeq";
/**
* Column name used to store the status of the running or stopped
* Aggregator.
*/
public static final String STATUS = "status";
/**
* Amount of read IOPS to provision for the Inventory table.
*/
public static final long READ_CAPACITY = 10L;
/**
* Amount of write IOPS to provision for the Inventory table.
*/
public static final long WRITE_CAPACITY = 10L;
/**
* Available states for an Aggregator to be in.
*/
public static enum STATE {
STARTING, RUNNING, STOPPED, SERIALISATION_ERROR, UNKNOWN_ERROR;
}
private AmazonDynamoDB dynamoClient;
public InventoryModel(AmazonDynamoDB dynamoClient) throws Exception {
this.dynamoClient = dynamoClient;
init();
}
public InventoryModel(AWSCredentialsProvider credentials) throws Exception {
this(new AmazonDynamoDBClient(credentials));
}
protected void init() throws Exception {
List<AttributeDefinition> attributes = new ArrayList<AttributeDefinition>() {
{
add(new AttributeDefinition().withAttributeName(InventoryModel.AGGREGATOR).withAttributeType(
"S"));
add(new AttributeDefinition().withAttributeName(InventoryModel.SHARD_ID).withAttributeType(
"S"));
}
};
List<KeySchemaElement> key = new ArrayList<KeySchemaElement>() {
{
add(new KeySchemaElement().withAttributeName(InventoryModel.AGGREGATOR).withKeyType(
KeyType.HASH));
add(new KeySchemaElement().withAttributeName(InventoryModel.SHARD_ID).withKeyType(
KeyType.RANGE));
}
};
DynamoUtils.initTable(dynamoClient, InventoryModel.TABLE_NAME,
InventoryModel.READ_CAPACITY, InventoryModel.WRITE_CAPACITY, attributes, key, null);
online = true;
}
private Map<String, AttributeValue> getKey(final String streamName,
final String applicationName, final String namespace, final String shardId) {
return new HashMap<String, AttributeValue>() {
{
put(InventoryModel.AGGREGATOR, new AttributeValue().withS(String.format("%s.%s.%s",
streamName, applicationName, namespace)));
put(InventoryModel.SHARD_ID, new AttributeValue().withS(shardId));
}
};
}
public void removeState(final String streamName, final String applicationName,
final String namespace, final String shardId) throws Exception {
DeleteItemRequest req = new DeleteItemRequest().withTableName(TABLE_NAME).withKey(
getKey(streamName, applicationName, namespace, shardId));
dynamoClient.deleteItem(req);
}
/**
* Update the Inventory table with the state of an Aggregator.
*
* @param streamName The Kinesis Stream being aggregated.
* @param applicationName The application name running the aggregator.
* @param workerId The worker ID which encapsulates an instance of an
* Aggregator.
* @param lastLowSeq The lowest sequence number observed in all records
* which were flushed prior to this update.
* @param lastHighSeq The highest sequence number for all records flushed in
* this update.
* @param lastWriteTime The write time of the data to Dynamo DB.
* @param status The {@link STATE} of the Aggregator.
* @throws Exception
*/
public void update(final String streamName, final String applicationName,
final String namespace, final String shardId, final String lastLowSeq,
final String lastHighSeq, final long lastWriteTime, final STATE status)
throws Exception {
// create the last write time value
final String lastUpdateDateLabel = StreamAggregator.dateFormatter.format(new Date(
lastWriteTime));
// generate the item update
Map<String, AttributeValueUpdate> inventoryUpdate = new HashMap<String, AttributeValueUpdate>() {
{
put(InventoryModel.LAST_WRITE_TIME,
new AttributeValueUpdate().withAction(AttributeAction.PUT).withValue(
new AttributeValue().withS(lastUpdateDateLabel)));
if (lastLowSeq != null)
put(InventoryModel.LAST_LOW_SEQ,
new AttributeValueUpdate().withAction(AttributeAction.PUT).withValue(
new AttributeValue().withS(lastLowSeq)));
if (lastHighSeq != null)
put(InventoryModel.LAST_HIGH_SEQ,
new AttributeValueUpdate().withAction(AttributeAction.PUT).withValue(
new AttributeValue().withS(lastHighSeq)));
if (status != null)
put(InventoryModel.STATUS,
new AttributeValueUpdate().withAction(AttributeAction.PUT).withValue(
new AttributeValue().withS(status.name())));
}
};
DynamoUtils.updateWithRetries(
dynamoClient,
new UpdateItemRequest().withTableName(InventoryModel.TABLE_NAME).withKey(
getKey(streamName, applicationName, namespace, shardId)).withAttributeUpdates(
inventoryUpdate));
}
/**
* Method which returns the update information for an Aggregator process.
*
* @param streamName The Stream name which is being aggregated.
* @param applicationName The application which is hosting the aggregator.
* @param workerId The worker ID which is running an aggregator instance.
* @return Tuple of Last Write Time (String), Last Low Sequence, and Last
* High Sequence
*/
public InventoryStatus getLastUpdate(final String streamName, final String applicationName,
final String namespace, final String shardId) {
GetItemResult response = dynamoClient.getItem(InventoryModel.TABLE_NAME,
getKey(streamName, applicationName, namespace, shardId));
if (response.getItem() != null) {
Map<String, AttributeValue> item = response.getItem();
AttributeValue lastTime, lowSeq, highSeq = null;
lastTime = item.get(InventoryModel.LAST_WRITE_TIME);
lowSeq = item.get(InventoryModel.LAST_LOW_SEQ);
highSeq = item.get(InventoryModel.LAST_HIGH_SEQ);
return new InventoryStatus(lastTime == null ? null : lastTime.getS(),
lowSeq == null ? null : lowSeq.getS(), highSeq == null ? null : highSeq.getS());
} else {
return null;
}
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/InventoryStatus.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
public class InventoryStatus {
private String lastTime, lowSeq, highSeq;
public InventoryStatus(String lastTime, String lowSeq, String highSeq) {
super();
this.lastTime = lastTime;
this.lowSeq = lowSeq;
this.highSeq = highSeq;
}
public String getLastTime() {
return lastTime;
}
public String getLowSeq() {
return lowSeq;
}
public String getHighSeq() {
return highSeq;
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/LabelSet.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.util.LinkedHashMap;
import java.util.List;
/**
* Object which encapsulates all label values to be tracked and managed within
* the Aggregator framework. Adheres to the properties of strict ordering by
* insert sequence, equality on the basis of values as well as hash on the basis
* of values, and a name property synthesized from the keyset, and a String
* value synthesized from the value set
*/
public class LabelSet extends LinkedHashMap<String, String> {
private final String setDelimiter = ".";
private String alias = null;
public LabelSet() {
super();
}
public static LabelSet fromIntegerKeys(List<Integer> keys) {
LabelSet labels = new LabelSet();
for (Integer i : keys) {
labels.put("" + i, null);
}
return labels;
}
public static LabelSet fromStringKeys(List<String> keys) {
LabelSet labels = new LabelSet();
for (String s : keys) {
labels.put(s, null);
}
return labels;
}
@Override
public String put(String key, String value) {
// wrap general map put with internal pre-processing of names
return super.put(StreamAggregatorUtils.methodToColumn(key), value);
}
public String valuesAsString() {
StringBuffer sb = new StringBuffer();
for (String s : this.values()) {
sb.append(s + setDelimiter);
}
return sb.substring(0, sb.length() - 1);
}
public String getName() {
if (this.alias == null) {
StringBuffer sb = new StringBuffer();
for (String s : this.keySet()) {
sb.append(StreamAggregatorUtils.methodToColumn(s) + setDelimiter);
}
return sb.substring(0, sb.length() - 1);
} else {
return this.alias;
}
}
public LabelSet withAlias(String alias) {
this.alias = alias;
return this;
}
@Override
public boolean equals(Object o) {
if (o == null)
return false;
if (!(o instanceof LabelSet))
return false;
LabelSet other = (LabelSet) o;
boolean matched = false;
// match on keys
for (String s : this.keySet()) {
matched = false;
for (String k : other.keySet()) {
if (k.equals(s)) {
matched = true;
break;
}
}
if (!matched)
return false;
}
// must match on values
for (String t : this.values()) {
matched = false;
for (String v : other.values()) {
if (t.equals(v)) {
matched = true;
break;
}
}
if (!matched)
return false;
}
return true;
}
@Override
public int hashCode() {
int res = 17;
for (String s : this.keySet()) {
res = 31 * res + (s == null ? 0 : s.hashCode());
}
for (String t : this.values()) {
res = 31 * res + (t == null ? 0 : t.hashCode());
}
return res;
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/StreamAggregator.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.math.BigInteger;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBAsyncClient;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
import com.amazonaws.services.dynamodbv2.model.ComparisonOperator;
import com.amazonaws.services.dynamodbv2.model.Condition;
import com.amazonaws.services.kinesis.AmazonKinesisClient;
import com.amazonaws.services.kinesis.aggregators.cache.AggregateCache;
import com.amazonaws.services.kinesis.aggregators.datastore.DynamoDataStore;
import com.amazonaws.services.kinesis.aggregators.datastore.DynamoQueryEngine.QueryKeyScope;
import com.amazonaws.services.kinesis.aggregators.datastore.IDataStore;
import com.amazonaws.services.kinesis.aggregators.exception.InvalidConfigurationException;
import com.amazonaws.services.kinesis.aggregators.exception.SerializationException;
import com.amazonaws.services.kinesis.aggregators.idempotency.DefaultIdempotencyCheck;
import com.amazonaws.services.kinesis.aggregators.idempotency.IIdempotencyCheck;
import com.amazonaws.services.kinesis.aggregators.metrics.CloudWatchMetricsEmitter;
import com.amazonaws.services.kinesis.aggregators.metrics.IMetricsEmitter;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisClientLibConfiguration;
import com.amazonaws.services.kinesis.io.IDataExtractor;
import com.amazonaws.services.kinesis.model.Record;
/**
* StreamAggregator is the main implementation of the Kinesis Aggregators
* framework. It provides the ability to create dynamic aggregations in Dynamo
* DB for data being streamed through Kinesis. Objects are aggregated on the
* basis of the unique values contained in the Aggregate Label by Date, storing
* event counts. Additionally, by configuring a set of summary values on the
* StreamAggregator with AggregatorType set to SUM, an additional data element
* is aggregated tracking the total value observed for each element of the
* stream.Data in DynamoDB is aggregated on the basis of the configured
* TimeHorizon, from granularity of SECOND to FOREVER.
*
* @author meyersi
*/
public class StreamAggregator implements IStreamAggregator {
public static final String AWSApplication = "AmazonKinesisAggregators";
public static final String version = ".9.2.7.3";
/**
* The default column name for the aggregated value, if none is provided.
*/
public static final String DEFAULT_AGGREGATE_COLUMN = "aggregatedValue";
/**
* The default attribute name for the date value of an aggregate, if none is
* provided.
*/
public static final String DEFAULT_DATE_VALUE = "dateValue";
/**
* The default attribute name for the count of events observed for an
* aggregate value and date, if none is provided.
*/
public static final String EVENT_COUNT = "eventCount";
/**
* The attribute name for the time horizon marker
*/
public static final String TIME_HORIZON_ATTR = "timeHorizonType";
/**
* The attribute name used for the last write sequence value in the table.
*/
public static final String LAST_WRITE_SEQ = "lastWriteSeq";
/**
* The attribute name used for the timestamp of the update of the aggregate.
*/
public static final String LAST_WRITE_TIME = "lastWriteTime";
/**
* The attribute used to refer to the partition key
*/
public static final String REF_PARTITION_KEY = "__partition_key";
/**
* The attribute used to refer to the event sequence number
*/
public static final String REF_SEQUENCE = "__sequence";
public static final SimpleDateFormat dateFormatter = new SimpleDateFormat(
"yyyy-MM-dd HH:mm:ss");
protected String namespace;
private KinesisClientLibConfiguration config;
private String environment;
private AmazonDynamoDB dynamoClient;
private AmazonKinesisClient kinesisClient;
private InventoryModel inventory;
protected String tableName;
protected boolean withTimeHierarchy = false;
protected List<TimeHorizon> timeHorizons = new ArrayList<>();
protected AggregatorType aggregatorType = AggregatorType.COUNT;
protected long readCapacity;
protected long writeCapacity;
protected final String streamName;
protected final String applicationName;
protected String shardId = null;
private boolean isFirstShardWorker = false;
private final Log LOG = LogFactory.getLog(StreamAggregator.class);
private Region region = null;
protected AggregateCache cache;
protected boolean online = false;
protected String lowSeq;
protected BigInteger highSeq = null;
protected long start;
private IDataExtractor dataExtractor;
private IDataStore dataStore;
private IIdempotencyCheck idempotencyCheck = new DefaultIdempotencyCheck();
private IMetricsEmitter metricsEmitter;
private boolean raiseExceptionOnDataExtractionErrors = true;
private int ignoredRecordsBelowHWM = 0;
private boolean publishMetrics = false;
/**
* Copy Constructor
*
* @param template
*/
public StreamAggregator(StreamAggregator template) throws Exception {
this.streamName = template.streamName;
this.applicationName = template.applicationName;
this.namespace = template.namespace;
this.config = template.config;
this.dataExtractor = template.dataExtractor.copy();
this.withDataStore(template.getDataStore());
this.withAggregatorType(template.aggregatorType);
this.withRaiseExceptionOnDataExtractionErrors(template.raiseExceptionOnDataExtractionErrors);
this.withStorageCapacity(template.readCapacity, template.writeCapacity);
this.withTableName(template.tableName);
this.withTimeHorizon(template.timeHorizons);
this.withIdempotencyCheck(template.idempotencyCheck);
if (template.publishMetrics) {
this.publishMetrics = true;
this.metricsEmitter = template.metricsEmitter;
}
}
public StreamAggregator(String streamName, String applicationName,
String namespace, KinesisClientLibConfiguration config,
IDataExtractor dataExtractor) {
this.streamName = streamName;
this.applicationName = applicationName;
this.namespace = namespace;
this.config = config;
this.dataExtractor = dataExtractor;
}
public void checkpoint() throws Exception {
cache.flush();
lowSeq = null;
// update the worker inventory showing progress to the last sequence
// value
inventory.update(this.streamName, this.applicationName, this.namespace,
this.shardId, this.lowSeq, this.highSeq.toString(),
System.currentTimeMillis(), InventoryModel.STATE.RUNNING);
// warn and reset if there were any ignored records
if (ignoredRecordsBelowHWM > 0) {
logWarn(String
.format("Processed %s records which were ignored due to being below the current processing HWM",
ignoredRecordsBelowHWM));
ignoredRecordsBelowHWM = 0;
}
LOG.debug("Aggregator Checkpoint for Shard " + this.shardId
+ " Complete");
}
/*
* builder methods
*/
public StreamAggregator withStorageCapacity(Long readCapacity,
Long writeCapacity) {
if (readCapacity != null)
this.readCapacity = readCapacity;
if (writeCapacity != null)
this.writeCapacity = writeCapacity;
return this;
}
private void logInfo(String message) {
LOG.info("[" + this.shardId + "] " + message);
}
private void logWarn(String message) {
LOG.warn("[" + this.shardId + "] " + message);
}
private void logWarn(String message, Exception e) {
LOG.warn("[" + this.shardId + "] " + message);
LOG.error(e);
}
public void initialize(String shardId) throws Exception {
// Set System properties to allow entity expansion of unlimited items in
// response documents from AWS API
//
// see https://blogs.oracle.com/joew/entry/jdk_7u45_aws_issue_123 for
// more information
System.setProperty("entityExpansionLimit", "0");
System.setProperty("jdk.xml.entityExpansionLimit", "0");
this.shardId = shardId;
// establish we are running on the lowest shard on the basis of hash
// range
AmazonKinesisClient kinesisClient = new AmazonKinesisClient(
this.config.getKinesisCredentialsProvider());
if (this.config.getRegionName() != null) {
region = Region.getRegion(Regions.fromName(this.config
.getRegionName()));
kinesisClient.setRegion(region);
}
try {
if (this.shardId.equals(StreamAggregatorUtils.getFirstShardName(
kinesisClient, this.config.getStreamName()))) {
this.isFirstShardWorker = true;
logInfo("Aggregator taking Primary Thread Responsibility");
}
} catch (Exception e) {
logWarn("Unable to establish if Worker Thread is Primary");
}
validateConfig();
// set the default aggregator type
if (this.aggregatorType == null) {
this.aggregatorType = AggregatorType.COUNT;
}
if (this.dataExtractor == null)
throw new InvalidConfigurationException(
"Unable to create Aggregator Instance without a configured IDataStore");
// set the aggregator type on the data extractor
this.dataExtractor.setAggregatorType(this.aggregatorType);
this.dataExtractor.validate();
// create connections to dynamo and kinesis
ClientConfiguration clientConfig = new ClientConfiguration()
.withSocketTimeout(60000);
this.dynamoClient = new AmazonDynamoDBAsyncClient(
this.config.getDynamoDBCredentialsProvider(), clientConfig);
if (region != null)
this.dynamoClient.setRegion(region);
this.kinesisClient = new AmazonKinesisClient(
this.config.getKinesisCredentialsProvider());
if (region != null)
this.kinesisClient.setRegion(region);
inventory = new InventoryModel(this.dynamoClient);
// get the latest sequence number checkpointed for this named aggregator
// on this shard
InventoryStatus lastUpdate = inventory.getLastUpdate(this.streamName,
this.applicationName, this.namespace, this.shardId);
if (lastUpdate != null && lastUpdate.getHighSeq() != null) {
// set the current high sequence to the last high sequence
this.highSeq = new BigInteger(lastUpdate.getHighSeq());
}
// log that we are now starting up
inventory.update(this.streamName, this.applicationName, this.namespace,
this.shardId, null, null, System.currentTimeMillis(),
InventoryModel.STATE.STARTING);
// set the table name we will use for aggregated values
if (this.tableName == null) {
this.tableName = StreamAggregatorUtils.getTableName(
config.getApplicationName(), this.getNamespace());
}
if (this.environment != null && !this.environment.equals(""))
this.tableName = String.format("%s.%s", this.environment,
this.tableName);
// resolve the basic data being aggregated
String labelColumn = StreamAggregatorUtils.methodToColumn(dataExtractor
.getAggregateLabelName());
String dateColumn = dataExtractor.getDateValueName() == null ? DEFAULT_DATE_VALUE
: dataExtractor.getDateValueName();
// configure the default dynamo data store
if (this.dataStore == null) {
this.dataStore = new DynamoDataStore(this.dynamoClient,
this.kinesisClient, this.aggregatorType, this.streamName,
this.tableName, labelColumn, dateColumn)
.withStorageCapacity(this.readCapacity, this.writeCapacity);
this.dataStore.setRegion(region);
}
this.dataStore.initialise();
// configure the cache so it can do its work
cache = new AggregateCache(this.shardId)
.withCredentials(this.config.getKinesisCredentialsProvider())
.withAggregateType(this.aggregatorType)
.withTableName(this.tableName).withLabelColumn(labelColumn)
.withDateColumn(dateColumn).withDataStore(this.dataStore);
// create a cloudwatch client for the cache to publish against if needed
if (this.publishMetrics && this.metricsEmitter == null) {
this.metricsEmitter = new CloudWatchMetricsEmitter(this.tableName,
this.config.getCloudWatchCredentialsProvider());
}
if (this.metricsEmitter != null) {
if (this.config.getRegionName() != null)
this.metricsEmitter.setRegion(region);
}
// add the metrics publisher to the cache if we are bound to the lowest
// shard
if (this.metricsEmitter != null) {
cache.withMetricsEmitter(this.metricsEmitter);
}
cache.initialise();
// set the user agent
StringBuilder userAgent = new StringBuilder(
ClientConfiguration.DEFAULT_USER_AGENT);
userAgent.append(" ");
userAgent.append(this.AWSApplication);
userAgent.append("/");
userAgent.append(this.version);
this.config.getKinesisClientConfiguration().setUserAgent(
userAgent.toString());
// log startup state
StringBuffer sb = new StringBuffer();
for (TimeHorizon t : timeHorizons) {
sb.append(String.format("%s,", t.name()));
}
sb.deleteCharAt(sb.length() - 1);
logInfo(String
.format("Amazon Kinesis Stream Aggregator Online\nStream: %s\nApplication: %s\nNamespace: %s\nWorker: %s\nGranularity: %s\nContent Extracted With: %s",
streamName, applicationName, this.namespace,
this.config.getWorkerIdentifier(), sb.toString(),
dataExtractor.getClass().getName()));
if (this.highSeq != null)
logInfo(String.format("Processing Data from Seq: %s", this.highSeq));
online = true;
}
private void validateConfig() throws Exception {
// this would only be null if the containing worker IRecordProcessor has
// not called initialise()
if (this.shardId == null) {
throw new Exception(
"Aggregator Not Online - Call Initialise to establish System State on Shard");
}
// default to Hourly granularity if the customer has not configured it
if (this.timeHorizons == null) {
withTimeHorizon(TimeHorizon.HOUR);
}
}
/**
* Add a single
* {@link com.amazonaws.services.kinesis.aggregators.TimeHorizon} to the
* configuration of the Aggregator
*
* @param horizon
* TimeHorizon value to be used for aggregated data
* @return
*/
public StreamAggregator withTimeHorizon(TimeHorizon horizon) {
if (this.timeHorizons == null)
this.timeHorizons = new ArrayList<>();
this.timeHorizons.add(horizon);
return this;
}
/**
* Add a set of
* {@link com.amazonaws.services.kinesis.aggregators.TimeHorizon} values to
* the configuration of the Aggregator
*
* @param horizon
* TimeHorizon value to be used for aggregated data
* @return
*/
public StreamAggregator withTimeHorizon(List<TimeHorizon> horizons) {
if (this.timeHorizons == null) {
this.timeHorizons = horizons;
} else {
this.timeHorizons.addAll(horizons);
}
return this;
}
/**
* Add a set of
* {@link com.amazonaws.services.kinesis.aggregators.TimeHorizon} values to
* the configuration of the Aggregator
*
* @param horizon
* TimeHorizon value to be used for aggregated data
* @return
*/
public StreamAggregator withTimeHorizon(TimeHorizon... horizons) {
if (this.timeHorizons == null)
this.timeHorizons = new ArrayList<>();
for (TimeHorizon t : horizons) {
this.timeHorizons.add(t);
}
return this;
}
/**
* Set the name of the data store in Dynamo DB for the Aggregated Data
*
* @param tableName
* The table name to use for data storage
* @return
*/
public StreamAggregator withTableName(String tableName) {
this.tableName = tableName;
return this;
}
/**
* Select an explicit
* {@link com.amazonaws.servies.kinesis.aggregators.AggregatorType} for the
* Aggregator. Default is COUNT
*
* @param t
* The Aggregator Type to use
* @return
*/
public StreamAggregator withAggregatorType(AggregatorType t) {
if (t != null) {
this.aggregatorType = t;
}
return this;
}
/**
* Override the default behaviour of an Aggregator to fail when the data
* stream cannot be deserialised. When setting this value to 'true', then
* the Aggregator stream will be able to deal with bad data that cannot be
* aggregated, and will simply continue working
*
* @param bool
* Boolean indicating whether to fail when bad data is received
* on the stream and cannot be deserialised
* @return
*/
public StreamAggregator withRaiseExceptionOnDataExtractionErrors(
boolean bool) {
this.raiseExceptionOnDataExtractionErrors = bool;
return this;
}
/**
* Should we publish CloudWatch metrics for all captured data?
*
* @param bool
* @return
*/
public StreamAggregator withCloudWatchMetrics() {
this.publishMetrics = true;
return this;
}
/**
* Allow configuring a non-Default data store
*
* @param dataStore
* @return
*/
public StreamAggregator withDataStore(IDataStore dataStore) {
if (dataStore != null) {
this.dataStore = dataStore;
}
return this;
}
/**
* Allow configuring a non-Default metrics emitter
*
* @param metricsEmitter
* @return
*/
public StreamAggregator withMetricsEmitter(IMetricsEmitter metricsEmitter) {
if (metricsEmitter != null) {
this.metricsEmitter = metricsEmitter;
}
return this;
}
/**
* Allow configuring a non-Default idempotency check
*
* @param idempotencyCheck
* @return
*/
public StreamAggregator withIdempotencyCheck(
IIdempotencyCheck idempotencyCheck) {
if (idempotencyCheck != null) {
this.idempotencyCheck = idempotencyCheck;
}
return this;
}
public StreamAggregator withEnvironment(EnvironmentType environment) {
this.environment = environment.name();
return this;
}
public StreamAggregator withEnvironment(String environment) {
this.environment = environment;
return this;
}
/* Simple property accessors */
public String getNamespace() {
return this.namespace;
}
public IDataExtractor getDataExtractor() {
return this.dataExtractor;
}
public IDataStore getDataStore() {
return this.dataStore;
}
public String getTableName() {
return this.tableName;
}
public String getLabelAttribute() {
return this.dataExtractor.getAggregateLabelName();
}
public String getDateAttribute() {
return this.dataExtractor.getDateValueName();
}
public AggregatorType getAggregatorType() {
return this.aggregatorType;
}
public long getReadCapacity() {
return this.readCapacity;
}
public long getWriteCapacity() {
return this.writeCapacity;
}
public List<TimeHorizon> getTimeHorizon() {
return this.timeHorizons;
}
/**
* Shut down an aggregator and mark its state as Stopped in the Inventory
* Table
*
* @param flushState
* Should the aggregator clear it's pending updates prior to
* shutting down
* @param withState
* Final status for the aggregator
* @throws Exception
*/
public void shutdown() throws Exception {
shutdown(true);
}
public void shutdown(boolean flushState) throws Exception {
shutdown(flushState, null);
}
public void shutdown(boolean flushState, InventoryModel.STATE withState)
throws Exception {
if (flushState)
checkpoint();
if (inventory != null)
inventory.update(this.streamName, this.applicationName,
this.namespace, this.shardId, null, null, System
.currentTimeMillis(),
withState == null ? InventoryModel.STATE.STOPPED
: withState);
}
/**
* {@inheritDoc}
*/
public void aggregate(List<Record> records) throws Exception {
List<InputEvent> events = new ArrayList<>();
for (Record r : records) {
events.add(new InputEvent(r));
}
aggregateEvents(events);
}
/**
* {@inheritDoc}
*/
public void aggregateEvents(List<InputEvent> events) throws Exception {
start = System.currentTimeMillis();
int aggregatedEventCount = 0;
int aggregatedElementCount = 0;
if (!online) {
throw new Exception("Aggregator Not Initialised");
}
BigInteger thisSequence;
List<AggregateData> extractedItems = null;
Date eventDate = null;
try {
for (InputEvent event : events) {
// reset extracted items
extractedItems = null;
if (event.getSequenceNumber() != null) {
thisSequence = new BigInteger(event.getSequenceNumber());
// ignore any records which are going backward with regard
// to
// the current hwm
if (highSeq != null
&& highSeq.compareTo(thisSequence) != -1) {
ignoredRecordsBelowHWM++;
continue;
}
}
// set the low sequence if this is the first record received
// after a flush
if (lowSeq == null)
lowSeq = event.getSequenceNumber();
// high sequence is always the latest value
highSeq = new BigInteger(event.getSequenceNumber());
// extract the data from the input event
try {
extractedItems = dataExtractor.getData(event);
} catch (SerializationException se) {
// customer may have elected to suppress serialisation
// errors if the stream is expected have heterogenous data
// on it
if (this.raiseExceptionOnDataExtractionErrors) {
throw se;
} else {
logWarn(String.format(
"Serialisation Exception Sequence %s Partition Key %s",
event.getSequenceNumber(),
event.getPartitionKey()), se);
}
}
// data extractor may have returned multiple data elements, or
// be empty if there were serialisation problems which are
// suppressed
if (extractedItems != null) {
aggregatedEventCount++;
for (AggregateData data : extractedItems) {
// run the idempotency check
if (!this.idempotencyCheck.doProcess(
event.getPartitionKey(),
event.getSequenceNumber(), data,
event.getData())) {
logInfo(String
.format("Ignoring Event %s as it failed Idempotency Check",
event.getPartitionKey()));
continue;
}
aggregatedElementCount++;
// if the data extractor didn't have a date value to
// extract, then use the current time
eventDate = data.getDate();
if (eventDate == null) {
eventDate = new Date(System.currentTimeMillis());
}
// generate the local updates, one per time horizon that
// is requested
for (TimeHorizon h : timeHorizons) {
// atomically update the aggregate table with event
// count or count + summaries
cache.update(
aggregatorType,
data.getLabels(),
(timeHorizons.size() > 1 ? h
.getItemWithMultiValueFormat(eventDate)
: h.getValue(eventDate)), h, event
.getSequenceNumber(), 1, data
.getSummaries(), dataExtractor
.getSummaryConfig());
}
}
}
}
logInfo(String
.format("Aggregation Complete - %s Records and %s Elements in %s ms",
aggregatedEventCount, aggregatedElementCount,
(System.currentTimeMillis() - start)));
} catch (SerializationException se) {
shutdown(true, InventoryModel.STATE.SERIALISATION_ERROR);
LOG.error(se);
throw se;
} catch (Exception e) {
shutdown(true, InventoryModel.STATE.UNKNOWN_ERROR);
LOG.error(e);
throw e;
}
}
/**
* Return the stored value for a label and date value at the configured time
* granularity
*
* @param label
* The Aggregated Label Value to get data for
* @param dateValue
* The Date Value to obtain data from
* @param h
* The Time Horizon to query
* @return
*/
public List<Map<String, AttributeValue>> queryValue(String label,
Date dateValue, ComparisonOperator comp) throws Exception {
if (!(this.dataStore instanceof DynamoDataStore)) {
throw new Exception(
"Unable to Query by Date unless Data Store is Dynamo DB");
}
if (comp != null && comp.equals(ComparisonOperator.BETWEEN)) {
throw new InvalidConfigurationException(
"Between Operator Not Supported");
}
return ((DynamoDataStore) this.dataStore).queryEngine().queryByKey(
label, dateValue, comp);
}
/**
* Query all data in the data store for a given range of date values and
* time horizon
*
* @param dateValue
* The date to search relative to
* @param h
* The Time Horizon to limit search to
* @param comp
* The Comparison Operator to be applied to the dateValue, such
* as 'equal' EQ or 'greater than' GT
* @return A list of data stored in Dynamo DB for the time range
* @throws Exception
*/
public List<Map<String, AttributeValue>> queryByDate(Date dateValue,
TimeHorizon h, ComparisonOperator comp, int threads)
throws Exception {
if (!(this.dataStore instanceof DynamoDataStore)) {
throw new Exception(
"Unable to Query by Date unless Data Store is Dynamo DB");
}
if (comp.equals(ComparisonOperator.BETWEEN)) {
throw new InvalidConfigurationException(
"Between Operator Not Supported");
}
// resolve the query date based on if we are managing multiple time
// values or a single
String queryDate = null;
if (this.timeHorizons.size() > 1) {
queryDate = h.getItemWithMultiValueFormat(dateValue);
} else {
queryDate = h.getValue(dateValue);
}
// setup the query condition on date
Map<String, Condition> conditions = new HashMap<>();
Condition dateCondition = new Condition().withComparisonOperator(comp)
.withAttributeValueList(new AttributeValue().withS(queryDate));
conditions.put(this.dataExtractor.getDateValueName(), dateCondition);
List<Map<String, AttributeValue>> items = ((DynamoDataStore) this.dataStore)
.queryEngine().parallelQueryDate(
this.dataExtractor.getDateValueName(), conditions,
threads);
return items;
}
public List<TableKeyStructure> parallelQueryKeys(QueryKeyScope scope,
int threads) throws Exception {
if (!(this.dataStore instanceof DynamoDataStore)) {
throw new Exception(
"Unable to Query Keys unless Data Store is Dynamo DB");
}
logInfo(String
.format("Executing Unique Key Scan on %s with Scope %s using %s Threads",
this.tableName, scope.toString(), threads));
return ((DynamoDataStore) this.dataStore).queryEngine()
.parallelQueryKeys(scope, threads);
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/StreamAggregatorUtils.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.io.File;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
import com.amazonaws.services.dynamodbv2.model.GetItemRequest;
import com.amazonaws.services.kinesis.AmazonKinesisClient;
import com.amazonaws.services.kinesis.aggregators.cache.UpdateKey;
import com.amazonaws.services.kinesis.aggregators.datastore.DynamoUtils;
import com.amazonaws.services.kinesis.model.LimitExceededException;
import com.amazonaws.services.kinesis.model.ResourceNotFoundException;
import com.amazonaws.services.kinesis.model.Shard;
import com.amazonaws.services.kinesis.model.StreamDescription;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
/**
* Utility methods used across the Amazon Kinesis Aggregators framework.
*/
public class StreamAggregatorUtils {
private static final Log LOG = LogFactory
.getLog(StreamAggregatorUtils.class);
private static final String rsTimeformat = "yyyy-mm-dd hh:mi:ss";
private static final ObjectMapper mapper = new ObjectMapper();
static {
mapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY,
true);
mapper.configure(DeserializationFeature.WRAP_EXCEPTIONS, false);
}
private StreamAggregatorUtils() {
}
/**
* Helper method which converts input values from Aggregator configurations
* into names for attributes in Dynamo DB. In particular this supports
* Object based Aggregators who will be configured using get methods. For
* example, this methods will turn 'getValue' into 'value' and 'isSomething'
* to 'isSomething'.
*
* @param methodName
* The name of the method to be converted into an Attribute Name.
* @return A string value to be used as the corresponding attribute name.
*/
public static String methodToColumn(String methodName) {
if (methodName.startsWith("get")) {
return methodName.substring(3, 4).toLowerCase()
+ methodName.substring(4);
} else {
return methodName.substring(0, 1).toLowerCase()
+ methodName.substring(1);
}
}
/**
* Returns a statement which can be used to create an External Table in Hive
* which wraps the Aggregator Table indicated, using the required name in
* Hive.
*
* @param dynamoClient
* Dynamo DB Client to use for connection to Dynamo DB.
* @param hiveTableName
* The table name to generate for the Hive Table.
* @param dynamoTable
* The name of the aggregator table in Dynamo DB.
* @return A CREATE EXTERNAL TABLE statement to be used in Hive
* @throws Exception
*/
public static String getDynamoHiveWrapper(AmazonDynamoDB dynamoClient,
String hiveTableName, String dynamoTable) throws Exception {
LOG.info("Generating Hive Integration Statement");
StringBuffer sb = new StringBuffer();
sb.append(String.format("CREATE EXTERNAL TABLE %s(", hiveTableName));
// add the hive table spec
List<String> tableDefinition = DynamoUtils.getDictionaryEntry(
dynamoClient, dynamoTable);
for (String s : tableDefinition) {
sb.append(String.format("%s string,", s));
}
sb.replace(sb.length() - 1, sb.length(), "");
sb.append(String
.format(") STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler' TBLPROPERTIES (\"dynamodb.table.name\" = \"%s\", \"dynamodb.column.mapping\" = \"",
dynamoTable));
for (String s : tableDefinition) {
sb.append(String.format("%s:%s,", s, s));
}
sb.replace(sb.length() - 1, sb.length(), "");
sb.append("))");
return sb.toString();
}
/**
* Helper method to generate a Redshift CREATE TABLE command which matches
* the structure of the aggregate table, and a COPY command which will load
* the table data from an Aggregator table into a Redshift Table. * @param
* dynamoClient Dynamo DB Client to use for connection to Dynamo DB.
*
* @param redshiftTableName
* The table name to use in Redshift.
* @param dynamoTable
* The Aggregator table name in Dynamo DB.
* @return A String which contains the create table and copy commands to be
* issued against redshift.
*/
public static String getRedshiftCopyCommand(
final AmazonDynamoDB dynamoClient, String redshiftTableName,
String dynamoTable) throws Exception {
LOG.info("Generating Redshift Copy Command");
StringBuffer sb = new StringBuffer();
// generate the create table statement
sb.append(String.format("CREATE TABLE %S(\n", redshiftTableName));
int i = 0;
List<String> tableStructure = DynamoUtils.getDictionaryEntry(
dynamoClient, dynamoTable);
String columnSpec = null;
String dataType = null;
for (String s : tableStructure) {
i++;
switch (s) {
case StreamAggregator.LAST_WRITE_SEQ:
dataType = "BIGINT";
break;
case StreamAggregator.LAST_WRITE_TIME:
dataType = "TIMESTAMP";
break;
case StreamAggregator.EVENT_COUNT:
dataType = "INT";
break;
default:
if (s.contains("-SUM") || s.contains("-MIN")
|| s.contains("-MAX")) {
dataType = "INT";
} else {
dataType = "VARCHAR(1000)";
}
break;
}
;
columnSpec = s + " " + dataType;
if (i == tableStructure.size()) {
sb.append(columnSpec);
} else {
sb.append(columnSpec + ",");
}
}
sb.append(");\n\n");
// generate the copy command
sb.append(String
.format("copy %s from 'dynamodb://%s' credentials 'aws_access_key_id=<Your-Access-Key-ID>;aws_secret_access_key=<Your-Secret-Access-Key>' readratio 50 timeformat 'yyyy-MM-dd hh:mi:ss';",
redshiftTableName, dynamoTable, rsTimeformat));
return sb.toString();
}
/**
* Index name which should be used for the last write sequence GSI on a
* table
*
* @param dynamoTable
* The table name in Dynamo DB.
* @return The name for the global secondary index on the table for last
* write sequence.
*/
public static final String getLastWriteSeqIndexName(String dynamoTable) {
return dynamoTable + "-seq";
}
/**
* Index name which should be used for the last write sequence GSI on a
* table
*
* @param dynamoTable
* The table name in Dynamo DB.
* @return The name for the global secondary index on the table for last
* write sequence.
*/
public static final String getDateDimensionIndexName(String dynamoTable,
String dateAttribute) {
return String.format("%s-%s", dynamoTable, dateAttribute);
}
/**
* Method which will generate a correctly formatted primary key for a dynamo
* table hosting aggregated data.
*
* @param updateKey
* An {@link UpdateKey} which should be pivoted into a key.
* @return
*/
public static Map<String, AttributeValue> getTableKey(UpdateKey updateKey) {
return getTableKey(updateKey.getAggregateColumnName(),
updateKey.getAggregatedValue(),
updateKey.getDateValueColumnName(), updateKey.getDateValue());
}
/**
* Method which will generate a correctly formatted primary key for a dynamo
* table hosting aggregated data.
*
* @param keyColumnName
* The attribute name in the table to be used as the first part
* of a hash key.
* @param fieldValue
* The value of the hash key to query for.
* @param dateColumnName
* The attribute name of the date column to be used as the range
* key.
* @param dateValue
* The value of the range key value to query for.
* @return
*/
protected static Map<String, AttributeValue> getTableKey(
String keyColumnName, String fieldValue, String dateColumnName,
String dateValue) {
HashMap<String, AttributeValue> key = new HashMap<>();
key.put(keyColumnName, new AttributeValue().withS(fieldValue));
key.put(dateColumnName, new AttributeValue().withS(dateValue));
return key;
}
protected static Map<String, AttributeValue> getValue(
final AmazonDynamoDB dynamoClient, final String tableName,
final UpdateKey key) {
GetItemRequest req = new GetItemRequest().withTableName(tableName)
.withKey(getTableKey(key));
return dynamoClient.getItem(req).getItem();
}
protected static String getTableName(final String applicationName,
final String namespace) {
return String.format("%s-%s", applicationName, namespace);
}
public static JsonNode asJsonNode(String s) throws Exception {
return mapper.readTree(s);
}
public static JsonNode asJsonNode(File f) throws Exception {
return mapper.readTree(f);
}
public static JsonNode readJsonValue(JsonNode json, String atPath) {
if (!atPath.contains(".")) {
return json.get(atPath);
} else {
String[] path = atPath.split("\\.");
JsonNode node = json.get(path[0]);
for (int i = 1; i < path.length; i++) {
node = node.path(path[i]);
}
return node;
}
}
public static String readValueAsString(JsonNode json, String atPath) {
JsonNode node = readJsonValue(json, atPath);
return node == null ? null : node.asText();
}
/**
* Get a list of all Open shards ordered by their start hash
*
* @param streamName
* @return A Map of only Open Shards indexed by the Shard ID
*/
public static Map<String, Shard> getOpenShards(
AmazonKinesisClient kinesisClient, String streamName)
throws Exception {
Map<String, Shard> shardMap = new LinkedHashMap<>();
final int BACKOFF_MILLIS = 10;
final int MAX_DESCRIBE_ATTEMPTS = 10;
int describeAttempts = 0;
StreamDescription stream = null;
try {
do {
try {
stream = kinesisClient.describeStream(streamName)
.getStreamDescription();
} catch (LimitExceededException e) {
Thread.sleep(2 ^ describeAttempts * BACKOFF_MILLIS);
describeAttempts++;
}
} while (stream == null && describeAttempts < MAX_DESCRIBE_ATTEMPTS);
} catch (InterruptedException e) {
LOG.error(e);
throw e;
}
if (stream == null) {
throw new Exception(String.format(
"Unable to describe Stream after %s attempts",
MAX_DESCRIBE_ATTEMPTS));
}
Collection<String> openShardNames = new ArrayList<String>();
// load all the shards on the stream
for (Shard shard : stream.getShards()) {
openShardNames.add(shard.getShardId());
shardMap.put(shard.getShardId(), shard);
// remove this shard's parents from the set of active shards -
// we
// can't do anything to them
if (shard.getParentShardId() != null) {
openShardNames.remove(shard.getParentShardId());
}
if (shard.getAdjacentParentShardId() != null) {
openShardNames.remove(shard.getAdjacentParentShardId());
}
}
// create a List of Open shards for sorting
List<Shard> shards = new ArrayList<Shard>();
for (String s : openShardNames) {
shards.add(shardMap.get(s));
}
// sort the list into lowest start hash order
Collections.sort(shards, new Comparator<Shard>() {
public int compare(Shard o1, Shard o2) {
return new BigInteger(o1.getHashKeyRange().getStartingHashKey())
.compareTo(new BigInteger(o2.getHashKeyRange()
.getStartingHashKey()));
}
});
// rebuild the shard map into the correct order
shardMap.clear();
for (Shard s : shards) {
shardMap.put(s.getShardId(), s);
}
return shardMap;
}
public static Shard getFirstShard(AmazonKinesisClient kinesisClient,
String streamName) throws Exception {
return getOpenShards(kinesisClient, streamName).values().iterator()
.next();
}
public static String getFirstShardName(AmazonKinesisClient kinesisClient,
String streamName) throws Exception {
return getFirstShard(kinesisClient, streamName).getShardId();
}
public static int getShardCount(AmazonKinesisClient kinesisClient,
String streamName) throws Exception {
return getOpenShards(kinesisClient, streamName).keySet().size();
}
/**
* Strip the TimeHorizon abbreviation from a dateValueItem used in DynamoDB
* with multi-value format
*/
public static String extractDateFromMultivalue(TimeHorizon t, String date) {
if (date.startsWith(t.getAbbrev())) {
return date.replaceAll("^" + t.getAbbrev() + "-", "");
} else {
return date;
}
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/TableKeyStructure.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.util.HashSet;
import java.util.Set;
public class TableKeyStructure {
private String labelAttributeName, labelAttributeValue, dateAttributeName;
private Set<String> dateValues;
public TableKeyStructure() {
}
public TableKeyStructure(String labelAttributeName, String labelAttributeValue,
String dateAttributeName) {
this.labelAttributeName = labelAttributeName;
this.labelAttributeValue = labelAttributeValue;
this.dateAttributeName = dateAttributeName;
}
public TableKeyStructure(String labelAttributeName, String labelAttributeValue,
String dateAttributeName, String dateAttributeValue) {
this.labelAttributeName = labelAttributeName;
this.labelAttributeValue = labelAttributeValue;
this.dateAttributeName = dateAttributeName;
this.dateValues = new HashSet<>();
this.dateValues.add(dateAttributeValue);
}
public TableKeyStructure withDateValue(String dateValue) {
if (this.dateValues == null) {
this.dateValues = new HashSet<>();
}
this.dateValues.add(dateValue);
return this;
}
public String getLabelAttributeName() {
return this.labelAttributeName;
}
public String getLabelAttributeValue() {
return this.labelAttributeValue;
}
public String getDateAttributeName() {
return this.dateAttributeName;
}
public Set<String> getDateValues() {
return this.dateValues;
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/TimeHorizon.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Date;
import java.util.List;
public enum TimeHorizon {
SECOND(0, "MM-dd HH:mm:ss", "s"), MINUTE(1, "MM-dd HH:mm:00", "m"), MINUTES_GROUPED(1, null,
"mb") {
private Calendar calendar = Calendar.getInstance();
private int scope;
@Override
public int getGranularity() {
return this.scope;
}
@Override
public void setGranularity(int bucketSize) {
this.scope = bucketSize;
}
@Override
public String getValue(Date forDate) {
calendar.setTime(forDate);
int minutes = calendar.get(Calendar.MINUTE);
int bucket = new Double(Math.floor(minutes / scope) * scope).intValue();
return String.format("%s:%02d:00",
new SimpleDateFormat("yyyy-MM-dd HH").format(forDate), bucket);
}
},
HOUR(2, "MM-dd HH:00:00", "H"), DAY(3, "MM-dd 00:00:00", "d"), WEEK(4, "ww", "W"), MONTH(5, "MM-01 00:00:00", "M"), YEAR(
6, "01-01 00:00:00", "Y"), FOREVER(999, "", "*") {
/**
* Override the getValue method, as TimeHorizon.FOREVER is for all
* values regardless of time period. We'll set the value to '*' as
* Dynamo wont allow an empty value
*/
@Override
public String getValue(Date forDate) {
return "*";
}
};
private TimeHorizon(int placemark, String mask, String abbrev) {
this.placemark = placemark;
this.mask = mask;
this.abbrev = abbrev;
}
private int placemark;
private String mask;
private String abbrev;
private SimpleDateFormat getMask() {
return new SimpleDateFormat("yyyy-" + this.mask);
}
public String getAbbrev() {
return this.abbrev;
}
public String getItemWithMultiValueFormat(Date dateValue) {
return getAbbrev() + "-" + getValue(dateValue);
}
public String getValue(Date forDate) {
return getMask().format(forDate);
}
/**
* Returns the full hierarchy of TimeHorizon values from this Horizon to
* FOREVER
*
* @return
*/
public List<TimeHorizon> getFullHierarchy() {
return getHierarchyTo(TimeHorizon.FOREVER);
}
/**
* Get a list of all TimeHorizons in decreasing granularity, to the
* indicated Time Horizon. For example, if we requested
* TimeHorizon.MINUTE.getHierarchyTo(TimeHorizon.MONTH), we would receive a
* list of MINUTE, HOUR, DAY, MONTH
*
* @param t
* @return
*/
public List<TimeHorizon> getHierarchyTo(TimeHorizon t) {
List<TimeHorizon> hierarchy = new ArrayList<>();
for (TimeHorizon h : TimeHorizon.values()) {
// don't include Minutes Group in automated hierarchies as they are
// a peer to Minutes
if (h.placemark >= this.placemark && h.placemark <= t.placemark
&& !h.equals(TimeHorizon.MINUTES_GROUPED)) {
hierarchy.add(h);
}
}
return hierarchy;
}
public int getGranularity() throws Exception {
throw new Exception("Not Implemented");
}
public void setGranularity(int scope) throws Exception {
throw new Exception("Not Implemented");
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/Aggregate.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.annotations;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import com.amazonaws.services.kinesis.aggregators.AggregatorType;
import com.amazonaws.services.kinesis.aggregators.TimeHorizon;
import com.amazonaws.services.kinesis.aggregators.datastore.DynamoDataStore;
import com.amazonaws.services.kinesis.aggregators.metrics.CloudWatchMetricsEmitter;
/**
* Annotations to indicate that a Class contains an Aggregator Configuration
*/
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface Aggregate {
/**
* The type of Aggregator to create. Default is COUNT.
*
* @return
*/
AggregatorType type() default AggregatorType.COUNT;
/** The list of Time Horizons to Aggregate on */
TimeHorizon[] timeHorizons() default TimeHorizon.HOUR;
int[] timeGranularity() default -1;
/**
* The namespace for the Aggregation Data.
*
* @return
*/
String namespace() default "";
/**
* Should the Aggregator fail on errors in reading data from the stream for
* Aggregation.
*
* @return
*/
boolean failOnDataExtractionErrors() default true;
/**
* Should the aggregator publish intrumentation metrics? The default metrics
* emitter is CloudWatch
*
* @return
*/
boolean emitMetrics() default false;
/**
* Configure an IDataStore other than the default Dynamo DB Datastore
*
* @return
*/
Class dataStore() default DynamoDataStore.class;
/**
* Configure an IMetricsEmitter other than the default CloudWatch metrics
* service
*/
Class metricsEmitter() default CloudWatchMetricsEmitter.class;
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/AnnotationProcessor.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.annotations;
import java.lang.annotation.Annotation;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.amazonaws.services.kinesis.aggregators.AggregatorType;
import com.amazonaws.services.kinesis.aggregators.LabelSet;
import com.amazonaws.services.kinesis.aggregators.StreamAggregatorUtils;
import com.amazonaws.services.kinesis.aggregators.TimeHorizon;
import com.amazonaws.services.kinesis.aggregators.datastore.IDataStore;
import com.amazonaws.services.kinesis.aggregators.exception.ClassNotAnnotatedException;
import com.amazonaws.services.kinesis.aggregators.exception.InvalidConfigurationException;
import com.amazonaws.services.kinesis.aggregators.metrics.IMetricsEmitter;
import com.amazonaws.services.kinesis.aggregators.summary.SummaryCalculation;
import com.amazonaws.services.kinesis.aggregators.summary.SummaryConfiguration;
import com.amazonaws.services.kinesis.aggregators.summary.SummaryElement;
/**
* AnnotationProcess provides a helper mechanism to extract information from an
* Annotationed Class which will be used to configure an Object Serialisation
* based Aggregation. See
* {@link com.amazonaws.services.kinesis.aggregators.factory.ObjectAggregatorFactory}
* .
*/
public class AnnotationProcessor {
@SuppressWarnings("rawtypes")
private Class clazz;
private LabelSet labelSet = new LabelSet();
private List<String> labelMethodNames = new ArrayList<>();
private Map<String, Method> labelMethodMap = new LinkedHashMap<>();
private String dateMethodName;
private Method dateMethod;
private Map<String, Method> summaryMethods = new HashMap<>();
private SummaryConfiguration summaryConfig = new SummaryConfiguration();
private AggregatorType type;
private List<TimeHorizon> timeHorizons;
private boolean timeHierarchy;
private String namespace;
private boolean failOnDataExtractionErrors = true;
private boolean emitMetrics = false;
private Class<IDataStore> dataStore;
private Class<IMetricsEmitter> metricsEmitter;
private AnnotationProcessor() {
}
/**
* Create a new Annotation Processor for an Annotated Class.
*
* @param clazz The Class to extract annotation information from.
* @throws Exception
*/
public AnnotationProcessor(@SuppressWarnings("rawtypes") Class clazz) throws Exception {
this.clazz = clazz;
boolean isAnnotated = false;
// get the class annotations
for (Annotation a : this.clazz.getAnnotations()) {
if (a.annotationType().equals(Aggregate.class)) {
isAnnotated = true;
Aggregate annotatedObject = (Aggregate) a;
this.namespace = annotatedObject.namespace();
if (this.namespace.contains(" "))
throw new ClassNotAnnotatedException("Namespace may not contain spaces");
this.type = annotatedObject.type();
// process time horizon annotations
int[] timeGranularities = annotatedObject.timeGranularity();
TimeHorizon[] horizons = annotatedObject.timeHorizons();
this.timeHorizons = new ArrayList<>();
int i = 0;
for (TimeHorizon h : horizons) {
if (h.equals(TimeHorizon.MINUTES_GROUPED)) {
try {
// prevent use of the default time granularity
if (timeGranularities[i] == -1) {
throw new ArrayIndexOutOfBoundsException();
}
h.setGranularity(timeGranularities[i]);
} catch (ArrayIndexOutOfBoundsException e) {
throw new InvalidConfigurationException(
"Unable to generate a MINUTES_GROUPED Time Horizon without configuration of timeGranularity");
}
}
this.timeHorizons.add(h);
i++;
}
this.failOnDataExtractionErrors = annotatedObject.failOnDataExtractionErrors();
this.emitMetrics = annotatedObject.emitMetrics();
this.dataStore = annotatedObject.dataStore();
this.metricsEmitter = annotatedObject.metricsEmitter();
}
}
if (!isAnnotated)
throw new ClassNotAnnotatedException(
"Cannot get Aggregator Config from non-Annotated Class");
// process the method annotations
if (isAnnotated) {
for (Method m : this.clazz.getDeclaredMethods()) {
// label method
if (m.getAnnotation(Label.class) != null) {
this.labelMethodNames.add(m.getName());
m.setAccessible(true);
this.labelMethodMap.put(m.getName(), m);
this.labelSet.put(m.getName(), null);
}
// date method
if (m.getAnnotation(DateValue.class) != null) {
this.dateMethodName = m.getName();
m.setAccessible(true);
this.dateMethod = m;
}
// summary methods
Annotation summary = m.getAnnotation(Summary.class);
if (summary != null) {
m.setAccessible(true);
this.summaryMethods.put(m.getName(), m);
// process the summary configuration
SummaryCalculation[] requestedCalcs = ((Summary) summary).type();
if (requestedCalcs != null) {
for (SummaryCalculation c : requestedCalcs) {
this.summaryConfig.add(m.getName(), new SummaryElement(m.getName(), c));
}
} else {
this.summaryConfig.add(m.getName(), new SummaryElement(m.getName(),
SummaryCalculation.SUM));
}
}
}
}
}
public List<String> getLabelMethodNames() {
return this.labelMethodNames;
}
public Map<String, Method> getLabelMethods() {
return this.labelMethodMap;
}
public String getDateMethodName() {
return this.dateMethodName;
}
public Method getDateMethod() {
return this.dateMethod;
}
public Map<String, Method> getSummaryMethods() {
return this.summaryMethods;
}
public SummaryConfiguration getSummaryConfig() {
return this.summaryConfig;
}
public AggregatorType getType() {
return this.type;
}
public List<TimeHorizon> getTimeHorizon() {
return this.timeHorizons;
}
public boolean hasTimeHierarchy() {
return this.timeHierarchy;
}
public boolean shouldFailOnDataExtractionErrors() {
return this.failOnDataExtractionErrors;
}
public boolean shouldEmitMetrics() {
return this.emitMetrics;
}
public Class<IMetricsEmitter> getMetricsEmitter() {
return this.metricsEmitter;
}
public Class<IDataStore> getDataStore() {
return this.dataStore;
}
public String getNamespace() {
return this.namespace;
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/DateValue.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.annotations;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
/**
* Marker Annotation indicating that a method should be used as the date value
* for Aggregation.
*/
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface DateValue {
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/Label.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.annotations;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
/**
* Marker Annotation indicating that this indicated method is the label to be
* used for Aggregation.
*/
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Label {
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/Summary.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.annotations;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import java.util.Arrays;
import com.amazonaws.services.kinesis.aggregators.summary.SummaryCalculation;
/**
* Annotation which indicates that a method should be used as a summary
* aggregation. If no type is indicated then it will be used as a
* {@link com.amazonaws.services.kinesis.aggregators.SummaryCalculation.SUM}.
*/
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Summary {
/**
* The type of summary calculations to apply to the method.
*
* @return
*/
public SummaryCalculation[] type() default SummaryCalculation.SUM;
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/AbstractQueryServlet.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.app;
import java.io.IOException;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public abstract class AbstractQueryServlet extends HttpServlet {
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
doAction(request, response);
}
public void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
doAction(request, response);
}
protected abstract void doAction(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException;
protected void doError(HttpServletResponse response, String message) throws ServletException {
try {
response.getWriter().print(message);
response.setStatus(400);
} catch (IOException e) {
throw new ServletException(e);
}
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/AggregatorsBeanstalkApp.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.app;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.services.kinesis.aggregators.AggregatorGroup;
import com.amazonaws.services.kinesis.aggregators.AggregatorsConstants;
import com.amazonaws.services.kinesis.aggregators.consumer.AggregatorConsumer;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
public class AggregatorsBeanstalkApp implements ServletContextListener {
private static final Log LOG = LogFactory.getLog(AggregatorsBeanstalkApp.class);
protected static final String AGGREGATOR_GROUP_PARAM = "aggregator-group";
private AggregatorConsumer consumer;
private Thread t;
@Override
public void contextDestroyed(ServletContextEvent arg0) {
try {
consumer.shutdown();
t.interrupt();
} catch (Exception e) {
LOG.error(e);
}
}
@SuppressWarnings({ "unchecked" })
@Override
public void contextInitialized(ServletContextEvent contextEvent) {
String configPath = System.getProperty(AggregatorsConstants.CONFIG_URL_PARAM);
if (configPath != null && !configPath.equals("")) {
LOG.info("Starting Managed Beanstalk Aggregators Worker");
String streamNameParam = System.getProperty(AggregatorsConstants.STREAM_NAME_PARAM);
String appNameParam = System.getProperty(AggregatorsConstants.APP_NAME_PARAM);
String regionNameParam = System.getProperty(AggregatorsConstants.REGION_PARAM);
String streamPosParam = System.getProperty(AggregatorsConstants.STREAM_POSITION_PARAM);
String maxRecordsParam = System.getProperty(AggregatorsConstants.MAX_RECORDS_PARAM);
String environmentParam = System.getProperty(AggregatorsConstants.ENVIRONMENT_PARAM);
String failuresToleratedParam = System.getProperty(AggregatorsConstants.FAILURES_TOLERATED_PARAM);
if (streamNameParam == null || streamNameParam.equals("") || appNameParam == null
|| appNameParam.equals("")) {
LOG.error(String.format(
"Unable to run Beanstalk Managed Aggregator Consumer without Configuration of Parameters %s and %s. Application is Idle.",
AggregatorsConstants.STREAM_NAME_PARAM, AggregatorsConstants.APP_NAME_PARAM));
return;
}
InitialPositionInStream initialPosition = null;
if (streamPosParam != null) {
try {
initialPosition = InitialPositionInStream.valueOf(streamPosParam);
LOG.info(String.format("Starting from %s Position in Stream", streamPosParam));
} catch (Exception e) {
LOG.error(String.format("%s is an invalid Initial Position in Stream",
streamPosParam));
return;
}
}
try {
AggregatorConsumer consumer = new AggregatorConsumer(streamNameParam, appNameParam,
configPath);
// add consumer parameters, if set from System Properties
if (regionNameParam != null && !regionNameParam.equals("")) {
consumer.withRegionName(regionNameParam);
}
if (initialPosition != null) {
consumer.withInitialPositionInStream(initialPosition.name());
}
if (maxRecordsParam != null && !maxRecordsParam.equals("")) {
consumer.withMaxRecords(Integer.parseInt(maxRecordsParam));
}
if (environmentParam != null && !environmentParam.equals("")) {
consumer.withEnvironment(environmentParam);
}
if (failuresToleratedParam != null && !failuresToleratedParam.equals("")) {
consumer.withToleratedWorkerFailures(Integer.parseInt(failuresToleratedParam));
}
// configure the consumer so that the aggregators get
// instantiated
consumer.configure();
AggregatorGroup aggGroup = consumer.getAggregators();
// put the aggregator group reference and configureation
// references into the application context
contextEvent.getServletContext().setAttribute(AGGREGATOR_GROUP_PARAM, aggGroup);
contextEvent.getServletContext().setAttribute(
AggregatorsConstants.STREAM_NAME_PARAM, streamNameParam);
LOG.info("Registered Stream and Aggregator Group with Servlet Context");
// start the consumer
final class ConsumerRunner implements Runnable {
final AggregatorConsumer consumer;
public ConsumerRunner(AggregatorConsumer consumer) {
this.consumer = consumer;
}
@Override
public void run() {
try {
consumer.run();
} catch (Exception e) {
e.printStackTrace();
LOG.error(e);
}
}
}
t = new Thread(new ConsumerRunner(consumer));
t.start();
} catch (Exception e) {
LOG.error(e);
}
} else {
LOG.warn(String.format(
"No Aggregators Configuration File found in Beanstalk Configuration %s. Application is Idle",
AggregatorsConstants.CONFIG_URL_PARAM));
}
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/DateQueryServlet.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.app;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.Date;
import java.util.List;
import java.util.Map;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
import com.amazonaws.services.dynamodbv2.model.ComparisonOperator;
import com.amazonaws.services.kinesis.aggregators.AggregatorGroup;
import com.amazonaws.services.kinesis.aggregators.AggregatorsConstants;
import com.amazonaws.services.kinesis.aggregators.StreamAggregator;
import com.amazonaws.services.kinesis.aggregators.TimeHorizon;
public class DateQueryServlet extends AbstractQueryServlet {
public static final String NAMESPACE_PARAM = "namespace";
public static final String DATE_VALUE_PARAM = "date-value";
public static final String OPERATOR_PARAM = "operator";
public static final String GRANULARITY_PARAM = "granularity";
public static final int QUERY_THREADS = 10;
private void respondWith(HttpServletResponse response,
List<Map<String, AttributeValue>> queryResult) throws IOException {
response.setStatus(200);
// cors grant
response.setHeader("Access-Control-Allow-Origin", "*");
PrintWriter w = response.getWriter();
w.println("[");
int i = 0;
// write out the response values as json
if (queryResult != null) {
for (Map<String, AttributeValue> map : queryResult) {
i++;
int j = 0;
w.print("{");
for (String s : map.keySet()) {
j++;
String toPrint = map.get(s).getS();
if (toPrint == null) {
toPrint = map.get(s).getN();
}
w.print(String.format("\"%s\":\"%s\"", s, toPrint));
if (j != map.keySet().size()) {
w.println(",");
}
}
w.print("}");
if (i != queryResult.size()) {
w.println(",");
}
}
}
w.print("]");
}
public void doAction(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String namespace = request.getParameter(NAMESPACE_PARAM);
String dateValue = request.getParameter(DATE_VALUE_PARAM);
String operator = request.getParameter(OPERATOR_PARAM);
String granularity = request.getParameter(GRANULARITY_PARAM);
// create the date item
Date d = null;
try {
d = StreamAggregator.dateFormatter.parse(dateValue);
} catch (Exception e) {
doError(response, String.format("Date Parameter must be in format %s",
StreamAggregator.dateFormatter.getDateFormatSymbols().toString()));
return;
}
// create the ComparisonOperator for Dynamo from the argument
ComparisonOperator c = null;
try {
c = ComparisonOperator.fromValue(operator);
} catch (Exception e) {
doError(response, String.format("%s is an invalid Comparison Operator", operator));
return;
}
// create the Time Horizon value from the argument
TimeHorizon h = null;
try {
h = TimeHorizon.valueOf(granularity);
} catch (Exception e) {
doError(response, String.format("%s is an invalid Granularity", granularity));
return;
}
String streamName = (String) request.getServletContext().getAttribute(
AggregatorsConstants.STREAM_NAME_PARAM);
AggregatorGroup aggGroup = (AggregatorGroup) request.getServletContext().getAttribute(
AggregatorsBeanstalkApp.AGGREGATOR_GROUP_PARAM);
if (aggGroup == null) {
doError(response, "Aggregator Application Not Initialised");
return;
}
// initialise the aggregator group onto shard 'none' for this operation
// - it may already be initialised
try {
aggGroup.initialize("none");
} catch (Exception e) {
throw new ServletException(e);
}
// put the initialised aggregator group back into the context
request.getServletContext().setAttribute(AggregatorsBeanstalkApp.AGGREGATOR_GROUP_PARAM,
aggGroup);
// acquire the correct aggregator by namespace
for (StreamAggregator agg : aggGroup.getAggregators()) {
if (agg.getNamespace().equals(namespace)) {
// run the query
try {
respondWith(response, agg.queryByDate(d, h, c, QUERY_THREADS));
return;
} catch (Exception e) {
throw new ServletException(e);
}
}
}
// shouldn't get here, so bail with a meaning error on namespace
doError(response,
String.format("Unable to acquire Aggregator with Namespace %s", namespace));
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/FetchConfigurationServlet.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.app;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.HashMap;
import java.util.Map;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import com.amazonaws.services.kinesis.aggregators.AggregatorsConstants;
import com.amazonaws.services.kinesis.aggregators.configuration.ConfigFileUtils;
public class FetchConfigurationServlet extends AbstractQueryServlet {
private void respondWith(HttpServletResponse response, Map<String, String> configItems)
throws IOException {
response.setStatus(200);
// cors grant
response.setHeader("Access-Control-Allow-Origin", "*");
PrintWriter w = response.getWriter();
int i = 0;
// write out the response values as json
w.println("{");
int resultCount = 0;
for (String s : configItems.keySet()) {
resultCount++;
String value = configItems.get(s);
w.print(String.format("\"%s\":%s", s,
value == null ? "null" : String.format("\"%s\"", value)));
if (resultCount != configItems.size()) {
w.println(",");
}
}
w.print("}");
}
@Override
protected void doAction(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
Map<String, String> config = new HashMap<>();
// required items
config.put(AggregatorsConstants.REGION_PARAM,
System.getProperty(AggregatorsConstants.REGION_PARAM));
config.put(AggregatorsConstants.STREAM_NAME_PARAM,
System.getProperty(AggregatorsConstants.STREAM_NAME_PARAM));
config.put(AggregatorsConstants.APP_NAME_PARAM,
System.getProperty(AggregatorsConstants.APP_NAME_PARAM));
config.put(AggregatorsConstants.CONFIG_URL_PARAM,
System.getProperty(AggregatorsConstants.CONFIG_URL_PARAM));
config.put(
"fetch-config-url",
ConfigFileUtils.makeConfigFileURL(System.getProperty(AggregatorsConstants.CONFIG_URL_PARAM)));
// optional items
config.put(AggregatorsConstants.ENVIRONMENT_PARAM,
System.getProperty(AggregatorsConstants.ENVIRONMENT_PARAM));
config.put(AggregatorsConstants.MAX_RECORDS_PARAM,
System.getProperty(AggregatorsConstants.MAX_RECORDS_PARAM));
config.put(AggregatorsConstants.FAILURES_TOLERATED_PARAM,
System.getProperty(AggregatorsConstants.FAILURES_TOLERATED_PARAM));
respondWith(response, config);
} catch (Exception e) {
throw new ServletException(e);
}
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/ListAggregateKeysServlet.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.app;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.List;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import com.amazonaws.services.kinesis.aggregators.AggregatorGroup;
import com.amazonaws.services.kinesis.aggregators.AggregatorsConstants;
import com.amazonaws.services.kinesis.aggregators.StreamAggregator;
import com.amazonaws.services.kinesis.aggregators.TableKeyStructure;
import com.amazonaws.services.kinesis.aggregators.datastore.DynamoQueryEngine.QueryKeyScope;
public class ListAggregateKeysServlet extends AbstractQueryServlet {
public static final String NAMESPACE_PARAM = "namespace";
public static final String SCOPE_PARAM = "scope";
public static final int QUERY_THREADS = 3;
private void respondWith(HttpServletResponse response, List<TableKeyStructure> queryResult)
throws IOException {
response.setStatus(200);
// cors grant
response.setHeader("Access-Control-Allow-Origin", "*");
PrintWriter w = response.getWriter();
w.println("{");
int i = 0;
// write out the response values as json
if (queryResult != null) {
int result = 0;
for (TableKeyStructure t : queryResult) {
if (result == 0) {
w.println(String.format("\"labelName\":\"%s\",", t.getLabelAttributeName()));
w.println(String.format("\"dateName\":\"%s\",", t.getDateAttributeName()));
w.println("\"values\":[");
}
// write the value as a struct
w.print("{");
w.print(String.format("\"value\":\"%s\"", t.getLabelAttributeValue()));
int dateItem = 0;
if (t.getDateValues() != null) {
if (dateItem == 0) {
w.print(",\n\"dates\":[");
}
for (String s : t.getDateValues()) {
// write the date value
w.print(String.format("\"%s\"", s));
if (dateItem != t.getDateValues().size() - 1) {
w.println(",");
} else {
w.print("]");
}
dateItem++;
}
}
w.print("}");
if (result != queryResult.size() - 1) {
w.println(",");
}
result++;
}
}
w.print("]}");
}
@Override
protected void doAction(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String namespace = request.getParameter(NAMESPACE_PARAM);
String scope = request.getParameter(SCOPE_PARAM);
// resolve the scope
QueryKeyScope queryScope = null;
try {
queryScope = QueryKeyScope.valueOf(scope);
} catch (Exception e) {
doError(response, String.format("Invalid Query Scope %s", scope));
return;
}
String streamName = (String) request.getServletContext().getAttribute(
AggregatorsConstants.STREAM_NAME_PARAM);
AggregatorGroup aggGroup = (AggregatorGroup) request.getServletContext().getAttribute(
AggregatorsBeanstalkApp.AGGREGATOR_GROUP_PARAM);
if (aggGroup == null) {
doError(response, "Aggregator Application Not Initialised");
return;
}
// initialise the aggregator group onto shard 'none' for this operation
// - it may already be initialised
try {
aggGroup.initialize("none");
} catch (Exception e) {
throw new ServletException(e);
}
// put the initialised aggregator group back into the context
request.getServletContext().setAttribute(AggregatorsBeanstalkApp.AGGREGATOR_GROUP_PARAM,
aggGroup);
// acquire the correct aggregator by namespace
for (StreamAggregator agg : aggGroup.getAggregators()) {
if (agg.getNamespace().equals(namespace)) {
// run the query
try {
respondWith(response, agg.parallelQueryKeys(queryScope, QUERY_THREADS));
return;
} catch (Exception e) {
throw new ServletException(e);
}
}
}
// shouldn't get here, so bail with a meaning error on namespace
doError(response,
String.format("Unable to acquire Aggregator with Namespace %s", namespace));
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/QueryByLabelServlet.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.app;
import java.io.IOException;
import java.io.PrintWriter;
import java.text.ParseException;
import java.util.Date;
import java.util.List;
import java.util.Map;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import com.amazonaws.services.dynamodbv2.model.AttributeValue;
import com.amazonaws.services.dynamodbv2.model.ComparisonOperator;
import com.amazonaws.services.kinesis.aggregators.AggregatorGroup;
import com.amazonaws.services.kinesis.aggregators.AggregatorsConstants;
import com.amazonaws.services.kinesis.aggregators.StreamAggregator;
public class QueryByLabelServlet extends AbstractQueryServlet {
public static final String NAMESPACE_PARAM = "namespace";
public static final String LABEL_VALUE_PARAM = "label-value";
public static final String DATE_VALUE_PARAM = "date-value";
public static final String OPERATOR_PARAM = "operator";
private void respondWith(HttpServletResponse response,
List<Map<String, AttributeValue>> queryResult) throws IOException {
response.setStatus(200);
// cors grant
response.setHeader("Access-Control-Allow-Origin", "*");
PrintWriter w = response.getWriter();
w.println("[");
int i = 0;
// write out the response values as json
if (queryResult != null) {
int resultCount = 0;
for (Map<String, AttributeValue> map : queryResult) {
resultCount++;
int mapCount = 0;
if (map != null) {
w.println("{");
for (String s : map.keySet()) {
mapCount++;
if (map.get(s).getN() == null) {
w.print(String.format("\"%s\":\"%s\"", s, map.get(s).getS()));
} else {
w.print(String.format("\"%s\":%s", s, map.get(s).getN()));
}
if (mapCount != map.size()) {
w.println(",");
}
}
w.print("}");
if (resultCount != queryResult.size()) {
w.println(",");
}
}
}
}
w.print("]");
}
@Override
protected void doAction(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String namespace = request.getParameter(NAMESPACE_PARAM);
String labelValue = request.getParameter(LABEL_VALUE_PARAM);
String dateValue = request.getParameter(DATE_VALUE_PARAM);
String operator = request.getParameter(OPERATOR_PARAM);
// have to provide namespace and label
if (namespace == null) {
doError(response, String.format("Argument '%s' must not be null", NAMESPACE_PARAM));
return;
}
if (labelValue == null || labelValue.equals("")) {
doError(response, String.format("Argument '%s' must not be null", LABEL_VALUE_PARAM));
return;
}
// if date value is provided, the so too must operator and granularity
ComparisonOperator setOperator = null;
if (dateValue != null && operator == null) {
setOperator = ComparisonOperator.EQ;
}
if (operator != null) {
try {
setOperator = ComparisonOperator.fromValue(operator);
} catch (Exception e) {
doError(response, String.format("%s is an invalid Comparison Operator", operator));
return;
}
}
String streamName = (String) request.getServletContext().getAttribute(
AggregatorsConstants.STREAM_NAME_PARAM);
AggregatorGroup aggGroup = (AggregatorGroup) request.getServletContext().getAttribute(
AggregatorsBeanstalkApp.AGGREGATOR_GROUP_PARAM);
if (aggGroup == null) {
doError(response, "Aggregator Application Not Initialised");
return;
} else {
// initialise the aggregator group onto shard 'none' for this
// operation
// - it may already be initialised
try {
aggGroup.initialize("none");
} catch (Exception e) {
throw new ServletException(e);
}
}
Date dateValueAsDate = null;
if (dateValue != null) {
try {
dateValueAsDate = StreamAggregator.dateFormatter.parse(dateValue);
} catch (ParseException e1) {
throw new ServletException(e1);
}
}
// put the initialised aggregator group back into the context
request.getServletContext().setAttribute(AggregatorsBeanstalkApp.AGGREGATOR_GROUP_PARAM,
aggGroup);
// acquire the correct aggregator by namespace
for (StreamAggregator agg : aggGroup.getAggregators()) {
if (agg.getNamespace().equals(namespace)) {
// run the query
try {
respondWith(response, agg.queryValue(labelValue, dateValueAsDate, setOperator));
return;
} catch (Exception e) {
throw new ServletException(e);
}
}
}
// shouldn't get here, so bail with a meaning error on namespace
doError(response,
String.format("Unable to acquire Aggregator with Namespace %s", namespace));
}
}
================================================
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/ShowConfigFileServlet.java
================================================
/**
* Amazon Kinesis Aggregators
*
* Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Amazon Software License (the "License").
* You may not use this file except in compliance with the License.
* A copy of the License is located at
*
* http://aws.amazon.com/asl/
*
* or in the "license" file accompanying this file. This file is distributed
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language governing
* permissions and limitations under the License.
*/
package com.amazonaws.services.kinesis.aggregators.app;
import java.io.IOException;
import java.net.URL;
import java.util.Date;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import com.amazonaws.HttpMethod;
import com.amazonaws.services.kinesis.aggregators.AggregatorsConstants;
import com.amazonaws.services.kinesis.aggregators.configuration.ConfigFileUtils;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
public class ShowConfigFileServlet extends AbstractQueryServlet {
private static final Log LOG = LogFactory.getLog(ShowConfigFileServlet.class);
@Override
protected void doAction(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
String configUrl = System.getProperty(AggregatorsConstants.CONFIG_URL_PARAM);
String url = null;
if (configUrl == null) {
response.setStatus(404);
} else {
url = ConfigFileUtils.makeConfigFileURL(configUrl);
LOG.info(String.format("Sending Redirect for Config File to S3 Temporary URL %s",
url));
response.setHeader("Access-Control-Allow-Origin", "*");
response.sendRedirect(url);
}
} catch (Exception e) {
throw new ServletException(e);
}
}
}
================================================
FILE: src/main/java/com/amazonaws/service
gitextract_tmef7pg6/
├── .gitignore
├── LICENSE.txt
├── NOTICE.txt
├── README.md
├── assembly.xml
├── dist/
│ ├── AmazonKinesisAggregators.jar-complete.jar
│ ├── AmazonKinesisAggregators.war
│ ├── amazon-kinesis-aggregators-.9.2.8.jar
│ ├── amazon-kinesis-aggregators-.9.2.9-sources.jar
│ └── amazon-kinesis-aggregators-.9.2.9.jar
├── pom.xml
├── sample/
│ ├── bin/
│ │ └── run-producer.sh
│ ├── java/
│ │ ├── model/
│ │ │ ├── SensorReading.java
│ │ │ └── SensorState.java
│ │ └── producer/
│ │ └── SensorReadingProducer.java
│ └── resources/
│ ├── BySegment-CSV.json
│ ├── BySegment-Json.json
│ └── BySegment-Regex.json
└── src/
├── .gitkeep
├── log4j.properties
└── main/
├── WebContent/
│ ├── .ebextensions/
│ │ └── as.config
│ ├── META-INF/
│ │ └── MANIFEST.MF
│ ├── WEB-INF/
│ │ └── web.xml
│ ├── index.html
│ └── styles/
│ └── styles.css
└── java/
└── com/
└── amazonaws/
└── services/
└── kinesis/
├── aggregators/
│ ├── AggregateData.java
│ ├── AggregatorGroup.java
│ ├── AggregatorType.java
│ ├── AggregatorsConstants.java
│ ├── EnvironmentType.java
│ ├── IStreamAggregator.java
│ ├── InputEvent.java
│ ├── InventoryModel.java
│ ├── InventoryStatus.java
│ ├── LabelSet.java
│ ├── StreamAggregator.java
│ ├── StreamAggregatorUtils.java
│ ├── TableKeyStructure.java
│ ├── TimeHorizon.java
│ ├── annotations/
│ │ ├── Aggregate.java
│ │ ├── AnnotationProcessor.java
│ │ ├── DateValue.java
│ │ ├── Label.java
│ │ └── Summary.java
│ ├── app/
│ │ ├── AbstractQueryServlet.java
│ │ ├── AggregatorsBeanstalkApp.java
│ │ ├── DateQueryServlet.java
│ │ ├── FetchConfigurationServlet.java
│ │ ├── ListAggregateKeysServlet.java
│ │ ├── QueryByLabelServlet.java
│ │ ├── ShowConfigFileServlet.java
│ │ └── ShowConfigurationServlet.java
│ ├── cache/
│ │ ├── AggregateCache.java
│ │ ├── UpdateKey.java
│ │ └── UpdateValue.java
│ ├── cli/
│ │ └── AggregatorsCli.java
│ ├── configuration/
│ │ ├── ConfigFileUtils.java
│ │ ├── DataExtractor.java
│ │ ├── ExternalConfigurationModel.java
│ │ └── json.schema
│ ├── consumer/
│ │ └── AggregatorConsumer.java
│ ├── datastore/
│ │ ├── AggregateAttributeModification.java
│ │ ├── DevNullDataStore.java
│ │ ├── DynamoDataStore.java
│ │ ├── DynamoQueryEngine.java
│ │ ├── DynamoUtils.java
│ │ └── IDataStore.java
│ ├── exception/
│ │ ├── ClassNotAnnotatedException.java
│ │ ├── InvalidConfigurationException.java
│ │ ├── SerializationException.java
│ │ └── UnsupportedCalculationException.java
│ ├── factory/
│ │ ├── CSVAggregatorFactory.java
│ │ ├── ExternallyConfiguredAggregatorFactory.java
│ │ ├── JsonAggregatorFactory.java
│ │ ├── ObjectAggregatorFactory.java
│ │ └── RegexAggregatorFactory.java
│ ├── idempotency/
│ │ ├── DefaultIdempotencyCheck.java
│ │ └── IIdempotencyCheck.java
│ ├── metrics/
│ │ ├── CloudWatchMetricsEmitter.java
│ │ ├── IMetricsEmitter.java
│ │ └── MetricsEmitterThrottledException.java
│ ├── processor/
│ │ ├── AggregatorProcessor.java
│ │ └── AggregatorProcessorFactory.java
│ └── summary/
│ ├── SummaryCalculation.java
│ ├── SummaryConfiguration.java
│ └── SummaryElement.java
└── io/
├── AbstractDataExtractor.java
├── CsvDataExtractor.java
├── IDataExtractor.java
├── JsonDataExtractor.java
├── ObjectExtractor.java
├── RegexDataExtractor.java
├── StringDataExtractor.java
└── serializer/
├── CsvSerializer.java
├── IKinesisSerializer.java
├── JavaSerializationSerializer.java
├── JsonSerializer.java
├── RegexSerializer.java
├── SerializationUtils.java
└── StringSerializer.java
SYMBOL INDEX (637 symbols across 73 files)
FILE: sample/java/model/SensorReading.java
class SensorReading (line 21) | public class SensorReading {
type OutputFormat (line 24) | public enum OutputFormat {
method SensorReading (line 41) | private SensorReading() {
method SensorReading (line 44) | public SensorReading(String id, String segment, long captureTs, double...
method getId (line 59) | public String getId() {
method getSegment (line 63) | public String getSegment() {
method getCaptureTs (line 67) | public long getCaptureTs() {
method getLat (line 71) | public double getLat() {
method getLng (line 75) | public double getLng() {
method getPressure (line 79) | public double getPressure() {
method getTemp (line 83) | public double getTemp() {
method getFlowRate (line 87) | public double getFlowRate() {
method getCorrosionIndex (line 91) | public double getCorrosionIndex() {
method getSegmentIncline (line 95) | public double getSegmentIncline() {
method withOutputFormat (line 99) | public SensorReading withOutputFormat(OutputFormat format) {
method asJson (line 104) | public String asJson() throws Exception {
method asString (line 108) | public String asString() throws Exception {
method asCSV (line 115) | public String asCSV() throws Exception {
method toString (line 122) | @Override
FILE: sample/java/model/SensorState.java
class SensorState (line 19) | public class SensorState {
method SensorState (line 29) | public SensorState(String segment, double lat, double lng, double pres...
method getSegment (line 41) | public String getSegment() {
method getLat (line 45) | public double getLat() {
method getLng (line 49) | public double getLng() {
method getPressure (line 53) | public double getPressure() {
method getFlowRate (line 57) | public double getFlowRate() {
method getTemp (line 61) | public double getTemp() {
method getCorrosion (line 65) | public double getCorrosion() {
method getIncline (line 69) | public double getIncline() {
FILE: sample/java/producer/SensorReadingProducer.java
class SensorReadingProducer (line 39) | public class SensorReadingProducer {
method SensorReadingProducer (line 68) | public SensorReadingProducer() {
method getLinePoint (line 71) | public double[] getLinePoint() {
method nextSensorReading (line 85) | public SensorReading nextSensorReading(final OutputFormat format) {
method nextSensorReading (line 89) | public SensorReading nextSensorReading(final OutputFormat format,
method run (line 136) | private void run(final int events, final OutputFormat format,
method main (line 161) | public static void main(String[] args) throws Exception {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/AggregateData.java
class AggregateData (line 22) | public class AggregateData {
method AggregateData (line 31) | public AggregateData(String uniqueId, LabelSet labels, Date date, Map<...
method getUniqueId (line 38) | public String getUniqueId() {
method getLabel (line 42) | public String getLabel() {
method getLabels (line 46) | public LabelSet getLabels() {
method getDate (line 50) | public Date getDate() {
method getSummaries (line 54) | public Map<String, Double> getSummaries() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorGroup.java
class AggregatorGroup (line 31) | public class AggregatorGroup implements IStreamAggregator {
method AggregatorGroup (line 34) | public AggregatorGroup() {
method AggregatorGroup (line 37) | public AggregatorGroup(AggregatorGroup template) throws Exception {
method registerAggregator (line 46) | public void registerAggregator(StreamAggregator agg) {
method getAggregators (line 50) | public List<StreamAggregator> getAggregators() {
method aggregate (line 57) | @Override
method aggregateEvents (line 64) | public void aggregateEvents(List<InputEvent> events) throws Exception {
method checkpoint (line 73) | @Override
method initialize (line 80) | @Override
method shutdown (line 90) | @Override
method getTableName (line 100) | @Override
method getTableNames (line 105) | public List<String> getTableNames() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorType.java
type AggregatorType (line 22) | public enum AggregatorType {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorsConstants.java
class AggregatorsConstants (line 19) | public class AggregatorsConstants {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/EnvironmentType.java
type EnvironmentType (line 19) | public enum EnvironmentType {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/IStreamAggregator.java
type IStreamAggregator (line 32) | public interface IStreamAggregator {
method aggregate (line 40) | public void aggregate(List<Record> records) throws Exception;
method aggregateEvents (line 42) | public void aggregateEvents(List<InputEvent> events) throws Exception;
method checkpoint (line 47) | public void checkpoint() throws Exception;
method initialize (line 53) | public void initialize(String shardId) throws Exception;
method shutdown (line 59) | public void shutdown(boolean flushState) throws Exception;
method getTableName (line 62) | public String getTableName();
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/InputEvent.java
class InputEvent (line 21) | public class InputEvent {
method InputEvent (line 28) | public InputEvent(Record record) {
method withSequence (line 34) | public InputEvent withSequence(String sequence) {
method withPartitionKey (line 39) | public InputEvent withPartitionKey(String partitionKey) {
method getSequenceNumber (line 44) | public String getSequenceNumber() {
method getPartitionKey (line 48) | public String getPartitionKey() {
method getData (line 52) | public byte[] getData() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/InventoryModel.java
class InventoryModel (line 43) | @SuppressWarnings("serial")
type STATE (line 98) | public static enum STATE {
method InventoryModel (line 104) | public InventoryModel(AmazonDynamoDB dynamoClient) throws Exception {
method InventoryModel (line 109) | public InventoryModel(AWSCredentialsProvider credentials) throws Excep...
method init (line 113) | protected void init() throws Exception {
method getKey (line 138) | private Map<String, AttributeValue> getKey(final String streamName,
method removeState (line 149) | public void removeState(final String streamName, final String applicat...
method update (line 171) | public void update(final String streamName, final String applicationName,
method getLastUpdate (line 214) | public InventoryStatus getLastUpdate(final String streamName, final St...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/InventoryStatus.java
class InventoryStatus (line 19) | public class InventoryStatus {
method InventoryStatus (line 22) | public InventoryStatus(String lastTime, String lowSeq, String highSeq) {
method getLastTime (line 29) | public String getLastTime() {
method getLowSeq (line 33) | public String getLowSeq() {
method getHighSeq (line 37) | public String getHighSeq() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/LabelSet.java
class LabelSet (line 29) | public class LabelSet extends LinkedHashMap<String, String> {
method LabelSet (line 34) | public LabelSet() {
method fromIntegerKeys (line 38) | public static LabelSet fromIntegerKeys(List<Integer> keys) {
method fromStringKeys (line 46) | public static LabelSet fromStringKeys(List<String> keys) {
method put (line 54) | @Override
method valuesAsString (line 60) | public String valuesAsString() {
method getName (line 69) | public String getName() {
method withAlias (line 82) | public LabelSet withAlias(String alias) {
method equals (line 88) | @Override
method hashCode (line 128) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/StreamAggregator.java
class StreamAggregator (line 66) | public class StreamAggregator implements IStreamAggregator {
method StreamAggregator (line 181) | public StreamAggregator(StreamAggregator template) throws Exception {
method StreamAggregator (line 200) | public StreamAggregator(String streamName, String applicationName,
method checkpoint (line 210) | public void checkpoint() throws Exception {
method withStorageCapacity (line 235) | public StreamAggregator withStorageCapacity(Long readCapacity,
method logInfo (line 245) | private void logInfo(String message) {
method logWarn (line 249) | private void logWarn(String message) {
method logWarn (line 253) | private void logWarn(String message, Exception e) {
method initialize (line 258) | public void initialize(String shardId) throws Exception {
method validateConfig (line 410) | private void validateConfig() throws Exception {
method withTimeHorizon (line 433) | public StreamAggregator withTimeHorizon(TimeHorizon horizon) {
method withTimeHorizon (line 451) | public StreamAggregator withTimeHorizon(List<TimeHorizon> horizons) {
method withTimeHorizon (line 470) | public StreamAggregator withTimeHorizon(TimeHorizon... horizons) {
method withTableName (line 488) | public StreamAggregator withTableName(String tableName) {
method withAggregatorType (line 502) | public StreamAggregator withAggregatorType(AggregatorType t) {
method withRaiseExceptionOnDataExtractionErrors (line 520) | public StreamAggregator withRaiseExceptionOnDataExtractionErrors(
method withCloudWatchMetrics (line 532) | public StreamAggregator withCloudWatchMetrics() {
method withDataStore (line 543) | public StreamAggregator withDataStore(IDataStore dataStore) {
method withMetricsEmitter (line 556) | public StreamAggregator withMetricsEmitter(IMetricsEmitter metricsEmit...
method withIdempotencyCheck (line 569) | public StreamAggregator withIdempotencyCheck(
method withEnvironment (line 577) | public StreamAggregator withEnvironment(EnvironmentType environment) {
method withEnvironment (line 582) | public StreamAggregator withEnvironment(String environment) {
method getNamespace (line 588) | public String getNamespace() {
method getDataExtractor (line 592) | public IDataExtractor getDataExtractor() {
method getDataStore (line 596) | public IDataStore getDataStore() {
method getTableName (line 600) | public String getTableName() {
method getLabelAttribute (line 604) | public String getLabelAttribute() {
method getDateAttribute (line 608) | public String getDateAttribute() {
method getAggregatorType (line 612) | public AggregatorType getAggregatorType() {
method getReadCapacity (line 616) | public long getReadCapacity() {
method getWriteCapacity (line 620) | public long getWriteCapacity() {
method getTimeHorizon (line 624) | public List<TimeHorizon> getTimeHorizon() {
method shutdown (line 639) | public void shutdown() throws Exception {
method shutdown (line 643) | public void shutdown(boolean flushState) throws Exception {
method shutdown (line 647) | public void shutdown(boolean flushState, InventoryModel.STATE withState)
method aggregate (line 664) | public void aggregate(List<Record> records) throws Exception {
method aggregateEvents (line 677) | public void aggregateEvents(List<InputEvent> events) throws Exception {
method queryValue (line 805) | public List<Map<String, AttributeValue>> queryValue(String label,
method queryByDate (line 835) | public List<Map<String, AttributeValue>> queryByDate(Date dateValue,
method parallelQueryKeys (line 871) | public List<TableKeyStructure> parallelQueryKeys(QueryKeyScope scope,
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/StreamAggregatorUtils.java
class StreamAggregatorUtils (line 50) | public class StreamAggregatorUtils {
method StreamAggregatorUtils (line 64) | private StreamAggregatorUtils() {
method methodToColumn (line 78) | public static String methodToColumn(String methodName) {
method getDynamoHiveWrapper (line 102) | public static String getDynamoHiveWrapper(AmazonDynamoDB dynamoClient,
method getRedshiftCopyCommand (line 142) | public static String getRedshiftCopyCommand(
method getLastWriteSeqIndexName (line 205) | public static final String getLastWriteSeqIndexName(String dynamoTable) {
method getDateDimensionIndexName (line 218) | public static final String getDateDimensionIndexName(String dynamoTable,
method getTableKey (line 231) | public static Map<String, AttributeValue> getTableKey(UpdateKey update...
method getTableKey (line 253) | protected static Map<String, AttributeValue> getTableKey(
method getValue (line 263) | protected static Map<String, AttributeValue> getValue(
method getTableName (line 271) | protected static String getTableName(final String applicationName,
method asJsonNode (line 276) | public static JsonNode asJsonNode(String s) throws Exception {
method asJsonNode (line 280) | public static JsonNode asJsonNode(File f) throws Exception {
method readJsonValue (line 284) | public static JsonNode readJsonValue(JsonNode json, String atPath) {
method readValueAsString (line 299) | public static String readValueAsString(JsonNode json, String atPath) {
method getOpenShards (line 311) | public static Map<String, Shard> getOpenShards(
method getFirstShard (line 382) | public static Shard getFirstShard(AmazonKinesisClient kinesisClient,
method getFirstShardName (line 388) | public static String getFirstShardName(AmazonKinesisClient kinesisClient,
method getShardCount (line 393) | public static int getShardCount(AmazonKinesisClient kinesisClient,
method extractDateFromMultivalue (line 402) | public static String extractDateFromMultivalue(TimeHorizon t, String d...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/TableKeyStructure.java
class TableKeyStructure (line 22) | public class TableKeyStructure {
method TableKeyStructure (line 27) | public TableKeyStructure() {
method TableKeyStructure (line 30) | public TableKeyStructure(String labelAttributeName, String labelAttrib...
method TableKeyStructure (line 37) | public TableKeyStructure(String labelAttributeName, String labelAttrib...
method withDateValue (line 46) | public TableKeyStructure withDateValue(String dateValue) {
method getLabelAttributeName (line 54) | public String getLabelAttributeName() {
method getLabelAttributeValue (line 58) | public String getLabelAttributeValue() {
method getDateAttributeName (line 62) | public String getDateAttributeName() {
method getDateValues (line 66) | public Set<String> getDateValues() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/TimeHorizon.java
type TimeHorizon (line 25) | public enum TimeHorizon {
method getGranularity (line 32) | @Override
method setGranularity (line 37) | @Override
method getValue (line 42) | @Override
method getValue (line 59) | @Override
method TimeHorizon (line 65) | private TimeHorizon(int placemark, String mask, String abbrev) {
method getMask (line 77) | private SimpleDateFormat getMask() {
method getAbbrev (line 81) | public String getAbbrev() {
method getItemWithMultiValueFormat (line 85) | public String getItemWithMultiValueFormat(Date dateValue) {
method getValue (line 89) | public String getValue(Date forDate) {
method getFullHierarchy (line 99) | public List<TimeHorizon> getFullHierarchy() {
method getHierarchyTo (line 112) | public List<TimeHorizon> getHierarchyTo(TimeHorizon t) {
method getGranularity (line 127) | public int getGranularity() throws Exception {
method setGranularity (line 131) | public void setGranularity(int scope) throws Exception {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/AnnotationProcessor.java
class AnnotationProcessor (line 46) | public class AnnotationProcessor {
method AnnotationProcessor (line 80) | private AnnotationProcessor() {
method AnnotationProcessor (line 89) | public AnnotationProcessor(@SuppressWarnings("rawtypes") Class clazz) ...
method getLabelMethodNames (line 182) | public List<String> getLabelMethodNames() {
method getLabelMethods (line 186) | public Map<String, Method> getLabelMethods() {
method getDateMethodName (line 190) | public String getDateMethodName() {
method getDateMethod (line 194) | public Method getDateMethod() {
method getSummaryMethods (line 198) | public Map<String, Method> getSummaryMethods() {
method getSummaryConfig (line 202) | public SummaryConfiguration getSummaryConfig() {
method getType (line 206) | public AggregatorType getType() {
method getTimeHorizon (line 210) | public List<TimeHorizon> getTimeHorizon() {
method hasTimeHierarchy (line 214) | public boolean hasTimeHierarchy() {
method shouldFailOnDataExtractionErrors (line 218) | public boolean shouldFailOnDataExtractionErrors() {
method shouldEmitMetrics (line 222) | public boolean shouldEmitMetrics() {
method getMetricsEmitter (line 226) | public Class<IMetricsEmitter> getMetricsEmitter() {
method getDataStore (line 230) | public Class<IDataStore> getDataStore() {
method getNamespace (line 234) | public String getNamespace() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/AbstractQueryServlet.java
class AbstractQueryServlet (line 26) | public abstract class AbstractQueryServlet extends HttpServlet {
method doGet (line 27) | public void doGet(HttpServletRequest request, HttpServletResponse resp...
method doPost (line 32) | public void doPost(HttpServletRequest request, HttpServletResponse res...
method doAction (line 37) | protected abstract void doAction(HttpServletRequest request, HttpServl...
method doError (line 40) | protected void doError(HttpServletResponse response, String message) t...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/AggregatorsBeanstalkApp.java
class AggregatorsBeanstalkApp (line 30) | public class AggregatorsBeanstalkApp implements ServletContextListener {
method contextDestroyed (line 39) | @Override
method contextInitialized (line 49) | @SuppressWarnings({ "unchecked" })
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/DateQueryServlet.java
class DateQueryServlet (line 36) | public class DateQueryServlet extends AbstractQueryServlet {
method respondWith (line 47) | private void respondWith(HttpServletResponse response,
method doAction (line 89) | public void doAction(HttpServletRequest request, HttpServletResponse r...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/FetchConfigurationServlet.java
class FetchConfigurationServlet (line 31) | public class FetchConfigurationServlet extends AbstractQueryServlet {
method respondWith (line 32) | private void respondWith(HttpServletResponse response, Map<String, Str...
method doAction (line 61) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/ListAggregateKeysServlet.java
class ListAggregateKeysServlet (line 33) | public class ListAggregateKeysServlet extends AbstractQueryServlet {
method respondWith (line 40) | private void respondWith(HttpServletResponse response, List<TableKeySt...
method doAction (line 96) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/QueryByLabelServlet.java
class QueryByLabelServlet (line 36) | public class QueryByLabelServlet extends AbstractQueryServlet {
method respondWith (line 45) | private void respondWith(HttpServletResponse response,
method doAction (line 90) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/ShowConfigFileServlet.java
class ShowConfigFileServlet (line 37) | public class ShowConfigFileServlet extends AbstractQueryServlet {
method doAction (line 40) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/app/ShowConfigurationServlet.java
class ShowConfigurationServlet (line 16) | public class ShowConfigurationServlet extends AbstractQueryServlet {
method doAction (line 18) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/cache/AggregateCache.java
class AggregateCache (line 50) | public class AggregateCache {
method AggregateCache (line 83) | public AggregateCache(String shardId) {
method logInfo (line 87) | private void logInfo(String message) {
method logWarn (line 91) | private void logWarn(String message) {
method initialise (line 100) | public void initialise() throws Exception {
method getReportUpdatesPendingCount (line 129) | protected long getReportUpdatesPendingCount() {
method getWarnUpdatesPendingCount (line 133) | protected long getWarnUpdatesPendingCount() {
method getForceCheckpointOnPendingUpdateCount (line 137) | protected long getForceCheckpointOnPendingUpdateCount() {
method withEnvironment (line 142) | public AggregateCache withEnvironment(EnvironmentType environment) {
method withEnvironment (line 147) | public AggregateCache withEnvironment(String environment) {
method withTableName (line 152) | public AggregateCache withTableName(String tableName) {
method withStreamName (line 157) | public AggregateCache withStreamName(String streamName) {
method withRegion (line 162) | public AggregateCache withRegion(Region region) {
method withLabelColumn (line 167) | public AggregateCache withLabelColumn(String labelColumn) {
method withDateColumn (line 172) | public AggregateCache withDateColumn(String dateColumn) {
method withCredentials (line 177) | public AggregateCache withCredentials(AWSCredentialsProvider credentia...
method withAggregateType (line 182) | public AggregateCache withAggregateType(AggregatorType type) {
method withMetricsEmitter (line 187) | public AggregateCache withMetricsEmitter(IMetricsEmitter metricsEmitte...
method withDataStore (line 192) | public AggregateCache withDataStore(IDataStore dataStore) {
method setCheckpointForcingThresholds (line 198) | protected void setCheckpointForcingThresholds() throws Exception {
method update (line 243) | public synchronized void update(final AggregatorType aggregatorType,
method get (line 329) | public UpdateValue get(UpdateKey key) {
method getDataStore (line 333) | protected IDataStore getDataStore() {
method flush (line 345) | public synchronized void flush() throws Exception {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/cache/UpdateKey.java
class UpdateKey (line 32) | public class UpdateKey {
method UpdateKey (line 43) | public UpdateKey(LabelSet labelValues, String dateAttribute,
method getAggregateColumnName (line 51) | public String getAggregateColumnName() {
method getDateValueColumnName (line 55) | public String getDateValueColumnName() {
method getAggregatedValue (line 59) | public String getAggregatedValue() {
method getDateValue (line 63) | public String getDateValue() {
method getDateValueAsDate (line 67) | public Date getDateValueAsDate() throws ParseException {
method getTimeHorizon (line 79) | public TimeHorizon getTimeHorizon() {
method equals (line 83) | @Override
method hashCode (line 100) | @Override
method toString (line 110) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/cache/UpdateValue.java
class UpdateValue (line 31) | public class UpdateValue {
method UpdateValue (line 45) | public UpdateValue() {
method incrementCount (line 50) | public void incrementCount(int count) {
method updateSummary (line 54) | public void updateSummary(String label, double withValue, SummaryEleme...
method lastWrite (line 72) | public void lastWrite(String lastSeq, long lastTime) {
method getAggregateCount (line 77) | public double getAggregateCount() {
method getSummaryValue (line 81) | public double getSummaryValue(String label) {
method getSummary (line 85) | public AggregateAttributeModification getSummary(String label) {
method getValueByOriginal (line 89) | public AggregateAttributeModification getValueByOriginal(String attrib...
method getSummaryValues (line 95) | public Map<String, AggregateAttributeModification> getSummaryValues() {
method getLastWriteSeq (line 99) | public String getLastWriteSeq() {
method getLastWriteTime (line 103) | public long getLastWriteTime() {
method toString (line 107) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/cli/AggregatorsCli.java
class AggregatorsCli (line 31) | public class AggregatorsCli {
method validateAction (line 40) | private static void validateAction(String actionRequested) throws Exce...
method main (line 47) | public static void main(String[] args) throws Exception {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/configuration/ConfigFileUtils.java
class ConfigFileUtils (line 27) | public class ConfigFileUtils {
method makeConfigFileURL (line 28) | public static final String makeConfigFileURL(String configUrl) throws ...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/configuration/DataExtractor.java
type DataExtractor (line 19) | public enum DataExtractor {
method DataExtractor (line 25) | private DataExtractor(String linkedClass) {
method getLinkedClass (line 31) | public String getLinkedClass() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/configuration/ExternalConfigurationModel.java
class ExternalConfigurationModel (line 47) | public class ExternalConfigurationModel {
method configureCsv (line 94) | private static void configureCsv(JsonNode document, ExternalConfigurat...
method configureStringCommon (line 98) | private static void configureStringCommon(JsonNode document, ExternalC...
method configureRegex (line 103) | private static void configureRegex(JsonNode document, ExternalConfigur...
method configureObject (line 113) | private static void configureObject(JsonNode document, ExternalConfigu...
method addTimeHorizons (line 139) | private static void addTimeHorizons(JsonNode document, ExternalConfigu...
method setAggregatorType (line 177) | private static void setAggregatorType(JsonNode document, ExternalConfi...
method buildFromConfig (line 192) | public static List<ExternalConfigurationModel> buildFromConfig(String ...
method getNamespace (line 360) | public String getNamespace() {
method getTimeHorizons (line 364) | public List<TimeHorizon> getTimeHorizons() {
method getFilterRegex (line 368) | public String getFilterRegex() {
method getRegularExpression (line 372) | public String getRegularExpression() {
method getTableName (line 376) | public String getTableName() {
method getReadIOPs (line 380) | public Long getReadIOPs() {
method getWriteIOPs (line 384) | public Long getWriteIOPs() {
method getLabelAttributeAlias (line 388) | public String getLabelAttributeAlias() {
method getDateAttributeAlias (line 392) | public String getDateAttributeAlias() {
method isAnnotatedClass (line 396) | public boolean isAnnotatedClass() {
method addTimeHorizon (line 400) | public void addTimeHorizon(TimeHorizon timeHorizon) {
method getAggregatorType (line 407) | public AggregatorType getAggregatorType() {
method getLabelItems (line 411) | public List<String> getLabelItems() {
method getDateItem (line 415) | public String getDateItem() {
method getDateAlias (line 419) | public String getDateAlias() {
method getDateFormat (line 423) | public String getDateFormat() {
method getSummaryItems (line 427) | public List<String> getSummaryItems() {
method getDelimiter (line 431) | public String getDelimiter() {
method getItemTerminator (line 435) | public String getItemTerminator() {
method addSummaryItem (line 439) | public void addSummaryItem(String summaryItem) {
method getClazz (line 446) | public Class getClazz() {
method getDataExtractor (line 450) | public DataExtractor getDataExtractor() {
method shouldFailOnDataExtraction (line 454) | public boolean shouldFailOnDataExtraction() {
method shouldEmitMetrics (line 458) | public boolean shouldEmitMetrics() {
method getMetricsEmitter (line 462) | public Class<IMetricsEmitter> getMetricsEmitter() {
method getDataStore (line 466) | public Class getDataStore() {
method setNamespace (line 470) | private void setNamespace(String namespace) {
method setAggregatorType (line 474) | private void setAggregatorType(AggregatorType aggregatorType) {
method addLabelItems (line 478) | private void addLabelItems(String labelItem) {
method setLabelItems (line 482) | private void setLabelItems(List<String> labelItems) {
method setDateItem (line 486) | private void setDateItem(String dateItem) {
method setDateFormat (line 490) | private void setDateFormat(String dateFormat) {
method setDelimiter (line 494) | private void setDelimiter(String delimiter) {
method setItemTerminator (line 499) | private void setItemTerminator(String itemTerminator) {
method setFilterRegex (line 504) | private void setFilterRegex(String filterRegex) {
method setRegularExpression (line 508) | private void setRegularExpression(String regularExpression) {
method setClazz (line 512) | private void setClazz(Class clazz) {
method setDataExtractor (line 516) | private void setDataExtractor(DataExtractor dataExtractor) {
method setAnnotatedClass (line 520) | private void setAnnotatedClass(boolean isAnnotatedClass) {
method setTableName (line 524) | private void setTableName(String tableName) {
method setReadIOPs (line 529) | private void setReadIOPs(Long readIOPs) {
method setWriteIOPs (line 533) | private void setWriteIOPs(Long writeIOPs) {
method setFailOnDataExtraction (line 537) | private void setFailOnDataExtraction(boolean failOnDataExtraction) {
method setEmitMetrics (line 541) | private void setEmitMetrics(boolean emitMetrics) {
method setMetricsEmitter (line 545) | private void setMetricsEmitter(Class<IMetricsEmitter> metricsEmitter) {
method setDataStore (line 549) | private void setDataStore(Class<IDataStore> dataStore) {
method setLabelAttributeAlias (line 553) | private void setLabelAttributeAlias(String labelAttributeAlias) {
method setDateAttributeAlias (line 557) | private void setDateAttributeAlias(String dateAttributeAlias) {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/consumer/AggregatorConsumer.java
class AggregatorConsumer (line 40) | public final class AggregatorConsumer {
method AggregatorConsumer (line 64) | public AggregatorConsumer(String streamName, String appName,
method buildAggregatorsFromConfig (line 71) | private AggregatorGroup buildAggregatorsFromConfig() throws Exception {
method shutdown (line 77) | public void shutdown() throws Exception {
method run (line 82) | public int run() throws Exception {
method assertThat (line 117) | private void assertThat(boolean condition, String message) throws Exce...
method validateConfig (line 123) | private void validateConfig() throws InvalidConfigurationException {
method configure (line 132) | public void configure() throws Exception {
method withKinesisEndpoint (line 194) | public AggregatorConsumer withKinesisEndpoint(String kinesisEndpoint) {
method withToleratedWorkerFailures (line 199) | public AggregatorConsumer withToleratedWorkerFailures(int failuresToTo...
method withMaxRecords (line 204) | public AggregatorConsumer withMaxRecords(int maxRecords) {
method withRegionName (line 209) | public AggregatorConsumer withRegionName(String regionName) {
method withEnvironment (line 214) | public AggregatorConsumer withEnvironment(String environmentName) {
method withCredentialsProvider (line 219) | public AggregatorConsumer withCredentialsProvider(
method withInitialPositionInStream (line 225) | public AggregatorConsumer withInitialPositionInStream(
method withMetricsEmitter (line 231) | public AggregatorConsumer withMetricsEmitter() {
method getAggregators (line 236) | public AggregatorGroup getAggregators() {
method main (line 240) | public static void main(String[] args) throws Exception {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/AggregateAttributeModification.java
class AggregateAttributeModification (line 21) | public class AggregateAttributeModification {
method AggregateAttributeModification (line 30) | private AggregateAttributeModification() {
method AggregateAttributeModification (line 33) | public AggregateAttributeModification(String attributeName, String ori...
method AggregateAttributeModification (line 38) | public AggregateAttributeModification(String attributeName, String ori...
method AggregateAttributeModification (line 44) | public AggregateAttributeModification(String attributeName, String ori...
method getAttributeName (line 56) | public String getAttributeName() {
method getOriginatingValueName (line 60) | public String getOriginatingValueName() {
method getOldValue (line 64) | public Double getOldValue() {
method getNewValue (line 68) | public Double getNewValue() {
method getFinalValue (line 72) | public Double getFinalValue() {
method getCalculationApplied (line 76) | public SummaryCalculation getCalculationApplied() {
method getWritesSoFar (line 80) | public int getWritesSoFar() {
method toString (line 84) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/DevNullDataStore.java
class DevNullDataStore (line 28) | public class DevNullDataStore implements IDataStore {
method write (line 30) | @Override
method initialise (line 56) | @Override
method refreshForceCheckpointThresholds (line 60) | @Override
method setRegion (line 65) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/DynamoDataStore.java
class DynamoDataStore (line 62) | public class DynamoDataStore implements IDataStore {
type DynamoSummaryUpdateMethod (line 63) | public enum DynamoSummaryUpdateMethod {
method DynamoSummaryUpdateMethod (line 67) | private DynamoSummaryUpdateMethod(AttributeAction a) {
method getAction (line 71) | public AttributeAction getAction() {
method DynamoDataStore (line 118) | public DynamoDataStore(AmazonDynamoDB dynamoClient, AmazonKinesisClien...
method DynamoDataStore (line 130) | public DynamoDataStore(AWSCredentialsProvider credentials, AggregatorT...
method initialise (line 136) | @Override
method write (line 155) | @Override
method updateConditionalValue (line 327) | public UpdateItemResult updateConditionalValue(final AmazonDynamoDB dy...
method getTableStructure (line 403) | public String getTableStructure() throws Exception {
method getDictionaryEntry (line 423) | protected List<String> getDictionaryEntry() throws Exception {
method initAggTable (line 448) | public void initAggTable(final String keyColumn, final String dateColu...
method refreshForceCheckpointThresholds (line 495) | public long refreshForceCheckpointThresholds() {
method getProvisionedWrites (line 518) | private long getProvisionedWrites() {
method queryEngine (line 522) | public DynamoQueryEngine queryEngine() {
method getRegion (line 526) | public Region getRegion() {
method setRegion (line 530) | @Override
method withStorageCapacity (line 535) | public DynamoDataStore withStorageCapacity(long readCapacity, long wri...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/DynamoQueryEngine.java
class DynamoQueryEngine (line 52) | public class DynamoQueryEngine {
method DynamoQueryEngine (line 61) | public DynamoQueryEngine(AmazonDynamoDB dynamoClient, String tableName,
type QueryKeyScope (line 69) | public enum QueryKeyScope {
method parallelQueryKeys (line 73) | public List<TableKeyStructure> parallelQueryKeys(QueryKeyScope scope,
method queryByKey (line 115) | public List<Map<String, AttributeValue>> queryByKey(String label,
class ParallelKeyScanWorker (line 183) | private class ParallelKeyScanWorker implements Runnable {
method ParallelKeyScanWorker (line 196) | public ParallelKeyScanWorker(String tableName, int workerInstance,
method getResultCount (line 207) | public int getResultCount() {
method getResultsProcessed (line 215) | public int getResultsProcessed() {
method throwExceptions (line 219) | public void throwExceptions() throws Exception {
method run (line 225) | @Override
method getOutput (line 383) | public List<TableKeyStructure> getOutput() {
class ParallelDateQueryWorker (line 388) | private class ParallelDateQueryWorker implements Runnable {
method throwException (line 399) | public void throwException() throws Exception {
method ParallelDateQueryWorker (line 404) | public ParallelDateQueryWorker(String tableName, String indexName,
method run (line 416) | @Override
method getResultKeys (line 505) | public Map<String, Set<String>> getResultKeys() {
method convertResultKeys (line 510) | private KeysAndAttributes convertResultKeys(
method batchGetDataByKeys (line 528) | private List<Map<String, AttributeValue>> batchGetDataByKeys(
method parallelQueryDate (line 546) | @SuppressWarnings("unchecked")
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/DynamoUtils.java
class DynamoUtils (line 57) | public class DynamoUtils {
method DynamoUtils (line 60) | private DynamoUtils() {
method safeCreateTable (line 72) | public static CreateTableResult safeCreateTable(
method initTable (line 120) | public static void initTable(final AmazonDynamoDB dynamoClient,
method safeDescribeTable (line 172) | public static DescribeTableResult safeDescribeTable(
method waitForTableActive (line 206) | public static void waitForTableActive(final AmazonDynamoDB dynamoClient,
method waitForTableState (line 223) | private static void waitForTableState(final AmazonDynamoDB dynamoClient,
method dropTable (line 240) | public static void dropTable(final AmazonDynamoDB dynamoClient,
method cleanupAggTable (line 254) | public static void cleanupAggTable(AWSCredentialsProvider credentials,
method updateWithRetries (line 320) | public static UpdateItemResult updateWithRetries(
method getDynamoTableStructure (line 376) | public static String getDynamoTableStructure(AmazonDynamoDB dynamoClient,
method getDictionaryEntry (line 399) | public static List<String> getDictionaryEntry(
method queryUntilDone (line 423) | public static List<Map<String, AttributeValue>> queryUntilDone(
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/IDataStore.java
type IDataStore (line 29) | public interface IDataStore {
method write (line 39) | public Map<UpdateKey, Map<String, AggregateAttributeModification>> write(
method initialise (line 47) | public void initialise() throws Exception;
method refreshForceCheckpointThresholds (line 56) | public long refreshForceCheckpointThresholds() throws Exception;
method setRegion (line 63) | public void setRegion(Region region);
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/exception/ClassNotAnnotatedException.java
class ClassNotAnnotatedException (line 19) | @SuppressWarnings("serial")
method ClassNotAnnotatedException (line 23) | public ClassNotAnnotatedException(Exception e) {
method ClassNotAnnotatedException (line 27) | public ClassNotAnnotatedException(String message, Exception e) {
method ClassNotAnnotatedException (line 31) | public ClassNotAnnotatedException(String message) {
method getMessage (line 35) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/exception/InvalidConfigurationException.java
class InvalidConfigurationException (line 19) | public class InvalidConfigurationException extends Exception {
method InvalidConfigurationException (line 21) | public InvalidConfigurationException() {
method InvalidConfigurationException (line 24) | public InvalidConfigurationException(String message) {
method InvalidConfigurationException (line 29) | public InvalidConfigurationException(Throwable cause) {
method InvalidConfigurationException (line 33) | public InvalidConfigurationException(String message, Throwable cause) {
method InvalidConfigurationException (line 37) | public InvalidConfigurationException(String message, Throwable cause,
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/exception/SerializationException.java
class SerializationException (line 19) | @SuppressWarnings("serial")
method SerializationException (line 23) | public SerializationException(String message) {
method SerializationException (line 27) | public SerializationException(Exception e) {
method SerializationException (line 31) | public SerializationException(String message, Exception e) {
method getMessage (line 35) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/exception/UnsupportedCalculationException.java
class UnsupportedCalculationException (line 25) | public class UnsupportedCalculationException extends Exception {
method UnsupportedCalculationException (line 28) | public UnsupportedCalculationException(String message) {
method getMessage (line 33) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/factory/CSVAggregatorFactory.java
class CSVAggregatorFactory (line 34) | public class CSVAggregatorFactory {
method CSVAggregatorFactory (line 35) | private CSVAggregatorFactory() {
method newInstance (line 61) | public static final StreamAggregator newInstance(String streamName, St...
method newInstance (line 72) | public static final StreamAggregator newInstance(String streamName, St...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/factory/ExternallyConfiguredAggregatorFactory.java
class ExternallyConfiguredAggregatorFactory (line 34) | public class ExternallyConfiguredAggregatorFactory {
method ExternallyConfiguredAggregatorFactory (line 35) | private ExternallyConfiguredAggregatorFactory() {
method intList (line 38) | private static List<Integer> intList(List<String> stringList) {
method buildFromConfig (line 48) | public static AggregatorGroup buildFromConfig(String streamName, Strin...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/factory/JsonAggregatorFactory.java
class JsonAggregatorFactory (line 29) | public class JsonAggregatorFactory {
method JsonAggregatorFactory (line 30) | private JsonAggregatorFactory() {
method newInstance (line 61) | public static final StreamAggregator newInstance(String streamName, St...
method newInstance (line 100) | public static final StreamAggregator newInstance(String streamName, St...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/factory/ObjectAggregatorFactory.java
class ObjectAggregatorFactory (line 33) | public class ObjectAggregatorFactory {
method ObjectAggregatorFactory (line 34) | private ObjectAggregatorFactory() {
method newInstance (line 51) | public static final StreamAggregator newInstance(String streamName, St...
method newInstance (line 114) | public static final StreamAggregator newInstance(String streamName, St...
method newInstance (line 152) | public static final StreamAggregator newInstance(String streamName, St...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/factory/RegexAggregatorFactory.java
class RegexAggregatorFactory (line 33) | public class RegexAggregatorFactory {
method RegexAggregatorFactory (line 34) | private RegexAggregatorFactory() {
method newInstance (line 60) | public static final StreamAggregator newInstance(String streamName, St...
method newInstance (line 97) | public static final StreamAggregator newInstance(String streamName, St...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/idempotency/DefaultIdempotencyCheck.java
class DefaultIdempotencyCheck (line 25) | public class DefaultIdempotencyCheck implements IIdempotencyCheck {
method DefaultIdempotencyCheck (line 26) | public DefaultIdempotencyCheck() {
method doProcess (line 29) | public boolean doProcess(String partitionKey, String sequenceNumber, A...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/idempotency/IIdempotencyCheck.java
type IIdempotencyCheck (line 25) | public interface IIdempotencyCheck {
method doProcess (line 32) | public boolean doProcess(String partitionKey, String sequenceNumber, A...
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/metrics/CloudWatchMetricsEmitter.java
class CloudWatchMetricsEmitter (line 41) | public class CloudWatchMetricsEmitter implements IMetricsEmitter {
method CloudWatchMetricsEmitter (line 56) | public CloudWatchMetricsEmitter() {
method CloudWatchMetricsEmitter (line 59) | public CloudWatchMetricsEmitter(String metricsNamespace,
method emit (line 65) | @Override
method setRegion (line 153) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/metrics/IMetricsEmitter.java
type IMetricsEmitter (line 30) | public interface IMetricsEmitter {
method emit (line 37) | public void emit(Map<UpdateKey, Map<String, AggregateAttributeModifica...
method setRegion (line 45) | public void setRegion(Region region);
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/metrics/MetricsEmitterThrottledException.java
class MetricsEmitterThrottledException (line 19) | public class MetricsEmitterThrottledException extends Exception {
method MetricsEmitterThrottledException (line 20) | public MetricsEmitterThrottledException() {
method MetricsEmitterThrottledException (line 24) | public MetricsEmitterThrottledException(String message) {
method MetricsEmitterThrottledException (line 28) | public MetricsEmitterThrottledException(Exception e) {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/processor/AggregatorProcessor.java
class AggregatorProcessor (line 39) | public class AggregatorProcessor implements IRecordProcessor {
method AggregatorProcessor (line 50) | public AggregatorProcessor(IStreamAggregator agg) {
method initialize (line 58) | @Override
method processRecords (line 72) | @Override
method shutdown (line 97) | @Override
method checkpoint (line 126) | private void checkpoint(IRecordProcessorCheckpointer checkpointer) {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/processor/AggregatorProcessorFactory.java
class AggregatorProcessorFactory (line 31) | public class AggregatorProcessorFactory implements IRecordProcessorFacto...
method AggregatorProcessorFactory (line 36) | private AggregatorProcessorFactory() {
method AggregatorProcessorFactory (line 45) | public AggregatorProcessorFactory(StreamAggregator agg) {
method AggregatorProcessorFactory (line 50) | public AggregatorProcessorFactory(AggregatorGroup group) {
method createProcessor (line 57) | public IRecordProcessor createProcessor() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/summary/SummaryCalculation.java
type SummaryCalculation (line 22) | public enum SummaryCalculation {
method apply (line 28) | @Override
method apply (line 40) | @Override
method apply (line 53) | @Override
method apply (line 69) | @Override
method apply (line 92) | @Override
method SummaryCalculation (line 110) | private SummaryCalculation(ComparisonOperator c, DynamoSummaryUpdateMe...
method SummaryCalculation (line 115) | private SummaryCalculation() {
method apply (line 127) | public abstract Double apply(Double currentValue, Double newValue);
method getDynamoComparisonOperator (line 136) | public ComparisonOperator getDynamoComparisonOperator() {
method getSummaryUpdateMethod (line 140) | public DynamoSummaryUpdateMethod getSummaryUpdateMethod() {
method nvl (line 144) | private static double nvl(Double val) {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/summary/SummaryConfiguration.java
class SummaryConfiguration (line 38) | public class SummaryConfiguration {
class ConfigWriter (line 42) | final class ConfigWriter {
method write (line 43) | public void write(String s, SummaryElement e) {
method SummaryConfiguration (line 59) | public SummaryConfiguration() {
method SummaryConfiguration (line 62) | public SummaryConfiguration(List<String> summaries) throws Unsupported...
method add (line 75) | public void add(String value, SummaryElement e) {
method addConfig (line 79) | private void addConfig(String summary) throws UnsupportedCalculationEx...
method withConfigItem (line 91) | public SummaryConfiguration withConfigItem(String summary)
method getRequestedCalculations (line 105) | public List<SummaryElement> getRequestedCalculations(String s) {
method getItemSet (line 114) | public Set<String> getItemSet() {
FILE: src/main/java/com/amazonaws/services/kinesis/aggregators/summary/SummaryElement.java
class SummaryElement (line 22) | public class SummaryElement {
method SummaryElement (line 27) | public SummaryElement(String streamDataElement, SummaryCalculation cal...
method SummaryElement (line 32) | public SummaryElement(String streamDataElement, SummaryCalculation cal...
method SummaryElement (line 54) | public SummaryElement(String s) throws UnsupportedCalculationException {
method makeStoreAttributeName (line 85) | public static String makeStoreAttributeName(String attribute, SummaryC...
method getStreamDataElement (line 90) | public String getStreamDataElement() {
method getAttributeAlias (line 94) | public String getAttributeAlias() {
method getCalculation (line 98) | public SummaryCalculation getCalculation() {
FILE: src/main/java/com/amazonaws/services/kinesis/io/AbstractDataExtractor.java
class AbstractDataExtractor (line 31) | public abstract class AbstractDataExtractor implements IDataExtractor {
method getAggregateLabelName (line 36) | public abstract String getAggregateLabelName();
method getDateValueName (line 38) | public abstract String getDateValueName();
method getData (line 40) | public abstract List<AggregateData> getData(InputEvent event) throws S...
method validate (line 42) | public abstract void validate() throws Exception;
method getAggregatorType (line 44) | public AggregatorType getAggregatorType() {
method setAggregatorType (line 48) | public void setAggregatorType(AggregatorType type) {
method getSummaryConfig (line 52) | public SummaryConfiguration getSummaryConfig() {
FILE: src/main/java/com/amazonaws/services/kinesis/io/CsvDataExtractor.java
class CsvDataExtractor (line 22) | public class CsvDataExtractor extends StringDataExtractor<CsvDataExtract...
method CsvDataExtractor (line 39) | public CsvDataExtractor(List<Integer> labelIndicies) {
method CsvDataExtractor (line 46) | public CsvDataExtractor(List<Integer> labelIndicies, String labelAttri...
method withRegexFilter (line 66) | public CsvDataExtractor withRegexFilter(String filterRegex) {
method withDelimiter (line 80) | public CsvDataExtractor withDelimiter(String delimiter) {
method withItemTerminator (line 94) | public CsvDataExtractor withItemTerminator(String lineTerminator) {
method withSerialiser (line 108) | public CsvDataExtractor withSerialiser(CsvSerializer serialiser) {
method withDateValueIndex (line 122) | public CsvDataExtractor withDateValueIndex(Integer dateValueIndex) {
method copy (line 129) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/io/IDataExtractor.java
type IDataExtractor (line 34) | public interface IDataExtractor {
method getUniqueIdName (line 39) | public String getUniqueIdName();
method getAggregateLabelName (line 46) | public String getAggregateLabelName();
method getDateValueName (line 53) | public String getDateValueName();
method getData (line 63) | public List<AggregateData> getData(InputEvent event) throws Serializat...
method setAggregatorType (line 72) | public void setAggregatorType(AggregatorType type);
method validate (line 79) | public void validate() throws Exception;
method getSummaryConfig (line 87) | public SummaryConfiguration getSummaryConfig();
method copy (line 89) | public IDataExtractor copy() throws Exception;
FILE: src/main/java/com/amazonaws/services/kinesis/io/JsonDataExtractor.java
class JsonDataExtractor (line 42) | public class JsonDataExtractor extends AbstractDataExtractor implements
method JsonDataExtractor (line 59) | private JsonDataExtractor() {
method JsonDataExtractor (line 62) | public JsonDataExtractor(List<String> labelAttributes) {
method JsonDataExtractor (line 67) | public JsonDataExtractor(List<String> labelAttributes,
method getData (line 76) | @Override
method withUniqueIdAttribute (line 191) | public JsonDataExtractor withUniqueIdAttribute(String uniqueIdAttribut...
method withDateValueAttribute (line 205) | public JsonDataExtractor withDateValueAttribute(String dateValueAttrib...
method withDateFormat (line 220) | public JsonDataExtractor withDateFormat(String dateFormat) {
method withSerialiser (line 228) | public JsonDataExtractor withSerialiser(JsonSerializer serialiser) {
method withSummaryAttributes (line 243) | public JsonDataExtractor withSummaryAttributes(
method withRegexFilter (line 264) | public JsonDataExtractor withRegexFilter(String filterRegex) {
method withItemTerminator (line 278) | public JsonDataExtractor withItemTerminator(String lineTerminator) {
method getAggregateLabelName (line 288) | @Override
method getDateValueName (line 296) | @Override
method getUniqueIdName (line 305) | @Override
method validate (line 313) | @Override
method copy (line 327) | public IDataExtractor copy() throws Exception {
FILE: src/main/java/com/amazonaws/services/kinesis/io/ObjectExtractor.java
class ObjectExtractor (line 49) | public class ObjectExtractor extends AbstractDataExtractor implements ID...
method ObjectExtractor (line 81) | private ObjectExtractor() {
method ObjectExtractor (line 84) | public ObjectExtractor(Class clazz) throws Exception {
method ObjectExtractor (line 112) | public ObjectExtractor(List<String> aggregateLabelMethods, Class clazz...
method ObjectExtractor (line 128) | public ObjectExtractor(List<String> aggregateLabelMethodNames, Class c...
method validate (line 159) | @Override
method getData (line 191) | @Override
method withDateMethod (line 289) | public ObjectExtractor withDateMethod(String dateMethodName) throws No...
method withUniqueIdMethod (line 297) | public ObjectExtractor withUniqueIdMethod(String uniqueIdMethodName)
method withSummaryMethods (line 326) | public ObjectExtractor withSummaryMethods(List<String> summaryMethodName)
method withSummaryConfig (line 346) | public ObjectExtractor withSummaryConfig(SummaryConfiguration config) {
method getClazz (line 364) | @SuppressWarnings("rawtypes")
method getAggregateLabelName (line 372) | @Override
method getDateValueName (line 380) | @Override
method copy (line 386) | public IDataExtractor copy() throws Exception {
method getUniqueIdName (line 393) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/io/RegexDataExtractor.java
class RegexDataExtractor (line 29) | public class RegexDataExtractor extends StringDataExtractor<RegexDataExt...
method RegexDataExtractor (line 44) | public RegexDataExtractor(String regex, List<Integer> labelIndicies) {
method RegexDataExtractor (line 48) | public RegexDataExtractor(String regex, List<Integer> labelIndicies, i...
method RegexDataExtractor (line 52) | public RegexDataExtractor(String regex, List<Integer> labelIndicies,
method withItemTerminator (line 75) | public RegexDataExtractor withItemTerminator(String lineTerminator) {
method withSerialiser (line 89) | public RegexDataExtractor withSerialiser(CsvSerializer serialiser) {
method withDateValueIndex (line 102) | public RegexDataExtractor withDateValueIndex(Integer dateValueIndex) {
method copy (line 109) | @Override
FILE: src/main/java/com/amazonaws/services/kinesis/io/StringDataExtractor.java
class StringDataExtractor (line 46) | public class StringDataExtractor<T extends StringDataExtractor<T>> exten...
method StringDataExtractor (line 76) | protected StringDataExtractor() {
method validate (line 82) | @Override
method getData (line 105) | @Override
method withDateFormat (line 193) | @SuppressWarnings("unchecked")
method withSummaryIndicies (line 216) | @SuppressWarnings("unchecked")
method withStringSummaryIndicies (line 238) | @SuppressWarnings("unchecked")
method withIntegerSummaryIndicies (line 250) | @SuppressWarnings("unchecked")
method withSummaryIndex (line 262) | @SuppressWarnings("unchecked")
method withSummaryIndex (line 276) | @SuppressWarnings("unchecked")
method withLabelAttributeAlias (line 293) | public T withLabelAttributeAlias(String alias) {
method withUniqueIdIndex (line 299) | public T withUniqueIdIndex(String index) {
method withDateAttributeAlias (line 316) | public T withDateAttributeAlias(String alias) {
method getAggregateLabelName (line 325) | public String getAggregateLabelName() {
method getUniqueIdName (line 332) | @Override
method getDateValueName (line 340) | public String getDateValueName() {
method getOriginalSummaryExpressions (line 344) | public List<Object> getOriginalSummaryExpressions() {
method copy (line 348) | public IDataExtractor copy() throws Exception {
FILE: src/main/java/com/amazonaws/services/kinesis/io/serializer/CsvSerializer.java
class CsvSerializer (line 27) | public class CsvSerializer extends StringSerializer<CsvSerializer> imple...
method toClass (line 43) | public List<List<String>> toClass(InputEvent event) throws IOException {
method fromClass (line 70) | public byte[] fromClass(List<List<String>> csv) throws IOException {
method withFieldDelimiter (line 91) | public CsvSerializer withFieldDelimiter(String delimiter) {
method withFilterRegex (line 103) | public CsvSerializer withFilterRegex(String regex) {
FILE: src/main/java/com/amazonaws/services/kinesis/io/serializer/IKinesisSerializer.java
type IKinesisSerializer (line 30) | public interface IKinesisSerializer<T, U> {
method toClass (line 38) | public T toClass(InputEvent event) throws IOException;
method fromClass (line 46) | public U fromClass(T record) throws IOException;
FILE: src/main/java/com/amazonaws/services/kinesis/io/serializer/JavaSerializationSerializer.java
class JavaSerializationSerializer (line 30) | public class JavaSerializationSerializer implements IKinesisSerializer<O...
method toClass (line 34) | public Object toClass(InputEvent event) throws IOException {
method fromClass (line 57) | public byte[] fromClass(Object o) throws IOException {
FILE: src/main/java/com/amazonaws/services/kinesis/io/serializer/JsonSerializer.java
class JsonSerializer (line 29) | @SuppressWarnings("rawtypes")
method JsonSerializer (line 49) | public JsonSerializer() {
method JsonSerializer (line 58) | public JsonSerializer(String itemTerminator) {
method JsonSerializer (line 67) | public JsonSerializer(Class clazz) {
method toClass (line 72) | @SuppressWarnings("unchecked")
method fromClass (line 114) | public byte[] fromClass(final Object o) throws IOException {
method withFilterRegex (line 129) | public JsonSerializer withFilterRegex(String regex) {
method withCharset (line 143) | public JsonSerializer withCharset(String charset) {
method withItemTerminator (line 159) | public JsonSerializer withItemTerminator(String itemTerminator) {
FILE: src/main/java/com/amazonaws/services/kinesis/io/serializer/RegexSerializer.java
class RegexSerializer (line 27) | public class RegexSerializer extends StringSerializer<RegexSerializer> i...
method RegexSerializer (line 35) | public RegexSerializer(String regexPattern) {
method toClass (line 40) | public List<List<String>> toClass(InputEvent event) throws IOException {
method fromClass (line 69) | public byte[] fromClass(List<List<String>> content) throws IOException {
FILE: src/main/java/com/amazonaws/services/kinesis/io/serializer/SerializationUtils.java
class SerializationUtils (line 26) | public class SerializationUtils {
method safeReturnData (line 38) | public static byte[] safeReturnData(byte[] check) throws IOException {
FILE: src/main/java/com/amazonaws/services/kinesis/io/serializer/StringSerializer.java
class StringSerializer (line 23) | public abstract class StringSerializer<T extends StringSerializer<T>> {
method withCharset (line 35) | @SuppressWarnings("unchecked")
method withItemTerminator (line 52) | @SuppressWarnings("unchecked")
method getItems (line 58) | protected String[] getItems(InputEvent event) throws Exception {
Condensed preview — 100 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (490K chars).
[
{
"path": ".gitignore",
"chars": 46,
"preview": "/bin/\n/target/\n.classpath\n.project\n.settings/\n"
},
{
"path": "LICENSE.txt",
"chars": 9520,
"preview": "http://www.apache.org/licenses/LICENSE-2.0.html\n\nApache License\n\nVersion 2.0, January 2004\n\nhttp://www.apache.org/licens"
},
{
"path": "NOTICE.txt",
"chars": 572,
"preview": "amazon-kinesis-aggregators\n\nCopyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.\n\nLicensed under the "
},
{
"path": "README.md",
"chars": 37464,
"preview": "# Amazon Kinesis Aggregators\n\n----\n\n*This project is now deprecated, and only updates for security vulnerabilities in de"
},
{
"path": "assembly.xml",
"chars": 575,
"preview": "<assembly\n\txmlns=\"http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0\"\n\txmlns:xsi=\"http://www.w3.org/20"
},
{
"path": "pom.xml",
"chars": 6346,
"preview": "\n<project xmlns=\"http://maven.apache.org/POM/4.0.0\"\n\txmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n\txsi:schemaLo"
},
{
"path": "sample/bin/run-producer.sh",
"chars": 996,
"preview": "#!/bin/bash\n#\n# Amazon Kinesis Aggregators\n#\n# Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n"
},
{
"path": "sample/java/model/SensorReading.java",
"chars": 3241,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "sample/java/model/SensorState.java",
"chars": 1590,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "sample/java/producer/SensorReadingProducer.java",
"chars": 5686,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "sample/resources/BySegment-CSV.json",
"chars": 517,
"preview": "[{\"dataExtractor\": \"CSV\",\n \"dateFormat\": \"\",\n \"dateItem\": 2,\n \"dateAttributeAlias\":\"sensorTS\",\n \"labelItems\""
},
{
"path": "sample/resources/BySegment-Json.json",
"chars": 441,
"preview": "[{\"dataExtractor\": \"JSON\",\n \"dateFormat\": \"\",\n \"dateAttribute\": \"captureTs\",\n \"labelItems\": [\n \"segment\"\n "
},
{
"path": "sample/resources/BySegment-Regex.json",
"chars": 670,
"preview": "[{\"dataExtractor\": \"REGEX\",\n \"dateFormat\": \"\",\n \"dateItem\": 2,\n \"dateAttributeAlias\":\"sensorTS\",\n \"labelItem"
},
{
"path": "src/.gitkeep",
"chars": 86,
"preview": "Feel free to delete this file as soon as actual Java code is added to this\ndirectory.\n"
},
{
"path": "src/log4j.properties",
"chars": 331,
"preview": "# Root logger option\nlog4j.rootLogger=INFO, stdout\n \n# Direct log messages to stdout\nlog4j.appender.stdout=org.apache.lo"
},
{
"path": "src/main/WebContent/.ebextensions/as.config",
"chars": 353,
"preview": "option_settings:\n - namespace: aws:autoscaling:asg\n option_name: MinSize\n value: 2\n - namespace: aws:autoscaling"
},
{
"path": "src/main/WebContent/META-INF/MANIFEST.MF",
"chars": 39,
"preview": "Manifest-Version: 1.0\r\nClass-Path: \r\n\r\n"
},
{
"path": "src/main/WebContent/WEB-INF/web.xml",
"chars": 2326,
"preview": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<web-app xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n\txmlns=\"http://jav"
},
{
"path": "src/main/WebContent/index.html",
"chars": 811,
"preview": "<!--\n\n Amazon Kinesis Aggregators\n\n Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n\n "
},
{
"path": "src/main/WebContent/styles/styles.css",
"chars": 2016,
"preview": "/*************************************\nGENERAL\n*************************************/\nbody {\n\tmargin: 0;\n\tpadding: 0;\n\tf"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/AggregateData.java",
"chars": 1469,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorGroup.java",
"chars": 3193,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorType.java",
"chars": 1228,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/AggregatorsConstants.java",
"chars": 1388,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/EnvironmentType.java",
"chars": 723,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/IStreamAggregator.java",
"chars": 2133,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/InputEvent.java",
"chars": 1503,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/InventoryModel.java",
"chars": 9646,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/InventoryStatus.java",
"chars": 1127,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/LabelSet.java",
"chars": 3889,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/StreamAggregator.java",
"chars": 26752,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/StreamAggregatorUtils.java",
"chars": 13155,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/TableKeyStructure.java",
"chars": 2184,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/TimeHorizon.java",
"chars": 4084,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/Aggregate.java",
"chars": 2463,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/AnnotationProcessor.java",
"chars": 8172,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/DateValue.java",
"chars": 1036,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/Label.java",
"chars": 1036,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/annotations/Summary.java",
"chars": 1429,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/app/AbstractQueryServlet.java",
"chars": 1720,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/app/AggregatorsBeanstalkApp.java",
"chars": 6559,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/app/DateQueryServlet.java",
"chars": 5889,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/app/FetchConfigurationServlet.java",
"chars": 3537,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/app/ListAggregateKeysServlet.java",
"chars": 5464,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/app/QueryByLabelServlet.java",
"chars": 6330,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/app/ShowConfigFileServlet.java",
"chars": 2258,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/app/ShowConfigurationServlet.java",
"chars": 2018,
"preview": "package com.amazonaws.services.kinesis.aggregators.app;\n\nimport java.io.IOException;\nimport java.io.PrintWriter;\nimport "
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/cache/AggregateCache.java",
"chars": 11733,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/cache/UpdateKey.java",
"chars": 3128,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/cache/UpdateValue.java",
"chars": 4378,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/cli/AggregatorsCli.java",
"chars": 4793,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/configuration/ConfigFileUtils.java",
"chars": 2139,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/configuration/DataExtractor.java",
"chars": 1195,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/configuration/ExternalConfigurationModel.java",
"chars": 20537,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/configuration/json.schema",
"chars": 3488,
"preview": "{\n \"title\": \"Amazon Kinesis Aggregators Configuration Schema\",\n \"type\": \"object\",\n \"properties\": {\n \"nam"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/consumer/AggregatorConsumer.java",
"chars": 8948,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/AggregateAttributeModification.java",
"chars": 3109,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/DevNullDataStore.java",
"chars": 2367,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/DynamoDataStore.java",
"chars": 24659,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/DynamoQueryEngine.java",
"chars": 19590,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/DynamoUtils.java",
"chars": 15145,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/datastore/IDataStore.java",
"chars": 2094,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/exception/ClassNotAnnotatedException.java",
"chars": 1138,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/exception/InvalidConfigurationException.java",
"chars": 1347,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/exception/SerializationException.java",
"chars": 1122,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/exception/UnsupportedCalculationException.java",
"chars": 1205,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/factory/CSVAggregatorFactory.java",
"chars": 4701,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/factory/ExternallyConfiguredAggregatorFactory.java",
"chars": 7122,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/factory/JsonAggregatorFactory.java",
"chars": 5917,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/factory/ObjectAggregatorFactory.java",
"chars": 8339,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/factory/RegexAggregatorFactory.java",
"chars": 4930,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/idempotency/DefaultIdempotencyCheck.java",
"chars": 1141,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/idempotency/IIdempotencyCheck.java",
"chars": 1273,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/metrics/CloudWatchMetricsEmitter.java",
"chars": 5098,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/metrics/IMetricsEmitter.java",
"chars": 1547,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/metrics/MetricsEmitterThrottledException.java",
"chars": 983,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/processor/AggregatorProcessor.java",
"chars": 4855,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/processor/AggregatorProcessorFactory.java",
"chars": 2327,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/summary/SummaryCalculation.java",
"chars": 4990,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/summary/SummaryConfiguration.java",
"chars": 3792,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/aggregators/summary/SummaryElement.java",
"chars": 3874,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/AbstractDataExtractor.java",
"chars": 1895,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/CsvDataExtractor.java",
"chars": 4940,
"preview": "/**\n * Amazon Kinesis Aggregators Copyright 2014, Amazon.com, Inc. or its\n * affiliates. All Rights Reserved. Licensed u"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/IDataExtractor.java",
"chars": 2927,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/JsonDataExtractor.java",
"chars": 10289,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/ObjectExtractor.java",
"chars": 14431,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/RegexDataExtractor.java",
"chars": 4159,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/StringDataExtractor.java",
"chars": 12354,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/serializer/CsvSerializer.java",
"chars": 3308,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/serializer/IKinesisSerializer.java",
"chars": 1604,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/serializer/JavaSerializationSerializer.java",
"chars": 2530,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/serializer/JsonSerializer.java",
"chars": 5066,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/serializer/RegexSerializer.java",
"chars": 2325,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/serializer/SerializationUtils.java",
"chars": 1403,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
},
{
"path": "src/main/java/com/amazonaws/services/kinesis/io/serializer/StringSerializer.java",
"chars": 1982,
"preview": "/**\n * Amazon Kinesis Aggregators\n *\n * Copyright 2014, Amazon.com, Inc. or its affiliates. All Rights Reserved.\n *\n * L"
}
]
// ... and 5 more files (download for full content)
About this extraction
This page contains the full source code of the awslabs/amazon-kinesis-aggregators GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 100 files (29.7 MB), approximately 101.1k tokens, and a symbol index with 637 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.