Repository: aws-samples/serverless-data-analytics
Branch: master
Commit: 7797580a55b2
Files: 10
Total size: 112.2 KB
Directory structure:
gitextract_vqvmwjvn/
├── .github/
│ └── PULL_REQUEST_TEMPLATE.md
├── LICENSE
├── Lab1/
│ └── README.md
├── Lab2/
│ ├── README.md
│ └── img/
│ └── README.md
├── Lab3/
│ └── README.md
├── Lab4/
│ ├── README.md
│ └── redshiftspectrumglue-lab4.template
├── NOTICE
└── README.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/PULL_REQUEST_TEMPLATE.md
================================================
*Issue #, if available:*
*Description of changes:*
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: Lab1/README.md
================================================
# Lab 1: Serverless Analysis of data in Amazon S3 using Amazon Athena
* [Creating Amazon Athena Database and Table](#creating-amazon-athena-database-and-table)
* [Create Athena Database](#create-database)
* [Create Athena Table](#create-a-table)
* [Querying data from Amazon S3 using Amazon Athena](#querying-data-from-amazon-s3-using-amazon-athena)
* [Querying partitioned data using Amazon Athena](#querying-partitioned-data-using-amazon-athena)
* [Create Athena Table with Partitions](#create-a-table-with-partitions)
* [Adding partition metadata to Amazon Athena](#adding-partition-metadata-to-amazon-athena)
* [Querying partitioned data set](#querying-partitioned-data-set)
* [Creating Views with Amazon Athena](#creating-views-with-amazon-athena)
* [CTAS Query with Amazon Athena](#ctas-query-with-amazon-athena)
* [Create an Amazon S3 Bucket](#create-an-amazon-s3-bucket)
* [Repartitioning the dataset using CTAS Query](#repartitioning-the-dataset-using-ctas-query)
* [Repartitioning and Bucketing the dataset using CTAS Query](#repartitioning-and-bucketing-the-dataset-using-ctas-query)
## Architectural Diagram
## Creating Amazon Athena Database and Table
Amazon Athena uses Apache Hive to define tables and create databases. Databases are a logical grouping of tables. When you create a database and table in Athena, you are simply describing the schema and location of the table data in Amazon S3\. In case of Hive, databases and tables don’t store the data along with the schema definition unlike traditional relational database systems. The data is read from Amazon S3 only when you query the table. The other benefit of using Hive is that the metastore found in Hive can be used in many other big data applications such as Spark, Hadoop, and Presto. With Athena catalog, you can now have Hive-compatible metastore in the cloud without the need for provisioning a Hadoop cluster or RDS instance. For guidance on databases and tables creation refer [Apache Hive documentation](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL). The following steps provide guidance specifically for Amazon Athena.
### Create Database
1. Open the [AWS Management Console for Athena](https://console.aws.amazon.com/athena/home).
2. If this is your first time visiting the AWS Management Console for Athena, you will get a Getting Started page. Choose **Get Started** to open the Query Editor. If this isn't your first time, the Athena **Query Editor** opens.
3. Make a note of the AWS region name, for example, for this lab you will need to choose the **US West (Oregon)** region.
4. In the Athena **Query Editor**, you will see a query pane with an example query. Now you can start entering your query in the query pane.
5. To create a database named *mydatabase*, copy the following statement, and then choose **Run Query**:
````sql
CREATE DATABASE mydatabase
````
6. Ensure *mydatabase* appears in the DATABASE list on the **Catalog** dashboard
### Create a Table
Now that you have a database, you are ready to create a table that is based on the New York taxi sample data. You define columns that map to the data, specify how the data is delimited, and provide the location in Amazon S3 for the file.
>**Note:**
>When creating the table, you need to consider the following:
>- You must have the appropriate permissions to work with data in the Amazon S3 location. For more information, refer [Setting User and Amazon S3 Bucket Permissions](http://docs.aws.amazon.com/athena/latest/ug/access.html).
>- The data can be in a different region from the primary region where you run Athena as long as the data is not encrypted in Amazon S3. Standard inter-region data transfer rates for Amazon S3 apply in addition to standard Athena charges.
>- If the data is encrypted in Amazon S3, it must be in the same region, and the user or principal who creates the table must have the appropriate permissions to decrypt the data. For more information, refer [Configuring Encryption Options](http://docs.aws.amazon.com/athena/latest/ug/encryption.html).
>- Athena does not support different storage classes within the bucket specified by the LOCATION clause, does not support the GLACIER storage class, and does not support Requester Pays buckets. For more information, see [Storage Classes](http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html),[Changing the Storage Class of an Object in Amazon S3](http://docs.aws.amazon.com/AmazonS3/latest/dev/ChgStoClsOfObj.html), and [Requester Pays Buckets](http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html) in the Amazon Simple Storage Service Developer Guide.
1. Ensure that current AWS region is **US West (Oregon)** region
2. Ensure **mydatabase** is selected from the **DATABASE** list and then choose **New Query**.
3. In the query pane, copy the following statement to create TaxiDataYellow table, and then choose **Run Query**:
````sql
CREATE EXTERNAL TABLE IF NOT EXISTS TaxiDataYellow (
VendorID STRING,
tpep_pickup_datetime TIMESTAMP,
tpep_dropoff_datetime TIMESTAMP,
passenger_count INT,
trip_distance DOUBLE,
pickup_longitude DOUBLE,
pickup_latitude DOUBLE,
RatecodeID INT,
store_and_fwd_flag STRING,
dropoff_longitude DOUBLE,
dropoff_latitude DOUBLE,
payment_type INT,
fare_amount DOUBLE,
extra DOUBLE,
mta_tax DOUBLE,
tip_amount DOUBLE,
tolls_amount DOUBLE,
improvement_surcharge DOUBLE,
total_amount DOUBLE
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 's3://us-west-2.serverless-analytics/NYC-Pub/yellow/'
````
>**Note:**
>- If you use CREATE TABLE without the EXTERNAL keyword, you will get an error as only tables with the EXTERNAL keyword can be created in Amazon Athena. We recommend that you always use the EXTERNAL keyword. When you drop a table, only the table metadata is removed and the data remains in Amazon S3.
>- You can also query data in regions other than the region where you are running Amazon Athena. Standard inter-region data transfer rates for Amazon S3 apply in addition to standard Amazon Athena charges.
>- Ensure the table you just created appears on the Catalog dashboard for the selected database.
## Querying data from Amazon S3 using Amazon Athena
Now that you have created the table, you can run queries on the data set and see the results in AWS Management Console for Amazon Athena.
1. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query**.
````sql
SELECT * FROM TaxiDataYellow limit 10
````
Results for the above query look like the following:
2. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query** to get the total number of taxi rides for yellow cabs.
````sql
SELECT COUNT(1) as TotalCount FROM TaxiDataYellow
````
Results for the above query look like the following:
>**Note:**
The current data format is CSV and this query is scanning **~207GB** of data and takes **~20.06** seconds to execute the query.
3. Make a note of query execution time for later comparison while querying the data set in Apache Parquet format.
4. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query** to query for the number of rides per vendor, along with the average fair amount for yellow taxi rides
````sql
SELECT
CASE vendorid
WHEN '1' THEN 'Creative Mobile Technologies'
WHEN '2' THEN 'VeriFone Inc'
ELSE vendorid END AS Vendor,
COUNT(1) as RideCount,
avg(total_amount) as AverageAmount
FROM TaxiDataYellow
WHERE total_amount > 0
GROUP BY (1)
````
Results for the above query look like the following:
## Querying partitioned data using Amazon Athena
By partitioning your data, you can restrict the amount of data scanned by each query, thus improving performance and reducing cost. Athena leverages Hive for [partitioning](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterPartition) data. You can partition your data by any key. A common practice is to partition the data based on time, often leading to a multi-level partitioning scheme. For example, a customer who has data coming in every hour might decide to partition by year, month, date, and hour. Another customer, who has data coming from many different sources but loaded one time per day, may partition by a data source identifier and date.
### Create a Table with Partitions
1. Ensure that current AWS region is **US West (Oregon)** region
2. Ensure **mydatabase** is selected from the DATABASE list and then choose **New Query**.
3. In the query pane, copy the following statement to create a the NYTaxiRides table, and then choose **Run Query**:
````sql
CREATE EXTERNAL TABLE NYTaxiRides (
vendorid STRING,
pickup_datetime TIMESTAMP,
dropoff_datetime TIMESTAMP,
ratecode INT,
passenger_count INT,
trip_distance DOUBLE,
fare_amount DOUBLE,
total_amount DOUBLE,
payment_type INT
)
PARTITIONED BY (YEAR INT, MONTH INT, TYPE string)
STORED AS PARQUET
LOCATION 's3://us-west-2.serverless-analytics/canonical/NY-Pub'
````
4.Ensure the table you just created appears on the Catalog dashboard for the selected database.
>**Note:**
> Running the following sample query on the NYTaxiRides table you just created will not return any result as no metadata about the partition is added to the Amazon Athena table catalog.
>```sql
> SELECT * FROM NYTaxiRides limit 10
>```
### Adding partition metadata to Amazon Athena
Now that you have created the table you need to add the partition metadata to the Amazon Athena Catalog.
1. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query** to add partition metadata.
```sql
MSCK REPAIR TABLE NYTaxiRides
```
The returned result will contain information for the partitions that are added to NYTaxiRides for each taxi type (yellow, green, fhv) for every month for the year from 2009 to 2016
>**Note:**
> The MSCK REPAIR TABLE automatically adds partition data based on the New York taxi ride data to in the Amazon S3 bucket is because the data is already converted to Apache Parquet format partitioned by year, month and type, where type is the taxi type (yellow, green or fhv). If the data layout does not confirm with the requirements of MSCK REPAIR TABLE the alternate approach is to add each partition manually using ALTER TABLE ADD PARTITION. You can also automate adding partitions by using the JDBC driver.
### Querying partitioned data set
Now that you have added the partition metadata to the Athena data catalog you can now run your query.
1. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query** to get the total number of taxi rides
```sql
SELECT count(1) as TotalCount from NYTaxiRides
```
Results for the above query look like the following:
>**Note:**
> This query executes much faster because the data set is partitioned and it in optimal format - Apache Parquet (an open source columnar). Following is a comparison of the execution time and amount of data scanned between the data formats:
>
>>**CSV Format:**
>>```sql
>> SELECT count(*) as count FROM TaxiDataYellow
>>```
>>Run time: **~20.06 seconds**, Data scanned: **~207.54GB**, Count: **1,310,911,060**
>>```sql
>>SELECT * FROM TaxiDataYellow limit 1000
>>```
>>Run time: **~3.13 seconds**, Data scanned: **~328.82MB**
>
>>**Parquet Format:**
>>```sql
>>SELECT count(*) as count FROM NYTaxiRides
>>```
>>Run time: **~5.76 seconds**, Data scanned: **0KB**, Count: **2,870,781,820**
>>```sql
>>SELECT * FROM NYTaxiRides limit 1000
>>```
>>Run time: **~1.13 seconds**, Data scanned: **5.2MB**
2. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query** to get the total number of taxi rides by year
```sql
SELECT YEAR, count(1) as TotalCount from NYTaxiRides GROUP BY YEAR
```
Results for the above query look like the following:
3. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query** to get the top 12 months by total number of rides across all the years
```sql
SELECT YEAR, MONTH, COUNT(1) as TotalCount
FROM NYTaxiRides
GROUP BY (1), (2)
ORDER BY (3) DESC LIMIT 12
```
Results for the above query look like the following:
4. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query** to get the monthly ride counts per taxi time for the year 2016.
```sql
SELECT MONTH, TYPE, COUNT(1) as TotalCount
FROM NYTaxiRides
WHERE YEAR = 2016
GROUP BY (1), (2)
ORDER BY (1), (2)
```
Results for the above query look like the following:
>**Note:**
Now the execution time is ~ 3 second, as the amount of data scanned by the query is restricted thus improving performance. This is because the data set is partitioned and it in optimal format – Apache Parquet, an open source columnar format.
5. Choose **New Query**, copy the following statement anywhere into the query pane, and then choose **Run Query**.
```sql
SELECT MONTH,
TYPE,
avg(trip_distance) as avgDistance,
avg(total_amount/trip_distance) as avgCostPerMile,
avg(total_amount) as avgCost,
approx_percentile(total_amount, .99) percentile99
FROM NYTaxiRides
WHERE YEAR = 2016 AND (TYPE = 'yellow' OR TYPE = 'green') AND trip_distance > 0 AND total_amount > 0
GROUP BY MONTH, TYPE
ORDER BY MONTH
```
Results for the above query look like the following:
## Creating Views with Amazon Athena
A view in Amazon Athena is a logical, not a physical table. The query that defines a view runs each time the view is referenced in a query. You can create a view from a SELECT query and then reference this view in future queries. For more information, see [CREATE VIEW](https://docs.aws.amazon.com/athena/latest/ug/create-view.html).
1. Ensure that current AWS region is **US West (Oregon)** region
2. Ensure **mydatabase** is selected from the DATABASE list.
3. Choose **New Query**, copy the following statement anywhere into the query pane, and then choose **Run Query**.
```sql
CREATE VIEW nytaxiridesmonthly AS
SELECT
year,
month,
vendorid,
avg(total_amount) as avg_Amt,
sum (total_amount) as sum_Amt
FROM nytaxirides
where total_amount > 0
group by vendorid, year, month
```
You will see a new view called **nytaxiridesmonthly** created under **Views** under **Database** section in the left.
4. Choose **New Query**, copy the following statement anywhere into the query pane, and then choose **Run Query**.
```sql
SELECT * FROM nytaxiridesmonthly WHERE vendorid = '1'
```
Some of the view specific commands to try out are [SHOW COLUMNS](https://docs.aws.amazon.com/athena/latest/ug/show-columns.html), [SHOW CREATE VIEW](https://docs.aws.amazon.com/athena/latest/ug/show-create-view.html), [DESCRIBE VIEW](https://docs.aws.amazon.com/athena/latest/ug/describe-view.html), and [DROP VIEW](https://docs.aws.amazon.com/athena/latest/ug/drop-view.html).
## CTAS Query with Amazon Athena
A CREATE TABLE AS SELECT (CTAS) query creates a new table in Athena from the results of a SELECT statement from another query. Athena stores data files created by the CTAS statement in a specified location in Amazon S3. For syntax, see [CREATE TABLE AS](https://docs.aws.amazon.com/athena/latest/ug/create-table-as.html).
Use CTAS queries to:
Create tables from query results in one step, without repeatedly querying raw data sets. This makes it easier to work with raw data sets.
Transform query results into other storage formats, such as Parquet and ORC. This improves query performance and reduces query costs in Athena. For information, see [Columnar Storage Formats](https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html).
Create copies of existing tables that contain only the data you need.
### Create an Amazon S3 Bucket
1. Open the [AWS Management console for Amazon S3](https://s3.console.aws.amazon.com/s3/home?region=us-west-2)
2. On the S3 Dashboard, Click on **Create Bucket**.
3. In the **Create Bucket** pop-up page, input a unique **Bucket name**. So it’s advised to choose a large bucket name, with many random characters and numbers (no spaces).
1. Select the region as **Oregon**.
2. Click **Next** to navigate to next tab.
3. In the **Set properties** tab, leave all options as default.
4. In the **Set permissions** tag, leave all options as default.
5. In the **Review** tab, click on **Create Bucket**
### Repartitioning the dataset using CTAS Query
1. Ensure that current AWS region is **US West (Oregon)** region
2. Ensure **mydatabase** is selected from the DATABASE list.
3. Choose **New Query**, copy the following statement anywhere into the query pane, and then choose **Run Query**.
```sql
CREATE TABLE ctas_nytaxride_partitioned
WITH (
format = 'PARQUET',
external_location = 's3://<name-of-the-bucket-your-created>/ctas_nytaxride_partitioned/',
partitioned_by = ARRAY['month','type','vendorid']
)
AS select
ratecode, passenger_count, trip_distance, fare_amount, total_amount, month, type, vendorid
FROM nytaxirides where year = 2016 and (vendorid = '1' or vendorid = '2')
```
Go to the Amazon S3 bucket specified as the external location and inspect the format and key structure in which the new objects are written in.
### Repartitioning and Bucketing the dataset using CTAS Query
4. Choose **New Query**, copy the following statement anywhere into the query pane, and then choose **Run Query**.
```sql
CREATE TABLE ctas_nytaxride_bucketed_partitioned
WITH (
format = 'PARQUET',
external_location = 's3://<name-of-the-bucket-your-created>/ctas_nytaxride_bucketed/',
partitioned_by = ARRAY['month', 'type'],
bucketed_by = ARRAY['vendorid'],
bucket_count = 3)
AS select
ratecode, passenger_count, trip_distance, fare_amount, total_amount,vendorid, month, type
FROM nytaxirides where year = 2016
```
>**Note:**
> This query will take approximately 6 minutes.
Go to the Amazon S3 bucket specified as the external location and inspect the format and key structure in which the new objects are written in.
Please refer to [Partitioning Vs. Bucketing](https://docs.aws.amazon.com/athena/latest/ug/bucketing-vs-partitioning.html) for more details.
---
## License
This library is licensed under the Apache 2.0 License.
================================================
FILE: Lab2/README.md
================================================
# Lab 2: Visualization using Amazon QuickSight
* [Create an Amazon S3 bucket](#create-an-amazon-s3-bucket)
* [Creating Amazon Athena Database and Table](#creating-amazon-athena-database-and-table)
* [Create Athena Database](#create-database)
* [Create Athena Table](#create-a-table)
* [Signing up for Amazon Quicksight Standard Edition](#signing-up-for-amazon-quicksight-standard-edition)
* [Configuring Amazon QuickSight to use Amazon Athena as data source](#configuring-amazon-quicksight-to-use-amazon-athena-as-data-source)
* [Visualizing the data using Amazon QuickSight](#visualizing-the-data-using-amazon-quicksight)
* [Add year based filter to visualize the dataset for the year 2016](#add-year-based-filter-to-visualize-the-dataset-for-the-year-2016)
* [Add the month based filter for the month of January](#add-the-month-based-filter-for-the-month-of-january)
* [Visualize the data by hour of day for the month of January 2016](#visualize-the-data-by-hour-of-day-for-the-month-of-january-2016)
* [Visualize the data for the month of January 2016 for all taxi types(yellow, green, fhv)](#visualize-the-data-for-the-month-of-january-2016-for-all-taxi-typesyellow-green-fhv)
## Architectural Diagram
## Create an Amazon S3 bucket
> Note: If you have already have an S3 bucket in your AWS Account you can skip this section.
1. Open the [AWS Management console for Amazon S3](https://s3.console.aws.amazon.com/s3/home?region=us-west-2)
2. On the S3 Dashboard, Click on **Create Bucket**.
3. In the **Create Bucket** pop-up page, input a unique **Bucket name**. It is advised to choose a large bucket name, with many random characters and numbers (no spaces).
1. Select the region as **Oregon**.
2. Click **Next** to navigate to next tab.
3. In the **Set properties** tab, leave all options as default.
4. In the **Set permissions** tag, leave all options as default.
5. In the **Review** tab, click on **Create Bucket**
## Creating Amazon Athena Database and Table
> Note: If you have complete the [Lab 1: Serverless Analysis of data in Amazon S3 using Amazon Athena](../Lab1) you can skip this section and go to the next section [Signing up for Amazon Quicksight Standard Edition](#signing-up-for-amazon-quicksight-standard-edition)
Amazon Athena uses Apache Hive to define tables and create databases. Databases are a logical grouping of tables. When you create a database and table in Athena, you are simply describing the schema and location of the table data in Amazon S3\. In case of Hive, databases and tables don’t store the data along with the schema definition unlike traditional relational database systems. The data is read from Amazon S3 only when you query the table. The other benefit of using Hive is that the metastore found in Hive can be used in many other big data applications such as Spark, Hadoop, and Presto. With Athena catalog, you can now have Hive-compatible metastore in the cloud without the need for provisioning a Hadoop cluster or RDS instance. For guidance on databases and tables creation refer [Apache Hive documentation](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL). The following steps provides guidance specifically for Amazon Athena.
1. In the **Create Bucket** pop-up page, input a unique **Bucket name**. It is advised to choose a large bucket name, with many random characters and numbers (no spaces).
1. Select the region as **Oregon**.
2. Click **Next** to navigate to next tab.
3. In the **Set properties** tab, leave all options as default.
4. In the **Set permissions** tag, leave all options as default.
5. In the **Review** tab, click on **Create Bucket**
### Create Database
1. Open the [AWS Management Console for Athena](https://console.aws.amazon.com/athena/home).
2. If this is your first time visiting the AWS Management Console for Athena, you will get a Getting Started page. Choose **Get Started** to open the Query Editor. If this isn't your first time, the Athena **Query Editor** opens.
3. Make a note of the AWS region name, for example, for this lab you will need to choose the **US West (Oregon)** region.
4. In the Athena **Query Editor**, you will see a query pane with an example query. Now you can start entering your query in the query pane.
5. To create a database named *mydatabase*, copy the following statement, and then choose **Run Query**:
````sql
CREATE DATABASE mydatabase
````
6. Ensure *mydatabase* appears in the DATABASE list on the **Catalog** dashboard
### Create a Table
1. Ensure that current AWS region is **US West (Oregon)** region
2. Ensure **mydatabase** is selected from the DATABASE list and then choose **New Query**.
3. In the query pane, copy the following statement to create a the NYTaxiRides table, and then choose **Run Query**:
````sql
CREATE EXTERNAL TABLE NYTaxiRides (
vendorid STRING,
pickup_datetime TIMESTAMP,
dropoff_datetime TIMESTAMP,
ratecode INT,
passenger_count INT,
trip_distance DOUBLE,
fare_amount DOUBLE,
total_amount DOUBLE,
payment_type INT
)
PARTITIONED BY (YEAR INT, MONTH INT, TYPE string)
STORED AS PARQUET
LOCATION 's3://us-west-2.serverless-analytics/canonical/NY-Pub'
````
4.Ensure the table you just created appears on the Catalog dashboard for the selected database.
Now that you have created the table you need to add the partition metadata to the Amazon Athena Catalog.
1. Choose **New Query**, copy the following statement into the query pane, and then choose **Run Query** to add partition metadata.
```sql
MSCK REPAIR TABLE NYTaxiRides
```
The returned result will contain information for the partitions that are added to NYTaxiRides for each taxi type (yellow, green, fhv) for every month for the year from 2009 to 2016
## Signing up for Amazon Quicksight Standard Edition
1. Open the [AWS Management Console for QuickSight](https://us-east-1.quicksight.aws.amazon.com/sn/start).
2. If this is the first time you are accessing QuickSight, you will see a sign-up landing page for QuickSight.
3. Click on **Sign up for QuickSight**.
> **Note:** Chrome browser might timeout at this step. If that's the case, try this step in Firefox/Microsoft Edge/Safari.
4. On the next page, for the subscription type select the **"Standard Edition"** and click **Continue**.
5. On the next page,
i. Enter a unique **QuickSight account name.**
ii. Enter a valid email for **Notification email address**.
iii. Just for this step, leave the **QuickSight capacity region** as **N.Virginia**.
iv. Ensure that **Enable autodiscovery of your data and users in your Amazon Redshift, Amazon RDS and AWS IAM Services** and **Amazon Athena** boxes are checked.
v. **Click Finish**.
vi. You will be presented with a message **Congratulations**! **You are signed up for Amazon QuickSight!** on successful sign up. Click on **Go to Amazon QuickSight**.
6. **Before continuing with the following steps, make sure you are in the N. Virginia Region to edit permissions.**
Now, on the Amazon QuickSight dashboard, navigate to User Settings page on the Top-Right section and click **Manage QuickSight**.
7. In this section, click on **Security & permissions** and then click **Add or remove**.
<p align="center"><img src="img/updated1.png" /></p>
8. Click on **Amazon S3** and on the tab that says **S3 buckets linked to QuickSight account**.
9. Ensure **Select All** is checked.
10. Click on **Select buckets**.
11. Now, select the **S3 Buckets you can access across AWS** tab on the top right. Make sure **Use a different bucket** is selected. Insert _us-west-2.serverless-analytics_ as the bucket name and select **Add S3 bucket**. It should look similar to below:
<p align="center"><img src="img/updated2.png" /></p>
12. When you are done doing all this, click **Update** to bring you back to the user settings back.
## Configuring Amazon QuickSight to use Amazon Athena as data source
> For this lab, you will need to choose the **US West (Oregon)** region.
1. Click on the region icon on the top-right corner of the page, and select **US West (Oregon)**.
2. Click on **Manage data** on the top-right corner of the webpage to review existing data sets.
3. Click on **New data set** on the top-left corner of the webpage and review the options.
4. Select **Athena** as a Data source.
5. Enter the **Data source** **name** (e.g. *AthenaDataSource*).
6. Click **Create data source**.
7. Select the **mydatabase** database.
8. Choose the **nytaxirides** table.
9. Choose **Edit/Preview** data.
> This is a crucial step. Please ensure you choose **Edit/Preview** data.
10. Under **Fields** on the left column, choose **Add calculate field**
i. Select the **extract** operation from Function list.
ii. Select **pickup_datetime** from the **Field list**.
iii. For **Calculated field name**, type **hourofday**.
iv. Type ‘HH’ so the Formula is **extract('HH',{pickup_datetime})**
v. Choose **Create** to add a field which is calculated from an existing field. In this case, the **hourofday** field is calculated from the **pickup_datetime field** based on the specified formula.
11. Choose **Save and Visualize** on top of the page.
## Visualizing the data using Amazon QuickSight
Now that you have configured the data source and created a new field to represent the hour of the day, in this section you will filter the data by year followed by month to visualize the taxi data for the entire month of January 2016 based on the **pickup_datetime** field.
### Add year based filter to visualize the dataset for the year 2016
1. Ensure that current AWS region is **US West (Oregon)** region.
2. Under the **Fields List**, select the **year** field to show the distribution of fares per year.
3. To reformat the **year** without comma
i. Select the dropdown arrow for the **year** field.
ii. Select **Format 1,234.5678** from the dropdown menu.
iii. Select **1235**.
4. To add a filter on the **year** field,
i. Select the dropdown for **year** field from the **Fields list**.
ii. Select **Add filter to the field** from the dropdown menu.
5. To filter the data only for the year 2016
i. Choose the new filter that you just created by clicking on **#** next to filter name **year** under the **Edit filter** menu.
ii. Select **Filter list** for the two dropdowns under the filter name.
iii. Deselect **Select All**.
iv. Select only **2016**.
v. Click **Apply**.
vi. Click **Close**.
### Add the month based filter for the month of January
1. Ensure that current AWS region is **US West(Oregon)** region.
2. Select **Visualize** from the navigation menu in the left-hand corner.
3. Under the **Fields list**, deselect **year** by clicking on **year** field name.
4. Select **month** by clicking on the **month** field name from the **Fields list**.
5. To filter the data set for the month of January (Month 1)
i. Select the dropdown arrow for **month** field under the **Fields List**.
ii. Select **Add filter to the field**.
6. To filter the data for month of January 2016 (Month 1),
i. Choose the new filter that you just created by clicking on **#** next to filter name **month** under the **Edit Filter** menu.
ii. Select **Filter list** for the two dropdowns under the filter name.
iii. Deselect **ALL**.
iv. Select only **1**.
v. Click **Apply**
vi. Click **Close**.
### Visualize the data by hour of day for the month of January 2016
1. Select **Visualize** from the navigation menu in the left-hand corner.
2. Under the **Fields list**, deselect **month** by clicking on **month** field name.
3. Select **hourofday** by clicking on the **hourofday** field name from the **Fields list**.
4. Change the visual type to a line chart by selecting the line chart icon highlighted in the screenshot below under **Visual types**.
5. Using the slider on x-axis, select the entire range [0,23] for **hourofday** field.
### Visualize the data for the month of January 2016 for all taxi types(yellow, green, fhv)
1. Click on the double drop-down arrow underneath your username at the top-right corner of the page to reveal **X-axis**, **Value** and **Color** under **Field wells**.
2. Under the **Fields list**, deselect **hourofday** by clicking on **hourofday** field name.
3. Select **pickup_datetime** for x-axis by clicking on the **pickup_datetime** field name from **Fields list**.
4. Select **type** for Color by clicking on the **type** field name from **Fields list.**
5. Click on the field name **pickup_datetime** in x-axis to reveal a sub-menu.
6. Select **Aggregate:Day** to aggregate by day.
8. Using the slider on x-axis, select the entire month of January 2016 for **pickup_datetime** field.
> Note: The interesting outlier in the above graph is that on Jan23rd, 2016, you see the dip in the number of taxis across all types. Doing a quick google search for that date, gets us this weather article from NBC New York
> 
*Using Amazon QuickSight, you were able to see patterns across a time-series data by building visualizations, performing ad-hoc analysis, and quickly generating insights.*
---
## License
This library is licensed under the Apache 2.0 License.
================================================
FILE: Lab2/img/README.md
================================================
adding images
================================================
FILE: Lab3/README.md
================================================
# Lab 3: Serverless ETL and Data Discovery using Amazon Glue
* [Create an IAM Role](#create-an-iam-role)
* [Create an Amazon S3 bucket](#create-an-amazon-s3-bucket)
* [Discover the Data](#discover-the-data)
* [Optimize the Queries and convert into Parquet](#optimize-the-queries-and-convert-into-parquet)
* [Query the Partitioned Data using Amazon Athena](#query-the-partitioned-data-using-amazon-athena)
* [Deleting the Glue database, crawlers and ETL Jobs created for this Lab](#deleting-the-glue-database-crawlers-and-etl-jobs-created-for-this-lab)
* [Summary](#summary)
## Architectural Diagram
## Create an IAM Role
Create an IAM role that has permission to your Amazon S3 sources, targets, temporary directory, scripts, **AWSGlueServiceRole** and any libraries used by the job. You can click [here](https://console.aws.amazon.com/iam/home?region=us-west-2#/roles) to create a new role. For additional documentation to create a role click [here](docs.aws.amazon.com/cli/latest/reference/iam/create-role.html).
1. On the IAM page, click on **Create Role**.
2. Choose the service as **Glue** and click on **Next: Permissions** on the bottom.
3. On the Attach permissions policies, search policies for S3 and check the box for **AmazonS3FullAccess**.
> Do not click on the policy, you just have to check the corresponding checkbox.
4. On the same page, now search policies for Glue and check the box for **AWSGlueServiceRole** and **AWSGlueConsoleFullAccess**.
> Do not click on the policy, you just have to check the corresponding checkbox.
5. Click on **Next: Review**.
6. Enter Role name as
```
nycitytaxianalysis-reinv
```
and click **Create role**.
## Create an Amazon S3 bucket
1. Open the [AWS Management console for Amazon S3](https://s3.console.aws.amazon.com/s3/home?region=us-west-2)
2. On the S3 Dashboard, Click on **Create Bucket**.
1. In the **Create Bucket** pop-up page, input a unique **Bucket name**. So it’s advised to choose a large bucket name, with many random characters and numbers (no spaces). It will be easier to name your bucket
```
aws-glue-scripts-<YOURAWSACCOUNTID>-us-west-2
```
and it would be easier to choose/select this bucket for the remainder of this Lab3.
1. Select the region as **Oregon**.
2. Click **Next** to navigate to the next tab.
3. In the **Set properties** tab, leave all options as default.
4. In the **Set permissions** tag, leave all options as default.
5. In the **Review** tab, click on **Create Bucket**
2. Now, in this newly created bucket, create two sub-folders **tmp** and **target** using the same instructions as the above step. We will use these buckets as part of Lab3 later on.
## Discover the Data
During this workshop, we will focus on one month of the New York City Taxi Records dataset, however you could easily do this for the entire eight years of data. As you crawl this unknown dataset, you discover that the data is in different formats, depending on the type of taxi. You then convert the data to a canonical form, start to analyze it, and build a set of visualizations. All without launching a single server.
> For this lab, you will need to choose the **US West (Oregon)** region.
1. Open the [AWS Management console for Amazon Glue](https://us-west-2.console.aws.amazon.com/glue/home?region=us-west-2#).
2. To analyze all the taxi rides for January 2016, you start with a set of data in S3. First, create a database for this workshop within AWS Glue. A database is a set of associated table definitions, organized into a logical group. In Athena, database names are all lowercase, no matter what you type.
i. Click on **Databases** under Data Catalog column on the left.
ii. Click on the **Add Database** button.
iii. Enter the Database name as **nycitytaxianalysis-reinv17**. You can skip the description and location fields and click on **Create**.
3. Click on **Crawlers** under Data Catalog column on the left.
i. Click on **Add Crawler** button.
ii. Under Add information about your crawler, for Crawler name type **nycitytaxianalysis-crawler-reinv17**. You can skip the Description and Classifiers field and click on **Next**.
iii. Under Specify crawler source type, make sure **Data stores** is selected. Click **Next**.
iv. When choosing a data store, make sure S3 is selected. Ensure the radio button for **Crawl Data in Specified path** is checked. For Include path, enter the following S3 path and click on **Next**.
```
s3://serverless-analytics/glue-blog
```
v. For Add Another data store, choose **No** and click on **Next**.
vi. For Choose an IAM Role, select **Create an IAM role** and enter the role name as following and click on **Next**.
```
nycitytaxianalysis-reinv17-crawler
```
vii. For Create a schedule for this crawler, choose Frequency as **Run on Demand** and click on **Next**.
viii. Configure the crawler output database and prefix:
a. For **Database**, select the database created earlier, **nycitytaxianalysis-reinv17**.
b. For **Prefix added to tables (optional)**, type **reinv17_** and click on **Next**.
c. Review configuration and click on **Finish** and on the next page, click on **Run it now** in the green box on the top.
d. The crawler runs and indicates that it found three tables.
4. Click on **Tables** under Data Catalog on the left column.
5. If you look under **Tables**, you can see the three new tables that were created under the database nycitytaxianalysis-reinv17.
6. The crawler used the built-in classifiers and identified the tables as CSV, inferred the columns/data types, and collected a set of properties for each table. If you look in each of those table definitions, you see the number of rows for each dataset found and that the columns don’t match between tables. As an example, clicking on the reinv17_yellow table, you can see the yellow dataset for January 2017 with 8.7 million rows, the location on S3, and the various columns found.
## Optimize the Queries and convert into Parquet
Create an ETL job to move this data into a query-optimized form. You convert the data into a column format, changing the storage type to Parquet, and writing the data to a bucket that you own.
1. Open the [AWS Management console for Amazon Glue](https://us-west-2.console.aws.amazon.com/glue/home?region=us-west-2#).
2. Click on **Jobs** under ETL on the left column and then click on the **Add Job** button.
3. Under Job properties, input name as **nycitytaxianalysis-reinv17-yellow**. Since we will be working with only the yellow dataset for this workshop.
i. Under IAM Role, Choose the IAM role created at the beginning of this lab which should be named **nycitytaxianalysis-reinv**.
x. Under This job runs, choose the radio button for **A proposed script generated by AWS Glue**.
xi. For Script file name, enter **nycitytaxianalysis-reinv17-yellow**.
> For this workshop, we are only working on the yellow dataset. Feel free to run through these steps to also convert the green and FHV dataset.
xii. For S3 path where script is stored, click on the Folder icon and choose the S3 bucket created at the beginning of this workshop. **Choose the newly created S3 bucket via the Folder icon and click Select**.
xiii. For Temporary directory, choose the tmp folder created at the beginning of this workshop. **Choose the S3 bucket via the Folder icon** and click **Select**.
> Ensure the temporary bucket is already created/available in your S3 bucket.
xiv. Click on Advanced properties, and select **Enable** for Job bookmark.
xv. Here's a screenshot of a finished job properties window:
4. Click **Next**.
5. Under Choose your data sources, select **reinv17_yellow** table as the data source and click on **Next**.
6. Under Choose a transform type, make sure **Change schema** is selected and click on **Next**.
> For this workshop, we are only working on the yellow dataset. Feel free to run through these steps to also convert the green and FHV dataset.
6. Under Choose your data targets, select the radio button for **Create tables in your data target**.
i. For Data store, Choose **Amazon S3**.
ii. For Format, choose **Parquet**.
iii. For Target path, **click on the folder icon** and choose the target folder previously created. **This S3 Bucket/Folder will contain the transformed Parquet data**.
7. Under Map the source columns to target columns page,
i. Under Target, change the Column name **tpep_pickup_datetime** to **pickup_date**. Click on its respective **data type** field string and change the Column type to **TIMESTAMP** and click on **Update**.
ii. Under Target, change the Column name **tpep_dropoff_datetime** to **dropoff_date**. Click on its respective **data type** field string and change the Column type to **TIMESTAMP** and click on **Update**.
iii. Choose **Save job and edit script**.
8. On the auto-generated script page, click on **Save** and **Run Job**.
8. In the parameters pop-up under Advanced properties, ensure Job bookmark is **Enabled** and click on **Run Job**.
9. This job will run for roughly around 30 minutes.
10. You can view logs on the bottom page of the same page.
11. The target folder (S3 Bucket) specified above (step 6 iii) will now have the converted parquet data.
## Query the Partitioned Data using Amazon Athena
In regions where AWS Glue is supported, Athena uses the AWS Glue Data Catalog as a central location to store and retrieve table metadata throughout an AWS account. The Athena execution engine requires table metadata that instructs it where to read data, how to read it, and other information necessary to process the data. The AWS Glue Data Catalog provides a unified metadata repository across a variety of data sources and data formats, integrating not only with Athena, but with Amazon S3, Amazon RDS, Amazon Redshift, Amazon Redshift Spectrum, Amazon EMR, and any application compatible with the Apache Hive metastore.
1. Open the [AWS Management console for Amazon Athena](https://us-west-2.console.aws.amazon.com/athena/home?force®ion=us-west-2).
> Ensure you are in the **US West (Oregon)** region.
2. Under Database, you should see the database **nycitytaxianalysis-reinv17** which was created during the previous section.
3. Click on **Create Table** right below the drop-down for Database and click on **From AWS Glue Crawler**. Click **Continue**.
4. You will now be re-directed to the AWS Glue console to set up a crawler. The crawler connects to your data store and automatically determines its structure to create the metadata for your table.
5. Enter Crawler name as **nycitytaxianalysis-crawlerparquet-reinv17** and Click **Next**.
6. Under Specify crawler source type, make sure **Data stores** is selected. Hit **Next** and on the next page, select the Data store as **S3**.
7. Choose Crawl data in **Specified path in my account**.
8. For Include path, click on the folder Icon and choose the **target** folder previously made which contains the parquet data and click on **Next**.
9. In Add another data store, choose **No** and click on **Next**.
10. For Choose an IAM role, select Choose an existing IAM role, and in the drop-down pick the role made in the previous section which should look similar to **AWSGlueServiceRole-nycitytaxianalysis-reinv17-crawler**. Click on **Next**.
11. In Create a schedule for this crawler, pick frequency as **Run on demand** and click on **Next**.
12. For Configure the crawler's output, Click **Add Database** to create a new database. Enter **nycitytaxianalysis-reinv17-parquet** as the database name and click **Create**. For Prefix added to tables, you can enter a prefix **parq_** and click **Next**.
13. Review the Crawler Info and click **Finish**. Click on **Run it Now?**.
14. Click on **Tables** on the left, and for database nycitytaxianalysis-reinv17-parquet you should see the table parq_target. Click on the table name and you will see the MetaData for this converted table.
15. Open the [AWS Management console for Amazon Athena](https://us-west-2.console.aws.amazon.com/athena/home?force®ion=us-west-2).
> Ensure you are in the **US West (Oregon)** region.
16. Under Database, you should see the database **nycitytaxianalysis-reinv17-parquet** which was just created. Select this database and you should see under Tables **parq_target**.
17. In the query editor on the right, type
```
select count(*) from parq_target;
```
and take note the Run Time and Data scanned numbers here.
What we see is the Run time and Data scanned numbers for Amazon Athena to **query and scan the parquet data**.
18. Under Database, you should see the earlier made database **nycitytaxianalysis-reinv17** which was created in a previous section. Select this database and you should see under Tables **reinv17_yellow**.
19. In the query editor on the right, type
```
select count(*) from reinv17_yellow;
```
and take note the Run Time and Data scanned numbers here.
20. What we see is the Run time and Data scanned numbers for Amazon Athena to query and scan the uncompressed data from the previous section.
> Note: Athena charges you by the amount of data scanned per query. You can save on costs and get better performance if you partition the data, compress data, or convert it to columnar formats such as Apache Parquet.
## Deleting the Glue database, crawlers and ETL Jobs created for this Lab
Now that you have successfully discovered and analyzed the dataset using Amazon Glue and Amazon Athena, you need to delete the resources created as part of this lab.
1. Open the [AWS Management console for Amazon Glue](https://us-west-2.console.aws.amazon.com/glue/home?region=us-west-2#). Ensure you are in the Oregon region (as part of this lab).
2. Click on **Databases** under Data Catalog column on the left.
3. Check the box for the Database that were created as part of this lab. Click on **Action** and select **Delete Database**. And click on **Delete**. This will also delete the tables under this database.
4. Click on **Crawlers** under Data Catalog column on the left.
5. Check the box for the crawler that were created as part of this lab. Click on **Action** and select **Delete Crawler**. And click on **Delete**.
6. Click on **Jobs** under ETL column on the left.
7. Check the box for the jobs that were created as part of this lab. Click on **Action** and select **Delete**. And click on **Delete**.
8. Open the [AWS Management console for Amazon S3](https://s3.console.aws.amazon.com/s3/home).
9. Click on the S3 bucket that was created as part of this lab. You need to click on its corresponding **Bucket icon** to select the bucket instead of opening the bucket. Click on **Delete bucket** button on the top, to delete the S3 bucket. In the pop-up window, Type the name of the bucket (that was created as part of this lab), and click **Confirm**.
## Summary
In the lab, you went from data discovery to analyzing a canonical dataset, without starting and setting up a single server. You started by crawling a dataset you didn’t know anything about and the crawler told you the structure, columns, and counts of records.
From there, you saw the datasets were in different formats, but represented the same thing: NY City Taxi rides. You then converted them into a canonical (or normalized) form that is easily queried through Athena and possible in QuickSight, in addition to a wide number of different tools not covered in this post.
---
## License
This library is licensed under the Apache 2.0 License.
================================================
FILE: Lab4/README.md
================================================
# Lab 4: Analysis of data in Amazon S3 using Amazon Redshift Spectrum
* [Deploying Amazon Redshift Cluster](#deploying-amazon-redshift-cluster)
* [Running AWS Glue Crawlers](#running-aws-glue-crawlers---csv--parquet-crawler)
* [Create Redshift Spectrum Scehma and reference external table from AWS Glue Data Catalog Database](#create-redshift-spectrum-scehma-and-reference-external-table-form-aws-glue-data-catalog-database)
* [Querying data from Amazon S3 using Amazon Redshift Spectrum](#querying-data-from-amazon-s3-using-amazon-redshift-spectrum)
* [Querying partitioned data using Amazon Redshift Spectrum](#querying-partitioned-data-using-amazon-redshift-spectrum)
## Architectural Diagram
## Deploying Amazon Redshift Cluster
In this section you will use the CloudFormation template to create Amazon RedShift cluster resources. The template will also install [pgweb](https://github.com/sosedoff/pgweb), an SQL Client for PostgreSQL, in an Amazon EC2 instance to connect and run your queries on the launched Amazon Redshift cluster. Alternatively, you can connect to the Amazon Redshift cluster using standard SQL Clients such as SQL Workbench/J. For more information refer http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-using-workbench.html.
1. Login in to your AWS console and open the [Amazon CloudFormation Dashboard](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2])
2. Make a note of the AWS region name, for example, for this lab you will need to choose the **US West (Oregon)** region.
3. Click **Create Stack**
4. Copy the contents of the file here [redshiftspectrumglue-lab4.template](../Lab4/redshiftspectrumglue-lab4.template) and save it to your local machine as **redshiftspectrumglue-lab4.json**. This file is the Amazon CloudFormation template file. Select **Upload a template file** and upload this file by clicking **Choose file**.
6. Click **Next**
8. Type a name *(e.g. RedshiftSpectrumLab)* for the **Stack Name**
9. Enter the following **Parameters** for **Redshift Cluster Configuration**
1. Choose *multi-node* for **ClusterType**
2. Type *2* for the **NumberOfNodes**
3. For **NodeType** select *dc1.large*
10. Enter the following **Parameters** for **Redshift Database Configuration**.
1. Type a name (e.g. dbadmin) for **MasterUserName**.
2. Type a password for **MasterUserPassword**. Make sure this password has at least 1 uppercase letter, 1 lowercase letter, and 1 number.
3. Type the a name (e.g. taxidb) for **DatabaseName**.
4. Type the IP address of your local machine for **ClientIP**.
11. Enter the following **Parameters** for **Glue Crawler Configuration**
1. Type the name (e.g. taxi-spectrum-db) for **GlueCatalogDBName**.
2. Type the name (e.g. csvCrawler) for **CSVCrawler**.
3. Type the name (e.g. parquetCrawler) for **ParquetCrawler**.
12. Click **Next**
13. [Optional] In the **Tags** sub-sections in **Options** type a **Key** name *(e.g. Name)* and **Value** for key.
14. Click **Next**
15. Check **I acknowledge that AWS CloudFormation might create IAM resources.**
16. Click **Create**
> **Note:** This is may take approximately 15 minutes
17. Ensure that status of the Amazon CloudFormation stack that you just created is **CREATE_COMPLETE**
18. Select your Amazon CloudFormation stack *(RedshiftSpectrumLab)*
19. Click on the **Outputs** tab
20. Review the list of **Key** and **Value** pairs, which will look like the following.
## Running AWS Glue Crawlers - CSV & Parquet Crawler
1. Open [AWS Management Console for Glue](https://us-west-2.console.aws.amazon.com/glue/home?region=us-west-2#)
2. Go to AWS Glues Crawlers page by clicking on **Crawlers** in the navigation pane
3. Select the AWS Glue Crawler for CSV (e.g. csvCrawler)
4. Click **Run crawler**
5. Select the AWS Glue Crawler for Parquet (e.g. parquetCrawler)
6. Click **Run crawler**
> Note: This may take approximately 5 min for both the crawlers to parse the data in CSV and Parquet format.
7. Wait for the **Status** of both the crawlers to return to the *Ready* state
Now that you have run the crawlers lets ensure that new tables *taxi* and *ny_pub* been created.
8. Go to the list of databases in the AWS Glue Data Catalog by clicking on **Databases** in the navigation pane.
9. Click on **taxi-spectrum-db**
10. Click on **Tables in taxi-spectrum-db**
11. Click on **taxi** to review the table definition and schema
12. Navigate back and click on **ny_pub** to review the table definition and schema
>**Note:**
>The good news is that you don’t have to create a new table or definition to read the CSV document we just looked at. With AWS Glue crawlers, you have already inferred the schema and created tables namely taxi and ny_pub.
13. Click on **View partitions** to review the partition metadata
>**Note:**
> The major advantage of Glue Crawlers is that they understand the partitions based on the S3 object prefix and automatically create the table with partitions as part of the crawling.
## Create Redshift Spectrum Scehma and reference external table from AWS Glue Data Catalog Database
1. Open the [Amazon CloudFormation Dashboard](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2])
2. Make a note of the AWS region name, for example, for this lab you will need to choose the **US West (Oregon)** region.
3. Select your Amazon CloudFormation stack *(RedshiftSpectrumLab)*
4. Click on the **Outputs** tab
5. Navigate to the **pgWeb** URL
6. In the pgWeb console ensure that the **SQL Query** tab is selected
7. Copy the following statement to create a database *(e.g. taxispectrum)* in Redshift Spectrum
```sql
create external schema taxispectrum from data catalog
database 'taxi-spectrum-db'
iam_role '<specify the redshift IAM Role arn from the CloudFormation outputs section>'
```
8. Replace the *<specify the redshift IAM Role arn from the CloudFormation output section'>* in the statment with the value of **redshiftIAMRole** from the **Outputs** tab of the Amazon CloudFromation stack *(RedshiftSpectrumLab)* you created as part of the lab.
> Note: The IAM role must be in single quotes
9. Click **Run Query**
> Note: You can create an external table in Amazon Redshift, AWS Glue, Amazon Athena, or an Apache Hive metastore. For more information, see [Getting Started Using AWS Glue](http://docs.aws.amazon.com/glue/latest/dg/getting-started.html) in the AWS Glue Developer Guide, [Getting Started](http://docs.aws.amazon.com/athena/latest/ug/getting-started.html) in the Amazon Athena User Guide, or [Apache Hive](http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive.html) in the Amazon EMR Developer Guide. If your external table is defined in AWS Glue, Athena, or a Hive metastore, you first create an external schema that references the external database. Then you can reference the external table in your SELECT statement by prefixing the table name with the schema name, without needing to create the table in Amazon Redshift. For more information, see [Creating External Schemas for Amazon Redshift Spectrum](http://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-external-schemas.html.)
## Querying data from Amazon S3 using Amazon Redshift Spectrum
Now that you have created the schema, you can run queries on the data set and see the results in PGWeb Console.
1. Copy the following statement into the query pane, and then choose **Run Query**.
```sql
SELECT * FROM taxispectrum.taxi limit 10
```
Results for the above query look like the following:
2. Copy the following statement into the query pane, and then choose **Run Query** to get the total number of taxi rides for yellow cabs.
```sql
SELECT COUNT(1) as TotalCount FROM taxispectrum.taxi
```
Results for the above query look like the following:
3. Copy the following statement into the query pane, and then choose **Run Query** to query for the number of rides per vendor, along with the average fair amount for yellow taxi rides
```sql
SELECT
CASE vendorid
WHEN '1' THEN 'Creative Mobile Technologies'
WHEN '2' THEN 'VeriFone Inc'
ELSE CAST(vendorid as VARCHAR) END AS Vendor,
COUNT(1) as RideCount,
avg(total_amount) as AverageAmount
FROM taxispectrum.taxi
WHERE total_amount > 0
GROUP BY (1)
```
Results for the above query look like the following:
## Querying partitioned data using Amazon Redshift Spectrum
By partitioning your data, you can restrict the amount of data scanned by each query, thus improving performance and reducing cost. Amazon Redshift Spectrum leverages Hive for [partitioning](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterPartition) data. You can partition your data by any key. A common practice is to partition the data based on time, often leading to a multi-level partitioning scheme. For example, a customer who has data coming in every hour might decide to partition by year, month, date, and hour. Another customer, who has data coming from many different sources but loaded one time per day, may partition by a data source identifier and date.
Now that you have added the partition metadata to the Athena data catalog you can now run your query.
1. Copy the following statement into the query pane, and then choose **Run Query** to get the total number of taxi rides
```sql
SELECT count(1) as TotalCount from taxispectrum.ny_pub
```
Results for the above query look like the following:
>**Note:**
> This query executes much faster because the data set is partitioned and it in optimal format - Apache Parquet (an open source columnar).
2. Copy the following statement into the query pane, and then choose **Run Query** to get the total number of taxi rides by year
```sql
SELECT YEAR, count(1) as TotalCount from taxispectrum.ny_pub GROUP BY YEAR
```
Results for the above query look like the following:
3. Copy the following statement into the query pane, and then choose **Run Query** to get the top 12 months by total number of rides across all the years
```sql
SELECT YEAR, MONTH, COUNT(1) as TotalCount
FROM taxispectrum.ny_pub
GROUP BY (1), (2)
ORDER BY (3) DESC LIMIT 12
```
Results for the above query look like the following:
4. Copy the following statement into the query pane, and then choose **Run Query** to get the monthly ride counts per taxi time for the year 2016.
```sql
SELECT MONTH, TYPE, COUNT(1) as TotalCount
FROM taxispectrum.ny_pub
WHERE YEAR = 2016
GROUP BY (1), (2)
ORDER BY (1), (2)
```
Results for the above query look like the following:
5. Copy the following statement anywhere into the query pane, and then choose **Run Query**.
```sql
SELECT MONTH, TYPE,
avg(trip_distance) avgDistance,
avg(total_amount/trip_distance) avgCostPerMile,
avg(total_amount) avgCost,
percentile_cont(0.99)
within group (order by total_amount)
FROM taxispectrum.ny_pub
WHERE YEAR = 2016 AND (TYPE = 'yellow' OR TYPE = 'green')
AND trip_distance > 0 AND total_amount > 0
GROUP BY MONTH, TYPE
ORDER BY MONTH
```
Results for the above query look like the following:
## Deleting the Amazon CloudFormation Stack
Now that you have successfully queried the dataset using Amazon Redshift Spectrum, you need to tear down the stack that you deployed using the Amazon CloudFormation template.
1. Open the [Amazon CloudFormation Dashboard](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2)
2. Enable the check box next to the name of the stack *(e.g. RedshiftSpectrumLab)* that you deployed at the beginning of the Lab.
3. Click on **Actions** drop down button.
4. Select **Delete Stack**.
5. Click **Yes, Delete** on the *Delete Stack* pop dialog
6. Ensure that Amazon CloudFormation stack name *(e.g. RedshiftSpectrumLab)* is no longer showing in the list of stacks.
---
## License
This library is licensed under the Apache 2.0 License.
================================================
FILE: Lab4/redshiftspectrumglue-lab4.template
================================================
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "This CloudFormation template redshift_spectrum_analytics lab launches the redshift instances in the default VPC and assign the required roles for testing spectrum feature. It will also create Glue catalog database and use crawlers for infering schema from NY taxi data located in S3 bucket. You will be billed for the AWS resources used if you create a stack from this template",
"Parameters" : {
"ClusterType": {
"Description": "The type of the cluster",
"Type": "String",
"Default": "single-node",
"AllowedValues": [ "single-node", "multi-node" ],
"ConstraintDescription" : "must be single-node or multi-node."
},
"NumberOfNodes": {
"Description": "The nuber of compute nodes in the redshift cluster. When cluster type is specified as: 1) single-node, the NumberOfNodes parameter should be specified as 1, 2) multi-node, the NumberOfNodes parameter should be greater than 1",
"Type": "Number",
"Default": "1"
},
"NodeType": {
"Description": "The node type to be provisioned for the redshift cluster",
"Type": "String",
"Default": "dc1.large",
"AllowedValues" : [ "dc1.large", "ds2.xlarge"],
"ConstraintDescription" : "must be a valid RedShift node type."
},
"MasterUsername": {
"Description": "The user name associated with the master user account for the redshift cluster that is being created",
"Type": "String",
"Default": "dbadmin",
"AllowedPattern": "([a-z])([a-z]|[0-9])*",
"NoEcho": "false",
"ConstraintDescription" : "must start with a-z and contain only a-z or 0-9."
},
"MasterUserPassword": {
"Description": "The password associated with the master user account for the redshift cluster that is being created. ",
"Type": "String",
"NoEcho": "true",
"MinLength": "1",
"MaxLength": "41",
"AllowedPattern" : "[a-zA-Z0-9]*",
"ConstraintDescription" : "must contain only alphanumeric characters."
},
"DatabaseName": {
"Description": "The name of the database to be created when the redshift cluster is created",
"Type": "String",
"Default": "taxidb",
"AllowedPattern": "([a-z]|[0-9])+",
"ConstraintDescription" : "must contain a-z or 0-9 only."
},
"GlueCatalogDBName": {
"Description": "The DB name for Glue Catalog. You can create a external schema using this database in Redshift spectrum and query the tables that are created by the crawlers",
"Type": "String",
"MinLength": "4",
"MaxLength": "255",
"Default": "taxi-spectrum-db",
"AllowedPattern" : "[a-zA-Z0-9_-]*" ,
"ConstraintDescription" : "must contain only alphanumeric characters."
},
"CSVCrawler": {
"Description": "Crawler Name for CSV File. This crawler will crawls the CSV file located in s3://us-west-2.serverless-analytics/NYC-transportation/taxi/ and infer the schema",
"Type": "String",
"MinLength": "4",
"MaxLength": "255",
"Default": "csvCrawler",
"AllowedPattern" : "[a-zA-Z0-9_-]*" ,
"ConstraintDescription" : "must contain only alphanumeric characters."
},
"ParquetCrawler": {
"Description": "Crawler Name for Parquer files. This will crawls s3://us-west-2.serverless-analytics/canonical/NY-Pub files and infer the schema",
"Type": "String",
"MinLength": "4",
"MaxLength": "255",
"Default": "parquetCrawler",
"AllowedPattern" : "[a-zA-Z0-9_-]*" ,
"ConstraintDescription" : "must contain only alphanumeric characters."
},
"ClientIP" : {
"Description" : "The IP address range that can be used to connect to the RedShift instance from your local machine either directly or via pgWeb.It must be a valid IP CIDR range of the form x.x.x.x/x.Pls get your address using checkip.amazonaws.com or whatsmyip.org",
"Type": "String",
"MinLength": "9",
"MaxLength": "18",
"Default": "0.0.0.0/0",
"AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
"ConstraintDescription": "It must be a valid IP CIDR range of the form x.x.x.x/x. Suggest to enable access to your IP address only. Pls get your address using checkip.amazonaws.com or whatsmyip.org."
}
},
"Metadata" : {
"AWS::CloudFormation::Interface" : {
"ParameterGroups" : [
{
"Label" : { "default" : "Redshift Cluster Configuration" },
"Parameters" : [ "ClusterType", "NumberOfNodes","NodeType" ]
},
{
"Label" : { "default":"Redshift Database Configuration" },
"Parameters" : ["MasterUsername","MasterUserPassword", "DatabaseName" ]
},
{
"Label" : { "default" : "Enter IP address for the Redshift Security group configuration" },
"Parameters" : [ "ClientIP" ]
},
{
"Label" : { "default" : "Glue Crawler Configurations" },
"Parameters" : [ "GlueCatalogDBName", "CSVCrawler","ParquetCrawler" ]
}
]
}
},
"Conditions": {
"IsMultiNodeCluster": { "Fn::Equals": [ { "Ref": "ClusterType" }, "multi-node" ] }
},
"Mappings" : {
"AMIid" : {
"us-east-1" : { "ver" : "ami-6057e21a" },
"us-east-2" : { "ver" : "ami-5ec1673e" },
"us-west-2" : { "ver" : "ami-32d8124a" },
"eu-west-1" : { "ver" : "ami-760aaa0f" }
}
},
"Resources" : {
"RedshiftSpectrumRole" : {
"Type" : "AWS::IAM::Role",
"Properties" : {
"ManagedPolicyArns" : ["arn:aws:iam::aws:policy/AmazonAthenaFullAccess","arn:aws:iam::aws:policy/AmazonS3FullAccess"],
"AssumeRolePolicyDocument" : {
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Principal" : { "Service" : "redshift.amazonaws.com" },
"Action" : "sts:AssumeRole"
}
]
}
}
},
"RedshiftCluster" : {
"Type" : "AWS::Redshift::Cluster",
"Properties" : {
"ClusterType" : { "Ref": "ClusterType" },
"NumberOfNodes": { "Fn::If": [ "IsMultiNodeCluster", { "Ref": "NumberOfNodes" }, { "Ref": "AWS::NoValue" } ] },
"NodeType": { "Ref": "NodeType" },
"DBName" : { "Ref" : "DatabaseName" },
"MasterUsername" : { "Ref" : "MasterUsername" },
"MasterUserPassword" : { "Ref" : "MasterUserPassword" },
"PubliclyAccessible" : "true",
"VpcSecurityGroupIds" : [ { "Fn::GetAtt": [ "RedshiftSecurityGroup" , "GroupId" ] } ],
"Port" : "5439",
"IamRoles": [ { "Fn::GetAtt" : ["RedshiftSpectrumRole", "Arn"] } ]
}
},
"RedshiftSecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "This is the security group for Redshift",
"SecurityGroupIngress" : [
{
"IpProtocol" : "tcp",
"FromPort" : "5439",
"ToPort" : "5439",
"CidrIp" : { "Ref" : "ClientIP"}
},
{
"IpProtocol" : "tcp",
"FromPort" : "5439",
"ToPort" : "5439",
"CidrIp" : "172.31.0.0/16"
}
]
}
},
"PGWebInstance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": {"Fn::FindInMap" : [ "AMIid", { "Ref" : "AWS::Region" }, "ver" ]},
"InstanceType": "m3.medium",
"SecurityGroups" : [ { "Ref": "PGWebSecurityGroup" } ],
"Tags": [{
"Key": "Name",
"Value": "PGWeb"
}],
"UserData": { "Fn::Base64": { "Fn::Join": [ "", [
"#!/bin/bash -xe\n",
"wget https://github.com/sosedoff/pgweb/releases/download/v0.9.6/pgweb_linux_386.zip\n",
"unzip pgweb_linux_386.zip\n",
"/pgweb_linux_386 --bind 0.0.0.0 --listen 80",
" --port ", { "Fn::GetAtt" : [ "RedshiftCluster", "Endpoint.Port" ] },
" --host ", { "Fn::GetAtt" : [ "RedshiftCluster", "Endpoint.Address" ] },
" --user ",{ "Ref" : "MasterUsername" },
" --pass ", { "Ref" : "MasterUserPassword" },
" --db taxidb &"
]]}}
}
},
"PGWebSecurityGroup": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "This is the security group to allow web access",
"SecurityGroupIngress" : [
{
"IpProtocol" : "tcp",
"FromPort" : "80",
"ToPort" : "80",
"CidrIp" : { "Ref" : "ClientIP"}
}
]
}
},
"GlueParseS3Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"ManagedPolicyArns" : ["arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"],
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"glue.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Path": "/service-role/",
"Policies": [
{
"PolicyName": "Glue-access-taxi-data",
"PolicyDocument":
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::us-west-2.serverless-analytics/NYC-transportation/taxi/*",
"arn:aws:s3:::us-west-2.serverless-analytics/canonical/NY-Pub/*"
]
}
]
}
}]
}
},
"GlueDatabase": {
"Type": "AWS::Glue::Database",
"Properties": {
"CatalogId": {
"Ref": "AWS::AccountId"
},
"DatabaseInput":
{
"Name": { "Ref" : "GlueCatalogDBName" },
"Description": "Database used for Redshift Spectrum"
}
}
},
"MyCrawler2": {
"Type": "AWS::Glue::Crawler",
"Properties": {
"Name": { "Ref" : "ParquetCrawler" },
"Role": { "Fn::GetAtt": [ "GlueParseS3Role", "Arn"] },
"DatabaseName": {"Ref": "GlueDatabase"},
"Targets": {
"S3Targets": [
{
"Path": "s3://us-west-2.serverless-analytics/canonical/NY-Pub"
}
]
},
"SchemaChangePolicy": {
"UpdateBehavior": "UPDATE_IN_DATABASE",
"DeleteBehavior": "LOG"
}
}
},
"MyCrawler1": {
"Type": "AWS::Glue::Crawler",
"Properties": {
"Name": { "Ref" : "CSVCrawler" },
"Role": { "Fn::GetAtt": [ "GlueParseS3Role", "Arn"] },
"DatabaseName": { "Ref": "GlueDatabase" },
"Targets": {
"S3Targets": [
{
"Path": "s3://us-west-2.serverless-analytics/NYC-transportation/taxi/"
}
]
},
"SchemaChangePolicy": {
"UpdateBehavior": "UPDATE_IN_DATABASE",
"DeleteBehavior": "LOG"
}
}
}
},
"Outputs": {
"redshiftHost" : {
"Value": { "Fn::GetAtt" : [ "RedshiftCluster", "Endpoint.Address" ]}
},
"redshiftPort" : {
"Value": { "Fn::GetAtt" : [ "RedshiftCluster", "Endpoint.Port" ]}
},
"redshiftUser" : {
"Value": { "Ref" : "MasterUsername" }
},
"redshiftDatabase" : {
"Value": { "Ref" : "DatabaseName" }
},
"redshiftEndpoint" : {
"Value": { "Fn::Join": [ ":", [ { "Fn::GetAtt": [ "RedshiftCluster", "Endpoint.Address" ] }, { "Fn::GetAtt": [ "RedshiftCluster", "Endpoint.Port" ] } ] ] }},
"pgWeb" : {
"Value": { "Fn::Join" : [ "", [ "http://", { "Fn::GetAtt" : [ "PGWebInstance", "PublicDnsName" ]} ]] }
},
"redshiftIAMRole" : {
"Value": { "Fn::GetAtt" : ["RedshiftSpectrumRole", "Arn"] }
},
"GlueCatalogDBName" : {
"Value": {"Ref" : "GlueCatalogDBName" }
},
"CSVCrawlerName" : {
"Value": { "Ref" : "CSVCrawler" }
},
"ParquetCrawlerName" : {
"Value": { "Ref" : "ParquetCrawler" }
}
}
}
================================================
FILE: NOTICE
================================================
Serverless Data Analytics
Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
================================================
FILE: README.md
================================================
# Building an End-to-End Serverless Data Analytics Solution on AWS
## Overview
In this lab, you are going to build a serverless architecture to analyze the data directly from Amazon S3 using [Amazon Athena](https://aws.amazon.com/athena/) and visualize the data in [Amazon QuickSight](https://quicksight.aws/). The data set that you are going to use is a public data set that includes trip records from all trips completed in Yellow and Green taxis in NYC from 2009 to 2016, and all trips in for-hire vehicles (FHV) from 2015 to 2016\. Records include fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts. The data set is already partitioned and converted from CSV to Apache Parquet. In the first part of the lab you will be building SQL like queries using Amazon Athena. You will query both data formats directly from Amazon S3 and compare the query performance. In the second part of the lab, you will use Amazon QuickSight to generate visualizations and meaningful insights from the data set in Amazon S3 using Athena tables you create during the first part of the lab. An optional lab is included to incorporate serverless ETL using AWS Glue to optimize query performance. We also give you access to a take-home lab for you to reapply the same design and directly query the same dataset in Amazon S3 from an Amazon Redshift data warehouse using Redshift Spectrum\.
---
## AWS Console
### Verifying your Region in the AWS Management Console
With Amazon EC2, you can place instances in multiple locations. Amazon EC2 locations are composed of regions that contain more than one Availability Zones. Regions are dispersed and located in separate geographic areas (US, EU, etc.). Availability Zones are distinct locations within a region. They are engineered to be isolated from failures in other Availability Zones and to provide inexpensive, low-latency network connectivity to other Availability Zones in the same region.
By launching instances in separate regions, you can design your application to be closer to specific customers or to meet legal or other requirements. By launching instances in separate Availability Zones, you can protect your application from localized regional failures.
### Verify your Region
The AWS region name is always listed in the upper-right corner of the AWS Management Console, in the navigation bar.
* Make a note of the AWS region name, for example, for this lab you will need to choose the **US West (Oregon)** region.
* Use the chart below to determine the region code. Choose **us-west-2 for this lab.**
| Region Name |Region Code|
|---|---|
|US East (Northern Virginia) Region|us-east-1 |
|US West (Oregon) Region|us-west-2|
|Asia Pacific (Tokyo) Region|ap-northeast-1|
|Asia Pacific (Seoul) Region|ap-northeast-2|
|Asia Pacific (Singapore) Region|ap-southeast-1|
|Asia Pacific (Sydney) Region|ap-southeast-2|
|EU (Ireland) Region|eu-west-1|
|EU (Frankfurt) Region|eu-central-1|
---
## Labs
### Prerequisites
[Create a new AWS Account](https://aws.amazon.com/free/) if you don't have one.
|Lab|Name|
|---|----|
|Lab 1|[Serverless Analysis of data in Amazon S3 using Amazon Athena](./Lab1)|
|Lab 2|[Visualization using Amazon QuickSight](./Lab2)|
|Lab 3|[Serverless ETL and Data Discovery using Amazon Glue](./Lab3)|
|Lab 4|[Analysis of data in Amazon S3 using Amazon Redshift Spectrum](./Lab4)|
## AMAZON ATHENA
### What is Amazon Athena?
Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to setup or manage, and you can start analyzing data immediately. You don’t even need to load your data into Athena, it works directly with data stored in S3\. To get started, just log into the Athena Management Console, define your schema, and start querying. Amazon Athena uses Presto with full standard SQL support and works with a variety of standard data formats, including CSV, JSON, ORC, Apache Parquet and Avro. While Amazon Athena is ideal for quick, ad-hoc querying and integrates with Amazon QuickSight for easy visualization, it can also handle complex analysis, including large joins, window functions, and arrays.
### What can I do with Amazon Athena?
Amazon Athena helps you analyze data stored in Amazon S3\. You can use Athena to run ad-hoc queries using ANSI SQL, without the need to aggregate or load the data into Athena. Amazon Athena can process unstructured, semi-structured, and structured data sets. Examples include CSV, JSON, Avro or columnar data formats such as Apache Parquet and Apache ORC. Amazon Athena integrates with Amazon QuickSight for easy visualization. You can also use Amazon Athena to generate reports or to explore data with business intelligence tools or SQL clients, connected via a [JDBC driver](http://docs.aws.amazon.com/athena/latest/ug/connect-with-jdbc.html).
### How do you access Amazon Athena?
Amazon Athena can be accessed via the AWS Management Console and a JDBC driver. You can programmatically run queries, add tables or partitions using the [JDBC driver](http://docs.aws.amazon.com/athena/latest/ug/connect-with-jdbc.html).
### What is the underlying technology behind Amazon Athena?
Amazon Athena uses Presto with full standard SQL support and works with a variety of standard data formats, including CSV, JSON, ORC, Avro, and Parquet. Athena can handle complex analysis, including large joins, window functions, and arrays. Because Amazon Athena uses Amazon S3 as the underlying data store, it is highly available and durable with data redundantly stored across multiple facilities and multiple devices in each facility.
### How do I create tables and schemas for my data on Amazon S3?
Amazon Athena uses Apache Hive DDL to define tables. You can run DDL statements using the Athena console, via a JDBC driver, or using the Athena create table wizard. When you create a new table schema in Amazon Athena the schema is stored in the data catalog and used when executing queries, but it does not modify your data in S3\. Athena uses an approach known as schema-on-read, which allows you to project your schema onto your data at the time you execute a query. This eliminates the need for any data loading or transformation. Learn more about [creating tables](http://docs.aws.amazon.com/athena/latest/ug/creating-tables.html).
### What data formats does Amazon Athena support?
Amazon Athena supports a wide variety of data formats like CSV, TSV, JSON, or Textfiles and also supports open source columnar formats such as Apache ORC and Apache Parquet. Athena also supports compressed data in Snappy, Zlib, LZO, and GZIP formats. By compressing, partitioning, and using columnar formats you can improve performance and reduce your costs.
For more details refer [Amazon Athena FAQ](https://aws.amazon.com/athena/faqs/).
## AMAZON QUICKSIGHT
### What is Amazon QuickSight?
Amazon QuickSight is a fast, cloud-powered business analytics service that makes it easy to build visualizations, perform ad-hoc analysis, and quickly get business insights from your data. Using our cloud-based service you can easily connect to your data, perform advanced analysis, and create stunning visualizations and rich dashboards that can be accessed from any browser or mobile device.
### How is Amazon QuickSight different from traditional Business Intelligence (BI) solutions?
Traditional BI solutions often require teams of data engineers to spend months building complex data models before generating a report. They typically lack interactive ad-hoc data exploration and visualization, limiting users to canned reports and pre-selected queries. Traditional BI solutions also require significant up-front investment in complex and costly hardware and software, and then customers to invest in even more infrastructure to maintain fast query performance as database sizes grow. This cost and complexity makes it difficult for companies to enable analytics solutions across their organizations. Amazon QuickSight has been designed to solve these problems by bringing the scale and flexibility of the AWS Cloud to business analytics. Unlike traditional BI or data discovery solutions, getting started with Amazon QuickSight is simple and fast. When you log in, Amazon QuickSight seamlessly discovers your data sources in AWS services such as Amazon Redshift, Amazon RDS, Amazon Athena, and Amazon Simple Storage Service (Amazon S3). You can connect to any of the data sources discovered by Amazon QuickSight and get insights from this data in minutes. You can choose for Amazon QuickSight to keep the data in SPICE up-to-date as the data in the underlying sources change. SPICE supports rich data discovery and business analytics capabilities to help customers derive valuable insights from their data without worrying about provisioning or managing infrastructure. Organizations pay a low monthly fee for each Amazon QuickSight user, eliminating the cost of long-term licenses. With Amazon QuickSight, organizations can deliver rich business analytics functionality to all employees without incurring a huge cost upfront.
### Which data sources does Amazon QuickSight support?
You can connect to AWS data sources including Amazon RDS, Amazon Aurora, Amazon Redshift, Amazon Athena and Amazon S3\. You can also upload Excel spreadsheets or flat files (CSV, TSV, CLF, and ELF), connect to on-premises databases like SQL Server, MySQL and PostgreSQL and import data from SaaS applications like Salesforce.
For more details refer [Amazon QuickSight FAQ](https://quicksight.aws/resources/faq/).
## Amazon Redshift Spectrum
### What is Amazon Redshift Spectrum?
Amazon Redshift Spectrum is a feature of Amazon Redshift that enables you to run queries against exabytes of unstructured data in Amazon S3, with no loading or ETL required. When you issue a query, it goes to the Amazon Redshift SQL endpoint, which generates and optimizes a query plan. Amazon Redshift determines what data is local and what is in Amazon S3, generates a plan to minimize the amount of Amazon S3 data that needs to be read, requests Redshift Spectrum workers out of a shared resource pool to read and process data from Amazon S3.
Redshift Spectrum scales out to thousands of instances if needed, so queries run quickly regardless of data size. And, you can use the exact same SQL for Amazon S3 data as you do for your Amazon Redshift queries today and connect to the same Amazon Redshift endpoint using your same BI tools. Redshift Spectrum lets you separate storage and compute, allowing you to scale each independently. You can setup as many Amazon Redshift clusters as you need to query your Amazon S3 data lake, providing high availability and limitless concurrency. Redshift Spectrum gives you the freedom to store your data where you want, in the format you want, and have it available for processing when you need it.
### Can Redshift Spectrum replace Amazon EMR?
No. While Redshift Spectrum is great for running queries against data in Amazon Redshift and S3, it really isn’t a fit for the types of use cases that enterprises typically ask from processing frameworks like Amazon EMR.
Amazon EMR goes far beyond just running SQL queries. Amazon EMR is a managed service that lets you process and analyze extremely large data sets using the latest versions of popular big data processing frameworks, such as Spark, Hadoop, and Presto, on fully customizable clusters. With Amazon EMR you can run a wide variety of scale-out data processing tasks for applications such as machine learning, graph analytics, data transformation, streaming data, and virtually anything you can code. You can also use Redshift Spectrum together with EMR. Amazon Redshift Spectrum uses the same approach to store table definitions as Amazon EMR. So, if you’re already using EMR to process a large data store, you can use Redshift Spectrum to query that data right at the same time without interfering with your Amazon EMR jobs.
Query services, data warehouses, and complex data processing frameworks all have their place, and they are used for different things. You just need to choose the right tool for the job.
### When should I use Amazon Athena vs. Redshift Spectrum?
Amazon Athena is the simplest way to give any employee the ability to run ad-hoc queries on data in Amazon S3. Athena is serverless, so there is no infrastructure to setup or manage, and you can start analyzing your data immediately.
If you have frequently accessed data, that needs to be stored in a consistent, highly structured format, then you should use a data warehouse like Amazon Redshift. This gives you the flexibility to store your structured, frequently accessed data in Amazon Redshift, and use Redshift Spectrum to extend your Amazon Redshift queries out to the entire universe of data in your Amazon S3 data lake. This gives you the freedom to store your data where you want, in the format you want, and have it available for processing when you need.
### Can I use Redshift Spectrum to query data that I process using Amazon EMR?
Yes, Redshift Spectrum can support the same Apache Hive Metastore used by Amazon EMR to locate data and table definitions. If you’re using Amazon EMR and have a Hive Metastore already, you just have to configure your Amazon Redshift cluster to use it. You can then start querying that data right away along with your Amazon EMR jobs.
For more details refer [Amazon Redshift Spectrum FAQ](https://aws.amazon.com/redshift/faqs/).
## AWS Glue
### What is AWS Glue?
AWS Glue is a fully-managed, pay-as-you-go, extract, transform, and load (ETL) service that automates the time-consuming steps of data preparation for analytics. AWS Glue automatically discovers and profiles your data via the Glue Data Catalog, recommends and generates ETL code to transform your source data into target schemas, and runs the ETL jobs on a fully managed, scale-out Apache Spark environment to load your data into its destination. It also allows you to setup, orchestrate, and monitor complex data flows.
### What are the main components of AWS Glue?
AWS Glue consists of a Data Catalog which is a central metadata repository, an ETL engine that can automatically generate Python code, and a flexible scheduler that handles dependency resolution, job monitoring, and retries. Together, these automate much of the undifferentiated heavy lifting involved with discovering, categorizing, cleaning, enriching, and moving data, so you can spend more time analyzing your data.
### When should I use AWS Glue?
You should use AWS Glue to discover properties of the data you own, transform it, and prepare it for analytics. Glue can automatically discover both structured and semi-structured data stored in your data lake on Amazon S3, data warehouse in Amazon Redshift, and various databases running on AWS. It provides a unified view of your data via the Glue Data Catalog that is available for ETL, querying and reporting using services like Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum. Glue automatically generates Python code for your ETL jobs that you can further customize using tools you are already familiar with. AWS Glue is serverless, so there are no compute resources to configure and manage.
### What data sources does AWS Glue support?
AWS Glue natively supports data stored in Amazon Aurora, Amazon RDS for MySQL, Amazon RDS for Oracle, Amazon RDS for PostgreSQL, Amazon RDS for SQL Server, Amazon Redshift, and Amazon S3, as well as MySQL, Oracle, Microsoft SQL Server, and PostgreSQL databases in your Virtual Private Cloud (Amazon VPC) running on Amazon EC2. The metadata stored in the AWS Glue Data Catalog can be readily accessed from Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum. You can also write custom PySpark code and import custom libraries in your Glue ETL jobs to access data sources not natively supported by AWS Glue. For more details on importing custom libraries, refer to our documentation.
### What is the AWS Glue Data Catalog?
The AWS Glue Data Catalog is a central repository to store structural and operational metadata for all your data assets. For a given data set, you can store its table definition, physical location, add business relevant attributes, as well as track how this data has changed over time.
The AWS Glue Data Catalog is Apache Hive Metastore compatible and is a drop-in replacement for the Apache Hive Metastore for Big Data applications running on Amazon EMR. For more information on setting up your EMR cluster to use AWS Glue Data Catalog as an Apache Hive Metastore, click here.
The AWS Glue Data Catalog also provides out-of-box integration with Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum. Once you add your table definitions to the Glue Data Catalog, they are available for ETL and also readily available for querying in Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum so that you can have a common view of your data between these services.
### How can I customize the ETL code generated by AWS Glue?
AWS Glue’s ETL script recommendation system generates PySpark code. It leverages Glue’s custom ETL library to simplify access to data sources as well as manage job execution. You can find more details about the library in our documentation. You can write ETL code using AWS Glue’s custom library or write arbitrary Spark code in Python (PySpark code) by using inline editing via the AWS Glue Console script editor, downloading the auto-generated code, and editing it in your own IDE. You can also start with one of the many samples hosted in our Github repository and customize that code.
### When should I use AWS Glue vs. AWS Data Pipeline?
AWS Glue provides a managed ETL service that runs on a serverless Apache Spark environment. This allows you to focus on your ETL job and not worry about configuring and managing the underlying compute resources. AWS Glue takes a data first approach and allows you to focus on the data properties and data manipulation to transform the data to a form where you can derive business insights. It provides an integrated data catalog that makes metadata available for ETL as well as querying via Amazon Athena and Amazon Redshift Spectrum.
AWS Data Pipeline provides a managed orchestration service that gives you greater flexibility in terms of the execution environment, access and control over the compute resources that run your code, as well as the code itself that does data processing. AWS Data Pipeline launches compute resources in your account allowing you direct access to the Amazon EC2 instances or Amazon EMR clusters.
Furthermore, AWS Glue ETL jobs are PySpark based. If your use case requires you to use an engine other than Apache Spark or if you want to run a heterogeneous set of jobs that run on a variety of engines like Hive, Pig, etc., then AWS Data Pipeline would be a better choice.
For more details refer [AWS Glue FAQ](https://aws.amazon.com/glue/faqs/).
## **ADDITIONAL RESOURCES**
### Amazon Athena:
- <https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/>
- <http://docs.aws.amazon.com/athena/latest/ug/convert-to-columnar.html>
### Redshift Spectrum
- https://aws.amazon.com/blogs/big-data/10-best-practices-for-amazon-redshift-spectrum/
### Serverless Analysis Architecture Blogs:
- <https://aws.amazon.com/blogs/big-data/derive-insights-from-iot-in-minutes-using-aws-iot-amazon-kinesis-firehose-amazon-athena-and-amazon-quicksight/>
- <https://aws.amazon.com/blogs/big-data/build-a-serverless-architecture-to-analyze-amazon-cloudfront-access-logs-using-aws-lambda-amazon-athena-and-amazon-kinesis-analytics/>
---
## License
This library is licensed under the Apache 2.0 License.
gitextract_vqvmwjvn/ ├── .github/ │ └── PULL_REQUEST_TEMPLATE.md ├── LICENSE ├── Lab1/ │ └── README.md ├── Lab2/ │ ├── README.md │ └── img/ │ └── README.md ├── Lab3/ │ └── README.md ├── Lab4/ │ ├── README.md │ └── redshiftspectrumglue-lab4.template ├── NOTICE └── README.md
Condensed preview — 10 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (119K chars).
[
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"chars": 169,
"preview": "*Issue #, if available:*\n\n*Description of changes:*\n\n\nBy submitting this pull request, I confirm that my contribution is"
},
{
"path": "LICENSE",
"chars": 11358,
"preview": "\n Apache License\n Version 2.0, January 2004\n "
},
{
"path": "Lab1/README.md",
"chars": 20969,
"preview": "# Lab 1: Serverless Analysis of data in Amazon S3 using Amazon Athena\n\n* [Creating Amazon Athena Database and Table](#cr"
},
{
"path": "Lab2/README.md",
"chars": 16611,
"preview": "# Lab 2: Visualization using Amazon QuickSight\n\n* [Create an Amazon S3 bucket](#create-an-amazon-s3-bucket)\n* [Creating "
},
{
"path": "Lab2/img/README.md",
"chars": 14,
"preview": "adding images\n"
},
{
"path": "Lab3/README.md",
"chars": 18031,
"preview": "# Lab 3: Serverless ETL and Data Discovery using Amazon Glue\n\n* [Create an IAM Role](#create-an-iam-role)\n* [Create an A"
},
{
"path": "Lab4/README.md",
"chars": 15260,
"preview": "# Lab 4: Analysis of data in Amazon S3 using Amazon Redshift Spectrum\n\n* [Deploying Amazon Redshift Cluster](#deploying-"
},
{
"path": "Lab4/redshiftspectrumglue-lab4.template",
"chars": 12339,
"preview": "{\n \"AWSTemplateFormatVersion\" : \"2010-09-09\",\n \"Description\" : \"This CloudFormation template redshift_spectrum_analyti"
},
{
"path": "NOTICE",
"chars": 99,
"preview": "Serverless Data Analytics\nCopyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. \n"
},
{
"path": "README.md",
"chars": 20081,
"preview": "\n# Building an End-to-End Serverless Data Analytics Solution on AWS\n\n\n## Overview\n\nIn this lab, you are going to build a"
}
]
About this extraction
This page contains the full source code of the aws-samples/serverless-data-analytics GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 10 files (112.2 KB), approximately 28.4k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.