[ { "path": "README.md", "content": "# O'Reilly SQL Fundamentals (Online Training)\n\nResources for O'Reilly \"SQL Fundamentals for Data\" online training\n\nRecommended SQLite Environments: \n\n*SQLiteOnline*\nhttp://sqliteonline.com/\n\n*SQLiteStudio*\nhttps://sqlitestudio.pl/features/\n\n# Open Databases Directly on SQliteOnline\n\n#### rexon_metals.db\nhttps://sqliteonline.com/#urldb=https://raw.githubusercontent.com/thomasnield/oreilly_sql_fundamentals_for_data/master/databases/rexon_metals.db\n\n#### weather_stations.db\nhttps://sqliteonline.com/#urldb=https://raw.githubusercontent.com/thomasnield/oreilly_sql_fundamentals_for_data/master/databases/weather_stations.db\n\n# Resources \n\n#### JetBrains Promotion\nGet a free 3 month license for JetBrains developer tools (including PyCharm, DataGrip) using code 3min_datascience: https://www.jetbrains.com/all/\n\n#### Links Mentioned in Class\n\n[Asianometry - The History of SQL](https://youtu.be/z8L202FlmD4?si=ObwEtRU2ND0SNSG1)\n\n[Asianometry - The History of Oracle](https://youtu.be/zSn8il5Mo5s?si=1G4lJ9Umb0xgopHd)\n\n[Big Data is Dead](https://motherduck.com/blog/big-data-is-dead/)\n\n[Silver Bullet Syndrome - Hadi Hariri](https://youtu.be/qamzvLfX-Zo?si=fClyuZepv5zxvjd9)\n\n[Silver Bullet Syndrome Part II - Hadi Hariri](https://youtu.be/WN3CSOai_ZU?si=MkxCn92pfeXoSUD8)\n\n[Silver Bullet Syndrome 2025 - Hadi Hariri](https://youtu.be/TIu6rQVwTkM?si=953rhVOAjiOiR4Gv)\n\n[This couple can’t do the simplest things online because their last name is ‘Null’](https://thenextweb.com/news/last-name-null-is-tough-for-computers)\n\n[SQL Injection Demo](https://youtu.be/6JfS8rHanAQ?si=uPhcKdK9SNTS9g75)\n\n#### Other SQL and SQL-Related Resources\n\n[Using SQL with Python (Video Course)](https://learning.oreilly.com/course/using-sql-in/0642572107871/)\n\n[Other Online Trainings/Books by Thomas Nield](https://www.oreilly.com/pub/au/6658)\n\n[Getting Started with SQL (O'Reilly Book)](https://learning.oreilly.com/library/view/getting-started-with/9781491938607/)\n\n![](https://images-na.ssl-images-amazon.com/images/I/51A7fbsp0EL.jpg)\n\n[Using SQL with Python](https://learning.oreilly.com/course/using-sql-in/0642572107871/)\n\n[SQL Interactive Scenarios on O'Reilly](https://learning.oreilly.com/search/?q=thomas%20nield%20sql&type=cloud-scenario&type=sandbox&type=scenario)\n\n[SQL Fundamentals for Data Video Course on O'Reilly](https://learning.oreilly.com/videos/-/9781491963876/)\n\n[SQL for Analytics Course on O'Reilly](https://learning.oreilly.com/videos/sql-for-analytics/9781492058212/)\n\n[SQL with Python, R, and Java (Code Examples)](https://github.com/thomasnield/oreilly_programming_with_sql/tree/master/code)\n\n[Intermediate SQL (Code Examples)](https://github.com/thomasnield/oreilly_intermediate_sql_for_data/blob/master/intermediate_sql_class_notes.md) \n\n[An Introduction to Regular Expressions](https://learning.oreilly.com/library/view/an-introduction-to/9781492082569/) \n\n[Regular Expressions Interactive Labs](https://learning.oreilly.com/search/?q=thomas%20nield%20regular%20expressions&type=sandbox&type=scenario&type=cloud-scenario&rows=100&language_with_transcripts=en)\n\n" }, { "path": "notes_and_slides/outline.md", "content": "# Day I\r\n1. Understanding Databases (15 minutes)\r\n Definition of database\r\n Relational Databases\r\n Normalization\r\n Lightweight vs Centralized Databases\r\n Exercises\r\n\r\n >0:15\r\n\r\n2. Using SQLite (15 minutes)\r\n Introduction to SQLite\r\n Setting up SQLiteStudio\r\n Importing and Navigating databases\r\n **Q&A/Break (10 minutes)**\r\n\r\n >0:40\r\n\r\n3. SELECT (20 minutes)\r\n Retrieving and Viewing Data with SELECT\r\n Expressions in SELECT statements\r\n Text concatenation\r\n Knowledge Exercise (5 minutes)\r\n\r\n >1:05\r\n\r\n4. WHERE (20 minutes)\r\n Filtering records with WHERE\r\n Using WHERE on numbers\r\n AND, OR, and IN statements\r\n Using WHERE on text\r\n Understanding True/False (boolean) values\r\n Handling NULL\r\n Grouping Conditions\r\n Knowledge Exercise (5 minutes)\r\n **Q&A/Break (10 minutes)**\r\n\r\n >1:40\r\n\r\n5. GROUP BY and ORDER BY (20 minutes)\r\n Grouping Records\r\n Ordering Records\r\n Aggregate Functions\r\n Filtering Aggregates with HAVING\r\n Getting DISTINCT records\r\n Knowledge Exercise (10 minutes)\r\n\r\n >2:10\r\n\r\n6. CASE Statements (20 minutes)\r\n The CASE Statement\r\n Grouping CASE Statements\r\n The \"Zero/Null\" CASE trick\r\n Knowledge Exercise (10 minutes)\r\n **Q&A (10 minutes)**\r\n\r\n >3:00\r\n\r\n# Day II\r\n\r\n7. JOIN (35 minutes)\r\n Stitching Multiple Tables Together\r\n INNER JOIN\r\n LEFT JOIN\r\n Other JOIN types\r\n Joining Multiple Tables\r\n Using GROUP BY with a JOIN\r\n Knowledge Exercise (10 minutes)\r\n **Q&A (15 minutes)**\r\n\r\n >1:00\r\n\r\n8. Database Design (25 minutes)\r\n Decisions in Planning a Database\r\n SQL Injection\r\n The SurgeTech Conference\r\n Turning SurgeTech Entities into Tables\r\n Primary and Foreign Keys\r\n The Final Schema\r\n\r\n> 1:25\r\n\r\n9. Writing Data\r\n Creating and Managing a Database (30 minutes)\r\n Using CREATE TABLE to Build the SurgeTech Database\r\n Setting the Primary/Foreign Keys\r\n Creating Views\r\n Adding data with INSERT\r\n Changing data with UPDATE\r\n Deleting data with DELETE\r\n Truncating and Dropping Tables\r\n Knowledge Exercise (15 minutes)\r\n **Q&A (10 minutes)**\r\n\r\n >2:20\r\n\r\n10. Going Forward and Closing (20 minutes)\r\n Summarize material covered\r\n Point out possible career paths and opportunities using SQL\r\n" }, { "path": "notes_and_slides/sql_fundamentals_notes.html", "content": "Document Name\nSection III - SELECT.md\nMarkdownPreviewToggle Mode\n\n\n\n

Section III - SELECT

\n

3.1: Selecting all columns

\n
SELECT * FROM CUSTOMER;\n
\n

To limit the number of records returned, use a LIMIT. To limit the results to just 2 records:

\n
SELECT * FROM CUSTOMER LIMIT 2;\n
\n

3.2: Selecting specific columns

\n
SELECT CUSTOMER_ID, NAME FROM CUSTOMER;\n
\n

3.3: Expressions

\n

First, select everything from PRODUCT

\n
SELECT * FROM PRODUCT;\n
\n

You can use expressions by declaring a TAXED_PRICE. This is not a column, but rather something that is calculated every time this query is executed.

\n
SELECT PRODUCT_ID,\nDESCRIPTION,\nPRICE,\nPRICE * 1.07 AS TAXED_PRICE\nFROM PRODUCT;\n
\n
\n

In SQliteStudio, you can hit CTRL + SPACE on Windows and Linux to show an autocomplete box with available fields. For Mac, you will need to enable that configuration in preferences.

\n
\n

You can also use aliases to declare an UNTAXED_PRICE column off the PRICE, without any expression.

\n
SELECT PRODUCT_ID,\nDESCRIPTION,\nPRICE as UNTAXED_PRICE,\nPRICE * 1.07 AS TAXED_PRICE\nFROM PRODUCT;\n
\n

SWITCH TO SLIDES FOR MATHEMATICAL OPERATORS

\n

3.4: Using round() Function

\n
SELECT PRODUCT_ID,\nDESCRIPTION,\nPRICE,\nround(PRICE * 1.07, 2) AS TAXED_PRICE\n\nFROM PRODUCT;\n
\n

3.5: Text Concatenation

\n

You can slap a dollar sign to our result using concatenation.

\n
SELECT PRODUCT_ID,\nDESCRIPTION,\nPRICE AS UNTAXED_PRICE,\n'$' || round(PRICE * 1.07, 2) AS TAXED_PRICE\nFROM PRODUCT\n
\n

You can merge text via concatenation. For instance, you can concatenate two fields and put a comma and space , in between.

\n
SELECT NAME,\nCITY || ', ' || STATE AS LOCATION\nFROM CUSTOMER;\n
\n

You can concatenate several fields to create an address.

\n
SELECT NAME,\nSTREET_ADDRESS || ' ' || CITY || ', ' || STATE || ' ' || ZIP AS SHIP_ADDRESS\nFROM CUSTOMER;\n
\n

This works with any data types, like numbers, texts, and dates. Also note that some platforms use concat() function instead of double pipes ||

\n

SWITCH TO SLIDES FOR EXERCISE

\n

3.6: Comments

\n

To make a comments in SQL, use commenting dashes or blocks:

\n
-- this is a comment\n\n/*\nThis is a\nmultiline comment\n*/\n
\n

Section IV- WHERE

\n

4.1: Getting year 2010 records

\n
SELECT * FROM station_data\nWHERE year = 2010;\n
\n

4.2: Getting non-2010 records

\n
SELECT * FROM station_data\nWHERE year != 2010;\n
\n
SELECT * FROM station_data\nWHERE year <> 2010;\n
\n

4.3: Getting records between 2005 and 2010

\n
SELECT * FROM station_data\nWHERE year BETWEEN 2005 AND 2010\n
\n

4.4: Using AND

\n
SELECT * FROM station_data\nWHERE year >= 2005 AND year <= 2010\n
\n

4.5: Exclusive Range

\n

This will get the years between 2005 and 2010, but exclude 2005 and 2010

\n
SELECT * FROM station_data\nWHERE year > 2005 AND year < 2010\n
\n

4.6: Using OR

\n
SELECT * FROM station_data\nWHERE MONTH = 3\nOR MONTH = 6\nOR MONTH = 9\nOR MONTH = 12\n
\n

4.7: Using IN

\n
SELECT * FROM station_data\nWHERE MONTH IN (3,6,9,12);\n
\n

4.8: Using NOT IN

\n
SELECT * FROM station_data\nWHERE MONTH NOT IN (3,6,9,12);\n
\n

4.9: Using Modulus

\n

The modulus will perform division but return the remainder. So a remainder of 0 means the two numbers divide evenly.

\n
SELECT * FROM station_data\nWHERE MONTH % 3 = 0;\n
\n

4.10: Using WHERE on TEXT

\n
SELECT * FROM station_data\nWHERE report_code = '513A63'\n
\n

4.11: Using IN with text

\n
SELECT * FROM station_data\nWHERE report_code IN ('513A63','1F8A7B','EF616A')\n
\n

4.12: Using length() function

\n
SELECT * FROM station_data\nWHERE length(report_code) != 6\n
\n

4.13A: Using LIKE for any characters

\n
SELECT * FROM station_data\nWHERE report_code LIKE 'A%';\n
\n

4.13B: Using Regular Expressions

\n

If you are familiar with regular expressions, you can use those to identify and qualify text patterns.

\n
SELECT * FROM STATION_DATA\nWHERE report_code REGEXP '^A.*$'\n
\n

4.14: Using LIKE for one character

\n
SELECT * FROM station_data\nWHERE report_code LIKE 'B_C%';\n
\n
\n

For LIKE, % is used in a different context than modulus %

\n
\n

4.15: True Booleans 1

\n
SELECT * FROM station_data\nWHERE tornado = 1 AND hail = 1;\n
\n

4.16: True Booleans 2

\n
SELECT * FROM station_data\nWHERE tornado AND hail\n
\n

4.17: False Booleans 1

\n
SELECT * FROM station_data\nWHERE tornado = 0 AND hail = 1;\n
\n

4.18: False Booleans 2

\n
SELECT * FROM station_data\nWHERE NOT tornado AND hail;\n
\n

4.19: Handling NULL

\n

A NULL is an absent value. It is not zero, empty text ’ ', or any value. It is blank.

\n

To check for a null value:

\n
SELECT * FROM station_data\nWHERE snow_depth IS NULL;\n
\n

4.20: Handling NULL in conditions

\n

Nulls will not qualify with any condition that doesn’t explicitly handle it.

\n
SELECT * FROM station_data\nWHERE precipitation <= 0.5;\n
\n

If you want to include nulls, do this:

\n
SELECT * FROM station_data\nWHERE precipitation IS NULL OR precipitation <= 0.5;\n
\n

You can also use a coalesce() function to turn a null value into a default value, if it indeed is null.

\n

This will treat all null values as a 0.

\n
SELECT * FROM station_data\nWHERE coalesce(precipitation, 0) <= 0.5;\n
\n

4.21: Combining AND and OR

\n

Querying for sleet or snow

\n

Problematic. What belongs to the AND and what belongs to the OR?

\n
SELECT * FROM station_data\nWHERE rain = 1 AND temperature <= 32\nOR snow_depth > 0;\n
\n

You must group up the sleet condition in parenthesis so it is treated as one unit.

\n
SELECT * FROM station_data\nWHERE (rain = 1 AND temperature <= 32)\nOR snow_depth > 0;\n
\n

Exercises

\n
-- SELECT all records where TEMPERATURE is between 30 and 50 degrees\n\nSELECT * FROM station_data\nWHERE temperature BETWEEN 30 AND 50;\n-- OR\nSELECT * FROM station_data\nWHERE temperature >= 30 and temperature <= 50;\n
\n
-- SELECT all records where station_pressure is greater than 1000 and a tornado was present\n\nSELECT * FROM STATION_DATA\nWHERE station_pressure > 1000 AND tornado;\n-- OR\nSELECT * FROM STATION_DATA\nWHERE station_pressure > 1000 AND tornado = 1;\n
\n
-- SELECT all records with report codes E6AED7, B950A1, 98DDAD\n\nSELECT * FROM STATION_DATA\nWHERE report_code IN ('E6AED7','B950A1','98DDAD')\n-- OR\nSELECT * FROM STATION_DATA\nWHERE report_code = 'E6AED7'\nOR report_code = 'B950A1'\nOR report_code = '98DDAD'\n
\n
-- SELECT all records WHERE station_pressure is null\n\nSELECT * FROM STATION_DATA\nWHERE station_pressure IS NULL;\n
\n

Section V- GROUP BY and ORDER BY

\n

5.1: Getting a count of records

\n
SELECT count(*) as record_count FROM station_data\n
\n

5.2 Getting a count of records with a condition

\n
SELECT count(*) as record_count FROM station_data\nWHERE tornado = 1\n
\n

5.3 Getting a count by year

\n
SELECT year, count(*) as record_count\nFROM station_data\nWHERE tornado = 1\nGROUP BY year\n
\n

5.4 Getting a count by year, month

\n
SELECT year, month, count(*) as record_count\nFROM station_data\nWHERE tornado = 1\nGROUP BY year, month\n
\n

5.5 Getting a count by year, month with ordinal index

\n
SELECT year, month, count(*) as record_count\nFROM station_data\nWHERE tornado = 1\nGROUP BY 1, 2\n
\n

5.6 Using ORDER BY

\n
SELECT year, month, count(*) as record_count\nFROM station_data\nWHERE tornado = 1\nGROUP BY year, month\nORDER BY year, month\n
\n

5.7 Using ORDER BY with DESC

\n
SELECT year, month, count(*) as record_count\nFROM station_data\nWHERE tornado = 1\nGROUP BY year, month\nORDER BY year DESC, month\n
\n

5.8 Counting non-null values

\n
SELECT COUNT(snow_depth) as recorded_snow_depth_count\nFROM station_data\n
\n

5.9 Average temperature by month since year 2000

\n
SELECT month, AVG(temperature) as avg_temp\nFROM station_data\nWHERE year >= 2000\nGROUP BY month\n
\n

5.10 Average temperature (with rounding) by month since year 2000

\n
SELECT month, round(AVG(temperature),2) as avg_temp\nFROM station_data\nWHERE year >= 2000\nGROUP BY month\n
\n

5.11 Sum of snow depth

\n
SELECT year, SUM(snow_depth) as total_snow\nFROM station_data\nWHERE year >= 2005\nGROUP BY year\n
\n

5.12 Multiple aggregations

\n
SELECT year,\nSUM(snow_depth) as total_snow,\nSUM(precipitation) as total_precipitation,\nMAX(precipitation) as max_precipitation\n\nFROM station_data\nWHERE year >= 2005\nGROUP BY year\n
\n

EXERCISES

\n

Flip to slides

\n

5.13 Using HAVING

\n

You cannot use WHERE on aggregations. This will result in an error.

\n
SELECT year,\nSUM(precipitation) as total_precipitation\nFROM station_data\nWHERE total_precipitation > 30\nGROUP BY year\n
\n

You can however, use HAVING.

\n
SELECT year,\nSUM(precipitation) as total_precipitation\nFROM station_data\nGROUP BY year\nHAVING total_precipitation > 30\n
\n

Note that some platforms like Oracle do not support aliasing in GROUP BY and HAVING.

\n

Therefore you have to rewrite the entire expression each time

\n
SELECT year,\nSUM(precipitation) as total_precipitation\nFROM station_data\nGROUP BY year\nHAVING SUM(precipitation) > 30\n
\n

5.14 Getting Distinct values

\n

You can get DISTINCT values for one or more columns

\n
SELECT DISTINCT station_number FROM station_data\n
\n

You can also get distinct combinations of values for multiple columns

\n
SELECT DISTINCT station_number, year FROM station_data\n
\n

Exercise

\n
-- Find the SUM of precipitation by year when a tornado was present, and sort by year descending.\n\nSELECT year, \nSUM(precipitation) as tornado_precipitation\nFROM station_data\nWHERE tornado = 1\nGROUP BY year\nORDER BY year DESC\n
\n
-- SELECT the year and max snow depth, but only years where the max snow depth is at least 50.\n\nSELECT year, \nmax(snow_depth) AS max_snow_depth\nFROM STATION_DATA\nGROUP BY year\nHAVING max_snow_depth >= 50\n
\n

Section VI - CASE Statements

\n

6.1 Categorizing Wind Speed

\n

You can use a CASE statement to turn a column value into another value based on conditions. For instance, we can turn different wind_speed ranges into HIGH, MODERATE, and LOW categories.

\n
SELECT report_code, year, month, day, wind_speed, \n\nCASE \n    WHEN wind_speed >= 40 THEN 'HIGH'\n    WHEN wind_speed >= 30 THEN 'MODERATE'\n    WHEN wind_speed >= 0 THEN 'LOW'\n    ELSE 'N/A'\nEND AS wind_severity\n\nFROM station_data\n\nORDER by wind_speed DESC\n
\n

6.2 More Efficient Way To Categorize Wind Speed

\n

We can actually omit AND wind_speed < 40 from the previous example because each WHEN/THEN is evaluated from top-to-bottom. The first one it finds to be true is the one it will go with, and stop evaluating subsequent conditions.

\n
SELECT report_code, year, month, day, wind_speed,\n\nCASE\n   WHEN wind_speed >= 40 THEN 'HIGH'\n   WHEN wind_speed >= 30 THEN 'MODERATE'\n   ELSE 'LOW'\nEND as wind_severity\n\nFROM station_data\n
\n

6.3 Using CASE with GROUP BY

\n

We can use GROUP BY in conjunction with a CASE statement to slice data in more ways, such as getting the record count by wind_severity.

\n
SELECT \n\nCASE \n    WHEN wind_speed >= 40 THEN 'HIGH'\n    WHEN wind_speed >= 30 THEN 'MODERATE'\n    WHEN wind_speed >= 0 THEN 'LOW'\n    ELSE 'N/A'\nEND AS wind_severity,\n\nCOUNT(*) AS record_count\n\nFROM station_data\n\nGROUP BY wind_severity\n
\n

Also, some wind_speed values are NULL, so without an ELSE any records that do not meet a condition will turn out to be NULL.

\n
SELECT\n\nCASE\n    WHEN wind_speed >= 40 THEN 'HIGH'\n    WHEN wind_speed >= 30 THEN 'MODERATE'\n    WHEN wind_speed >= 0 THEN 'LOW'\nEND AS wind_severity,\n\nCOUNT(*) AS record_count\n\nFROM station_data\n\nGROUP BY wind_severity\n
\n

6.4 “Zero/Null” Case Trick

\n

There is really no way to create multiple aggregations with different conditions unless you know a trick with the CASE statement. If you want to find two total precipitation, with and without tornado precipitations, for each year and month, you have to do separate queries.

\n

Tornado Precipitation

\n
SELECT year, month,\nSUM(precipitation) as tornado_precipitation\nFROM station_data\nWHERE tornado = 1\nAND year >= 1990\nGROUP BY year, month\n
\n

Non-Tornado Precipitation

\n
SELECT year, month,\nSUM(precipitation) as non_tornado_precipitation\nFROM station_data\nWHERE tornado = 0\nAND year >= 1990\nGROUP BY year, month\n
\n

But you can use a single query using a CASE statement that sets a value to 0 if the condition is not met. That way it will not impact the sum.

\n
SELECT year, month,\nSUM(CASE WHEN tornado = 1 THEN precipitation ELSE 0 END) as tornado_precipitation,\nSUM(CASE WHEN tornado = 0 THEN precipitation ELSE 0 END) as non_tornado_precipitation\n\nFROM station_data\nWHERE year >= 1990\n\nGROUP BY year, month\n
\n

Many folks who are not aware of the zero/null case trick will resort to derived tables (not covered in this class but covered in Advanced SQL for Data Analysis), which adds an unnecessary amount of effort and mess.

\n
SELECT t.year,\nt.month,\nt.tornado_precipitation,\nnon_t.non_tornado_precipitation\n\nFROM (\n    SELECT year, month,\n    SUM(precipitation) as tornado_precipitation\n    FROM station_data\n    WHERE tornado = 1\n    AND year >= 1990\n    GROUP BY year, month\n) t\n\nINNER JOIN\n\n(\n    SELECT year, month,\n    SUM(precipitation) as non_tornado_precipitation\n    FROM station_data\n    WHERE tornado = 0\n    AND year >= 1990\n    GROUP BY year, month\n) non_t\n
\n

6.5 Using Null in a CASE to conditionalize MIN/MAX

\n

Since NULL is ignored in SUM, MIN, MAX, and other aggregate functions, you can use it in a CASE statement to conditionally control whether or not a value should be included in that aggregation.

\n

For instance, we can split up max precipitation when a tornado was present vs not present.

\n
SELECT year,\nMAX(CASE WHEN tornado = 0 THEN precipitation ELSE NULL END) as max_non_tornado_precipitation,\nMAX(CASE WHEN tornado = 1 THEN precipitation ELSE NULL END) as max_tornado_precipitation\nFROM station_data\nWHERE year >= 1990\nGROUP BY year\n
\n

Switch to slides for exercise

\n

Exercise 6.1

\n

SELECT the report_code, year, quarter, and temperature, where a “quarter” is “Q1”, “Q2”, “Q3”, or “Q4” reflecting months 1-3, 4-6, 7-9, and 10-12 respectively.

\n

ANSWER:

\n
SELECT\n\nreport_code,\nyear,\n\nCASE\n    WHEN month BETWEEN 1 and 3 THEN 'Q1'\n    WHEN month BETWEEN 4 and 6 THEN 'Q2'\n    WHEN month BETWEEN 7 and 9 THEN 'Q3'\n    WHEN month BETWEEN 10 and 12 THEN 'Q4'\nEND as quarter,\n\ntemperature\n\nFROM STATION_DATA\n
\n

Exercise 6.2

\n

Get the average temperature by quarter and year, where a “quarter” is “Q1”, “Q2”, “Q3”, or “Q4” reflecting months 1-3, 4-6, 7-9, and 10-12 respectively.

\n

ANSWER

\n
SELECT\nyear,\n\nCASE\n    WHEN month BETWEEN 1 and 3 THEN 'Q1'\n    WHEN month BETWEEN 4 and 6 THEN 'Q2'\n    WHEN month BETWEEN 7 and 9 THEN 'Q3'\n    WHEN month BETWEEN 10 and 12 THEN 'Q4'\nEND as quarter,\n\nAVG(temperature) as avg_temp\n\nFROM STATION_DATA\nGROUP BY 1,2\n
\n

Section VII - JOIN

\n

7.1A INNER JOIN

\n

(Refer to slides Section VII)

\n

View customer address information with each order by joining tables CUSTOMER and CUSTOMER_ORDER.

\n
SELECT ORDER_ID,\nCUSTOMER.CUSTOMER_ID,\nORDER_DATE,\nSHIP_DATE,\nNAME,\nSTREET_ADDRESS,\nCITY,\nSTATE,\nZIP,\nPRODUCT_ID,\nORDER_QTY\n\nFROM CUSTOMER INNER JOIN CUSTOMER_ORDER\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n
\n

Joins allow us to keep stored data normalized and simple, but we can get more descriptive views of our data by using joins.

\n

Notice how two customers are omitted since they don’t have any orders (refer to slides).

\n

7.2B A BAD APPROACH

\n

You may come across a style of joining where commas are used to select the needed tables, and a WHERE defines the join condition as shown below:

\n
SELECT ORDER_ID,\nCUSTOMER.CUSTOMER_ID,\nORDER_DATE,\nSHIP_DATE,\nNAME,\nSTREET_ADDRESS,\nCITY,\nSTATE,\nZIP,\nPRODUCT_ID,\nORDER_QTY\n\nFROM CUSTOMER, CUSTOMER_ORDER\nWHERE CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n
\n

Do not use this approach no matter how much your colleagues use it (and educate them not to use it either). It is extremely inefficient as it will generate a cartesian product across both tables (every possible combination of records between both), and then filter it based on the WHERE. It does not work with LEFT JOIN either, which we will look at shortly.

\n

Using the INNER JOIN with an ON condition avoids the cartesian product and is more efficient. Therefore, always use that approach.

\n

7.2 LEFT OUTER JOIN

\n

To include all customers, regardless of whether they have orders, you can use a left outer join via LEFT JOIN (refer to slides).

\n

If any customers do not have any orders, they will get one record where the CUSTOMER_ORDER fields will be null.

\n
SELECT CUSTOMER.CUSTOMER_ID,\nNAME,\nSTREET_ADDRESS,\nCITY,\nSTATE,\nZIP,\nORDER_DATE,\nSHIP_DATE,\nORDER_ID,\nPRODUCT_ID,\nORDER_QTY\n\nFROM CUSTOMER LEFT JOIN CUSTOMER_ORDER\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n
\n

7.3 Finding Customers with No Orders

\n

With a left outer join, you can filter for NULL values on the CUSTOMER_ORDER table to find customers that have no orders.

\n
SELECT CUSTOMER.CUSTOMER_ID,\nNAME AS CUSTOMER_NAME\n\nFROM CUSTOMER LEFT JOIN CUSTOMER_ORDER\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n\nWHERE ORDER_ID IS NULL\n
\n

You can use a left outer join to find child records with no parent, or parent records with no children (e.g. a CUSTOMER_ORDER with no CUSTOMER, or a CUSTOMER with no CUSTOMER_ORDERs).

\n

7.4 Joining Multiple Tables

\n

Bring in PRODUCT to supply product information for each CUSTOMER_ORDER, on top of CUSTOMER information.

\n
SELECT ORDER_ID,\nCUSTOMER.CUSTOMER_ID,\nNAME AS CUSTOMER_NAME,\nSTREET_ADDRESS,\nCITY,\nSTATE,\nZIP,\nORDER_DATE,\nPRODUCT.PRODUCT_ID,\nDESCRIPTION,\nORDER_QTY\n\nFROM CUSTOMER INNER JOIN CUSTOMER_ORDER\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n\nINNER JOIN PRODUCT\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\n
\n

7.7 Using Expressions with JOINs

\n

You can use expressions combining any fields on any of the joined tables. For instance, we can now get the total revenue for each customer.

\n
SELECT ORDER_ID,\nCUSTOMER.CUSTOMER_ID,\nNAME AS CUSTOMER_NAME,\nSTREET_ADDRESS,\nCITY,\nSTATE,\nZIP,\nORDER_DATE,\nPRODUCT.PRODUCT_ID,\nDESCRIPTION,\nORDER_QTY,\nORDER_QTY * PRICE as REVENUE\n\nFROM CUSTOMER INNER JOIN CUSTOMER_ORDER\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n\nINNER JOIN PRODUCT\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\n
\n

7.6 Using GROUP BY with JOINs

\n

You can use GROUP BY with a join. For instance, you can find the total revenue for each customer by leveraging all three joined tables, and aggregating the REVENUE expression we created earlier.

\n
SELECT\nCUSTOMER.CUSTOMER_ID,\nNAME AS CUSTOMER_NAME,\nsum(ORDER_QTY * PRICE) as TOTAL_REVENUE\n\nFROM CUSTOMER INNER JOIN CUSTOMER_ORDER\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n\nINNER JOIN PRODUCT\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\n\nGROUP BY 1,2\n
\n

To see all customers even if they had no orders, use a LEFT JOIN

\n
SELECT\nCUSTOMER.CUSTOMER_ID,\nNAME AS CUSTOMER_NAME,\nsum(ORDER_QTY * PRICE) as TOTAL_REVENUE\n\nFROM CUSTOMER LEFT JOIN CUSTOMER_ORDER\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n\nLEFT JOIN PRODUCT\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\n\nGROUP BY 1,2\n
\n

You can also use a coalesce() function to turn null sums into zeros.

\n
SELECT\nCUSTOMER.CUSTOMER_ID,\nNAME AS CUSTOMER_NAME,\ncoalesce(sum(ORDER_QTY * PRICE), 0) as TOTAL_REVENUE\n\nFROM CUSTOMER LEFT JOIN CUSTOMER_ORDER\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\n\nLEFT JOIN PRODUCT\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\n\nGROUP BY 1,2\n
\n

Exercise

\n
/*\nSELECT the ORDER_ID, ORDER_DATE, and DESCRIPTION (from PRODUCT)\n(hint, you will need to INNER JOIN CUSTOMER_ORDER and PRODUCT)\n*/\nSELECT ORDER_ID, ORDER_DATE, DESCRIPTION\n\nFROM CUSTOMER_ORDER INNER JOIN PRODUCT\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\n
\n
/*\nFind the total revenue by product. Include the fields PRODUCT_ID, DESCRIPTION, and then the TOTAL_REVENUE.\n(Hint: you will need to join CUSTOMER_ORDER and PRODUCT. Then do a GROUP BY)\n*/\nSELECT PRODUCT.PRODUCT_ID,\nDESCRIPTION,\nCOALESCE(SUM (ORDER_QTY * PRICE), 0) AS TOTAL_REVENUE\n\nFROM PRODUCT LEFT JOIN CUSTOMER_ORDER\nON PRODUCT.PRODUCT_ID = CUSTOMER_ORDER.PRODUCT_ID\nGROUP BY 1, 2\n
\n

Section VIII - Database Design

\n

Refer to slides for database design concepts

\n

To view source code for SQL Injection Demo, here is the GitHub page:
\nhttps://github.com/thomasnield/sql-injection-demo

\n

To read about normalized forms (which we do not cover in favor of a more intuitive approach), you can read this article:

\n

http://www.dummies.com/programming/sql/sql-first-second-and-third-normal-forms/

\n

7.1 - Creating a Table

\n

In SQLiteStudio, navigate to Database -> Add a Database and click the green plus icon to create a new database. Choose a location and name it surgetech_conference.db.

\n

Create the COMPANY table. To create a new table, use the SQLiteStudio wizard by right-clicking the surgetech_conference database and selecting Create a table. You can also just execute the following SQL.

\n
CREATE TABLE COMPANY (\n  COMPANY_ID INTEGER PRIMARY KEY AUTOINCREMENT,\n  NAME VARCHAR(30) NOT NULL,\n  DESCRIPTION VARCHAR(60),\n  PRIMARY_CONTACT_ATTENDEE_ID INTEGER NOT NULL,\n  FOREIGN KEY (PRIMARY_CONTACT_ATTENDEE_ID) REFERENCES ATTENDEE(ATTENDEE_ID)\n);\n
\n

After each field declaration, we create “rules” for that field. For example, COMPANY_ID must be an INTEGER, it is a PRIMARY KEY, and it will AUTOINCREMENT to automatically generate a consecutive integer ID for each new record. The NAME field holds text because it is VARCHAR (a variable number of characters), and it is limited to 30 characters and cannot be NULL.

\n

Lastly, we declare any FOREIGN KEY constraints, specifying which field is a FOREIGN KEY and what PRIMARY KEY it references. In this example, PRIMARY_CONTACT_ATTENDEE_ID “references” the ATTENDEE_ID in the ATTENDEE table, and it can only be those values.

\n

7.2 - Creating the other tables

\n

Create the other tables using the SQLiteStudio New table wizard, or just executing the following SQL code.

\n
CREATE TABLE ROOM (\n  ROOM_ID INTEGER PRIMARY KEY AUTOINCREMENT,\n  FLOOR_NUMBER INTEGER NOT NULL,\n  SEAT_CAPACITY INTEGER NOT NULL\n);\n\nCREATE TABLE PRESENTATION (\n  PRESENTATION_ID INTEGER PRIMARY KEY AUTOINCREMENT,\n  BOOKED_COMPANY_ID INTEGER NOT NULL,\n  BOOKED_ROOM_ID INTEGER NOT NULL,\n  START_TIME TIME,\n  END_TIME TIME,\n  FOREIGN KEY (BOOKED_COMPANY_ID) REFERENCES COMPANY(COMPANY_ID)\n  FOREIGN KEY (BOOKED_ROOM_ID) REFERENCES ROOM(ROOM_ID)\n);\n\nCREATE TABLE ATTENDEE (\n   ATTENDEE_ID INTEGER PRIMARY KEY AUTOINCREMENT,\n   FIRST_NAME VARCHAR (30) NOT NULL,\n   LAST_NAME VARCHAR (30) NOT NULL,\n   PHONE INTEGER,\n   EMAIL VARCHAR (30),\n   VIP BOOLEAN DEFAULT (0)\n);\n\nCREATE TABLE PRESENTATION_ATTENDANCE (\n  TICKET_ID INTEGER PRIMARY KEY AUTOINCREMENT,\n  PRESENTATION_ID INTEGER,\n  ATTENDEE_ID INTEGER,\n  FOREIGN KEY (PRESENTATION_ID) REFERENCES PRESENTATION(PRESENTATION_ID)\n  FOREIGN KEY (ATTENDEE_ID) REFERENCES ATTENDEE(ATTENDEE_ID)\n);\n
\n

Creating Views

\n

It is not uncommon to save SELECT queries that are used frequently into a database. These are known as Views and act very similarly to tables. You can essentially save a SELECT query and work with it just like a table.

\n

For instance, say we wanted to save this SQL query that includes ROOM and COMPANY info with each PRESENTATION record.

\n
SELECT COMPANY.NAME as BOOKED_COMPANY,\nROOM.ROOM_ID as ROOM_NUMBER,\nROOM.FLOOR_NUMBER as FLOOR,\nROOM.SEAT_CAPACITY as SEATS,\nSTART_TIME, END_TIME\n\nFROM PRESENTATION\n\nINNER JOIN COMPANY\nON PRESENTATION.BOOKED_COMPANY_ID = COMPANY.COMPANY_ID\n\nINNER JOIN ROOM\nON PRESENTATION.BOOKED_ROOM_ID = ROOM.ROOM_ID\n
\n

You can save this as a view by right-clicking Views in the database navigator, and then Create a view. You can then paste the SQL as the body and give the view a name, such as PRESENTATION_VW (where “VW” means “View”).

\n

You can also just execute the following SQL syntax: CREATE [view name] AS [a SELECT query]. For this example, this is what it would look like.

\n
CREATE VIEW PRESENTATION_VW AS\n\nSELECT COMPANY.NAME as BOOKED_COMPANY,\nROOM.ROOM_ID as ROOM_NUMBER,\nROOM.FLOOR_NUMBER as FLOOR,\nROOM.SEAT_CAPACITY as SEATS,\nSTART_TIME, END_TIME\n\nFROM PRESENTATION\n\nINNER JOIN COMPANY\nON PRESENTATION.BOOKED_COMPANY_ID = COMPANY.COMPANY_ID\n\nINNER JOIN ROOM\nON PRESENTATION.BOOKED_ROOM_ID = ROOM.ROOM_ID\n
\n

You will then see the PRESENTATION_VW in your database navigator, and you can query it just like a table.

\n
SELECT * FROM PRESENTATION_VW\nWHERE SEATS >= 30\n
\n

Obviously, there is no data yet so you will not get any results. But there will be once you populate data into this database.

\n

Section IX - Writing Data

\n

In this section, we will learn how to write, modify, and delete data in a database.

\n

9.1 Using INSERT

\n

To create a new record in a table, use the INSERT command and supply the values for the needed columns.

\n

Put yourself into the ATTENDEE table.

\n
INSERT INTO ATTENDEE (FIRST_NAME, LAST_NAME)\nVALUES ('Thomas','Nield')\n
\n

Notice above that we declare the table we are writing to, which is ATTENDEE. Then we declare the columns we are supplying values for (FIRST_NAME, LAST_NAME), followed by the values for this new record ('Thomas','Nield').

\n

Notice we did not have to supply a value for ATTENDEE_ID as we have set it in the previous section to generate its own value. PHONE, EMAIL, and VIP fields have default values or are nullable, and therefore optional.

\n

9.2 Multiple INSERT records

\n

You can insert multiple rows in an INSERT. This will add three people to the ATTENDEE table.

\n
INSERT INTO ATTENDEE (FIRST_NAME, LAST_NAME, PHONE, EMAIL, VIP)\nVALUES ('Jon', 'Skeeter', 4802185842,'john.skeeter@rex.net', 1),\n  ('Sam','Scala', 2156783401,'sam.scala@gmail.com', 0),\n  ('Brittany','Fisher', 5932857296,'brittany.fisher@outlook.com', 0)\n
\n

9.3 Testing the foreign keys

\n

Let’s test our design and make sure our primary/foreign keys are working.

\n

Try to INSERT a COMPANY with a PRIMARY_CONTACT_ATTENDEE_ID that does not exist in the ATTENDEE table.

\n
INSERT INTO COMPANY (NAME, DESCRIPTION, PRIMARY_CONTACT_ATTENDEE_ID)\nVALUES ('RexApp Solutions','A mobile app delivery service', 5)\n
\n

Currently, there is no ATTENDEE with an ATTENDEE_ID of 5, this should error out which is good. It means we kept bad data out.

\n

If you use an ATTENDEE_ID value that does exist and supply it as a PRIMARY_CONTACT_ATTENDEE_ID, we should be good to go.

\n
INSERT INTO COMPANY (NAME, DESCRIPTION, PRIMARY_CONTACT_ATTENDEE_ID)\nVALUES ('RexApp Solutions', 'A mobile app delivery service', 3)\n
\n

9.3 DELETE records

\n

The DELETE command is dangerously simple. To delete records from both the COMPANY and ATTENDEE tables, execute the following SQL commands.

\n
DELETE FROM COMPANY;\nDELETE FROM ATTENDEE;\n
\n

Note that the COMPANY table has a foreign key relationship with the ATTENDEE table. Therefore we will have to delete records from COMPANY first before it allows us to delete data from ATTENDEE. Otherwise we will get a “FOREIGN KEY constraint failed effort” due to the COMPANY record we just added which is tied to the ATTENDEE with the ATTENDEE_ID of 3.

\n

You can also use a WHERE to only delete records that meet a conditional. To delete all ATTENDEE records with no PHONE or EMAIL, you can run this command.

\n
DELETE FROM ATTENDEE\nWHERE PHONE IS NULL AND EMAIL IS NULL\n
\n

A good practice is to use a SELECT * in place of the DELETE first. That way you can get a preview of what records will be deleted with that WHERE condition.

\n
SELECT * FROM ATTENDEE\nWHERE PHONE IS NULL AND EMAIL IS NULL\n
\n

UPDATE records

\n

Say we wanted to change the phone number for the ATTENDEE with the ATTENDEE_ID value of 3, which is Sam Scala. We can do this with an UPDATE statement.

\n
UPDATE ATTENDEE SET PHONE = 4802735872\nWHERE ATTENDEE_ID = 3\n
\n

Using a WHERE is important, otherwise it will update all records with the specified SET assignment. This can be handy if you wanted to say, make all EMAIL values uppercase.

\n
UPDATE ATTENDEE SET EMAIL = UPPER(EMAIL)\n
\n

9.4 Dropping Tables

\n

If you want to delete a table, it also is dangerously simple. Be very careful and sure before you delete any table, because it will remove it permanently.

\n
DROP TABLE MY_UNWANTED_TABLE\n
\n

9.5 Transactions

\n

Transactions are helpful when you want a series of writes to succeed.

\n

Below, we execute two successful write operations within a transaction.

\n
BEGIN TRANSACTION;\n\nINSERT INTO ROOM (FLOOR_NUMBER, SEAT_CAPACITY) VALUES (9, 80);\nINSERT INTO ROOM (FLOOR_NUMBER, SEAT_CAPACITY) VALUES (10, 110);\n\nEND TRANSACTION;\n
\n

But if we ever encountered a failure with our write operations, we can call ROLLBACK instead of END TRANSACTION to go back to the database state when BEGIN TRANSACTION was called.

\n

Below, we have a failed operation due to a broken INSERT.

\n
BEGIN TRANSACTION;\n\nINSERT INTO ROOM (FLOOR_NUMBER, SEAT_CAPACITY) VALUES (12, 210);\nINSERT INTO ROOM (FLOOR_NUMBER, SEAT_CAPACITY) VALUES (13); --failure\n
\n

So we can call ROLLBACK to “rewind” to the database state when BEGIN TRANSACTION was called.

\n
ROLLBACK;\n
\n

9.6 Creating Indexes

\n

You can create an index on a certain column to speed up SELECT performance, such as the EMAIL column on the ATTENDEE table.

\n
CREATE INDEX email_index ON ATTENDEE(EMAIL);\n
\n

You can also create an index for a column that has unique values, and it will make a special optimization for that case.

\n
CREATE UNIQUE INDEX email_index ON ATTENDEE(EMAIL);\n
\n

To remove an index, use the DROP command.

\n
DROP INDEX email_index;\n
\n

9.7 Working with Dates and Times

\n

Use the ISO ‘yyyy-mm-dd’ syntax with strings to treat them as dates easily.

\n

Keep in mind much of this functionality is proprietary to SQLite. Make sure you learn the date and time functionality for your specific database platform.

\n
SELECT * FROM CUSTOMER_ORDER\nWHERE SHIP_DATE < '2015-05-21'\n
\n

To get today’s date:

\n
SELECT DATE('now')\n
\n

To shift a date:

\n
SELECT DATE('now','-1 day')\nSELECT DATE('2015-12-07','+3 month','-1 day')\n
\n

To work with times, use hh:mm:ss format.

\n
SELECT '16:31' < '08:31'\n
\n

To get today’s GMT time:

\n
SELECT TIME('now')\n
\n

To shift a time:

\n
SELECT TIME('16:31','+1 minute')\n
\n

To merge a date and time, use a DateTime type.

\n
SELECT '2015-12-13 16:04:11'\nSELECT DATETIME('2015-12-13 16:04:11','-1 day','+3 hour')\n
\n

To format dates and times a certain way:

\n
SELECT strftime('%d-%m-%Y', 'now')\n
\n

Refer to SQLite documentation
\nhttp://www.sqlite.org/lang_datefunc.html

\n

Another helpful tutorial on using dates and times with SQLite.
\nhttps://www.tutorialspoint.com/sqlite/sqlite_date_time.htm

" }, { "path": "notes_and_slides/sql_fundamentals_notes.md", "content": "# Section III - SELECT\r\n\r\n### 3.1: Selecting all columns\r\n\r\n```sql\r\nSELECT * FROM CUSTOMER;\r\n```\r\n\r\n\r\nTo limit the number of records returned, use a LIMIT. To limit the results to just 2 records:\r\n\r\n```sql\r\nSELECT * FROM CUSTOMER LIMIT 2;\r\n```\r\n\r\n### 3.2: Selecting specific columns\r\n\r\n```sql\r\nSELECT CUSTOMER_ID, NAME FROM CUSTOMER;\r\n```\r\n\r\n### 3.3: Expressions\r\n\r\nFirst, select everything from `PRODUCT`\r\n\r\n```sql\r\nSELECT * FROM PRODUCT;\r\n```\r\n\r\nYou can use expressions by declaring a `TAXED_PRICE`. This is not a column, but rather something that is calculated every time this query is executed.\r\n\r\n```sql\r\nSELECT PRODUCT_ID,\r\nDESCRIPTION,\r\nPRICE,\r\nPRICE * 1.07 AS TAXED_PRICE\r\nFROM PRODUCT;\r\n```\r\n\r\n> In SQliteStudio, you can hit CTRL + SPACE on Windows and Linux to show an autocomplete box with available fields. For Mac, you will need to enable that configuration in preferences.\r\n\r\nYou can also use aliases to declare an `UNTAXED_PRICE` column off the `PRICE`, without any expression.\r\n\r\n```sql\r\nSELECT PRODUCT_ID,\r\nDESCRIPTION,\r\nPRICE as UNTAXED_PRICE,\r\nPRICE * 1.07 AS TAXED_PRICE\r\nFROM PRODUCT;\r\n```\r\n\r\n**SWITCH TO SLIDES** FOR MATHEMATICAL OPERATORS\r\n\r\n### 3.4: Using `round()` Function\r\n\r\n```sql\r\nSELECT PRODUCT_ID,\r\nDESCRIPTION,\r\nPRICE,\r\nround(PRICE * 1.07, 2) AS TAXED_PRICE\r\n\r\nFROM PRODUCT;\r\n```\r\n\r\n### 3.5: Text Concatenation\r\n\r\nYou can slap a dollar sign to our result using concatenation.\r\n\r\n```sql\r\nSELECT PRODUCT_ID,\r\nDESCRIPTION,\r\nPRICE AS UNTAXED_PRICE,\r\n'$' || round(PRICE * 1.07, 2) AS TAXED_PRICE\r\nFROM PRODUCT\r\n```\r\n\r\nYou can merge text via concatenation. For instance, you can concatenate two fields and put a comma and space ` ,` in between.\r\n\r\n```sql\r\nSELECT NAME,\r\nCITY || ', ' || STATE AS LOCATION\r\nFROM CUSTOMER;\r\n```\r\n\r\nYou can concatenate several fields to create an address.\r\n\r\n```sql\r\nSELECT NAME,\r\nSTREET_ADDRESS || ' ' || CITY || ', ' || STATE || ' ' || ZIP AS SHIP_ADDRESS\r\nFROM CUSTOMER;\r\n```\r\n\r\nThis works with any data types, like numbers, texts, and dates. Also note that some platforms use `concat()` function instead of double pipes `||`\r\n\r\n**SWITCH TO SLIDES** FOR EXERCISE\r\n\r\n\r\n## 3.6: Comments\r\n\r\nTo make a comments in SQL, use commenting dashes or blocks:\r\n\r\n```sql\r\n-- this is a comment\r\n\r\n/*\r\nThis is a\r\nmultiline comment\r\n*/\r\n```\r\n\r\n## Section IV- WHERE\r\n\r\n### 4.1: Getting year 2010 records\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE year = 2010;\r\n```\r\n\r\n### 4.2: Getting non-2010 records\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE year != 2010;\r\n```\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE year <> 2010;\r\n```\r\n\r\n### 4.3: Getting records between 2005 and 2010\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE year BETWEEN 2005 AND 2010\r\n```\r\n\r\n### 4.4: Using `AND`\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE year >= 2005 AND year <= 2010\r\n```\r\n\r\n### 4.5: Exclusive Range\r\n\r\nThis will get the years between 2005 and 2010, but exclude 2005 and 2010\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE year > 2005 AND year < 2010\r\n```\r\n\r\n### 4.6: Using `OR`\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE MONTH = 3\r\nOR MONTH = 6\r\nOR MONTH = 9\r\nOR MONTH = 12\r\n```\r\n\r\n### 4.7: Using `IN`\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE MONTH IN (3,6,9,12);\r\n```\r\n\r\n### 4.8: Using `NOT IN`\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE MONTH NOT IN (3,6,9,12);\r\n```\r\n\r\n### 4.9: Using Modulus\r\n\r\nThe modulus will perform division but return the remainder. So a remainder of 0 means the two numbers divide evenly.\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE MONTH % 3 = 0;\r\n```\r\n\r\n### 4.10: Using `WHERE` on TEXT\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE report_code = '513A63'\r\n```\r\n\r\n### 4.11: Using `IN` with text\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE report_code IN ('513A63','1F8A7B','EF616A')\r\n```\r\n\r\n### 4.12: Using `length()` function\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE length(report_code) != 6\r\n```\r\n\r\n### 4.13A: Using `LIKE` for any characters\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE report_code LIKE 'A%';\r\n```\r\n\r\n### 4.13B: Using Regular Expressions\r\n\r\n\r\nIf you are familiar with regular expressions, you can use those to identify and qualify text patterns.\r\n\r\n```sql\r\nSELECT * FROM STATION_DATA\r\nWHERE report_code REGEXP '^A.*$'\r\n```\r\n\r\n### 4.14: Using `LIKE` for one character\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE report_code LIKE 'B_C%';\r\n```\r\n\r\n>For `LIKE`, `%` is used in a different context than modulus `%`\r\n\r\n### 4.15: True Booleans 1\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE tornado = 1 AND hail = 1;\r\n```\r\n\r\n### 4.16: True Booleans 2\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE tornado AND hail\r\n```\r\n\r\n### 4.17: False Booleans 1\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE tornado = 0 AND hail = 1;\r\n```\r\n\r\n### 4.18: False Booleans 2\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE NOT tornado AND hail;\r\n```\r\n\r\n### 4.19: Handling `NULL`\r\n\r\nA `NULL` is an absent value. It is not zero, empty text ' ', or any value. It is blank.\r\n\r\nTo check for a null value:\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE snow_depth IS NULL;\r\n```\r\n\r\n\r\n### 4.20: Handling `NULL` in conditions\r\n\r\nNulls will not qualify with any condition that doesn't explicitly handle it.\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE precipitation <= 0.5;\r\n```\r\n\r\nIf you want to include nulls, do this:\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE precipitation IS NULL OR precipitation <= 0.5;\r\n```\r\n\r\nYou can also use a `coalesce()` function to turn a null value into a default value, if it indeed is null.\r\n\r\nThis will treat all null values as a 0.\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE coalesce(precipitation, 0) <= 0.5;\r\n```\r\n\r\n### 4.21: Combining `AND` and `OR`\r\n\r\nQuerying for sleet or snow\r\n\r\nProblematic. What belongs to the `AND` and what belongs to the `OR`?\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE rain = 1 AND temperature <= 32\r\nOR snow_depth > 0;\r\n```\r\n\r\nYou must group up the sleet condition in parenthesis so it is treated as one unit.\r\n\r\n```sql\r\nSELECT * FROM station_data\r\nWHERE (rain = 1 AND temperature <= 32)\r\nOR snow_depth > 0;\r\n```\r\n### Exercises\r\n\r\n```sql\r\n-- SELECT all records where TEMPERATURE is between 30 and 50 degrees\r\n\r\nSELECT * FROM station_data\r\nWHERE temperature BETWEEN 30 AND 50;\r\n-- OR\r\nSELECT * FROM station_data\r\nWHERE temperature >= 30 and temperature <= 50;\r\n```\r\n\r\n```sql\r\n-- SELECT all records where station_pressure is greater than 1000 and a tornado was present\r\n\r\nSELECT * FROM STATION_DATA\r\nWHERE station_pressure > 1000 AND tornado;\r\n-- OR\r\nSELECT * FROM STATION_DATA\r\nWHERE station_pressure > 1000 AND tornado = 1;\r\n```\r\n\r\n```sql\r\n-- SELECT all records with report codes E6AED7, B950A1, 98DDAD\r\n\r\nSELECT * FROM STATION_DATA\r\nWHERE report_code IN ('E6AED7','B950A1','98DDAD')\r\n-- OR\r\nSELECT * FROM STATION_DATA\r\nWHERE report_code = 'E6AED7'\r\nOR report_code = 'B950A1'\r\nOR report_code = '98DDAD'\r\n```\r\n\r\n```sql\r\n-- SELECT all records WHERE station_pressure is null\r\n\r\nSELECT * FROM STATION_DATA\r\nWHERE station_pressure IS NULL;\r\n```\r\n\r\n# Section V- GROUP BY and ORDER BY\r\n\r\n\r\n### 5.1: Getting a count of records\r\n\r\n```sql\r\nSELECT count(*) as record_count FROM station_data\r\n```\r\n\r\n### 5.2 Getting a count of records with a condition\r\n\r\n```sql\r\nSELECT count(*) as record_count FROM station_data\r\nWHERE tornado = 1\r\n```\r\n\r\n### 5.3 Getting a count by year\r\n\r\n```sql\r\nSELECT year, count(*) as record_count\r\nFROM station_data\r\nWHERE tornado = 1\r\nGROUP BY year\r\n```\r\n\r\n### 5.4 Getting a count by year, month\r\n\r\n```sql\r\nSELECT year, month, count(*) as record_count\r\nFROM station_data\r\nWHERE tornado = 1\r\nGROUP BY year, month\r\n```\r\n\r\n### 5.5 Getting a count by year, month with ordinal index\r\n\r\n```sql\r\nSELECT year, month, count(*) as record_count\r\nFROM station_data\r\nWHERE tornado = 1\r\nGROUP BY 1, 2\r\n```\r\n\r\n### 5.6 Using ORDER BY\r\n\r\n```sql\r\nSELECT year, month, count(*) as record_count\r\nFROM station_data\r\nWHERE tornado = 1\r\nGROUP BY year, month\r\nORDER BY year, month\r\n```\r\n\r\n### 5.7 Using ORDER BY with DESC\r\n\r\n```sql\r\nSELECT year, month, count(*) as record_count\r\nFROM station_data\r\nWHERE tornado = 1\r\nGROUP BY year, month\r\nORDER BY year DESC, month\r\n```\r\n\r\n### 5.8 Counting non-null values\r\n\r\n```sql\r\nSELECT COUNT(snow_depth) as recorded_snow_depth_count\r\nFROM station_data\r\n```\r\n\r\n### 5.9 Average temperature by month since year 2000\r\n\r\n```sql\r\nSELECT month, AVG(temperature) as avg_temp\r\nFROM station_data\r\nWHERE year >= 2000\r\nGROUP BY month\r\n```\r\n\r\n### 5.10 Average temperature (with rounding) by month since year 2000\r\n\r\n\r\n```sql\r\nSELECT month, round(AVG(temperature),2) as avg_temp\r\nFROM station_data\r\nWHERE year >= 2000\r\nGROUP BY month\r\n```\r\n\r\n### 5.11 Sum of snow depth\r\n\r\n```sql\r\nSELECT year, SUM(snow_depth) as total_snow\r\nFROM station_data\r\nWHERE year >= 2005\r\nGROUP BY year\r\n```\r\n\r\n### 5.12 Multiple aggregations\r\n\r\n```sql\r\nSELECT year,\r\nSUM(snow_depth) as total_snow,\r\nSUM(precipitation) as total_precipitation,\r\nMAX(precipitation) as max_precipitation\r\n\r\nFROM station_data\r\nWHERE year >= 2005\r\nGROUP BY year\r\n```\r\n\r\n### EXERCISES\r\nFlip to slides\r\n\r\n\r\n### 5.13 Using HAVING\r\n\r\nYou cannot use WHERE on aggregations. This will result in an error.\r\n\r\n```sql\r\nSELECT year,\r\nSUM(precipitation) as total_precipitation\r\nFROM station_data\r\nWHERE total_precipitation > 30\r\nGROUP BY year\r\n```\r\n\r\nYou can however, use HAVING.\r\n\r\n```sql\r\nSELECT year,\r\nSUM(precipitation) as total_precipitation\r\nFROM station_data\r\nGROUP BY year\r\nHAVING total_precipitation > 30\r\n```\r\n\r\nNote that some platforms like Oracle do not support aliasing in GROUP BY and HAVING.\r\n\r\nTherefore you have to rewrite the entire expression each time\r\n\r\n```sql\r\nSELECT year,\r\nSUM(precipitation) as total_precipitation\r\nFROM station_data\r\nGROUP BY year\r\nHAVING SUM(precipitation) > 30\r\n```\r\n\r\n\r\n### 5.14 Getting Distinct values\r\n\r\nYou can get DISTINCT values for one or more columns\r\n\r\n```sql\r\nSELECT DISTINCT station_number FROM station_data\r\n```\r\n\r\nYou can also get distinct combinations of values for multiple columns\r\n\r\n```sql\r\nSELECT DISTINCT station_number, year FROM station_data\r\n```\r\n\r\n### Exercise\r\n\r\n```sql\r\n-- Find the SUM of precipitation by year when a tornado was present, and sort by year descending.\r\n\r\nSELECT year, \r\nSUM(precipitation) as tornado_precipitation\r\nFROM station_data\r\nWHERE tornado = 1\r\nGROUP BY year\r\nORDER BY year DESC\r\n```\r\n\r\n```sql\r\n-- SELECT the year and max snow depth, but only years where the max snow depth is at least 50.\r\n\r\nSELECT year, \r\nmax(snow_depth) AS max_snow_depth\r\nFROM STATION_DATA\r\nGROUP BY year\r\nHAVING max_snow_depth >= 50\r\n```\r\n\r\n# Section VI - CASE Statements\r\n\r\n### 6.1 Categorizing Wind Speed\r\n\r\nYou can use a `CASE` statement to turn a column value into another value based on conditions. For instance, we can turn different `wind_speed` ranges into `HIGH`, `MODERATE`, and `LOW` categories.\r\n\r\n```sql\r\nSELECT report_code, year, month, day, wind_speed, \r\n\r\nCASE \r\n WHEN wind_speed >= 40 THEN 'HIGH'\r\n WHEN wind_speed >= 30 THEN 'MODERATE'\r\n WHEN wind_speed >= 0 THEN 'LOW'\r\n ELSE 'N/A'\r\nEND AS wind_severity\r\n\r\nFROM station_data\r\n\r\nORDER by wind_speed DESC\r\n```\r\n\r\n### 6.2 More Efficient Way To Categorize Wind Speed\r\n\r\nWe can actually omit `AND wind_speed < 40` from the previous example because each `WHEN`/`THEN` is evaluated from top-to-bottom. The first one it finds to be true is the one it will go with, and stop evaluating subsequent conditions.\r\n\r\n```sql\r\nSELECT report_code, year, month, day, wind_speed,\r\n\r\nCASE\r\n WHEN wind_speed >= 40 THEN 'HIGH'\r\n WHEN wind_speed >= 30 THEN 'MODERATE'\r\n ELSE 'LOW'\r\nEND as wind_severity\r\n\r\nFROM station_data\r\n```\r\n\r\n### 6.3 Using CASE with GROUP BY\r\n\r\nWe can use `GROUP BY` in conjunction with a `CASE` statement to slice data in more ways, such as getting the record count by `wind_severity`.\r\n\r\n```sql\r\nSELECT \r\n\r\nCASE \r\n WHEN wind_speed >= 40 THEN 'HIGH'\r\n WHEN wind_speed >= 30 THEN 'MODERATE'\r\n WHEN wind_speed >= 0 THEN 'LOW'\r\n ELSE 'N/A'\r\nEND AS wind_severity,\r\n\r\nCOUNT(*) AS record_count\r\n\r\nFROM station_data\r\n\r\nGROUP BY wind_severity\r\n```\r\n\r\nAlso, some wind_speed values are NULL, so without an `ELSE` any records that do not meet a condition will turn out to be NULL. \r\n\r\n```sql \r\nSELECT\r\n\r\nCASE\r\n WHEN wind_speed >= 40 THEN 'HIGH'\r\n WHEN wind_speed >= 30 THEN 'MODERATE'\r\n WHEN wind_speed >= 0 THEN 'LOW'\r\nEND AS wind_severity,\r\n\r\nCOUNT(*) AS record_count\r\n\r\nFROM station_data\r\n\r\nGROUP BY wind_severity\r\n```\r\n\r\n### 6.4 \"Zero/Null\" Case Trick\r\n\r\nThere is really no way to create multiple aggregations with different conditions unless you know a trick with the `CASE` statement. If you want to find two total precipitation, with and without tornado precipitations, for each year and month, you have to do separate queries.\r\n\r\n**Tornado Precipitation**\r\n```sql\r\nSELECT year, month,\r\nSUM(precipitation) as tornado_precipitation\r\nFROM station_data\r\nWHERE tornado = 1\r\nAND year >= 1990\r\nGROUP BY year, month\r\n```\r\n\r\n**Non-Tornado Precipitation**\r\n```sql\r\nSELECT year, month,\r\nSUM(precipitation) as non_tornado_precipitation\r\nFROM station_data\r\nWHERE tornado = 0\r\nAND year >= 1990\r\nGROUP BY year, month\r\n```\r\n\r\nBut you can use a single query using a `CASE` statement that sets a value to 0 if the condition is not met. That way it will not impact the sum.\r\n\r\n```sql\r\nSELECT year, month,\r\nSUM(CASE WHEN tornado = 1 THEN precipitation ELSE 0 END) as tornado_precipitation,\r\nSUM(CASE WHEN tornado = 0 THEN precipitation ELSE 0 END) as non_tornado_precipitation\r\n\r\nFROM station_data\r\nWHERE year >= 1990\r\n\r\nGROUP BY year, month\r\n```\r\n\r\nMany folks who are not aware of the zero/null case trick will resort to derived tables (not covered in this class but covered in _Advanced SQL for Data Analysis_), which adds an unnecessary amount of effort and mess.\r\n\r\n```sql\r\nSELECT t.year,\r\nt.month,\r\nt.tornado_precipitation,\r\nnon_t.non_tornado_precipitation\r\n\r\nFROM (\r\n SELECT year, month,\r\n SUM(precipitation) as tornado_precipitation\r\n FROM station_data\r\n WHERE tornado = 1\r\n AND year >= 1990\r\n GROUP BY year, month\r\n) t\r\n\r\nINNER JOIN\r\n\r\n(\r\n SELECT year, month,\r\n SUM(precipitation) as non_tornado_precipitation\r\n FROM station_data\r\n WHERE tornado = 0\r\n AND year >= 1990\r\n GROUP BY year, month\r\n) non_t\r\n```\r\n\r\n\r\n### 6.5 Using Null in a CASE to conditionalize MIN/MAX\r\n\r\nSince `NULL` is ignored in SUM, MIN, MAX, and other aggregate functions, you can use it in a `CASE` statement to conditionally control whether or not a value should be included in that aggregation.\r\n\r\nFor instance, we can split up max precipitation when a tornado was present vs not present.\r\n\r\n```sql\r\nSELECT year,\r\nMAX(CASE WHEN tornado = 0 THEN precipitation ELSE NULL END) as max_non_tornado_precipitation,\r\nMAX(CASE WHEN tornado = 1 THEN precipitation ELSE NULL END) as max_tornado_precipitation\r\nFROM station_data\r\nWHERE year >= 1990\r\nGROUP BY year\r\n```\r\n\r\n*Switch to slides for exercise*\r\n\r\n\r\n### Exercise 6.1\r\n\r\nSELECT the report_code, year, quarter, and temperature, where a “quarter” is “Q1”, “Q2”, “Q3”, or “Q4” reflecting months 1-3, 4-6, 7-9, and 10-12 respectively.\r\n\r\n**ANSWER:**\r\n\r\n```sql\r\nSELECT\r\n\r\nreport_code,\r\nyear,\r\n\r\nCASE\r\n WHEN month BETWEEN 1 and 3 THEN 'Q1'\r\n WHEN month BETWEEN 4 and 6 THEN 'Q2'\r\n WHEN month BETWEEN 7 and 9 THEN 'Q3'\r\n WHEN month BETWEEN 10 and 12 THEN 'Q4'\r\nEND as quarter,\r\n\r\ntemperature\r\n\r\nFROM STATION_DATA\r\n```\r\n\r\n### Exercise 6.2\r\n\r\nGet the average temperature by quarter and year, where a “quarter” is “Q1”, “Q2”, “Q3”, or “Q4” reflecting months 1-3, 4-6, 7-9, and 10-12 respectively.\r\n\r\n**ANSWER**\r\n\r\n```sql\r\nSELECT\r\nyear,\r\n\r\nCASE\r\n WHEN month BETWEEN 1 and 3 THEN 'Q1'\r\n WHEN month BETWEEN 4 and 6 THEN 'Q2'\r\n WHEN month BETWEEN 7 and 9 THEN 'Q3'\r\n WHEN month BETWEEN 10 and 12 THEN 'Q4'\r\nEND as quarter,\r\n\r\nAVG(temperature) as avg_temp\r\n\r\nFROM STATION_DATA\r\nGROUP BY 1,2\r\n```\r\n\r\n\r\n# Section VII - JOIN\r\n\r\n### 7.1A INNER JOIN\r\n\r\n(Refer to slides Section VII)\r\n\r\nView customer address information with each order by joining tables `CUSTOMER` and `CUSTOMER_ORDER`.\r\n\r\n```sql\r\nSELECT ORDER_ID,\r\nCUSTOMER.CUSTOMER_ID,\r\nORDER_DATE,\r\nSHIP_DATE,\r\nNAME,\r\nSTREET_ADDRESS,\r\nCITY,\r\nSTATE,\r\nZIP,\r\nPRODUCT_ID,\r\nORDER_QTY\r\n\r\nFROM CUSTOMER INNER JOIN CUSTOMER_ORDER\r\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n```\r\n\r\nJoins allow us to keep stored data normalized and simple, but we can get more descriptive views of our data by using joins.\r\n\r\nNotice how two customers are omitted since they don't have any orders (refer to slides).\r\n\r\n\r\n### 7.2B A BAD APPROACH\r\n\r\nYou may come across a style of joining where commas are used to select the needed tables, and a `WHERE` defines the join condition as shown below:\r\n\r\n```sql\r\nSELECT ORDER_ID,\r\nCUSTOMER.CUSTOMER_ID,\r\nORDER_DATE,\r\nSHIP_DATE,\r\nNAME,\r\nSTREET_ADDRESS,\r\nCITY,\r\nSTATE,\r\nZIP,\r\nPRODUCT_ID,\r\nORDER_QTY\r\n\r\nFROM CUSTOMER, CUSTOMER_ORDER\r\nWHERE CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n```\r\n\r\nDo not use this approach no matter how much your colleagues use it (and educate them not to use it either). It is extremely inefficient as it will generate a cartesian product across both tables (every possible combination of records between both), and then filter it based on the WHERE. It does not work with `LEFT JOIN` either, which we will look at shortly.\r\n\r\nUsing the `INNER JOIN` with an `ON` condition avoids the cartesian product and is more efficient. Therefore, always use that approach.\r\n\r\n### 7.2 LEFT OUTER JOIN\r\n\r\nTo include all customers, regardless of whether they have orders, you can use a left outer join via `LEFT JOIN` (refer to slides).\r\n\r\nIf any customers do not have any orders, they will get one record where the `CUSTOMER_ORDER` fields will be null.\r\n\r\n```sql\r\nSELECT CUSTOMER.CUSTOMER_ID,\r\nNAME,\r\nSTREET_ADDRESS,\r\nCITY,\r\nSTATE,\r\nZIP,\r\nORDER_DATE,\r\nSHIP_DATE,\r\nORDER_ID,\r\nPRODUCT_ID,\r\nORDER_QTY\r\n\r\nFROM CUSTOMER LEFT JOIN CUSTOMER_ORDER\r\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n```\r\n\r\n\r\n## 7.3 Finding Customers with No Orders\r\n\r\nWith a left outer join, you can filter for NULL values on the `CUSTOMER_ORDER` table to find customers that have no orders.\r\n\r\n```sql\r\nSELECT CUSTOMER.CUSTOMER_ID,\r\nNAME AS CUSTOMER_NAME\r\n\r\nFROM CUSTOMER LEFT JOIN CUSTOMER_ORDER\r\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n\r\nWHERE ORDER_ID IS NULL\r\n```\r\n\r\nYou can use a left outer join to find child records with no parent, or parent records with no children (e.g. a `CUSTOMER_ORDER` with no `CUSTOMER`, or a `CUSTOMER` with no `CUSTOMER_ORDER`s).\r\n\r\n\r\n## 7.4 Joining Multiple Tables\r\n\r\nBring in `PRODUCT` to supply product information for each `CUSTOMER_ORDER`, on top of `CUSTOMER` information.\r\n\r\n```sql\r\nSELECT ORDER_ID,\r\nCUSTOMER.CUSTOMER_ID,\r\nNAME AS CUSTOMER_NAME,\r\nSTREET_ADDRESS,\r\nCITY,\r\nSTATE,\r\nZIP,\r\nORDER_DATE,\r\nPRODUCT.PRODUCT_ID,\r\nDESCRIPTION,\r\nORDER_QTY\r\n\r\nFROM CUSTOMER INNER JOIN CUSTOMER_ORDER\r\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n\r\nINNER JOIN PRODUCT\r\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\r\n```\r\n\r\n## 7.7 Using Expressions with JOINs\r\n\r\nYou can use expressions combining any fields on any of the joined tables. For instance, we can now get the total revenue for each customer.\r\n\r\n```sql\r\nSELECT ORDER_ID,\r\nCUSTOMER.CUSTOMER_ID,\r\nNAME AS CUSTOMER_NAME,\r\nSTREET_ADDRESS,\r\nCITY,\r\nSTATE,\r\nZIP,\r\nORDER_DATE,\r\nPRODUCT.PRODUCT_ID,\r\nDESCRIPTION,\r\nORDER_QTY,\r\nORDER_QTY * PRICE as REVENUE\r\n\r\nFROM CUSTOMER INNER JOIN CUSTOMER_ORDER\r\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n\r\nINNER JOIN PRODUCT\r\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\r\n```\r\n\r\n\r\n## 7.6 Using GROUP BY with JOINs\r\n\r\nYou can use `GROUP BY` with a join. For instance, you can find the total revenue for each customer by leveraging all three joined tables, and aggregating the `REVENUE` expression we created earlier.\r\n\r\n```sql\r\nSELECT\r\nCUSTOMER.CUSTOMER_ID,\r\nNAME AS CUSTOMER_NAME,\r\nsum(ORDER_QTY * PRICE) as TOTAL_REVENUE\r\n\r\nFROM CUSTOMER INNER JOIN CUSTOMER_ORDER\r\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n\r\nINNER JOIN PRODUCT\r\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\r\n\r\nGROUP BY 1,2\r\n```\r\n\r\nTo see all customers even if they had no orders, use a `LEFT JOIN`\r\n\r\n```sql\r\nSELECT\r\nCUSTOMER.CUSTOMER_ID,\r\nNAME AS CUSTOMER_NAME,\r\nsum(ORDER_QTY * PRICE) as TOTAL_REVENUE\r\n\r\nFROM CUSTOMER LEFT JOIN CUSTOMER_ORDER\r\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n\r\nLEFT JOIN PRODUCT\r\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\r\n\r\nGROUP BY 1,2\r\n```\r\n\r\nYou can also use a `coalesce()` function to turn null sums into zeros.\r\n\r\n```sql\r\nSELECT\r\nCUSTOMER.CUSTOMER_ID,\r\nNAME AS CUSTOMER_NAME,\r\ncoalesce(sum(ORDER_QTY * PRICE), 0) as TOTAL_REVENUE\r\n\r\nFROM CUSTOMER LEFT JOIN CUSTOMER_ORDER\r\nON CUSTOMER.CUSTOMER_ID = CUSTOMER_ORDER.CUSTOMER_ID\r\n\r\nLEFT JOIN PRODUCT\r\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\r\n\r\nGROUP BY 1,2\r\n```\r\n\r\n\r\n## Exercise\r\n\r\n```sql\r\n/*\r\nSELECT the ORDER_ID, ORDER_DATE, and DESCRIPTION (from PRODUCT)\r\n(hint, you will need to INNER JOIN CUSTOMER_ORDER and PRODUCT)\r\n*/\r\nSELECT ORDER_ID, ORDER_DATE, DESCRIPTION\r\n\r\nFROM CUSTOMER_ORDER INNER JOIN PRODUCT\r\nON CUSTOMER_ORDER.PRODUCT_ID = PRODUCT.PRODUCT_ID\r\n```\r\n\r\n```sql\r\n/*\r\nFind the total revenue by product. Include the fields PRODUCT_ID, DESCRIPTION, and then the TOTAL_REVENUE.\r\n(Hint: you will need to join CUSTOMER_ORDER and PRODUCT. Then do a GROUP BY)\r\n*/\r\nSELECT PRODUCT.PRODUCT_ID,\r\nDESCRIPTION,\r\nCOALESCE(SUM (ORDER_QTY * PRICE), 0) AS TOTAL_REVENUE\r\n\r\nFROM PRODUCT LEFT JOIN CUSTOMER_ORDER\r\nON PRODUCT.PRODUCT_ID = CUSTOMER_ORDER.PRODUCT_ID\r\nGROUP BY 1, 2\r\n```\r\n\r\n\r\n# Section VIII - Database Design\r\n\r\nRefer to slides for database design concepts\r\n\r\nTo view source code for SQL Injection Demo, here is the GitHub page:\r\nhttps://github.com/thomasnield/sql-injection-demo\r\n\r\n\r\nTo read about normalized forms (which we do not cover in favor of a more intuitive approach), you can read this article:\r\n\r\nhttp://www.dummies.com/programming/sql/sql-first-second-and-third-normal-forms/\r\n\r\n\r\n## 7.1 - Creating a Table\r\n\r\nIn SQLiteStudio, navigate to *Database* -> *Add a Database* and click the green plus icon to create a new database. Choose a location and name it `surgetech_conference.db`.\r\n\r\nCreate the `COMPANY` table. To create a new table, use the SQLiteStudio wizard by right-clicking the `surgetech_conference` database and selecting `Create a table`. You can also just execute the following SQL.\r\n\r\n```sql\r\nCREATE TABLE COMPANY (\r\n COMPANY_ID INTEGER PRIMARY KEY AUTOINCREMENT,\r\n NAME VARCHAR(30) NOT NULL,\r\n DESCRIPTION VARCHAR(60),\r\n PRIMARY_CONTACT_ATTENDEE_ID INTEGER NOT NULL,\r\n FOREIGN KEY (PRIMARY_CONTACT_ATTENDEE_ID) REFERENCES ATTENDEE(ATTENDEE_ID)\r\n);\r\n```\r\n\r\nAfter each field declaration, we create \"rules\" for that field. For example, `COMPANY_ID` must be an `INTEGER`, it is a `PRIMARY KEY`, and it will `AUTOINCREMENT` to automatically generate a consecutive integer ID for each new record. The `NAME` field holds text because it is `VARCHAR` (a variable number of characters), and it is limited to 30 characters and cannot be `NULL`.\r\n\r\nLastly, we declare any `FOREIGN KEY` constraints, specifying which field is a `FOREIGN KEY` and what `PRIMARY KEY` it references. In this example, `PRIMARY_CONTACT_ATTENDEE_ID` \"references\" the `ATTENDEE_ID` in the `ATTENDEE` table, and it can only be those values.\r\n\r\n## 7.2 - Creating the other tables\r\n\r\nCreate the other tables using the SQLiteStudio *New table* wizard, or just executing the following SQL code.\r\n\r\n```sql\r\nCREATE TABLE ROOM (\r\n ROOM_ID INTEGER PRIMARY KEY AUTOINCREMENT,\r\n FLOOR_NUMBER INTEGER NOT NULL,\r\n SEAT_CAPACITY INTEGER NOT NULL\r\n);\r\n\r\nCREATE TABLE PRESENTATION (\r\n PRESENTATION_ID INTEGER PRIMARY KEY AUTOINCREMENT,\r\n BOOKED_COMPANY_ID INTEGER NOT NULL,\r\n BOOKED_ROOM_ID INTEGER NOT NULL,\r\n START_TIME TIME,\r\n END_TIME TIME,\r\n FOREIGN KEY (BOOKED_COMPANY_ID) REFERENCES COMPANY(COMPANY_ID)\r\n FOREIGN KEY (BOOKED_ROOM_ID) REFERENCES ROOM(ROOM_ID)\r\n);\r\n\r\nCREATE TABLE ATTENDEE (\r\n ATTENDEE_ID INTEGER PRIMARY KEY AUTOINCREMENT,\r\n FIRST_NAME VARCHAR (30) NOT NULL,\r\n LAST_NAME VARCHAR (30) NOT NULL,\r\n PHONE INTEGER,\r\n EMAIL VARCHAR (30),\r\n VIP BOOLEAN DEFAULT (0)\r\n);\r\n\r\nCREATE TABLE PRESENTATION_ATTENDANCE (\r\n TICKET_ID INTEGER PRIMARY KEY AUTOINCREMENT,\r\n PRESENTATION_ID INTEGER,\r\n ATTENDEE_ID INTEGER,\r\n FOREIGN KEY (PRESENTATION_ID) REFERENCES PRESENTATION(PRESENTATION_ID)\r\n FOREIGN KEY (ATTENDEE_ID) REFERENCES ATTENDEE(ATTENDEE_ID)\r\n);\r\n```\r\n\r\n## Creating Views\r\n\r\nIt is not uncommon to save `SELECT` queries that are used frequently into a database. These are known as **Views** and act very similarly to tables. You can essentially save a `SELECT` query and work with it just like a table.\r\n\r\nFor instance, say we wanted to save this SQL query that includes `ROOM` and `COMPANY` info with each `PRESENTATION` record.\r\n\r\n```sql\r\nSELECT COMPANY.NAME as BOOKED_COMPANY,\r\nROOM.ROOM_ID as ROOM_NUMBER,\r\nROOM.FLOOR_NUMBER as FLOOR,\r\nROOM.SEAT_CAPACITY as SEATS,\r\nSTART_TIME, END_TIME\r\n\r\nFROM PRESENTATION\r\n\r\nINNER JOIN COMPANY\r\nON PRESENTATION.BOOKED_COMPANY_ID = COMPANY.COMPANY_ID\r\n\r\nINNER JOIN ROOM\r\nON PRESENTATION.BOOKED_ROOM_ID = ROOM.ROOM_ID\r\n```\r\n\r\nYou can save this as a view by right-clicking *Views* in the database navigator, and then *Create a view*. You can then paste the SQL as the body and give the view a name, such as `PRESENTATION_VW` (where \"VW\" means \"View\").\r\n\r\nYou can also just execute the following SQL syntax: `CREATE [view name] AS [a SELECT query]`. For this example, this is what it would look like.\r\n\r\n```sql\r\nCREATE VIEW PRESENTATION_VW AS\r\n\r\nSELECT COMPANY.NAME as BOOKED_COMPANY,\r\nROOM.ROOM_ID as ROOM_NUMBER,\r\nROOM.FLOOR_NUMBER as FLOOR,\r\nROOM.SEAT_CAPACITY as SEATS,\r\nSTART_TIME, END_TIME\r\n\r\nFROM PRESENTATION\r\n\r\nINNER JOIN COMPANY\r\nON PRESENTATION.BOOKED_COMPANY_ID = COMPANY.COMPANY_ID\r\n\r\nINNER JOIN ROOM\r\nON PRESENTATION.BOOKED_ROOM_ID = ROOM.ROOM_ID\r\n```\r\n\r\nYou will then see the `PRESENTATION_VW` in your database navigator, and you can query it just like a table.\r\n\r\n```sql\r\nSELECT * FROM PRESENTATION_VW\r\nWHERE SEATS >= 30\r\n```\r\n\r\nObviously, there is no data yet so you will not get any results. But there will be once you populate data into this database.\r\n\r\n# Section IX - Writing Data\r\n\r\nIn this section, we will learn how to write, modify, and delete data in a database.\r\n\r\n\r\n## 9.1 Using `INSERT`\r\n\r\nTo create a new record in a table, use the `INSERT` command and supply the values for the needed columns.\r\n\r\nPut yourself into the `ATTENDEE` table.\r\n\r\n```sql\r\nINSERT INTO ATTENDEE (FIRST_NAME, LAST_NAME)\r\nVALUES ('Thomas','Nield')\r\n```\r\n\r\nNotice above that we declare the table we are writing to, which is `ATTENDEE`. Then we declare the columns we are supplying values for `(FIRST_NAME, LAST_NAME)`, followed by the values for this new record `('Thomas','Nield')`.\r\n\r\nNotice we did not have to supply a value for `ATTENDEE_ID` as we have set it in the previous section to generate its own value. `PHONE`, `EMAIL`, and `VIP` fields have default values or are nullable, and therefore optional.\r\n\r\n\r\n## 9.2 Multiple `INSERT` records\r\n\r\nYou can insert multiple rows in an `INSERT`. This will add three people to the `ATTENDEE` table.\r\n\r\n```sql\r\nINSERT INTO ATTENDEE (FIRST_NAME, LAST_NAME, PHONE, EMAIL, VIP)\r\nVALUES ('Jon', 'Skeeter', 4802185842,'john.skeeter@rex.net', 1),\r\n ('Sam','Scala', 2156783401,'sam.scala@gmail.com', 0),\r\n ('Brittany','Fisher', 5932857296,'brittany.fisher@outlook.com', 0)\r\n```\r\n\r\n## 9.3 Testing the foreign keys\r\n\r\nLet's test our design and make sure our primary/foreign keys are working.\r\n\r\nTry to `INSERT` a `COMPANY` with a `PRIMARY_CONTACT_ATTENDEE_ID` that does not exist in the `ATTENDEE` table.\r\n\r\n```sql\r\nINSERT INTO COMPANY (NAME, DESCRIPTION, PRIMARY_CONTACT_ATTENDEE_ID)\r\nVALUES ('RexApp Solutions','A mobile app delivery service', 5)\r\n```\r\n\r\nCurrently, there is no `ATTENDEE` with an `ATTENDEE_ID` of 5, this should error out which is good. It means we kept bad data out.\r\n\r\nIf you use an `ATTENDEE_ID` value that does exist and supply it as a `PRIMARY_CONTACT_ATTENDEE_ID`, we should be good to go.\r\n\r\n```sql\r\nINSERT INTO COMPANY (NAME, DESCRIPTION, PRIMARY_CONTACT_ATTENDEE_ID)\r\nVALUES ('RexApp Solutions', 'A mobile app delivery service', 3)\r\n```\r\n\r\n### 9.3 `DELETE` records\r\n\r\nThe `DELETE` command is dangerously simple. To delete records from both the `COMPANY` and `ATTENDEE` tables, execute the following SQL commands.\r\n\r\n```sql\r\nDELETE FROM COMPANY;\r\nDELETE FROM ATTENDEE;\r\n```\r\n\r\nNote that the `COMPANY` table has a foreign key relationship with the `ATTENDEE` table. Therefore we will have to delete records from `COMPANY` first before it allows us to delete data from `ATTENDEE`. Otherwise we will get a \"FOREIGN KEY constraint failed effort\" due to the `COMPANY` record we just added which is tied to the `ATTENDEE` with the `ATTENDEE_ID` of 3.\r\n\r\nYou can also use a `WHERE` to only delete records that meet a conditional. To delete all `ATTENDEE` records with no `PHONE` or `EMAIL`, you can run this command.\r\n\r\n```sql\r\nDELETE FROM ATTENDEE\r\nWHERE PHONE IS NULL AND EMAIL IS NULL\r\n```\r\n\r\nA good practice is to use a `SELECT *` in place of the `DELETE` first. That way you can get a preview of what records will be deleted with that `WHERE` condition.\r\n\r\n\r\n```sql\r\nSELECT * FROM ATTENDEE\r\nWHERE PHONE IS NULL AND EMAIL IS NULL\r\n```\r\n\r\n### `UPDATE` records\r\n\r\nSay we wanted to change the phone number for the `ATTENDEE` with the `ATTENDEE_ID` value of 3, which is Sam Scala. We can do this with an `UPDATE` statement.\r\n\r\n```sql\r\nUPDATE ATTENDEE SET PHONE = 4802735872\r\nWHERE ATTENDEE_ID = 3\r\n```\r\n\r\nUsing a `WHERE` is important, otherwise it will update all records with the specified `SET` assignment. This can be handy if you wanted to say, make all `EMAIL` values uppercase.\r\n\r\n```sql\r\nUPDATE ATTENDEE SET EMAIL = UPPER(EMAIL)\r\n```\r\n\r\n### 9.4 Dropping Tables\r\n\r\nIf you want to delete a table, it also is dangerously simple. Be very careful and sure before you delete any table, because it will remove it permanently.\r\n\r\n```sql\r\nDROP TABLE MY_UNWANTED_TABLE\r\n```\r\n\r\n### 9.5 Transactions\r\n\r\nTransactions are helpful when you want a series of writes to succeed.\r\n\r\n\r\nBelow, we execute two successful write operations within a transaction.\r\n\r\n```sql\r\nBEGIN TRANSACTION;\r\n\r\nINSERT INTO ROOM (FLOOR_NUMBER, SEAT_CAPACITY) VALUES (9, 80);\r\nINSERT INTO ROOM (FLOOR_NUMBER, SEAT_CAPACITY) VALUES (10, 110);\r\n\r\nEND TRANSACTION;\r\n```\r\n\r\nBut if we ever encountered a failure with our write operations, we can call `ROLLBACK` instead of `END TRANSACTION` to go back to the database state when `BEGIN TRANSACTION` was called.\r\n\r\nBelow, we have a failed operation due to a broken `INSERT`.\r\n\r\n```sql\r\nBEGIN TRANSACTION;\r\n\r\nINSERT INTO ROOM (FLOOR_NUMBER, SEAT_CAPACITY) VALUES (12, 210);\r\nINSERT INTO ROOM (FLOOR_NUMBER, SEAT_CAPACITY) VALUES (13); --failure\r\n```\r\n\r\nSo we can call `ROLLBACK` to \"rewind\" to the database state when `BEGIN TRANSACTION` was called.\r\n\r\n```sql\r\nROLLBACK;\r\n```\r\n\r\n### 9.6 Creating Indexes\r\n\r\nYou can create an index on a certain column to speed up SELECT performance, such as the `EMAIL` column on the `ATTENDEE` table.\r\n\r\n```sql\r\nCREATE INDEX email_index ON ATTENDEE(EMAIL);\r\n```\r\n\r\nYou can also create an index for a column that has unique values, and it will make a special optimization for that case.\r\n\r\n```sql\r\nCREATE UNIQUE INDEX email_index ON ATTENDEE(EMAIL);\r\n```\r\n\r\nTo remove an index, use the `DROP` command.\r\n\r\n```sql\r\nDROP INDEX email_index;\r\n```\r\n\r\n\r\n### 9.7 Working with Dates and Times\r\n\r\nUse the ISO 'yyyy-mm-dd' syntax with strings to treat them as dates easily.\r\n\r\nKeep in mind much of this functionality is proprietary to SQLite. Make sure you learn the date and time functionality for your specific database platform.\r\n\r\n```sql\r\nSELECT * FROM CUSTOMER_ORDER\r\nWHERE SHIP_DATE < '2015-05-21'\r\n```\r\n\r\nTo get today's date:\r\n\r\n```sql\r\nSELECT DATE('now')\r\n```\r\n\r\nTo shift a date:\r\n\r\n```sql\r\nSELECT DATE('now','-1 day')\r\nSELECT DATE('2015-12-07','+3 month','-1 day')\r\n```\r\n\r\nTo work with times, use `hh:mm:ss` format.\r\n\r\n```sql\r\nSELECT '16:31' < '08:31'\r\n```\r\n\r\nTo get today's GMT time:\r\n\r\n```sql\r\nSELECT TIME('now')\r\n```\r\n\r\nTo shift a time:\r\n\r\n```sql\r\nSELECT TIME('16:31','+1 minute')\r\n```\r\n\r\nTo merge a date and time, use a DateTime type.\r\n\r\n```sql\r\nSELECT '2015-12-13 16:04:11'\r\nSELECT DATETIME('2015-12-13 16:04:11','-1 day','+3 hour')\r\n```\r\n\r\nTo format dates and times a certain way: \r\n\r\n```sql\r\nSELECT strftime('%d-%m-%Y', 'now')\r\n```\r\n\r\nRefer to SQLite documentation\r\nhttp://www.sqlite.org/lang_datefunc.html\r\n\r\nAnother helpful tutorial on using dates and times with SQLite.\r\nhttps://www.tutorialspoint.com/sqlite/sqlite_date_time.htm\r\n\r\n" } ]