Add Flows to Kestra programmatically
If you prefer to add flows programmatically using Kestra's API, run the following commands: ```bash # Import all flows: assuming username admin@kestra.io and password Admin1234! (adjust to match your username and password) curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/01_hello_world.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/02_python.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/03_getting_started_data_pipeline.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/04_postgres_taxi.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/05_postgres_taxi_scheduled.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/06_gcp_kv.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/07_gcp_setup.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/08_gcp_taxi.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/09_gcp_taxi_scheduled.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/10_chat_without_rag.yaml curl -X POST -u 'admin@kestra.io:Admin1234!' http://localhost:8080/api/v1/flows/import -F fileUpload=@flows/11_chat_with_rag.yaml ```Did you take notes? You can share them here
* [Notes by Alvaro Navas](https://github.com/ziritrion/dataeng-zoomcamp/blob/main/notes/3_data_warehouse.md) * [Isaac Kargar's blog post](https://kargarisaac.github.io/blog/data%20engineering/jupyter/2022/01/30/data-engineering-w3.html) * [Marcos Torregrosa's blog post](https://www.n4gash.com/2023/data-engineering-zoomcamp-semana-3/) * [Notes by Victor Padilha](https://github.com/padilha/de-zoomcamp/tree/master/week3) * [Notes from Xia He-Bleinagel](https://xiahe-bleinagel.com/2023/02/week-3-data-engineering-zoomcamp-notes-data-warehouse-and-bigquery/) * [Bigger picture summary on Data Lakes, Data Warehouses, and tooling](https://medium.com/@verazabeida/zoomcamp-week-4-b8bde661bf98), by Vera * [Notes by froukje](https://github.com/froukje/de-zoomcamp/blob/main/week_3_data_warehouse/notes/notes_week_03.md) * [Notes by Alain Boisvert](https://github.com/boisalai/de-zoomcamp-2023/blob/main/week3.md) * [Notes from Vincenzo Galante](https://binchentso.notion.site/Data-Talks-Club-Data-Engineering-Zoomcamp-8699af8e7ff94ec49e6f9bdec8eb69fd) * [2024 videos transcript week3](https://drive.google.com/drive/folders/1quIiwWO-tJCruqvtlqe_Olw8nvYSmmDJ?usp=sharing) by Maria Fisher * [Notes by Linda](https://github.com/inner-outer-space/de-zoomcamp-2024/blob/main/3a-data-warehouse/readme.md) * [Jonah Oliver's blog post](https://www.jonahboliver.com/blog/de-zc-w3) * [2024 - steps to send data from Mage to GCS + creating external table](https://drive.google.com/file/d/1GIi6xnS4070a8MUlIg-ozITt485_-ePB/view?usp=drive_link) by Maria Fisher * [2024 - mage dataloader script to load the parquet files from a remote URL and push it to Google bucket as parquet file](https://github.com/amohan601/dataengineering-zoomcamp2024/blob/main/week_3_data_warehouse/mage_scripts/green_taxi_2022_v2.py) by Anju Mohan * [2024 - steps to send data from Mage to GCS + creating external table](https://drive.google.com/file/d/1GIi6xnS4070a8MUlIg-ozITt485_-ePB/view?usp=drive_link) by Maria Fisher * [Notes by HongWei](https://github.com/hwchua0209/data-engineering-zoomcamp-submission/blob/main/03-data-warehouse/README.md) * [2025 Notes by Manuel Guerra](https://github.com/ManuelGuerra1987/data-engineering-zoomcamp-notes/blob/main/3_Data-Warehouse/README.md) * [Notes from Horeb SEIDOU](https://spotted-hardhat-eea.notion.site/Week-3-Data-Warehouse-and-BigQuery-17c29780dc4a80c8a226f372543ae388) * [2025 - Notes by Gabi Fonseca](https://github.com/fonsecagabriella/data_engineering/blob/main/03_data_warehouse/00_notes.md) * [2025 Gitbook Notes Tinker0425](https://data-engineering-zoomcamp-2025-t.gitbook.io/tinker0425/module-3/introduction-to-module-3) * [2025 Notes from Daniel Lachner](https://drive.google.com/file/d/105zjtLFi0sRqqFFgdMSCTzfcLPx2rfv4/view?usp=sharing) * [2026 Notes from Catherine Frost](https://docs.google.com/document/d/1j3jeNnBI2fw1nq7JwEauPx2G8FybDfTqmMk7eRu0vSo/edit?tab=t.0) * Add your notes here (above this line)Did you take notes? You can share them here
* [Slides used in previous years](https://docs.google.com/presentation/d/1xSll_jv0T8JF4rYZvLHfkJXYqUjPtThA/edit?usp=sharing&ouid=114544032874539580154&rtpof=true&sd=true) * [Notes by Alvaro Navas](https://github.com/ziritrion/dataeng-zoomcamp/blob/main/notes/4_analytics.md) * [Sandy's DE learning blog](https://learningdataengineering540969211.wordpress.com/2022/02/17/week-4-setting-up-dbt-cloud-with-bigquery/) * [Notes by Victor Padilha](https://github.com/padilha/de-zoomcamp/tree/master/week4) * [Marcos Torregrosa's blog (spanish)](https://www.n4gash.com/2023/data-engineering-zoomcamp-semana-4/) * [Notes by froukje](https://github.com/froukje/de-zoomcamp/blob/main/week_4_analytics_engineering/notes/notes_week_04.md) * [Notes by Alain Boisvert](https://github.com/boisalai/de-zoomcamp-2023/blob/main/week4.md) * [Setting up Prefect with dbt by Vera](https://medium.com/@verazabeida/zoomcamp-week-5-5b6a9d53a3a0) * [Blog by Xia He-Bleinagel](https://xiahe-bleinagel.com/2023/02/week-4-data-engineering-zoomcamp-notes-analytics-engineering-and-dbt/) * [Setting up DBT with BigQuery by Tofag](https://medium.com/@fagbuyit/setting-up-your-dbt-cloud-dej-9-d18e5b7c96ba) * [Blog post by Dewi Oktaviani](https://medium.com/@oktavianidewi/de-zoomcamp-2023-learning-week-4-analytics-engineering-with-dbt-53f781803d3e) * [Notes from Vincenzo Galante](https://binchentso.notion.site/Data-Talks-Club-Data-Engineering-Zoomcamp-8699af8e7ff94ec49e6f9bdec8eb69fd) * [Notes from Balaji](https://github.com/Balajirvp/DE-Zoomcamp/blob/main/Week%204/Data%20Engineering%20Zoomcamp%20Week%204.ipynb) * [Notes by Linda](https://github.com/inner-outer-space/de-zoomcamp-2024/blob/main/4-analytics-engineering/readme.md) * [2024 - Videos transcript week4](https://drive.google.com/drive/folders/1V2sHWOotPEMQTdMT4IMki1fbMPTn3jOP?usp=drive) * [Blog Post](https://www.jonahboliver.com/blog/de-zc-w4) by Jonah Oliver * [2025 Notes by Manuel Guerra](https://github.com/ManuelGuerra1987/data-engineering-zoomcamp-notes/blob/main/4_Analytics-Engineering/README.md) * [2025 Notes by Horeb SEIDOU](https://spotted-hardhat-eea.notion.site/Week-4-Analytics-Engineering-18929780dc4a808692e4e0ee488bf49c?pvs=74) * [2025 Notes by Daniel Lachner](https://github.com/mossdet/dlp_data_eng/blob/main/Notes/04_01_Analytics_Engineering.pdf) * [2026 Notes by Sharad K. Gupta](https://github.com/sharadgupta27/data-engineering/blob/main/Notes/dbt_commands.md) * [Analytical Engineering overview](https://github.com/khanhnguyen7802/DataEngineer101/tree/main/week4-analytics-engineering#readme) * [2026 Notes about dbt](https://github.com/khanhnguyen7802/DataEngineer101/blob/main/week4-analytics-engineering/dbt_installation.md) | [dbt + Duckdb setup using Docker](https://github.com/khanhnguyen7802/DataEngineer101/blob/main/week4-analytics-engineering/dbt_installation.md) by Khanh Nguyen * Add your notes here (above this line)
[](https://www.getdbt.com/)
[](https://cloud.google.com/bigquery)
> [!NOTE]
> This guide assumes you've completed [Module 3: Data Warehouse](https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main/03-data-warehouse) where you:
> - Created a GCP project and enabled the BigQuery API
> - Created a service account with BigQuery permissions
> - Learned how to load data into BigQuery (in the `nytaxi` dataset)
>
> Module 4 uses **different data** than Module 3 (green and yellow taxi data for 2019-2020 instead of yellow-only 2024). You'll load the new data in [Step 1](#load-the-taxi-data) below.
## Step 1: Verify Your BigQuery Setup
Before setting up dbt Cloud, confirm you have the required data and credentials from Module 3.
### Check Your Service Account
You should already have a service account JSON key file from Module 3. Make sure it has these permissions:
- **BigQuery Data Editor**
- **BigQuery Job User**
- **BigQuery User**
If you need to create a new service account or download a new key, follow the instructions below.
### How to Download Service Account JSON Key
If you don't have the JSON key file or need to download a new one:
1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Navigate to **IAM & Admin** > **Service Accounts**
- Or use the search bar and type "Service Accounts"
3. Find your service account in the list
- It should look like: `service-account-name@project-id.iam.gserviceaccount.com`
- If you don't have a service account yet, click **+ CREATE SERVICE ACCOUNT** and:
- Enter a name (e.g., `dbt-bigquery-service-account`)
- Click **CREATE AND CONTINUE**
- Add these roles:
- **BigQuery Admin** (or at minimum: BigQuery Data Editor, BigQuery Job User, BigQuery User)
- Click **CONTINUE** > **DONE**
4. Click on your service account name to open its details
5. Go to the **KEYS** tab
6. Click **ADD KEY** > **Create new key**
7. Select **JSON** as the key type
8. Click **CREATE**
9. The JSON key file will automatically download to your computer
- Save it in a secure location
- **Never commit this file to Git or share it publicly** - it contains credentials to access your GCP resources
The downloaded JSON file will look something like this:
```json
{
"type": "service_account",
"project_id": "your-project-id",
"private_key_id": "...",
"private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-name@project-id.iam.gserviceaccount.com",
...
}
```
You'll use this JSON file in Step 4 to connect dbt Cloud to BigQuery.
### Load the Taxi Data
This module uses **yellow and green taxi data for 2019-2020**, which is different from the data you loaded in Module 3. Using the same approach you learned in Module 3, load the following data into your BigQuery `nytaxi` dataset:
- **Yellow taxi trip records** for all months of 2019 and 2020
- **Green taxi trip records** for all months of 2019 and 2020
> [!IMPORTANT]
> Download the data from the [DataTalksClub NYC TLC Data repository](https://github.com/DataTalksClub/nyc-tlc-data/releases), **not** from the official NYC TLC website. The official site has been retroactively updated over the years, so its data differs from what the homework answers are based on.
After loading, verify your data:
1. Go to [BigQuery Console](https://console.cloud.google.com/bigquery)
2. In the Explorer panel on the left, expand your project
3. You should see the `nytaxi` dataset
4. Expand the `nytaxi` dataset - you should see tables:
- `green_tripdata`
- `yellow_tripdata`
### Note Your Dataset Location
When you created your BigQuery datasets in Module 3, you chose a location (e.g., `US`, `EU`, `us-central1`). You'll need to use the same location when configuring dbt.
**To check your dataset location:**
1. In BigQuery Console, click on the `nytaxi` dataset
2. Look for **Data location** in the dataset details
## Step 2: Sign Up for dbt Platform
dbt Platform is dbt's cloud-based development environment with a web IDE, scheduler, and collaboration features. dbt offers a **free Developer plan**. This should be more than enough to learn dbt and follow the course.
## Step 3: Create a New dbt Project
Now you'll create a fresh dbt project from scratch in dbt Cloud.
1. Navigate to **Account settings** (gear icon in the top-right corner) and click **+ New Project**
2. Enter a project name:
- Project name: `taxi_rides_ny`
3. Click **Continue**
## Step 4: Configure BigQuery Connection
After clicking **Continue** in the previous step, dbt Cloud will prompt you to configure your data warehouse connection.
> [!TIP]
> If you're not automatically taken to the connection setup, you can also configure it from **Account settings** > **Projects** > **taxi_rides_ny** > **Connection**.
### Upload Service Account JSON
1. For the connection type, select **BigQuery**
2. Click **Upload a Service Account JSON file**
3. Select the service account JSON key file from Module 3
4. dbt will automatically extract:
- Your GCP project ID
- Authentication credentials
### Configure Connection Settings
1. **Dataset**: Enter `dbt_prod`
- This is the base schema name where dbt will create datasets
- dbt will organize your models into schemas like:
- `dbt_prod_staging` - for staging models
- `dbt_prod_intermediate` - for intermediate models
- `dbt_prod_marts` - for final analytics tables
2. **Location**: Select the same location as your `nytaxi` dataset from Module 3
- Example: `US`, `EU`, or `us-central1`
- **This must match your nytaxi dataset location**
- You can find this under **Optional Settings** or **Advanced Settings** depending on your UI version
3. **Timeout**: `300` seconds
4. **Maximum Bytes Billed**: (optional)
- Leave blank for unlimited, OR
- Set a limit like `1000000000` (1 GB) to prevent runaway queries
### Test the Connection
1. Click **Test Connection**
2. You should see a success message: "Connection test succeeded"
3. Click **Continue**
## Step 5: Set Up Your Repository
dbt Cloud needs a Git repository to store your project code. You have two options:
- Let dbt Manage the Repository (Recommended for Beginners)
- Connect Your Own GitHub Repository (Recommended for Production)
It doesn't matter which one you prefer for this course.
## Step 6: Verify Your Development Environment
### What Are Environments in dbt?
In dbt, **environments** define different contexts where your data transformations run:
- **Development Environment**: Your personal workspace for building and testing models
- Uses your personal credentials
- Creates temporary schemas with your name (e.g., `dbt_
[](https://www.getdbt.com/)
[](https://duckdb.org/)
>[!NOTE]
>*This guide will explain how to do the setup manually. If you want an additional challenge, try to run this setup using Docker Compose or a Python virtual environment.*
**Important**: All dbt commands must be run from inside the `taxi_rides_ny/` directory. The setup steps below will guide you through:
1. Installing the necessary tools
2. Configuring your connection to DuckDB
3. Loading the NYC taxi data
4. Verifying everything works
## Step 1: Install DuckDB
DuckDB is a fast, in-process SQL database that works great for local analytics workloads. To install DuckDB, follow the instruction on the [official site](https://duckdb.org/docs/installation) for your specific operating system.
> [!TIP]
> *You can install DuckDB in two ways. You can install the CLI or install the client API for your favorite programming language (in the case of Python, you can use `pip install duckdb`). I personally prefer installing the CLI, but either way is fine.*
## Step 2: Install dbt
```bash
pip install dbt-duckdb
```
This installs:
* `dbt-core`: The core dbt framework
* `dbt-duckdb`: The DuckDB adapter for dbt
## Step 3: Configure dbt Profile
Since this repository already contains a dbt project (`taxi_rides_ny/`), you don't need to run `dbt init`. Instead, you need to configure your dbt profile to connect to DuckDB.
### Create or Update `~/.dbt/profiles.yml`
The dbt profile tells dbt how to connect to your database. Create or update the file `~/.dbt/profiles.yml` with the following content:
```yaml
taxi_rides_ny:
target: dev
outputs:
# DuckDB Development profile
dev:
type: duckdb
path: taxi_rides_ny.duckdb
schema: dev
threads: 1
extensions:
- parquet
settings:
memory_limit: '2GB'
preserve_insertion_order: false
# DuckDB Production profile
prod:
type: duckdb
path: taxi_rides_ny.duckdb
schema: prod
threads: 1
extensions:
- parquet
settings:
memory_limit: '2GB'
preserve_insertion_order: false
# Troubleshooting:
# - If you have less than 4GB RAM, try setting memory_limit to '1GB'
# - If you have 16GB+ RAM, you can increase to '4GB' for faster builds
# - Expected build time: 5-10 minutes on most systems
```
## Step 4: Download and Ingest Data
Now that your dbt profile is configured, let's load the taxi data into DuckDB. Navigate to the dbt project directory and run the ingestion script
```python
import duckdb
import requests
from pathlib import Path
BASE_URL = "https://github.com/DataTalksClub/nyc-tlc-data/releases/download"
def download_and_convert_files(taxi_type):
data_dir = Path("data") / taxi_type
data_dir.mkdir(exist_ok=True, parents=True)
for year in [2019, 2020]:
for month in range(1, 13):
parquet_filename = f"{taxi_type}_tripdata_{year}-{month:02d}.parquet"
parquet_filepath = data_dir / parquet_filename
if parquet_filepath.exists():
print(f"Skipping {parquet_filename} (already exists)")
continue
# Download CSV.gz file
csv_gz_filename = f"{taxi_type}_tripdata_{year}-{month:02d}.csv.gz"
csv_gz_filepath = data_dir / csv_gz_filename
response = requests.get(f"{BASE_URL}/{taxi_type}/{csv_gz_filename}", stream=True)
response.raise_for_status()
with open(csv_gz_filepath, 'wb') as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)
print(f"Converting {csv_gz_filename} to Parquet...")
con = duckdb.connect()
con.execute(f"""
COPY (SELECT * FROM read_csv_auto('{csv_gz_filepath}'))
TO '{parquet_filepath}' (FORMAT PARQUET)
""")
con.close()
# Remove the CSV.gz file to save space
csv_gz_filepath.unlink()
print(f"Completed {parquet_filename}")
def update_gitignore():
gitignore_path = Path(".gitignore")
# Read existing content or start with empty string
content = gitignore_path.read_text() if gitignore_path.exists() else ""
# Add data/ if not already present
if 'data/' not in content:
with open(gitignore_path, 'a') as f:
f.write('\n# Data directory\ndata/\n' if content else '# Data directory\ndata/\n')
if __name__ == "__main__":
# Update .gitignore to exclude data directory
update_gitignore()
for taxi_type in ["yellow", "green"]:
download_and_convert_files(taxi_type)
con = duckdb.connect("taxi_rides_ny.duckdb")
con.execute("CREATE SCHEMA IF NOT EXISTS prod")
for taxi_type in ["yellow", "green"]:
con.execute(f"""
CREATE OR REPLACE TABLE prod.{taxi_type}_tripdata AS
SELECT * FROM read_parquet('data/{taxi_type}/*.parquet', union_by_name=true)
""")
con.close()
```
This script downloads yellow and green taxi data from 2019-2020, creates the `prod` schema, and loads the raw data into DuckDB. The download may take several minutes depending on your internet connection.
## Step 5: Test the dbt Connection
Verify dbt can connect to your DuckDB database:
```bash
dbt debug
```
## Step 6: Install dbt Power User Extension (VS Code Users)
If you're using Visual Studio Code, install the **dbt Power User** extension to enhance your dbt development experience.
### What is dbt Power User?
dbt Power User is a VS Code extension that provides:
* SQL syntax highlighting and formatting for dbt models
* Inline column-level lineage visualization
* Auto-completion for dbt models, sources, and macros
* Interactive documentation preview
* Model compilation and execution directly from the editor
### Why Not Use the Official dbt Extension?
dbt Labs released an official VS Code extension called [dbt Extension](https://marketplace.visualstudio.com/items?itemName=dbtLabsInc.dbt) powered by the new dbt Fusion engine. However, this extension **requires dbt Fusion** and does not support dbt Core.
Since we're using **dbt Core** with DuckDB for local development, we need the community-maintained **dbt Power User by AltimateAI** extension instead. This extension:
* Works seamlessly with dbt Core (not just dbt Cloud)
* Supports all dbt adapters, including DuckDB
* Is actively maintained and open source
* Provides a rich feature set for local development
### Installation
1. Open VS Code
2. Go to Extensions (Ctrl+Shift+X / Cmd+Shift+X)
3. Search for "dbt Power User"
4. Install **dbt Power User by AltimateAI** (not the dbt Labs version)
Alternatively, install it from the [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user).
> [!NOTE]
> At this point, your local dbt environment is fully configured and ready to use. The next steps (running models, tests, and building documentation) will be covered in the tutorial videos.
## Additional Resources
* [DuckDB Documentation](https://duckdb.org/docs/)
* [dbt Documentation](https://docs.getdbt.com/)
* [dbt-duckdb Adapter](https://github.com/duckdb/dbt-duckdb)
* [NYC Taxi Data Dictionary](https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf)
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/.gitignore
================================================
# you shouldn't commit these into source control
# these are the default directory names, adjust/add to fit your needs
target/
dbt_packages/
logs/
profiles.yml
.user.yml
# Data files for DuckDB
data/green_tripdata/
data/yellow_tripdata/
data/
*.duckdb
*.duckdb.wal
.duckdb_temp/
# Parquet data files
*.parquet
# Python artifacts
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# Virtual environments
venv/
env/
ENV/
env.bak/
venv.bak/
.venv/
# PyCharm
.idea/
# VS Code
.vscode/
# Jupyter Notebook
.ipynb_checkpoints
*.ipynb
# pyenv
.python-version
# pytest
.pytest_cache/
.coverage
htmlcov/
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# GCP credentials and service account keys
*-key.json
*-keys.json
*key*.json
*credential*.json
*service-account*.json
*serviceaccount*.json
service-account.json
serviceaccount.json
gcp-*.json
google-*.json
# Environment variables
.env
.env.local
.env.*.local
*.env
dbt_internal_packages/
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/dbt_project.yml
================================================
name: 'taxi_rides_ny'
version: '1.0.0'
# Require a specific dbt version for reproducibility
require-dbt-version: [">=1.7.0", "<3.0.0"]
# This setting configures which "profile" dbt uses for this project.
profile: 'taxi_rides_ny'
# These configurations specify where dbt should look for different types of files.
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
clean-targets:
- "target"
- "dbt_packages"
# Project-level variables
vars:
# Date range for dev environment sampling
dev_start_date: '2019-01-01'
dev_end_date: '2019-02-01'
# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models
models:
taxi_rides_ny:
staging:
+materialized: view
intermediate:
+materialized: table
marts:
+materialized: table
flags:
require_generic_test_arguments_property: true
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/macros/get_trip_duration_minutes.sql
================================================
{#
Calculate trip duration in minutes from pickup and dropoff timestamps.
Uses dbts built-in cross-database datediff macro.
This works seamlessly across DuckDB, BigQuery, Snowflake, Redshift, PostgreSQL, etc.
Returns: Trip duration as a numeric value in minutes
#}
{% macro get_trip_duration_minutes(pickup_datetime, dropoff_datetime) %}
{{ dbt.datediff(pickup_datetime, dropoff_datetime, 'minute') }}
{% endmacro %}
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/macros/get_vendor_data.sql
================================================
{#
Macro to generate vendor_name column using Jinja dictionary.
This approach works seamlessly across BigQuery, DuckDB, Snowflake, etc.
by generating a CASE statement at compile time.
Usage: {{ get_vendor_data('vendor_id') }}
Returns: SQL CASE expression that maps vendor_id to vendor_name
#}
{% macro get_vendor_data(vendor_id_column) %}
{% set vendors = {
1: 'Creative Mobile Technologies',
2: 'VeriFone Inc.',
4: 'Unknown/Other'
} %}
case {{ vendor_id_column }}
{% for vendor_id, vendor_name in vendors.items() %}
when {{ vendor_id }} then '{{ vendor_name }}'
{% endfor %}
end
{% endmacro %}
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/macros/macros_properties.yml
================================================
macros:
- name: get_trip_duration_minutes
description: >
Calculates trip duration in minutes from pickup and dropoff timestamps.
This macro is cross-database compatible, supporting both DuckDB and BigQuery.
Returns a numeric value representing the duration in minutes.
arguments:
- name: pickup_datetime
type: timestamp
description: The pickup timestamp
- name: dropoff_datetime
type: timestamp
description: The dropoff timestamp
- name: get_vendor_data
description: >
Generates a CASE statement that maps vendor_id to vendor_name.
This macro is cross-database compatible and generates SQL at compile time using a Jinja dictionary.
Supports vendor IDs: 1 (Creative Mobile Technologies), 2 (VeriFone Inc.), 4 (Unknown/Other).
arguments:
- name: vendor_id_column
type: integer
description: The column name containing the vendor ID
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/macros/safe_cast.sql
================================================
{% macro safe_cast(column, data_type) %}
{% if target.type == 'bigquery' %}
safe_cast({{ column }} as {{ data_type }})
{% else %}
cast({{ column }} as {{ data_type }})
{% endif %}
{% endmacro %}
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/intermediate/int_trips.sql
================================================
-- Enrich and deduplicate trip data
-- Demonstrates enrichment and surrogate key generation
-- Note: Data quality analysis available in analyses/trips_data_quality.sql
with unioned as (
select * from {{ ref('int_trips_unioned') }}
),
payment_types as (
select * from {{ ref('payment_type_lookup') }}
),
cleaned_and_enriched as (
select
-- Generate unique trip identifier (surrogate key pattern)
{{ dbt_utils.generate_surrogate_key(['u.vendor_id', 'u.pickup_datetime', 'u.pickup_location_id', 'u.service_type']) }} as trip_id,
-- Identifiers
u.vendor_id,
u.service_type,
u.rate_code_id,
-- Location IDs
u.pickup_location_id,
u.dropoff_location_id,
-- Timestamps
u.pickup_datetime,
u.dropoff_datetime,
-- Trip details
u.store_and_fwd_flag,
u.passenger_count,
u.trip_distance,
u.trip_type,
-- Payment breakdown
u.fare_amount,
u.extra,
u.mta_tax,
u.tip_amount,
u.tolls_amount,
u.ehail_fee,
u.improvement_surcharge,
u.total_amount,
-- Enrich with payment type description
coalesce(u.payment_type, 0) as payment_type,
coalesce(pt.description, 'Unknown') as payment_type_description
from unioned u
left join payment_types pt
on coalesce(u.payment_type, 0) = pt.payment_type
)
select * from cleaned_and_enriched
-- Deduplicate: if multiple trips match (same vendor, second, location, service), keep first
qualify row_number() over(
partition by vendor_id, pickup_datetime, pickup_location_id, service_type
order by dropoff_datetime
) = 1
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/intermediate/int_trips_unioned.sql
================================================
-- Union green and yellow taxi data into a single dataset
-- Demonstrates how to combine data from multiple sources with slightly different schemas
with green_trips as (
select
vendor_id,
rate_code_id,
pickup_location_id,
dropoff_location_id,
pickup_datetime,
dropoff_datetime,
store_and_fwd_flag,
passenger_count,
trip_distance,
trip_type,
fare_amount,
extra,
mta_tax,
tip_amount,
tolls_amount,
ehail_fee,
improvement_surcharge,
total_amount,
payment_type,
'Green' as service_type
from {{ ref('stg_green_tripdata') }}
),
yellow_trips as (
select
vendor_id,
rate_code_id,
pickup_location_id,
dropoff_location_id,
pickup_datetime,
dropoff_datetime,
store_and_fwd_flag,
passenger_count,
trip_distance,
cast(1 as integer) as trip_type, -- Yellow taxis only do street-hail (code 1)
fare_amount,
extra,
mta_tax,
tip_amount,
tolls_amount,
cast(0 as numeric) as ehail_fee, -- Yellow taxis don't have ehail_fee
improvement_surcharge,
total_amount,
payment_type,
'Yellow' as service_type
from {{ ref('stg_yellow_tripdata') }}
)
select * from green_trips
union all
select * from yellow_trips
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/intermediate/schema.yml
================================================
models:
- name: int_trips_unioned
description: Union of green and yellow taxi trip data with normalized schema
columns:
- name: vendor_id
description: Taxi technology provider ID
- name: rate_code_id
description: Rate code at end of trip (1=Standard, 2=JFK, 3=Newark, 4=Nassau/Westchester, 5=Negotiated, 6=Group)
- name: pickup_location_id
description: TLC Taxi Zone where trip started
- name: dropoff_location_id
description: TLC Taxi Zone where trip ended
- name: pickup_datetime
description: Timestamp when meter was engaged
- name: dropoff_datetime
description: Timestamp when meter was disengaged
- name: store_and_fwd_flag
description: Trip record stored in vehicle memory (Y/N)
- name: passenger_count
description: Number of passengers in the vehicle
- name: trip_distance
description: Trip distance in miles
- name: trip_type
description: Trip type (1=Street-hail, 2=Dispatch)
- name: fare_amount
description: Time and distance fare
- name: extra
description: Miscellaneous extras and surcharges
- name: mta_tax
description: MTA tax
- name: tip_amount
description: Tip amount (credit card only)
- name: tolls_amount
description: Total tolls paid
- name: ehail_fee
description: E-hail service fee
- name: improvement_surcharge
description: Improvement surcharge
- name: total_amount
description: Total amount charged to passenger
- name: payment_type
description: Payment method code
- name: service_type
description: Type of taxi service (Green or Yellow)
- name: int_trips
description: Cleaned, enriched, and deduplicated trip data ready for marts
columns:
- name: trip_id
description: Unique trip identifier (surrogate key)
data_tests:
- unique
- not_null
- name: vendor_id
description: Taxi technology provider ID
data_tests:
- not_null
- name: service_type
description: Type of taxi service (Green or Yellow)
data_tests:
- not_null
- accepted_values:
arguments:
values: ['Green', 'Yellow']
- name: rate_code_id
description: Rate code at end of trip
- name: pickup_location_id
description: TLC Taxi Zone where trip started
- name: dropoff_location_id
description: TLC Taxi Zone where trip ended
- name: pickup_datetime
description: Timestamp when meter was engaged
data_tests:
- not_null
- name: dropoff_datetime
description: Timestamp when meter was disengaged
- name: store_and_fwd_flag
description: Trip record stored in vehicle memory (Y/N)
- name: passenger_count
description: Number of passengers in the vehicle
- name: trip_distance
description: Trip distance in miles
- name: trip_type
description: Trip type (1=Street-hail, 2=Dispatch)
- name: fare_amount
description: Time and distance fare
- name: extra
description: Miscellaneous extras and surcharges
- name: mta_tax
description: MTA tax
- name: tip_amount
description: Tip amount (credit card only)
- name: tolls_amount
description: Total tolls paid
- name: ehail_fee
description: E-hail service fee
- name: improvement_surcharge
description: Improvement surcharge
- name: total_amount
description: Total amount charged to passenger
data_tests:
- not_null
- name: payment_type
description: Payment method code
- name: payment_type_description
description: Human-readable payment method description
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/marts/dim_vendors.sql
================================================
-- Dimension table for taxi technology vendors
-- Small static dimension defining vendor codes and their company names
with trips as (
select * from {{ ref('fct_trips') }}
),
vendors as (
select distinct
vendor_id,
{{ get_vendor_data('vendor_id') }} as vendor_name
from trips
)
select * from vendors
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/marts/dim_zones.sql
================================================
-- Dimension table for NYC taxi zones
-- This is a simple pass-through from the seed file, but having it as a model
-- allows for future enhancements (e.g., adding calculated fields, filtering)
select
locationid as location_id,
borough,
zone,
service_zone
from {{ ref('taxi_zone_lookup') }}
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/marts/fct_trips.sql
================================================
{{
config(
materialized='incremental',
unique_key='trip_id',
incremental_strategy='merge',
on_schema_change='append_new_columns' )
}}
-- Fact table containing all taxi trips enriched with zone information
-- This is a classic star schema design: fact table (trips) joined to dimension table (zones)
-- Materialized incrementally to handle large datasets efficiently
select
-- Trip identifiers
trips.trip_id,
trips.vendor_id,
trips.service_type,
trips.rate_code_id,
-- Location details (enriched with human-readable zone names from dimension)
trips.pickup_location_id,
pz.borough as pickup_borough,
pz.zone as pickup_zone,
trips.dropoff_location_id,
dz.borough as dropoff_borough,
dz.zone as dropoff_zone,
-- Trip timing
trips.pickup_datetime,
trips.dropoff_datetime,
trips.store_and_fwd_flag,
-- Trip metrics
trips.passenger_count,
trips.trip_distance,
trips.trip_type,
{{ get_trip_duration_minutes('trips.pickup_datetime', 'trips.dropoff_datetime') }} as trip_duration_minutes,
-- Payment breakdown
trips.fare_amount,
trips.extra,
trips.mta_tax,
trips.tip_amount,
trips.tolls_amount,
trips.ehail_fee,
trips.improvement_surcharge,
trips.total_amount,
trips.payment_type,
trips.payment_type_description
from {{ ref('int_trips') }} as trips
-- LEFT JOIN preserves all trips even if zone information is missing or unknown
left join {{ ref('dim_zones') }} as pz
on trips.pickup_location_id = pz.location_id
left join {{ ref('dim_zones') }} as dz
on trips.dropoff_location_id = dz.location_id
{% if is_incremental() %}
-- Only process new trips based on pickup datetime
where trips.pickup_datetime > (select max(pickup_datetime) from {{ this }})
{% endif %}
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/marts/reporting/fct_monthly_zone_revenue.sql
================================================
-- Data mart for monthly revenue analysis by pickup zone and service type
-- This aggregation is optimized for business reporting and dashboards
-- Enables analysis of revenue trends across different zones and taxi types
select
-- Grouping dimensions
coalesce(pickup_zone, 'Unknown Zone') as pickup_zone,
{% if target.type == 'bigquery' %}cast(date_trunc(pickup_datetime, month) as date)
{% elif target.type == 'duckdb' %}date_trunc('month', pickup_datetime)
{% endif %} as revenue_month,
service_type,
-- Revenue breakdown (summed by zone, month, and service type)
sum(fare_amount) as revenue_monthly_fare,
sum(extra) as revenue_monthly_extra,
sum(mta_tax) as revenue_monthly_mta_tax,
sum(tip_amount) as revenue_monthly_tip_amount,
sum(tolls_amount) as revenue_monthly_tolls_amount,
sum(ehail_fee) as revenue_monthly_ehail_fee,
sum(improvement_surcharge) as revenue_monthly_improvement_surcharge,
sum(total_amount) as revenue_monthly_total_amount,
-- Additional metrics for operational analysis
count(trip_id) as total_monthly_trips,
avg(passenger_count) as avg_monthly_passenger_count,
avg(trip_distance) as avg_monthly_trip_distance
from {{ ref('fct_trips') }}
group by pickup_zone, revenue_month, service_type
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/marts/reporting/schema.yml
================================================
models:
- name: fct_monthly_zone_revenue
description: Monthly revenue aggregation by pickup zone and service type for business reporting
data_tests:
- dbt_utils.unique_combination_of_columns:
arguments:
combination_of_columns:
- pickup_zone
- revenue_month
- service_type
columns:
- name: pickup_zone
description: Pickup zone where revenue was generated
data_tests:
- not_null
- name: revenue_month
description: Month for revenue aggregation
data_tests:
- not_null
- name: service_type
description: Service type (Green or Yellow)
data_tests:
- not_null
- accepted_values:
arguments:
values: ['Green', 'Yellow']
- name: revenue_monthly_total_amount
description: Monthly sum of total fares
data_tests:
- not_null
- name: total_monthly_trips
description: Count of trips in the month
data_tests:
- not_null
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/marts/schema.yml
================================================
models:
- name: dim_zones
description: Taxi zone dimension table with location details
columns:
- name: location_id
description: Unique identifier for each taxi zone
data_tests:
- unique
- not_null
- name: borough
description: NYC borough name
- name: zone
description: Specific zone name within the borough
- name: service_zone
description: Service zone classification
- name: dim_vendors
description: Taxi technology vendor dimension table
columns:
- name: vendor_id
description: Unique vendor identifier
data_tests:
- unique
- not_null
- name: vendor_name
description: Company name of the vendor
- name: fct_trips
description: Fact table with all taxi trips including trip and payment details
config:
contract:
enforced: true
columns:
- name: trip_id
description: Unique trip identifier
data_type: string
data_tests:
- unique
- not_null
- name: vendor_id
description: Taxi technology provider
data_type: integer
data_tests:
- not_null
- name: service_type
description: Type of taxi service (Green or Yellow)
data_type: string
data_tests:
- accepted_values:
arguments:
values: ['Green', 'Yellow']
- not_null
- name: rate_code_id
description: Final rate code
data_type: integer
- name: pickup_location_id
description: TLC Taxi Zone where trip started
data_type: integer
data_tests:
- relationships:
arguments:
to: ref('dim_zones')
field: location_id
- name: pickup_borough
description: NYC borough where trip started
data_type: string
- name: pickup_zone
description: Specific zone where trip started
data_type: string
- name: dropoff_location_id
description: TLC Taxi Zone where trip ended
data_type: integer
data_tests:
- relationships:
arguments:
to: ref('dim_zones')
field: location_id
- name: dropoff_borough
description: NYC borough where trip ended
data_type: string
- name: dropoff_zone
description: Specific zone where trip ended
data_type: string
- name: pickup_datetime
description: Timestamp when meter was engaged
data_type: timestamp
data_tests:
- not_null
- name: dropoff_datetime
description: Timestamp when meter was disengaged
data_type: timestamp
- name: store_and_fwd_flag
description: Trip record stored in vehicle memory (Y/N)
data_type: string
- name: passenger_count
description: Number of passengers
data_type: integer
- name: trip_distance
description: Trip distance in miles
data_type: numeric
- name: trip_type
description: Trip type (1=Street-hail, 2=Dispatch)
data_type: integer
- name: trip_duration_minutes
description: Trip duration in minutes (calculated using cross-database macro)
data_type: bigint
- name: fare_amount
description: Time and distance fare
data_type: numeric
- name: extra
description: Miscellaneous extras and surcharges
data_type: numeric
- name: mta_tax
description: MTA tax
data_type: numeric
- name: tip_amount
description: Tip amount (credit card only)
data_type: numeric
- name: tolls_amount
description: Total tolls paid
data_type: numeric
- name: ehail_fee
description: E-hail service fee
data_type: numeric
- name: improvement_surcharge
description: Improvement surcharge
data_type: numeric
- name: total_amount
description: Total amount charged
data_type: numeric
data_tests:
- not_null
- name: payment_type
description: Payment method code
data_type: integer
- name: payment_type_description
description: Human-readable payment method description
data_type: string
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/staging/schema.yml
================================================
models:
- name: stg_green_tripdata
description: >
Staging model for green taxi trip data. This model standardizes column names
and data types from the raw green_tripdata source, providing a clean foundation
for downstream transformations.
columns:
- name: vendor_id
description: Taxi technology provider (1 = Creative Mobile Technologies, 2 = VeriFone Inc.)
data_tests:
- not_null
- name: rate_code_id
description: Rate code at end of trip (1=Standard, 2=JFK, 3=Newark, 4=Nassau/Westchester, 5=Negotiated, 6=Group)
- name: pickup_location_id
description: TLC Taxi Zone where the meter was engaged
- name: dropoff_location_id
description: TLC Taxi Zone where the meter was disengaged
- name: pickup_datetime
description: Date and time when the meter was engaged
data_tests:
- not_null
- name: dropoff_datetime
description: Date and time when the meter was disengaged
- name: store_and_fwd_flag
description: Flag indicating if trip record was held in vehicle memory (Y/N)
- name: passenger_count
description: Number of passengers in the vehicle (driver-entered value)
- name: trip_distance
description: Trip distance in miles reported by the taximeter
- name: trip_type
description: Code for trip type (1=Street-hail, 2=Dispatch)
- name: fare_amount
description: Time and distance fare calculated by the meter
- name: extra
description: Miscellaneous extras and surcharges (rush hour, overnight)
- name: mta_tax
description: $0.50 MTA tax automatically triggered based on meter rate
- name: tip_amount
description: Tip amount (credit card tips only, cash tips not included)
- name: tolls_amount
description: Total amount of all tolls paid during the trip
- name: ehail_fee
description: E-hail service fee
- name: improvement_surcharge
description: Improvement surcharge assessed on hailed trips
- name: total_amount
description: Total amount charged to passengers (does not include cash tips)
- name: payment_type
description: Payment method code (1=Credit card, 2=Cash, 3=No charge, 4=Dispute, 5=Unknown, 6=Voided)
- name: stg_yellow_tripdata
description: >
Staging model for yellow taxi trip data. This model standardizes column names
and data types from the raw yellow_tripdata source, providing a clean foundation
for downstream transformations.
columns:
- name: vendor_id
description: Taxi technology provider (1 = Creative Mobile Technologies, 2 = VeriFone Inc.)
data_tests:
- not_null
- name: rate_code_id
description: Rate code at end of trip (1=Standard, 2=JFK, 3=Newark, 4=Nassau/Westchester, 5=Negotiated, 6=Group)
- name: pickup_location_id
description: TLC Taxi Zone where the meter was engaged
- name: dropoff_location_id
description: TLC Taxi Zone where the meter was disengaged
- name: pickup_datetime
description: Date and time when the meter was engaged
data_tests:
- not_null
- name: dropoff_datetime
description: Date and time when the meter was disengaged
- name: store_and_fwd_flag
description: Flag indicating if trip record was held in vehicle memory (Y/N)
- name: passenger_count
description: Number of passengers in the vehicle (driver-entered value)
- name: trip_distance
description: Trip distance in miles reported by the taximeter
- name: fare_amount
description: Time and distance fare calculated by the meter
- name: extra
description: Miscellaneous extras and surcharges (rush hour, overnight)
- name: mta_tax
description: $0.50 MTA tax automatically triggered based on meter rate
- name: tip_amount
description: Tip amount (credit card tips only, cash tips not included)
- name: tolls_amount
description: Total amount of all tolls paid during the trip
- name: improvement_surcharge
description: Improvement surcharge assessed on hailed trips
- name: total_amount
description: Total amount charged to passengers (does not include cash tips)
- name: payment_type
description: Payment method code (1=Credit card, 2=Cash, 3=No charge, 4=Dispute, 5=Unknown, 6=Voided)
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/staging/sources.yml
================================================
sources:
- name: raw
description: Raw taxi trip data from NYC TLC
database: |
{%- if target.type == 'bigquery' -%}
{{ env_var('GCP_PROJECT_ID', 'please-add-your-gcp-project-id-here') }}
{%- else -%}
taxi_rides_ny
{%- endif -%}
schema: |
{%- if target.type == 'bigquery' -%}
nytaxi
{%- else -%}
prod
{%- endif -%}
tables:
- name: green_tripdata
description: Raw green taxi trip records
columns:
- name: vendorid
description: "Taxi technology provider (1 = Creative Mobile Technologies, 2 = VeriFone Inc.) - Note: Raw data may contain nulls, filtered in staging"
- name: lpep_pickup_datetime
description: Date and time when the meter was engaged
- name: lpep_dropoff_datetime
description: Date and time when the meter was disengaged
- name: passenger_count
description: Number of passengers in the vehicle
- name: trip_distance
description: Trip distance in miles
- name: pulocationid
description: TLC Taxi Zone where the meter was engaged
- name: dolocationid
description: TLC Taxi Zone where the meter was disengaged
- name: ratecodeid
description: Rate code (1=Standard, 2=JFK, 3=Newark, 4=Nassau/Westchester, 5=Negotiated, 6=Group)
- name: store_and_fwd_flag
description: Trip record held in vehicle memory (Y/N)
- name: payment_type
description: Payment method (1=Credit card, 2=Cash, 3=No charge, 4=Dispute, 5=Unknown, 6=Voided)
- name: fare_amount
description: Time and distance fare
- name: extra
description: Miscellaneous extras and surcharges
- name: mta_tax
description: MTA tax
- name: tip_amount
description: Tip amount (credit card only)
- name: tolls_amount
description: Total tolls paid
- name: improvement_surcharge
description: Improvement surcharge
- name: total_amount
description: Total amount charged
- name: trip_type
description: Trip type (1=Street-hail, 2=Dispatch)
- name: ehail_fee
description: E-hail fee
config:
loaded_at_field: lpep_pickup_datetime
- name: yellow_tripdata
description: Raw yellow taxi trip records
columns:
- name: vendorid
description: "Taxi technology provider (1 = Creative Mobile Technologies, 2 = VeriFone Inc.) - Note: Raw data may contain nulls, filtered in staging"
- name: tpep_pickup_datetime
description: Date and time when the meter was engaged
- name: tpep_dropoff_datetime
description: Date and time when the meter was disengaged
- name: passenger_count
description: Number of passengers in the vehicle
- name: trip_distance
description: Trip distance in miles
- name: pulocationid
description: TLC Taxi Zone where the meter was engaged
- name: dolocationid
description: TLC Taxi Zone where the meter was disengaged
- name: ratecodeid
description: Rate code (1=Standard, 2=JFK, 3=Newark, 4=Nassau/Westchester, 5=Negotiated, 6=Group)
- name: store_and_fwd_flag
description: Trip record held in vehicle memory (Y/N)
- name: payment_type
description: Payment method (1=Credit card, 2=Cash, 3=No charge, 4=Dispute, 5=Unknown, 6=Voided)
- name: fare_amount
description: Time and distance fare
- name: extra
description: Miscellaneous extras and surcharges
- name: mta_tax
description: MTA tax
- name: tip_amount
description: Tip amount (credit card only)
- name: tolls_amount
description: Total tolls paid
- name: improvement_surcharge
description: Improvement surcharge
- name: total_amount
description: Total amount charged
config:
loaded_at_field: tpep_pickup_datetime
config:
freshness:
warn_after: {count: 24, period: hour}
error_after: {count: 48, period: hour}
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/staging/stg_green_tripdata.sql
================================================
with source as (
select * from {{ source('raw', 'green_tripdata') }}
),
renamed as (
select
-- identifiers
cast(vendorid as integer) as vendor_id,
{{ safe_cast('ratecodeid', 'integer') }} as rate_code_id,
cast(pulocationid as integer) as pickup_location_id,
cast(dolocationid as integer) as dropoff_location_id,
-- timestamps
cast(lpep_pickup_datetime as timestamp) as pickup_datetime, -- lpep = Licensed Passenger Enhancement Program (green taxis)
cast(lpep_dropoff_datetime as timestamp) as dropoff_datetime,
-- trip info
cast(store_and_fwd_flag as string) as store_and_fwd_flag,
cast(passenger_count as integer) as passenger_count,
cast(trip_distance as numeric) as trip_distance,
{{ safe_cast('trip_type', 'integer') }} as trip_type,
-- payment info
cast(fare_amount as numeric) as fare_amount,
cast(extra as numeric) as extra,
cast(mta_tax as numeric) as mta_tax,
cast(tip_amount as numeric) as tip_amount,
cast(tolls_amount as numeric) as tolls_amount,
cast(ehail_fee as numeric) as ehail_fee,
cast(improvement_surcharge as numeric) as improvement_surcharge,
cast(total_amount as numeric) as total_amount,
{{ safe_cast('payment_type', 'integer') }} as payment_type
from source
-- Filter out records with null vendor_id (data quality requirement)
where vendorid is not null
)
select * from renamed
-- Sample records for dev environment using deterministic date filter
{% if target.name == 'dev' %}
where pickup_datetime >= '2019-01-01' and pickup_datetime < '2019-02-01'
{% endif %}
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/models/staging/stg_yellow_tripdata.sql
================================================
with source as (
select * from {{ source('raw', 'yellow_tripdata') }}
),
renamed as (
select
-- identifiers (standardized naming for consistency across yellow/green)
cast(vendorid as integer) as vendor_id,
cast(ratecodeid as integer) as rate_code_id,
cast(pulocationid as integer) as pickup_location_id,
cast(dolocationid as integer) as dropoff_location_id,
-- timestamps (standardized naming)
cast(tpep_pickup_datetime as timestamp) as pickup_datetime, -- tpep = Taxicab Passenger Enhancement Program (yellow taxis)
cast(tpep_dropoff_datetime as timestamp) as dropoff_datetime,
-- trip info
cast(store_and_fwd_flag as string) as store_and_fwd_flag,
cast(passenger_count as integer) as passenger_count,
cast(trip_distance as numeric) as trip_distance,
-- payment info
cast(fare_amount as numeric) as fare_amount,
cast(extra as numeric) as extra,
cast(mta_tax as numeric) as mta_tax,
cast(tip_amount as numeric) as tip_amount,
cast(tolls_amount as numeric) as tolls_amount,
cast(improvement_surcharge as numeric) as improvement_surcharge,
cast(total_amount as numeric) as total_amount,
cast(payment_type as integer) as payment_type
from source
-- Filter out records with null vendor_id (data quality requirement)
where vendorid is not null
)
select * from renamed
-- Sample records for dev environment using deterministic date filter
{% if target.name == 'dev' %}
where pickup_datetime >= '2019-01-01' and pickup_datetime < '2019-02-01'
{% endif %}
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/package-lock.yml
================================================
packages:
- name: dbt_utils
package: dbt-labs/dbt_utils
version: 1.3.3
- name: codegen
package: dbt-labs/codegen
version: 0.14.0
sha1_hash: 01f31e0d658d76121f50e62b998342ebf138df11
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/packages.yml
================================================
packages:
- package: dbt-labs/dbt_utils
version: [">=1.3.0", "<2.0.0"]
- package: dbt-labs/codegen
version: [">=0.14.0", "<1.0.0"]
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/seeds/seeds_properties.yml
================================================
seeds:
- name: taxi_zone_lookup
description: >
Taxi Zones roughly based on NYC Department of City Planning's Neighborhood
Tabulation Areas (NTAs) and are meant to approximate neighborhoods, so you can see which
neighborhood a passenger was picked up in, and which neighborhood they were dropped off in.
Includes associated service_zone (EWR, Boro Zone, Yellow Zone)
- name: payment_type_lookup
description: >
Payment type reference data mapping payment type codes to their descriptions.
Used as a dimension table for payment method analysis.
columns:
- name: payment_type
description: Numeric code for payment type
data_tests:
- unique
- not_null
- name: description
description: Human-readable description of payment method
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/snapshots/.gitkeep
================================================
================================================
FILE: 04-analytics-engineering/taxi_rides_ny/tests/.gitkeep
================================================
================================================
FILE: 05-data-platforms/README.md
================================================
# Module 5: Data Platforms
## Overview
In this module, you'll learn about data platforms - tools that help you manage the entire data lifecycle from ingestion to analytics.
We'll use [Bruin](https://getbruin.com/) as an example of a data platform. Bruin puts multiple tools under one platform:
- Data ingestion (extract from sources to your warehouse)
- Data transformation (cleaning, modeling, aggregating)
- Data orchestration (scheduling and dependency management)
- Data quality (built-in checks and validation)
- Metadata management (lineage, documentation)
## Tutorial
Follow the complete hands-on tutorial at:
[Bruin Data Engineering Zoomcamp Template](https://github.com/bruin-data/bruin/tree/main/templates/zoomcamp)
The template is a TODO-based learning exercise — run `bruin init zoomcamp my-taxi-pipeline` and fill in the configuration and code guided by inline comments. The [notes](notes/) contain completed reference implementations.
## Videos
### :movie_camera: 5.1 - Introduction to Bruin
[](https://youtu.be/f6vg7lGqZx0&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&index=1)
Introduction to the Bruin data platform: what it is, what a modern data stack looks like (ETL/ELT, orchestration, data quality), and how Bruin brings all of these together into a single project.
- [Notes](notes/01-introduction.md)
### :movie_camera: 5.2 - Getting Started with Bruin
[](https://youtu.be/JJwHKSidX_c&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&index=2)
Install Bruin, set up the VS Code/Cursor extension and Bruin MCP, and create a first project using `bruin init`. Walk through environments, connections (DuckDB, Chess.com), pipeline YAML configuration, and running Python, YAML ingestor, and SQL assets.
- [Notes](notes/02-getting-started.md)
### :movie_camera: 5.3 - Building an End-to-End Pipeline with NYC Taxi Data
[](https://youtu.be/q0k_iz9kWsI&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&index=3)
Build a full pipeline with a three-layered architecture (ingestion, staging, reports) using NYC taxi data and DuckDB.
- [Notes](notes/03-nyc-taxi-pipeline.md)
### :movie_camera: 5.4 - Using Bruin MCP with AI Agents
[](https://youtu.be/224xH7h8OaQ&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&index=4)
Install the Bruin MCP in Cursor/VS Code and use an AI agent to build the entire NYC taxi pipeline end to end. Query data conversationally, ask questions about pipeline logic, and troubleshoot issues — all through natural language.
- [Notes](notes/04-bruin-mcp.md)
### :movie_camera: 5.5 - Deploying to Bruin Cloud
[](https://youtu.be/uBqjLEwF8rc&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&index=5)
Register for Bruin Cloud, connect your GitHub repository, set up data warehouse connections, deploy and monitor your pipelines with a fully managed infrastructure.
- [Notes](notes/05-bruin-cloud.md)
## Bruin Core Concepts
Short videos covering the fundamental concepts of Bruin: projects, pipelines, assets, variables, and commands.
### :movie_camera: Projects
[](https://www.youtube.com/watch?v=YWDjnSxbBtY)
The root directory where you create your Bruin data pipeline. Learn about project initialization, the `.bruin.yml` configuration file, environments, and connections.
- [Notes](notes/06-core-01-projects.md)
### :movie_camera: Pipelines
[](https://www.youtube.com/watch?v=uzp_DiR4Sok)
A grouping mechanism for organizing assets based on their execution schedule. Each pipeline has a single schedule and its own configuration file.
- [Notes](notes/06-core-02-pipelines.md)
### :movie_camera: Assets
[](https://www.youtube.com/watch?v=ZElY5SoqrwI)
Single files that perform specific tasks, creating or updating tables/views in your database. Covers SQL, Python, and YAML asset types with examples.
- [Notes](notes/06-core-03-assets.md)
### :movie_camera: Variables
[](https://www.youtube.com/watch?v=XCx0nDmhhxA)
Dynamic values initialized at each pipeline run. Learn about built-in variables (start_date, end_date) and custom variables for parameterizing your pipelines.
- [Notes](notes/06-core-04-variables.md)
### :movie_camera: Commands
[](https://www.youtube.com/watch?v=3nykPEs_V7E)
CLI commands for interacting with your Bruin project: `bruin run`, `bruin validate`, `bruin lineage`, and more with practical examples.
- [Notes](notes/06-core-05-commands.md)
## Resources
- [Bruin Documentation](https://getbruin.com/docs)
- [Bruin GitHub Repository](https://github.com/bruin-data/bruin)
- [Bruin MCP (AI Integration)](https://getbruin.com/docs/bruin/getting-started/bruin-mcp)
- [Bruin Cloud](https://getbruin.com/) — managed deployment and monitoring
# Homework
* [2026 Homework](../cohorts/2026/05-data-platforms/homework.md)
# Community notes
Did you take notes? You can share them here
* Add your notes here (above this line)Did you take notes? You can share them here
* [Notes by Alvaro Navas](https://github.com/ziritrion/dataeng-zoomcamp/blob/main/notes/5_batch_processing.md) * [Sandy's DE Learning Blog](https://learningdataengineering540969211.wordpress.com/2022/02/24/week-5-de-zoomcamp-5-2-1-installing-spark-on-linux/) * [Notes by Alain Boisvert](https://github.com/boisalai/de-zoomcamp-2023/blob/main/week5.md) * [Alternative : Using docker-compose to launch spark by rafik](https://gist.github.com/rafik-rahoui/f98df941c4ccced9c46e9ccbdef63a03) * [Marcos Torregrosa's blog (spanish)](https://www.n4gash.com/2023/data-engineering-zoomcamp-semana-5-batch-spark) * [Notes by Victor Padilha](https://github.com/padilha/de-zoomcamp/tree/master/week5) * [Notes by Oscar Garcia](https://github.com/ozkary/Data-Engineering-Bootcamp/tree/main/Step5-Batch-Processing) * [Notes by HongWei](https://github.com/hwchua0209/data-engineering-zoomcamp-submission/blob/main/05-batch-processing/README.md) * [2024 videos transcript](https://drive.google.com/drive/folders/1XMmP4H5AMm1qCfMFxc_hqaPGw31KIVcb?usp=drive_link) by Maria Fisher * [2025 Notes by Manuel Guerra](https://github.com/ManuelGuerra1987/data-engineering-zoomcamp-notes/blob/main/5_Batch-Processing-Spark/README.md) * [2025 Notes by Gabi Fonseca](https://github.com/fonsecagabriella/data_engineering/blob/main/05_batch_processing/00_notes.md) * [2025 Notes on Installing Spark on MacOS (with Anaconda + brew) by Gabi Fonseca](https://github.com/fonsecagabriella/data_engineering/blob/main/05_batch_processing/01_env_setup.md) * [2025 Notes by Daniel Lachner](https://github.com/mossdet/dlp_data_eng/blob/main/Notes/05_01_Batch_Processing_Spark_GCP.pdf) * [2026 Notes by Ajay Katte](https://github.com/mushroomsandchai/dtdez/tree/main/06_batch_processing/notes) * Add your notes here (above this line)Did you take notes? You can share them here
* [Notes by Alvaro Navas](https://github.com/ziritrion/dataeng-zoomcamp/blob/main/notes/6_streaming.md ) * [Marcos Torregrosa's blog (spanish)](https://www.n4gash.com/2023/data-engineering-zoomcamp-semana-6-stream-processing/) * [Notes by Oscar Garcia](https://github.com/ozkary/Data-Engineering-Bootcamp/tree/main/Step6-Streaming) * [Notes by Shayan Shafiee Moghadam](https://github.com/shayansm2/eng-notebook/blob/main/kafka/readme.md) * Add your notes here (above this line)newBuilder().
*/
public RideRecord() {}
/**
* All-args constructor.
* @param vendor_id The new value for vendor_id
* @param passenger_count The new value for passenger_count
* @param trip_distance The new value for trip_distance
*/
public RideRecord(java.lang.String vendor_id, java.lang.Integer passenger_count, java.lang.Double trip_distance) {
this.vendor_id = vendor_id;
this.passenger_count = passenger_count;
this.trip_distance = trip_distance;
}
@Override
public org.apache.avro.specific.SpecificData getSpecificData() { return MODEL$; }
@Override
public org.apache.avro.Schema getSchema() { return SCHEMA$; }
// Used by DatumWriter. Applications should not call.
@Override
public java.lang.Object get(int field$) {
switch (field$) {
case 0: return vendor_id;
case 1: return passenger_count;
case 2: return trip_distance;
default: throw new IndexOutOfBoundsException("Invalid index: " + field$);
}
}
// Used by DatumReader. Applications should not call.
@Override
@SuppressWarnings(value="unchecked")
public void put(int field$, java.lang.Object value$) {
switch (field$) {
case 0: vendor_id = value$ != null ? value$.toString() : null; break;
case 1: passenger_count = (java.lang.Integer)value$; break;
case 2: trip_distance = (java.lang.Double)value$; break;
default: throw new IndexOutOfBoundsException("Invalid index: " + field$);
}
}
/**
* Gets the value of the 'vendor_id' field.
* @return The value of the 'vendor_id' field.
*/
public java.lang.String getVendorId() {
return vendor_id;
}
/**
* Sets the value of the 'vendor_id' field.
* @param value the value to set.
*/
public void setVendorId(java.lang.String value) {
this.vendor_id = value;
}
/**
* Gets the value of the 'passenger_count' field.
* @return The value of the 'passenger_count' field.
*/
public int getPassengerCount() {
return passenger_count;
}
/**
* Sets the value of the 'passenger_count' field.
* @param value the value to set.
*/
public void setPassengerCount(int value) {
this.passenger_count = value;
}
/**
* Gets the value of the 'trip_distance' field.
* @return The value of the 'trip_distance' field.
*/
public double getTripDistance() {
return trip_distance;
}
/**
* Sets the value of the 'trip_distance' field.
* @param value the value to set.
*/
public void setTripDistance(double value) {
this.trip_distance = value;
}
/**
* Creates a new RideRecord RecordBuilder.
* @return A new RideRecord RecordBuilder
*/
public static schemaregistry.RideRecord.Builder newBuilder() {
return new schemaregistry.RideRecord.Builder();
}
/**
* Creates a new RideRecord RecordBuilder by copying an existing Builder.
* @param other The existing builder to copy.
* @return A new RideRecord RecordBuilder
*/
public static schemaregistry.RideRecord.Builder newBuilder(schemaregistry.RideRecord.Builder other) {
if (other == null) {
return new schemaregistry.RideRecord.Builder();
} else {
return new schemaregistry.RideRecord.Builder(other);
}
}
/**
* Creates a new RideRecord RecordBuilder by copying an existing RideRecord instance.
* @param other The existing instance to copy.
* @return A new RideRecord RecordBuilder
*/
public static schemaregistry.RideRecord.Builder newBuilder(schemaregistry.RideRecord other) {
if (other == null) {
return new schemaregistry.RideRecord.Builder();
} else {
return new schemaregistry.RideRecord.Builder(other);
}
}
/**
* RecordBuilder for RideRecord instances.
*/
@org.apache.avro.specific.AvroGenerated
public static class Builder extends org.apache.avro.specific.SpecificRecordBuilderBasenewBuilder().
*/
public RideRecordCompatible() {}
/**
* All-args constructor.
* @param vendorId The new value for vendorId
* @param passenger_count The new value for passenger_count
* @param trip_distance The new value for trip_distance
* @param pu_location_id The new value for pu_location_id
*/
public RideRecordCompatible(java.lang.String vendorId, java.lang.Integer passenger_count, java.lang.Double trip_distance, java.lang.Long pu_location_id) {
this.vendorId = vendorId;
this.passenger_count = passenger_count;
this.trip_distance = trip_distance;
this.pu_location_id = pu_location_id;
}
@Override
public org.apache.avro.specific.SpecificData getSpecificData() { return MODEL$; }
@Override
public org.apache.avro.Schema getSchema() { return SCHEMA$; }
// Used by DatumWriter. Applications should not call.
@Override
public java.lang.Object get(int field$) {
switch (field$) {
case 0: return vendorId;
case 1: return passenger_count;
case 2: return trip_distance;
case 3: return pu_location_id;
default: throw new IndexOutOfBoundsException("Invalid index: " + field$);
}
}
// Used by DatumReader. Applications should not call.
@Override
@SuppressWarnings(value="unchecked")
public void put(int field$, java.lang.Object value$) {
switch (field$) {
case 0: vendorId = value$ != null ? value$.toString() : null; break;
case 1: passenger_count = (java.lang.Integer)value$; break;
case 2: trip_distance = (java.lang.Double)value$; break;
case 3: pu_location_id = (java.lang.Long)value$; break;
default: throw new IndexOutOfBoundsException("Invalid index: " + field$);
}
}
/**
* Gets the value of the 'vendorId' field.
* @return The value of the 'vendorId' field.
*/
public java.lang.String getVendorId() {
return vendorId;
}
/**
* Sets the value of the 'vendorId' field.
* @param value the value to set.
*/
public void setVendorId(java.lang.String value) {
this.vendorId = value;
}
/**
* Gets the value of the 'passenger_count' field.
* @return The value of the 'passenger_count' field.
*/
public int getPassengerCount() {
return passenger_count;
}
/**
* Sets the value of the 'passenger_count' field.
* @param value the value to set.
*/
public void setPassengerCount(int value) {
this.passenger_count = value;
}
/**
* Gets the value of the 'trip_distance' field.
* @return The value of the 'trip_distance' field.
*/
public double getTripDistance() {
return trip_distance;
}
/**
* Sets the value of the 'trip_distance' field.
* @param value the value to set.
*/
public void setTripDistance(double value) {
this.trip_distance = value;
}
/**
* Gets the value of the 'pu_location_id' field.
* @return The value of the 'pu_location_id' field.
*/
public java.lang.Long getPuLocationId() {
return pu_location_id;
}
/**
* Sets the value of the 'pu_location_id' field.
* @param value the value to set.
*/
public void setPuLocationId(java.lang.Long value) {
this.pu_location_id = value;
}
/**
* Creates a new RideRecordCompatible RecordBuilder.
* @return A new RideRecordCompatible RecordBuilder
*/
public static schemaregistry.RideRecordCompatible.Builder newBuilder() {
return new schemaregistry.RideRecordCompatible.Builder();
}
/**
* Creates a new RideRecordCompatible RecordBuilder by copying an existing Builder.
* @param other The existing builder to copy.
* @return A new RideRecordCompatible RecordBuilder
*/
public static schemaregistry.RideRecordCompatible.Builder newBuilder(schemaregistry.RideRecordCompatible.Builder other) {
if (other == null) {
return new schemaregistry.RideRecordCompatible.Builder();
} else {
return new schemaregistry.RideRecordCompatible.Builder(other);
}
}
/**
* Creates a new RideRecordCompatible RecordBuilder by copying an existing RideRecordCompatible instance.
* @param other The existing instance to copy.
* @return A new RideRecordCompatible RecordBuilder
*/
public static schemaregistry.RideRecordCompatible.Builder newBuilder(schemaregistry.RideRecordCompatible other) {
if (other == null) {
return new schemaregistry.RideRecordCompatible.Builder();
} else {
return new schemaregistry.RideRecordCompatible.Builder(other);
}
}
/**
* RecordBuilder for RideRecordCompatible instances.
*/
@org.apache.avro.specific.AvroGenerated
public static class Builder extends org.apache.avro.specific.SpecificRecordBuilderBasenewBuilder().
*/
public RideRecordNoneCompatible() {}
/**
* All-args constructor.
* @param vendorId The new value for vendorId
* @param passenger_count The new value for passenger_count
* @param trip_distance The new value for trip_distance
*/
public RideRecordNoneCompatible(java.lang.Integer vendorId, java.lang.Integer passenger_count, java.lang.Double trip_distance) {
this.vendorId = vendorId;
this.passenger_count = passenger_count;
this.trip_distance = trip_distance;
}
@Override
public org.apache.avro.specific.SpecificData getSpecificData() { return MODEL$; }
@Override
public org.apache.avro.Schema getSchema() { return SCHEMA$; }
// Used by DatumWriter. Applications should not call.
@Override
public java.lang.Object get(int field$) {
switch (field$) {
case 0: return vendorId;
case 1: return passenger_count;
case 2: return trip_distance;
default: throw new IndexOutOfBoundsException("Invalid index: " + field$);
}
}
// Used by DatumReader. Applications should not call.
@Override
@SuppressWarnings(value="unchecked")
public void put(int field$, java.lang.Object value$) {
switch (field$) {
case 0: vendorId = (java.lang.Integer)value$; break;
case 1: passenger_count = (java.lang.Integer)value$; break;
case 2: trip_distance = (java.lang.Double)value$; break;
default: throw new IndexOutOfBoundsException("Invalid index: " + field$);
}
}
/**
* Gets the value of the 'vendorId' field.
* @return The value of the 'vendorId' field.
*/
public int getVendorId() {
return vendorId;
}
/**
* Sets the value of the 'vendorId' field.
* @param value the value to set.
*/
public void setVendorId(int value) {
this.vendorId = value;
}
/**
* Gets the value of the 'passenger_count' field.
* @return The value of the 'passenger_count' field.
*/
public int getPassengerCount() {
return passenger_count;
}
/**
* Sets the value of the 'passenger_count' field.
* @param value the value to set.
*/
public void setPassengerCount(int value) {
this.passenger_count = value;
}
/**
* Gets the value of the 'trip_distance' field.
* @return The value of the 'trip_distance' field.
*/
public double getTripDistance() {
return trip_distance;
}
/**
* Sets the value of the 'trip_distance' field.
* @param value the value to set.
*/
public void setTripDistance(double value) {
this.trip_distance = value;
}
/**
* Creates a new RideRecordNoneCompatible RecordBuilder.
* @return A new RideRecordNoneCompatible RecordBuilder
*/
public static schemaregistry.RideRecordNoneCompatible.Builder newBuilder() {
return new schemaregistry.RideRecordNoneCompatible.Builder();
}
/**
* Creates a new RideRecordNoneCompatible RecordBuilder by copying an existing Builder.
* @param other The existing builder to copy.
* @return A new RideRecordNoneCompatible RecordBuilder
*/
public static schemaregistry.RideRecordNoneCompatible.Builder newBuilder(schemaregistry.RideRecordNoneCompatible.Builder other) {
if (other == null) {
return new schemaregistry.RideRecordNoneCompatible.Builder();
} else {
return new schemaregistry.RideRecordNoneCompatible.Builder(other);
}
}
/**
* Creates a new RideRecordNoneCompatible RecordBuilder by copying an existing RideRecordNoneCompatible instance.
* @param other The existing instance to copy.
* @return A new RideRecordNoneCompatible RecordBuilder
*/
public static schemaregistry.RideRecordNoneCompatible.Builder newBuilder(schemaregistry.RideRecordNoneCompatible other) {
if (other == null) {
return new schemaregistry.RideRecordNoneCompatible.Builder();
} else {
return new schemaregistry.RideRecordNoneCompatible.Builder(other);
}
}
/**
* RecordBuilder for RideRecordNoneCompatible instances.
*/
@org.apache.avro.specific.AvroGenerated
public static class Builder extends org.apache.avro.specific.SpecificRecordBuilderBasewindow [00:00, 00:10) not published yet
A added to window Note over P: 5 seconds pass, phone reconnects P->>K: Event B (ts=14:00:04, 8s late) K->>F: Event B Note over F: watermark = 00:07
window [00:00, 00:10) still not published
B added to window Note over F: more events arrive
watermark reaches 00:10
time to publish F->>PG: INSERT (window=00:00, PU=79, trips=2) Note over PG: both events counted ``` Event B arrived late, but within Flink's patience window. Flink hadn't published the result yet, so B was included in the count. Now what if Event B were 20 seconds late - arriving after Flink already published? ```mermaid sequenceDiagram participant P as Producer participant K as Kafka participant F as Flink participant PG as PostgreSQL P->>K: Event A (ts=14:00:07, on time) K->>F: Event A Note over F: A added to window [00:00, 00:10) Note over F: watermark reaches 00:10
time to publish F->>PG: INSERT (window=00:00, PU=79, trips=1) Note over PG: published with trips=1 Note over P: 20 seconds later, phone reconnects P->>K: Event B (ts=14:00:04, 20s late) K->>F: Event B Note over F: window [00:00, 00:10) already published
but B still belongs to it F->>PG: UPDATE (window=00:00, PU=79, trips=2) Note over PG: upsert via PRIMARY KEY
corrected from 1 to 2 ``` Flink already published trips=1, but when Event B finally arrives, the PRIMARY KEY lets Flink send a correction. PostgreSQL updates the row from 1 to 2. Without the PRIMARY KEY (an append-only sink), Event B would be lost - Flink can't re-open a published window in append mode. The trade-off is latency vs completeness. A larger watermark means more patience for late events, but you wait longer before seeing any results. 5 seconds is a reasonable default. In production, you'd tune this based on how out-of-order your data actually is. Other differences from the pass-through job: - The sink has a `PRIMARY KEY` with `NOT ENFORCED` - this enables upsert behavior in the Flink JDBC connector. - `earliest-offset` - reads all existing data from Kafka. - `env.set_parallelism(3)` - runs 3 copies processing data in parallel. - The `TUMBLE` function creates fixed-size, non-overlapping windows. `DESCRIPTOR(event_timestamp)` must reference the column with the `WATERMARK` defined on it, and `INTERVAL '1' HOUR` sets the window size. Submit and test: ```bash docker compose exec jobmanager ./bin/flink run \ -py /opt/src/job/aggregation_job.py \ --pyFiles /opt/src -d ``` Send data: ```bash uv run python src/producers/producer.py ``` Wait ~15 seconds for the windows to close, then check: ```sql SELECT window_start, count(*) as locations, sum(num_trips) as total_trips, round(sum(total_revenue)::numeric, 2) as revenue FROM processed_events_aggregated GROUP BY window_start ORDER BY window_start; ``` ``` window_start | locations | total_trips | revenue ----------------------+-----------+-------------+--------- 2025-11-01 00:00:00 | ... 2025-11-01 01:00:00 | ... ... ``` The 1000 taxi trips were grouped into 1-hour tumbling windows by pickup location. Each row shows how many locations had trips in that hour and the total number of trips. Try this with a plain Python consumer - you'd need to implement the windowing logic, handle late events, manage state, and write the upsert SQL yourself. With Flink, it's a SQL query. ## Late events and upserts The CSV producer sends events in order, so the watermark never has to handle late arrivals. Let's use a real-time producer that generates synthetic events with occasional delays to see what happens. Download and run the real-time producer: ```bash PREFIX="https://raw.githubusercontent.com/DataTalksClub/data-engineering-zoomcamp/main/07-streaming/workshop" wget ${PREFIX}/src/producers/producer_realtime.py -P src/producers/ ``` ```bash uv run python src/producers/producer_realtime.py ``` It generates random taxi trips with current timestamps, but ~20% of events are sent with a timestamp 3-10 seconds in the past (simulating network delays). The output labels each event: ``` on time -> PU=79 ts=14:23:05 on time -> PU=107 ts=14:23:05 LATE (8s) -> PU=234 ts=14:22:58 on time -> PU=48 ts=14:23:06 ``` With our 5-second watermark and 1-hour windows, no events will be dropped - even an event 10 seconds late lands well within the current hour window. But the watermark + upsert behavior is still visible: Flink first emits window results when the watermark passes the window end, then late events update those results via the PRIMARY KEY. To see this in action, open two terminals: Terminal 1 - run the real-time producer: ```bash uv run python src/producers/producer_realtime.py ``` Terminal 2 - watch aggregation counts change: ```bash watch -n 1 'PGPASSWORD=postgres docker compose exec postgres psql -U postgres -d postgres -c "SELECT window_start, sum(num_trips) as trips, round(sum(total_revenue)::numeric, 2) as revenue FROM processed_events_aggregated GROUP BY window_start ORDER BY window_start;"' ``` You'll see the counts for older windows increase as late events arrive and update the aggregation via upsert. This is why we set up the PRIMARY KEY - without it, late events would either be dropped or create duplicates. ## Understanding window types We used tumbling windows above. Flink supports three types: ### Tumbling windows Fixed-size, non-overlapping. Every event belongs to exactly one window. If you come from the batch world, tumbling windows are the most familiar - they just cut up your data into fixed segments. It's essentially a way to speed up batch processing. ``` | Window 1 | Window 2 | Window 3 | | 1 hour | 1 hour | 1 hour | ``` Use case: Counting trips per hour, daily revenue summaries. ### Sliding windows Fixed-size, overlapping. An event can belong to multiple windows. When you think of a 1-hour window, most people think of 00:00-01:00. But there's also 00:15-01:15, 00:30-01:30 - those are also 1-hour windows, just starting at different points. Sliding windows capture all of them. ``` |--- Window 1 (1 hour) ---| |--- Window 2 (1 hour) ---| |--- Window 3 (1 hour) ---| <- 15 min slide -> ``` ```sql HOP(TABLE events, DESCRIPTOR(event_timestamp), INTERVAL '15' MINUTE, INTERVAL '1' HOUR) ``` Use case: finding peaks and valleys - "what was our peak traffic in any 1-hour window?" These overlapping windows let you find the moment in time where you have the highest or lowest values. Good for min-maxing, moving averages, and surge detection (e.g., ride-share surge pricing). ### Session windows Dynamic windows based on inactivity gaps. Unlike tumbling and sliding windows, the window size isn't fixed - the window doesn't close at a specified time, it closes after a specified amount of inactivity. ``` |--events--| gap |--events------| gap |--events--| | Session 1| | Session 2 | | Session 3| ``` Use case: grouping user behavior together. Imagine a user logs into an app, clicks a bunch of buttons, leaves for 2 minutes, then comes back - that's still technically the same session. You set a session gap (say, 30 minutes of inactivity) and Flink groups all the events within that session together. Sessionization is very powerful for behavioral analytics. ## Cleanup Stop and remove all containers: ```bash docker compose down ``` To also remove the PostgreSQL data volume: ```bash docker compose down -v ``` ## Q&A Questions and answers from the [2025 stream with Zach Wilson](https://www.youtube.com/watch?v=P2loELMUUeI). ### What happens when a Flink job dies and restarts? Does it reprocess everything? The `earliest` offset setting is only for the initial startup. If the job restarts (not re-submitted as a new job), it uses checkpointing to resume from the last snapshot. Without checkpointing, you either reprocess everything (with `earliest`) or skip data (with `latest`). The catch: checkpoints are scoped to a specific job instance. If you completely kill a job and submit a new one, the new job has no knowledge of the previous checkpoints. To preserve state across redeployments, restart the existing job rather than creating a new one. ### Why can't we just use Kafka consumers? What does Flink actually add? For simple pass-through (read a message, write it somewhere), a Kafka consumer is fine. For anything involving time windows, watermarks, checkpointing, or parallel processing, Flink saves you from building all that yourself. You can do windowing, watermarking, late data handling, and job recovery with a plain consumer - go ahead and manage it yourself. But as Zach puts it: "good luck." With a plain consumer, you'd also need to track checkpoints yourself - save the latest processed timestamp to a file or database and manage it on every restart. Flink keeps the state for you. It's like asking "why use Spark when you can use Pandas?" You can, but Pandas won't work at higher scale in a distributed way. ### What happens with events delayed beyond the watermark (the "tunnel" scenario)? There are two types of lateness. The watermark handles acceptable lateness - small delays where events arrive a few seconds late. For events arriving much later (like after a 5-minute tunnel), Flink has an allowed lateness parameter. By default, allowed lateness is zero - events arriving after the watermark closes a window are discarded. If you set allowed lateness to 10 minutes, Flink will go back, find the old closed window, create a new aggregation with the late event, and send it to the sink as a brand new record. This means you need deduplication logic on the sink side (a primary key with upsert behavior - exactly what we set up in the aggregation section). The trade-off: allowed lateness requires Flink to hold all those windows on disk for the duration of the tolerance. ### When do we actually need streaming? For many things micro-batch is enough. The key question: is something going to happen in real time on the other side? If there is an automated process that will change something based on the data, streaming is a great choice. If a human is just looking at data, real-time is unnecessary and micro-batch is easier to maintain. In 10 years as a data engineer, Zach had literally two use cases that genuinely needed streaming - Netflix fraud/security detection (5 minutes of delay means 5 more minutes of a hacked account) and Airbnb surge pricing (supply and demand changes rapidly). Everything else was daily batch, or hourly/every-15-minute micro-batch for lower latency needs. Before committing to streaming, consider the operational cost. A streaming job runs 24/7 - if it breaks at 3 AM, someone needs to fix it. If you're the only person on the team who understands Flink, you'll be on-call for it forever. Talk to your manager before implementing streaming - you'll need to teach your entire team before you can share the on-call burden. ### Spark Streaming vs Flink Streaming? They are fundamentally different today but will likely converge. The key difference: Spark Streaming is micro-batch - it pulses every 15-30 seconds, pulling data in small batches (pull architecture). Flink is genuine continuous processing - events flow through as they arrive (push architecture). For most use cases the difference is negligible, but Flink has lower latency for truly real-time needs. For micro-batch intervals, Zach finds every-5-minutes too frequent with Spark because startup alone takes about a minute, making the overhead-to-work ratio poor. His sweet spots are hourly and every 15 minutes. ### How does job submission work in production? In this workshop we mount local files into Docker and submit jobs with `docker compose exec` - that's a development convenience. In production, job submission looks different depending on the deployment: - Managed services (AWS Kinesis Data Analytics, Google Cloud Dataflow, Confluent Cloud) - you upload a JAR or Python zip through a web console or CLI. The service handles the cluster. - Self-hosted Flink on Kubernetes - you typically build a Docker image with your job code baked in, or use the Flink Kubernetes Operator which pulls job artifacts from S3/GCS at startup. - Standalone Flink cluster - you use the `flink run` CLI pointing to a local file or an HTTP/S3 URL. CI/CD pipelines often upload the job artifact to S3 and then call `flink run` with that URL. The common pattern: your code lives in git, CI builds an artifact (JAR, Python zip, or Docker image), pushes it to a registry or object store, and then triggers the Flink cluster to pick it up. ================================================ FILE: 07-streaming/workshop/docker-compose.yml ================================================ services: redpanda: image: redpandadata/redpanda:v25.3.9 command: - redpanda - start - --smp - '1' - --reserve-memory - 0M - --overprovisioned - --node-id - '1' - --kafka-addr - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092 - --advertise-kafka-addr - PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092 - --pandaproxy-addr - PLAINTEXT://0.0.0.0:28082,OUTSIDE://0.0.0.0:8082 - --advertise-pandaproxy-addr - PLAINTEXT://redpanda:28082,OUTSIDE://localhost:8082 - --rpc-addr - 0.0.0.0:33145 - --advertise-rpc-addr - redpanda:33145 ports: - 8082:8082 - 9092:9092 - 28082:28082 - 29092:29092 jobmanager: build: context: . dockerfile: ./Dockerfile.flink image: pyflink-workshop pull_policy: never expose: - "6123" ports: - "8081:8081" volumes: - ./:/opt/flink/usrlib - ./src/:/opt/src command: jobmanager environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager jobmanager.memory.process.size: 1600m taskmanager: image: pyflink-workshop pull_policy: never expose: - "6121" - "6122" volumes: - ./:/opt/flink/usrlib - ./src/:/opt/src depends_on: - jobmanager command: taskmanager --taskmanager.registration.timeout 5 min environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager taskmanager.memory.process.size: 1728m taskmanager.numberOfTaskSlots: 15 parallelism.default: 3 postgres: image: postgres:18 restart: on-failure environment: - POSTGRES_DB=postgres - POSTGRES_USER=postgres - POSTGRES_PASSWORD=postgres ports: - "5432:5432" ================================================ FILE: 07-streaming/workshop/flink-config.yaml ================================================ # Custom Flink config for PyFlink workshop. # Original: https://github.com/apache/flink/blob/release-2.2/flink-dist/src/main/resources/config.yaml # Changes from default: # 1. Added taskmanager.memory.jvm-metaspace.size: 512m (PyFlink needs more metaspace) # 2. Removed --add-exports=jdk.compiler/... from env.java.opts.all # (jdk.compiler module is not present in the JRE, causing warnings on every command) blob: server: port: '6124' taskmanager: memory: process: size: 1728m jvm-metaspace: size: 512m # added for PyFlink bind-host: 0.0.0.0 numberOfTaskSlots: 15 jobmanager: execution: failover-strategy: region rpc: address: jobmanager port: 6123 memory: process: size: 1600m bind-host: 0.0.0.0 query: server: port: '6125' parallelism: default: 1 rest: address: 0.0.0.0 env: java: opts: all: >- --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED ================================================ FILE: 07-streaming/workshop/live/.gitignore ================================================ # Byte-compiled / optimized / DLL files __pycache__/ *.py[codz] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py.cover .hypothesis/ .pytest_cache/ cover/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder .pybuilder/ target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: # .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # UV # Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control. # This is especially recommended for binary packages to ensure reproducibility, and is more # commonly ignored for libraries. #uv.lock # poetry # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. # This is especially recommended for binary packages to ensure reproducibility, and is more # commonly ignored for libraries. # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control #poetry.lock #poetry.toml # pdm # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. # pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python. # https://pdm-project.org/en/latest/usage/project/#working-with-version-control #pdm.lock #pdm.toml .pdm-python .pdm-build/ # pixi # Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control. #pixi.lock # Pixi creates a virtual environment in the .pixi directory, just like venv module creates one # in the .venv directory. It is recommended not to include this directory in version control. .pixi # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .envrc .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # pytype static type analyzer .pytype/ # Cython debug symbols cython_debug/ # PyCharm # JetBrains specific template is maintained in a separate JetBrains.gitignore that can # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. #.idea/ # Abstra # Abstra is an AI-powered process automation framework. # Ignore directories containing user credentials, local state, and settings. # Learn more at https://abstra.io/docs .abstra/ # Visual Studio Code # Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore # that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore # and can be added to the global gitignore or merged into this file. However, if you prefer, # you could uncomment the following to ignore the entire vscode folder # .vscode/ # Ruff stuff: .ruff_cache/ # PyPI configuration file .pypirc # Cursor # Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to # exclude from AI features like autocomplete and code analysis. Recommended for sensitive data # refer to https://docs.cursor.com/context/ignore-files .cursorignore .cursorindexingignore # Marimo marimo/_static/ marimo/_lsp/ __marimo__/ ================================================ FILE: 07-streaming/workshop/live/.python-version ================================================ 3.12 ================================================ FILE: 07-streaming/workshop/live/Dockerfile.flink ================================================ FROM flink:2.2.0-scala_2.12-java17 COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/ # ref: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker WORKDIR /opt/pyflink COPY pyproject.flink.toml pyproject.toml RUN uv python install 3.12 && uv sync ENV PATH="/opt/pyflink/.venv/bin:$PATH" # Download connector libraries WORKDIR /opt/flink/lib RUN wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-json/2.2.0/flink-json-2.2.0.jar; \ wget https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka/4.0.1-2.0/flink-sql-connector-kafka-4.0.1-2.0.jar; \ wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-connector-jdbc-core/4.0.0-2.0/flink-connector-jdbc-core-4.0.0-2.0.jar; \ wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-connector-jdbc-postgres/4.0.0-2.0/flink-connector-jdbc-postgres-4.0.0-2.0.jar; \ wget https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.10/postgresql-42.7.10.jar COPY flink-config.yaml /opt/flink/conf/config.yaml WORKDIR /opt/flink ================================================ FILE: 07-streaming/workshop/live/README.md ================================================ # streaming-workshop ================================================ FILE: 07-streaming/workshop/live/docker-compose.yaml ================================================ services: redpanda: image: redpandadata/redpanda:v25.3.9 command: - redpanda - start - --smp - '1' - --reserve-memory - 0M - --overprovisioned - --node-id - '1' - --kafka-addr - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092 - --advertise-kafka-addr - PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092 - --pandaproxy-addr - PLAINTEXT://0.0.0.0:28082,OUTSIDE://0.0.0.0:8082 - --advertise-pandaproxy-addr - PLAINTEXT://redpanda:28082,OUTSIDE://localhost:8082 - --rpc-addr - 0.0.0.0:33145 - --advertise-rpc-addr - redpanda:33145 ports: - 8082:8082 - 9092:9092 - 28082:28082 - 29092:29092 postgres: image: postgres:18 restart: on-failure environment: - POSTGRES_DB=postgres - POSTGRES_USER=postgres - POSTGRES_PASSWORD=postgres ports: - "5432:5432" jobmanager: build: context: . dockerfile: ./Dockerfile.flink image: pyflink-workshop pull_policy: never expose: - "6123" ports: - "8081:8081" volumes: - ./:/opt/flink/usrlib - ./src/:/opt/src command: jobmanager environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager jobmanager.memory.process.size: 1600m taskmanager: image: pyflink-workshop pull_policy: never expose: - "6121" - "6122" volumes: - ./:/opt/flink/usrlib - ./src/:/opt/src depends_on: - jobmanager command: taskmanager --taskmanager.registration.timeout 5 min environment: - | FLINK_PROPERTIES= jobmanager.rpc.address: jobmanager taskmanager.memory.process.size: 1728m taskmanager.numberOfTaskSlots: 15 parallelism.default: 3 ================================================ FILE: 07-streaming/workshop/live/flink-config.yaml ================================================ # Custom Flink config for PyFlink workshop. # Original: https://github.com/apache/flink/blob/release-2.2/flink-dist/src/main/resources/config.yaml # Changes from default: # 1. Added taskmanager.memory.jvm-metaspace.size: 512m (PyFlink needs more metaspace) # 2. Removed --add-exports=jdk.compiler/... from env.java.opts.all # (jdk.compiler module is not present in the JRE, causing warnings on every command) blob: server: port: '6124' taskmanager: memory: process: size: 1728m jvm-metaspace: size: 512m # added for PyFlink bind-host: 0.0.0.0 numberOfTaskSlots: 15 jobmanager: execution: failover-strategy: region rpc: address: jobmanager port: 6123 memory: process: size: 1600m bind-host: 0.0.0.0 query: server: port: '6125' parallelism: default: 1 rest: address: 0.0.0.0 env: java: opts: all: >- --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED ================================================ FILE: 07-streaming/workshop/live/main.py ================================================ def main(): print("Hello from streaming-workshop!") if __name__ == "__main__": main() ================================================ FILE: 07-streaming/workshop/live/notebooks/consumer_db.ipynb ================================================ { "cells": [ { "cell_type": "code", "execution_count": 2, "id": "c77749d8", "metadata": {}, "outputs": [], "source": [ "from kafka import KafkaConsumer\n", "\n", "server = 'localhost:9092'\n", "topic_name = 'rides'" ] }, { "cell_type": "code", "execution_count": 3, "id": "74dcdffe", "metadata": {}, "outputs": [], "source": [ "from models import Ride, ride_deserializer" ] }, { "cell_type": "code", "execution_count": null, "id": "00726e41", "metadata": {}, "outputs": [], "source": [ "consumer = KafkaConsumer(\n", " topic_name,\n", " bootstrap_servers=[server],\n", " auto_offset_reset='earliest',\n", " group_id='rides-database',\n", " value_deserializer=ride_deserializer\n", ")" ] }, { "cell_type": "code", "execution_count": 1, "id": "a2cf7106", "metadata": {}, "outputs": [], "source": [ "import psycopg2\n", "\n", "conn = psycopg2.connect(\n", " host='localhost',\n", " port=5432,\n", " database='postgres',\n", " user='postgres',\n", " password='postgres'\n", ")\n", "\n", "conn.autocommit = True\n", "cur = conn.cursor()" ] }, { "cell_type": "code", "execution_count": 5, "id": "f0902406", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Listening to rides and writing to PostgreSQL...\n", "Inserted 100 rows...\n", "Inserted 200 rows...\n", "Inserted 300 rows...\n", "Inserted 400 rows...\n", "Inserted 500 rows...\n", "Inserted 600 rows...\n", "Inserted 700 rows...\n", "Inserted 800 rows...\n", "Inserted 900 rows...\n", "Inserted 1000 rows...\n" ] }, { "ename": "KeyboardInterrupt", "evalue": "", "output_type": "error", "traceback": [ "\u001b[31m---------------------------------------------------------------------------\u001b[39m", "\u001b[31mKeyboardInterrupt\u001b[39m Traceback (most recent call last)", "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[5]\u001b[39m\u001b[32m, line 6\u001b[39m\n\u001b[32m 3\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mListening to \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mtopic_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m and writing to PostgreSQL...\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m 5\u001b[39m count = \u001b[32m0\u001b[39m\n\u001b[32m----> \u001b[39m\u001b[32m6\u001b[39m \u001b[38;5;28;43;01mfor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mmessage\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01min\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mconsumer\u001b[49m\u001b[43m:\u001b[49m\n\u001b[32m 7\u001b[39m \u001b[43m \u001b[49m\u001b[43mride\u001b[49m\u001b[43m \u001b[49m\u001b[43m=\u001b[49m\u001b[43m \u001b[49m\u001b[43mmessage\u001b[49m\u001b[43m.\u001b[49m\u001b[43mvalue\u001b[49m\n\u001b[32m 8\u001b[39m \u001b[43m \u001b[49m\u001b[43mpickup_dt\u001b[49m\u001b[43m \u001b[49m\u001b[43m=\u001b[49m\u001b[43m \u001b[49m\u001b[43mdatetime\u001b[49m\u001b[43m.\u001b[49m\u001b[43mfromtimestamp\u001b[49m\u001b[43m(\u001b[49m\u001b[43mride\u001b[49m\u001b[43m.\u001b[49m\u001b[43mtpep_pickup_datetime\u001b[49m\u001b[43m \u001b[49m\u001b[43m/\u001b[49m\u001b[43m \u001b[49m\u001b[32;43m1000\u001b[39;49m\u001b[43m)\u001b[49m\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/consumer/group.py:1213\u001b[39m, in \u001b[36mKafkaConsumer.__next__\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 1211\u001b[39m \u001b[38;5;28mself\u001b[39m._iterator = \u001b[38;5;28mself\u001b[39m._message_generator_v2()\n\u001b[32m 1212\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1213\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mnext\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_iterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 1214\u001b[39m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mStopIteration\u001b[39;00m:\n\u001b[32m 1215\u001b[39m \u001b[38;5;28mself\u001b[39m._iterator = \u001b[38;5;28;01mNone\u001b[39;00m\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/consumer/group.py:1185\u001b[39m, in \u001b[36mKafkaConsumer._message_generator_v2\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 1183\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34m_message_generator_v2\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[32m 1184\u001b[39m timeout_ms = \u001b[32m1000\u001b[39m * \u001b[38;5;28mmax\u001b[39m(\u001b[32m0\u001b[39m, \u001b[38;5;28mself\u001b[39m._consumer_timeout - time.time())\n\u001b[32m-> \u001b[39m\u001b[32m1185\u001b[39m record_map = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mpoll\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout_ms\u001b[49m\u001b[43m=\u001b[49m\u001b[43mtimeout_ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mupdate_offsets\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[32m 1186\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m tp, records \u001b[38;5;129;01min\u001b[39;00m six.iteritems(record_map):\n\u001b[32m 1187\u001b[39m \u001b[38;5;66;03m# Generators are stateful, and it is possible that the tp / records\u001b[39;00m\n\u001b[32m 1188\u001b[39m \u001b[38;5;66;03m# here may become stale during iteration -- i.e., we seek to a\u001b[39;00m\n\u001b[32m 1189\u001b[39m \u001b[38;5;66;03m# different offset, pause consumption, or lose assignment.\u001b[39;00m\n\u001b[32m 1190\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m record \u001b[38;5;129;01min\u001b[39;00m records:\n\u001b[32m 1191\u001b[39m \u001b[38;5;66;03m# is_fetchable(tp) should handle assignment changes and offset\u001b[39;00m\n\u001b[32m 1192\u001b[39m \u001b[38;5;66;03m# resets; for all other changes (e.g., seeks) we'll rely on the\u001b[39;00m\n\u001b[32m 1193\u001b[39m \u001b[38;5;66;03m# outer function destroying the existing iterator/generator\u001b[39;00m\n\u001b[32m 1194\u001b[39m \u001b[38;5;66;03m# via self._iterator = None\u001b[39;00m\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/consumer/group.py:708\u001b[39m, in \u001b[36mKafkaConsumer.poll\u001b[39m\u001b[34m(self, timeout_ms, max_records, update_offsets)\u001b[39m\n\u001b[32m 706\u001b[39m timer = Timer(timeout_ms)\n\u001b[32m 707\u001b[39m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m._closed:\n\u001b[32m--> \u001b[39m\u001b[32m708\u001b[39m records = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_poll_once\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimer\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_records\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mupdate_offsets\u001b[49m\u001b[43m=\u001b[49m\u001b[43mupdate_offsets\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 709\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m records:\n\u001b[32m 710\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m records\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/consumer/group.py:757\u001b[39m, in \u001b[36mKafkaConsumer._poll_once\u001b[39m\u001b[34m(self, timer, max_records, update_offsets)\u001b[39m\n\u001b[32m 754\u001b[39m log.debug(\u001b[33m'\u001b[39m\u001b[33mpoll: do not have all fetch positions...\u001b[39m\u001b[33m'\u001b[39m)\n\u001b[32m 755\u001b[39m poll_timeout_ms = \u001b[38;5;28mmin\u001b[39m(poll_timeout_ms, \u001b[38;5;28mself\u001b[39m.config[\u001b[33m'\u001b[39m\u001b[33mretry_backoff_ms\u001b[39m\u001b[33m'\u001b[39m])\n\u001b[32m--> \u001b[39m\u001b[32m757\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_client\u001b[49m\u001b[43m.\u001b[49m\u001b[43mpoll\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout_ms\u001b[49m\u001b[43m=\u001b[49m\u001b[43mpoll_timeout_ms\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 758\u001b[39m \u001b[38;5;66;03m# after the long poll, we should check whether the group needs to rebalance\u001b[39;00m\n\u001b[32m 759\u001b[39m \u001b[38;5;66;03m# prior to returning data so that the group can stabilize faster\u001b[39;00m\n\u001b[32m 760\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m._coordinator.need_rejoin():\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/client_async.py:685\u001b[39m, in \u001b[36mKafkaClient.poll\u001b[39m\u001b[34m(self, timeout_ms, future)\u001b[39m\n\u001b[32m 678\u001b[39m timeout = \u001b[38;5;28mmin\u001b[39m(\n\u001b[32m 679\u001b[39m user_timeout_ms,\n\u001b[32m 680\u001b[39m metadata_timeout_ms,\n\u001b[32m 681\u001b[39m idle_connection_timeout_ms,\n\u001b[32m 682\u001b[39m request_timeout_ms)\n\u001b[32m 683\u001b[39m timeout = \u001b[38;5;28mmax\u001b[39m(\u001b[32m0\u001b[39m, timeout) \u001b[38;5;66;03m# avoid negative timeouts\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m685\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_poll\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout\u001b[49m\u001b[43m \u001b[49m\u001b[43m/\u001b[49m\u001b[43m \u001b[49m\u001b[32;43m1000\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[32m 687\u001b[39m \u001b[38;5;66;03m# called without the lock to avoid deadlock potential\u001b[39;00m\n\u001b[32m 688\u001b[39m \u001b[38;5;66;03m# if handlers need to acquire locks\u001b[39;00m\n\u001b[32m 689\u001b[39m responses.extend(\u001b[38;5;28mself\u001b[39m._fire_pending_completed_requests())\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/client_async.py:781\u001b[39m, in \u001b[36mKafkaClient._poll\u001b[39m\u001b[34m(self, timeout)\u001b[39m\n\u001b[32m 778\u001b[39m \u001b[38;5;28;01mcontinue\u001b[39;00m\n\u001b[32m 780\u001b[39m \u001b[38;5;28mself\u001b[39m._idle_expiry_manager.update(conn.node_id)\n\u001b[32m--> \u001b[39m\u001b[32m781\u001b[39m \u001b[38;5;28mself\u001b[39m._pending_completion.extend(\u001b[43mconn\u001b[49m\u001b[43m.\u001b[49m\u001b[43mrecv\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[32m 783\u001b[39m \u001b[38;5;66;03m# Check for additional pending SSL bytes\u001b[39;00m\n\u001b[32m 784\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m.config[\u001b[33m'\u001b[39m\u001b[33msecurity_protocol\u001b[39m\u001b[33m'\u001b[39m] \u001b[38;5;129;01min\u001b[39;00m (\u001b[33m'\u001b[39m\u001b[33mSSL\u001b[39m\u001b[33m'\u001b[39m, \u001b[33m'\u001b[39m\u001b[33mSASL_SSL\u001b[39m\u001b[33m'\u001b[39m):\n\u001b[32m 785\u001b[39m \u001b[38;5;66;03m# TODO: optimize\u001b[39;00m\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/conn.py:1131\u001b[39m, in \u001b[36mBrokerConnection.recv\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 1126\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mrecv\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[32m 1127\u001b[39m \u001b[38;5;250m \u001b[39m\u001b[33;03m\"\"\"Non-blocking network receive.\u001b[39;00m\n\u001b[32m 1128\u001b[39m \n\u001b[32m 1129\u001b[39m \u001b[33;03m Return list of (response, future) tuples\u001b[39;00m\n\u001b[32m 1130\u001b[39m \u001b[33;03m \"\"\"\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m1131\u001b[39m responses = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_recv\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 1132\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m responses \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mself\u001b[39m.requests_timed_out():\n\u001b[32m 1133\u001b[39m timed_out = \u001b[38;5;28mself\u001b[39m.timed_out_ifrs()\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/conn.py:1202\u001b[39m, in \u001b[36mBrokerConnection._recv\u001b[39m\u001b[34m(self)\u001b[39m\n\u001b[32m 1200\u001b[39m recvd_data = \u001b[33mb\u001b[39m\u001b[33m'\u001b[39m\u001b[33m'\u001b[39m.join(recvd)\n\u001b[32m 1201\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m._sensors:\n\u001b[32m-> \u001b[39m\u001b[32m1202\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_sensors\u001b[49m\u001b[43m.\u001b[49m\u001b[43mbytes_received\u001b[49m\u001b[43m.\u001b[49m\u001b[43mrecord\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mrecvd_data\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 1204\u001b[39m \u001b[38;5;66;03m# We need to keep the lock through protocol receipt\u001b[39;00m\n\u001b[32m 1205\u001b[39m \u001b[38;5;66;03m# so that we ensure that the processed byte order is the\u001b[39;00m\n\u001b[32m 1206\u001b[39m \u001b[38;5;66;03m# same as the received byte order\u001b[39;00m\n\u001b[32m 1207\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/metrics/stats/sensor.py:77\u001b[39m, in \u001b[36mSensor.record\u001b[39m\u001b[34m(self, value, time_ms)\u001b[39m\n\u001b[32m 74\u001b[39m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m._lock: \u001b[38;5;66;03m# XXX high volume, might be performance issue\u001b[39;00m\n\u001b[32m 75\u001b[39m \u001b[38;5;66;03m# increment all the stats\u001b[39;00m\n\u001b[32m 76\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m stat \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m._stats:\n\u001b[32m---> \u001b[39m\u001b[32m77\u001b[39m \u001b[43mstat\u001b[49m\u001b[43m.\u001b[49m\u001b[43mrecord\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_config\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mvalue\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtime_ms\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 78\u001b[39m \u001b[38;5;28mself\u001b[39m._check_quotas(time_ms)\n\u001b[32m 79\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m parent \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m._parents:\n", "\u001b[36mFile \u001b[39m\u001b[32m/workspaces/streaming-workshop/.venv/lib/python3.12/site-packages/kafka/metrics/stats/rate.py:49\u001b[39m, in \u001b[36mRate.record\u001b[39m\u001b[34m(self, config, value, time_ms)\u001b[39m\n\u001b[32m 46\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34munit_name\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[32m 47\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m TimeUnit.get_name(\u001b[38;5;28mself\u001b[39m._unit)\n\u001b[32m---> \u001b[39m\u001b[32m49\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mrecord\u001b[39m(\u001b[38;5;28mself\u001b[39m, config, value, time_ms):\n\u001b[32m 50\u001b[39m \u001b[38;5;28mself\u001b[39m._stat.record(config, value, time_ms)\n\u001b[32m 52\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mmeasure\u001b[39m(\u001b[38;5;28mself\u001b[39m, config, now):\n", "\u001b[31mKeyboardInterrupt\u001b[39m: " ] } ], "source": [ "from datetime import datetime\n", "\n", "print(f\"Listening to {topic_name} and writing to PostgreSQL...\")\n", "\n", "count = 0\n", "for message in consumer:\n", " ride = message.value\n", " pickup_dt = datetime.fromtimestamp(ride.tpep_pickup_datetime / 1000)\n", " cur.execute(\n", " \"\"\"INSERT INTO processed_events\n", " (PULocationID, DOLocationID, trip_distance, total_amount, pickup_datetime)\n", " VALUES (%s, %s, %s, %s, %s)\"\"\",\n", " (ride.PULocationID, ride.DOLocationID,\n", " ride.trip_distance, ride.total_amount, pickup_dt)\n", " )\n", " count += 1\n", " if count % 100 == 0:\n", " print(f\"Inserted {count} rows...\")\n", "\n", "consumer.close()\n", "cur.close()\n", "conn.close()" ] }, { "cell_type": "code", "execution_count": null, "id": "66840c80", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "2bec0472", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "streaming-workshop", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.1" } }, "nbformat": 4, "nbformat_minor": 5 } ================================================ FILE: 07-streaming/workshop/live/notebooks/models.py ================================================ import json import dataclasses from dataclasses import dataclass @dataclass class Ride: PULocationID: int DOLocationID: int trip_distance: float total_amount: float tpep_pickup_datetime: int # epoch milliseconds def ride_from_row(row): return Ride( PULocationID=int(row['PULocationID']), DOLocationID=int(row['DOLocationID']), trip_distance=float(row['trip_distance']), total_amount=float(row['total_amount']), tpep_pickup_datetime=int(row['tpep_pickup_datetime'].timestamp() * 1000), ) def ride_serializer(ride): ride_dict = dataclasses.asdict(ride) ride_json = json.dumps(ride_dict).encode('utf-8') return ride_json def ride_deserializer(data): json_str = data.decode('utf-8') ride_dict = json.loads(json_str) return Ride(**ride_dict) ================================================ FILE: 07-streaming/workshop/live/notebooks/producer.ipynb ================================================ { "cells": [ { "cell_type": "code", "execution_count": 2, "id": "eebfcff0", "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 3, "id": "1e3c198b", "metadata": {}, "outputs": [], "source": [ "url = 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-11.parquet'" ] }, { "cell_type": "code", "execution_count": 4, "id": "2113c0a9", "metadata": {}, "outputs": [], "source": [ "columns = ['PULocationID', 'DOLocationID', 'trip_distance', 'total_amount', 'tpep_pickup_datetime']\n", "df = pd.read_parquet(url, columns=columns).head(1000)" ] }, { "cell_type": "code", "execution_count": 39, "id": "05ed66d7", "metadata": {}, "outputs": [], "source": [ "from models import Ride, ride_from_row, ride_serializer" ] }, { "cell_type": "code", "execution_count": 40, "id": "26950bac", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Ride(PULocationID=142, DOLocationID=237, trip_distance=2.28, total_amount=24.94, tpep_pickup_datetime=1761958147000)" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ride = ride_from_row(df.iloc[1])\n", "ride" ] }, { "cell_type": "code", "execution_count": 41, "id": "05cfce95", "metadata": {}, "outputs": [], "source": [ "from kafka import KafkaProducer\n", "\n", "server = 'localhost:9092'\n", "\n", "producer = KafkaProducer(\n", " bootstrap_servers=[server],\n", " value_serializer=ride_serializer\n", ")" ] }, { "cell_type": "code", "execution_count": 46, "id": "21f5fff3", "metadata": {}, "outputs": [], "source": [ "topic_name = 'rides'\n", "\n", "producer.send(topic_name, value=ride)\n", "producer.flush()" ] }, { "cell_type": "code", "execution_count": 48, "id": "b17a175a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sent: Ride(PULocationID=43, DOLocationID=186, trip_distance=1.68, total_amount=22.15, tpep_pickup_datetime=1761956005000)\n", "Sent: Ride(PULocationID=142, DOLocationID=237, trip_distance=2.28, total_amount=24.94, tpep_pickup_datetime=1761958147000)\n", "Sent: Ride(PULocationID=163, DOLocationID=238, trip_distance=2.7, total_amount=25.62, tpep_pickup_datetime=1761955639000)\n", "Sent: Ride(PULocationID=138, DOLocationID=261, trip_distance=12.87, total_amount=86.14, tpep_pickup_datetime=1761955200000)\n", "Sent: Ride(PULocationID=138, DOLocationID=37, trip_distance=8.4, total_amount=48.65, tpep_pickup_datetime=1761956330000)\n", "Sent: Ride(PULocationID=90, DOLocationID=100, trip_distance=0.85, total_amount=16.45, tpep_pickup_datetime=1761956471000)\n", "Sent: Ride(PULocationID=142, DOLocationID=170, trip_distance=3.01, total_amount=25.85, tpep_pickup_datetime=1761955651000)\n", "Sent: Ride(PULocationID=237, DOLocationID=144, trip_distance=3.82, total_amount=57.54, tpep_pickup_datetime=1761958012000)\n", "Sent: Ride(PULocationID=162, DOLocationID=161, trip_distance=0.89, total_amount=12.95, tpep_pickup_datetime=1761958619000)\n", "Sent: Ride(PULocationID=234, DOLocationID=162, trip_distance=2.28, total_amount=38.68, tpep_pickup_datetime=1761955843000)\n", "Sent: Ride(PULocationID=158, DOLocationID=88, trip_distance=3.3, total_amount=44.0, tpep_pickup_datetime=1761955203000)\n", "Sent: Ride(PULocationID=88, DOLocationID=148, trip_distance=1.5, total_amount=19.55, tpep_pickup_datetime=1761957833000)\n", "Sent: Ride(PULocationID=148, DOLocationID=236, trip_distance=4.7, total_amount=47.65, tpep_pickup_datetime=1761958682000)\n", "Sent: Ride(PULocationID=87, DOLocationID=255, trip_distance=5.61, total_amount=38.85, tpep_pickup_datetime=1761958368000)\n", "Sent: Ride(PULocationID=231, DOLocationID=43, trip_distance=3.9, total_amount=46.55, tpep_pickup_datetime=1761955553000)\n", "Sent: Ride(PULocationID=141, DOLocationID=262, trip_distance=1.14, total_amount=14.9, tpep_pickup_datetime=1761956024000)\n", "Sent: Ride(PULocationID=238, DOLocationID=24, trip_distance=0.6, total_amount=9.12, tpep_pickup_datetime=1761955398000)\n", "Sent: Ride(PULocationID=236, DOLocationID=147, trip_distance=4.3, total_amount=29.2, tpep_pickup_datetime=1761956395000)\n", "Sent: Ride(PULocationID=231, DOLocationID=137, trip_distance=3.0, total_amount=32.75, tpep_pickup_datetime=1761957955000)\n", "Sent: Ride(PULocationID=237, DOLocationID=237, trip_distance=0.69, total_amount=11.5, tpep_pickup_datetime=1761955872000)\n", "Sent: Ride(PULocationID=132, DOLocationID=265, trip_distance=15.47, total_amount=106.63, tpep_pickup_datetime=1761955521000)\n", "Sent: Ride(PULocationID=79, DOLocationID=125, trip_distance=1.29, total_amount=22.26, tpep_pickup_datetime=1761955760000)\n", "Sent: Ride(PULocationID=158, DOLocationID=79, trip_distance=1.66, total_amount=32.34, tpep_pickup_datetime=1761957539000)\n", "Sent: Ride(PULocationID=79, DOLocationID=90, trip_distance=1.25, total_amount=22.25, tpep_pickup_datetime=1761958533000)\n", "Sent: Ride(PULocationID=142, DOLocationID=249, trip_distance=2.68, total_amount=48.68, tpep_pickup_datetime=1761956184000)\n", "Sent: Ride(PULocationID=4, DOLocationID=48, trip_distance=3.16, total_amount=33.15, tpep_pickup_datetime=1761958409000)\n", "Sent: Ride(PULocationID=48, DOLocationID=24, trip_distance=2.8, total_amount=24.55, tpep_pickup_datetime=1761956650000)\n", "Sent: Ride(PULocationID=143, DOLocationID=169, trip_distance=7.45, total_amount=44.04, tpep_pickup_datetime=1761956178000)\n", "Sent: Ride(PULocationID=140, DOLocationID=142, trip_distance=2.02, total_amount=17.8, tpep_pickup_datetime=1761957454000)\n", "Sent: Ride(PULocationID=107, DOLocationID=90, trip_distance=3.46, total_amount=35.7, tpep_pickup_datetime=1761957360000)\n", "Sent: Ride(PULocationID=50, DOLocationID=263, trip_distance=2.89, total_amount=26.46, tpep_pickup_datetime=1761956052000)\n", "Sent: Ride(PULocationID=234, DOLocationID=68, trip_distance=1.2, total_amount=27.3, tpep_pickup_datetime=1761956041000)\n", "Sent: Ride(PULocationID=68, DOLocationID=68, trip_distance=0.2, total_amount=13.02, tpep_pickup_datetime=1761957285000)\n", "Sent: Ride(PULocationID=68, DOLocationID=233, trip_distance=2.57, total_amount=26.15, tpep_pickup_datetime=1761957826000)\n", "Sent: Ride(PULocationID=140, DOLocationID=233, trip_distance=1.4, total_amount=24.75, tpep_pickup_datetime=1761956818000)\n", "Sent: Ride(PULocationID=233, DOLocationID=170, trip_distance=0.3, total_amount=13.0, tpep_pickup_datetime=1761957934000)\n", "Sent: Ride(PULocationID=170, DOLocationID=137, trip_distance=0.7, total_amount=17.7, tpep_pickup_datetime=1761958462000)\n", "Sent: Ride(PULocationID=141, DOLocationID=236, trip_distance=0.92, total_amount=13.8, tpep_pickup_datetime=1761956649000)\n", "Sent: Ride(PULocationID=43, DOLocationID=151, trip_distance=2.21, total_amount=17.4, tpep_pickup_datetime=1761957030000)\n", "Sent: Ride(PULocationID=151, DOLocationID=116, trip_distance=2.62, total_amount=21.75, tpep_pickup_datetime=1761957624000)\n", "Sent: Ride(PULocationID=164, DOLocationID=114, trip_distance=1.58, total_amount=35.25, tpep_pickup_datetime=1761957104000)\n", "Sent: Ride(PULocationID=138, DOLocationID=74, trip_distance=6.51, total_amount=52.08, tpep_pickup_datetime=1761958025000)\n", "Sent: Ride(PULocationID=166, DOLocationID=262, trip_distance=3.19, total_amount=27.24, tpep_pickup_datetime=1761956016000)\n", "Sent: Ride(PULocationID=238, DOLocationID=238, trip_distance=0.46, total_amount=12.12, tpep_pickup_datetime=1761956726000)\n", "Sent: Ride(PULocationID=238, DOLocationID=166, trip_distance=1.2, total_amount=17.16, tpep_pickup_datetime=1761957605000)\n", "Sent: Ride(PULocationID=66, DOLocationID=246, trip_distance=4.4, total_amount=30.35, tpep_pickup_datetime=1761955924000)\n", "Sent: Ride(PULocationID=68, DOLocationID=239, trip_distance=3.5, total_amount=29.85, tpep_pickup_datetime=1761958430000)\n", "Sent: Ride(PULocationID=233, DOLocationID=161, trip_distance=0.66, total_amount=14.25, tpep_pickup_datetime=1761956248000)\n", "Sent: Ride(PULocationID=43, DOLocationID=239, trip_distance=1.13, total_amount=16.38, tpep_pickup_datetime=1761956861000)\n", "Sent: Ride(PULocationID=238, DOLocationID=166, trip_distance=1.22, total_amount=13.32, tpep_pickup_datetime=1761957749000)\n", "Sent: Ride(PULocationID=138, DOLocationID=48, trip_distance=11.29, total_amount=81.18, tpep_pickup_datetime=1761958324000)\n", "Sent: Ride(PULocationID=237, DOLocationID=262, trip_distance=0.86, total_amount=14.2, tpep_pickup_datetime=1761955542000)\n", "Sent: Ride(PULocationID=234, DOLocationID=249, trip_distance=1.47, total_amount=24.78, tpep_pickup_datetime=1761958550000)\n", "Sent: Ride(PULocationID=137, DOLocationID=164, trip_distance=0.52, total_amount=20.47, tpep_pickup_datetime=1761955095000)\n", "Sent: Ride(PULocationID=164, DOLocationID=142, trip_distance=3.99, total_amount=38.22, tpep_pickup_datetime=1761955769000)\n", "Sent: Ride(PULocationID=107, DOLocationID=164, trip_distance=1.03, total_amount=16.35, tpep_pickup_datetime=1761955355000)\n", "Sent: Ride(PULocationID=100, DOLocationID=141, trip_distance=2.47, total_amount=27.75, tpep_pickup_datetime=1761956835000)\n", "Sent: Ride(PULocationID=161, DOLocationID=237, trip_distance=1.6, total_amount=18.45, tpep_pickup_datetime=1761958690000)\n", "Sent: Ride(PULocationID=140, DOLocationID=75, trip_distance=2.11, total_amount=20.52, tpep_pickup_datetime=1761955763000)\n", "Sent: Ride(PULocationID=132, DOLocationID=216, trip_distance=4.7, total_amount=24.3, tpep_pickup_datetime=1761957345000)\n", "Sent: Ride(PULocationID=148, DOLocationID=170, trip_distance=2.08, total_amount=31.5, tpep_pickup_datetime=1761958185000)\n", "Sent: Ride(PULocationID=113, DOLocationID=4, trip_distance=0.9, total_amount=19.72, tpep_pickup_datetime=1761957574000)\n", "Sent: Ride(PULocationID=4, DOLocationID=233, trip_distance=2.2, total_amount=22.05, tpep_pickup_datetime=1761958544000)\n", "Sent: Ride(PULocationID=231, DOLocationID=209, trip_distance=1.1, total_amount=19.7, tpep_pickup_datetime=1761958275000)\n", "Sent: Ride(PULocationID=186, DOLocationID=238, trip_distance=4.98, total_amount=45.75, tpep_pickup_datetime=1761955208000)\n", "Sent: Ride(PULocationID=138, DOLocationID=164, trip_distance=9.43, total_amount=59.54, tpep_pickup_datetime=1761955156000)\n", "Sent: Ride(PULocationID=162, DOLocationID=141, trip_distance=1.14, total_amount=17.22, tpep_pickup_datetime=1761957361000)\n", "Sent: Ride(PULocationID=158, DOLocationID=68, trip_distance=1.43, total_amount=23.21, tpep_pickup_datetime=1761955824000)\n", "Sent: Ride(PULocationID=90, DOLocationID=68, trip_distance=0.36, total_amount=12.15, tpep_pickup_datetime=1761956775000)\n", "Sent: Ride(PULocationID=186, DOLocationID=186, trip_distance=0.01, total_amount=-10.85, tpep_pickup_datetime=1761957695000)\n", "Sent: Ride(PULocationID=186, DOLocationID=186, trip_distance=0.01, total_amount=10.85, tpep_pickup_datetime=1761957695000)\n", "Sent: Ride(PULocationID=68, DOLocationID=265, trip_distance=4.31, total_amount=90.81, tpep_pickup_datetime=1761958059000)\n", "Sent: Ride(PULocationID=90, DOLocationID=141, trip_distance=3.92, total_amount=42.42, tpep_pickup_datetime=1761956086000)\n", "Sent: Ride(PULocationID=162, DOLocationID=173, trip_distance=7.98, total_amount=50.25, tpep_pickup_datetime=1761958460000)\n", "Sent: Ride(PULocationID=238, DOLocationID=48, trip_distance=3.03, total_amount=35.44, tpep_pickup_datetime=1761956112000)\n", "Sent: Ride(PULocationID=141, DOLocationID=140, trip_distance=0.54, total_amount=13.86, tpep_pickup_datetime=1761956344000)\n", "Sent: Ride(PULocationID=237, DOLocationID=239, trip_distance=1.46, total_amount=20.5, tpep_pickup_datetime=1761957032000)\n", "Sent: Ride(PULocationID=229, DOLocationID=48, trip_distance=1.8, total_amount=26.45, tpep_pickup_datetime=1761957521000)\n", "Sent: Ride(PULocationID=48, DOLocationID=233, trip_distance=0.91, total_amount=17.85, tpep_pickup_datetime=1761956210000)\n", "Sent: Ride(PULocationID=233, DOLocationID=113, trip_distance=1.61, total_amount=27.3, tpep_pickup_datetime=1761957229000)\n", "Sent: Ride(PULocationID=79, DOLocationID=137, trip_distance=1.33, total_amount=19.25, tpep_pickup_datetime=1761957297000)\n", "Sent: Ride(PULocationID=132, DOLocationID=265, trip_distance=45.7, total_amount=284.39, tpep_pickup_datetime=1761956656000)\n", "Sent: Ride(PULocationID=158, DOLocationID=231, trip_distance=1.75, total_amount=20.55, tpep_pickup_datetime=1761956254000)\n", "Sent: Ride(PULocationID=231, DOLocationID=229, trip_distance=4.32, total_amount=45.78, tpep_pickup_datetime=1761957567000)\n", "Sent: Ride(PULocationID=148, DOLocationID=224, trip_distance=1.99, total_amount=22.26, tpep_pickup_datetime=1761955509000)\n", "Sent: Ride(PULocationID=224, DOLocationID=141, trip_distance=2.85, total_amount=29.82, tpep_pickup_datetime=1761956184000)\n", "Sent: Ride(PULocationID=141, DOLocationID=239, trip_distance=1.32, total_amount=18.84, tpep_pickup_datetime=1761957687000)\n", "Sent: Ride(PULocationID=148, DOLocationID=79, trip_distance=1.0, total_amount=17.75, tpep_pickup_datetime=1761956921000)\n", "Sent: Ride(PULocationID=79, DOLocationID=170, trip_distance=1.29, total_amount=23.75, tpep_pickup_datetime=1761957629000)\n", "Sent: Ride(PULocationID=125, DOLocationID=186, trip_distance=1.81, total_amount=31.15, tpep_pickup_datetime=1761956520000)\n", "Sent: Ride(PULocationID=186, DOLocationID=249, trip_distance=1.14, total_amount=21.25, tpep_pickup_datetime=1761958570000)\n", "Sent: Ride(PULocationID=138, DOLocationID=237, trip_distance=10.18, total_amount=69.71, tpep_pickup_datetime=1761957894000)\n", "Sent: Ride(PULocationID=236, DOLocationID=43, trip_distance=2.16, total_amount=17.1, tpep_pickup_datetime=1761956402000)\n", "Sent: Ride(PULocationID=237, DOLocationID=263, trip_distance=1.7, total_amount=18.0, tpep_pickup_datetime=1761955253000)\n", "Sent: Ride(PULocationID=263, DOLocationID=7, trip_distance=4.5, total_amount=32.34, tpep_pickup_datetime=1761955970000)\n", "Sent: Ride(PULocationID=79, DOLocationID=234, trip_distance=1.4, total_amount=27.55, tpep_pickup_datetime=1761955608000)\n", "Sent: Ride(PULocationID=113, DOLocationID=244, trip_distance=10.5, total_amount=57.05, tpep_pickup_datetime=1761957376000)\n", "Sent: Ride(PULocationID=68, DOLocationID=255, trip_distance=7.32, total_amount=56.35, tpep_pickup_datetime=1761955670000)\n", "Sent: Ride(PULocationID=230, DOLocationID=261, trip_distance=6.49, total_amount=43.85, tpep_pickup_datetime=1761958702000)\n", "Sent: Ride(PULocationID=45, DOLocationID=97, trip_distance=3.2, total_amount=23.45, tpep_pickup_datetime=1761956745000)\n", "Sent: Ride(PULocationID=144, DOLocationID=186, trip_distance=2.9, total_amount=24.85, tpep_pickup_datetime=1761958647000)\n", "Sent: Ride(PULocationID=79, DOLocationID=234, trip_distance=0.97, total_amount=23.94, tpep_pickup_datetime=1761956073000)\n", "Sent: Ride(PULocationID=234, DOLocationID=237, trip_distance=1.73, total_amount=22.26, tpep_pickup_datetime=1761957100000)\n", "Sent: Ride(PULocationID=45, DOLocationID=79, trip_distance=1.32, total_amount=22.26, tpep_pickup_datetime=1761957425000)\n", "Sent: Ride(PULocationID=79, DOLocationID=90, trip_distance=1.64, total_amount=24.72, tpep_pickup_datetime=1761958384000)\n", "Sent: Ride(PULocationID=113, DOLocationID=113, trip_distance=0.6, total_amount=23.1, tpep_pickup_datetime=1761956460000)\n", "Sent: Ride(PULocationID=113, DOLocationID=41, trip_distance=7.6, total_amount=49.4, tpep_pickup_datetime=1761957518000)\n", "Sent: Ride(PULocationID=140, DOLocationID=233, trip_distance=1.1, total_amount=22.3, tpep_pickup_datetime=1761956203000)\n", "Sent: Ride(PULocationID=162, DOLocationID=75, trip_distance=2.7, total_amount=23.2, tpep_pickup_datetime=1761957674000)\n", "Sent: Ride(PULocationID=170, DOLocationID=125, trip_distance=3.43, total_amount=49.14, tpep_pickup_datetime=1761957040000)\n", "Sent: Ride(PULocationID=209, DOLocationID=90, trip_distance=3.28, total_amount=31.15, tpep_pickup_datetime=1761956343000)\n", "Sent: Ride(PULocationID=90, DOLocationID=125, trip_distance=2.08, total_amount=34.86, tpep_pickup_datetime=1761958287000)\n", "Sent: Ride(PULocationID=48, DOLocationID=113, trip_distance=2.01, total_amount=41.58, tpep_pickup_datetime=1761956311000)\n", "Sent: Ride(PULocationID=237, DOLocationID=237, trip_distance=0.15, total_amount=10.44, tpep_pickup_datetime=1761955368000)\n", "Sent: Ride(PULocationID=246, DOLocationID=48, trip_distance=1.4, total_amount=22.75, tpep_pickup_datetime=1761957364000)\n", "Sent: Ride(PULocationID=142, DOLocationID=68, trip_distance=2.4, total_amount=24.8, tpep_pickup_datetime=1761956279000)\n", "Sent: Ride(PULocationID=262, DOLocationID=90, trip_distance=4.3, total_amount=52.5, tpep_pickup_datetime=1761956236000)\n", "Sent: Ride(PULocationID=68, DOLocationID=246, trip_distance=0.72, total_amount=13.86, tpep_pickup_datetime=1761956524000)\n", "Sent: Ride(PULocationID=68, DOLocationID=68, trip_distance=0.38, total_amount=12.55, tpep_pickup_datetime=1761957465000)\n", "Sent: Ride(PULocationID=68, DOLocationID=68, trip_distance=0.95, total_amount=17.22, tpep_pickup_datetime=1761958160000)\n", "Sent: Ride(PULocationID=68, DOLocationID=236, trip_distance=4.25, total_amount=46.62, tpep_pickup_datetime=1761955309000)\n", "Sent: Ride(PULocationID=236, DOLocationID=75, trip_distance=0.83, total_amount=11.5, tpep_pickup_datetime=1761955738000)\n", "Sent: Ride(PULocationID=141, DOLocationID=137, trip_distance=2.0, total_amount=20.75, tpep_pickup_datetime=1761957902000)\n", "Sent: Ride(PULocationID=140, DOLocationID=140, trip_distance=0.49, total_amount=10.1, tpep_pickup_datetime=1761956499000)\n", "Sent: Ride(PULocationID=263, DOLocationID=238, trip_distance=1.3, total_amount=17.16, tpep_pickup_datetime=1761957396000)\n", "Sent: Ride(PULocationID=142, DOLocationID=249, trip_distance=3.49, total_amount=30.05, tpep_pickup_datetime=1761958009000)\n", "Sent: Ride(PULocationID=87, DOLocationID=229, trip_distance=6.31, total_amount=40.74, tpep_pickup_datetime=1761958328000)\n", "Sent: Ride(PULocationID=50, DOLocationID=68, trip_distance=1.9, total_amount=20.55, tpep_pickup_datetime=1761955870000)\n", "Sent: Ride(PULocationID=87, DOLocationID=49, trip_distance=3.71, total_amount=25.55, tpep_pickup_datetime=1761957107000)\n", "Sent: Ride(PULocationID=97, DOLocationID=256, trip_distance=3.39, total_amount=27.6, tpep_pickup_datetime=1761958421000)\n", "Sent: Ride(PULocationID=148, DOLocationID=236, trip_distance=5.79, total_amount=40.95, tpep_pickup_datetime=1761955769000)\n", "Sent: Ride(PULocationID=236, DOLocationID=143, trip_distance=3.23, total_amount=23.7, tpep_pickup_datetime=1761957945000)\n", "Sent: Ride(PULocationID=231, DOLocationID=232, trip_distance=1.6, total_amount=19.25, tpep_pickup_datetime=1761956045000)\n", "Sent: Ride(PULocationID=141, DOLocationID=229, trip_distance=0.7, total_amount=13.85, tpep_pickup_datetime=1761956383000)\n", "Sent: Ride(PULocationID=263, DOLocationID=262, trip_distance=0.3, total_amount=13.4, tpep_pickup_datetime=1761957107000)\n", "Sent: Ride(PULocationID=263, DOLocationID=263, trip_distance=0.7, total_amount=12.8, tpep_pickup_datetime=1761957502000)\n", "Sent: Ride(PULocationID=263, DOLocationID=229, trip_distance=1.7, total_amount=18.9, tpep_pickup_datetime=1761958191000)\n", "Sent: Ride(PULocationID=107, DOLocationID=148, trip_distance=1.04, total_amount=24.78, tpep_pickup_datetime=1761958597000)\n", "Sent: Ride(PULocationID=233, DOLocationID=265, trip_distance=0.0, total_amount=124.25, tpep_pickup_datetime=1761956790000)\n", "Sent: Ride(PULocationID=79, DOLocationID=229, trip_distance=2.57, total_amount=28.14, tpep_pickup_datetime=1761956518000)\n", "Sent: Ride(PULocationID=170, DOLocationID=48, trip_distance=1.06, total_amount=19.57, tpep_pickup_datetime=1761956704000)\n", "Sent: Ride(PULocationID=68, DOLocationID=68, trip_distance=0.32, total_amount=13.02, tpep_pickup_datetime=1761955678000)\n", "Sent: Ride(PULocationID=68, DOLocationID=162, trip_distance=1.73, total_amount=26.46, tpep_pickup_datetime=1761956006000)\n", "Sent: Ride(PULocationID=74, DOLocationID=236, trip_distance=1.55, total_amount=17.16, tpep_pickup_datetime=1761958603000)\n", "Sent: Ride(PULocationID=140, DOLocationID=230, trip_distance=2.16, total_amount=33.69, tpep_pickup_datetime=1761955305000)\n", "Sent: Ride(PULocationID=158, DOLocationID=231, trip_distance=1.99, total_amount=31.5, tpep_pickup_datetime=1761955569000)\n", "Sent: Ride(PULocationID=231, DOLocationID=148, trip_distance=1.04, total_amount=19.95, tpep_pickup_datetime=1761956993000)\n", "Sent: Ride(PULocationID=148, DOLocationID=170, trip_distance=1.53, total_amount=19.95, tpep_pickup_datetime=1761958077000)\n", "Sent: Ride(PULocationID=263, DOLocationID=234, trip_distance=2.63, total_amount=25.62, tpep_pickup_datetime=1761958784000)\n", "Sent: Ride(PULocationID=148, DOLocationID=141, trip_distance=4.28, total_amount=38.22, tpep_pickup_datetime=1761958147000)\n", "Sent: Ride(PULocationID=113, DOLocationID=230, trip_distance=3.19, total_amount=42.42, tpep_pickup_datetime=1761955883000)\n", "Sent: Ride(PULocationID=48, DOLocationID=246, trip_distance=1.38, total_amount=18.9, tpep_pickup_datetime=1761958302000)\n", "Sent: Ride(PULocationID=261, DOLocationID=186, trip_distance=3.2, total_amount=34.0, tpep_pickup_datetime=1761957372000)\n", "Sent: Ride(PULocationID=158, DOLocationID=68, trip_distance=0.85, total_amount=26.81, tpep_pickup_datetime=1761955469000)\n", "Sent: Ride(PULocationID=68, DOLocationID=237, trip_distance=3.15, total_amount=33.8, tpep_pickup_datetime=1761956993000)\n", "Sent: Ride(PULocationID=170, DOLocationID=114, trip_distance=1.4, total_amount=31.5, tpep_pickup_datetime=1761955287000)\n", "Sent: Ride(PULocationID=114, DOLocationID=230, trip_distance=2.8, total_amount=31.05, tpep_pickup_datetime=1761956826000)\n", "Sent: Ride(PULocationID=163, DOLocationID=43, trip_distance=0.55, total_amount=13.02, tpep_pickup_datetime=1761955330000)\n", "Sent: Ride(PULocationID=142, DOLocationID=143, trip_distance=0.7, total_amount=13.8, tpep_pickup_datetime=1761955615000)\n", "Sent: Ride(PULocationID=87, DOLocationID=141, trip_distance=6.07, total_amount=34.55, tpep_pickup_datetime=1761956252000)\n", "Sent: Ride(PULocationID=148, DOLocationID=186, trip_distance=2.8, total_amount=33.18, tpep_pickup_datetime=1761956258000)\n", "Sent: Ride(PULocationID=186, DOLocationID=48, trip_distance=1.54, total_amount=21.35, tpep_pickup_datetime=1761958653000)\n", "Sent: Ride(PULocationID=239, DOLocationID=48, trip_distance=1.3, total_amount=17.35, tpep_pickup_datetime=1761957861000)\n", "Sent: Ride(PULocationID=48, DOLocationID=141, trip_distance=1.6, total_amount=23.95, tpep_pickup_datetime=1761958511000)\n", "Sent: Ride(PULocationID=234, DOLocationID=236, trip_distance=4.26, total_amount=39.9, tpep_pickup_datetime=1761956321000)\n", "Sent: Ride(PULocationID=141, DOLocationID=164, trip_distance=2.85, total_amount=28.14, tpep_pickup_datetime=1761958363000)\n", "Sent: Ride(PULocationID=113, DOLocationID=140, trip_distance=3.86, total_amount=42.42, tpep_pickup_datetime=1761956960000)\n", "Sent: Ride(PULocationID=263, DOLocationID=68, trip_distance=4.44, total_amount=34.02, tpep_pickup_datetime=1761958780000)\n", "Sent: Ride(PULocationID=48, DOLocationID=234, trip_distance=2.0, total_amount=28.45, tpep_pickup_datetime=1761956085000)\n", "Sent: Ride(PULocationID=234, DOLocationID=48, trip_distance=2.1, total_amount=37.2, tpep_pickup_datetime=1761957500000)\n", "Sent: Ride(PULocationID=140, DOLocationID=159, trip_distance=4.96, total_amount=30.4, tpep_pickup_datetime=1761956905000)\n", "Sent: Ride(PULocationID=170, DOLocationID=237, trip_distance=1.48, total_amount=21.42, tpep_pickup_datetime=1761955928000)\n", "Sent: Ride(PULocationID=79, DOLocationID=137, trip_distance=1.0, total_amount=16.45, tpep_pickup_datetime=1761956435000)\n", "Sent: Ride(PULocationID=79, DOLocationID=137, trip_distance=0.6, total_amount=13.85, tpep_pickup_datetime=1761957093000)\n", "Sent: Ride(PULocationID=170, DOLocationID=229, trip_distance=1.5, total_amount=24.75, tpep_pickup_datetime=1761957404000)\n", "Sent: Ride(PULocationID=263, DOLocationID=224, trip_distance=4.26, total_amount=50.85, tpep_pickup_datetime=1761955719000)\n", "Sent: Ride(PULocationID=224, DOLocationID=233, trip_distance=1.07, total_amount=17.22, tpep_pickup_datetime=1761958748000)\n", "Sent: Ride(PULocationID=249, DOLocationID=162, trip_distance=2.38, total_amount=34.86, tpep_pickup_datetime=1761956435000)\n", "Sent: Ride(PULocationID=162, DOLocationID=24, trip_distance=3.81, total_amount=28.55, tpep_pickup_datetime=1761958415000)\n", "Sent: Ride(PULocationID=246, DOLocationID=246, trip_distance=0.77, total_amount=15.05, tpep_pickup_datetime=1761956032000)\n", "Sent: Ride(PULocationID=262, DOLocationID=262, trip_distance=0.0, total_amount=8.0, tpep_pickup_datetime=1761956477000)\n", "Sent: Ride(PULocationID=263, DOLocationID=75, trip_distance=0.8, total_amount=13.5, tpep_pickup_datetime=1761957782000)\n", "Sent: Ride(PULocationID=236, DOLocationID=238, trip_distance=2.0, total_amount=18.84, tpep_pickup_datetime=1761958251000)\n", "Sent: Ride(PULocationID=230, DOLocationID=141, trip_distance=1.78, total_amount=18.09, tpep_pickup_datetime=1761955958000)\n", "Sent: Ride(PULocationID=237, DOLocationID=74, trip_distance=2.49, total_amount=21.6, tpep_pickup_datetime=1761956602000)\n", "Sent: Ride(PULocationID=231, DOLocationID=87, trip_distance=2.61, total_amount=24.78, tpep_pickup_datetime=1761958149000)\n", "Sent: Ride(PULocationID=161, DOLocationID=237, trip_distance=1.84, total_amount=17.15, tpep_pickup_datetime=1761955513000)\n", "Sent: Ride(PULocationID=237, DOLocationID=262, trip_distance=1.05, total_amount=15.48, tpep_pickup_datetime=1761956227000)\n", "Sent: Ride(PULocationID=163, DOLocationID=163, trip_distance=0.45, total_amount=15.54, tpep_pickup_datetime=1761954619000)\n", "Sent: Ride(PULocationID=113, DOLocationID=68, trip_distance=1.8, total_amount=49.98, tpep_pickup_datetime=1761955361000)\n", "Sent: Ride(PULocationID=246, DOLocationID=114, trip_distance=1.88, total_amount=31.94, tpep_pickup_datetime=1761958526000)\n", "Sent: Ride(PULocationID=164, DOLocationID=87, trip_distance=4.4, total_amount=57.3, tpep_pickup_datetime=1761956034000)\n", "Sent: Ride(PULocationID=230, DOLocationID=237, trip_distance=0.89, total_amount=15.39, tpep_pickup_datetime=1761958187000)\n", "Sent: Ride(PULocationID=231, DOLocationID=229, trip_distance=4.88, total_amount=43.73, tpep_pickup_datetime=1761958624000)\n", "Sent: Ride(PULocationID=239, DOLocationID=151, trip_distance=1.0, total_amount=14.6, tpep_pickup_datetime=1761956087000)\n", "Sent: Ride(PULocationID=48, DOLocationID=261, trip_distance=5.0, total_amount=50.15, tpep_pickup_datetime=1761957798000)\n", "Sent: Ride(PULocationID=148, DOLocationID=148, trip_distance=0.39, total_amount=23.8, tpep_pickup_datetime=1761955384000)\n", "Sent: Ride(PULocationID=148, DOLocationID=223, trip_distance=8.64, total_amount=60.06, tpep_pickup_datetime=1761956689000)\n", "Sent: Ride(PULocationID=256, DOLocationID=107, trip_distance=3.25, total_amount=40.35, tpep_pickup_datetime=1761955754000)\n", "Sent: Ride(PULocationID=107, DOLocationID=68, trip_distance=1.47, total_amount=18.55, tpep_pickup_datetime=1761957878000)\n", "Sent: Ride(PULocationID=161, DOLocationID=7, trip_distance=4.12, total_amount=31.8, tpep_pickup_datetime=1761955283000)\n", "Sent: Ride(PULocationID=138, DOLocationID=113, trip_distance=9.36, total_amount=75.5, tpep_pickup_datetime=1761957063000)\n", "Sent: Ride(PULocationID=90, DOLocationID=158, trip_distance=0.96, total_amount=23.94, tpep_pickup_datetime=1761957835000)\n", "Sent: Ride(PULocationID=237, DOLocationID=230, trip_distance=1.28, total_amount=20.56, tpep_pickup_datetime=1761955072000)\n", "Sent: Ride(PULocationID=100, DOLocationID=107, trip_distance=1.7, total_amount=24.95, tpep_pickup_datetime=1761956114000)\n", "Sent: Ride(PULocationID=107, DOLocationID=137, trip_distance=0.64, total_amount=13.63, tpep_pickup_datetime=1761957625000)\n", "Sent: Ride(PULocationID=170, DOLocationID=107, trip_distance=1.17, total_amount=27.3, tpep_pickup_datetime=1761958042000)\n", "Sent: Ride(PULocationID=79, DOLocationID=68, trip_distance=3.48, total_amount=37.38, tpep_pickup_datetime=1761956556000)\n", "Sent: Ride(PULocationID=161, DOLocationID=75, trip_distance=3.25, total_amount=25.62, tpep_pickup_datetime=1761955328000)\n", "Sent: Ride(PULocationID=43, DOLocationID=237, trip_distance=1.06, total_amount=18.35, tpep_pickup_datetime=1761956831000)\n", "Sent: Ride(PULocationID=43, DOLocationID=236, trip_distance=2.03, total_amount=22.26, tpep_pickup_datetime=1761957676000)\n", "Sent: Ride(PULocationID=68, DOLocationID=246, trip_distance=0.54, total_amount=16.55, tpep_pickup_datetime=1761955938000)\n", "Sent: Ride(PULocationID=246, DOLocationID=236, trip_distance=3.49, total_amount=35.7, tpep_pickup_datetime=1761956552000)\n", "Sent: Ride(PULocationID=75, DOLocationID=238, trip_distance=1.23, total_amount=12.9, tpep_pickup_datetime=1761958460000)\n", "Sent: Ride(PULocationID=138, DOLocationID=229, trip_distance=10.1, total_amount=72.24, tpep_pickup_datetime=1761956382000)\n", "Sent: Ride(PULocationID=162, DOLocationID=230, trip_distance=1.0, total_amount=17.15, tpep_pickup_datetime=1761958646000)\n", "Sent: Ride(PULocationID=132, DOLocationID=177, trip_distance=9.77, total_amount=54.55, tpep_pickup_datetime=1761957231000)\n", "Sent: Ride(PULocationID=239, DOLocationID=262, trip_distance=1.9, total_amount=21.35, tpep_pickup_datetime=1761958531000)\n", "Sent: Ride(PULocationID=125, DOLocationID=239, trip_distance=4.72, total_amount=47.46, tpep_pickup_datetime=1761955991000)\n", "Sent: Ride(PULocationID=239, DOLocationID=262, trip_distance=2.02, total_amount=20.5, tpep_pickup_datetime=1761958263000)\n", "Sent: Ride(PULocationID=224, DOLocationID=231, trip_distance=3.16, total_amount=27.65, tpep_pickup_datetime=1761957209000)\n", "Sent: Ride(PULocationID=239, DOLocationID=90, trip_distance=3.67, total_amount=34.02, tpep_pickup_datetime=1761956212000)\n", "Sent: Ride(PULocationID=90, DOLocationID=90, trip_distance=0.18, total_amount=19.74, tpep_pickup_datetime=1761957726000)\n", "Sent: Ride(PULocationID=90, DOLocationID=79, trip_distance=0.98, total_amount=27.3, tpep_pickup_datetime=1761958631000)\n", "Sent: Ride(PULocationID=263, DOLocationID=90, trip_distance=4.7, total_amount=40.74, tpep_pickup_datetime=1761956974000)\n", "Sent: Ride(PULocationID=249, DOLocationID=68, trip_distance=0.85, total_amount=19.55, tpep_pickup_datetime=1761955241000)\n", "Sent: Ride(PULocationID=186, DOLocationID=161, trip_distance=1.37, total_amount=24.94, tpep_pickup_datetime=1761956360000)\n", "Sent: Ride(PULocationID=231, DOLocationID=232, trip_distance=2.09, total_amount=22.26, tpep_pickup_datetime=1761956603000)\n", "Sent: Ride(PULocationID=142, DOLocationID=78, trip_distance=8.9, total_amount=35.0, tpep_pickup_datetime=1761957013000)\n", "Sent: Ride(PULocationID=148, DOLocationID=236, trip_distance=7.09, total_amount=43.65, tpep_pickup_datetime=1761955507000)\n", "Sent: Ride(PULocationID=237, DOLocationID=239, trip_distance=2.0, total_amount=18.81, tpep_pickup_datetime=1761957857000)\n", "Sent: Ride(PULocationID=239, DOLocationID=50, trip_distance=1.81, total_amount=20.58, tpep_pickup_datetime=1761958528000)\n", "Sent: Ride(PULocationID=148, DOLocationID=113, trip_distance=1.25, total_amount=24.78, tpep_pickup_datetime=1761956381000)\n", "Sent: Ride(PULocationID=113, DOLocationID=107, trip_distance=1.44, total_amount=24.78, tpep_pickup_datetime=1761957581000)\n", "Sent: Ride(PULocationID=50, DOLocationID=90, trip_distance=2.4, total_amount=33.2, tpep_pickup_datetime=1761955786000)\n", "Sent: Ride(PULocationID=90, DOLocationID=68, trip_distance=0.6, total_amount=17.05, tpep_pickup_datetime=1761957365000)\n", "Sent: Ride(PULocationID=246, DOLocationID=68, trip_distance=0.8, total_amount=16.75, tpep_pickup_datetime=1761958508000)\n", "Sent: Ride(PULocationID=249, DOLocationID=246, trip_distance=2.2, total_amount=27.3, tpep_pickup_datetime=1761955399000)\n", "Sent: Ride(PULocationID=48, DOLocationID=90, trip_distance=1.5, total_amount=18.45, tpep_pickup_datetime=1761957057000)\n", "Sent: Ride(PULocationID=90, DOLocationID=107, trip_distance=1.1, total_amount=24.8, tpep_pickup_datetime=1761957738000)\n", "Sent: Ride(PULocationID=148, DOLocationID=234, trip_distance=2.5, total_amount=34.0, tpep_pickup_datetime=1761956224000)\n", "Sent: Ride(PULocationID=48, DOLocationID=186, trip_distance=1.2, total_amount=18.9, tpep_pickup_datetime=1761955938000)\n", "Sent: Ride(PULocationID=234, DOLocationID=166, trip_distance=6.1, total_amount=43.3, tpep_pickup_datetime=1761957437000)\n", "Sent: Ride(PULocationID=238, DOLocationID=239, trip_distance=0.7, total_amount=15.5, tpep_pickup_datetime=1761956589000)\n", "Sent: Ride(PULocationID=90, DOLocationID=113, trip_distance=0.37, total_amount=17.15, tpep_pickup_datetime=1761957323000)\n", "Sent: Ride(PULocationID=79, DOLocationID=90, trip_distance=1.2, total_amount=24.8, tpep_pickup_datetime=1761956590000)\n", "Sent: Ride(PULocationID=234, DOLocationID=107, trip_distance=1.0, total_amount=20.65, tpep_pickup_datetime=1761957748000)\n", "Sent: Ride(PULocationID=79, DOLocationID=90, trip_distance=1.34, total_amount=24.15, tpep_pickup_datetime=1761958718000)\n", "Sent: Ride(PULocationID=231, DOLocationID=137, trip_distance=2.4, total_amount=27.78, tpep_pickup_datetime=1761956073000)\n", "Sent: Ride(PULocationID=100, DOLocationID=232, trip_distance=3.1, total_amount=34.85, tpep_pickup_datetime=1761955678000)\n", "Sent: Ride(PULocationID=232, DOLocationID=263, trip_distance=5.4, total_amount=36.55, tpep_pickup_datetime=1761957916000)\n", "Sent: Ride(PULocationID=236, DOLocationID=43, trip_distance=0.51, total_amount=12.96, tpep_pickup_datetime=1761957600000)\n", "Sent: Ride(PULocationID=237, DOLocationID=237, trip_distance=1.02, total_amount=17.88, tpep_pickup_datetime=1761958142000)\n", "Sent: Ride(PULocationID=230, DOLocationID=75, trip_distance=3.62, total_amount=27.3, tpep_pickup_datetime=1761957421000)\n", "Sent: Ride(PULocationID=263, DOLocationID=141, trip_distance=0.54, total_amount=11.4, tpep_pickup_datetime=1761957775000)\n", "Sent: Ride(PULocationID=162, DOLocationID=140, trip_distance=0.89, total_amount=13.02, tpep_pickup_datetime=1761958544000)\n", "Sent: Ride(PULocationID=246, DOLocationID=17, trip_distance=7.47, total_amount=65.94, tpep_pickup_datetime=1761956096000)\n", "Sent: Ride(PULocationID=79, DOLocationID=170, trip_distance=1.8, total_amount=23.94, tpep_pickup_datetime=1761956686000)\n", "Sent: Ride(PULocationID=148, DOLocationID=229, trip_distance=3.43, total_amount=34.02, tpep_pickup_datetime=1761955666000)\n", "Sent: Ride(PULocationID=229, DOLocationID=262, trip_distance=1.62, total_amount=17.05, tpep_pickup_datetime=1761957259000)\n", "Sent: Ride(PULocationID=262, DOLocationID=237, trip_distance=1.32, total_amount=18.0, tpep_pickup_datetime=1761957709000)\n", "Sent: Ride(PULocationID=237, DOLocationID=263, trip_distance=0.9, total_amount=16.32, tpep_pickup_datetime=1761958396000)\n", "Sent: Ride(PULocationID=4, DOLocationID=107, trip_distance=1.3, total_amount=19.7, tpep_pickup_datetime=1761957455000)\n", "Sent: Ride(PULocationID=107, DOLocationID=170, trip_distance=0.93, total_amount=16.38, tpep_pickup_datetime=1761955645000)\n", "Sent: Ride(PULocationID=249, DOLocationID=164, trip_distance=1.12, total_amount=24.15, tpep_pickup_datetime=1761955455000)\n", "Sent: Ride(PULocationID=236, DOLocationID=236, trip_distance=0.39, total_amount=11.11, tpep_pickup_datetime=1761957759000)\n", "Sent: Ride(PULocationID=79, DOLocationID=231, trip_distance=1.82, total_amount=24.95, tpep_pickup_datetime=1761955330000)\n", "Sent: Ride(PULocationID=231, DOLocationID=162, trip_distance=6.21, total_amount=43.05, tpep_pickup_datetime=1761956607000)\n", "Sent: Ride(PULocationID=237, DOLocationID=237, trip_distance=0.27, total_amount=11.75, tpep_pickup_datetime=1761956212000)\n", "Sent: Ride(PULocationID=263, DOLocationID=43, trip_distance=1.46, total_amount=15.0, tpep_pickup_datetime=1761958020000)\n", "Sent: Ride(PULocationID=237, DOLocationID=141, trip_distance=0.5, total_amount=14.95, tpep_pickup_datetime=1761956740000)\n", "Sent: Ride(PULocationID=234, DOLocationID=87, trip_distance=4.1, total_amount=51.35, tpep_pickup_datetime=1761958460000)\n", "Sent: Ride(PULocationID=132, DOLocationID=116, trip_distance=18.5, total_amount=96.24, tpep_pickup_datetime=1761957126000)\n", "Sent: Ride(PULocationID=138, DOLocationID=17, trip_distance=9.96, total_amount=61.39, tpep_pickup_datetime=1761957438000)\n", "Sent: Ride(PULocationID=231, DOLocationID=170, trip_distance=4.21, total_amount=34.55, tpep_pickup_datetime=1761957583000)\n", "Sent: Ride(PULocationID=4, DOLocationID=263, trip_distance=3.7, total_amount=31.5, tpep_pickup_datetime=1761956232000)\n", "Sent: Ride(PULocationID=263, DOLocationID=263, trip_distance=0.0, total_amount=8.0, tpep_pickup_datetime=1761957635000)\n", "Sent: Ride(PULocationID=234, DOLocationID=229, trip_distance=1.9, total_amount=27.3, tpep_pickup_datetime=1761958640000)\n", "Sent: Ride(PULocationID=68, DOLocationID=164, trip_distance=1.05, total_amount=15.75, tpep_pickup_datetime=1761956099000)\n", "Sent: Ride(PULocationID=164, DOLocationID=170, trip_distance=1.13, total_amount=15.75, tpep_pickup_datetime=1761956710000)\n", "Sent: Ride(PULocationID=170, DOLocationID=256, trip_distance=5.72, total_amount=60.9, tpep_pickup_datetime=1761957394000)\n", "Sent: Ride(PULocationID=255, DOLocationID=49, trip_distance=4.19, total_amount=46.08, tpep_pickup_datetime=1761956379000)\n", "Sent: Ride(PULocationID=170, DOLocationID=148, trip_distance=2.1, total_amount=29.76, tpep_pickup_datetime=1761955716000)\n", "Sent: Ride(PULocationID=144, DOLocationID=249, trip_distance=1.3, total_amount=18.9, tpep_pickup_datetime=1761957598000)\n", "Sent: Ride(PULocationID=249, DOLocationID=87, trip_distance=2.2, total_amount=26.45, tpep_pickup_datetime=1761958322000)\n", "Sent: Ride(PULocationID=79, DOLocationID=79, trip_distance=0.35, total_amount=15.54, tpep_pickup_datetime=1761956616000)\n", "Sent: Ride(PULocationID=164, DOLocationID=246, trip_distance=1.21, total_amount=15.75, tpep_pickup_datetime=1761955343000)\n", "Sent: Ride(PULocationID=246, DOLocationID=246, trip_distance=0.83, total_amount=19.95, tpep_pickup_datetime=1761955909000)\n", "Sent: Ride(PULocationID=79, DOLocationID=263, trip_distance=4.38, total_amount=34.02, tpep_pickup_datetime=1761958565000)\n", "Sent: Ride(PULocationID=132, DOLocationID=107, trip_distance=16.64, total_amount=93.44, tpep_pickup_datetime=1761956975000)\n", "Sent: Ride(PULocationID=107, DOLocationID=233, trip_distance=1.44, total_amount=18.45, tpep_pickup_datetime=1761955379000)\n", "Sent: Ride(PULocationID=79, DOLocationID=246, trip_distance=2.65, total_amount=35.55, tpep_pickup_datetime=1761956312000)\n", "Sent: Ride(PULocationID=233, DOLocationID=229, trip_distance=0.71, total_amount=13.86, tpep_pickup_datetime=1761958144000)\n", "Sent: Ride(PULocationID=48, DOLocationID=79, trip_distance=3.44, total_amount=40.74, tpep_pickup_datetime=1761957545000)\n", "Sent: Ride(PULocationID=68, DOLocationID=90, trip_distance=0.6, total_amount=17.22, tpep_pickup_datetime=1761955431000)\n", "Sent: Ride(PULocationID=90, DOLocationID=68, trip_distance=1.1, total_amount=19.85, tpep_pickup_datetime=1761956413000)\n", "Sent: Ride(PULocationID=107, DOLocationID=62, trip_distance=8.16, total_amount=53.55, tpep_pickup_datetime=1761956385000)\n", "Sent: Ride(PULocationID=234, DOLocationID=79, trip_distance=1.11, total_amount=26.46, tpep_pickup_datetime=1761955381000)\n", "Sent: Ride(PULocationID=79, DOLocationID=4, trip_distance=0.77, total_amount=21.42, tpep_pickup_datetime=1761956586000)\n", "Sent: Ride(PULocationID=4, DOLocationID=107, trip_distance=1.11, total_amount=18.06, tpep_pickup_datetime=1761957433000)\n", "Sent: Ride(PULocationID=79, DOLocationID=68, trip_distance=2.16, total_amount=28.95, tpep_pickup_datetime=1761956224000)\n", "Sent: Ride(PULocationID=68, DOLocationID=90, trip_distance=0.8, total_amount=19.74, tpep_pickup_datetime=1761957722000)\n", "Sent: Ride(PULocationID=90, DOLocationID=246, trip_distance=1.77, total_amount=21.95, tpep_pickup_datetime=1761958441000)\n", "Sent: Ride(PULocationID=87, DOLocationID=79, trip_distance=2.87, total_amount=34.02, tpep_pickup_datetime=1761957364000)\n", "Sent: Ride(PULocationID=43, DOLocationID=140, trip_distance=2.05, total_amount=17.15, tpep_pickup_datetime=1761956821000)\n", "Sent: Ride(PULocationID=263, DOLocationID=263, trip_distance=0.6, total_amount=13.8, tpep_pickup_datetime=1761957600000)\n", "Sent: Ride(PULocationID=43, DOLocationID=249, trip_distance=3.17, total_amount=35.82, tpep_pickup_datetime=1761958522000)\n", "Sent: Ride(PULocationID=107, DOLocationID=170, trip_distance=0.7, total_amount=17.75, tpep_pickup_datetime=1761955603000)\n", "Sent: Ride(PULocationID=138, DOLocationID=162, trip_distance=8.9, total_amount=64.74, tpep_pickup_datetime=1761958130000)\n", "Sent: Ride(PULocationID=230, DOLocationID=142, trip_distance=1.9, total_amount=23.9, tpep_pickup_datetime=1761956855000)\n", "Sent: Ride(PULocationID=239, DOLocationID=41, trip_distance=2.4, total_amount=22.2, tpep_pickup_datetime=1761958017000)\n", "Sent: Ride(PULocationID=166, DOLocationID=151, trip_distance=0.71, total_amount=11.7, tpep_pickup_datetime=1761955955000)\n", "Sent: Ride(PULocationID=166, DOLocationID=243, trip_distance=5.58, total_amount=34.32, tpep_pickup_datetime=1761956353000)\n", "Sent: Ride(PULocationID=236, DOLocationID=236, trip_distance=0.64, total_amount=12.12, tpep_pickup_datetime=1761955402000)\n", "Sent: Ride(PULocationID=236, DOLocationID=170, trip_distance=3.58, total_amount=34.02, tpep_pickup_datetime=1761955588000)\n", "Sent: Ride(PULocationID=161, DOLocationID=236, trip_distance=1.3, total_amount=17.2, tpep_pickup_datetime=1761956232000)\n", "Sent: Ride(PULocationID=231, DOLocationID=74, trip_distance=7.55, total_amount=64.26, tpep_pickup_datetime=1761958722000)\n", "Sent: Ride(PULocationID=246, DOLocationID=90, trip_distance=1.0, total_amount=24.78, tpep_pickup_datetime=1761955226000)\n", "Sent: Ride(PULocationID=90, DOLocationID=137, trip_distance=1.51, total_amount=26.95, tpep_pickup_datetime=1761956284000)\n", "Sent: Ride(PULocationID=162, DOLocationID=263, trip_distance=2.0, total_amount=19.15, tpep_pickup_datetime=1761958009000)\n", "Sent: Ride(PULocationID=234, DOLocationID=170, trip_distance=1.0, total_amount=19.74, tpep_pickup_datetime=1761955806000)\n", "Sent: Ride(PULocationID=170, DOLocationID=233, trip_distance=0.15, total_amount=19.75, tpep_pickup_datetime=1761956644000)\n", "Sent: Ride(PULocationID=113, DOLocationID=256, trip_distance=4.57, total_amount=45.05, tpep_pickup_datetime=1761955504000)\n", "Sent: Ride(PULocationID=237, DOLocationID=161, trip_distance=0.67, total_amount=13.02, tpep_pickup_datetime=1761956393000)\n", "Sent: Ride(PULocationID=158, DOLocationID=158, trip_distance=0.07, total_amount=10.15, tpep_pickup_datetime=1761956927000)\n", "Sent: Ride(PULocationID=158, DOLocationID=107, trip_distance=0.99, total_amount=33.69, tpep_pickup_datetime=1761957291000)\n", "Sent: Ride(PULocationID=234, DOLocationID=100, trip_distance=1.46, total_amount=20.95, tpep_pickup_datetime=1761956040000)\n", "Sent: Ride(PULocationID=100, DOLocationID=233, trip_distance=1.13, total_amount=25.05, tpep_pickup_datetime=1761956974000)\n", "Sent: Ride(PULocationID=239, DOLocationID=68, trip_distance=2.89, total_amount=24.13, tpep_pickup_datetime=1761956628000)\n", "Sent: Ride(PULocationID=100, DOLocationID=233, trip_distance=1.9, total_amount=23.1, tpep_pickup_datetime=1761957676000)\n", "Sent: Ride(PULocationID=246, DOLocationID=68, trip_distance=0.82, total_amount=13.95, tpep_pickup_datetime=1761956420000)\n", "Sent: Ride(PULocationID=90, DOLocationID=142, trip_distance=3.3, total_amount=36.54, tpep_pickup_datetime=1761957077000)\n", "Sent: Ride(PULocationID=143, DOLocationID=79, trip_distance=7.5, total_amount=59.05, tpep_pickup_datetime=1761956774000)\n", "Sent: Ride(PULocationID=231, DOLocationID=148, trip_distance=1.08, total_amount=16.38, tpep_pickup_datetime=1761957693000)\n", "Sent: Ride(PULocationID=148, DOLocationID=263, trip_distance=4.19, total_amount=35.44, tpep_pickup_datetime=1761958288000)\n", "Sent: Ride(PULocationID=162, DOLocationID=229, trip_distance=0.52, total_amount=15.54, tpep_pickup_datetime=1761955463000)\n", "Sent: Ride(PULocationID=141, DOLocationID=238, trip_distance=2.32, total_amount=23.04, tpep_pickup_datetime=1761956228000)\n", "Sent: Ride(PULocationID=263, DOLocationID=141, trip_distance=0.48, total_amount=12.12, tpep_pickup_datetime=1761956375000)\n", "Sent: Ride(PULocationID=141, DOLocationID=140, trip_distance=1.2, total_amount=16.32, tpep_pickup_datetime=1761956646000)\n", "Sent: Ride(PULocationID=79, DOLocationID=142, trip_distance=4.38, total_amount=38.41, tpep_pickup_datetime=1761955989000)\n", "Sent: Ride(PULocationID=48, DOLocationID=246, trip_distance=1.42, total_amount=15.05, tpep_pickup_datetime=1761958537000)\n", "Sent: Ride(PULocationID=68, DOLocationID=68, trip_distance=0.3, total_amount=11.55, tpep_pickup_datetime=1761957747000)\n", "Sent: Ride(PULocationID=68, DOLocationID=249, trip_distance=0.98, total_amount=19.74, tpep_pickup_datetime=1761958266000)\n", "Sent: Ride(PULocationID=79, DOLocationID=79, trip_distance=0.86, total_amount=20.58, tpep_pickup_datetime=1761955096000)\n", "Sent: Ride(PULocationID=107, DOLocationID=229, trip_distance=2.06, total_amount=23.94, tpep_pickup_datetime=1761956339000)\n", "Sent: Ride(PULocationID=141, DOLocationID=141, trip_distance=0.89, total_amount=15.54, tpep_pickup_datetime=1761955468000)\n", "Sent: Ride(PULocationID=141, DOLocationID=262, trip_distance=0.97, total_amount=11.5, tpep_pickup_datetime=1761956055000)\n", "Sent: Ride(PULocationID=236, DOLocationID=158, trip_distance=5.59, total_amount=45.78, tpep_pickup_datetime=1761956318000)\n", "Sent: Ride(PULocationID=158, DOLocationID=68, trip_distance=1.17, total_amount=19.57, tpep_pickup_datetime=1761958429000)\n", "Sent: Ride(PULocationID=24, DOLocationID=75, trip_distance=1.38, total_amount=15.0, tpep_pickup_datetime=1761956097000)\n", "Sent: Ride(PULocationID=137, DOLocationID=224, trip_distance=0.83, total_amount=19.25, tpep_pickup_datetime=1761958771000)\n", "Sent: Ride(PULocationID=68, DOLocationID=48, trip_distance=1.34, total_amount=24.78, tpep_pickup_datetime=1761955846000)\n", "Sent: Ride(PULocationID=48, DOLocationID=41, trip_distance=3.75, total_amount=31.4, tpep_pickup_datetime=1761957073000)\n", "Sent: Ride(PULocationID=142, DOLocationID=263, trip_distance=2.33, total_amount=23.04, tpep_pickup_datetime=1761958753000)\n", "Sent: Ride(PULocationID=249, DOLocationID=61, trip_distance=7.04, total_amount=58.36, tpep_pickup_datetime=1761957360000)\n", "Sent: Ride(PULocationID=186, DOLocationID=163, trip_distance=1.1, total_amount=21.4, tpep_pickup_datetime=1761955669000)\n", "Sent: Ride(PULocationID=163, DOLocationID=261, trip_distance=5.9, total_amount=52.5, tpep_pickup_datetime=1761956719000)\n", "Sent: Ride(PULocationID=140, DOLocationID=233, trip_distance=1.16, total_amount=16.45, tpep_pickup_datetime=1761956976000)\n", "Sent: Ride(PULocationID=233, DOLocationID=107, trip_distance=0.98, total_amount=18.06, tpep_pickup_datetime=1761957777000)\n", "Sent: Ride(PULocationID=48, DOLocationID=24, trip_distance=3.75, total_amount=26.25, tpep_pickup_datetime=1761957045000)\n", "Sent: Ride(PULocationID=141, DOLocationID=170, trip_distance=2.2, total_amount=21.17, tpep_pickup_datetime=1761957744000)\n", "Sent: Ride(PULocationID=161, DOLocationID=48, trip_distance=1.2, total_amount=18.55, tpep_pickup_datetime=1761955558000)\n", "Sent: Ride(PULocationID=74, DOLocationID=42, trip_distance=2.21, total_amount=15.3, tpep_pickup_datetime=1761955487000)\n", "Sent: Ride(PULocationID=151, DOLocationID=238, trip_distance=0.71, total_amount=11.7, tpep_pickup_datetime=1761957229000)\n", "Sent: Ride(PULocationID=143, DOLocationID=151, trip_distance=2.15, total_amount=19.58, tpep_pickup_datetime=1761958183000)\n", "Sent: Ride(PULocationID=239, DOLocationID=230, trip_distance=1.9, total_amount=21.4, tpep_pickup_datetime=1761957928000)\n", "Sent: Ride(PULocationID=230, DOLocationID=230, trip_distance=0.3, total_amount=11.55, tpep_pickup_datetime=1761958796000)\n", "Sent: Ride(PULocationID=113, DOLocationID=233, trip_distance=2.3, total_amount=31.05, tpep_pickup_datetime=1761957266000)\n", "Sent: Ride(PULocationID=233, DOLocationID=262, trip_distance=3.0, total_amount=19.25, tpep_pickup_datetime=1761958773000)\n", "Sent: Ride(PULocationID=162, DOLocationID=50, trip_distance=1.26, total_amount=23.05, tpep_pickup_datetime=1761958750000)\n", "Sent: Ride(PULocationID=90, DOLocationID=186, trip_distance=0.59, total_amount=15.54, tpep_pickup_datetime=1761956784000)\n", "Sent: Ride(PULocationID=186, DOLocationID=48, trip_distance=1.39, total_amount=16.45, tpep_pickup_datetime=1761957244000)\n", "Sent: Ride(PULocationID=48, DOLocationID=238, trip_distance=2.04, total_amount=25.62, tpep_pickup_datetime=1761958076000)\n", "Sent: Ride(PULocationID=68, DOLocationID=141, trip_distance=3.93, total_amount=40.74, tpep_pickup_datetime=1761957260000)\n", "Sent: Ride(PULocationID=186, DOLocationID=4, trip_distance=1.98, total_amount=36.54, tpep_pickup_datetime=1761957796000)\n", "Sent: Ride(PULocationID=234, DOLocationID=112, trip_distance=3.96, total_amount=49.91, tpep_pickup_datetime=1761956053000)\n", "Sent: Ride(PULocationID=211, DOLocationID=137, trip_distance=2.41, total_amount=29.25, tpep_pickup_datetime=1761955544000)\n", "Sent: Ride(PULocationID=239, DOLocationID=141, trip_distance=1.48, total_amount=18.06, tpep_pickup_datetime=1761958664000)\n", "Sent: Ride(PULocationID=170, DOLocationID=138, trip_distance=11.3, total_amount=71.29, tpep_pickup_datetime=1761956418000)\n", "Sent: Ride(PULocationID=138, DOLocationID=162, trip_distance=11.04, total_amount=74.46, tpep_pickup_datetime=1761958340000)\n", "Sent: Ride(PULocationID=90, DOLocationID=79, trip_distance=1.82, total_amount=44.1, tpep_pickup_datetime=1761957003000)\n", "Sent: Ride(PULocationID=236, DOLocationID=237, trip_distance=0.64, total_amount=14.2, tpep_pickup_datetime=1761955771000)\n", "Sent: Ride(PULocationID=237, DOLocationID=162, trip_distance=1.19, total_amount=19.74, tpep_pickup_datetime=1761956127000)\n", "Sent: Ride(PULocationID=233, DOLocationID=80, trip_distance=5.8, total_amount=52.43, tpep_pickup_datetime=1761956989000)\n", "Sent: Ride(PULocationID=262, DOLocationID=87, trip_distance=6.6, total_amount=66.8, tpep_pickup_datetime=1761955277000)\n", "Sent: Ride(PULocationID=132, DOLocationID=141, trip_distance=14.9, total_amount=81.0, tpep_pickup_datetime=1761956893000)\n", "Sent: Ride(PULocationID=107, DOLocationID=230, trip_distance=2.16, total_amount=34.02, tpep_pickup_datetime=1761955901000)\n", "Sent: Ride(PULocationID=230, DOLocationID=236, trip_distance=2.67, total_amount=23.65, tpep_pickup_datetime=1761957546000)\n", "Sent: Ride(PULocationID=163, DOLocationID=179, trip_distance=3.9, total_amount=29.0, tpep_pickup_datetime=1761955359000)\n", "Sent: Ride(PULocationID=79, DOLocationID=113, trip_distance=1.0, total_amount=17.85, tpep_pickup_datetime=1761955818000)\n", "Sent: Ride(PULocationID=113, DOLocationID=48, trip_distance=3.0, total_amount=34.85, tpep_pickup_datetime=1761956723000)\n", "Sent: Ride(PULocationID=234, DOLocationID=113, trip_distance=0.87, total_amount=29.95, tpep_pickup_datetime=1761955542000)\n", "Sent: Ride(PULocationID=113, DOLocationID=236, trip_distance=3.63, total_amount=41.56, tpep_pickup_datetime=1761957744000)\n", "Sent: Ride(PULocationID=48, DOLocationID=170, trip_distance=1.92, total_amount=25.75, tpep_pickup_datetime=1761956455000)\n", "Sent: Ride(PULocationID=170, DOLocationID=48, trip_distance=1.12, total_amount=21.42, tpep_pickup_datetime=1761957962000)\n", "Sent: Ride(PULocationID=79, DOLocationID=170, trip_distance=1.2, total_amount=15.05, tpep_pickup_datetime=1761957985000)\n", "Sent: Ride(PULocationID=233, DOLocationID=263, trip_distance=2.0, total_amount=16.45, tpep_pickup_datetime=1761958639000)\n", "Sent: Ride(PULocationID=263, DOLocationID=141, trip_distance=1.4, total_amount=16.35, tpep_pickup_datetime=1761956025000)\n", "Sent: Ride(PULocationID=234, DOLocationID=141, trip_distance=2.8, total_amount=27.3, tpep_pickup_datetime=1761958394000)\n", "Sent: Ride(PULocationID=113, DOLocationID=186, trip_distance=1.2, total_amount=24.05, tpep_pickup_datetime=1761956352000)\n", "Sent: Ride(PULocationID=170, DOLocationID=142, trip_distance=2.1, total_amount=22.25, tpep_pickup_datetime=1761957923000)\n", "Sent: Ride(PULocationID=50, DOLocationID=246, trip_distance=1.25, total_amount=16.38, tpep_pickup_datetime=1761958378000)\n", "Sent: Ride(PULocationID=246, DOLocationID=75, trip_distance=5.25, total_amount=37.35, tpep_pickup_datetime=1761958709000)\n", "Sent: Ride(PULocationID=170, DOLocationID=170, trip_distance=0.59, total_amount=14.65, tpep_pickup_datetime=1761958613000)\n", "Sent: Ride(PULocationID=141, DOLocationID=161, trip_distance=1.56, total_amount=18.9, tpep_pickup_datetime=1761956052000)\n", "Sent: Ride(PULocationID=161, DOLocationID=7, trip_distance=3.47, total_amount=31.5, tpep_pickup_datetime=1761956801000)\n", "Sent: Ride(PULocationID=113, DOLocationID=107, trip_distance=0.54, total_amount=14.15, tpep_pickup_datetime=1761955290000)\n", "Sent: Ride(PULocationID=107, DOLocationID=229, trip_distance=1.92, total_amount=30.65, tpep_pickup_datetime=1761955702000)\n", "Sent: Ride(PULocationID=237, DOLocationID=232, trip_distance=5.7, total_amount=56.7, tpep_pickup_datetime=1761955310000)\n", "Sent: Ride(PULocationID=34, DOLocationID=263, trip_distance=8.81, total_amount=54.9, tpep_pickup_datetime=1761956216000)\n", "Sent: Ride(PULocationID=238, DOLocationID=263, trip_distance=1.8, total_amount=17.15, tpep_pickup_datetime=1761956756000)\n", "Sent: Ride(PULocationID=263, DOLocationID=233, trip_distance=2.3, total_amount=23.2, tpep_pickup_datetime=1761957345000)\n", "Sent: Ride(PULocationID=233, DOLocationID=163, trip_distance=1.2, total_amount=17.2, tpep_pickup_datetime=1761958050000)\n", "Sent: Ride(PULocationID=237, DOLocationID=42, trip_distance=3.5, total_amount=24.8, tpep_pickup_datetime=1761958674000)\n", "Sent: Ride(PULocationID=143, DOLocationID=81, trip_distance=13.98, total_amount=62.65, tpep_pickup_datetime=1761957611000)\n", "Sent: Ride(PULocationID=148, DOLocationID=263, trip_distance=7.04, total_amount=47.46, tpep_pickup_datetime=1761958101000)\n", "Sent: Ride(PULocationID=230, DOLocationID=230, trip_distance=0.28, total_amount=11.55, tpep_pickup_datetime=1761955440000)\n", "Sent: Ride(PULocationID=265, DOLocationID=265, trip_distance=0.0, total_amount=71.0, tpep_pickup_datetime=1761957594000)\n", "Sent: Ride(PULocationID=265, DOLocationID=265, trip_distance=0.0, total_amount=85.2, tpep_pickup_datetime=1761957770000)\n", "Sent: Ride(PULocationID=246, DOLocationID=87, trip_distance=4.0, total_amount=47.35, tpep_pickup_datetime=1761958153000)\n", "Sent: Ride(PULocationID=231, DOLocationID=231, trip_distance=0.3, total_amount=12.15, tpep_pickup_datetime=1761955983000)\n", "Sent: Ride(PULocationID=237, DOLocationID=239, trip_distance=1.9, total_amount=21.35, tpep_pickup_datetime=1761958793000)\n", "Sent: Ride(PULocationID=236, DOLocationID=238, trip_distance=1.6, total_amount=17.0, tpep_pickup_datetime=1761956346000)\n", "Sent: Ride(PULocationID=239, DOLocationID=143, trip_distance=0.8, total_amount=13.8, tpep_pickup_datetime=1761956873000)\n", "Sent: Ride(PULocationID=239, DOLocationID=263, trip_distance=2.3, total_amount=22.25, tpep_pickup_datetime=1761957322000)\n", "Sent: Ride(PULocationID=79, DOLocationID=66, trip_distance=4.8, total_amount=31.85, tpep_pickup_datetime=1761954981000)\n", "Sent: Ride(PULocationID=234, DOLocationID=239, trip_distance=4.01, total_amount=37.38, tpep_pickup_datetime=1761955907000)\n", "Sent: Ride(PULocationID=237, DOLocationID=249, trip_distance=4.0, total_amount=52.5, tpep_pickup_datetime=1761955461000)\n", "Sent: Ride(PULocationID=249, DOLocationID=236, trip_distance=5.02, total_amount=42.95, tpep_pickup_datetime=1761958553000)\n", "Sent: Ride(PULocationID=144, DOLocationID=262, trip_distance=5.8, total_amount=40.75, tpep_pickup_datetime=1761957082000)\n", "Sent: Ride(PULocationID=239, DOLocationID=68, trip_distance=3.74, total_amount=34.56, tpep_pickup_datetime=1761956465000)\n", "Sent: Ride(PULocationID=237, DOLocationID=75, trip_distance=1.98, total_amount=15.7, tpep_pickup_datetime=1761955693000)\n", "Sent: Ride(PULocationID=107, DOLocationID=158, trip_distance=2.0, total_amount=24.8, tpep_pickup_datetime=1761957491000)\n", "Sent: Ride(PULocationID=43, DOLocationID=170, trip_distance=1.8, total_amount=23.9, tpep_pickup_datetime=1761955415000)\n", "Sent: Ride(PULocationID=107, DOLocationID=113, trip_distance=1.0, total_amount=24.75, tpep_pickup_datetime=1761956410000)\n", "Sent: Ride(PULocationID=90, DOLocationID=68, trip_distance=1.4, total_amount=25.45, tpep_pickup_datetime=1761957751000)\n", "Sent: Ride(PULocationID=263, DOLocationID=229, trip_distance=1.5, total_amount=16.75, tpep_pickup_datetime=1761956645000)\n", "Sent: Ride(PULocationID=229, DOLocationID=263, trip_distance=1.9, total_amount=19.7, tpep_pickup_datetime=1761957347000)\n", "Sent: Ride(PULocationID=263, DOLocationID=79, trip_distance=3.7, total_amount=36.5, tpep_pickup_datetime=1761957967000)\n", "Sent: Ride(PULocationID=186, DOLocationID=4, trip_distance=2.35, total_amount=45.04, tpep_pickup_datetime=1761955695000)\n", "Sent: Ride(PULocationID=79, DOLocationID=230, trip_distance=3.06, total_amount=34.02, tpep_pickup_datetime=1761958084000)\n", "Sent: Ride(PULocationID=151, DOLocationID=164, trip_distance=3.53, total_amount=37.38, tpep_pickup_datetime=1761956326000)\n", "Sent: Ride(PULocationID=164, DOLocationID=79, trip_distance=1.62, total_amount=24.05, tpep_pickup_datetime=1761958296000)\n", "Sent: Ride(PULocationID=170, DOLocationID=141, trip_distance=1.78, total_amount=22.35, tpep_pickup_datetime=1761956815000)\n", "Sent: Ride(PULocationID=90, DOLocationID=161, trip_distance=3.12, total_amount=37.45, tpep_pickup_datetime=1761956190000)\n", "Sent: Ride(PULocationID=161, DOLocationID=137, trip_distance=0.71, total_amount=14.7, tpep_pickup_datetime=1761958680000)\n", "Sent: Ride(PULocationID=158, DOLocationID=239, trip_distance=4.03, total_amount=36.15, tpep_pickup_datetime=1761957513000)\n", "Sent: Ride(PULocationID=236, DOLocationID=239, trip_distance=1.43, total_amount=17.7, tpep_pickup_datetime=1761955297000)\n", "Sent: Ride(PULocationID=107, DOLocationID=263, trip_distance=4.9, total_amount=33.85, tpep_pickup_datetime=1761955712000)\n", "Sent: Ride(PULocationID=141, DOLocationID=249, trip_distance=4.3, total_amount=55.0, tpep_pickup_datetime=1761957577000)\n", "Sent: Ride(PULocationID=186, DOLocationID=234, trip_distance=0.47, total_amount=15.7, tpep_pickup_datetime=1761955763000)\n", "Sent: Ride(PULocationID=234, DOLocationID=170, trip_distance=2.07, total_amount=29.82, tpep_pickup_datetime=1761956324000)\n", "Sent: Ride(PULocationID=232, DOLocationID=232, trip_distance=0.07, total_amount=8.75, tpep_pickup_datetime=1761957625000)\n", "Sent: Ride(PULocationID=232, DOLocationID=224, trip_distance=1.45, total_amount=19.74, tpep_pickup_datetime=1761957883000)\n", "Sent: Ride(PULocationID=224, DOLocationID=229, trip_distance=1.92, total_amount=20.58, tpep_pickup_datetime=1761958559000)\n", "Sent: Ride(PULocationID=234, DOLocationID=255, trip_distance=4.34, total_amount=54.18, tpep_pickup_datetime=1761955952000)\n", "Sent: Ride(PULocationID=68, DOLocationID=249, trip_distance=1.0, total_amount=40.74, tpep_pickup_datetime=1761955307000)\n", "Sent: Ride(PULocationID=249, DOLocationID=148, trip_distance=1.88, total_amount=22.97, tpep_pickup_datetime=1761957572000)\n", "Sent: Ride(PULocationID=148, DOLocationID=25, trip_distance=2.62, total_amount=24.78, tpep_pickup_datetime=1761958769000)\n", "Sent: Ride(PULocationID=162, DOLocationID=263, trip_distance=1.95, total_amount=15.75, tpep_pickup_datetime=1761955485000)\n", "Sent: Ride(PULocationID=261, DOLocationID=13, trip_distance=0.83, total_amount=16.83, tpep_pickup_datetime=1761958153000)\n", "Sent: Ride(PULocationID=79, DOLocationID=261, trip_distance=1.9, total_amount=20.75, tpep_pickup_datetime=1761955780000)\n", "Sent: Ride(PULocationID=45, DOLocationID=170, trip_distance=3.1, total_amount=39.9, tpep_pickup_datetime=1761957020000)\n", "Sent: Ride(PULocationID=243, DOLocationID=116, trip_distance=2.99, total_amount=21.72, tpep_pickup_datetime=1761955822000)\n", "Sent: Ride(PULocationID=234, DOLocationID=137, trip_distance=1.41, total_amount=27.55, tpep_pickup_datetime=1761956665000)\n", "Sent: Ride(PULocationID=48, DOLocationID=41, trip_distance=3.29, total_amount=25.45, tpep_pickup_datetime=1761955794000)\n", "Sent: Ride(PULocationID=113, DOLocationID=170, trip_distance=1.17, total_amount=24.78, tpep_pickup_datetime=1761955535000)\n", "Sent: Ride(PULocationID=234, DOLocationID=107, trip_distance=0.91, total_amount=23.94, tpep_pickup_datetime=1761957348000)\n", "Sent: Ride(PULocationID=107, DOLocationID=100, trip_distance=0.93, total_amount=18.06, tpep_pickup_datetime=1761958539000)\n", "Sent: Ride(PULocationID=249, DOLocationID=107, trip_distance=1.65, total_amount=28.85, tpep_pickup_datetime=1761956687000)\n", "Sent: Ride(PULocationID=186, DOLocationID=113, trip_distance=1.3, total_amount=24.78, tpep_pickup_datetime=1761955497000)\n", "Sent: Ride(PULocationID=113, DOLocationID=100, trip_distance=2.19, total_amount=30.45, tpep_pickup_datetime=1761957097000)\n", "Sent: Ride(PULocationID=186, DOLocationID=24, trip_distance=5.13, total_amount=44.1, tpep_pickup_datetime=1761957239000)\n", "Sent: Ride(PULocationID=264, DOLocationID=90, trip_distance=1.4, total_amount=41.55, tpep_pickup_datetime=1761957062000)\n", "Sent: Ride(PULocationID=138, DOLocationID=162, trip_distance=10.22, total_amount=65.24, tpep_pickup_datetime=1761955208000)\n", "Sent: Ride(PULocationID=161, DOLocationID=234, trip_distance=1.9, total_amount=23.1, tpep_pickup_datetime=1761956752000)\n", "Sent: Ride(PULocationID=113, DOLocationID=113, trip_distance=0.0, total_amount=-8.75, tpep_pickup_datetime=1761957929000)\n", "Sent: Ride(PULocationID=113, DOLocationID=113, trip_distance=0.0, total_amount=8.75, tpep_pickup_datetime=1761957929000)\n", "Sent: Ride(PULocationID=113, DOLocationID=75, trip_distance=5.69, total_amount=35.95, tpep_pickup_datetime=1761958337000)\n", "Sent: Ride(PULocationID=144, DOLocationID=148, trip_distance=0.8, total_amount=22.25, tpep_pickup_datetime=1761955358000)\n", "Sent: Ride(PULocationID=148, DOLocationID=170, trip_distance=2.0, total_amount=25.6, tpep_pickup_datetime=1761956269000)\n", "Sent: Ride(PULocationID=107, DOLocationID=262, trip_distance=3.6, total_amount=30.65, tpep_pickup_datetime=1761957679000)\n", "Sent: Ride(PULocationID=263, DOLocationID=75, trip_distance=0.7, total_amount=11.1, tpep_pickup_datetime=1761958752000)\n", "Sent: Ride(PULocationID=48, DOLocationID=238, trip_distance=2.0, total_amount=22.31, tpep_pickup_datetime=1761957155000)\n", "Sent: Ride(PULocationID=48, DOLocationID=229, trip_distance=1.5, total_amount=19.85, tpep_pickup_datetime=1761958389000)\n", "Sent: Ride(PULocationID=68, DOLocationID=144, trip_distance=1.58, total_amount=30.66, tpep_pickup_datetime=1761955512000)\n", "Sent: Ride(PULocationID=163, DOLocationID=143, trip_distance=1.2, total_amount=18.06, tpep_pickup_datetime=1761956341000)\n", "Sent: Ride(PULocationID=48, DOLocationID=141, trip_distance=2.19, total_amount=23.94, tpep_pickup_datetime=1761957714000)\n", "Sent: Ride(PULocationID=234, DOLocationID=233, trip_distance=1.26, total_amount=25.25, tpep_pickup_datetime=1761956020000)\n", "Sent: Ride(PULocationID=137, DOLocationID=79, trip_distance=1.64, total_amount=27.65, tpep_pickup_datetime=1761956631000)\n", "Sent: Ride(PULocationID=79, DOLocationID=107, trip_distance=0.91, total_amount=14.45, tpep_pickup_datetime=1761958456000)\n", "Sent: Ride(PULocationID=237, DOLocationID=68, trip_distance=2.5, total_amount=29.82, tpep_pickup_datetime=1761955599000)\n", "Sent: Ride(PULocationID=90, DOLocationID=87, trip_distance=4.67, total_amount=50.82, tpep_pickup_datetime=1761957483000)\n", "Sent: Ride(PULocationID=237, DOLocationID=164, trip_distance=1.85, total_amount=23.94, tpep_pickup_datetime=1761958438000)\n", "Sent: Ride(PULocationID=143, DOLocationID=162, trip_distance=1.3, total_amount=13.65, tpep_pickup_datetime=1761956802000)\n", "Sent: Ride(PULocationID=162, DOLocationID=158, trip_distance=3.0, total_amount=41.65, tpep_pickup_datetime=1761957357000)\n", "Sent: Ride(PULocationID=37, DOLocationID=143, trip_distance=8.61, total_amount=65.88, tpep_pickup_datetime=1761956548000)\n", "Sent: Ride(PULocationID=87, DOLocationID=148, trip_distance=1.67, total_amount=28.14, tpep_pickup_datetime=1761956271000)\n", "Sent: Ride(PULocationID=79, DOLocationID=107, trip_distance=0.93, total_amount=18.9, tpep_pickup_datetime=1761957806000)\n", "Sent: Ride(PULocationID=137, DOLocationID=236, trip_distance=3.45, total_amount=26.45, tpep_pickup_datetime=1761958735000)\n", "Sent: Ride(PULocationID=148, DOLocationID=87, trip_distance=3.04, total_amount=26.05, tpep_pickup_datetime=1761958064000)\n", "Sent: Ride(PULocationID=238, DOLocationID=236, trip_distance=1.77, total_amount=20.5, tpep_pickup_datetime=1761955394000)\n", "Sent: Ride(PULocationID=141, DOLocationID=265, trip_distance=6.26, total_amount=92.1, tpep_pickup_datetime=1761956388000)\n", "Sent: Ride(PULocationID=152, DOLocationID=82, trip_distance=7.39, total_amount=50.94, tpep_pickup_datetime=1761957298000)\n", "Sent: Ride(PULocationID=79, DOLocationID=234, trip_distance=1.38, total_amount=19.25, tpep_pickup_datetime=1761958530000)\n", "Sent: Ride(PULocationID=137, DOLocationID=90, trip_distance=0.83, total_amount=18.9, tpep_pickup_datetime=1761956852000)\n", "Sent: Ride(PULocationID=263, DOLocationID=237, trip_distance=1.47, total_amount=14.3, tpep_pickup_datetime=1761955429000)\n", "Sent: Ride(PULocationID=162, DOLocationID=233, trip_distance=0.55, total_amount=19.74, tpep_pickup_datetime=1761956098000)\n", "Sent: Ride(PULocationID=158, DOLocationID=75, trip_distance=6.2, total_amount=53.35, tpep_pickup_datetime=1761956596000)\n", "Sent: Ride(PULocationID=164, DOLocationID=231, trip_distance=2.93, total_amount=35.55, tpep_pickup_datetime=1761958267000)\n", "Sent: Ride(PULocationID=100, DOLocationID=246, trip_distance=0.73, total_amount=14.25, tpep_pickup_datetime=1761957445000)\n", "Sent: Ride(PULocationID=246, DOLocationID=48, trip_distance=1.48, total_amount=23.94, tpep_pickup_datetime=1761955900000)\n", "Sent: Ride(PULocationID=48, DOLocationID=246, trip_distance=1.6, total_amount=20.41, tpep_pickup_datetime=1761955372000)\n", "Sent: Ride(PULocationID=90, DOLocationID=137, trip_distance=1.6, total_amount=25.9, tpep_pickup_datetime=1761956526000)\n", "Sent: Ride(PULocationID=107, DOLocationID=233, trip_distance=1.4, total_amount=22.05, tpep_pickup_datetime=1761958784000)\n", "Sent: Ride(PULocationID=107, DOLocationID=234, trip_distance=0.4, total_amount=14.7, tpep_pickup_datetime=1761955781000)\n", "Sent: Ride(PULocationID=234, DOLocationID=170, trip_distance=2.1, total_amount=44.9, tpep_pickup_datetime=1761956134000)\n", "Sent: Ride(PULocationID=170, DOLocationID=87, trip_distance=5.1, total_amount=53.34, tpep_pickup_datetime=1761956209000)\n", "Sent: Ride(PULocationID=262, DOLocationID=145, trip_distance=1.73, total_amount=16.45, tpep_pickup_datetime=1761955736000)\n", "Sent: Ride(PULocationID=68, DOLocationID=158, trip_distance=0.68, total_amount=21.95, tpep_pickup_datetime=1761955728000)\n", "Sent: Ride(PULocationID=158, DOLocationID=249, trip_distance=0.22, total_amount=15.54, tpep_pickup_datetime=1761956324000)\n", "Sent: Ride(PULocationID=239, DOLocationID=229, trip_distance=2.49, total_amount=24.95, tpep_pickup_datetime=1761955362000)\n", "Sent: Ride(PULocationID=229, DOLocationID=140, trip_distance=1.08, total_amount=15.54, tpep_pickup_datetime=1761956228000)\n", "Sent: Ride(PULocationID=249, DOLocationID=163, trip_distance=4.6, total_amount=47.85, tpep_pickup_datetime=1761955422000)\n", "Sent: Ride(PULocationID=163, DOLocationID=237, trip_distance=1.41, total_amount=19.74, tpep_pickup_datetime=1761958654000)\n", "Sent: Ride(PULocationID=229, DOLocationID=68, trip_distance=3.5, total_amount=26.25, tpep_pickup_datetime=1761958702000)\n", "Sent: Ride(PULocationID=90, DOLocationID=48, trip_distance=1.1, total_amount=24.8, tpep_pickup_datetime=1761955292000)\n", "Sent: Ride(PULocationID=163, DOLocationID=90, trip_distance=1.3, total_amount=21.45, tpep_pickup_datetime=1761956729000)\n", "Sent: Ride(PULocationID=90, DOLocationID=249, trip_distance=0.5, total_amount=20.55, tpep_pickup_datetime=1761957601000)\n", "Sent: Ride(PULocationID=249, DOLocationID=137, trip_distance=2.7, total_amount=36.3, tpep_pickup_datetime=1761958362000)\n", "Sent: Ride(PULocationID=232, DOLocationID=87, trip_distance=1.54, total_amount=22.29, tpep_pickup_datetime=1761957878000)\n", "Sent: Ride(PULocationID=87, DOLocationID=148, trip_distance=1.66, total_amount=29.82, tpep_pickup_datetime=1761958604000)\n", "Sent: Ride(PULocationID=65, DOLocationID=49, trip_distance=0.9, total_amount=11.4, tpep_pickup_datetime=1761956206000)\n", "Sent: Ride(PULocationID=148, DOLocationID=249, trip_distance=1.43, total_amount=21.95, tpep_pickup_datetime=1761958184000)\n", "Sent: Ride(PULocationID=162, DOLocationID=186, trip_distance=1.66, total_amount=26.15, tpep_pickup_datetime=1761955322000)\n", "Sent: Ride(PULocationID=186, DOLocationID=68, trip_distance=0.75, total_amount=16.05, tpep_pickup_datetime=1761956736000)\n", "Sent: Ride(PULocationID=68, DOLocationID=164, trip_distance=1.11, total_amount=21.42, tpep_pickup_datetime=1761957396000)\n", "Sent: Ride(PULocationID=79, DOLocationID=234, trip_distance=2.22, total_amount=49.85, tpep_pickup_datetime=1761956131000)\n", "Sent: Ride(PULocationID=138, DOLocationID=68, trip_distance=11.1, total_amount=79.9, tpep_pickup_datetime=1761955488000)\n", "Sent: Ride(PULocationID=68, DOLocationID=246, trip_distance=0.9, total_amount=33.45, tpep_pickup_datetime=1761958334000)\n", "Sent: Ride(PULocationID=141, DOLocationID=74, trip_distance=2.9, total_amount=23.6, tpep_pickup_datetime=1761955491000)\n", "Sent: Ride(PULocationID=74, DOLocationID=244, trip_distance=3.7, total_amount=22.2, tpep_pickup_datetime=1761956515000)\n", "Sent: Ride(PULocationID=263, DOLocationID=209, trip_distance=7.04, total_amount=51.66, tpep_pickup_datetime=1761957054000)\n", "Sent: Ride(PULocationID=48, DOLocationID=256, trip_distance=7.1, total_amount=61.95, tpep_pickup_datetime=1761955765000)\n", "Sent: Ride(PULocationID=48, DOLocationID=239, trip_distance=1.18, total_amount=17.22, tpep_pickup_datetime=1761955439000)\n", "Sent: Ride(PULocationID=239, DOLocationID=161, trip_distance=2.44, total_amount=24.78, tpep_pickup_datetime=1761956004000)\n", "Sent: Ride(PULocationID=237, DOLocationID=236, trip_distance=1.22, total_amount=17.22, tpep_pickup_datetime=1761957596000)\n", "Sent: Ride(PULocationID=236, DOLocationID=142, trip_distance=1.74, total_amount=18.84, tpep_pickup_datetime=1761958175000)\n", "Sent: Ride(PULocationID=137, DOLocationID=229, trip_distance=3.03, total_amount=37.38, tpep_pickup_datetime=1761955292000)\n", "Sent: Ride(PULocationID=162, DOLocationID=75, trip_distance=2.3, total_amount=20.58, tpep_pickup_datetime=1761957307000)\n", "Sent: Ride(PULocationID=79, DOLocationID=137, trip_distance=1.27, total_amount=20.15, tpep_pickup_datetime=1761956056000)\n", "Sent: Ride(PULocationID=233, DOLocationID=243, trip_distance=9.04, total_amount=55.02, tpep_pickup_datetime=1761956831000)\n", "Sent: Ride(PULocationID=239, DOLocationID=142, trip_distance=0.69, total_amount=15.86, tpep_pickup_datetime=1761955394000)\n", "Sent: Ride(PULocationID=48, DOLocationID=100, trip_distance=1.52, total_amount=21.42, tpep_pickup_datetime=1761957177000)\n", "Sent: Ride(PULocationID=230, DOLocationID=163, trip_distance=0.76, total_amount=14.44, tpep_pickup_datetime=1761957724000)\n", "Sent: Ride(PULocationID=263, DOLocationID=141, trip_distance=0.84, total_amount=13.8, tpep_pickup_datetime=1761958711000)\n", "Sent: Ride(PULocationID=68, DOLocationID=113, trip_distance=1.8, total_amount=34.45, tpep_pickup_datetime=1761957122000)\n", "Sent: Ride(PULocationID=162, DOLocationID=233, trip_distance=0.38, total_amount=10.85, tpep_pickup_datetime=1761957862000)\n", "Sent: Ride(PULocationID=186, DOLocationID=233, trip_distance=1.63, total_amount=30.66, tpep_pickup_datetime=1761955249000)\n", "Sent: Ride(PULocationID=162, DOLocationID=79, trip_distance=3.63, total_amount=35.25, tpep_pickup_datetime=1761957099000)\n", "Sent: Ride(PULocationID=148, DOLocationID=87, trip_distance=1.2, total_amount=19.75, tpep_pickup_datetime=1761956635000)\n", "Sent: Ride(PULocationID=209, DOLocationID=170, trip_distance=4.5, total_amount=39.05, tpep_pickup_datetime=1761957901000)\n", "Sent: Ride(PULocationID=249, DOLocationID=249, trip_distance=0.27, total_amount=15.05, tpep_pickup_datetime=1761955403000)\n", "Sent: Ride(PULocationID=68, DOLocationID=239, trip_distance=3.5, total_amount=48.3, tpep_pickup_datetime=1761956136000)\n", "Sent: Ride(PULocationID=48, DOLocationID=170, trip_distance=2.16, total_amount=24.78, tpep_pickup_datetime=1761957922000)\n", "Sent: Ride(PULocationID=238, DOLocationID=166, trip_distance=1.0, total_amount=13.0, tpep_pickup_datetime=1761958113000)\n", "Sent: Ride(PULocationID=263, DOLocationID=262, trip_distance=0.56, total_amount=12.96, tpep_pickup_datetime=1761956773000)\n", "Sent: Ride(PULocationID=263, DOLocationID=74, trip_distance=1.91, total_amount=17.25, tpep_pickup_datetime=1761957198000)\n", "Sent: Ride(PULocationID=138, DOLocationID=161, trip_distance=9.31, total_amount=70.26, tpep_pickup_datetime=1761958431000)\n", "Sent: Ride(PULocationID=246, DOLocationID=243, trip_distance=10.4, total_amount=54.25, tpep_pickup_datetime=1761957210000)\n", "Sent: Ride(PULocationID=148, DOLocationID=231, trip_distance=1.17, total_amount=21.42, tpep_pickup_datetime=1761956227000)\n", "Sent: Ride(PULocationID=79, DOLocationID=137, trip_distance=1.57, total_amount=19.55, tpep_pickup_datetime=1761958599000)\n", "Sent: Ride(PULocationID=163, DOLocationID=145, trip_distance=1.97, total_amount=22.26, tpep_pickup_datetime=1761956297000)\n", "Sent: Ride(PULocationID=148, DOLocationID=209, trip_distance=1.4, total_amount=18.9, tpep_pickup_datetime=1761958782000)\n", "Sent: Ride(PULocationID=48, DOLocationID=163, trip_distance=0.96, total_amount=13.95, tpep_pickup_datetime=1761958014000)\n", "Sent: Ride(PULocationID=163, DOLocationID=234, trip_distance=1.35, total_amount=19.55, tpep_pickup_datetime=1761958413000)\n", "Sent: Ride(PULocationID=100, DOLocationID=145, trip_distance=4.79, total_amount=42.26, tpep_pickup_datetime=1761956569000)\n", "Sent: Ride(PULocationID=107, DOLocationID=152, trip_distance=7.57, total_amount=57.79, tpep_pickup_datetime=1761957412000)\n", "Sent: Ride(PULocationID=142, DOLocationID=163, trip_distance=0.64, total_amount=12.25, tpep_pickup_datetime=1761957651000)\n", "Sent: Ride(PULocationID=48, DOLocationID=239, trip_distance=1.16, total_amount=15.75, tpep_pickup_datetime=1761958651000)\n", "Sent: Ride(PULocationID=148, DOLocationID=80, trip_distance=3.27, total_amount=24.85, tpep_pickup_datetime=1761956017000)\n", "Sent: Ride(PULocationID=138, DOLocationID=70, trip_distance=1.16, total_amount=24.09, tpep_pickup_datetime=1761958172000)\n", "Sent: Ride(PULocationID=138, DOLocationID=252, trip_distance=6.74, total_amount=49.63, tpep_pickup_datetime=1761958767000)\n", "Sent: Ride(PULocationID=234, DOLocationID=236, trip_distance=3.51, total_amount=37.38, tpep_pickup_datetime=1761955439000)\n", "Sent: Ride(PULocationID=162, DOLocationID=75, trip_distance=2.72, total_amount=18.55, tpep_pickup_datetime=1761958052000)\n", "Sent: Ride(PULocationID=238, DOLocationID=151, trip_distance=0.53, total_amount=12.12, tpep_pickup_datetime=1761957724000)\n", "Sent: Ride(PULocationID=113, DOLocationID=48, trip_distance=2.82, total_amount=38.22, tpep_pickup_datetime=1761955250000)\n", "Sent: Ride(PULocationID=48, DOLocationID=74, trip_distance=5.21, total_amount=37.38, tpep_pickup_datetime=1761957341000)\n", "Sent: Ride(PULocationID=141, DOLocationID=144, trip_distance=4.22, total_amount=42.73, tpep_pickup_datetime=1761956330000)\n", "Sent: Ride(PULocationID=132, DOLocationID=142, trip_distance=21.73, total_amount=87.69, tpep_pickup_datetime=1761956883000)\n", "Sent: Ride(PULocationID=144, DOLocationID=141, trip_distance=4.5, total_amount=46.6, tpep_pickup_datetime=1761955852000)\n", "Sent: Ride(PULocationID=107, DOLocationID=262, trip_distance=3.44, total_amount=33.18, tpep_pickup_datetime=1761956238000)\n", "Sent: Ride(PULocationID=263, DOLocationID=229, trip_distance=1.63, total_amount=19.74, tpep_pickup_datetime=1761958132000)\n", "Sent: Ride(PULocationID=229, DOLocationID=236, trip_distance=1.45, total_amount=16.38, tpep_pickup_datetime=1761958766000)\n", "Sent: Ride(PULocationID=114, DOLocationID=107, trip_distance=1.3, total_amount=21.35, tpep_pickup_datetime=1761955718000)\n", "Sent: Ride(PULocationID=107, DOLocationID=79, trip_distance=1.0, total_amount=20.65, tpep_pickup_datetime=1761956865000)\n", "Sent: Ride(PULocationID=79, DOLocationID=48, trip_distance=3.6, total_amount=29.05, tpep_pickup_datetime=1761957987000)\n", "Sent: Ride(PULocationID=230, DOLocationID=238, trip_distance=2.1, total_amount=21.4, tpep_pickup_datetime=1761957886000)\n", "Sent: Ride(PULocationID=144, DOLocationID=141, trip_distance=3.71, total_amount=32.34, tpep_pickup_datetime=1761958179000)\n", "Sent: Ride(PULocationID=144, DOLocationID=170, trip_distance=1.92, total_amount=28.14, tpep_pickup_datetime=1761958413000)\n", "Sent: Ride(PULocationID=138, DOLocationID=162, trip_distance=9.95, total_amount=76.61, tpep_pickup_datetime=1761956513000)\n", "Sent: Ride(PULocationID=140, DOLocationID=107, trip_distance=2.2, total_amount=24.15, tpep_pickup_datetime=1761957310000)\n", "Sent: Ride(PULocationID=164, DOLocationID=17, trip_distance=8.9, total_amount=67.49, tpep_pickup_datetime=1761956182000)\n", "Sent: Ride(PULocationID=263, DOLocationID=238, trip_distance=1.33, total_amount=17.16, tpep_pickup_datetime=1761956654000)\n", "Sent: Ride(PULocationID=234, DOLocationID=230, trip_distance=1.8, total_amount=22.25, tpep_pickup_datetime=1761957156000)\n", "Sent: Ride(PULocationID=236, DOLocationID=48, trip_distance=2.3, total_amount=28.35, tpep_pickup_datetime=1761958637000)\n", "Sent: Ride(PULocationID=141, DOLocationID=79, trip_distance=4.6, total_amount=53.34, tpep_pickup_datetime=1761957453000)\n", "Sent: Ride(PULocationID=90, DOLocationID=249, trip_distance=1.3, total_amount=30.45, tpep_pickup_datetime=1761955450000)\n", "Sent: Ride(PULocationID=249, DOLocationID=238, trip_distance=4.5, total_amount=50.0, tpep_pickup_datetime=1761957468000)\n", "Sent: Ride(PULocationID=249, DOLocationID=246, trip_distance=1.83, total_amount=24.75, tpep_pickup_datetime=1761957761000)\n", "Sent: Ride(PULocationID=170, DOLocationID=148, trip_distance=2.5, total_amount=32.85, tpep_pickup_datetime=1761955826000)\n", "Sent: Ride(PULocationID=144, DOLocationID=249, trip_distance=1.9, total_amount=24.15, tpep_pickup_datetime=1761958126000)\n", "Sent: Ride(PULocationID=100, DOLocationID=143, trip_distance=1.98, total_amount=25.36, tpep_pickup_datetime=1761955308000)\n", "Sent: Ride(PULocationID=170, DOLocationID=246, trip_distance=2.17, total_amount=23.45, tpep_pickup_datetime=1761955684000)\n", "Sent: Ride(PULocationID=79, DOLocationID=234, trip_distance=1.79, total_amount=28.98, tpep_pickup_datetime=1761955920000)\n", "Sent: Ride(PULocationID=107, DOLocationID=237, trip_distance=2.55, total_amount=23.35, tpep_pickup_datetime=1761957446000)\n", "Sent: Ride(PULocationID=237, DOLocationID=263, trip_distance=1.66, total_amount=17.2, tpep_pickup_datetime=1761958370000)\n", "Sent: Ride(PULocationID=113, DOLocationID=255, trip_distance=5.69, total_amount=52.15, tpep_pickup_datetime=1761956236000)\n", "Sent: Ride(PULocationID=144, DOLocationID=249, trip_distance=1.0, total_amount=15.05, tpep_pickup_datetime=1761958649000)\n", "Sent: Ride(PULocationID=158, DOLocationID=43, trip_distance=4.24, total_amount=42.45, tpep_pickup_datetime=1761957905000)\n", "Sent: Ride(PULocationID=132, DOLocationID=256, trip_distance=16.31, total_amount=87.85, tpep_pickup_datetime=1761957025000)\n", "Sent: Ride(PULocationID=138, DOLocationID=231, trip_distance=11.65, total_amount=66.33, tpep_pickup_datetime=1761958443000)\n", "Sent: Ride(PULocationID=88, DOLocationID=261, trip_distance=0.43, total_amount=18.06, tpep_pickup_datetime=1761956169000)\n", "Sent: Ride(PULocationID=261, DOLocationID=186, trip_distance=5.42, total_amount=51.66, tpep_pickup_datetime=1761957049000)\n", "Sent: Ride(PULocationID=50, DOLocationID=68, trip_distance=0.93, total_amount=15.54, tpep_pickup_datetime=1761957235000)\n", "Sent: Ride(PULocationID=68, DOLocationID=13, trip_distance=3.6, total_amount=-36.05, tpep_pickup_datetime=1761958091000)\n", "Sent: Ride(PULocationID=68, DOLocationID=13, trip_distance=3.6, total_amount=36.05, tpep_pickup_datetime=1761958091000)\n", "Sent: Ride(PULocationID=246, DOLocationID=37, trip_distance=12.66, total_amount=83.09, tpep_pickup_datetime=1761956632000)\n", "Sent: Ride(PULocationID=148, DOLocationID=79, trip_distance=0.69, total_amount=14.65, tpep_pickup_datetime=1761955727000)\n", "Sent: Ride(PULocationID=79, DOLocationID=148, trip_distance=1.23, total_amount=25.62, tpep_pickup_datetime=1761956213000)\n", "Sent: Ride(PULocationID=148, DOLocationID=79, trip_distance=1.04, total_amount=16.75, tpep_pickup_datetime=1761957350000)\n", "Sent: Ride(PULocationID=79, DOLocationID=231, trip_distance=1.63, total_amount=19.15, tpep_pickup_datetime=1761958074000)\n", "Sent: Ride(PULocationID=79, DOLocationID=255, trip_distance=3.38, total_amount=49.14, tpep_pickup_datetime=1761956923000)\n", "Sent: Ride(PULocationID=234, DOLocationID=113, trip_distance=0.28, total_amount=16.38, tpep_pickup_datetime=1761956238000)\n", "Sent: Ride(PULocationID=113, DOLocationID=161, trip_distance=2.63, total_amount=29.75, tpep_pickup_datetime=1761956822000)\n", "Sent: Ride(PULocationID=43, DOLocationID=158, trip_distance=5.39, total_amount=36.25, tpep_pickup_datetime=1761956931000)\n", "Sent: Ride(PULocationID=158, DOLocationID=186, trip_distance=1.33, total_amount=17.75, tpep_pickup_datetime=1761958537000)\n", "Sent: Ride(PULocationID=50, DOLocationID=263, trip_distance=4.11, total_amount=26.25, tpep_pickup_datetime=1761955276000)\n", "Sent: Ride(PULocationID=231, DOLocationID=144, trip_distance=1.07, total_amount=24.06, tpep_pickup_datetime=1761955616000)\n", "Sent: Ride(PULocationID=144, DOLocationID=158, trip_distance=2.15, total_amount=27.3, tpep_pickup_datetime=1761956698000)\n", "Sent: Ride(PULocationID=158, DOLocationID=249, trip_distance=0.58, total_amount=25.03, tpep_pickup_datetime=1761957879000)\n", "Sent: Ride(PULocationID=107, DOLocationID=246, trip_distance=1.7, total_amount=25.6, tpep_pickup_datetime=1761958272000)\n", "Sent: Ride(PULocationID=100, DOLocationID=75, trip_distance=4.3, total_amount=36.5, tpep_pickup_datetime=1761955917000)\n", "Sent: Ride(PULocationID=68, DOLocationID=68, trip_distance=0.73, total_amount=17.22, tpep_pickup_datetime=1761955730000)\n", "Sent: Ride(PULocationID=230, DOLocationID=243, trip_distance=8.62, total_amount=52.5, tpep_pickup_datetime=1761957246000)\n", "Sent: Ride(PULocationID=236, DOLocationID=163, trip_distance=1.9, total_amount=20.6, tpep_pickup_datetime=1761956718000)\n", "Sent: Ride(PULocationID=163, DOLocationID=234, trip_distance=2.3, total_amount=29.8, tpep_pickup_datetime=1761957364000)\n", "Sent: Ride(PULocationID=141, DOLocationID=79, trip_distance=3.15, total_amount=48.56, tpep_pickup_datetime=1761955429000)\n", "Sent: Ride(PULocationID=79, DOLocationID=79, trip_distance=0.09, total_amount=13.02, tpep_pickup_datetime=1761957978000)\n", "Sent: Ride(PULocationID=48, DOLocationID=239, trip_distance=2.07, total_amount=23.94, tpep_pickup_datetime=1761955360000)\n", "Sent: Ride(PULocationID=237, DOLocationID=237, trip_distance=0.43, total_amount=12.12, tpep_pickup_datetime=1761956193000)\n", "Sent: Ride(PULocationID=237, DOLocationID=239, trip_distance=1.27, total_amount=15.48, tpep_pickup_datetime=1761956860000)\n", "Sent: Ride(PULocationID=239, DOLocationID=229, trip_distance=3.11, total_amount=28.98, tpep_pickup_datetime=1761957265000)\n", "Sent: Ride(PULocationID=229, DOLocationID=79, trip_distance=2.55, total_amount=32.45, tpep_pickup_datetime=1761958387000)\n", "Sent: Ride(PULocationID=226, DOLocationID=7, trip_distance=1.1, total_amount=14.75, tpep_pickup_datetime=1761958172000)\n", "Sent: Ride(PULocationID=107, DOLocationID=137, trip_distance=0.75, total_amount=16.35, tpep_pickup_datetime=1761955477000)\n", "Sent: Ride(PULocationID=161, DOLocationID=236, trip_distance=2.06, total_amount=19.85, tpep_pickup_datetime=1761957775000)\n", "Sent: Ride(PULocationID=132, DOLocationID=145, trip_distance=16.5, total_amount=90.45, tpep_pickup_datetime=1761958648000)\n", "Sent: Ride(PULocationID=186, DOLocationID=107, trip_distance=1.0, total_amount=21.42, tpep_pickup_datetime=1761955658000)\n", "Sent: Ride(PULocationID=107, DOLocationID=170, trip_distance=0.71, total_amount=12.95, tpep_pickup_datetime=1761956423000)\n", "Sent: Ride(PULocationID=170, DOLocationID=170, trip_distance=0.0, total_amount=-19.25, tpep_pickup_datetime=1761956816000)\n", "Sent: Ride(PULocationID=170, DOLocationID=170, trip_distance=0.0, total_amount=19.25, tpep_pickup_datetime=1761956816000)\n", "Sent: Ride(PULocationID=113, DOLocationID=224, trip_distance=1.0, total_amount=23.95, tpep_pickup_datetime=1761956447000)\n", "Sent: Ride(PULocationID=224, DOLocationID=79, trip_distance=0.9, total_amount=17.2, tpep_pickup_datetime=1761957800000)\n", "Sent: Ride(PULocationID=79, DOLocationID=170, trip_distance=1.7, total_amount=18.9, tpep_pickup_datetime=1761958452000)\n", "Sent: Ride(PULocationID=114, DOLocationID=113, trip_distance=0.8, total_amount=19.15, tpep_pickup_datetime=1761955618000)\n", "Sent: Ride(PULocationID=79, DOLocationID=7, trip_distance=5.5, total_amount=56.7, tpep_pickup_datetime=1761957926000)\n", "Sent: Ride(PULocationID=238, DOLocationID=48, trip_distance=1.8, total_amount=19.15, tpep_pickup_datetime=1761956880000)\n", "Sent: Ride(PULocationID=230, DOLocationID=164, trip_distance=2.4, total_amount=29.05, tpep_pickup_datetime=1761957816000)\n", "Sent: Ride(PULocationID=234, DOLocationID=237, trip_distance=2.28, total_amount=24.15, tpep_pickup_datetime=1761957069000)\n", "Sent: Ride(PULocationID=107, DOLocationID=164, trip_distance=0.62, total_amount=16.38, tpep_pickup_datetime=1761957817000)\n", "Sent: Ride(PULocationID=186, DOLocationID=238, trip_distance=4.1, total_amount=36.15, tpep_pickup_datetime=1761958381000)\n", "Sent: Ride(PULocationID=158, DOLocationID=125, trip_distance=0.99, total_amount=23.75, tpep_pickup_datetime=1761955374000)\n", "Sent: Ride(PULocationID=90, DOLocationID=158, trip_distance=2.28, total_amount=43.26, tpep_pickup_datetime=1761956336000)\n", "Sent: Ride(PULocationID=141, DOLocationID=237, trip_distance=0.88, total_amount=12.25, tpep_pickup_datetime=1761956351000)\n", "Sent: Ride(PULocationID=141, DOLocationID=263, trip_distance=0.58, total_amount=12.12, tpep_pickup_datetime=1761956881000)\n", "Sent: Ride(PULocationID=263, DOLocationID=262, trip_distance=0.34, total_amount=10.81, tpep_pickup_datetime=1761957100000)\n", "Sent: Ride(PULocationID=262, DOLocationID=140, trip_distance=1.3, total_amount=18.0, tpep_pickup_datetime=1761957520000)\n", "Sent: Ride(PULocationID=237, DOLocationID=236, trip_distance=0.51, total_amount=10.4, tpep_pickup_datetime=1761958424000)\n", "Sent: Ride(PULocationID=231, DOLocationID=79, trip_distance=1.71, total_amount=21.35, tpep_pickup_datetime=1761955817000)\n", "Sent: Ride(PULocationID=79, DOLocationID=79, trip_distance=0.77, total_amount=19.25, tpep_pickup_datetime=1761957630000)\n", "Sent: Ride(PULocationID=249, DOLocationID=230, trip_distance=2.52, total_amount=44.94, tpep_pickup_datetime=1761955596000)\n", "Sent: Ride(PULocationID=116, DOLocationID=166, trip_distance=1.05, total_amount=13.52, tpep_pickup_datetime=1761956317000)\n", "Sent: Ride(PULocationID=138, DOLocationID=230, trip_distance=11.01, total_amount=82.02, tpep_pickup_datetime=1761956511000)\n", "Sent: Ride(PULocationID=48, DOLocationID=45, trip_distance=4.35, total_amount=-38.85, tpep_pickup_datetime=1761958593000)\n", "Sent: Ride(PULocationID=48, DOLocationID=45, trip_distance=4.35, total_amount=38.85, tpep_pickup_datetime=1761958593000)\n", "Sent: Ride(PULocationID=140, DOLocationID=263, trip_distance=0.82, total_amount=13.7, tpep_pickup_datetime=1761956267000)\n", "Sent: Ride(PULocationID=263, DOLocationID=74, trip_distance=0.84, total_amount=13.8, tpep_pickup_datetime=1761956776000)\n", "Sent: Ride(PULocationID=48, DOLocationID=186, trip_distance=1.4, total_amount=18.9, tpep_pickup_datetime=1761958274000)\n", "Sent: Ride(PULocationID=138, DOLocationID=28, trip_distance=6.35, total_amount=50.47, tpep_pickup_datetime=1761955330000)\n", "Sent: Ride(PULocationID=138, DOLocationID=262, trip_distance=8.44, total_amount=63.48, tpep_pickup_datetime=1761957553000)\n", "Sent: Ride(PULocationID=246, DOLocationID=48, trip_distance=1.11, total_amount=22.26, tpep_pickup_datetime=1761956847000)\n", "Sent: Ride(PULocationID=48, DOLocationID=87, trip_distance=6.66, total_amount=50.15, tpep_pickup_datetime=1761957913000)\n", "Sent: Ride(PULocationID=239, DOLocationID=151, trip_distance=1.41, total_amount=16.32, tpep_pickup_datetime=1761956114000)\n", "Sent: Ride(PULocationID=239, DOLocationID=143, trip_distance=0.76, total_amount=10.8, tpep_pickup_datetime=1761957059000)\n", "Sent: Ride(PULocationID=142, DOLocationID=238, trip_distance=1.01, total_amount=13.8, tpep_pickup_datetime=1761957693000)\n", "Sent: Ride(PULocationID=239, DOLocationID=263, trip_distance=1.4, total_amount=18.0, tpep_pickup_datetime=1761958383000)\n", "Sent: Ride(PULocationID=263, DOLocationID=75, trip_distance=1.11, total_amount=14.64, tpep_pickup_datetime=1761958732000)\n", "Sent: Ride(PULocationID=261, DOLocationID=229, trip_distance=5.7, total_amount=39.55, tpep_pickup_datetime=1761957098000)\n", "Sent: Ride(PULocationID=107, DOLocationID=148, trip_distance=1.38, total_amount=29.05, tpep_pickup_datetime=1761956006000)\n", "Sent: Ride(PULocationID=144, DOLocationID=141, trip_distance=4.33, total_amount=44.1, tpep_pickup_datetime=1761957811000)\n", "Sent: Ride(PULocationID=132, DOLocationID=238, trip_distance=19.53, total_amount=98.88, tpep_pickup_datetime=1761956173000)\n", "Sent: Ride(PULocationID=234, DOLocationID=43, trip_distance=2.5, total_amount=37.2, tpep_pickup_datetime=1761957579000)\n", "Sent: Ride(PULocationID=138, DOLocationID=68, trip_distance=10.49, total_amount=73.3, tpep_pickup_datetime=1761955850000)\n", "Sent: Ride(PULocationID=90, DOLocationID=48, trip_distance=1.54, total_amount=33.25, tpep_pickup_datetime=1761955446000)\n", "Sent: Ride(PULocationID=50, DOLocationID=97, trip_distance=5.7, total_amount=56.65, tpep_pickup_datetime=1761958001000)\n", "Sent: Ride(PULocationID=90, DOLocationID=148, trip_distance=1.43, total_amount=38.22, tpep_pickup_datetime=1761956136000)\n", "Sent: Ride(PULocationID=148, DOLocationID=36, trip_distance=3.85, total_amount=37.45, tpep_pickup_datetime=1761958185000)\n", "Sent: Ride(PULocationID=48, DOLocationID=246, trip_distance=1.09, total_amount=15.15, tpep_pickup_datetime=1761955232000)\n", "Sent: Ride(PULocationID=246, DOLocationID=249, trip_distance=1.32, total_amount=23.45, tpep_pickup_datetime=1761955586000)\n", "Sent: Ride(PULocationID=249, DOLocationID=13, trip_distance=3.02, total_amount=39.06, tpep_pickup_datetime=1761956901000)\n", "Sent: Ride(PULocationID=263, DOLocationID=141, trip_distance=0.7, total_amount=12.95, tpep_pickup_datetime=1761956438000)\n", "Sent: Ride(PULocationID=140, DOLocationID=244, trip_distance=5.9, total_amount=38.9, tpep_pickup_datetime=1761956922000)\n", "Sent: Ride(PULocationID=107, DOLocationID=68, trip_distance=1.07, total_amount=19.74, tpep_pickup_datetime=1761954560000)\n", "Sent: Ride(PULocationID=68, DOLocationID=263, trip_distance=5.34, total_amount=49.14, tpep_pickup_datetime=1761955328000)\n", "Sent: Ride(PULocationID=48, DOLocationID=237, trip_distance=1.92, total_amount=19.25, tpep_pickup_datetime=1761958528000)\n", "Sent: Ride(PULocationID=132, DOLocationID=10, trip_distance=2.32, total_amount=26.76, tpep_pickup_datetime=1761958227000)\n", "Sent: Ride(PULocationID=79, DOLocationID=162, trip_distance=2.4, total_amount=26.45, tpep_pickup_datetime=1761956453000)\n", "Sent: Ride(PULocationID=162, DOLocationID=48, trip_distance=1.3, total_amount=23.2, tpep_pickup_datetime=1761957702000)\n", "Sent: Ride(PULocationID=48, DOLocationID=246, trip_distance=1.1, total_amount=17.05, tpep_pickup_datetime=1761958724000)\n", "Sent: Ride(PULocationID=48, DOLocationID=90, trip_distance=1.8, total_amount=27.3, tpep_pickup_datetime=1761955601000)\n", "Sent: Ride(PULocationID=90, DOLocationID=88, trip_distance=3.1, total_amount=41.6, tpep_pickup_datetime=1761956876000)\n", "Sent: Ride(PULocationID=68, DOLocationID=48, trip_distance=1.5, total_amount=26.45, tpep_pickup_datetime=1761955465000)\n", "Sent: Ride(PULocationID=48, DOLocationID=164, trip_distance=1.3, total_amount=20.85, tpep_pickup_datetime=1761956661000)\n", "Sent: Ride(PULocationID=48, DOLocationID=50, trip_distance=0.5, total_amount=15.5, tpep_pickup_datetime=1761957974000)\n", "Sent: Ride(PULocationID=48, DOLocationID=239, trip_distance=1.64, total_amount=18.06, tpep_pickup_datetime=1761957850000)\n", "Sent: Ride(PULocationID=229, DOLocationID=4, trip_distance=2.84, total_amount=39.06, tpep_pickup_datetime=1761958097000)\n", "Sent: Ride(PULocationID=113, DOLocationID=164, trip_distance=1.09, total_amount=17.15, tpep_pickup_datetime=1761958351000)\n", "Sent: Ride(PULocationID=68, DOLocationID=88, trip_distance=3.3, total_amount=35.7, tpep_pickup_datetime=1761957349000)\n", "Sent: Ride(PULocationID=142, DOLocationID=229, trip_distance=1.0, total_amount=17.0, tpep_pickup_datetime=1761955657000)\n", "Sent: Ride(PULocationID=163, DOLocationID=223, trip_distance=5.7, total_amount=55.0, tpep_pickup_datetime=1761957596000)\n", "Sent: Ride(PULocationID=148, DOLocationID=4, trip_distance=0.54, total_amount=16.05, tpep_pickup_datetime=1761956220000)\n", "Sent: Ride(PULocationID=4, DOLocationID=246, trip_distance=2.39, total_amount=35.7, tpep_pickup_datetime=1761956807000)\n", "Sent: Ride(PULocationID=148, DOLocationID=140, trip_distance=4.82, total_amount=42.42, tpep_pickup_datetime=1761957016000)\n", "Sent: Ride(PULocationID=249, DOLocationID=231, trip_distance=1.29, total_amount=18.06, tpep_pickup_datetime=1761958584000)\n", "Sent: Ride(PULocationID=230, DOLocationID=238, trip_distance=2.3, total_amount=20.65, tpep_pickup_datetime=1761955264000)\n", "Sent: Ride(PULocationID=143, DOLocationID=263, trip_distance=2.0, total_amount=18.85, tpep_pickup_datetime=1761956722000)\n", "Sent: Ride(PULocationID=263, DOLocationID=107, trip_distance=4.0, total_amount=34.0, tpep_pickup_datetime=1761957356000)\n", "Sent: Ride(PULocationID=107, DOLocationID=249, trip_distance=1.3, total_amount=24.95, tpep_pickup_datetime=1761958780000)\n", "Sent: Ride(PULocationID=163, DOLocationID=229, trip_distance=1.0, total_amount=18.05, tpep_pickup_datetime=1761955689000)\n", "Sent: Ride(PULocationID=229, DOLocationID=107, trip_distance=1.7, total_amount=32.15, tpep_pickup_datetime=1761956227000)\n", "Sent: Ride(PULocationID=107, DOLocationID=79, trip_distance=0.9, total_amount=27.55, tpep_pickup_datetime=1761958453000)\n", "Sent: Ride(PULocationID=231, DOLocationID=148, trip_distance=1.35, total_amount=23.94, tpep_pickup_datetime=1761955756000)\n", "Sent: Ride(PULocationID=230, DOLocationID=48, trip_distance=0.7, total_amount=16.38, tpep_pickup_datetime=1761956588000)\n", "Sent: Ride(PULocationID=48, DOLocationID=100, trip_distance=1.02, total_amount=15.75, tpep_pickup_datetime=1761957508000)\n", "Sent: Ride(PULocationID=100, DOLocationID=140, trip_distance=2.04, total_amount=22.75, tpep_pickup_datetime=1761958204000)\n", "Sent: Ride(PULocationID=48, DOLocationID=166, trip_distance=2.9, total_amount=23.35, tpep_pickup_datetime=1761956399000)\n", "Sent: Ride(PULocationID=151, DOLocationID=116, trip_distance=2.6, total_amount=22.55, tpep_pickup_datetime=1761957485000)\n", "Sent: Ride(PULocationID=209, DOLocationID=158, trip_distance=2.7, total_amount=26.95, tpep_pickup_datetime=1761955527000)\n", "Sent: Ride(PULocationID=158, DOLocationID=249, trip_distance=1.3, total_amount=32.8, tpep_pickup_datetime=1761957154000)\n", "Sent: Ride(PULocationID=48, DOLocationID=48, trip_distance=0.3, total_amount=10.15, tpep_pickup_datetime=1761955372000)\n", "Sent: Ride(PULocationID=48, DOLocationID=112, trip_distance=5.3, total_amount=39.9, tpep_pickup_datetime=1761955882000)\n", "Sent: Ride(PULocationID=237, DOLocationID=262, trip_distance=1.43, total_amount=-15.0, tpep_pickup_datetime=1761957456000)\n", "Sent: Ride(PULocationID=237, DOLocationID=263, trip_distance=1.43, total_amount=15.0, tpep_pickup_datetime=1761957456000)\n", "Sent: Ride(PULocationID=162, DOLocationID=48, trip_distance=0.7, total_amount=22.95, tpep_pickup_datetime=1761956063000)\n", "Sent: Ride(PULocationID=138, DOLocationID=161, trip_distance=10.23, total_amount=79.5, tpep_pickup_datetime=1761955406000)\n", "Sent: Ride(PULocationID=137, DOLocationID=87, trip_distance=4.43, total_amount=50.05, tpep_pickup_datetime=1761956344000)\n", "Sent: Ride(PULocationID=79, DOLocationID=113, trip_distance=0.5, total_amount=13.85, tpep_pickup_datetime=1761956358000)\n", "Sent: Ride(PULocationID=113, DOLocationID=144, trip_distance=0.9, total_amount=15.55, tpep_pickup_datetime=1761956781000)\n", "Sent: Ride(PULocationID=261, DOLocationID=236, trip_distance=9.4, total_amount=59.35, tpep_pickup_datetime=1761958586000)\n", "Sent: Ride(PULocationID=237, DOLocationID=140, trip_distance=0.87, total_amount=13.7, tpep_pickup_datetime=1761955339000)\n", "Sent: Ride(PULocationID=141, DOLocationID=7, trip_distance=3.94, total_amount=27.65, tpep_pickup_datetime=1761956046000)\n", "Sent: Ride(PULocationID=48, DOLocationID=107, trip_distance=2.58, total_amount=34.02, tpep_pickup_datetime=1761958777000)\n", "Sent: Ride(PULocationID=24, DOLocationID=74, trip_distance=1.82, total_amount=17.52, tpep_pickup_datetime=1761957474000)\n", "Sent: Ride(PULocationID=79, DOLocationID=107, trip_distance=0.77, total_amount=23.1, tpep_pickup_datetime=1761955737000)\n", "Sent: Ride(PULocationID=107, DOLocationID=87, trip_distance=3.99, total_amount=40.74, tpep_pickup_datetime=1761957025000)\n", "Sent: Ride(PULocationID=148, DOLocationID=113, trip_distance=0.66, total_amount=17.22, tpep_pickup_datetime=1761956685000)\n", "Sent: Ride(PULocationID=113, DOLocationID=170, trip_distance=1.08, total_amount=21.42, tpep_pickup_datetime=1761957289000)\n", "Sent: Ride(PULocationID=170, DOLocationID=48, trip_distance=1.59, total_amount=25.79, tpep_pickup_datetime=1761958164000)\n", "Sent: Ride(PULocationID=226, DOLocationID=181, trip_distance=9.59, total_amount=47.3, tpep_pickup_datetime=1761955971000)\n", "Sent: Ride(PULocationID=186, DOLocationID=246, trip_distance=0.55, total_amount=12.85, tpep_pickup_datetime=1761956644000)\n", "Sent: Ride(PULocationID=48, DOLocationID=265, trip_distance=2.75, total_amount=102.97, tpep_pickup_datetime=1761958631000)\n", "Sent: Ride(PULocationID=231, DOLocationID=13, trip_distance=1.22, total_amount=17.22, tpep_pickup_datetime=1761955724000)\n", "Sent: Ride(PULocationID=107, DOLocationID=140, trip_distance=3.69, total_amount=26.25, tpep_pickup_datetime=1761958367000)\n", "Sent: Ride(PULocationID=4, DOLocationID=144, trip_distance=0.95, total_amount=19.25, tpep_pickup_datetime=1761955694000)\n", "Sent: Ride(PULocationID=144, DOLocationID=79, trip_distance=0.82, total_amount=19.74, tpep_pickup_datetime=1761956737000)\n", "Sent: Ride(PULocationID=234, DOLocationID=49, trip_distance=6.77, total_amount=54.69, tpep_pickup_datetime=1761957574000)\n", "Sent: Ride(PULocationID=239, DOLocationID=41, trip_distance=2.43, total_amount=22.2, tpep_pickup_datetime=1761957454000)\n", "Sent: Ride(PULocationID=138, DOLocationID=148, trip_distance=9.41, total_amount=64.7, tpep_pickup_datetime=1761958000000)\n", "Sent: Ride(PULocationID=100, DOLocationID=211, trip_distance=3.51, total_amount=40.74, tpep_pickup_datetime=1761955201000)\n", "Sent: Ride(PULocationID=211, DOLocationID=158, trip_distance=1.21, total_amount=22.29, tpep_pickup_datetime=1761957218000)\n", "Sent: Ride(PULocationID=4, DOLocationID=239, trip_distance=5.75, total_amount=40.43, tpep_pickup_datetime=1761955800000)\n", "Sent: Ride(PULocationID=161, DOLocationID=262, trip_distance=2.3, total_amount=23.2, tpep_pickup_datetime=1761955464000)\n", "Sent: Ride(PULocationID=263, DOLocationID=79, trip_distance=4.2, total_amount=53.35, tpep_pickup_datetime=1761956651000)\n", "Sent: Ride(PULocationID=137, DOLocationID=233, trip_distance=1.18, total_amount=18.06, tpep_pickup_datetime=1761956725000)\n", "Sent: Ride(PULocationID=137, DOLocationID=79, trip_distance=1.34, total_amount=32.34, tpep_pickup_datetime=1761957567000)\n", "Sent: Ride(PULocationID=238, DOLocationID=263, trip_distance=1.73, total_amount=17.4, tpep_pickup_datetime=1761957998000)\n", "Sent: Ride(PULocationID=234, DOLocationID=163, trip_distance=2.2, total_amount=23.86, tpep_pickup_datetime=1761955626000)\n", "Sent: Ride(PULocationID=163, DOLocationID=141, trip_distance=1.11, total_amount=18.9, tpep_pickup_datetime=1761956732000)\n", "Sent: Ride(PULocationID=100, DOLocationID=68, trip_distance=0.4, total_amount=13.02, tpep_pickup_datetime=1761955846000)\n", "Sent: Ride(PULocationID=68, DOLocationID=144, trip_distance=4.01, total_amount=48.15, tpep_pickup_datetime=1761956664000)\n", "Sent: Ride(PULocationID=238, DOLocationID=107, trip_distance=5.4, total_amount=50.45, tpep_pickup_datetime=1761956704000)\n", "Sent: Ride(PULocationID=186, DOLocationID=33, trip_distance=4.16, total_amount=44.1, tpep_pickup_datetime=1761957441000)\n", "Sent: Ride(PULocationID=48, DOLocationID=196, trip_distance=8.15, total_amount=49.99, tpep_pickup_datetime=1761958446000)\n", "Sent: Ride(PULocationID=13, DOLocationID=107, trip_distance=4.97, total_amount=37.38, tpep_pickup_datetime=1761955283000)\n", "Sent: Ride(PULocationID=107, DOLocationID=140, trip_distance=2.6, total_amount=22.75, tpep_pickup_datetime=1761956605000)\n", "Sent: Ride(PULocationID=141, DOLocationID=143, trip_distance=1.47, total_amount=18.84, tpep_pickup_datetime=1761957687000)\n", "Sent: Ride(PULocationID=170, DOLocationID=79, trip_distance=1.59, total_amount=24.75, tpep_pickup_datetime=1761955644000)\n", "Sent: Ride(PULocationID=4, DOLocationID=113, trip_distance=0.99, total_amount=19.74, tpep_pickup_datetime=1761956917000)\n", "Sent: Ride(PULocationID=114, DOLocationID=4, trip_distance=0.88, total_amount=17.45, tpep_pickup_datetime=1761957745000)\n", "Sent: Ride(PULocationID=161, DOLocationID=263, trip_distance=3.05, total_amount=34.86, tpep_pickup_datetime=1761956135000)\n", "Sent: Ride(PULocationID=263, DOLocationID=237, trip_distance=0.79, total_amount=14.2, tpep_pickup_datetime=1761957780000)\n", "Sent: Ride(PULocationID=263, DOLocationID=90, trip_distance=3.1, total_amount=28.98, tpep_pickup_datetime=1761958374000)\n", "Sent: Ride(PULocationID=161, DOLocationID=262, trip_distance=2.46, total_amount=24.78, tpep_pickup_datetime=1761957461000)\n", "Sent: Ride(PULocationID=68, DOLocationID=79, trip_distance=6.25, total_amount=77.44, tpep_pickup_datetime=1761956039000)\n", "Sent: Ride(PULocationID=68, DOLocationID=137, trip_distance=1.4, total_amount=27.3, tpep_pickup_datetime=1761956403000)\n", "Sent: Ride(PULocationID=137, DOLocationID=170, trip_distance=0.8, total_amount=19.7, tpep_pickup_datetime=1761957702000)\n", "Sent: Ride(PULocationID=162, DOLocationID=229, trip_distance=0.8, total_amount=11.55, tpep_pickup_datetime=1761958542000)\n", "Sent: Ride(PULocationID=68, DOLocationID=42, trip_distance=6.31, total_amount=50.82, tpep_pickup_datetime=1761955373000)\n", "Sent: Ride(PULocationID=41, DOLocationID=263, trip_distance=1.92, total_amount=19.68, tpep_pickup_datetime=1761958166000)\n", "Sent: Ride(PULocationID=140, DOLocationID=209, trip_distance=6.1, total_amount=39.35, tpep_pickup_datetime=1761957889000)\n", "Sent: Ride(PULocationID=161, DOLocationID=164, trip_distance=0.97, total_amount=16.45, tpep_pickup_datetime=1761955699000)\n", "Sent: Ride(PULocationID=164, DOLocationID=231, trip_distance=2.31, total_amount=37.19, tpep_pickup_datetime=1761956454000)\n", "Sent: Ride(PULocationID=237, DOLocationID=236, trip_distance=1.53, total_amount=18.84, tpep_pickup_datetime=1761955789000)\n", "Sent: Ride(PULocationID=148, DOLocationID=148, trip_distance=0.0, total_amount=8.75, tpep_pickup_datetime=1761957945000)\n", "Sent: Ride(PULocationID=148, DOLocationID=68, trip_distance=2.8, total_amount=28.15, tpep_pickup_datetime=1761957966000)\n", "Sent: Ride(PULocationID=231, DOLocationID=87, trip_distance=0.7, total_amount=12.55, tpep_pickup_datetime=1761958490000)\n", "Sent: Ride(PULocationID=66, DOLocationID=37, trip_distance=3.77, total_amount=31.0, tpep_pickup_datetime=1761958073000)\n", "Sent: Ride(PULocationID=24, DOLocationID=262, trip_distance=2.63, total_amount=24.72, tpep_pickup_datetime=1761958756000)\n", "Sent: Ride(PULocationID=144, DOLocationID=186, trip_distance=3.0, total_amount=42.45, tpep_pickup_datetime=1761956480000)\n", "Sent: Ride(PULocationID=249, DOLocationID=231, trip_distance=1.12, total_amount=14.89, tpep_pickup_datetime=1761955444000)\n", "Sent: Ride(PULocationID=231, DOLocationID=148, trip_distance=1.18, total_amount=29.75, tpep_pickup_datetime=1761955863000)\n", "Sent: Ride(PULocationID=148, DOLocationID=234, trip_distance=1.1, total_amount=20.25, tpep_pickup_datetime=1761957725000)\n", "Sent: Ride(PULocationID=234, DOLocationID=246, trip_distance=1.9, total_amount=26.7, tpep_pickup_datetime=1761956765000)\n", "Sent: Ride(PULocationID=163, DOLocationID=142, trip_distance=1.04, total_amount=17.31, tpep_pickup_datetime=1761957754000)\n", "Sent: Ride(PULocationID=79, DOLocationID=107, trip_distance=0.75, total_amount=14.25, tpep_pickup_datetime=1761955857000)\n", "Sent: Ride(PULocationID=107, DOLocationID=163, trip_distance=2.13, total_amount=19.95, tpep_pickup_datetime=1761956490000)\n", "Sent: Ride(PULocationID=142, DOLocationID=142, trip_distance=0.4, total_amount=13.02, tpep_pickup_datetime=1761957827000)\n", "Sent: Ride(PULocationID=113, DOLocationID=113, trip_distance=0.38, total_amount=12.95, tpep_pickup_datetime=1761957395000)\n", "Sent: Ride(PULocationID=113, DOLocationID=151, trip_distance=7.92, total_amount=69.3, tpep_pickup_datetime=1761957823000)\n", "Sent: Ride(PULocationID=234, DOLocationID=163, trip_distance=1.44, total_amount=19.25, tpep_pickup_datetime=1761956112000)\n", "Sent: Ride(PULocationID=230, DOLocationID=256, trip_distance=5.77, total_amount=51.45, tpep_pickup_datetime=1761955515000)\n", "Sent: Ride(PULocationID=263, DOLocationID=236, trip_distance=0.33, total_amount=12.12, tpep_pickup_datetime=1761957509000)\n", "Sent: Ride(PULocationID=148, DOLocationID=4, trip_distance=1.05, total_amount=-19.25, tpep_pickup_datetime=1761957971000)\n", "Sent: Ride(PULocationID=148, DOLocationID=4, trip_distance=1.05, total_amount=19.25, tpep_pickup_datetime=1761957971000)\n", "Sent: Ride(PULocationID=88, DOLocationID=33, trip_distance=3.68, total_amount=28.14, tpep_pickup_datetime=1761956756000)\n", "Sent: Ride(PULocationID=33, DOLocationID=25, trip_distance=0.98, total_amount=13.0, tpep_pickup_datetime=1761957514000)\n", "Sent: Ride(PULocationID=148, DOLocationID=211, trip_distance=0.85, total_amount=18.65, tpep_pickup_datetime=1761955144000)\n", "Sent: Ride(PULocationID=211, DOLocationID=97, trip_distance=3.25, total_amount=31.15, tpep_pickup_datetime=1761956474000)\n", "Sent: Ride(PULocationID=48, DOLocationID=246, trip_distance=1.26, total_amount=18.06, tpep_pickup_datetime=1761958254000)\n", "Sent: Ride(PULocationID=239, DOLocationID=143, trip_distance=1.43, total_amount=15.3, tpep_pickup_datetime=1761955827000)\n", "Sent: Ride(PULocationID=50, DOLocationID=90, trip_distance=2.31, total_amount=27.3, tpep_pickup_datetime=1761956610000)\n", "Sent: Ride(PULocationID=234, DOLocationID=236, trip_distance=3.37, total_amount=31.5, tpep_pickup_datetime=1761957803000)\n", "Sent: Ride(PULocationID=148, DOLocationID=113, trip_distance=1.03, total_amount=20.58, tpep_pickup_datetime=1761955629000)\n", "Sent: Ride(PULocationID=137, DOLocationID=170, trip_distance=0.11, total_amount=9.45, tpep_pickup_datetime=1761957550000)\n", "Sent: Ride(PULocationID=79, DOLocationID=68, trip_distance=1.7, total_amount=21.33, tpep_pickup_datetime=1761957416000)\n", "Sent: Ride(PULocationID=68, DOLocationID=231, trip_distance=2.27, total_amount=33.45, tpep_pickup_datetime=1761958648000)\n", "Sent: Ride(PULocationID=90, DOLocationID=233, trip_distance=1.76, total_amount=27.33, tpep_pickup_datetime=1761956324000)\n", "Sent: Ride(PULocationID=231, DOLocationID=186, trip_distance=2.52, total_amount=35.7, tpep_pickup_datetime=1761955460000)\n", "Sent: Ride(PULocationID=107, DOLocationID=236, trip_distance=4.01, total_amount=28.95, tpep_pickup_datetime=1761955447000)\n", "Sent: Ride(PULocationID=231, DOLocationID=127, trip_distance=15.53, total_amount=81.65, tpep_pickup_datetime=1761955352000)\n", "Sent: Ride(PULocationID=161, DOLocationID=246, trip_distance=2.33, total_amount=28.98, tpep_pickup_datetime=1761955667000)\n", "Sent: Ride(PULocationID=161, DOLocationID=100, trip_distance=0.23, total_amount=13.1, tpep_pickup_datetime=1761955396000)\n", "Sent: Ride(PULocationID=48, DOLocationID=151, trip_distance=2.59, total_amount=22.65, tpep_pickup_datetime=1761956438000)\n", "Sent: Ride(PULocationID=143, DOLocationID=262, trip_distance=2.21, total_amount=25.4, tpep_pickup_datetime=1761958058000)\n", "Sent: Ride(PULocationID=234, DOLocationID=234, trip_distance=3.02, total_amount=52.32, tpep_pickup_datetime=1761955791000)\n", "Sent: Ride(PULocationID=234, DOLocationID=4, trip_distance=1.64, total_amount=34.86, tpep_pickup_datetime=1761958476000)\n", "Sent: Ride(PULocationID=263, DOLocationID=163, trip_distance=2.58, total_amount=25.62, tpep_pickup_datetime=1761956710000)\n", "Sent: Ride(PULocationID=148, DOLocationID=79, trip_distance=0.79, total_amount=20.25, tpep_pickup_datetime=1761956022000)\n", "Sent: Ride(PULocationID=79, DOLocationID=114, trip_distance=0.61, total_amount=17.45, tpep_pickup_datetime=1761957040000)\n", "Sent: Ride(PULocationID=79, DOLocationID=249, trip_distance=2.04, total_amount=32.34, tpep_pickup_datetime=1761957906000)\n", "Sent: Ride(PULocationID=234, DOLocationID=260, trip_distance=6.9, total_amount=50.09, tpep_pickup_datetime=1761956327000)\n", "Sent: Ride(PULocationID=230, DOLocationID=48, trip_distance=0.94, total_amount=15.54, tpep_pickup_datetime=1761957874000)\n", "Sent: Ride(PULocationID=161, DOLocationID=143, trip_distance=1.88, total_amount=17.85, tpep_pickup_datetime=1761953770000)\n", "Sent: Ride(PULocationID=143, DOLocationID=236, trip_distance=2.75, total_amount=26.4, tpep_pickup_datetime=1761954614000)\n", "Sent: Ride(PULocationID=140, DOLocationID=48, trip_distance=2.65, total_amount=34.02, tpep_pickup_datetime=1761956414000)\n", "Sent: Ride(PULocationID=170, DOLocationID=229, trip_distance=1.5, total_amount=22.25, tpep_pickup_datetime=1761955922000)\n", "Sent: Ride(PULocationID=48, DOLocationID=246, trip_distance=1.2, total_amount=15.69, tpep_pickup_datetime=1761958070000)\n", "Sent: Ride(PULocationID=234, DOLocationID=152, trip_distance=8.0, total_amount=58.38, tpep_pickup_datetime=1761956935000)\n", "Sent: Ride(PULocationID=166, DOLocationID=116, trip_distance=1.21, total_amount=11.96, tpep_pickup_datetime=1761956038000)\n", "Sent: Ride(PULocationID=263, DOLocationID=238, trip_distance=1.75, total_amount=18.8, tpep_pickup_datetime=1761957587000)\n", "Sent: Ride(PULocationID=48, DOLocationID=90, trip_distance=1.78, total_amount=26.46, tpep_pickup_datetime=1761955155000)\n", "Sent: Ride(PULocationID=237, DOLocationID=107, trip_distance=2.28, total_amount=28.44, tpep_pickup_datetime=1761956921000)\n", "Sent: Ride(PULocationID=107, DOLocationID=162, trip_distance=1.15, total_amount=16.38, tpep_pickup_datetime=1761958446000)\n", "Sent: Ride(PULocationID=236, DOLocationID=75, trip_distance=1.63, total_amount=18.0, tpep_pickup_datetime=1761955216000)\n", "Sent: Ride(PULocationID=68, DOLocationID=186, trip_distance=1.14, total_amount=17.31, tpep_pickup_datetime=1761956032000)\n", "Sent: Ride(PULocationID=234, DOLocationID=90, trip_distance=0.76, total_amount=26.25, tpep_pickup_datetime=1761958542000)\n", "Sent: Ride(PULocationID=151, DOLocationID=116, trip_distance=2.17, total_amount=20.04, tpep_pickup_datetime=1761957776000)\n", "Sent: Ride(PULocationID=239, DOLocationID=79, trip_distance=4.19, total_amount=47.35, tpep_pickup_datetime=1761957071000)\n", "Sent: Ride(PULocationID=138, DOLocationID=164, trip_distance=11.3, total_amount=69.85, tpep_pickup_datetime=1761958772000)\n", "Sent: Ride(PULocationID=164, DOLocationID=158, trip_distance=1.9, total_amount=26.85, tpep_pickup_datetime=1761958643000)\n", "Sent: Ride(PULocationID=141, DOLocationID=263, trip_distance=1.1, total_amount=15.45, tpep_pickup_datetime=1761956554000)\n", "Sent: Ride(PULocationID=141, DOLocationID=234, trip_distance=3.1, total_amount=33.15, tpep_pickup_datetime=1761957289000)\n", "Sent: Ride(PULocationID=138, DOLocationID=134, trip_distance=6.05, total_amount=41.65, tpep_pickup_datetime=1761956033000)\n", "Sent: Ride(PULocationID=138, DOLocationID=239, trip_distance=11.5, total_amount=82.86, tpep_pickup_datetime=1761958551000)\n", "Sent: Ride(PULocationID=79, DOLocationID=234, trip_distance=1.72, total_amount=26.25, tpep_pickup_datetime=1761956681000)\n", "Sent: Ride(PULocationID=87, DOLocationID=61, trip_distance=6.25, total_amount=41.45, tpep_pickup_datetime=1761957608000)\n", "Sent: Ride(PULocationID=4, DOLocationID=80, trip_distance=3.17, total_amount=25.55, tpep_pickup_datetime=1761958324000)\n", "Sent: Ride(PULocationID=263, DOLocationID=137, trip_distance=3.33, total_amount=31.45, tpep_pickup_datetime=1761955496000)\n", "Sent: Ride(PULocationID=137, DOLocationID=13, trip_distance=6.92, total_amount=64.26, tpep_pickup_datetime=1761957268000)\n", "Sent: Ride(PULocationID=249, DOLocationID=170, trip_distance=2.41, total_amount=41.58, tpep_pickup_datetime=1761955218000)\n", "Sent: Ride(PULocationID=107, DOLocationID=113, trip_distance=0.62, total_amount=23.94, tpep_pickup_datetime=1761955586000)\n", "Sent: Ride(PULocationID=79, DOLocationID=263, trip_distance=4.62, total_amount=49.14, tpep_pickup_datetime=1761956998000)\n", "Sent: Ride(PULocationID=79, DOLocationID=246, trip_distance=2.12, total_amount=25.45, tpep_pickup_datetime=1761956111000)\n", "Sent: Ride(PULocationID=246, DOLocationID=50, trip_distance=0.39, total_amount=22.14, tpep_pickup_datetime=1761957432000)\n", "Sent: Ride(PULocationID=48, DOLocationID=142, trip_distance=0.9, total_amount=14.7, tpep_pickup_datetime=1761958636000)\n", "Sent: Ride(PULocationID=263, DOLocationID=75, trip_distance=0.65, total_amount=13.8, tpep_pickup_datetime=1761957376000)\n", "Sent: Ride(PULocationID=141, DOLocationID=263, trip_distance=0.67, total_amount=11.5, tpep_pickup_datetime=1761958327000)\n", "Sent: Ride(PULocationID=48, DOLocationID=68, trip_distance=0.4, total_amount=13.02, tpep_pickup_datetime=1761956732000)\n", "Sent: Ride(PULocationID=186, DOLocationID=41, trip_distance=4.87, total_amount=-36.05, tpep_pickup_datetime=1761957722000)\n", "Sent: Ride(PULocationID=186, DOLocationID=74, trip_distance=4.87, total_amount=36.05, tpep_pickup_datetime=1761957722000)\n", "Sent: Ride(PULocationID=144, DOLocationID=239, trip_distance=5.79, total_amount=53.34, tpep_pickup_datetime=1761957109000)\n", "Sent: Ride(PULocationID=137, DOLocationID=79, trip_distance=0.99, total_amount=19.95, tpep_pickup_datetime=1761956287000)\n", "Sent: Ride(PULocationID=79, DOLocationID=137, trip_distance=1.73, total_amount=23.35, tpep_pickup_datetime=1761957380000)\n", "Sent: Ride(PULocationID=152, DOLocationID=151, trip_distance=1.39, total_amount=14.16, tpep_pickup_datetime=1761955467000)\n", "Sent: Ride(PULocationID=239, DOLocationID=107, trip_distance=3.63, total_amount=32.34, tpep_pickup_datetime=1761956364000)\n", "Sent: Ride(PULocationID=231, DOLocationID=263, trip_distance=6.34, total_amount=52.5, tpep_pickup_datetime=1761955711000)\n", "Sent: Ride(PULocationID=87, DOLocationID=74, trip_distance=8.5, total_amount=54.15, tpep_pickup_datetime=1761955795000)\n", "Sent: Ride(PULocationID=74, DOLocationID=42, trip_distance=2.2, total_amount=17.5, tpep_pickup_datetime=1761957673000)\n", "Sent: Ride(PULocationID=80, DOLocationID=34, trip_distance=4.3, total_amount=26.75, tpep_pickup_datetime=1761957260000)\n", "Sent: Ride(PULocationID=144, DOLocationID=112, trip_distance=4.4, total_amount=44.95, tpep_pickup_datetime=1761956555000)\n", "Sent: Ride(PULocationID=144, DOLocationID=68, trip_distance=2.0, total_amount=47.45, tpep_pickup_datetime=1761955568000)\n", "Sent: Ride(PULocationID=68, DOLocationID=114, trip_distance=1.7, total_amount=27.25, tpep_pickup_datetime=1761958278000)\n", "Sent: Ride(PULocationID=162, DOLocationID=179, trip_distance=3.86, total_amount=33.69, tpep_pickup_datetime=1761956794000)\n", "Sent: Ride(PULocationID=264, DOLocationID=229, trip_distance=1.38, total_amount=18.9, tpep_pickup_datetime=1761958759000)\n", "Sent: Ride(PULocationID=79, DOLocationID=233, trip_distance=2.18, total_amount=31.94, tpep_pickup_datetime=1761956161000)\n", "Sent: Ride(PULocationID=233, DOLocationID=262, trip_distance=1.85, total_amount=19.74, tpep_pickup_datetime=1761957605000)\n", "Sent: Ride(PULocationID=234, DOLocationID=170, trip_distance=1.47, total_amount=21.25, tpep_pickup_datetime=1761956512000)\n", "Sent: Ride(PULocationID=233, DOLocationID=140, trip_distance=2.02, total_amount=20.58, tpep_pickup_datetime=1761957479000)\n", "Sent: Ride(PULocationID=68, DOLocationID=164, trip_distance=1.9, total_amount=24.63, tpep_pickup_datetime=1761958326000)\n", "Sent: Ride(PULocationID=50, DOLocationID=252, trip_distance=15.65, total_amount=106.85, tpep_pickup_datetime=1761956198000)\n", "Sent: Ride(PULocationID=170, DOLocationID=141, trip_distance=2.03, total_amount=18.9, tpep_pickup_datetime=1761958441000)\n", "Sent: Ride(PULocationID=68, DOLocationID=243, trip_distance=7.66, total_amount=65.94, tpep_pickup_datetime=1761956564000)\n", "Sent: Ride(PULocationID=142, DOLocationID=262, trip_distance=2.73, total_amount=19.9, tpep_pickup_datetime=1761955822000)\n", "Sent: Ride(PULocationID=262, DOLocationID=162, trip_distance=2.05, total_amount=-16.45, tpep_pickup_datetime=1761956792000)\n", "Sent: Ride(PULocationID=262, DOLocationID=162, trip_distance=2.05, total_amount=16.45, tpep_pickup_datetime=1761956792000)\n", "Sent: Ride(PULocationID=229, DOLocationID=229, trip_distance=0.41, total_amount=12.48, tpep_pickup_datetime=1761957355000)\n", "Sent: Ride(PULocationID=68, DOLocationID=233, trip_distance=2.05, total_amount=24.35, tpep_pickup_datetime=1761956966000)\n", "Sent: Ride(PULocationID=137, DOLocationID=232, trip_distance=2.58, total_amount=34.86, tpep_pickup_datetime=1761955867000)\n", "Sent: Ride(PULocationID=232, DOLocationID=238, trip_distance=7.27, total_amount=45.21, tpep_pickup_datetime=1761957725000)\n", "Sent: Ride(PULocationID=132, DOLocationID=10, trip_distance=3.38, total_amount=24.31, tpep_pickup_datetime=1761957541000)\n", "Sent: Ride(PULocationID=107, DOLocationID=234, trip_distance=0.5, total_amount=20.58, tpep_pickup_datetime=1761955758000)\n", "Sent: Ride(PULocationID=234, DOLocationID=236, trip_distance=4.53, total_amount=48.3, tpep_pickup_datetime=1761957112000)\n", "Sent: Ride(PULocationID=48, DOLocationID=142, trip_distance=1.09, total_amount=17.22, tpep_pickup_datetime=1761955648000)\n", "Sent: Ride(PULocationID=239, DOLocationID=100, trip_distance=2.5, total_amount=27.3, tpep_pickup_datetime=1761956205000)\n", "Sent: Ride(PULocationID=100, DOLocationID=68, trip_distance=1.06, total_amount=20.58, tpep_pickup_datetime=1761957629000)\n", "Sent: Ride(PULocationID=125, DOLocationID=151, trip_distance=6.01, total_amount=38.85, tpep_pickup_datetime=1761956924000)\n", "Sent: Ride(PULocationID=107, DOLocationID=107, trip_distance=0.77, total_amount=16.38, tpep_pickup_datetime=1761955684000)\n", "Sent: Ride(PULocationID=113, DOLocationID=164, trip_distance=1.0, total_amount=18.8, tpep_pickup_datetime=1761955373000)\n", "Sent: Ride(PULocationID=164, DOLocationID=137, trip_distance=1.0, total_amount=23.2, tpep_pickup_datetime=1761956086000)\n", "Sent: Ride(PULocationID=233, DOLocationID=141, trip_distance=1.2, total_amount=12.95, tpep_pickup_datetime=1761956985000)\n", "Sent: Ride(PULocationID=163, DOLocationID=141, trip_distance=1.8, total_amount=19.75, tpep_pickup_datetime=1761958331000)\n", "Sent: Ride(PULocationID=161, DOLocationID=230, trip_distance=0.7, total_amount=16.05, tpep_pickup_datetime=1761956008000)\n", "Sent: Ride(PULocationID=161, DOLocationID=263, trip_distance=1.8, total_amount=21.4, tpep_pickup_datetime=1761956913000)\n", "Sent: Ride(PULocationID=263, DOLocationID=142, trip_distance=2.2, total_amount=19.8, tpep_pickup_datetime=1761957773000)\n", "Sent: Ride(PULocationID=163, DOLocationID=263, trip_distance=2.68, total_amount=24.78, tpep_pickup_datetime=1761955217000)\n", "Sent: Ride(PULocationID=263, DOLocationID=141, trip_distance=0.56, total_amount=11.62, tpep_pickup_datetime=1761955998000)\n", "Sent: Ride(PULocationID=263, DOLocationID=83, trip_distance=5.16, total_amount=31.85, tpep_pickup_datetime=1761957529000)\n", "Sent: Ride(PULocationID=113, DOLocationID=246, trip_distance=2.6, total_amount=39.9, tpep_pickup_datetime=1761957214000)\n", "Sent: Ride(PULocationID=211, DOLocationID=114, trip_distance=0.8, total_amount=15.65, tpep_pickup_datetime=1761956007000)\n", "Sent: Ride(PULocationID=144, DOLocationID=158, trip_distance=1.9, total_amount=24.78, tpep_pickup_datetime=1761956443000)\n", "Sent: Ride(PULocationID=158, DOLocationID=87, trip_distance=3.82, total_amount=28.45, tpep_pickup_datetime=1761957437000)\n", "Sent: Ride(PULocationID=87, DOLocationID=107, trip_distance=4.34, total_amount=40.69, tpep_pickup_datetime=1761958760000)\n", "Sent: Ride(PULocationID=114, DOLocationID=137, trip_distance=2.0, total_amount=25.6, tpep_pickup_datetime=1761957565000)\n", "Sent: Ride(PULocationID=265, DOLocationID=265, trip_distance=0.0, total_amount=96.0, tpep_pickup_datetime=1761958241000)\n", "Sent: Ride(PULocationID=230, DOLocationID=144, trip_distance=3.5, total_amount=32.55, tpep_pickup_datetime=1761956658000)\n", "Sent: Ride(PULocationID=148, DOLocationID=160, trip_distance=7.45, total_amount=40.45, tpep_pickup_datetime=1761958726000)\n", "Sent: Ride(PULocationID=48, DOLocationID=256, trip_distance=6.03, total_amount=68.46, tpep_pickup_datetime=1761955214000)\n", "Sent: Ride(PULocationID=24, DOLocationID=152, trip_distance=1.3, total_amount=10.4, tpep_pickup_datetime=1761956874000)\n", "Sent: Ride(PULocationID=236, DOLocationID=236, trip_distance=0.43, total_amount=11.75, tpep_pickup_datetime=1761956620000)\n", "Sent: Ride(PULocationID=236, DOLocationID=263, trip_distance=1.15, total_amount=16.32, tpep_pickup_datetime=1761956897000)\n", "Sent: Ride(PULocationID=263, DOLocationID=238, trip_distance=1.99, total_amount=19.1, tpep_pickup_datetime=1761957478000)\n", "Sent: Ride(PULocationID=237, DOLocationID=24, trip_distance=2.3, total_amount=19.7, tpep_pickup_datetime=1761955265000)\n", "Sent: Ride(PULocationID=238, DOLocationID=239, trip_distance=0.8, total_amount=13.8, tpep_pickup_datetime=1761956041000)\n", "Sent: Ride(PULocationID=142, DOLocationID=262, trip_distance=2.1, total_amount=20.5, tpep_pickup_datetime=1761956804000)\n", "Sent: Ride(PULocationID=263, DOLocationID=226, trip_distance=4.0, total_amount=23.45, tpep_pickup_datetime=1761957750000)\n", "Sent: Ride(PULocationID=239, DOLocationID=143, trip_distance=1.53, total_amount=17.95, tpep_pickup_datetime=1761958179000)\n", "Sent: Ride(PULocationID=144, DOLocationID=231, trip_distance=1.2, total_amount=18.9, tpep_pickup_datetime=1761955678000)\n", "Sent: Ride(PULocationID=113, DOLocationID=229, trip_distance=2.66, total_amount=30.66, tpep_pickup_datetime=1761958375000)\n", "Sent: Ride(PULocationID=113, DOLocationID=107, trip_distance=0.75, total_amount=21.42, tpep_pickup_datetime=1761955353000)\n", "Sent: Ride(PULocationID=107, DOLocationID=88, trip_distance=4.28, total_amount=36.05, tpep_pickup_datetime=1761957278000)\n", "Sent: Ride(PULocationID=263, DOLocationID=262, trip_distance=0.67, total_amount=13.5, tpep_pickup_datetime=1761956709000)\n", "Sent: Ride(PULocationID=148, DOLocationID=79, trip_distance=1.5, total_amount=23.1, tpep_pickup_datetime=1761955608000)\n", "Sent: Ride(PULocationID=41, DOLocationID=238, trip_distance=1.73, total_amount=19.68, tpep_pickup_datetime=1761955822000)\n", "Sent: Ride(PULocationID=163, DOLocationID=140, trip_distance=1.44, total_amount=18.11, tpep_pickup_datetime=1761958756000)\n", "Sent: Ride(PULocationID=107, DOLocationID=137, trip_distance=0.49, total_amount=12.55, tpep_pickup_datetime=1761957049000)\n", "Sent: Ride(PULocationID=137, DOLocationID=68, trip_distance=1.48, total_amount=23.05, tpep_pickup_datetime=1761957555000)\n", "Sent: Ride(PULocationID=166, DOLocationID=41, trip_distance=1.1, total_amount=11.1, tpep_pickup_datetime=1761955442000)\n", "Sent: Ride(PULocationID=239, DOLocationID=140, trip_distance=1.8, total_amount=18.85, tpep_pickup_datetime=1761956742000)\n", "Sent: Ride(PULocationID=140, DOLocationID=237, trip_distance=0.7, total_amount=14.31, tpep_pickup_datetime=1761957607000)\n", "Sent: Ride(PULocationID=237, DOLocationID=107, trip_distance=2.8, total_amount=30.65, tpep_pickup_datetime=1761958306000)\n", "Sent: Ride(PULocationID=100, DOLocationID=239, trip_distance=2.94, total_amount=38.22, tpep_pickup_datetime=1761958211000)\n", "Sent: Ride(PULocationID=48, DOLocationID=42, trip_distance=4.7, total_amount=34.85, tpep_pickup_datetime=1761955550000)\n", "Sent: Ride(PULocationID=263, DOLocationID=236, trip_distance=0.9, total_amount=13.8, tpep_pickup_datetime=1761957769000)\n", "Sent: Ride(PULocationID=238, DOLocationID=114, trip_distance=6.12, total_amount=56.35, tpep_pickup_datetime=1761955775000)\n", "Sent: Ride(PULocationID=138, DOLocationID=232, trip_distance=9.37, total_amount=55.4, tpep_pickup_datetime=1761955488000)\n", "Sent: Ride(PULocationID=232, DOLocationID=137, trip_distance=1.93, total_amount=22.05, tpep_pickup_datetime=1761957174000)\n", "Sent: Ride(PULocationID=137, DOLocationID=263, trip_distance=2.72, total_amount=20.65, tpep_pickup_datetime=1761958356000)\n", "Sent: Ride(PULocationID=246, DOLocationID=263, trip_distance=4.6, total_amount=39.9, tpep_pickup_datetime=1761956440000)\n", "Sent: Ride(PULocationID=263, DOLocationID=141, trip_distance=0.6, total_amount=11.92, tpep_pickup_datetime=1761958168000)\n", "Sent: Ride(PULocationID=263, DOLocationID=141, trip_distance=1.04, total_amount=14.64, tpep_pickup_datetime=1761955790000)\n", "Sent: Ride(PULocationID=237, DOLocationID=75, trip_distance=1.93, total_amount=18.7, tpep_pickup_datetime=1761956503000)\n", "Sent: Ride(PULocationID=233, DOLocationID=107, trip_distance=0.76, total_amount=21.42, tpep_pickup_datetime=1761956475000)\n", "Sent: Ride(PULocationID=262, DOLocationID=48, trip_distance=3.12, total_amount=28.14, tpep_pickup_datetime=1761955204000)\n", "Sent: Ride(PULocationID=48, DOLocationID=68, trip_distance=0.45, total_amount=13.02, tpep_pickup_datetime=1761956902000)\n", "Sent: Ride(PULocationID=246, DOLocationID=186, trip_distance=0.7, total_amount=13.95, tpep_pickup_datetime=1761957726000)\n", "Sent: Ride(PULocationID=100, DOLocationID=164, trip_distance=0.15, total_amount=9.45, tpep_pickup_datetime=1761958138000)\n", "Sent: Ride(PULocationID=234, DOLocationID=113, trip_distance=0.43, total_amount=18.06, tpep_pickup_datetime=1761958439000)\n", "Sent: Ride(PULocationID=239, DOLocationID=116, trip_distance=3.3, total_amount=28.35, tpep_pickup_datetime=1761955931000)\n", "Sent: Ride(PULocationID=107, DOLocationID=79, trip_distance=0.9, total_amount=18.9, tpep_pickup_datetime=1761958296000)\n", "took 11.49 seconds\n" ] } ], "source": [ "import time\n", "\n", "t0 = time.time()\n", "\n", "for _, row in df.iterrows():\n", " ride = ride_from_row(row)\n", " producer.send(topic_name, value=ride)\n", " print(f\"Sent: {ride}\")\n", " time.sleep(0.01)\n", "\n", "producer.flush()\n", "\n", "t1 = time.time()\n", "print(f'took {(t1 - t0):.2f} seconds')" ] }, { "cell_type": "code", "execution_count": null, "id": "a1ca66fe", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "streaming-workshop", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.1" } }, "nbformat": 4, "nbformat_minor": 5 } ================================================ FILE: 07-streaming/workshop/live/pyproject.flink.toml ================================================ [project] name = "pyflink-workshop" version = "0.1.0" requires-python = ">=3.12" dependencies = [ "apache-flink==2.2.0", ] ================================================ FILE: 07-streaming/workshop/live/pyproject.toml ================================================ [project] name = "streaming-workshop" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.12" dependencies = [ "kafka-python>=2.3.0", "pandas>=3.0.1", "psycopg2-binary>=2.9.11", "pyarrow>=23.0.1", ] [dependency-groups] dev = [ "jupyter>=1.1.1", ] ================================================ FILE: 07-streaming/workshop/live/src/job/aggregation_job.py ================================================ from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import EnvironmentSettings, StreamTableEnvironment def create_events_source_kafka(t_env): table_name = "events" source_ddl = f""" CREATE TABLE {table_name} ( PULocationID INTEGER, DOLocationID INTEGER, trip_distance DOUBLE, total_amount DOUBLE, tpep_pickup_datetime BIGINT, event_timestamp AS TO_TIMESTAMP_LTZ(tpep_pickup_datetime, 3), WATERMARK for event_timestamp as event_timestamp - INTERVAL '5' SECOND ) WITH ( 'connector' = 'kafka', 'properties.bootstrap.servers' = 'redpanda:29092', 'topic' = 'rides', 'scan.startup.mode' = 'earliest-offset', 'properties.auto.offset.reset' = 'earliest', 'format' = 'json' ); """ t_env.execute_sql(source_ddl) return table_name def create_events_aggregated_sink(t_env): table_name = 'processed_events_aggregated' sink_ddl = f""" CREATE TABLE {table_name} ( window_start TIMESTAMP(3), PULocationID INT, num_trips BIGINT, total_revenue DOUBLE, PRIMARY KEY (window_start, PULocationID) NOT ENFORCED ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:postgresql://postgres:5432/postgres', 'table-name' = '{table_name}', 'username' = 'postgres', 'password' = 'postgres', 'driver' = 'org.postgresql.Driver' ); """ t_env.execute_sql(sink_ddl) return table_name def log_aggregation(): env = StreamExecutionEnvironment.get_execution_environment() env.enable_checkpointing(10 * 1000) env.set_parallelism(3) settings = EnvironmentSettings.new_instance().in_streaming_mode().build() t_env = StreamTableEnvironment.create(env, environment_settings=settings) try: source_table = create_events_source_kafka(t_env) aggregated_table = create_events_aggregated_sink(t_env) t_env.execute_sql(f""" INSERT INTO {aggregated_table} SELECT window_start, PULocationID, COUNT(*) AS num_trips, SUM(total_amount) AS total_revenue FROM TABLE( TUMBLE(TABLE {source_table}, DESCRIPTOR(event_timestamp), INTERVAL '1' HOUR) ) GROUP BY window_start, PULocationID; """).wait() except Exception as e: print("Writing records from Kafka to JDBC failed:", str(e)) if __name__ == '__main__': log_aggregation() ================================================ FILE: 07-streaming/workshop/live/src/job/pass_through_job.py ================================================ from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import EnvironmentSettings, StreamTableEnvironment def create_events_source_kafka(t_env): table_name = "events" source_ddl = f""" CREATE TABLE {table_name} ( PULocationID INTEGER, DOLocationID INTEGER, trip_distance DOUBLE, total_amount DOUBLE, tpep_pickup_datetime BIGINT ) WITH ( 'connector' = 'kafka', 'properties.bootstrap.servers' = 'redpanda:29092', 'topic' = 'rides', 'scan.startup.mode' = 'latest-offset', 'properties.auto.offset.reset' = 'latest', 'format' = 'json' ); """ t_env.execute_sql(source_ddl) return table_name def create_processed_events_sink_postgres(t_env): table_name = 'processed_events' sink_ddl = f""" CREATE TABLE {table_name} ( PULocationID INTEGER, DOLocationID INTEGER, trip_distance DOUBLE, total_amount DOUBLE, pickup_datetime TIMESTAMP ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:postgresql://postgres:5432/postgres', 'table-name' = '{table_name}', 'username' = 'postgres', 'password' = 'postgres', 'driver' = 'org.postgresql.Driver' ); """ t_env.execute_sql(sink_ddl) return table_name def log_processing(): env = StreamExecutionEnvironment.get_execution_environment() env.enable_checkpointing(10 * 1000) # checkpoint every 10 seconds settings = EnvironmentSettings.new_instance().in_streaming_mode().build() t_env = StreamTableEnvironment.create(env, environment_settings=settings) source_table = create_events_source_kafka(t_env) postgres_sink = create_processed_events_sink_postgres(t_env) t_env.execute_sql( f""" INSERT INTO {postgres_sink} SELECT PULocationID, DOLocationID, trip_distance, total_amount, TO_TIMESTAMP_LTZ(tpep_pickup_datetime, 3) as pickup_datetime FROM {source_table} """ ).wait() if __name__ == '__main__': log_processing() ================================================ FILE: 07-streaming/workshop/live/src/producers/models.py ================================================ import json import dataclasses from dataclasses import dataclass @dataclass class Ride: PULocationID: int DOLocationID: int trip_distance: float total_amount: float tpep_pickup_datetime: int # epoch milliseconds def ride_from_row(row): return Ride( PULocationID=int(row['PULocationID']), DOLocationID=int(row['DOLocationID']), trip_distance=float(row['trip_distance']), total_amount=float(row['total_amount']), tpep_pickup_datetime=int(row['tpep_pickup_datetime'].timestamp() * 1000), ) def ride_serializer(ride): ride_dict = dataclasses.asdict(ride) ride_json = json.dumps(ride_dict).encode('utf-8') return ride_json def ride_deserializer(data): json_str = data.decode('utf-8') ride_dict = json.loads(json_str) return Ride(**ride_dict) ================================================ FILE: 07-streaming/workshop/live/src/producers/producer_realtime.py ================================================ import dataclasses import json import random import sys import time from datetime import datetime, timezone from pathlib import Path sys.path.insert(0, str(Path(__file__).parent.parent)) from kafka import KafkaProducer from models import Ride # Top pickup locations from the actual NYC yellow taxi data. # PULocationID is a taxi zone ID (1-263) defined by the NYC TLC. # See https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv PICKUP_LOCATIONS = [ 79, # East Village, Manhattan 107, # Gramercy, Manhattan 48, # Clinton East (Hell's Kitchen), Manhattan 132, # JFK Airport 234, # Union Sq, Manhattan 148, # Lower East Side, Manhattan 249, # West Village, Manhattan 68, # East Chelsea, Manhattan 90, # Flatiron, Manhattan 263, # Yorkville West, Manhattan 138, # LaGuardia Airport 230, # Times Sq/Theatre District, Manhattan 161, # Midtown Center, Manhattan 162, # Midtown East, Manhattan 170, # Murray Hill, Manhattan 237, # Upper East Side South, Manhattan 239, # Upper West Side South, Manhattan 186, # Penn Station/Madison Sq West, Manhattan 164, # Midtown South, Manhattan 236, # Upper East Side North, Manhattan ] DROPOFF_LOCATIONS = PICKUP_LOCATIONS # same pool for simplicity def make_ride(delay_seconds=0): now_ms = int(time.time() * 1000) - delay_seconds * 1000 return Ride( PULocationID=random.choice(PICKUP_LOCATIONS), DOLocationID=random.choice(DROPOFF_LOCATIONS), trip_distance=round(random.uniform(0.5, 20.0), 2), total_amount=round(random.uniform(5.0, 100.0), 2), tpep_pickup_datetime=now_ms, ) def ride_serializer(ride): return json.dumps(dataclasses.asdict(ride)).encode('utf-8') server = 'localhost:9092' producer = KafkaProducer( bootstrap_servers=[server], value_serializer=ride_serializer, ) topic_name = 'rides' count = 0 print("Sending events (Ctrl+C to stop)...") print() try: while True: # ~20% chance of a late event (3-10 seconds old) if random.random() < 0.2: delay = random.randint(3, 10) ride = make_ride(delay_seconds=delay) ts = datetime.fromtimestamp(ride.tpep_pickup_datetime / 1000, tz=timezone.utc) print(f" LATE ({delay}s) -> PU={ride.PULocationID} ts={ts:%H:%M:%S}") else: ride = make_ride() ts = datetime.fromtimestamp(ride.tpep_pickup_datetime / 1000, tz=timezone.utc) print(f" on time -> PU={ride.PULocationID} ts={ts:%H:%M:%S}") producer.send(topic_name, value=ride) count += 1 time.sleep(0.5) except KeyboardInterrupt: producer.flush() print(f"\nSent {count} events") ================================================ FILE: 07-streaming/workshop/pyproject.flink.toml ================================================ [project] name = "pyflink-workshop" version = "0.1.0" requires-python = ">=3.12" dependencies = [ "apache-flink==2.2.0", ] ================================================ FILE: 07-streaming/workshop/pyproject.toml ================================================ [project] name = "workshop" version = "0.1.0" description = "PyFlink Stream Processing Workshop" requires-python = ">=3.12" dependencies = [ "kafka-python>=2.3.0", "pandas>=2.2.0", "psycopg2-binary>=2.9.11", "pyarrow>=19.0.0", ] ================================================ FILE: 07-streaming/workshop/src/consumers/consumer.py ================================================ import sys from datetime import datetime from pathlib import Path sys.path.insert(0, str(Path(__file__).parent.parent)) from kafka import KafkaConsumer from models import ride_deserializer server = 'localhost:9092' topic_name = 'rides' consumer = KafkaConsumer( topic_name, bootstrap_servers=[server], auto_offset_reset='earliest', group_id='rides-console', value_deserializer=ride_deserializer ) print(f"Listening to {topic_name}...") count = 0 for message in consumer: ride = message.value pickup_dt = datetime.fromtimestamp(ride.tpep_pickup_datetime / 1000) print(f"Received: PU={ride.PULocationID}, DO={ride.DOLocationID}, " f"distance={ride.trip_distance}, amount=${ride.total_amount:.2f}, " f"pickup={pickup_dt}") count += 1 if count >= 10: print(f"\n... received {count} messages so far (stopping after 10 for demo)") break consumer.close() ================================================ FILE: 07-streaming/workshop/src/consumers/consumer_postgres.py ================================================ import sys from datetime import datetime from pathlib import Path sys.path.insert(0, str(Path(__file__).parent.parent)) import psycopg2 from kafka import KafkaConsumer from models import ride_deserializer server = 'localhost:9092' topic_name = 'rides' # Connect to PostgreSQL conn = psycopg2.connect( host='localhost', port=5432, database='postgres', user='postgres', password='postgres' ) conn.autocommit = True cur = conn.cursor() consumer = KafkaConsumer( topic_name, bootstrap_servers=[server], auto_offset_reset='earliest', group_id='rides-to-postgres', value_deserializer=ride_deserializer ) print(f"Listening to {topic_name} and writing to PostgreSQL...") count = 0 for message in consumer: ride = message.value pickup_dt = datetime.fromtimestamp(ride.tpep_pickup_datetime / 1000) cur.execute( """INSERT INTO processed_events (PULocationID, DOLocationID, trip_distance, total_amount, pickup_datetime) VALUES (%s, %s, %s, %s, %s)""", (ride.PULocationID, ride.DOLocationID, ride.trip_distance, ride.total_amount, pickup_dt) ) count += 1 if count % 100 == 0: print(f"Inserted {count} rows...") consumer.close() cur.close() conn.close() ================================================ FILE: 07-streaming/workshop/src/job/aggregation_job.py ================================================ from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import EnvironmentSettings, StreamTableEnvironment def create_events_aggregated_sink(t_env): table_name = 'processed_events_aggregated' sink_ddl = f""" CREATE TABLE {table_name} ( window_start TIMESTAMP(3), PULocationID INT, num_trips BIGINT, total_revenue DOUBLE, PRIMARY KEY (window_start, PULocationID) NOT ENFORCED ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:postgresql://postgres:5432/postgres', 'table-name' = '{table_name}', 'username' = 'postgres', 'password' = 'postgres', 'driver' = 'org.postgresql.Driver' ); """ t_env.execute_sql(sink_ddl) return table_name def create_events_source_kafka(t_env): table_name = "events" source_ddl = f""" CREATE TABLE {table_name} ( PULocationID INTEGER, DOLocationID INTEGER, trip_distance DOUBLE, total_amount DOUBLE, tpep_pickup_datetime BIGINT, event_timestamp AS TO_TIMESTAMP_LTZ(tpep_pickup_datetime, 3), WATERMARK for event_timestamp as event_timestamp - INTERVAL '5' SECOND ) WITH ( 'connector' = 'kafka', 'properties.bootstrap.servers' = 'redpanda:29092', 'topic' = 'rides', 'scan.startup.mode' = 'earliest-offset', 'properties.auto.offset.reset' = 'earliest', 'format' = 'json' ); """ t_env.execute_sql(source_ddl) return table_name def log_aggregation(): # Set up the execution environment env = StreamExecutionEnvironment.get_execution_environment() env.enable_checkpointing(10 * 1000) env.set_parallelism(3) # Set up the table environment settings = EnvironmentSettings.new_instance().in_streaming_mode().build() t_env = StreamTableEnvironment.create(env, environment_settings=settings) try: # Create Kafka table source_table = create_events_source_kafka(t_env) aggregated_table = create_events_aggregated_sink(t_env) t_env.execute_sql(f""" INSERT INTO {aggregated_table} SELECT window_start, PULocationID, COUNT(*) AS num_trips, SUM(total_amount) AS total_revenue FROM TABLE( TUMBLE(TABLE {source_table}, DESCRIPTOR(event_timestamp), INTERVAL '1' HOUR) ) GROUP BY window_start, PULocationID; """).wait() except Exception as e: print("Writing records from Kafka to JDBC failed:", str(e)) if __name__ == '__main__': log_aggregation() ================================================ FILE: 07-streaming/workshop/src/job/aggregation_job_demo.py ================================================ """ Demo aggregation job with 10-second tumbling windows. Use with producer_realtime.py to observe watermark behavior: - Watermark = event_timestamp - 5 seconds - Late events (<=5s) arrive before the watermark closes the window -> included - Late events (>5s) may arrive after the watermark closes the window -> dropped """ from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import EnvironmentSettings, StreamTableEnvironment def create_events_source_kafka(t_env): table_name = "events" source_ddl = f""" CREATE TABLE {table_name} ( PULocationID INTEGER, DOLocationID INTEGER, trip_distance DOUBLE, total_amount DOUBLE, tpep_pickup_datetime BIGINT, event_timestamp AS TO_TIMESTAMP_LTZ(tpep_pickup_datetime, 3), WATERMARK for event_timestamp as event_timestamp - INTERVAL '5' SECOND ) WITH ( 'connector' = 'kafka', 'properties.bootstrap.servers' = 'redpanda:29092', 'topic' = 'rides', 'scan.startup.mode' = 'latest-offset', 'properties.auto.offset.reset' = 'latest', 'format' = 'json' ); """ t_env.execute_sql(source_ddl) return table_name def create_events_aggregated_sink(t_env): table_name = 'processed_events_aggregated' sink_ddl = f""" CREATE TABLE {table_name} ( window_start TIMESTAMP(3), PULocationID INT, num_trips BIGINT, total_revenue DOUBLE, PRIMARY KEY (window_start, PULocationID) NOT ENFORCED ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:postgresql://postgres:5432/postgres', 'table-name' = '{table_name}', 'username' = 'postgres', 'password' = 'postgres', 'driver' = 'org.postgresql.Driver' ); """ t_env.execute_sql(sink_ddl) return table_name def log_aggregation(): env = StreamExecutionEnvironment.get_execution_environment() env.enable_checkpointing(10 * 1000) env.set_parallelism(1) settings = EnvironmentSettings.new_instance().in_streaming_mode().build() t_env = StreamTableEnvironment.create(env, environment_settings=settings) try: source_table = create_events_source_kafka(t_env) aggregated_table = create_events_aggregated_sink(t_env) # 10-second tumbling windows (instead of 1 hour) so we can # observe windows closing and late events being dropped t_env.execute_sql(f""" INSERT INTO {aggregated_table} SELECT window_start, PULocationID, COUNT(*) AS num_trips, SUM(total_amount) AS total_revenue FROM TABLE( TUMBLE(TABLE {source_table}, DESCRIPTOR(event_timestamp), INTERVAL '10' SECOND) ) GROUP BY window_start, PULocationID; """).wait() except Exception as e: print("Writing records from Kafka to JDBC failed:", str(e)) if __name__ == '__main__': log_aggregation() ================================================ FILE: 07-streaming/workshop/src/job/pass_through_job.py ================================================ from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import EnvironmentSettings, StreamTableEnvironment def create_processed_events_sink_postgres(t_env): table_name = 'processed_events' sink_ddl = f""" CREATE TABLE {table_name} ( PULocationID INTEGER, DOLocationID INTEGER, trip_distance DOUBLE, total_amount DOUBLE, pickup_datetime TIMESTAMP ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:postgresql://postgres:5432/postgres', 'table-name' = '{table_name}', 'username' = 'postgres', 'password' = 'postgres', 'driver' = 'org.postgresql.Driver' ); """ t_env.execute_sql(sink_ddl) return table_name def create_events_source_kafka(t_env): table_name = "events" source_ddl = f""" CREATE TABLE {table_name} ( PULocationID INTEGER, DOLocationID INTEGER, trip_distance DOUBLE, total_amount DOUBLE, tpep_pickup_datetime BIGINT ) WITH ( 'connector' = 'kafka', 'properties.bootstrap.servers' = 'redpanda:29092', 'topic' = 'rides', 'scan.startup.mode' = 'latest-offset', 'properties.auto.offset.reset' = 'latest', 'format' = 'json' ); """ t_env.execute_sql(source_ddl) return table_name def log_processing(): # Set up the execution environment env = StreamExecutionEnvironment.get_execution_environment() env.enable_checkpointing(10 * 1000) # Set up the table environment settings = EnvironmentSettings.new_instance().in_streaming_mode().build() t_env = StreamTableEnvironment.create(env, environment_settings=settings) try: # Create Kafka table source_table = create_events_source_kafka(t_env) postgres_sink = create_processed_events_sink_postgres(t_env) # write records to postgres t_env.execute_sql( f""" INSERT INTO {postgres_sink} SELECT PULocationID, DOLocationID, trip_distance, total_amount, TO_TIMESTAMP_LTZ(tpep_pickup_datetime, 3) as pickup_datetime FROM {source_table} """ ).wait() except Exception as e: print("Writing records from Kafka to JDBC failed:", str(e)) if __name__ == '__main__': log_processing() ================================================ FILE: 07-streaming/workshop/src/models.py ================================================ import json from dataclasses import dataclass @dataclass class Ride: PULocationID: int DOLocationID: int trip_distance: float total_amount: float tpep_pickup_datetime: int # epoch milliseconds def ride_from_row(row): return Ride( PULocationID=int(row['PULocationID']), DOLocationID=int(row['DOLocationID']), trip_distance=float(row['trip_distance']), total_amount=float(row['total_amount']), tpep_pickup_datetime=int(row['tpep_pickup_datetime'].timestamp() * 1000), ) def ride_deserializer(data): json_str = data.decode('utf-8') ride_dict = json.loads(json_str) return Ride(**ride_dict) ================================================ FILE: 07-streaming/workshop/src/producers/producer.py ================================================ import dataclasses import json import sys import time from pathlib import Path sys.path.insert(0, str(Path(__file__).parent.parent)) import pandas as pd from kafka import KafkaProducer from models import Ride, ride_from_row # Download NYC yellow taxi trip data (first 1000 rows) url = "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-11.parquet" columns = ['PULocationID', 'DOLocationID', 'trip_distance', 'total_amount', 'tpep_pickup_datetime'] df = pd.read_parquet(url, columns=columns).head(1000) def ride_serializer(ride): ride_dict = dataclasses.asdict(ride) json_str = json.dumps(ride_dict) return json_str.encode('utf-8') server = 'localhost:9092' producer = KafkaProducer( bootstrap_servers=[server], value_serializer=ride_serializer ) t0 = time.time() topic_name = 'rides' for _, row in df.iterrows(): ride = ride_from_row(row) producer.send(topic_name, value=ride) print(f"Sent: {ride}") time.sleep(0.01) producer.flush() t1 = time.time() print(f'took {(t1 - t0):.2f} seconds') ================================================ FILE: 07-streaming/workshop/src/producers/producer_realtime.py ================================================ import dataclasses import json import random import sys import time from datetime import datetime, timezone from pathlib import Path sys.path.insert(0, str(Path(__file__).parent.parent)) from kafka import KafkaProducer from models import Ride # Top pickup locations from the actual NYC yellow taxi data. # PULocationID is a taxi zone ID (1-263) defined by the NYC TLC. # See https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv PICKUP_LOCATIONS = [ 79, # East Village, Manhattan 107, # Gramercy, Manhattan 48, # Clinton East (Hell's Kitchen), Manhattan 132, # JFK Airport 234, # Union Sq, Manhattan 148, # Lower East Side, Manhattan 249, # West Village, Manhattan 68, # East Chelsea, Manhattan 90, # Flatiron, Manhattan 263, # Yorkville West, Manhattan 138, # LaGuardia Airport 230, # Times Sq/Theatre District, Manhattan 161, # Midtown Center, Manhattan 162, # Midtown East, Manhattan 170, # Murray Hill, Manhattan 237, # Upper East Side South, Manhattan 239, # Upper West Side South, Manhattan 186, # Penn Station/Madison Sq West, Manhattan 164, # Midtown South, Manhattan 236, # Upper East Side North, Manhattan ] DROPOFF_LOCATIONS = PICKUP_LOCATIONS # same pool for simplicity def make_ride(delay_seconds=0): now_ms = int(time.time() * 1000) - delay_seconds * 1000 return Ride( PULocationID=random.choice(PICKUP_LOCATIONS), DOLocationID=random.choice(DROPOFF_LOCATIONS), trip_distance=round(random.uniform(0.5, 20.0), 2), total_amount=round(random.uniform(5.0, 100.0), 2), tpep_pickup_datetime=now_ms, ) def ride_serializer(ride): return json.dumps(dataclasses.asdict(ride)).encode('utf-8') server = 'localhost:9092' producer = KafkaProducer( bootstrap_servers=[server], value_serializer=ride_serializer, ) topic_name = 'rides' count = 0 print("Sending events (Ctrl+C to stop)...") print() try: while True: # ~20% chance of a late event (3-10 seconds old) if random.random() < 0.2: delay = random.randint(3, 10) ride = make_ride(delay_seconds=delay) ts = datetime.fromtimestamp(ride.tpep_pickup_datetime / 1000, tz=timezone.utc) print(f" LATE ({delay}s) -> PU={ride.PULocationID} ts={ts:%H:%M:%S}") else: ride = make_ride() ts = datetime.fromtimestamp(ride.tpep_pickup_datetime / 1000, tz=timezone.utc) print(f" on time -> PU={ride.PULocationID} ts={ts:%H:%M:%S}") producer.send(topic_name, value=ride) count += 1 time.sleep(0.5) except KeyboardInterrupt: producer.flush() print(f"\nSent {count} events") ================================================ FILE: README.md ================================================
Data Engineering Zoomcamp: A Free 9-Week Course on Data Engineering Fundamentals
Master the fundamentals of data engineering by building an end-to-end data pipeline from scratch. Gain hands-on experience with industry-standard tools and best practices.
Join Slack • #course-data-engineering Channel • Telegram Announcements • Course Playlist • FAQ
## How to Enroll ### 2026 Cohort - **Start Date**: 12 January 2026 - **Register Here**: [Sign up](https://airtable.com/shr6oVXeQvSI5HuWD) ### Self-Paced Learning All course materials are freely available for independent study. Follow these steps: 1. Watch the course videos. 2. Join the [Slack community](https://datatalks.club/slack.html). 3. Refer to the [FAQ document](https://datatalks.club/faq/data-engineering-zoomcamp.html) for guidance. ## Syllabus Overview The course consists of structured modules, hands-on workshops, and a final project to reinforce your learning. ### **Prerequisites** To get the most out of this course, you should have: - Basic coding experience - Familiarity with SQL - Experience with Python (helpful but not required) No prior data engineering experience is necessary. ### **Modules** #### [Module 1: Containerization and Infrastructure as Code](01-docker-terraform/) - Introduction to GCP - Docker and Docker Compose - Running PostgreSQL with Docker - Infrastructure setup with Terraform - Homework #### [Module 2: Workflow Orchestration](02-workflow-orchestration/) - Data Lakes and Workflow Orchestration - Workflow orchestration with Kestra - Homework #### [Workshop 1: Data Ingestion](cohorts/2026/workshops/dlt.md) - API reading and pipeline scalability - Data normalization and incremental loading - Homework #### [Module 3: Data Warehousing](03-data-warehouse/) - Introduction to BigQuery - Partitioning, clustering, and best practices - Machine learning in BigQuery #### [Module 4: Analytics Engineering](04-analytics-engineering/) - Analytics Engineering and Data Modeling - dbt (data build tool) with DuckDB & BigQuery - Testing, documentation, and deployment #### [Module 5: Data Platforms](05-data-platforms/) - Building end-to-end data pipelines with Bruin - Data ingestion, transformation, and quality - Deployment to cloud (BigQuery) #### [Module 6: Batch Processing](06-batch/) - Introduction to Apache Spark - DataFrames and SQL - Internals of GroupBy and Joins #### [Module 7: Streaming](07-streaming/) - Introduction to Kafka - Kafka Streams and KSQL - Schema management with Avro #### [Final Project](projects/) - Apply all concepts learned in a real-world scenario - Peer review and feedback process ## Testimonials > Thank you for what you do! The Data Engineering Zoomcamp gave me skills that helped me land my first tech job. > > — [Tim Claytor](https://www.linkedin.com/in/claytor/) ([Source](https://www.linkedin.com/feed/update/urn:li:activity:7396882073308938240?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7396882073308938240%2C7396889959711793152%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287396889959711793152%2Curn%3Ali%3Aactivity%3A7396882073308938240%29)) > Three months might seem like a long time, but the growth and learning during this period are truly remarkable. It was a great experience with a lot of learning, connecting with like-minded people from all around the world, and having fun. I must admit, this was really hard. But the feeling of accomplishment and learning made it all worthwhile. And I would do it again! > > — [Nevenka Lukic](https://www.linkedin.com/in/nevenka-lukic/) ([Source](https://www.linkedin.com/posts/nevenka-lukic_data-engineering-zoomcamp-final-project-activity-7181985646033461248-Lc1O?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)) > One of the significant things I inferred from the Zoomcamp is to prioritize fundamentals and principles over ever-evolving tools and tech stacks. Hugely grateful to Alexey Grigorev for putting together this incredible course and offering it for free. > > — [Siddhartha Gogoi](https://www.linkedin.com/in/siddhartha-gogoi/) ([Source](https://www.linkedin.com/posts/activity-7325692407675604992-XSKI?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)) > Such a fun deep dive into data engineering, cloud automation, and orchestration. I learned so much along the way. Big shoutout to Alexey Grigorev and the DataTalksClub team for the opportunity and guidance throughout the 3 months of the free course. > > — [Assitan NIARE](https://www.linkedin.com/in/assitan-niar%C3%A9-data/) ([Source](https://www.linkedin.com/posts/activity-7317441554023874561-E3wm?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)) > If you’re serious about breaking into data engineering, start here. The repo’s structure, community, and hands-on focus make it unparalleled. > > — [Wady Osama](https://www.linkedin.com/in/wadyosama/) ([Source](https://www.linkedin.com/posts/wadyosama_dataengineering-zoomcamp-dezoomcamp-activity-7292126824711520258-puJm?utm_source=share&utm_medium=member_desktop&rcm=ACoAADJu9vMBW6iyIYswCQnN6t8UJLkXH2tQPi4)) ## Community & Support ### **Getting Help on Slack** Join the [`#course-data-engineering`](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) channel on [DataTalks.Club Slack](https://datatalks.club/slack.html) for discussions, troubleshooting, and networking. To keep discussions organized: - Follow [our guidelines](asking-questions.md) when posting questions. - Review the [community guidelines](https://datatalks.club/slack/guidelines.html). ## Meet the Instructors - [Alexey Grigorev](https://linkedin.com/in/agrigorev) - [Michael Shoemaker](https://www.linkedin.com/in/michaelshoemaker1/) - [Will Russell](https://www.linkedin.com/in/wrussell1999/) - [Anna Geller](https://www.linkedin.com/in/anna-geller-12a86811a/) - [Juan Manuel Perafan](https://www.linkedin.com/in/jmperafan/) - [Arsalan Noorafkan](https://www.linkedin.com/in/arsalan0/) Past instructors: - [Victoria Perez Mola](https://www.linkedin.com/in/victoriaperezmola/) - [Ankush Khanna](https://linkedin.com/in/ankushkhanna2) - [Sejal Vaidya](https://www.linkedin.com/in/vaidyasejal/) - [Irem Erturk](https://www.linkedin.com/in/iremerturk/) - [Luis Oliveira](https://www.linkedin.com/in/lgsoliveira/) - [Zach Wilson](https://www.linkedin.com/in/eczachly) ## Sponsors & Supporters A special thanks to our course sponsors for making this initiative possible!
DataTalks.Club is a global online community of data enthusiasts. It's a place to discuss data, learn, share knowledge, ask and answer questions, and support each other.
Website • Join Slack Community • Newsletter • Upcoming Events • YouTube • GitHub • LinkedIn • Twitter
All the activity at DataTalks.Club mainly happens on [Slack](https://datatalks.club/slack.html). We post updates there and discuss different aspects of data, career questions, and more. At DataTalksClub, we organize online events, community activities, and free courses. You can learn more about what we do at [DataTalksClub Community Navigation](https://www.notion.so/DataTalksClub-Community-Navigation-bf070ad27ba44bf6bbc9222082f0e5a8?pvs=21). ================================================ FILE: after-sign-up.md ================================================ ## Thank you! Thanks for signing up for the course. The process of adding you to the mailing list is not automated yet, but you will hear from us closer to the course start. To make sure you don't miss any announcements - Register in [DataTalks.Club's Slack](https://datatalks.club/slack.html) and join the [`#course-data-engineering`](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) channel - Join the [course Telegram channel with announcements](https://t.me/dezoomcamp) - Subscribe to [DataTalks.Club's YouTube channel](https://www.youtube.com/c/DataTalksClub) and check [the course playlist](https://www.youtube.com/playlist?list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) See you in January! ================================================ FILE: asking-questions.md ================================================ ## Asking questions If you have any questions, ask them in the [`#course-data-engineering`](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) channel in [DataTalks.Club](https://datatalks.club) slack. To keep our discussion in Slack more organized, we ask you to follow these suggestions: * First, review How to troubleshoot issues listed below. * Before asking a question, check the [FAQ](https://datatalks.club/faq/data-engineering-zoomcamp.html). * Before asking a question review the [Slack Guidelines](#Ask-in-Slack). * If somebody helped you with your problem and it's not in [FAQ](https://datatalks.club/faq/data-engineering-zoomcamp.html), please add it there. It'll help other students. * Zed Shaw (of the Learn the Hard Way series) has [a great post on how to help others help you](https://learncodethehardway.com/blog/03-how-to-ask-for-help/) * Check [Stackoverflow guide on asking](https://stackoverflow.com/help/how-to-ask) ### How to troubleshoot issues The first step is to try to solve the issue on you own; get used to solving problems. This will be a real life skill you need when employed. 1. What does the error say? There will often be a description of the error or instructions on what is needed, I have even seen a link to the solution. Does it reference a specific line of your code? 2. Restart the application or server/pc. 3. Google it. It is going to be rare that you are the first to have the problem, someone out there has posted the issue and likely the solution. Search using: **technology** **problem statement**. Example: `pgcli error column c.relhasoids does not exist`. * There are often different solutions for the same problem due to variation in environments. 4. Check the tech’s documentation. Use its search if available or use the browser's search function. 5. Try uninstall (this may remove the bad actor) and reinstall of application or re-implementation of action. Don’t forget to restart the server/pc for reinstalls. * Sometimes reinstalling fails to resolve the issue but works if you uninstall first. 6. Ask in Slack 7. Take a break and come back to it later. You will be amazed at how often you figure out the solution after letting your brain rest. Get some fresh air, workout, play a video game, watch a tv show, whatever allows your brain to not think about it for a little while or even until the next day. 8. Remember technology issues in real life sometimes take days or even weeks to resolve ### Asking in Slack * Before asking a question, check the [FAQ](https://datatalks.club/faq/data-engineering-zoomcamp.html). * DO NOT use screenshots, especially don’t take pictures from a phone. * DO NOT tag instructors, it may discourage others from helping you. * Copy and paste errors; if it’s long, just post it in a reply to your thread. * Use ``` for formatting your code. * Use the same thread for the conversation (that means replying to your own thread). * DO NOT create multiple posts to discuss the issue. * You may create a new post if the issue reemerges down the road. Be sure to describe what has changed in the environment. * Provide additional information in the same thread of the steps you have taken for resolution. ================================================ FILE: awesome-data-engineering.md ================================================ Have you found any cool resources about data engineering? Put them here ## Learning Data Engineering ### Courses * [Data Engineering Zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp) by DataTalks.Club (free) * [Big Data Platforms, Autumn 2022: Introduction to Big Data Processing Frameworks](https://big-data-platforms-22.mooc.fi/) by the University of Helsinki (free) * [Awesome Data Engineering Learning Path](https://awesomedataengineering.com/) ### Books * [Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321) * [Big Data: Principles and Best Practices of Scalable Realtime Data Systems by Nathan Marz, James Warren](https://www.amazon.com/Big-Data-Principles-practices-scalable/dp/1617290343) * [Practical DataOps: Delivering Agile Data Science at Scale by Harvinder Atwal](https://www.amazon.com/Practical-DataOps-Delivering-Agile-Science/dp/1484251032) * [Data Pipelines Pocket Reference: Moving and Processing Data for Analytics by James Densmore](https://www.amazon.com/Data-Pipelines-Pocket-Reference-Processing/dp/1492087831) * [Best books for data engineering](https://awesomedataengineering.com/data_engineering_best_books) * [Fundamentals of Data Engineering: Plan and Build Robust Data Systems by Joe Reis, Matt Housley](https://www.amazon.com/Fundamentals-Data-Engineering-Robust-Systems/dp/1098108302) ### Introduction to Data Engineering Terms * [https://datatalks.club/podcast/s05e02-data-engineering-acronyms.html](https://datatalks.club/podcast/s05e02-data-engineering-acronyms.html) ### Data engineering in practice Conference talks from companies, blog posts, etc * [Uber Data Archives](https://eng.uber.com/category/articles/uberdata/) (Uber engineering blog) * [Data Engineering Weekly (DE-focused substack)](https://www.dataengineeringweekly.com/) * [Seattle Data Guy (DE-focused substack)](https://seattledataguy.substack.com/) ## Doing Data Engineering ### Coding & Python * [CS50's Introduction to Computer Science | edX](https://www.edx.org/course/introduction-computer-science-harvardx-cs50x) (course) * [Python for Everybody Specialization](https://www.coursera.org/specializations/python) (course) * [Practical Python programming](https://github.com/dabeaz-course/practical-python/blob/master/Notes/Contents.md) ### SQL * [Intro to SQL: Querying and managing data | Khan Academy](https://www.khanacademy.org/computing/computer-programming/sql) * [Mode SQL Tutorial](https://mode.com/sql-tutorial/) * [Use The Index, Luke](https://use-the-index-luke.com/) (SQL Indexing a nd Tuning e-Book)nfreffx * [SQL Performance Explained](https://sql-performance-explained.com/) (book) e ### Workflow orchestration * [What is DAG?](https://youtu.be/1Yh5S-S6wsI) (video) * [Airflow, Prefect, and Dagster: An Inside Look](https://towardsdatascience.com/airflow-prefect-and-dagster-an-inside-look-6074781c9b77) (blog post) * [Open-Source Spotlight - Prefect - Kevin Kho](https://www.youtube.com/watch?v=ISLV9JyqF1w) (video) * [Prefect as a Data Engineering Project Workflow Tool, with Mary Clair Thompson (Duke) - 11/6/2020](https://youtu.be/HuwA4wLQtCM) (video) ### ETL and ELT * [ETL vs. ELT: What’s the Difference?](https://rivery.io/blog/etl-vs-elt/) (blog post) (print version) ### Data lakes * [An Introduction to Modern Data Lake Storage Layers (Hodi, Iceberg, Delta Lake)](https://dacort.dev/posts/modern-data-lake-storage-layers/) (blog post) * [Lake House Architecture @ Halodoc: Data Platform 2.0](https://blogs.halodoc.io/lake-house-architecture-halodoc-data-platform-2-0/amp/) (blzog post) ### Data warehousing * [Guide to Data Warehousing. Short and comprehensive information… | by Tomas Peluritis](https://medium.com/towards-data-science/guide-to-data-warehousing-6fdcf30b6fbe) (blog post) * [Snowflake, Redshift, BigQuery, and Others: Cloud Data Warehouse Tools Compared](https://www.altexsoft.com/blog/snowflake-redshift-bigquery-data-warehouse-tools/) (blog post) ### Streaming * Building Streaming Analytics: The Journey and Learnings - Maxim Lukichev ### DataOps * [DataOps 101 with Lars Albertsson – DataTalks.Club](https://datatalks.club/podcast/s02e11-dataops.html) (podcast) * ### Monitoring and observability * [Data Observability: The Next Frontier of Data Engineering with Barr Moses](https://datatalks.club/podcast/s03e03-data-observability.html) (podcast) ### Analytics engineering * [Analytics Engineer: New Role in a Data Team with Victoria Perez Mola](https://datatalks.club/podcast/s03e11-analytics-engineer.html) (podcast) * [Modern Data Stack for Analytics Engineering - Kyle Shannon](https://www.youtube.com/watch?v=UmIZIkeOfi0) (video) * [Analytics Engineering vs Data Engineering | RudderStack Blog](https://www.rudderstack.com/blog/analytics-engineering-vs-data-engineering) (blog post) * [Learn the Fundamentals of Analytics Engineering with dbt](https://courses.getdbt.com/courses/fundamentals) (course) ### Data mesh * [Data Mesh in Practice - Max Schultze](https://www.youtube.com/watch?v=ekEc8D_D3zY) (video) ### Cloud * [https://acceldataio.medium.com/data-engineering-best-practices-how-netflix-keeps-its-data-infrastructure-cost-effective-dee310bcc910](https://acceldataio.medium.com/data-engineering-best-practices-how-netflix-keeps-its-data-infrastructure-cost-effective-dee310bcc910) ### Reverse ETL * TODO: What is reverse ETL? * [https://datatalks.club/podcast/s05e02-data-engineering-acronyms.html](https://datatalks.club/podcast/s05e02-data-engineering-acronyms.html) * [Open-Source Spotlight - Grouparoo - Brian Leonard](https://www.youtube.com/watch?v=hswlcgQZYuw) (video) * [Open-Source Spotlight - Castled.io (Reverse ETL) - Arun Thulasidharan](https://www.youtube.com/watch?v=iW0XhltAUJ8) (video) ## Career in Data Engineering * [From Data Science to Data Engineering with Ellen König – DataTalks.Club](https://datatalks.club/podcast/s07e08-from-data-science-to-data-engineering.html) (podcast) * [Big Data Engineer vs Data Scientist with Roksolana Diachuk – DataTalks.Club](https://datatalks.club/podcast/s04e03-big-data-engineer-vs-data-scientist.html) (podcast) * [What Skills Do You Need to Become a Data Engineer](https://www.linkedin.com/pulse/what-skills-do-you-need-become-data-engineer-peng-wang/) (blog post) * [The future history of Data Engineering](https://groupby1.substack.com/p/data-engineering?s=r) (blog post) * [What Skills Do Data Engineers Need](https://www.theseattledataguy.com/what-skills-do-data-engineers-need/) (blog post) ### Data Engineering Management * [Becoming a Data Engineering Manager with Rahul Jain – DataTalks.Club](https://datatalks.club/podcast/s07e07-becoming-a-data-engineering-manager.html) (podcast) ## Data engineering projects * [How To Start A Data Engineering Project - With Data Engineering Project Ideas](https://www.youtube.com/watch?v=WpN47Jddo7I) (video) * [Data Engineering Project for Beginners - Batch edition](https://www.startdataengineering.com/post/data-engineering-project-for-beginners-batch-edition/) (blog post) * [Building a Data Engineering Project in 20 Minutes](https://www.sspaeti.com/blog/data-engineering-project-in-twenty-minutes/) (blog post) * [Automating Nike Run Club Data Analysis with Python, Airflow and Google Data Studio | by Rich Martin | Medium](https://medium.com/@rich_23525/automating-nike-run-club-data-analysis-with-python-airflow-and-google-data-studio-3c9556478926) (blog post) ## Data Engineering Resources ### Blogs * [Start Data Engineering](https://www.startdataengineering.com/) ### Podcasts * [The Data Engineering Podcast](https://www.dataengineeringpodcast.com/) * [DataTalks.Club Podcast](https://datatalks.club/podcast.html) (only some episodes are about data engineering) * ### Communities * [DataTalks.Club](https://datatalks.club/) * [/r/dataengineering](https://www.reddit.com/r/dataengineering) ### Meetups * [Sydney Data Engineers](https://sydneydataengineers.github.io/) ### People to follow on Twitter and LinkedIn * TODO ### YouTube channels * [Karolina Sowinska - YouTube](https://www.youtube.com/channel/UCAxnMry1lETl47xQWABvH7g) x` * [Seattle Data Guy - YouTube](https://www.youtube.com/c/SeattleDataGuy) * [Andreas Kretz - YouTube](https://www.youtube.com/c/andreaskayy) * [DataTalksClub - YouTube](https://youtube.com/c/datatalksclub) (only some videos are about data engineering) ### Resource aggregators * [Reading List](https://www.scling.com/reading-list/) by Lars Albertsson * [GitHub - igorbarinov/awesome-data-engineering](https://github.com/igorbarinov/awesome-data-engineering) (focus is more on tools) * [GitHub - DataExpert-io/data-engineer-handbook](https://github.com/DataExpert-io/data-engineer-handbook) (contains tools,blogs and more) ## License This work is licensed under a Creative Commons Attribution 4.0 International License. CC BY 4.0 ================================================ FILE: certificates.md ================================================ ## Getting your certificate Congratulations on finishing the course! You can find your certificate in your enrollment profile (you need to be logged in): * For the 2025 edition, it's https://courses.datatalks.club/de-zoomcamp-2025/enrollment If you can't find a certificate in your profile, it means you didn't pass the project. If you believe it's a mistake, write in the course channel in Slack. ## Adding to LinkedIn You can add your certificate to LinkedIn: * Log in to your LinkedIn account, then go to your profile. * On the right, in the "Add profile" section dropdown, choose "Background" and then select the drop-down triangle next to "Licenses & Certifications". * In "Name", enter "Data Engineering Zoomcamp". * In "Issuing Organization", enter "DataTalksClub". * (Optional) In "Issue Date", enter the time when the certificate was created. * (Optional) Select the checkbox This certification does not expire. * Put your certificate ID. * In "Certification URL", enter the URL for your certificate. [Adapted from here](https://support.edx.org/hc/en-us/articles/206501938-How-can-I-add-my-certificate-to-my-LinkedIn-profile-) ================================================ FILE: cohorts/2022/README.md ================================================ ### 2022 Cohort * **Start**: 17 January 2022 * **Registration link**: https://airtable.com/shr6oVXeQvSI5HuWD * [Leaderboard](https://docs.google.com/spreadsheets/d/e/2PACX-1vR9oQiYnAVvzL4dagnhvp0sngqagF0AceD0FGjhS-dnzMTBzNQIal3-hOgkTibVQvfuqbQ69b0fvRnf/pubhtml) ================================================ FILE: cohorts/2022/project.md ================================================ ## Course Project The goal of this project is to apply everything we learned in this course and build an end-to-end data pipeline. Remember that to pass the project, you must evaluate 3 peers. If you don't do that, your project can't be considered compelete. ### Submitting #### Project Cohort #2 Project: * Form: https://forms.gle/JECXB9jYQ1vBXbsw6 * Deadline: 2 May, 22:00 CET Peer reviewing: * Peer review assignments: [link](https://docs.google.com/spreadsheets/d/e/2PACX-1vShnv8T4iY_5NA8h0nySIS8Wzr-DZGGigEikIW4ZMSi9HlvhaEB4RhwmepVIuIUGaQHS90r5iHR2YXV/pubhtml?gid=964123374&single=true) * Form: https://forms.gle/Pb2fBwYLQ3GGFsaK6 * Deadline: 9 May, 22:00 CET #### Project Cohort #1 Project: * Form: https://forms.gle/6aeVcEVJipqR2BqC8 * Deadline: 4 April, 22:00 CET Peer reviewing: * Peer review assignments: [link](https://docs.google.com/spreadsheets/d/e/2PACX-1vShnv8T4iY_5NA8h0nySIS8Wzr-DZGGigEikIW4ZMSi9HlvhaEB4RhwmepVIuIUGaQHS90r5iHR2YXV/pubhtml) * Form: https://forms.gle/AZ62bXMp4SGcVUmK7 * Deadline: 11 April, 22:00 CET Project feedback: [link](https://docs.google.com/spreadsheets/d/e/2PACX-1vRcVCkO-jes5mbPAcikn9X_s2laJ1KhsO8aibHYQxxKqdCUYMVTEJLJQdM8C5aAUWKFl_0SJW4rme7H/pubhtml) ================================================ FILE: cohorts/2022/week_1_basics_n_setup/homework.md ================================================ ## Week 1 Homework In this homework we'll prepare the environment and practice with terraform and SQL ## Question 1. Google Cloud SDK Install Google Cloud SDK. What's the version you have? To get the version, run `gcloud --version` ## Google Cloud account Create an account in Google Cloud and create a project. ## Question 2. Terraform Now install terraform and go to the terraform directory (`week_1_basics_n_setup/1_terraform_gcp/terraform`) After that, run * `terraform init` * `terraform plan` * `terraform apply` Apply the plan and copy the output (after running `apply`) to the form. It should be the entire output - from the moment you typed `terraform init` to the very end. ## Prepare Postgres Run Postgres and load data as shown in the videos We'll use the yellow taxi trips from January 2021: ```bash wget https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2021-01.csv ``` You will also need the dataset with zones: ```bash wget https://s3.amazonaws.com/nyc-tlc/misc/taxi+_zone_lookup.csv ``` Download this data and put it to Postgres ## Question 3. Count records How many taxi trips were there on January 15? Consider only trips that started on January 15. ## Question 4. Largest tip for each day Find the largest tip for each day. On which day it was the largest tip in January? Use the pick up time for your calculations. (note: it's not a typo, it's "tip", not "trip") ## Question 5. Most popular destination What was the most popular destination for passengers picked up in central park on January 14? Use the pick up time for your calculations. Enter the zone name (not id). If the zone name is unknown (missing), write "Unknown" ## Question 6. Most expensive locations What's the pickup-dropoff pair with the largest average price for a ride (calculated based on `total_amount`)? Enter two zone names separated by a slash For example: "Jamaica Bay / Clinton East" If any of the zone names are unknown (missing), write "Unknown". For example, "Unknown / Clinton East". ## Submitting the solutions * Form for submitting: https://forms.gle/yGQrkgRdVbiFs8Vd7 * You can submit your homework multiple times. In this case, only the last submission will be used. Deadline: 26 January (Wednesday), 22:00 CET ## Solution Here is the solution to questions 3-6: [video](https://www.youtube.com/watch?v=HxHqH2ARfxM&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) ================================================ FILE: cohorts/2022/week_2_data_ingestion/README.md ================================================ ## Week 2: Data Ingestion ### Data Lake (GCS) * What is a Data Lake * ELT vs. ETL * Alternatives to components (S3/HDFS, Redshift, Snowflake etc.) :movie_camera: [Video](https://www.youtube.com/watch?v=W3Zm6rjOq70&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) [Slides](https://docs.google.com/presentation/d/1RkH-YhBz2apIjYZAxUz2Uks4Pt51-fVWVN9CcH9ckyY/edit?usp=sharing) ### Introduction to Workflow orchestration * What is an Orchestration Pipeline? * What is a DAG? * [Video](https://www.youtube.com/watch?v=0yK7LXwYeD0&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) ### Setting up Airflow locally * Setting up Airflow with Docker-Compose * [Video](https://www.youtube.com/watch?v=lqDMzReAtrw&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) * More information in the [airflow folder](airflow) If you want to run a lighter version of Airflow with fewer services, check this [video](https://www.youtube.com/watch?v=A1p5LQ0zzaQ&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb). It's optional. ### Ingesting data to GCP with Airflow * Extraction: Download and unpack the data * Pre-processing: Convert this raw data to parquet * Upload the parquet files to GCS * Create an external table in BigQuery * [Video](https://www.youtube.com/watch?v=9ksX9REfL8w&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&index=19) ### Ingesting data to Local Postgres with Airflow * Converting the ingestion script for loading data to Postgres to Airflow DAG * [Video](https://www.youtube.com/watch?v=s2U8MWJH5xA&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) ### Transfer service (AWS -> GCP) Moving files from AWS to GCP. You will need an AWS account for this. This section is optional * [Video 1](https://www.youtube.com/watch?v=rFOFTfD1uGk&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) * [Video 2](https://www.youtube.com/watch?v=VhmmbqpIzeI&list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb) ### Homework In the homework, you'll create a few DAGs for processing the NY Taxi data for 2019-2021 More information [here](homework.md) ## Community notes Did you take notes? You can share them here. * [Notes from Alvaro Navas](https://github.com/ziritrion/dataeng-zoomcamp/blob/main/notes/2_data_ingestion.md) * [Notes from Aaron Wright](https://github.com/ABZ-Aaron/DataEngineerZoomCamp/blob/master/week_2_data_ingestion/README.md) * [Notes from Abd](https://itnadigital.notion.site/Week-2-Data-Ingestion-ec2d0d36c0664bc4b8be6a554b2765fd) * [Blog post by Isaac Kargar](https://kargarisaac.github.io/blog/data%20engineering/jupyter/2022/01/25/data-engineering-w2.html) * [Blog, notes, walkthroughs by Sandy Behrens](https://learningdataengineering540969211.wordpress.com/2022/01/30/week-2-de-zoomcamp-2-3-2-ingesting-data-to-gcp-with-airflow/) * [Notes from Apurva Hegde](https://github.com/apuhegde/Airflow-LocalExecutor-In-Docker#readme) * [Notes from Vincenzo Galante](https://binchentso.notion.site/Data-Talks-Club-Data-Engineering-Zoomcamp-8699af8e7ff94ec49e6f9bdec8eb69fd) * Add your notes here (above this line) ================================================ FILE: cohorts/2022/week_2_data_ingestion/airflow/.env_example ================================================ # Custom COMPOSE_PROJECT_NAME=dtc-de GOOGLE_APPLICATION_CREDENTIALS=/.google/credentials/google_credentials.json AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT=google-cloud-platform://?extra__google_cloud_platform__key_path=/.google/credentials/google_credentials.json # AIRFLOW_UID= GCP_PROJECT_ID= GCP_GCS_BUCKET= # Postgres POSTGRES_USER=airflow POSTGRES_PASSWORD=airflow POSTGRES_DB=airflow # Airflow AIRFLOW__CORE__EXECUTOR=LocalExecutor AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC=10 AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB} AIRFLOW_CONN_METADATA_DB=postgres+psycopg2://airflow:airflow@postgres:5432/airflow AIRFLOW_VAR__METADATA_DB_SCHEMA=airflow _AIRFLOW_WWW_USER_CREATE=True _AIRFLOW_WWW_USER_USERNAME=${_AIRFLOW_WWW_USER_USERNAME:airflow} _AIRFLOW_WWW_USER_PASSWORD=${_AIRFLOW_WWW_USER_PASSWORD:airflow} AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=True AIRFLOW__CORE__LOAD_EXAMPLES=False ================================================ FILE: cohorts/2022/week_2_data_ingestion/airflow/1_setup_official.md ================================================ ## Setup (Official) ### Pre-Reqs 1. For the sake of standardization across this workshop's config, rename your gcp-service-accounts-credentials file to `google_credentials.json` & store it in your `$HOME` directory ``` bash cd ~ && mkdir -p ~/.google/credentials/ mv| Name | Project | Social | Links and comments |
|---|---|---|---|
| Katharina Eichinger | Project |   |
|
| Alia Hamwi | Project | |
|
| Emmanuel Ikpesu | Project | |
More infoLinks: |
| Sanya Syed | Project | |
More infoLinks: > I am excited about the prospect of securing a challenging role as a Data Engineer, where I can utilise my skills and expertise to contribute meaningfully to an organisation's data-driven initiatives. |
| Aminu Lawal | Project | |
|
| Lisa Reiber | Project | |
More infoLinks: > always happy to connect with other data enthusiasts over topics like low-budget data engineering solutions for non-profits or AI solutions for non-profits |
| Vincenzo Galante | Project | |
More info> Thank you for having this course! |
| Grzegorz Gątkowski | Project | |
|
| Matt Young | Project | |
More infoLinks: > Experienced Developer | Cloud & Data Enthusiast | Open to Cloud & Data Engineering Roles 🌩️ ➜ C#, SQL, JavaScript, Python | BI, Data Analytics | AWS, Azure, GCP Passionate about data pipelines, storage, and processing. Excited to implement advanced cloud solutions and enable data-driven insights. Seeking Data Engineering opportunities to leverage my extensive SQL/Data Analytics experience and to transition into the world of cloud-based data solutions. Let's connect and collaborate on innovative data projects! #DataEngineering #CloudTechnology |
| Sam Hatley | Project | |
|
| Evan Hofmeister | Project | |
|
| Barys Kazarkin | Project | |
|
| Joshua Ati | Project | |
|
| Oleg Agapov | Project | |
|
| Mikhail Kuklin | Project | |
More infoLinks: |
| Emmanuel Letremble | Project | |
More infoLinks: > Thanks to the DataTalks.Club for completing my Full Stack & Machine Learning skill sets with some extra DE knowledge. |
| Victor Kuang | Project | |
|
| Antonis Angelakis | Project | |
|
| Christian Ruiz | |||
| Alex Pilugin | Project | |
|
| Ahmad Rizky | Project | |
|
| Juan Francisco Hernandez Hernandez | Project | |
More info> Thanks to Data Talks Club, it was amazing learning for me as a Career changer. |
| Iurii Chernigin | Project | |
|
| Franklyne Kibet | |||
| Federico Zambelli | Project | |
|
| Marilina Orihuela | Project | |
|
| Alejandro R. Mármol Ruiz | Project | |
|
| Daniel Takeshi | Project | |
|
| Xia He-Bleinagel | Project | |
More infoLinks: |
| Thorsten Foltz | |
||
| Danh Vo | Project | |
|
| Joseph Ologunja | Project | |
|
| Roman Zabolotin | Project | |
|
| Aditya Gupta | Project | |
More infoLinks: |
| Vladimir Bugaevskii | Project | |
|
| Fozan Talat | Project | |
|
| Alain Boisvert | Project | |
|
| reneboy garcia | Project | |
More info> "Success is not always about the grand achievements; it's about the small victories that accumulate over time." - Unknown |
| Svetlana Kononova | |||
| Dmitrii Nikolaev | |
More infoLinks: |
|
| Francis Romio | Project | |
|
| Saul Acevedo | Project | |
|
| Alina Li | Project | |
|
| Alexander Eryuzhev | Project | |
|
| Paul Nwosu | Project | |
More infoLinks:
|
| Param mirani | Project | |
|
| Oscar Garcia - ozkary | Project | |
|
| Hector Torres | Project | |
More infoLinks: > Currently looking for a position as data engineer |
| Dewi Nurfitri Oktaviani | Project | |
More infoLinks: |
| Ryno Marx | |
||
| Hidir Cem Altun | |
||
| Francis Mark Cayco | Project | |
|
| Adrian Baumann | Project | |
|
| Vladislav Garist | Project | |
|
| Gerald Ooi | |
||
| Roman | |
||
| Aleksandr Krasnov | |
More infoLinks: |
|
| Jaesung Ryu | Project | |
|
| António Damião Rodrigues | Project | |
|
| Alicia Escontrela | Project | |
|
| Chalermdej Lematavekul | Project | |
More info> Thank you so much for the course. Learn so many thing from here. |
| Muhammed Jimoh | Project | |
|
| Bartosz Skłodowski | Project | |
|
| Daniel Rigney | Project | |
|
| Daniel Gheorghita | Project | |
|
| Daniel Gheorghita | Project | |
|
| Niel Kemp | |
||
| Shahmir | Project | |
More infoLinks: > I've added a bunch of new features since the reviews! Check it out |
| Matt Bertrand | Project | |
|
| Nikolay Galkov | Project | |
|
| Hiroko Sakai | Project | |
|
| Rohit Joshi | Project | |
|
| Valerii Bazyrov | |
||
| Juan Pablo Ricapito | Project | |
|
| Ashraf Omara | Project | |
More info> I need to thank all of the data club community for this amazing contribution. |
| Wasawat Boonyarittikit | Project | |
|
| Fedor Faizov | Project | |
More info> Absolutly amazing course <3 |
You can load the data however you would like, but keep the files in .GZ Format. If you are using orchestration such as Airflow or Prefect do not load the data into Big Query using the orchestrator. Stop with loading the files into a bucket. NOTE: You can use the CSV option for the GZ files when creating an External Table SETUP: Create an external table using the fhv 2019 data. Create a table in BQ using the fhv 2019 data (do not partition or cluster this table). Data can be found here: https://github.com/DataTalksClub/nyc-tlc-data/releases/tag/fhv
## Question 1: What is the count for fhv vehicle records for year 2019? - 65,623,481 - 43,244,696 - 22,978,333 - 13,942,414 ## Question 2: Write a query to count the distinct number of affiliated_base_number for the entire dataset on both the tables. What is the estimated amount of data that will be read when this query is executed on the External Table and the Table? - 25.2 MB for the External Table and 100.87MB for the BQ Table - 225.82 MB for the External Table and 47.60MB for the BQ Table - 0 MB for the External Table and 0MB for the BQ Table - 0 MB for the External Table and 317.94MB for the BQ Table ## Question 3: How many records have both a blank (null) PUlocationID and DOlocationID in the entire dataset? - 717,748 - 1,215,687 - 5 - 20,332 ## Question 4: What is the best strategy to optimize the table if query always filter by pickup_datetime and order by affiliated_base_number? - Cluster on pickup_datetime Cluster on affiliated_base_number - Partition by pickup_datetime Cluster on affiliated_base_number - Partition by pickup_datetime Partition by affiliated_base_number - Partition by affiliated_base_number Cluster on pickup_datetime ## Question 5: Implement the optimized solution you chose for question 4. Write a query to retrieve the distinct affiliated_base_number between pickup_datetime 2019/03/01 and 2019/03/31 (inclusive). Use the BQ table you created earlier in your from clause and note the estimated bytes. Now change the table in the from clause to the partitioned table you created for question 4 and note the estimated bytes processed. What are these values? Choose the answer which most closely matches. - 12.82 MB for non-partitioned table and 647.87 MB for the partitioned table - 647.87 MB for non-partitioned table and 23.06 MB for the partitioned table - 582.63 MB for non-partitioned table and 0 MB for the partitioned table - 646.25 MB for non-partitioned table and 646.25 MB for the partitioned table ## Question 6: Where is the data stored in the External Table you created? - Big Query - GCP Bucket - Container Registry - Big Table ## Question 7: It is best practice in Big Query to always cluster your data: - True - False ## (Not required) Question 8: A better format to store these files may be parquet. Create a data pipeline to download the gzip files and convert them into parquet. Upload the files to your GCP Bucket and create an External and BQ Table. Note: Column types for all files used in an External Table must have the same datatype. While an External Table may be created and shown in the side panel in Big Query, this will need to be validated by running a count query on the External Table to check if any errors occur. ## Submitting the solutions * Form for submitting: https://forms.gle/rLdvQW2igsAT73HTA * You can submit your homework multiple times. In this case, only the last submission will be used. Deadline: 13 February (Monday), 22:00 CET ## Solution Solution: https://www.youtube.com/watch?v=j8r2OigKBWE ================================================ FILE: cohorts/2023/week_4_analytics_engineering/homework.md ================================================ ## Week 4 Homework In this homework, we'll use the models developed during the week 4 videos and enhance the already presented dbt project using the already loaded Taxi data for fhv vehicles for year 2019 in our DWH. This means that in this homework we use the following data [Datasets list](https://github.com/DataTalksClub/nyc-tlc-data/) * Yellow taxi data - Years 2019 and 2020 * Green taxi data - Years 2019 and 2020 * fhv data - Year 2019. We will use the data loaded for: * Building a source table: `stg_fhv_tripdata` * Building a fact table: `fact_fhv_trips` * Create a dashboard If you don't have access to GCP, you can do this locally using the ingested data from your Postgres database instead. If you have access to GCP, you don't need to do it for local Postgres - only if you want to. > **Note**: if your answer doesn't match exactly, select the closest option ### Question 1: **What is the count of records in the model fact_trips after running all models with the test run variable disabled and filtering for 2019 and 2020 data only (pickup datetime)?** You'll need to have completed the ["Build the first dbt models"](https://www.youtube.com/watch?v=UVI30Vxzd6c) video and have been able to run the models via the CLI. You should find the views and models for querying in your DWH. - 41648442 - 51648442 - 61648442 - 71648442 ### Question 2: **What is the distribution between service type filtering by years 2019 and 2020 data as done in the videos?** You will need to complete "Visualising the data" videos, either using [google data studio](https://www.youtube.com/watch?v=39nLTs74A3E) or [metabase](https://www.youtube.com/watch?v=BnLkrA7a6gM). - 89.9/10.1 - 94/6 - 76.3/23.7 - 99.1/0.9 ### Question 3: **What is the count of records in the model stg_fhv_tripdata after running all models with the test run variable disabled (:false)?** Create a staging model for the fhv data for 2019 and do not add a deduplication step. Run it via the CLI without limits (is_test_run: false). Filter records with pickup time in year 2019. - 33244696 - 43244696 - 53244696 - 63244696 ### Question 4: **What is the count of records in the model fact_fhv_trips after running all dependencies with the test run variable disabled (:false)?** Create a core model for the stg_fhv_tripdata joining with dim_zones. Similar to what we've done in fact_trips, keep only records with known pickup and dropoff locations entries for pickup and dropoff locations. Run it via the CLI without limits (is_test_run: false) and filter records with pickup time in year 2019. - 12998722 - 22998722 - 32998722 - 42998722 ### Question 5: **What is the month with the biggest amount of rides after building a tile for the fact_fhv_trips table?** Create a dashboard with some tiles that you find interesting to explore the data. One tile should show the amount of trips per month, as done in the videos for fact_trips, based on the fact_fhv_trips table. - March - April - January - December ## Submitting the solutions * Form for submitting: https://forms.gle/6A94GPutZJTuT5Y16 * You can submit your homework multiple times. In this case, only the last submission will be used. Deadline: 25 February (Saturday), 22:00 CET ## Solution * Video: https://www.youtube.com/watch?v=I_K0lNu9WQw&list=PL3MmuxUbc_hJjEePXIdE-LVUx_1ZZjYGW * Answers: * Question 1: 61648442, * Question 2: 89.9/10.1 * Question 3: 43244696 * Question 4: 22998722 * Question 5: January ================================================ FILE: cohorts/2023/week_5_batch_processing/homework.md ================================================ ## Week 5 Homework In this homework we'll put what we learned about Spark in practice. For this homework we will be using the FHVHV 2021-06 data found here. [FHVHV Data](https://github.com/DataTalksClub/nyc-tlc-data/releases/download/fhvhv/fhvhv_tripdata_2021-06.csv.gz ) ### Question 1: **Install Spark and PySpark** - Install Spark - Run PySpark - Create a local spark session - Execute spark.version. What's the output? - 3.3.2 - 2.1.4 - 1.2.3 - 5.4 ### Question 2: **HVFHW June 2021** Read it with Spark using the same schema as we did in the lessons. We will use this dataset for all the remaining questions. Repartition it to 12 partitions and save it to parquet. What is the average size of the Parquet (ending with .parquet extension) Files that were created (in MB)? Select the answer which most closely matches. - 2MB - 24MB - 100MB - 250MB ### Question 3: **Count records** How many taxi trips were there on June 15? Consider only trips that started on June 15. - 308,164 - 12,856 - 452,470 - 50,982 ### Question 4: **Longest trip for each day** Now calculate the duration for each trip. How long was the longest trip in Hours? - 66.87 Hours - 243.44 Hours - 7.68 Hours - 3.32 Hours ### Question 5: **User Interface** Spark’s User Interface which shows application's dashboard runs on which local port? - 80 - 443 - 4040 - 8080 ### Question 6: **Most frequent pickup location zone** Load the zone lookup data into a temp view in Spark [Zone Data](https://github.com/DataTalksClub/nyc-tlc-data/releases/download/misc/taxi_zone_lookup.csv) Using the zone lookup data and the fhvhv June 2021 data, what is the name of the most frequent pickup location zone? - East Chelsea - Astoria - Union Sq - Crown Heights North ## Submitting the solutions * Form for submitting: https://forms.gle/EcSvDs6vp64gcGuD8 * You can submit your homework multiple times. In this case, only the last submission will be used. Deadline: 06 March (Monday), 22:00 CET ## Solution * Video: https://www.youtube.com/watch?v=ldoDIT32pJs * Answers: * Question 1: 3.3.2 * Question 2: 24MB * Question 3: 452,470 * Question 4: 66.87 Hours * Question 5: 4040 * Question 6: Crown Heights North ================================================ FILE: cohorts/2023/week_6_stream_processing/client.properties ================================================ # Required connection configs for Kafka producer, consumer, and admin bootstrap.servers=For this homework we will be using the 2022 Green Taxi Trip Record Parquet Files from the New York City Taxi Data found here: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page If you are using orchestration such as Mage, Airflow or Prefect do not load the data into Big Query using the orchestrator. Stop with loading the files into a bucket. NOTE: You will need to use the PARQUET option files when creating an External Table SETUP: Create an external table using the Green Taxi Trip Records Data for 2022. Create a table in BQ using the Green Taxi Trip Records for 2022 (do not partition or cluster this table).
## Question 1: Question 1: What is count of records for the 2022 Green Taxi Data?? - 65,623,481 - 840,402 - 1,936,423 - 253,647 ## Question 2: Write a query to count the distinct number of PULocationIDs for the entire dataset on both the tables. What is the estimated amount of data that will be read when this query is executed on the External Table and the Table? - 0 MB for the External Table and 6.41MB for the Materialized Table - 18.82 MB for the External Table and 47.60 MB for the Materialized Table - 0 MB for the External Table and 0MB for the Materialized Table - 2.14 MB for the External Table and 0MB for the Materialized Table ## Question 3: How many records have a fare_amount of 0? - 12,488 - 128,219 - 112 - 1,622 ## Question 4: What is the best strategy to make an optimized table in Big Query if your query will always order the results by PUlocationID and filter based on lpep_pickup_datetime? (Create a new table with this strategy) - Cluster on lpep_pickup_datetime Partition by PUlocationID - Partition by lpep_pickup_datetime Cluster on PUlocationID - Partition by lpep_pickup_datetime and Partition by PUlocationID - Cluster on by lpep_pickup_datetime and Cluster on PUlocationID ## Question 5: Write a query to retrieve the distinct PULocationID between lpep_pickup_datetime 06/01/2022 and 06/30/2022 (inclusive) Use the materialized table you created earlier in your from clause and note the estimated bytes. Now change the table in the from clause to the partitioned table you created for question 4 and note the estimated bytes processed. What are these values? Choose the answer which most closely matches. - 22.82 MB for non-partitioned table and 647.87 MB for the partitioned table - 12.82 MB for non-partitioned table and 1.12 MB for the partitioned table - 5.63 MB for non-partitioned table and 0 MB for the partitioned table - 10.31 MB for non-partitioned table and 10.31 MB for the partitioned table ## Question 6: Where is the data stored in the External Table you created? - Big Query - GCP Bucket - Big Table - Container Registry ## Question 7: It is best practice in Big Query to always cluster your data: - True - False ## (Bonus: Not worth points) Question 8: No Points: Write a `SELECT count(*)` query FROM the materialized table you created. How many bytes does it estimate will be read? Why? ## Submitting the solutions * Form for submitting: https://courses.datatalks.club/de-zoomcamp-2024/homework/hw3 ================================================ FILE: cohorts/2024/04-analytics-engineering/homework.md ================================================ ## Module 4 Homework In this homework, we'll use the models developed during the week 4 videos and enhance the already presented dbt project using the already loaded Taxi data for fhv vehicles for year 2019 in our DWH. This means that in this homework we use the following data [Datasets list](https://github.com/DataTalksClub/nyc-tlc-data/) * Yellow taxi data - Years 2019 and 2020 * Green taxi data - Years 2019 and 2020 * fhv data - Year 2019. We will use the data loaded for: * Building a source table: `stg_fhv_tripdata` * Building a fact table: `fact_fhv_trips` * Create a dashboard If you don't have access to GCP, you can do this locally using the ingested data from your Postgres database instead. If you have access to GCP, you don't need to do it for local Postgres - only if you want to. > **Note**: if your answer doesn't match exactly, select the closest option ### Question 1: **What happens when we execute dbt build --vars '{'is_test_run':'true'}'** You'll need to have completed the ["Build the first dbt models"](https://www.youtube.com/watch?v=UVI30Vxzd6c) video. - It's the same as running *dbt build* - It applies a _limit 100_ to all of our models - It applies a _limit 100_ only to our staging models - Nothing ### Question 2: **What is the code that our CI job will run? Where is this code coming from?** - The code that has been merged into the main branch - The code that is behind the creation object on the dbt_cloud_pr_ schema - The code from any development branch that has been opened based on main - The code from the development branch we are requesting to merge to main ### Question 3 (2 points) **What is the count of records in the model fact_fhv_trips after running all dependencies with the test run variable disabled (:false)?** Create a staging model for the fhv data, similar to the ones made for yellow and green data. Add an additional filter for keeping only records with pickup time in year 2019. Do not add a deduplication step. Run this models without limits (is_test_run: false). Create a core model similar to fact trips, but selecting from stg_fhv_tripdata and joining with dim_zones. Similar to what we've done in fact_trips, keep only records with known pickup and dropoff locations entries for pickup and dropoff locations. Run the dbt model without limits (is_test_run: false). - 12998722 - 22998722 - 32998722 - 42998722 ### Question 4 (2 points) **What is the service that had the most rides during the month of July 2019 month with the biggest amount of rides after building a tile for the fact_fhv_trips table and the fact_trips tile as seen in the videos?** Create a dashboard with some tiles that you find interesting to explore the data. One tile should show the amount of trips per month, as done in the videos for fact_trips, including the fact_fhv_trips data. - FHV - Green - Yellow - FHV and Green ## Submitting the solutions * Form for submitting: https://courses.datatalks.club/de-zoomcamp-2024/homework/hw4 Deadline: 22 February (Thursday), 22:00 CET ## Solution (To be published after deadline) * Video: https://youtu.be/3OPggh5Rca8 * Answers: * Question 1: It applies a _limit 100_ only to our staging models * Question 2: The code from the development branch we are requesting to merge to main * Question 3: 22998722 * Question 4: Yellow ================================================ FILE: cohorts/2024/05-batch/homework.md ================================================ ## Module 5 Homework Solution: https://www.youtube.com/watch?v=YtddC7vJOgQ In this homework we'll put what we learned about Spark in practice. For this homework we will be using the FHV 2019-10 data found here. [FHV Data](https://github.com/DataTalksClub/nyc-tlc-data/releases/download/fhv/fhv_tripdata_2019-10.csv.gz) ### Question 1: **Install Spark and PySpark** - Install Spark - Run PySpark - Create a local spark session - Execute spark.version. What's the output? > [!NOTE] > To install PySpark follow this [guide](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/05-batch/setup/pyspark.md) ### Question 2: **FHV October 2019** Read the October 2019 FHV into a Spark Dataframe with a schema as we did in the lessons. Repartition the Dataframe to 6 partitions and save it to parquet. What is the average size of the Parquet (ending with .parquet extension) Files that were created (in MB)? Select the answer which most closely matches. - 1MB - 6MB - 25MB - 87MB ### Question 3: **Count records** How many taxi trips were there on the 15th of October? Consider only trips that started on the 15th of October. - 108,164 - 12,856 - 452,470 - 62,610 > [!IMPORTANT] > Be aware of columns order when defining schema ### Question 4: **Longest trip for each day** What is the length of the longest trip in the dataset in hours? - 631,152.50 Hours - 243.44 Hours - 7.68 Hours - 3.32 Hours ### Question 5: **User Interface** Spark’s User Interface which shows the application's dashboard runs on which local port? - 80 - 443 - 4040 - 8080 ### Question 6: **Least frequent pickup location zone** Load the zone lookup data into a temp view in Spark [Zone Data](https://github.com/DataTalksClub/nyc-tlc-data/releases/download/misc/taxi_zone_lookup.csv) Using the zone lookup data and the FHV October 2019 data, what is the name of the LEAST frequent pickup location Zone? - East Chelsea - Jamaica Bay - Union Sq - Crown Heights North ## Submitting the solutions - Form for submitting: https://courses.datatalks.club/de-zoomcamp-2024/homework/hw5 - Deadline: See the website ================================================ FILE: cohorts/2024/06-streaming/docker-compose.yml ================================================ version: '3.7' services: # Redpanda cluster redpanda-1: image: docker.redpanda.com/vectorized/redpanda:v22.3.5 container_name: redpanda-1 command: - redpanda - start - --smp - '1' - --reserve-memory - 0M - --overprovisioned - --node-id - '1' - --kafka-addr - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092 - --advertise-kafka-addr - PLAINTEXT://redpanda-1:29092,OUTSIDE://localhost:9092 - --pandaproxy-addr - PLAINTEXT://0.0.0.0:28082,OUTSIDE://0.0.0.0:8082 - --advertise-pandaproxy-addr - PLAINTEXT://redpanda-1:28082,OUTSIDE://localhost:8082 - --rpc-addr - 0.0.0.0:33145 - --advertise-rpc-addr - redpanda-1:33145 ports: # - 8081:8081 - 8082:8082 - 9092:9092 - 28082:28082 - 29092:29092 ================================================ FILE: cohorts/2024/06-streaming/homework.md ================================================ ## Module 6 Homework In this homework, we're going to extend Module 5 Homework and learn about streaming with PySpark. Instead of Kafka, we will use Red Panda, which is a drop-in replacement for Kafka. Ensure you have the following set up (if you had done the previous homework and the module): - Docker (see [module 1](https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main/01-docker-terraform)) - PySpark (see [module 5](https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main/05-batch/setup)) For this homework we will be using the files from Module 5 homework: - Green 2019-10 data from [here](https://github.com/DataTalksClub/nyc-tlc-data/releases/download/green/green_tripdata_2019-10.csv.gz) ## Start Red Panda Let's start redpanda in a docker container. There's a `docker-compose.yml` file in the homework folder (taken from [here](https://github.com/redpanda-data-blog/2023-python-gsg/blob/main/docker-compose.yml)) Copy this file to your homework directory and run ```bash docker-compose up ``` (Add `-d` if you want to run in detached mode) ## Question 1: Redpanda version Now let's find out the version of redpandas. For that, check the output of the command `rpk help` _inside the container_. The name of the container is `redpanda-1`. Find out what you need to execute based on the `help` output. What's the version, based on the output of the command you executed? (copy the entire version) ## Question 2. Creating a topic Before we can send data to the redpanda server, we need to create a topic. We do it also with the `rpk` command we used previously for figuring out the version of redpandas. Read the output of `help` and based on it, create a topic with name `test-topic` What's the output of the command for creating a topic? Include the entire output in your answer. ## Question 3. Connecting to the Kafka server We need to make sure we can connect to the server, so later we can send some data to its topics First, let's install the kafka connector (up to you if you want to have a separate virtual environment for that) ```bash pip install kafka-python ``` You can start a jupyter notebook in your solution folder or create a script Let's try to connect to our server: ```python import json import time from kafka import KafkaProducer def json_serializer(data): return json.dumps(data).encode('utf-8') server = 'localhost:9092' producer = KafkaProducer( bootstrap_servers=[server], value_serializer=json_serializer ) producer.bootstrap_connected() ``` Provided that you can connect to the server, what's the output of the last command? ## Question 4. Sending data to the stream Now we're ready to send some test data: ```python t0 = time.time() topic_name = 'test-topic' for i in range(10): message = {'number': i} producer.send(topic_name, value=message) print(f"Sent: {message}") time.sleep(0.05) producer.flush() t1 = time.time() print(f'took {(t1 - t0):.2f} seconds') ``` How much time did it take? Where did it spend most of the time? * Sending the messages * Flushing * Both took approximately the same amount of time (Don't remove `time.sleep` when answering this question) ## Reading data with `rpk` You can see the messages that you send to the topic with `rpk`: ```bash rpk topic consume test-topic ``` Run the command above and send the messages one more time to see them ## Sending the taxi data Now let's send our actual data: * Read the green csv.gz file * We will only need these columns: * `'lpep_pickup_datetime',` * `'lpep_dropoff_datetime',` * `'PULocationID',` * `'DOLocationID',` * `'passenger_count',` * `'trip_distance',` * `'tip_amount'` Iterate over the records in the dataframe ```python for row in df_green.itertuples(index=False): row_dict = {col: getattr(row, col) for col in row._fields} print(row_dict) break # TODO implement sending the data here ``` Note: this way of iterating over the records is more efficient compared to `iterrows` ## Question 5: Sending the Trip Data * Create a topic `green-trips` and send the data there * How much time in seconds did it take? (You can round it to a whole number) * Make sure you don't include sleeps in your code ## Creating the PySpark consumer Now let's read the data with PySpark. Spark needs a library (jar) to be able to connect to Kafka, so we need to tell PySpark that it needs to use it: ```python import pyspark from pyspark.sql import SparkSession pyspark_version = pyspark.__version__ kafka_jar_package = f"org.apache.spark:spark-sql-kafka-0-10_2.12:{pyspark_version}" spark = SparkSession \ .builder \ .master("local[*]") \ .appName("GreenTripsConsumer") \ .config("spark.jars.packages", kafka_jar_package) \ .getOrCreate() ``` Now we can connect to the stream: ```python green_stream = spark \ .readStream \ .format("kafka") \ .option("kafka.bootstrap.servers", "localhost:9092") \ .option("subscribe", "green-trips") \ .option("startingOffsets", "earliest") \ .load() ``` In order to test that we can consume from the stream, let's see what will be the first record there. In Spark streaming, the stream is represented as a sequence of small batches, each batch being a small RDD (or a small dataframe). So we can execute a function over each mini-batch. Let's run `take(1)` there to see what do we have in the stream: ```python def peek(mini_batch, batch_id): first_row = mini_batch.take(1) if first_row: print(first_row[0]) query = green_stream.writeStream.foreachBatch(peek).start() ``` You should see a record like this: ``` Row(key=None, value=bytearray(b'{"lpep_pickup_datetime": "2019-10-01 00:26:02", "lpep_dropoff_datetime": "2019-10-01 00:39:58", "PULocationID": 112, "DOLocationID": 196, "passenger_count": 1.0, "trip_distance": 5.88, "tip_amount": 0.0}'), topic='green-trips', partition=0, offset=0, timestamp=datetime.datetime(2024, 3, 12, 22, 42, 9, 411000), timestampType=0) ``` Now let's stop the query, so it doesn't keep consuming messages from the stream ```python query.stop() ``` ## Question 6. Parsing the data The data is JSON, but currently it's in binary format. We need to parse it and turn it into a streaming dataframe with proper columns. Similarly to PySpark, we define the schema ```python from pyspark.sql import types schema = types.StructType() \ .add("lpep_pickup_datetime", types.StringType()) \ .add("lpep_dropoff_datetime", types.StringType()) \ .add("PULocationID", types.IntegerType()) \ .add("DOLocationID", types.IntegerType()) \ .add("passenger_count", types.DoubleType()) \ .add("trip_distance", types.DoubleType()) \ .add("tip_amount", types.DoubleType()) ``` And apply this schema: ```python from pyspark.sql import functions as F green_stream = green_stream \ .select(F.from_json(F.col("value").cast('STRING'), schema).alias("data")) \ .select("data.*") ``` How does the record look after parsing? Copy the output. ### Question 7: Most popular destination Now let's finally do some streaming analytics. We will see what's the most popular destination currently based on our stream of data (which ideally we should have sent with delays like we did in workshop 2) This is how you can do it: * Add a column "timestamp" using the `current_timestamp` function * Group by: * 5 minutes window based on the timestamp column (`F.window(col("timestamp"), "5 minutes")`) * `"DOLocationID"` * Order by count You can print the output to the console using this code ```python query = popular_destinations \ .writeStream \ .outputMode("complete") \ .format("console") \ .option("truncate", "false") \ .start() query.awaitTermination() ``` Write the most popular destination, your answer should be *either* the zone ID or the zone name of this destination. (You will need to re-send the data for this to work) ## Submitting the solutions * Form for submitting: https://courses.datatalks.club/de-zoomcamp-2024/homework/hw6 ## Solution We will publish the solution here after deadline. ================================================ FILE: cohorts/2024/README.md ================================================ ## Data Engineering Zoomcamp 2024 Cohort * [Pre-launch Q&A stream](https://www.youtube.com/watch?v=91b8u9GmqB4) * [Launch stream with course overview](https://www.youtube.com/live/AtRhA-NfS24?si=5JzA_E8BmJjiLi8l) * [Deadline calendar](https://docs.google.com/spreadsheets/d/e/2PACX-1vQACMLuutV5rvXg5qICuJGL-yZqIV0FBD84CxPdC5eZHf8TfzB-CJT_3Mo7U7oGVTXmSihPgQxuuoku/pubhtml) * [FAQ](https://datatalks.club/faq/data-engineering-zoomcamp.html) * Course Playlist: Only 2024 Live videos & homeworks (TODO) * [Public Leaderboard of Top-100 Participants](leaderboard.md) [**Module 1: Introduction & Prerequisites**](01-docker-terraform/) * [Homework](01-docker-terraform/homework.md) [**Module 2: Workflow Orchestration**](02-workflow-orchestration) * [Homework](02-workflow-orchestration/homework.md) * Office hours [**Workshop 1: Data Ingestion**](workshops/dlt.md) * Workshop with dlt * [Homework](workshops/dlt.md) [**Module 3: Data Warehouse**](03-data-warehouse) * [Homework](03-data-warehouse/homework.md) [**Module 4: Analytics Engineering**](04-analytics-engineering/) * [Homework](04-analytics-engineering/homework.md) [**Module 5: Batch processing**](05-batch/) * [Homework](05-batch/homework.md) [**Module 6: Stream Processing**](06-streaming) * [Homework](06-streaming/homework.md) [**Project**](project.md) More information [here](project.md) ================================================ FILE: cohorts/2024/leaderboard.md ================================================ ## Leaderboard This is the top [100 leaderboard](https://courses.datatalks.club/de-zoomcamp-2024/leaderboard) of participants of Data Engineering Zoomcamp 2024 edition!| Name | Projects | Social | Comments |
|---|---|---|---|
| Ashraf Mohammad | |
|
commentReally Recommend this bootcamp , if you want to get hands on data engineering experience. My two Capstone project: www.github.com/Ashraf1395/supply_chain_finance, www.github.com/Ashraf1395/customer_retention_analytics |
| Jorge Vladimir Abrego Arevalo | |
|
|
| Purnendu Shekhar Shukla | |||
| Krishna Anand | |
|
|
| Abhijit Chakraborty | |||
| Hekmatullah Sajid | |
|
|
| Lottie Jane Pollard | |
|
|
| AviAnna | |||
| Ketut Garjita | |
|
commentI would like to express my thanks and appreciation to the Data Talks Club for organizing this excellent Data Engineering Zoomcamp training. This made me valuable experience in deepening new knowledge for me even though previously I had mostly worked as a Database Administrator for various platform databases. Thank you also to the community (datatalks-club.slack.com), especially slack course-data-engineering, as well as other slack communities such as mageai.slack.com. |
| Diogo Costa | |
|
commentGreat course! Check out my YouTube channel: https://www.youtube.com/@TechWithCosta |
| Francisco Ortiz Tena | |
|
commentIt is an awesome course! |
| Nevenka Lukic | |
|
commentThis DE Zoomcamp was fantastic learning and networking experiences. Many thanks to organizers and big recommendations to anyone! |
| Mukhammad Sofyan Rizka Akbar | |
|
commentThanks for providing this course, especially for Alexey and other Datatalk hosts and I hope I can join ML, ML Ops, and LLM Zoomcamp. See you soon :) |
| Mahmoud Mahdy Zaky | |
|
|
| Brilliant Pancake | |||
| Jobert M. Gutierrez | |
|
|
| Olusegun Samson Ayeni | |
|
|
| Lily Chau | |
|
commentBig thank you to Alexey and all other speakers. This is one of the best online learning platforms I have ever come across. |
| Aleksandr Kolmakov | |
|
|
| Kang Zhi Yong | |||
| Eduardo Muñoz Sala | |
|
|
| Kirill Bazarov | |
|
|
| Shayan Shafiee Moghadam | |
|
|
| Landry N. | |
|
commentThanks for the awsome course. |
| Condescending Austin | |||
| Lee Durbin | |||
| Loving Einstein | |||
| Carlos Vecina Tebar | |||
| Abiodun Oki | |
commentthoroughly enjoyed the course, great work Alexey & course team! |
|
| Jimoh | |||
| Sleepy Villani | |||
| Ella Cinders | |||
| Max Lutz | |||
| Jessica De Silva | |||
| Daniel Okello | |
||
| Kirill Sitnikov | |
|
commentThank you Alexey and all DTC team! I’m so glad that I knew about your courses and projects! |
| edumad | |||
| Duy Quoc Vo | |
|
commentNA |
| Xiang Li | |||
| Sugeng Wahyudi | |
|
commentThanks a lot, this was amazing. Can't miss another course and zoomcamp from datatalks.club |
| Anatolii Kryvko | |
|
|
| David Vanegas | |||
| Honey Badger | |||
| Abdelrahman Kamal | |||
| Jean Paul Rodriguez | |
|
|
| Eager Pasteur | |||
| Damian Pszczoła | |
|
|
| ManPrat | |||
| forrest_parnassus | |||
| Ramazan Abylkassov | |
|
commentLook mom, I am on leaderboard! |
| Digamber Deshmukh | |||
| Andrew Lee | |
||
| Matt R | |||
| Raul Antonio Catacora Grundy | |
|
commentI just want to thank everyone, all the instructors, collaborators for creating this amazing set of resources and such a solid community based on sharing and caring. Many many thanks and shout out to you guys |
| Ranga H. | |||
| Salma Gouda | |
|
|
| Artsiom Turevich | |
|
commentA long time ago in a galaxy far, far away... |
| Abhirup Ghosh | |||
| Sonny Pham | |||
| Peter Tran | |||
| Ritika Tilwalia | |
|
|
| Eager Yalow | |||
| Dave Samaniego | |
|
commentThank you DataTalksClub for the course. It was challenging learning many new things, but I had fun along the way too! |
| Lucid Keldysh | |||
| Isaac Ndirangu Muturi | |
|
commentAmazing learning experience |
| Agitated Wing | |||
| Hanaa HAMMAD | |
commentGrateful to this great course |
|
| Jonah Oliver | |||
| Paul Emilio Arizpe Colorado | |
|
commentDataTalksClub brought me the opportunity to learn data engineering. Thanks for all :D |
| Asma-Chloë FARAH | |
|
commentThank you for this amazing zoomcamp ! It was really fun ! |
| Happy Feistel | |||
| Luca Pugliese | |
|
commentit has been a crowdlearning experience! starting in thousands of us. 359 graduated in the end. Proud to have classified 59th. Thanks to all. |
| Jake Maund | |||
| Aditya Phulallwar | |||
| Dave Wilson | |||
| Haitham Hussein Hamad | |
|
|
| Alexandre Bergere aka Rocket | |||
| TOGBAN COKOUVI Joyce Elvis Mahoutondji | |
|
|
| Sad Robinson | |||
| Tetiana Omelchenko | |
|
|
| Amanda Kershaw | |
|
commentThis course was incredibly rewarding and absolutely worth the effort. |
| Kristjan Sert | |
|
|
| Murad Arfanyan | |
|
|
| Ecstatic Hofstadter | |||
| Chung Huu Tin | |
|
|
| Zen Mayer | |||
| Zhastay Yeltay | |
|
comment;) |
| AV3NII | |||
| Sebastian Alejandro Peralta Casafranca | |||
| Relaxed Williams | |||
| George Mouratos | |
|
comment- |
| mhmed ahmed rjb | |||
| Frosty Jackson | |||
| WANJOHI | |
|
|
| Ighorr Holstrom | |
|
|
| Jesse Delzio | |
||
| Khalil El Daou | |
|
commentAlready made a post about the zoomcamp |
| Juan Rojas | |||
| Gonçalo | |||
| Muhamad Farikhin | |||
| Bold Lederberg | |||
| Taras Shalaiko | |
|
\n",
"
\n"
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"\n",
"Sum of ages from generators `people_1()` and `people_2()` combined: 353\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"## Solution 3: Merge a generator\n",
"\n",
"A table's primary key needs to be created from the start, so load your data to a new table with primary key ID.\n",
"\n",
"Load your first generator first, and then load the second one with merge. Since they have overlapping IDs, some of the records from the first load should be replaced by the ones from the second load.\n",
"\n",
"After loading, you should have a total of 8 records, and ID 3 should have age 33."
],
"metadata": {
"id": "G-T-jR9qlzdB"
}
},
{
"cell_type": "code",
"source": [
"import dlt\n",
"\n",
"# Set up a DLT pipeline.\n",
"# Currently using DuckDB for local testing, but it can be switched to BigQuery for production.\n",
"generators_pipeline = dlt.pipeline(destination='duckdb', dataset_name='people_merge')\n",
"\n",
"# Load data from the first generator `people_1` into 'people_merge' table.\n",
"# This operation will replace any existing data in the table.\n",
"# A primary key 'ID' is specified for potential future merge operations.\n",
"info = generators_pipeline.run(people_1(),\n",
" table_name=\"people_v2\",\n",
" write_disposition=\"replace\",\n",
" primary_key=\"ID\")\n",
"\n",
"# Print metadata of the loading process for the first generator.\n",
"print(f\"{info}\\n\\n\")\n",
"\n",
"# Load data from the second generator `people_2` into the same 'people_merge' table.\n",
"# This operation will merge the new data with existing data based on the primary key 'ID'.\n",
"info = generators_pipeline.run(people_2(),\n",
" table_name=\"people_merged\",\n",
" write_disposition=\"merge\",\n",
" primary_key=\"ID\")\n",
"\n",
"# Print metadata of the loading process for the second generator.\n",
"print(f\"{info}\\n\\n\")\n",
"\n",
"import duckdb\n",
"\n",
"# Establish a connection to the DuckDB database created by the pipeline.\n",
"conn = duckdb.connect(f\"{generators_pipeline.pipeline_name}.duckdb\")\n",
"\n",
"# Set the search path to the dataset 'people_merge' and display the available tables.\n",
"conn.sql(f\"SET search_path = '{generators_pipeline.dataset_name}'\")\n",
"print('Loaded tables: ')\n",
"display(conn.sql(\"show tables\"))\n",
"\n",
"# Display the merged data from the 'people_merged' table.\n",
"print(\"\\n\\n\\nData from the 'people_merged' table:\")\n",
"data = conn.sql(\"SELECT * FROM people_merged\").df()\n",
"display(data)\n",
"\n",
"# Calculate and display the sum of ages from the merged data in 'people_merged' table.\n",
"sum_of_ages_p1_p2 = conn.execute(\"SELECT SUM(age) FROM people_merged\").fetchone()[0]\n",
"print(f\"\\n\\nSum of ages of people in generator `people_1()` merged with generator `people_2()` is: {sum_of_ages_p1_p2}\")\n"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 773
},
"id": "rXR-IN85kBtq",
"outputId": "c74a7ab7-aa77-4445-c2bc-e782054a7201"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Pipeline dlt_colab_kernel_launcher load step completed in 0.24 seconds\n",
"1 load package(s) were loaded to destination duckdb and into dataset people_merge\n",
"The duckdb destination used duckdb:////content/dlt_colab_kernel_launcher.duckdb location to store data\n",
"Load package 1706030294.0522 is LOADED and contains no failed jobs\n",
"\n",
"\n",
"Pipeline dlt_colab_kernel_launcher load step completed in 0.42 seconds\n",
"1 load package(s) were loaded to destination duckdb and into dataset people_merge\n",
"The duckdb destination used duckdb:////content/dlt_colab_kernel_launcher.duckdb location to store data\n",
"Load package 1706030294.7037766 is LOADED and contains no failed jobs\n",
"\n",
"\n",
"Loaded tables: \n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"┌─────────────────────┐\n",
"│ name │\n",
"│ varchar │\n",
"├─────────────────────┤\n",
"│ _dlt_loads │\n",
"│ _dlt_pipeline_state │\n",
"│ _dlt_version │\n",
"│ people_merged │\n",
"│ people_v2 │\n",
"└─────────────────────┘"
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"\n",
"\n",
"Data from the 'people_merged' table:\n"
]
},
{
"output_type": "display_data",
"data": {
"text/plain": [
" id name age city occupation _dlt_load_id _dlt_id\n",
"0 8 Person_8 38 City_B Job_8 1706030294.7037766 Q1k+DIAjXLL7cg\n",
"1 4 Person_4 34 City_B Job_4 1706030294.7037766 ewlZ3LjULEchiQ\n",
"2 5 Person_5 35 City_B Job_5 1706030294.7037766 X+LfQEa/X8GU9w\n",
"3 7 Person_7 37 City_B Job_7 1706030294.7037766 lQT0h7IL7E/wxg\n",
"4 3 Person_3 33 City_B Job_3 1706030294.7037766 gRBswCo8B/DJmw\n",
"5 6 Person_6 36 City_B Job_6 1706030294.7037766 M3IbNKfZZCtbcQ"
],
"text/html": [
"\n",
" \n",
"\n",
"\n",
" \n",
"
\n",
"
\n",
" \n",
" | \n", " | id | \n", "name | \n", "age | \n", "city | \n", "_dlt_load_id | \n", "_dlt_id | \n", "occupation | \n", "
|---|---|---|---|---|---|---|---|
| 0 | \n", "1 | \n", "Person_1 | \n", "26 | \n", "City_A | \n", "1706029306.7456656 | \n", "An8WyXL43/J1GQ | \n", "None | \n", "
| 1 | \n", "2 | \n", "Person_2 | \n", "27 | \n", "City_A | \n", "1706029306.7456656 | \n", "ZGI1S72CddPbJQ | \n", "None | \n", "
| 2 | \n", "3 | \n", "Person_3 | \n", "28 | \n", "City_A | \n", "1706029306.7456656 | \n", "+z4Pm5oCykL2Vg | \n", "None | \n", "
| 3 | \n", "4 | \n", "Person_4 | \n", "29 | \n", "City_A | \n", "1706029306.7456656 | \n", "0Vfr36JHZ34OJA | \n", "None | \n", "
| 4 | \n", "5 | \n", "Person_5 | \n", "30 | \n", "City_A | \n", "1706029306.7456656 | \n", "aA+9WOclw3YWpg | \n", "None | \n", "
| 5 | \n", "3 | \n", "Person_3 | \n", "33 | \n", "City_B | \n", "1706029307.9851513 | \n", "mEegoM7n4XujYw | \n", "Job_3 | \n", "
| 6 | \n", "4 | \n", "Person_4 | \n", "34 | \n", "City_B | \n", "1706029307.9851513 | \n", "FPrsrzXgz+E9Fw | \n", "Job_4 | \n", "
| 7 | \n", "5 | \n", "Person_5 | \n", "35 | \n", "City_B | \n", "1706029307.9851513 | \n", "ZaAOBa5EEqXU1Q | \n", "Job_5 | \n", "
| 8 | \n", "6 | \n", "Person_6 | \n", "36 | \n", "City_B | \n", "1706029307.9851513 | \n", "gmcktDnX6y4Fmg | \n", "Job_6 | \n", "
| 9 | \n", "7 | \n", "Person_7 | \n", "37 | \n", "City_B | \n", "1706029307.9851513 | \n", "960gdVKySsa4JA | \n", "Job_7 | \n", "
| 10 | \n", "8 | \n", "Person_8 | \n", "38 | \n", "City_B | \n", "1706029307.9851513 | \n", "+su5IfZQyFEsEw | \n", "Job_8 | \n", "
\n",
"
\n"
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"\n",
"\n",
"Sum of ages of people in generator `people_1()` merged with generator `people_2()` is: 213\n"
]
}
]
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "TApfkuNKtlt3"
},
"execution_count": null,
"outputs": []
}
]
}
================================================
FILE: cohorts/2024/workshops/dlt_resources/homework_starter.ipynb
================================================
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"source": [
"# **Homework**: Data talks club data engineering zoomcamp Data loading workshop\n",
"\n",
"Hello folks, let's practice what we learned - Loading data with the best practices of data engineering.\n",
"\n",
"Here are the exercises we will do\n",
"\n",
"\n"
],
"metadata": {
"id": "mrTFv5nPClXh"
}
},
{
"cell_type": "markdown",
"source": [
"# 1. Use a generator\n",
"\n",
"Remember the concept of generator? Let's practice using them to futher our understanding of how they work.\n",
"\n",
"Let's define a generator and then run it as practice.\n",
"\n",
"**Answer the following questions:**\n",
"\n",
"- **Question 1: What is the sum of the outputs of the generator for limit = 5?**\n",
"- **Question 2: What is the 13th number yielded**\n",
"\n",
"I suggest practicing these questions without GPT as the purpose is to further your learning."
],
"metadata": {
"id": "wLF4iXf-NR7t"
}
},
{
"cell_type": "code",
"source": [
"def square_root_generator(limit):\n",
" n = 1\n",
" while n <= limit:\n",
" yield n ** 0.5\n",
" n += 1\n",
"\n",
"# Example usage:\n",
"limit = 5\n",
"generator = square_root_generator(limit)\n",
"\n",
"for sqrt_value in generator:\n",
" print(sqrt_value)\n"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "wLng-bDJN4jf",
"outputId": "547683cb-5f56-4815-a903-d0d9578eb1f9"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"1.0\n",
"1.4142135623730951\n",
"1.7320508075688772\n",
"2.0\n",
"2.23606797749979\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [],
"metadata": {
"id": "xbe3q55zN43j"
}
},
{
"cell_type": "markdown",
"source": [
"# 2. Append a generator to a table with existing data\n",
"\n",
"\n",
"Below you have 2 generators. You will be tasked to load them to duckdb and answer some questions from the data\n",
"\n",
"1. Load the first generator and calculate the sum of ages of all people. Make sure to only load it once.\n",
"2. Append the second generator to the same table as the first.\n",
"3. **After correctly appending the data, calculate the sum of all ages of people.**\n",
"\n",
"\n"
],
"metadata": {
"id": "vjWhILzGJMpK"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "2MoaQcdLBEk6",
"outputId": "d2b93dc1-d83f-44ea-aeff-fdf51d75f7aa"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"{'ID': 1, 'Name': 'Person_1', 'Age': 26, 'City': 'City_A'}\n",
"{'ID': 2, 'Name': 'Person_2', 'Age': 27, 'City': 'City_A'}\n",
"{'ID': 3, 'Name': 'Person_3', 'Age': 28, 'City': 'City_A'}\n",
"{'ID': 4, 'Name': 'Person_4', 'Age': 29, 'City': 'City_A'}\n",
"{'ID': 5, 'Name': 'Person_5', 'Age': 30, 'City': 'City_A'}\n",
"{'ID': 3, 'Name': 'Person_3', 'Age': 33, 'City': 'City_B', 'Occupation': 'Job_3'}\n",
"{'ID': 4, 'Name': 'Person_4', 'Age': 34, 'City': 'City_B', 'Occupation': 'Job_4'}\n",
"{'ID': 5, 'Name': 'Person_5', 'Age': 35, 'City': 'City_B', 'Occupation': 'Job_5'}\n",
"{'ID': 6, 'Name': 'Person_6', 'Age': 36, 'City': 'City_B', 'Occupation': 'Job_6'}\n",
"{'ID': 7, 'Name': 'Person_7', 'Age': 37, 'City': 'City_B', 'Occupation': 'Job_7'}\n",
"{'ID': 8, 'Name': 'Person_8', 'Age': 38, 'City': 'City_B', 'Occupation': 'Job_8'}\n"
]
}
],
"source": [
"def people_1():\n",
" for i in range(1, 6):\n",
" yield {\"ID\": i, \"Name\": f\"Person_{i}\", \"Age\": 25 + i, \"City\": \"City_A\"}\n",
"\n",
"for person in people_1():\n",
" print(person)\n",
"\n",
"\n",
"def people_2():\n",
" for i in range(3, 9):\n",
" yield {\"ID\": i, \"Name\": f\"Person_{i}\", \"Age\": 30 + i, \"City\": \"City_B\", \"Occupation\": f\"Job_{i}\"}\n",
"\n",
"\n",
"for person in people_2():\n",
" print(person)\n"
]
},
{
"cell_type": "markdown",
"source": [],
"metadata": {
"id": "vtdTIm4fvQCN"
}
},
{
"cell_type": "markdown",
"source": [
"# 3. Merge a generator\n",
"\n",
"Re-use the generators from Exercise 2.\n",
"\n",
"A table's primary key needs to be created from the start, so load your data to a new table with primary key ID.\n",
"\n",
"Load your first generator first, and then load the second one with merge. Since they have overlapping IDs, some of the records from the first load should be replaced by the ones from the second load.\n",
"\n",
"After loading, you should have a total of 8 records, and ID 3 should have age 33.\n",
"\n",
"Question: **Calculate the sum of ages of all the people loaded as described above.**\n"
],
"metadata": {
"id": "pY4cFAWOSwN1"
}
},
{
"cell_type": "markdown",
"source": [
"# Solution: First make sure that the following modules are installed:"
],
"metadata": {
"id": "kKB2GTB9oVjr"
}
},
{
"cell_type": "code",
"source": [
"#Install the dependencies\n",
"%%capture\n",
"!pip install dlt[duckdb]"
],
"metadata": {
"id": "xTVvtyqrfVNq"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# to do: homework :)"
],
"metadata": {
"id": "a2-PRBAkGC2K"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Questions? difficulties? We are here to help.\n",
"- DTC data engineering course channel: https://datatalks-club.slack.com/archives/C01FABYF2RG\n",
"- dlt's DTC cohort channel: https://dlthub-community.slack.com/archives/C06GAEX2VNX"
],
"metadata": {
"id": "PoTJu4kbGG0z"
}
}
]
}
================================================
FILE: cohorts/2024/workshops/dlt_resources/workshop.ipynb
================================================
[File too large to display: 10.7 MB]
================================================
FILE: cohorts/2024/workshops/rising-wave.md
================================================
\n",
"\n",
"\n",
" \n",
"
\n",
"
\n",
" \n",
" | \n", " | id | \n", "name | \n", "age | \n", "city | \n", "occupation | \n", "_dlt_load_id | \n", "_dlt_id | \n", "
|---|---|---|---|---|---|---|---|
| 0 | \n", "8 | \n", "Person_8 | \n", "38 | \n", "City_B | \n", "Job_8 | \n", "1706030294.7037766 | \n", "Q1k+DIAjXLL7cg | \n", "
| 1 | \n", "4 | \n", "Person_4 | \n", "34 | \n", "City_B | \n", "Job_4 | \n", "1706030294.7037766 | \n", "ewlZ3LjULEchiQ | \n", "
| 2 | \n", "5 | \n", "Person_5 | \n", "35 | \n", "City_B | \n", "Job_5 | \n", "1706030294.7037766 | \n", "X+LfQEa/X8GU9w | \n", "
| 3 | \n", "7 | \n", "Person_7 | \n", "37 | \n", "City_B | \n", "Job_7 | \n", "1706030294.7037766 | \n", "lQT0h7IL7E/wxg | \n", "
| 4 | \n", "3 | \n", "Person_3 | \n", "33 | \n", "City_B | \n", "Job_3 | \n", "1706030294.7037766 | \n", "gRBswCo8B/DJmw | \n", "
| 5 | \n", "6 | \n", "Person_6 | \n", "36 | \n", "City_B | \n", "Job_6 | \n", "1706030294.7037766 | \n", "M3IbNKfZZCtbcQ | \n", "
Documentation 📑 Hands-on Tutorials 🎯 RisingWave Cloud 🚀 Get Instant Help
Solution
```sql CREATE MATERIALIZED VIEW latest_dropoff_time AS WITH t AS ( SELECT MAX(tpep_dropoff_datetime) AS latest_dropoff_time FROM trip_data ) SELECT taxi_zone.Zone as taxi_zone, latest_dropoff_time FROM t, trip_data JOIN taxi_zone ON trip_data.DOLocationID = taxi_zone.location_id WHERE trip_data.tpep_dropoff_datetime = t.latest_dropoff_time; -- taxi_zone | latest_dropoff_time -- ----------------+--------------------- -- Midtown Center | 2022-01-03 17:24:54 -- (1 row) ```Docker Compose Example
This Docker Compose has the Zoomcamp DB container and pgAdmin container added to it, so it's all in one file. Changes include: - New `volume` for the Zoomcamp DB container - Zoomcamp DB container is added and renamed to prevent clashes with the Kestra DB container - Depends on condition is added to make sure Kestra is running before it starts - pgAdmin is added and running on Port 8085 so it doesn't clash wit Kestra which uses 8080 and 8081 ```yaml volumes: postgres-data: driver: local kestra-data: driver: local zoomcamp-data: driver: local services: postgres: image: postgres volumes: - postgres-data:/var/lib/postgresql/data environment: POSTGRES_DB: kestra POSTGRES_USER: kestra POSTGRES_PASSWORD: k3str4 healthcheck: test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"] interval: 30s timeout: 10s retries: 10 kestra: image: kestra/kestra:latest pull_policy: always # Note that this setup with a root user is intended for development purpose. # Our base image runs without root, but the Docker Compose implementation needs root to access the Docker socket # To run Kestra in a rootless mode in production, see: https://kestra.io/docs/installation/podman-compose user: "root" command: server standalone volumes: - kestra-data:/app/storage - /var/run/docker.sock:/var/run/docker.sock - /tmp/kestra-wd:/tmp/kestra-wd environment: KESTRA_CONFIGURATION: | datasources: postgres: url: jdbc:postgresql://postgres:5432/kestra driverClassName: org.postgresql.Driver username: kestra password: k3str4 kestra: server: basicAuth: enabled: false username: "admin@kestra.io" # it must be a valid email address password: kestra repository: type: postgres storage: type: local local: basePath: "/app/storage" queue: type: postgres tasks: tmpDir: path: /tmp/kestra-wd/tmp url: http://localhost:8080/ ports: - "8080:8080" - "8081:8081" depends_on: postgres: condition: service_started postgres_zoomcamp: image: postgres environment: POSTGRES_USER: kestra POSTGRES_PASSWORD: k3str4 POSTGRES_DB: postgres-zoomcamp ports: - "5432:5432" volumes: - zoomcamp-data:/var/lib/postgresql/data depends_on: kestra: condition: service_started pgadmin: image: dpage/pgadmin4 environment: - PGADMIN_DEFAULT_EMAIL=admin@admin.com - PGADMIN_DEFAULT_PASSWORD=root ports: - "8085:80" depends_on: postgres_zoomcamp: condition: service_started ``` For this homework we will be using the Yellow Taxi Trip Records for **January 2024 - June 2024 NOT the entire year of data**
Parquet Files from the New York
City Taxi Data found here: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
If you are using orchestration such as Kestra, Mage, Airflow or Prefect etc. do not load the data into Big Query using the orchestrator.
Stop with loading the files into a bucket.
**Load Script:** You can manually download the parquet files and upload them to your GCS Bucket or you can use the linked script [here](./load_yellow_taxi_data.py):
You will simply need to generate a Service Account with GCS Admin Priveleges or be authenticated with the Google SDK and update the bucket name in the script to the name of your bucket
Nothing is fool proof so make sure that all 6 files show in your GCS Bucket before beginning.
NOTE: You will need to use the PARQUET option files when creating an External Table
BIG QUERY SETUP:
Create an external table using the Yellow Taxi Trip Records.
Create a (regular/materialized) table in BQ using the Yellow Taxi Trip Records (do not partition or cluster this table).
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" VendorID lpep_pickup_datetime lpep_dropoff_datetime store_and_fwd_flag \\\n",
"0 2.0 2019-10-01 00:26:02 2019-10-01 00:39:58 N \n",
"1 1.0 2019-10-01 00:18:11 2019-10-01 00:22:38 N \n",
"2 1.0 2019-10-01 00:09:31 2019-10-01 00:24:47 N \n",
"3 1.0 2019-10-01 00:37:40 2019-10-01 00:41:49 N \n",
"4 2.0 2019-10-01 00:08:13 2019-10-01 00:17:56 N \n",
"\n",
" RatecodeID PULocationID DOLocationID passenger_count trip_distance \\\n",
"0 1.0 112 196 1.0 5.88 \n",
"1 1.0 43 263 1.0 0.80 \n",
"2 1.0 255 228 2.0 7.50 \n",
"3 1.0 181 181 1.0 0.90 \n",
"4 1.0 97 188 1.0 2.52 \n",
"\n",
" fare_amount extra mta_tax tip_amount tolls_amount ehail_fee \\\n",
"0 18.0 0.50 0.5 0.00 0.0 NaN \n",
"1 5.0 3.25 0.5 0.00 0.0 NaN \n",
"2 21.5 0.50 0.5 0.00 0.0 NaN \n",
"3 5.5 0.50 0.5 0.00 0.0 NaN \n",
"4 10.0 0.50 0.5 2.26 0.0 NaN \n",
"\n",
" improvement_surcharge total_amount payment_type trip_type \\\n",
"0 0.3 19.30 2.0 1.0 \n",
"1 0.3 9.05 2.0 1.0 \n",
"2 0.3 22.80 2.0 1.0 \n",
"3 0.3 6.80 2.0 1.0 \n",
"4 0.3 13.56 1.0 1.0 \n",
"\n",
" congestion_surcharge \n",
"0 0.0 \n",
"1 0.0 \n",
"2 0.0 \n",
"3 0.0 \n",
"4 0.0 "
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "d085b583-1609-41a9-a222-ff6ca495ee27",
"metadata": {},
"outputs": [],
"source": [
"columns = [\n",
" 'lpep_pickup_datetime',\n",
" 'lpep_dropoff_datetime',\n",
" 'PULocationID',\n",
" 'DOLocationID',\n",
" 'passenger_count',\n",
" 'trip_distance',\n",
" 'tip_amount'\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "66e9f47c-9284-4760-8011-3a8f48aaa49f",
"metadata": {},
"outputs": [],
"source": [
"df = df[columns]"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "7ae3f843-d428-43d2-9e47-7f9fb43acbad",
"metadata": {},
"outputs": [],
"source": [
"from time import time"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "f1ca1ac1-176c-4ccc-aa11-7e1cb5659d39",
"metadata": {},
"outputs": [],
"source": [
"from tqdm.auto import tqdm"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "0b3da4e1-2f1c-400f-bb67-82734c1193f4",
"metadata": {},
"outputs": [],
"source": [
"messages = df.to_dict(orient='records')"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "3bdc95d8-64e1-4819-a885-996813b4bf94",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"476386"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(messages)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "d6f15929-e928-464d-afc1-690343f4f780",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4dffdeb2a0064e1d9bd02dff9f9c49f0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/476386 [00:00 **Important**: update your "Certificate name" here: https://courses.datatalks.club/de-zoomcamp-2025/enrollment -
this is what we will use when generating certificates for you.
### Evaluation criteria
See [here](../../projects/README.md)
================================================
FILE: cohorts/2025/workshops/dlt/README.md
================================================
# Data ingestion with dlt
Homework: [dlt_homework.md](dlt_homework.md)
🎥 **Watch the workshop video**
[](https://www.youtube.com/watch?v=pgJWP_xqO1g "Watch the workshop video")
Welcome to this hands-on workshop, where you'll learn to build efficient and scalable data ingestion pipelines.
### **What will you learn in this workshop?**
In this workshop, you’ll learn the core skills required to build and manage data pipelines:
- **How to build robust, scalable, and self-maintaining pipelines**.
- **Best practices**, like built-in data governance, for ensuring clean and reliable data flows.
- **Incremental loading techniques** to refresh data quickly and cost-effectively.
- **How to build a Data Lake** with dlt.
By the end of this workshop, you'll be able to build data pipelines like a senior data engineer — quickly, concisely, and with best practices baked in.
---
## 📂 Navigation & Resources
- Workshop:
- [Workshop content](data_ingestion_workshop.md).
- [Workshop Colab Notebook](https://colab.research.google.com/drive/1FiAHNFenM8RyptyTPtDTfqPCi5W6KX_V?usp=sharing).
- Homework:
- [Homework Markdown](dlt_homework.md).
- [Homework Colab Notebook](https://colab.research.google.com/drive/1plqdl33K_HkVx0E0nGJrrkEUssStQsW7).
- 🌐 [Official dlt Documentation](https://dlthub.com/docs/intro).
- 💬 Join our [Slack Community](https://dlthub.com/community).
---
## 📖 Course overview
This workshop is structured into three key parts:
1️⃣ **[Extracting Data](data_ingestion_workshop.md#extracting-data)** – Learn scalable data extraction techniques.
2️⃣ **[Normalizing Data](data_ingestion_workshop.md#normalizing-data)** – Clean and structure data before loading.
3️⃣ **[Loading & Incremental Updates](data_ingestion_workshop.md#loading-data)** – Efficiently load and update data.
📌 **Find the full course file here**: [Course File](data_ingestion_workshop.md)
---
## 👩🏫 Teacher
Welcome to the DataTalks.Club Data Engineering Zoomcamp the data ingestion workshop!
I'm Violetta Mishechkina, Solutions Engineer at dltHub. 👋
- I’ve been working in the data field since 2018, with a background in machine learning.
- I started as a Data Scientist, training ML models and neural networks.
- Over time, I realized that in production, hitting the highest RMSE isn’t as important as model size, infrastructure, and data quality - so I transitioned into MLOps.
- A year ago, I joined dltHub’s Customer Success team and discovered dlt, a Python library that automates 90% of tedious data engineering tasks.
- Now, I work closely with customers and partners to help them integrate and optimize dlt in production.
- I also collaborate with our development team as the voice of the customer, ensuring our product meets real-world data engineering needs.
- My experience across ML, MLOps, and data engineering gives me a practical, hands-on perspective on solving data challenges.
---
## Homework
- [Homework Markdown](dlt_homework.md).
- [Homework Colab Notebook](https://colab.research.google.com/drive/1plqdl33K_HkVx0E0nGJrrkEUssStQsW7).
---
## Next steps
As you are learning the various concepts of data engineering,
consider creating a portfolio project that will further your own knowledge.
By demonstrating the ability to deliver end to end, you will have an easier time finding your first role.
This will help regardless of whether your hiring manager reviews your project, largely because you will have a better
understanding and will be able to talk the talk.
Here are some example projects that others did with dlt:
- Serverless dlt-dbt on cloud functions: [Article](https://docs.getdbt.com/blog/serverless-dlt-dbt-stack)
- Bird finder: [Part 1](https://publish.obsidian.md/lough-on-data/blogs/bird-finder-via-dlt-i), [Part 2](https://publish.obsidian.md/lough-on-data/blogs/bird-finder-via-dlt-ii)
- Event ingestion on GCP: [Article and repo](https://dlthub.com/docs/blog/streaming-pubsub-json-gcp)
- Event ingestion on AWS: [Article and repo](https://dlthub.com/docs/blog/dlt-aws-taktile-blog)
- Or see one of the many demos created by our working students: [Hacker news](https://dlthub.com/docs/blog/hacker-news-gpt-4-dashboard-demo),
[GA4 events](https://dlthub.com/docs/blog/ga4-internal-dashboard-demo),
[an E-Commerce](https://dlthub.com/docs/blog/postgresql-bigquery-metabase-demo),
[Google Sheets](https://dlthub.com/docs/blog/google-sheets-to-data-warehouse-pipeline),
[Motherduck](https://dlthub.com/docs/blog/dlt-motherduck-demo),
[MongoDB + Holistics](https://dlthub.com/docs/blog/MongoDB-dlt-Holistics),
[Deepnote](https://dlthub.com/docs/blog/deepnote-women-wellness-violence-tends),
[Prefect](https://dlthub.com/docs/blog/dlt-prefect),
[PowerBI vs GoodData vs Metabase](https://dlthub.com/docs/blog/semantic-modeling-tools-comparison),
[Dagster](https://dlthub.com/docs/blog/dlt-dagster),
[Ingesting events via gcp webhooks](https://dlthub.com/docs/blog/dlt-webhooks-on-cloud-functions-for-event-capture),
[SAP to snowflake replication](https://dlthub.com/docs/blog/sap-hana-to-snowflake-demo-blog),
[Read emails and send sumamry to slack with AI and Kestra](https://dlthub.com/docs/blog/dlt-kestra-demo-blog),
[Mode +dlt capabilities](https://dlthub.com/docs/blog/dlt-mode-blog),
[dbt on cloud functions](https://dlthub.com/docs/blog/dlt-dbt-runner-on-cloud-functions)
- If you want to use dlt in your project, [check this list of public APIs](https://dlthub.com/docs/blog/practice-api-sources)
If you create a personal project, consider submitting it to our blog - we will be happy to showcase it. Just drop us a line in the dlt Slack.
## **💛 If you enjoy dlt, support us!**
* ⭐ **Give us a [GitHub Star](https://github.com/dlt-hub/dlt)!**
* 💬 **Join our [Slack Community](https://dlthub.com/community)!**
* 🚀 **Let’s build great data pipelines together!**
---
# Community notes
Did you take notes? You can share them by creating a PR to this file!
* [Ingest Data to GCS by dlt from peatwan](https://github.com/peatwan/de-zoomcamp/tree/main/workshop/dlt/homework/load_to_gcs)
* Add your notes above this line
================================================
FILE: cohorts/2025/workshops/dlt/data_ingestion_workshop.md
================================================
# Data ingestion with dlt
* Sign up: https://lu.ma/quyfn4q8 (optional)
* Homework: [dlt_homework.md](dlt_homework.md)
## **What is data ingestion?**
Data ingestion is the process of **extracting** data from a source, transporting it to a suitable environment, and preparing it for use. This often includes **normalizing**, **cleaning**, and **adding metadata**.
---
### **“A wild dataset Magically appears!”**
In many data science teams, data seems to appear out of nowhere — because an engineer loads it.
For example, the well-known **NYC Taxi dataset** looks well-structured and ready to use, making it easy to query and analyze. However, not all datasets arrive in such a clean format.
- **Well-structured data** (with an explicit schema) can be used immediately.
- Examples: Parquet, Avro, or database tables where data types and structures are predefined.
- **Unstructured or weakly typed data** (without a defined schema) often needs cleaning and formatting first.
- Examples: CSV, JSON, where fields might be inconsistent, nested or missing key details.
💡 **What is a schema?**
A schema defines the expected format and structure of data, including field names, data types, and relationships.
---
### **Be the Magician! 😎**
Since you're here to learn data engineering, **you** will be the one making datasets magically appear!
To build effective pipelines, you need to master:
✅ **Extracting** data from various sources (APIs, databases, files).
✅ **Normalization** data by transforming, cleaning, and defining schemas.
✅ **Loading** data where it can be used (data warehouse, lake, or database).
---
### **Why are data pipelines so amazing?**
Data pipelines are the backbone of modern data-driven organizations, transforming raw, scattered data into actionable insights.
They ensure data flows seamlessly from its source to its final destination, where it can drive decision-making, analytics, and innovation.
But pipelines don’t just move data — they enable an entire ecosystem of functionality that makes them indispensable.
### **What makes data pipelines so essential?**
1. **Collect**:
Data pipelines gather information from a variety of sources, such as databases, data streams, and applications. This ensures no data is overlooked.
- Example: Retrieving sales data from an online store or capturing user activity logs from an app.
2. **Ingest**:
The collected data flows into an event queue, where it’s organized and prepared for the next steps.
- **Structured data** (like Parquet files or database tables) can be processed immediately.
- **Unstructured data** (like CSV or JSON files) often needs cleaning and normalization.
- Example: Cleaning a JSON response by standardizing its fields or formatting dates in a CSV file.
3. **Store**:
Pipelines send the processed data to **data lakes**, **data warehouses**, or **data lakehouses** for efficient storage and easy access.
- Example: Storing marketing campaign data in a data warehouse to analyze its performance.
4. **Compute**:
Data is processed either in **batches** (large chunks) or as **streams** (real-time updates) to make it ready for analysis.
- Example: Calculating monthly revenue or processing live stock market data.
5. **Consume**:
Finally, the prepared data is delivered to users in forms they can act on:
- **Dashboards** for executives and analysts.
- **Self-service analytics tools** for teams exploring trends.
- **Machine learning models** for predictions and automation.
---
### **Why are data engineers so important in this process?**
Data engineers are the architects behind these pipelines. They don’t just build pipelines—they make sure they’re reliable, efficient, and scalable. Beyond pipeline development, data engineers:
- **Optimize data storage** to keep costs low and performance high.
- **Ensure data quality and integrity**, addressing duplicates, inconsistencies, and missing values.
- **Implement governance** for secure, compliant, and well-managed data.
- **Adapt data architectures** to meet the changing needs of the organization.
Ultimately, their role is to strategically manage the entire **data lifecycle**, from collection to consumption.
---
### **What will you learn in this workshop?**
In this workshop, you’ll learn the core skills required to build and manage data pipelines:
- **How to build robust, scalable, and self-maintaining pipelines**.
- **Best practices**, like built-in data governance, for ensuring clean and reliable data flows.
- **Incremental loading techniques** to refresh data quickly and cost-effectively.
- **How to build a Data Lake** with dlt.
By the end, you’ll not only understand why data pipelines are amazing, but you’ll also know how to create them with best practices to power your organization’s data-driven success.🚀
---
## **Extracting data**
Most of the data you’ll work with is stored behind an **API**, which is like a doorway to the data. Here are the most common types:
- **RESTful APIs**: Provide records of data from business applications.
- Example: Getting a list of customers from a CRM system.
- **File-based APIs**: Return secure file paths to bulk data like JSON or Parquet files stored in buckets.
- Example: Downloading monthly sales reports.
- **Database APIs**: Connect to databases like MongoDB or SQL, often returning data as JSON, the most common interchange format.
As an engineer, you will need to build pipelines that “just work”.
So here’s what you need to consider on extraction, to prevent the pipelines from breaking, and to keep them running smoothly:
1. **Hardware limits**: Be mindful of memory (RAM) and storage (disk space). Overloading these can crash your system.
2. **Network reliability**: Networks can fail! Always account for retries to make your pipelines more robust.
- Tip: Use libraries like `dlt` that have built-in retry mechanisms.
3. **API rate limits**: APIs often restrict the number of requests you can make in a given time.
- Tip: Check the API documentation to understand its limits (e.g., [Zendesk](https://developer.zendesk.com/api-reference/introduction/rate-limits/), [Shopify](https://shopify.dev/docs/api/usage/rate-limits)).
There are even more challenges to consider when working with APIs — such as **pagination and authentication**. Let’s explore how to handle these effectively when working with **REST APIs**.
### **Working with REST APIs**
REST APIs (Representational State Transfer APIs) are one of the most common ways to extract data. They allow you to retrieve structured data using simple HTTP requests. However, working with APIs comes with its own challenges.
#### **Common Challenges**
#### **1. Rate limits**
Many APIs **limit the number of requests** you can make within a certain time frame to prevent overloading their servers. If you exceed this limit, the API may **reject your requests** temporarily or even block you for a period.
To avoid hitting these limits, we can:
- **Monitor API rate limits** – Some APIs provide headers that tell you how many requests you have left.
- **Pause requests when needed** – If we're close to the limit, we wait before making more requests.
- **Implement automatic retries** – If a request fails due to rate limiting, we can wait and retry after some time.
💡Some APIs provide a **retry-after** header, which tells you how long to wait before making another request. Always check the API documentation for best practices!
---
#### **2. Authentication**
Many APIs require an **API key or token** to access data securely. Without authentication, requests may be limited or denied.
🔐 **Types of Authentication in APIs:**
- **API Keys** – A simple token included in the request header or URL.
- **OAuth Tokens** – A more secure authentication method requiring user authorization.
- **Basic Authentication** – Using a username and password (less common today).
💡 Never share your API token publicly! Store it in environment variables or use a secure secrets manager.
----
#### **3. Pagination**
Many APIs return data in **chunks (or pages)** rather than sending everything at once. This prevents **overloading the server** and improves performance, especially for large datasets. To retrieve **all the data**, we need to make multiple requests and keep track of pages until we reach the last one.
📌 Example:
>In this example, we’ll request data from an API that serves the **NYC taxi dataset**.
For these purposes we created an API that can serve the data you are already familiar with. The API returns **1,000 records per page**, and we must request multiple pages to retrieve the full dataset.
```py
import requests
BASE_API_URL = "https://us-central1-dlthub-analytics.cloudfunctions.net/data_engineering_zoomcamp_api"
page_number = 1
while True:
params = {'page': page_number}
response = requests.get(BASE_API_URL, params=params)
page_data = response.json()
if not page_data:
break
print(page_data)
page_number += 1
# limit the number of pages for testing
if page_number > 2:
break
```
What happens here:
- Starts at page 1 and makes a GET request to the API.
- Retrieves JSON data and checks if the page contains records.
- If data exists, prints it and moves to the next page.
- If the page is empty, stops requesting more data.
💡 Different APIs handle pagination differently (some use offsets, cursors, or tokens instead of page numbers). Always check the API documentation for the correct method!
---
#### **4. Avoiding memory issues during extraction**
To prevent your pipeline from crashing, you need to control memory usage.
#### **Challenges with memory**
- Many pipelines run on systems with limited memory, like serverless functions or shared clusters.
- If you try to load all the data into memory at once, it can crash the entire system.
- Even disk space can become an issue if you’re storing large amounts of data.
#### **The solution: streaming data**
**Streaming** means processing data in small chunks or events, rather than loading everything at once. This keeps memory usage low and ensures your pipeline remains efficient.
As a data engineer, you’ll use streaming to transfer data between buffers, such as:
- from APIs to local files;
- from Webhooks to event queues;
- from Event queues (like Kafka) to storage buckets.
---
### **Example of extracting data: Grabbing data from an API**
In this example, we’ll request data from an API that serves the **NYC taxi dataset**. For these purposes we created an API that can serve the data you are already familiar with.
#### **API documentation**:
- **Data**: Comes in pages of 1,000 records.
- **Pagination**: When there’s no more data, the API returns an empty page.
- **Details**:
- **Method**: GET
- **URL**: `https://us-central1-dlthub-analytics.cloudfunctions.net/data_engineering_zoomcamp_api`
- **Parameters**:
- `page`: Integer (page number), defaults to 1.
Here’s how we design our requester:
1. **Request page by page** until we hit an empty page. Since we don’t know how much data is behind the API, we must assume it could be as little as 1,000 records or as much as 10GB.
2. **Use a generator** to handle this efficiently and avoid loading all data into memory.
```py
import requests
BASE_API_URL = "https://us-central1-dlthub-analytics.cloudfunctions.net/data_engineering_zoomcamp_api"
def paginated_getter():
page_number = 1
while True:
params = {'page': page_number}
response = requests.get(BASE_API_URL, params=params)
response.raise_for_status()
page_json = response.json()
print(f'Got page {page_number} with {len(page_json)} records')
if page_json:
yield page_json
page_number += 1
else:
break
for page_data in paginated_getter():
print(page_data)
```
In this approach to grabbing data from APIs, there are both pros and cons:
✅ Pros: **Easy memory management** since the API returns data in small pages or events.
❌ Cons: **Low throughput** because data transfer is limited by API constraints (rate limits, response time).
To simplify data extraction, use specialized tools that follow best practices like streaming — for example, [dlt (data load tool)](https://dlthub.com). It efficiently processes data while **keeping memory usage low** and **leveraging parallelism** for better performance.
### **Extracting data with dlt**
Extracting data from APIs manually requires handling
- **pagination**,
- **rate limits**,
- **authentication**,
- **errors**.
Instead of writing custom scripts, **[dlt](https://dlthub.com/)** simplifies the process with a built-in **[REST API Client](https://dlthub.com/docs/general-usage/http/rest-client)**, making extraction **efficient, scalable, and reliable**.
---
### **Why use dlt for extraction?**
✅ **Built-in REST API support** – Extract data from APIs with minimal code.
✅ **Automatic pagination handling** – No need to loop through pages manually.
✅ **Manages Rate Limits & Retries** – Prevents exceeding API limits and handles failures.
✅ **Streaming support** – Extracts and processes data without loading everything into memory.
✅ **Seamless integration** – Works with **normalization and loading** in a single pipeline.
### **Install dlt**
[Install](https://dlthub.com/docs/reference/installation) dlt with DuckDB as destination:
```shell
pip install dlt[duckdb]
```
### **Example of extracting data with dlt**
Instead of manually writing pagination logic, let’s use **dlt’s [`RESTClient` helper](https://dlthub.com/docs/general-usage/http/rest-client)** to extract NYC taxi ride data:
```py
import dlt
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.paginators import PageNumberPaginator
def paginated_getter():
client = RESTClient(
base_url="https://us-central1-dlthub-analytics.cloudfunctions.net",
# Define pagination strategy - page-based pagination
paginator=PageNumberPaginator( # <--- Pages are numbered (1, 2, 3, ...)
base_page=1, # <--- Start from page 1
total_path=None # <--- No total count of pages provided by API, pagination should stop when a page contains no result items
)
)
for page in client.paginate("data_engineering_zoomcamp_api"): # <--- API endpoint for retrieving taxi ride data
yield page # remember about memory management and yield data
for page_data in paginated_getter():
print(page_data)
```
**How dlt simplifies API extraction:**
🔹 **No manual pagination** – dlt **automatically** fetches **all pages** of data.
🔹 **Low memory usage** – Streams data **chunk by chunk**, avoiding RAM overflows.
🔹 **Handles rate limits & retries** – Ensures requests are sent efficiently **without failures**.
🔹 **Flexible destination support** – Load extracted data into **databases, warehouses, or data lakes**.
---
Well, you’ve successfully **extracted** the data — great! 🎉 But raw data isn’t always ready to use. Now, you need to **process**, **clean**, and **structure** it before it can be loaded into a data lake or data warehouse.
## **Normalizing data**
You often hear that data professionals spend most of their time **“cleaning” data** — but what does that actually mean?
Data cleaning typically involves two key steps:
1. **Normalizing data** – Structuring and standardizing data **without changing its meaning**.
2. **Filtering data for a specific use case** – Selecting or modifying data **in a way that changes its meaning** to fit the analysis.
### **Data cleaning: more than just fixing errors**
A big part of **data cleaning** is actually **metadata work** — ensuring data is structured and standardized so it can be used effectively.
#### **Metadata tasks in data cleaning:**
✅ **Add types** – Convert strings to numbers, timestamps, etc.
✅ **Rename columns** – Ensure names follow a standard format (e.g., no special characters).
✅ **Flatten nested dictionaries** – Bring values from nested dictionaries into the top-level row.
✅ **Unnest lists/arrays** – Convert lists into **child tables** since they can’t be stored directly in a flat format.
👉 **We’ll look at a practical example next, as these concepts are easier to understand with real data.**
---
### **Why prepare data? Why not use JSON directly?**
While JSON is a great format for **data transfer**, it’s not ideal for analysis. Here’s why:
❌ **No enforced schema** – We don’t always know what fields exist in a JSON document.
❌ **Inconsistent data types** – A field like `age` might appear as `25`, `"twenty five"`, or `25.00`, which can break downstream applications.
❌ **Hard to process** – If we need to group data by day, we must manually convert date strings to timestamps.
❌ **Memory-heavy** – JSON requires reading the entire file into memory, unlike databases or columnar formats that allow scanning just the necessary fields.
❌ **Slow for aggregation and search** – JSON is not optimized for quick lookups or aggregations like columnar formats (e.g., Parquet).
JSON is great for **data exchange** but **not for direct analytical use**. To make data useful, we need to **normalize it** — flattening, typing, and structuring it for efficiency.
---
### **Normalization example**
To understand what we’re working with, let’s look at a sample record from our API:
```py
item = page_data[0]
item
```
Output:
```json
{'End_Lat': 40.742963,
'End_Lon': -73.980072,
'Fare_Amt': 45.0,
'Passenger_Count': 1,
'Payment_Type': 'Credit',
'Rate_Code': None,
'Start_Lat': 40.641525,
'Start_Lon': -73.787442,
'Tip_Amt': 9.0,
'Tolls_Amt': 4.15,
'Total_Amt': 58.15,
'Trip_Distance': 17.52,
'Trip_Dropoff_DateTime': '2009-06-14 23:48:00',
'Trip_Pickup_DateTime': '2009-06-14 23:23:00',
'mta_tax': None,
'store_and_forward': None,
'surcharge': 0.0,
'vendor_name': 'VTS'}
```
The data we retrieved from the API has **already been processed and unnested**, meaning that any **nested structures** (like dictionaries and lists) have been flattened, making it easier to store and query in a database or a dataframe. However, let’s imagine we originally received the **raw data** in a more complex format.
---
### **How was this data processed?**
Before reaching this format, the raw data likely contained **nested structures** that had to be **flattened and transformed**.
1️⃣ **Flattened nested coordinates:**
- Originally, the latitude and longitude values might have been nested like this:
```json
"coordinates": {
"start": {"lat": 40.641525, "lon": -73.787442},
"end": {"lat": 40.742963, "lon": -73.980072}
}
```
- These were **flattened** into `Start_Lat`, `Start_Lon`, `End_Lat`, and `End_Lon`.
2️⃣ **Converted timestamps:**
- Originally, timestamps might have been stored as Unix timestamps or separate date/time fields:
```json
"Trip_Pickup": {"date": "2009-06-14", "time": "23:23:00"}
```
- Now, they are **formatted as ISO datetime strings**:
```json
"Trip_Pickup_DateTime": "2009-06-14 23:23:00"
```
3️⃣ **Unnested passenger & payment information:**
- The original structure might have included a nested list for passengers:
```json
"passengers": [
{"name": "John", "rating": 4.9},
{"name": "Jack", "rating": 3.9}
]
```
- Since lists **cannot be stored directly in a database table**, they were likely **moved to a separate table**.
💡 **However, real-world data is rarely this clean!** We often receive raw, nested, and inconsistent data. This is why the **normalization process** is so important—it **prepares** the data for efficient storage and analysis.
**[dlt (data load tool)](https://dlthub.com/docs/intro)** simplifies the **normalization process**, automatically transforming raw data into a **structured, clean format** that is ready for storage and analysis.
---
### **Normalizing data with dlt**
**Why use dlt for normalization?**
✅ **Automatically detects schema** – No need to define column types manually.
✅ **Flattens nested JSON** – Converts complex structures into table-ready formats.
✅ **Handles data type conversion** – Converts dates, numbers, and booleans correctly.
✅ **Splits lists into child tables** – Ensures relational integrity for better analysis.
✅ **Schema evolution support** – Adapts to changes in data structure over time.
---
### **Example**
Let's assume we extracted the following raw NYC taxi ride data, which contains **nested dictionaries** and **lists**:
```py
data = [
{
"vendor_name": "VTS",
"record_hash": "b00361a396177a9cb410ff61f20015ad",
"time": {
"pickup": "2009-06-14 23:23:00",
"dropoff": "2009-06-14 23:48:00"
},
"coordinates": {
"start": {"lon": -73.787442, "lat": 40.641525},
"end": {"lon": -73.980072, "lat": 40.742963}
},
"passengers": [
{"name": "John", "rating": 4.9},
{"name": "Jack", "rating": 3.9}
]
}
]
```
### **How dlt normalizes this data automatically**
Instead of manually flattening fields and extracting nested lists, we can **load it directly into dlt**:
```py
import dlt
# Define a dlt pipeline with automatic normalization
pipeline = dlt.pipeline(
pipeline_name="ny_taxi_data",
destination="duckdb",
dataset_name="taxi_rides",
)
# Run the pipeline with raw nested data
info = pipeline.run(data, table_name="rides", write_disposition="replace")
# Print the load summary
print(info)
print(pipeline.last_trace)
```
---
### **What happens behind the scenes?**
After running this pipeline, dlt automatically **transforms the data** into the following **normalized structure**:
**Main table: `rides`**
```py
pipeline.dataset(dataset_type="default").rides.df()
```
| vendor_name | record_hash | time__pickup | time__dropoff | coordinates__start__lon | coordinates__start__lat | coordinates__end__lon | coordinates__end__lat | _dlt_load_id | _dlt_id |
|-------------|------------------------------------|---------------------------|---------------------------|-------------------------|-------------------------|-----------------------|-----------------------|-------------------|---------------|
| VTS | b00361a396177a9cb410ff61f20015ad | 2009-06-14 23:23:00+00:00 | 2009-06-14 23:48:00+00:00 | -73.787442 | 40.641525 | -73.980072 | 40.742963 | 1738604244.2625916 | k+bnoLuti245ag |
This table **displays structured taxi ride data**, including **vendor details, timestamps, coordinates, and dlt metadata**.
**Child Table: `rides_passengers`**
```py
pipeline.dataset(dataset_type="default").rides__passengers.df()
```
| name | rating | _dlt_parent_id | _dlt_list_idx | _dlt_id |
|-------|--------|------------------|--------------|---------------|
| John | 4.9 | k+bnoLuti245ag | 0 | 8ppDh+8gQ7SSHg |
| Jack | 3.9 | k+bnoLuti245ag | 1 | oQnWuvkgHhxlaA |
✅ **Nested structures were flattened** into separate columns.
✅ **Lists were extracted into child tables**, preserving relationships.
✅ **Timestamps were converted to the correct format.**
---
### **Why dlt makes normalization easy**
🔹 **No manual transformations needed** – Just load the raw data, and dlt does the rest!
🔹 **Database-ready format** – Ensures clean, structured tables for easy querying.
🔹 **Handles schema evolution** – Adapts to new fields automatically.
🔹 **Scales effortlessly** – Works for small datasets and enterprise-scale pipelines.
💡 With dlt, normalization happens automatically, so you can focus on insights instead of data wrangling.
---
## **Loading data**
Now that we’ve covered **extracting** and **normalizing** data, the final step is **loading** the data **into a destination**. This is where the processed data is stored, making it ready for querying, analysis, or further transformations.
### **How data loading happens without dlt**
Before dlt, data engineers had to manually handle **schema validation, batch processing, error handling, and retries** for every destination. This process becomes especially complex when loading data into **data warehouses and data lakes**, where performance optimization, partitioning, and incremental updates are critical.
### **Example: Loading data into database without dlt**
A basic pipeline requires:
1. Setting up a database connection.
2. Creating tables and defining schemas.
3. Handling schema changes manually.
4. Writing queries to insert/update data.
```py
import duckdb
# 1. Create a connection to an in-memory DuckDB database
conn = duckdb.connect("ny_taxi_manual.db")
# 2. Create the rides Table
# Since our dataset has nested structures, we must manually flatten it before inserting data.
conn.execute("""
CREATE TABLE IF NOT EXISTS rides (
record_hash TEXT PRIMARY KEY,
vendor_name TEXT,
pickup_time TIMESTAMP,
dropoff_time TIMESTAMP,
start_lon DOUBLE,
start_lat DOUBLE,
end_lon DOUBLE,
end_lat DOUBLE
);
""")
# 3. Insert Data Manually
# Since JSON data has nested fields, we need to extract and transform them before inserting them into DuckDB.
data = [
{
"vendor_name": "VTS",
"record_hash": "b00361a396177a9cb410ff61f20015ad",
"time": {
"pickup": "2009-06-14 23:23:00",
"dropoff": "2009-06-14 23:48:00"
},
"coordinates": {
"start": {"lon": -73.787442, "lat": 40.641525},
"end": {"lon": -73.980072, "lat": 40.742963}
}
}
]
# Prepare data for insertion
flattened_data = [
(
ride["record_hash"],
ride["vendor_name"],
ride["time"]["pickup"],
ride["time"]["dropoff"],
ride["coordinates"]["start"]["lon"],
ride["coordinates"]["start"]["lat"],
ride["coordinates"]["end"]["lon"],
ride["coordinates"]["end"]["lat"]
)
for ride in data
]
# Insert into DuckDB
conn.executemany("""
INSERT INTO rides (record_hash, vendor_name, pickup_time, dropoff_time, start_lon, start_lat, end_lon, end_lat)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""", flattened_data)
print("Data successfully loaded into DuckDB!")
# 4. Query Data in DuckDB
# Now that the data is loaded, we can query it using DuckDB’s SQL engine.
df = conn.execute("SELECT * FROM rides").df()
conn.close()
```
Problems without dlt:
❌ **Schema management is manual** – If the schema changes, you need to update table structures manually.
❌ **No automatic retries** – If the network fails, data may be lost.
❌ **No incremental loading** – Every run reloads everything, making it slow and expensive.
❌ **More code to maintain** – A simple pipeline quickly becomes complex.
---
### **How dlt handles the load step automatically**
With dlt, loading data **requires just a few lines of code** — schema inference, error handling, and incremental updates are all handled automatically!
### **Why use dlt for loading?**
✅ **Supports multiple destinations** – Load data into **BigQuery, Redshift, Snowflake, Postgres, DuckDB, Parquet (S3, GCS)** and more.
✅ **Optimized for performance** – Uses **batch loading, parallelism, and streaming** for fast and scalable data transfer.
✅ **Schema-aware** – Ensures that **column names, data types, and structures match** the destination’s requirements.
✅ **Incremental loading** – Avoids unnecessary reloading by **only inserting new or updated records**.
✅ **Resilience & retries** – Automatically handles failures, ensuring data is loaded **without missing records**.
### **Example: Loading data into database with dlt**
To use all the power of dlt is better to wrap our API Client in the `@dlt.resource` decorator which denotes a logical grouping of data within a data source, typically holding data of similar structure and origin:
```py
import dlt
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.paginators import PageNumberPaginator
# Define the API resource for NYC taxi data
@dlt.resource(name="rides") # <--- The name of the resource (will be used as the table name)
def ny_taxi():
client = RESTClient(
base_url="https://us-central1-dlthub-analytics.cloudfunctions.net",
paginator=PageNumberPaginator(
base_page=1,
total_path=None
)
)
for page in client.paginate("data_engineering_zoomcamp_api"): # <--- API endpoint for retrieving taxi ride data
yield page # <--- yield data to manage memory
# define new dlt pipeline
pipeline = dlt.pipeline(destination="duckdb")
# run the pipeline with the new resource
load_info = pipeline.run(ny_taxi, write_disposition="replace")
print(load_info)
# explore loaded data
pipeline.dataset(dataset_type="default").rides.df()
```
**Done!** The data is now stored in **DuckDB**, with schema managed automatically!
---
### **Incremental Loading**
Incremental loading allows us to update datasets by **loading only new or changed data**, instead of replacing the entire dataset. This makes pipelines **faster and more cost-effective** by reducing redundant data processing.
### **How does incremental loading work?**
Incremental loading works alongside two key concepts:
- **Incremental extraction** – Only extracts the new or modified data rather than retrieving everything again.
- **State tracking** – Keeps track of what has already been loaded, ensuring that only new data is processed.
In dlt, **state** is stored in a **separate table** at the destination, allowing pipelines to track what has been processed.
🔹 **Want to learn more?** You can read about incremental extraction and state management in the [dlt documentation](https://dlthub.com/docs).
---
### **Incremental loading methods in dlt**
dlt provides two ways to load data incrementally:
#### **1. Append (adding new records)**
- Best for **immutable or stateless data**, such as taxi ride records.
- Each run **adds new records** without modifying previous data.
- Can also be used to create a **history of changes** (slowly changing dimensions).
**Example:**
- If taxi ride data is loaded daily, only **new rides** are added, rather than reloading the full history.
- If tracking changes in a list of vehicles, **each version** is stored as a new row for auditing.
---
#### **2. Merge (updating existing records)**
- Best for **updating existing records** (stateful data).
- Replaces old records with updated ones based on a **unique key**.
- Useful for tracking **status changes**, such as payment updates.
**Example:**
- A taxi ride's **payment status** could change from `"booked"` to `"cancelled"`, requiring an update.
- A **customer profile** might be updated with a new email or phone number.
---
### **Choosing between Append and Merge**
| **Scenario** | **Use Append** | **Use Merge** |
|-----------------------------------|--------------|--------------|
| Immutable records (e.g., ride history) | ✅ Yes | ❌ No |
| Tracking historical changes (slowly changing dimensions) | ✅ Yes | ❌ No |
| Updating existing records (e.g., payment status) | ❌ No | ✅ Yes |
| Keeping full change history | ✅ Yes | ❌ No |
### **Example: Incremental loading with dlt**
**The goal**: download only trips made after June 15, 2009, skipping the old ones.
Using `dlt`, we set up an [incremental filter](https://dlthub.com/docs/general-usage/incremental-loading%23incremental-loading-with-a-cursor-field) to only fetch trips made after a certain date:
```python
cursor_date = dlt.sources.incremental("Trip_Dropoff_DateTime", initial_value="2009-06-15")
```
This tells `dlt`:
- **Start date**: June 15, 2009 (`initial_value`).
- **Field to track**: `Trip_Dropoff_DateTime` (our timestamp).
As you run the pipeline repeatedly, `dlt` will keep track of the latest `Trip_Dropoff_DateTime` value processed. It will skip records older than this date in future runs.
Let's make the data resource incremental using `dlt.sources.incremental`:
```py
import dlt
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.paginators import PageNumberPaginator
@dlt.resource(name="rides", write_disposition="append")
def ny_taxi(
cursor_date=dlt.sources.incremental(
"Trip_Dropoff_DateTime", # <--- field to track, our timestamp
initial_value="2009-06-15", # <--- start date June 15, 2009
)
):
client = RESTClient(
base_url="https://us-central1-dlthub-analytics.cloudfunctions.net",
paginator=PageNumberPaginator(
base_page=1,
total_path=None
)
)
for page in client.paginate("data_engineering_zoomcamp_api"):
yield page
```
Finally, we run our pipeline and load the fresh taxi rides data:
```py
# define new dlt pipeline
pipeline = dlt.pipeline(pipeline_name="ny_taxi", destination="duckdb", dataset_name="ny_taxi_data")
# run the pipeline with the new resource
load_info = pipeline.run(ny_taxi)
print(pipeline.last_trace)
```
Only 5325 rows were flitered out and loaded into the `duckdb` destination. Let's take a look at the earliest date in the loaded data:
```py
with pipeline.sql_client() as client:
res = client.execute_sql(
"""
SELECT
MIN(trip_dropoff_date_time)
FROM rides;
"""
)
print(res)
```
Run the same pipeline again.
```py
# define new dlt pipeline
pipeline = dlt.pipeline(pipeline_name="ny_taxi", destination="duckdb", dataset_name="ny_taxi_data")
# run the pipeline with the new resource
load_info = pipeline.run(ny_taxi)
print(pipeline.last_trace)
```
The pipeline will detect that there are **no new records** based on the `Trip_Dropoff_DateTime` field and the incremental cursor. As a result, **no new data will be loaded** into the destination:
>0 load package(s) were loaded
💡 **With dlt, incremental loading is simple, scalable, and automatic!**
---
### **Example: Loading data into a Data Warehouse (BigQuery)**
First, install the dependencies, define the source, then change the destination name and run the pipeline.
```shell
pip install dlt[bigquery]
```
Let's use our NY Taxi API and load data from the source into destination.
```py
import dlt
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.paginators import PageNumberPaginator
@dlt.resource(name="rides", write_disposition="replace")
def ny_taxi():
client = RESTClient(
base_url="https://us-central1-dlthub-analytics.cloudfunctions.net",
paginator=PageNumberPaginator(
base_page=1,
total_path=None
)
)
for page in client.paginate("data_engineering_zoomcamp_api"):
yield page
```
**Choosing a destination**
Switching between **data warehouses (BigQuery, Snowflake, Redshift)** or **data lakes (S3, Google Cloud Storage, Parquet files)** in dlt is incredibly straightforward — simply modify the `destination` parameter in your pipeline configuration.
For example:
```py
pipeline = dlt.pipeline(
pipeline_name='taxi_data',
destination='duckdb', # <--- to test pipeline locally
dataset_name='taxi_rides',
)
pipeline = dlt.pipeline(
pipeline_name='taxi_data',
destination='bigquery', # <--- to run pipeline in production
dataset_name='taxi_rides',
)
```
This flexibility allows you to easily transition from local development to production-grade environments.
> 💡 No need to rewrite your pipeline — dlt adapts automatically!
**Set Credentials**
The next logical step is to [set credentials](https://dlthub.com/docs/general-usage/credentials/) using **dlt's TOML providers** or **environment variables (ENVs)**.
```py
import os
from google.colab import userdata
os.environ["DESTINATION__BIGQUERY__CREDENTIALS"] = userdata.get('BIGQUERY_CREDENTIALS')
```
Run the pipeline:
```py
pipeline = dlt.pipeline(
pipeline_name="taxi_data",
destination="bigquery",
dataset_name="taxi_rides",
dev_mode=True,
)
info = pipeline.run(ny_taxi)
print(info)
```
💡 **What’s different?**
- **dlt automatically adapts the schema** to fit BigQuery.
- **Partitioning & clustering** can be applied for performance optimization.
- **Efficient batch loading** ensures scalability.
---
### **Example: Loading data into a Data Lake (Parquet on Local FS or S3)**
**Why use a Data Lake?**
- **Cost-effective storage** – Cheaper than traditional databases.
- **Optimized for big data processing** – Works seamlessly with Spark, Databricks, and Presto.
- **Easy scalability** – Store petabytes of data efficiently.
The `filesystem` destination enables you to load data into **files stored locally** or in **cloud storage** solutions, making it an excellent choice for lightweight testing, prototyping, or file-based workflows.
Below is an **example** demonstrating how to use the `filesystem` destination to load data in **Parquet** format:
* Step 1: Set up a local bucket or cloud directory for storing files
```py
import os
os.environ["BUCKET_URL"] = "/content"
```
* Step 2: Define the data source (above)
* Step 3: Run the pipeline
```py
import dlt
pipeline = dlt.pipeline(
pipeline_name='fs_pipeline',
destination='filesystem', # <--- change destination to 'filesystem'
dataset_name='fs_data',
)
load_info = pipeline.run(ny_taxi, loader_file_format="parquet") # <--- choose a file format: parquet, csv or jsonl
print(load_info)
```
Look at the files:
```shell
! ls fs_data/rides
```
Look at the loaded data:
```py
# explore loaded data
pipeline.dataset(dataset_type="default").rides.df()
```
#### **Table formats: [Delta tables & Iceberg](https://dlthub.com/docs/dlt-ecosystem/destinations/delta-iceberg)**
dlt supports writing **Delta** and **Iceberg** tables when using the `filesystem` destination.
**How it works:**
dlt uses the `deltalake` and `pyiceberg` libraries to write Delta and Iceberg tables, respectively. One or multiple Parquet files are prepared during the extract and normalize steps. In the load step, these Parquet files are exposed as an Arrow data structure and fed into `deltalake` or `pyiceberg`.
```shell
!pip install "dlt[pyiceberg]"
```
```py
pipeline = dlt.pipeline(
pipeline_name='fs_pipeline',
destination='filesystem', # <--- change destination to 'filesystem'
dataset_name='fs_iceberg_data',
)
load_info = pipeline.run(
ny_taxi,
loader_file_format="parquet",
table_format="iceberg", # <--- choose a table format: delta or iceberg
)
print(load_info)
```
💡**Note:**
Open source version of dlt supports basic functionality for **iceberg**, but the dltHub team is currently working on an **extended** and **more powerful** integration with iceberg.
[Join the waiting list to learn more about dlt+ and Iceberg.](https://info.dlthub.com/waiting-list)
---
## **What’s Next?**
- **Try loading data into different [destinations](https://dlthub.com/docs/dlt-ecosystem/destinations/)** – Test Postgres, Snowflake, or Parquet.
- **Experiment with [incremental loading](https://dlthub.com/docs/general-usage/incremental-loading)** – Load only new records for better efficiency.
- **Explore dlt’s [schema evolution](https://dlthub.com/docs/general-usage/schema-evolution)** – Automatically adjust to data structure changes.
- **Join our [Slack community](https://dlthub.com/community)** to share your progress!
With **dlt’s automated load step**, you get **effortless, scalable, and resilient data loading**—so you can focus on insights instead of pipeline maintenance. 🚀
---
### Extra homework 💻
* [Data ingestion with DLT to Bigquery from Sara Sabater](https://github.com/saraisab/Data_Engineer/blob/main/courses/DE_zoomcamp/Homework/DLT-Workshop/extra_homework/Data_ingestion_with_DLT_to_bigquery.ipynb).
================================================
FILE: cohorts/2025/workshops/dlt/dlt_homework.md
================================================
Original file is located at
https://colab.research.google.com/drive/1plqdl33K_HkVx0E0nGJrrkEUssStQsW7
# **Workshop "Data Ingestion with dlt": Homework**
---
## **Dataset & API**
We’ll use **NYC Taxi data** via the same custom API from the workshop:
🔹 **Base API URL:**
```
https://us-central1-dlthub-analytics.cloudfunctions.net/data_engineering_zoomcamp_api
```
🔹 **Data format:** Paginated JSON (1,000 records per page).
🔹 **API Pagination:** Stop when an empty page is returned.
## **Question 1: dlt Version**
1. **Install dlt**:
```
!pip install dlt[duckdb]
```
> Or choose a different bracket—`bigquery`, `redshift`, etc.—if you prefer another primary destination. For this assignment, we’ll still do a quick test with DuckDB.
2. **Check** the version:
```
!dlt --version
```
or:
```py
import dlt
print("dlt version:", dlt.__version__)
```
Provide the **version** you see in the output.
## **Question 2: Define & Run the Pipeline (NYC Taxi API)**
Use dlt to extract all pages of data from the API.
Steps:
1️⃣ Use the `@dlt.resource` decorator to define the API source.
2️⃣ Implement automatic pagination using dlt's built-in REST client.
3️⃣ Load the extracted data into DuckDB for querying.
```py
import dlt
from dlt.sources.helpers.rest_client import RESTClient
from dlt.sources.helpers.rest_client.paginators import PageNumberPaginator
# your code is here
pipeline = dlt.pipeline(
pipeline_name="ny_taxi_pipeline",
destination="duckdb",
dataset_name="ny_taxi_data"
)
```
Load the data into DuckDB to test:
```py
load_info = pipeline.run(ny_taxi)
print(load_info)
```
Start a connection to your database using native `duckdb` connection and look what tables were generated:"""
```py
import duckdb
from google.colab import data_table
data_table.enable_dataframe_formatter()
# A database '| \n", " | VendorID | \n", "lpep_pickup_datetime | \n", "lpep_dropoff_datetime | \n", "store_and_fwd_flag | \n", "RatecodeID | \n", "PULocationID | \n", "DOLocationID | \n", "passenger_count | \n", "trip_distance | \n", "fare_amount | \n", "extra | \n", "mta_tax | \n", "tip_amount | \n", "tolls_amount | \n", "ehail_fee | \n", "improvement_surcharge | \n", "total_amount | \n", "payment_type | \n", "trip_type | \n", "congestion_surcharge | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "2.0 | \n", "2019-10-01 00:26:02 | \n", "2019-10-01 00:39:58 | \n", "N | \n", "1.0 | \n", "112 | \n", "196 | \n", "1.0 | \n", "5.88 | \n", "18.0 | \n", "0.50 | \n", "0.5 | \n", "0.00 | \n", "0.0 | \n", "NaN | \n", "0.3 | \n", "19.30 | \n", "2.0 | \n", "1.0 | \n", "0.0 | \n", "
| 1 | \n", "1.0 | \n", "2019-10-01 00:18:11 | \n", "2019-10-01 00:22:38 | \n", "N | \n", "1.0 | \n", "43 | \n", "263 | \n", "1.0 | \n", "0.80 | \n", "5.0 | \n", "3.25 | \n", "0.5 | \n", "0.00 | \n", "0.0 | \n", "NaN | \n", "0.3 | \n", "9.05 | \n", "2.0 | \n", "1.0 | \n", "0.0 | \n", "
| 2 | \n", "1.0 | \n", "2019-10-01 00:09:31 | \n", "2019-10-01 00:24:47 | \n", "N | \n", "1.0 | \n", "255 | \n", "228 | \n", "2.0 | \n", "7.50 | \n", "21.5 | \n", "0.50 | \n", "0.5 | \n", "0.00 | \n", "0.0 | \n", "NaN | \n", "0.3 | \n", "22.80 | \n", "2.0 | \n", "1.0 | \n", "0.0 | \n", "
| 3 | \n", "1.0 | \n", "2019-10-01 00:37:40 | \n", "2019-10-01 00:41:49 | \n", "N | \n", "1.0 | \n", "181 | \n", "181 | \n", "1.0 | \n", "0.90 | \n", "5.5 | \n", "0.50 | \n", "0.5 | \n", "0.00 | \n", "0.0 | \n", "NaN | \n", "0.3 | \n", "6.80 | \n", "2.0 | \n", "1.0 | \n", "0.0 | \n", "
| 4 | \n", "2.0 | \n", "2019-10-01 00:08:13 | \n", "2019-10-01 00:17:56 | \n", "N | \n", "1.0 | \n", "97 | \n", "188 | \n", "1.0 | \n", "2.52 | \n", "10.0 | \n", "0.50 | \n", "0.5 | \n", "2.26 | \n", "0.0 | \n", "NaN | \n", "0.3 | \n", "13.56 | \n", "1.0 | \n", "1.0 | \n", "0.0 | \n", "
In this notebook we will use:
\n", "\n", "- \n",
"
- dlt to extract, normalize, and load data \n", "
- DuckDB as the destination database (runs locally inside Colab) \n", "
\n", " DuckDB is great for beginners because it requires no setup and no credentials.\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "aQTSvnvnHWBd" }, "source": [ "## 📚 Step 1: Import Libraries" ] }, { "cell_type": "markdown", "metadata": { "id": "YFQGLTECWkpn" }, "source": [ "\n", "In this cell we import the libraries we will use throughout the notebook:
\n", "\n", "- \n",
"
- dlt is the main library for building and running the pipeline \n", "
- rest_api_source helps us define an API source using a simple configuration \n", "
- islice (from
itertools) is a small Python helper for previewing only a few records \n",
"
\n", " In dlt, a source is the part of your pipeline that knows how to fetch data from somewhere.\n", " In this notebook, our source fetches data from the Open Library Search API.\n", "
\n", "\n", "\n",
" We define the source using rest_api_source, which lets us describe an API in a simple\n",
" Python dictionary instead of writing lots of request code.\n",
"
\n",
" 📖 Open Library Search API docs:
\n",
" \n",
" https://openlibrary.org/dev/docs/api/search\n",
" \n",
"
What does pipeline.run() do?
\n",
"\n",
"\n",
" pipeline.run() simply combines the three steps we already executed manually:\n",
"
- \n",
"
- Extract – fetch data from the Open Library API \n", "
- Normalize – convert nested JSON into relational tables \n", "
- Load – write those tables into DuckDB \n", "
In other words, this:
\n", "\n", "pipeline.run(source)is equivalent to:
\n", "\n", "pipeline.extract(source)\n",
"pipeline.normalize()\n",
"pipeline.load()\n", " There is no hidden magic. It just runs the full ELT process in order.\n", "
\n" ] }, { "cell_type": "markdown", "metadata": { "id": "7ViMq6gIfJj_" }, "source": [ "## 🔎 Step 8: Inspect the Loaded Data\n", "\n", "Now that the data is loaded into DuckDB, we can inspect it using `pipeline.dataset()`.\n", "\n", "This gives us a convenient Python interface for exploring the tables that dlt created, without writing SQL.\n", "\n", "---\n", "\n", "### List available tables\n", "\n", "First, let’s see what tables exist in the dataset:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "bmnrK1aVZXPO" }, "outputs": [], "source": [ "ds = pipeline.dataset()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "SV6J6AtBf0xq", "outputId": "19ad26bf-f34a-4f8e-c30c-5acd3342c3c5" }, "outputs": [ { "data": { "text/plain": [ "['books',\n", " 'books__author_key',\n", " 'books__author_name',\n", " 'books__ia',\n", " 'books__ia_collection',\n", " 'books__language',\n", " 'books__id_standard_ebooks',\n", " 'books__id_librivox',\n", " 'books__id_project_gutenberg',\n", " '_dlt_version',\n", " '_dlt_loads',\n", " '_dlt_pipeline_state']" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ds.tables" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 315 }, "id": "WLa4yN7lf1TF", "outputId": "d2da841b-a8bf-461f-a011-eb1db644656f" }, "outputs": [ { "data": { "application/vnd.google.colaboratory.intrinsic+json": { "summary": "{\n \"name\": \"df\",\n \"rows\": 3756,\n \"fields\": [\n {\n \"column\": \"cover_edition_key\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 1192,\n \"samples\": [\n \"OL24951484M\",\n \"OL9131663M\",\n \"OL47198575M\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"cover_i\",\n \"properties\": {\n \"dtype\": \"Int64\",\n \"num_unique_values\": 1288,\n \"samples\": [\n 842156,\n 10365881,\n 3341732\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"ebook_access\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"printdisabled\",\n \"unclassified\",\n \"no_ebook\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"edition_count\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 108,\n \"min\": 0,\n \"max\": 3546,\n \"num_unique_values\": 62,\n \"samples\": [\n 44,\n 92,\n 396\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"first_publish_year\",\n \"properties\": {\n \"dtype\": \"Int64\",\n \"num_unique_values\": 127,\n \"samples\": [\n 2008,\n 1622,\n 1962\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"has_fulltext\",\n \"properties\": {\n \"dtype\": \"boolean\",\n \"num_unique_values\": 2,\n \"samples\": [\n false,\n true\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"key\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3756,\n \"samples\": [\n \"/works/OL34662215W\",\n \"/works/OL39702699W\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"lending_edition_s\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 281,\n \"samples\": [\n \"OL45637056M\",\n \"OL26064272M\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"lending_identifier_s\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 281,\n \"samples\": [\n \"alicesadventures0000unse_v7d2\",\n \"harrypottermagic0000unse_n5w6\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"public_scan_b\",\n \"properties\": {\n \"dtype\": \"boolean\",\n \"num_unique_values\": 2,\n \"samples\": [\n true,\n false\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 2984,\n \"samples\": [\n \"1000 Facts and Trivia about Marvel Cinematic Universe, Game of Thrones, Disney, Star Wars, Harry Potter 1\",\n \"The Unofficial Harry Potter Insults Handbook\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"_dlt_load_id\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 1,\n \"samples\": [\n \"1770819876.9353185\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"_dlt_id\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3756,\n \"samples\": [\n \"ZN3UfCkWBXFxSw\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"subtitle\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 59,\n \"samples\": [\n \"Hogwarts Through the Years\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", "type": "dataframe", "variable_name": "df" }, "text/html": [ "\n", "\n",
"
\n"
],
"text/plain": [
" cover_edition_key cover_i ebook_access edition_count \\\n",
"0 OL61027601M 15155833 borrowable 396 \n",
"1 OL26378158M 15158660 printdisabled 144 \n",
"2 OL26234270M 10580435 borrowable 278 \n",
"\n",
" first_publish_year has_fulltext key lending_edition_s \\\n",
"0 1997 True /works/OL82563W OL38565767M \n",
"1 2007 True /works/OL82586W None \n",
"2 1999 True /works/OL82536W OL48101764M \n",
"\n",
" lending_identifier_s public_scan_b \\\n",
"0 harrypotterylapi0000rowl_q5r6 False \n",
"1 None False \n",
"2 bdrc-W8LS66814 False \n",
"\n",
" title _dlt_load_id \\\n",
"0 Harry Potter and the Philosopher's Stone 1770819876.9353185 \n",
"1 Harry Potter and the Deathly Hallows 1770819876.9353185 \n",
"2 Harry Potter and the Prisoner of Azkaban 1770819876.9353185 \n",
"\n",
" _dlt_id subtitle \n",
"0 lGJrV2BS8Z9qJQ None \n",
"1 F9W0WQlLwgvsFw None \n",
"2 kSdfO1XbBVAjmQ None "
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df = ds.books.df() # main table\n",
"df.head(3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OWFqaH2wgCWR"
},
"source": [
"## 💡 Conclusion\n",
"\n",
"### What dlt handled for us\n",
"\n",
"✔ API requests \n",
"✔ JSON normalization \n",
"✔ Table creation \n",
"✔ Database loading \n",
"✔ Simple dataset inspection \n",
"\n",
"---\n",
"\n",
"### But there are still friction points\n",
"\n",
"• Getting the REST API config exactly right \n",
"• Remembering paginator syntax \n",
"• Remembering how to inspect tables \n",
"• Debugging schema or pagination issues \n",
"• Writing Python or SQL to get insights \n",
"\n",
"It works... but it still takes effort.\n",
"\n",
"---\n",
"\n",
"## 🚀 Next Up: LLM-Powered Workflows\n",
"\n",
"dlt now integrates LLMs directly into the workflow to make:\n",
"\n",
"• Pipeline runs easier \n",
"• Debugging faster \n",
"• Schema inspection simpler \n",
"• Data analysis more natural \n",
"\n",
"Instead of writing glue code, you can use natural language.\n",
"\n",
"In the workshop, we will see what that looks like.\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"id": "BweSVO3igErN"
},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
================================================
FILE: cohorts/2026/workshops/dlt/dlt_homework.md
================================================
# Homework: Build Your Own dlt Pipeline
You've seen how to build a pipeline with a scaffolded source. Now it's your turn to do it from scratch with a **custom API**.
## Workshop Content
* [Workshop README](README.md)
* [dlt Pipeline Overview Notebook (Google Colab)](https://colab.research.google.com/github/anair123/data-engineering-zoomcamp/blob/workshop/dlt_2026/cohorts/2026/workshops/dlt/dlt_Pipeline_Overview.ipynb)
* [Workshop registration page](https://luma.com/hzis1yzp)
## The Challenge
For this homework, build a dlt pipeline that loads NYC taxi trip data from a custom API into DuckDB and then answer some questions using the loaded data.
## Data Source
You'll be working with **NYC Yellow Taxi trip data** from a custom API (not available as a dlt scaffold). This dataset contains records of individual taxi trips in New York City.
| Property | Value |
|----------|-------|
| Base URL | `https://us-central1-dlthub-analytics.cloudfunctions.net/data_engineering_zoomcamp_api` |
| Format | Paginated JSON |
| Page Size | 1,000 records per page |
| Pagination | Stop when an empty page is returned |
## Setup Instructions
Since this API is custom (not one of the scaffolds in dlt workspace), the setup is slightly different.
### Step 1: Create a New Project (or Reuse Your Demo Project)
If you already created a project folder while following along with the workshop demo, you can reuse that folder. Otherwise, create a new one:
```bash
mkdir taxi-pipeline
cd taxi-pipeline
```
Open this folder in Cursor (or your preferred agentic IDE).
### Step 2: Set Up the dlt MCP Server (If Not Already Done)
Choose the setup for your IDE:
Cursor - go to **Settings → Tools & MCP → New MCP Server** and add:
```json
{
"mcpServers": {
"dlt": {
"command": "uv",
"args": [
"run",
"--with",
"dlt[duckdb]",
"--with",
"dlt-mcp[search]",
"python",
"-m",
"dlt_mcp"
]
}
}
}
```
VS Code (Copilot) - create `.vscode/mcp.json` in your project folder:
```json
{
"servers": {
"dlt": {
"command": "uv",
"args": [
"run",
"--with",
"dlt[duckdb]",
"--with",
"dlt-mcp[search]",
"python",
"-m",
"dlt_mcp"
]
}
}
}
```
Claude Code - run in your terminal:
```bash
claude mcp add dlt -- uv run --with "dlt[duckdb]" --with "dlt-mcp[search]" python -m dlt_mcp
```
This enables the dlt MCP server, giving the AI access to dlt documentation, code examples, and your pipeline metadata.
### Step 3: Install dlt
```bash
pip install "dlt[workspace]"
```
### Step 4: Initialize the Project
```bash
dlt init dlthub:taxi_pipeline duckdb
```
You can name the project whatever you like. Since this API has no scaffold, the command will create:
- The dlt project files
- Cursor rules for AI assistance
**But no YAML file with API metadata.** You will need to provide the API information yourself.
### Step 5: Prompt the Agent
Now use your AI assistant to build the pipeline. You'll need to provide the API details in your prompt since there's no scaffold.
Here's an example to get you started:
```
Build a REST API source for NYC taxi data.
API details:
- Base URL: https://us-central1-dlthub-analytics.cloudfunctions.net/data_engineering_zoomcamp_api
- Data format: Paginated JSON (1,000 records per page)
- Pagination: Stop when an empty page is returned
Place the code in taxi_pipeline.py and name the pipeline taxi_pipeline.
Use @dlt rest api as a tutorial.
```
### Step 6: Run and Debug
Run your pipeline and iterate with the agent until it works:
```bash
python taxi_pipeline.py
```
---
## Questions
Once your pipeline has run successfully, use the methods covered in the workshop to investigate the following:
- **dlt Dashboard**: `dlt pipeline taxi_pipeline show`
- **dlt MCP Server**: Ask the agent questions about your pipeline
- **Marimo Notebook**: Build visualizations and run queries
We challenge you to try out the different methods explored in the workshop when answering these questions to see what works best for you. Feel free to share your thoughts on what worked (or didn't) in your submission!
### Question 1: What is the start date and end date of the dataset?
- 2009-01-01 to 2009-01-31
- 2009-06-01 to 2009-07-01
- 2024-01-01 to 2024-02-01
- 2024-06-01 to 2024-07-01
### Question 2: What proportion of trips are paid with credit card?
- 16.66%
- 26.66%
- 36.66%
- 46.66%
### Question 3: What is the total amount of money generated in tips?
- $4,063.41
- $6,063.41
- $8,063.41
- $10,063.41
### Resources
| Resource | Link |
|----------|------|
| dlt Dashboard Docs | [dlthub.com/docs/general-usage/dashboard](https://dlthub.com/docs/general-usage/dashboard) |
| marimo + dlt Guide | [dlthub.com/docs/general-usage/dataset-access/marimo](https://dlthub.com/docs/general-usage/dataset-access/marimo) |
| dlt Documentation | [dlthub.com/docs](https://dlthub.com/docs) |
---
## Submitting the solutions
- Form for submitting: https://courses.datatalks.club/de-zoomcamp-2026/homework/dlt
- Deadline: See the website
## Tips
- The API returns paginated data. Make sure your pipeline handles pagination correctly.
- If the agent gets stuck, paste the error into the chat and let it debug.
- Use the dlt MCP server to ask questions about your pipeline metadata.
## Learning in Public
We encourage everyone to share what they learned. This is called "learning in public".
Read more about the benefits [here](https://alexeyondata.substack.com/p/benefits-of-learning-in-public-and).
### Example post for LinkedIn
```
🚀 dlt Workshop of Data Engineering Zoomcamp by @DataTalksClub complete!
Just finished the Data Ingestion workshop with @dltHub. Learned how to:
✅ Build REST API data pipelines with dlt
✅ Use AI-assisted development with dlt MCP Server
✅ Load paginated API data into DuckDB
✅ Inspect pipeline data with dlt Dashboard and marimo notebooks
Built a full NYC taxi data pipeline from a custom API - AI-assisted data engineering is the future!
Here's my homework solution:
Following along with this amazing free course - who else is learning data engineering?
You can sign up here: https://github.com/DataTalksClub/data-engineering-zoomcamp/
```
### Example post for Twitter/X
```
🔄 dlt Workshop of Data Engineering Zoomcamp done!
- REST API pipelines with @dltHub
- AI-assisted pipeline building
- DuckDB as local data warehouse
- dlt Dashboard & marimo notebooks
My solution:
Free course by @DataTalksClub: https://github.com/DataTalksClub/data-engineering-zoomcamp/
```
================================================
FILE: cohorts/2026/workshops/dlt/open_library_pipeline.py
================================================
"""Pipeline to ingest data from the Open Library Search API."""
import dlt
from dlt.sources.rest_api import rest_api_source
def open_library_source(query: str = "harry potter"):
"""
Create a dlt source for the Open Library Search API.
Args:
query: Search query string (default: "harry potter")
"""
return rest_api_source({
"client": {
"base_url": "https://openlibrary.org",
},
"resource_defaults": {
"primary_key": "key",
"write_disposition": "replace",
},
"resources": [
{
"name": "books",
"endpoint": {
"path": "search.json",
"params": {
"q": query,
"limit": 100,
},
"data_selector": "docs",
"paginator": {
"type": "offset",
"limit": 100,
"offset_param": "offset",
"limit_param": "limit",
"total_path": "numFound",
},
},
},
],
})
if __name__ == "__main__":
pipeline = dlt.pipeline(
pipeline_name="open_library_pipeline",
destination="duckdb",
dataset_name="open_library_data",
progress="log",
)
# Load Harry Potter books from Open Library
load_info = pipeline.run(open_library_source(query="harry potter"))
print(load_info)
================================================
FILE: cohorts/2026/workshops/dlt/pyproject.toml
================================================
[project]
name = "zoomcamp-workshop-prep"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
"altair>=6.0.0",
"dlt[workspace]>=1.21.0",
"ibis-framework[duckdb]>=12.0.0",
"jupyterlab>=4.5.4",
"marimo>=0.19.9",
]
================================================
FILE: cohorts/2026/workshops/dlt.md
================================================
# From APIs to Warehouses: AI-Assisted Data Ingestion with dlt
[Video](https://www.youtube.com/watch?v=5eMytPBgmVs)
This hands-on workshop focuses on building reliable data ingestion pipelines to data warehouses (for example, Snowflake) using dlt (data load tool), enhanced with LLMs, the dlt dashboard, and dlt MCP.
## What you'll learn
You'll work through the key building blocks of a production-ready ingestion setup, including:
- Extracting data from APIs, files, and databases
- Normalizing data into consistent schemas
- Writing data to a data warehouse (e.g. Snowflake)
- Using LLMs to accelerate dlt pipeline development
- Validating data and schema changes using the dlt dashboard and dlt MCP
The session is fully practical and code-driven. By the end of the workshop, you'll understand how to design maintainable, scalable ingestion pipelines and use AI and validation tools to build them faster and with confidence.
## Materials
* [Workshop instructions](dlt/README.md)
* [dlt Pipeline Overview Notebook (Google Colab)](https://colab.research.google.com/github/anair123/data-engineering-zoomcamp/blob/workshop/dlt_2026/cohorts/2026/workshops/dlt/dlt_Pipeline_Overview.ipynb)
* [Homework](dlt/dlt_homework.md)
* [Homework submission form](https://courses.datatalks.club/de-zoomcamp-2026/homework/dlt)
## About the Speaker
**Aashish Nair** is a Data Engineer at dltHub and the creator of the famous _dlt deployment_ course, where he teaches best practices for running dlt pipelines in production.
================================================
FILE: learning-in-public.md
================================================
# Learning in public
Most people learn in private: they consume content but don't tell
anyone about it. There's nothing wrong with it.
But we want to encourage you to document your progress and
share it publicly on social media.
It helps you get noticed and will lead to:
* Expanding your network: meeting new people and making new friends
* Being invited to meetups, conferences and podcasts
* Landing a job or getting clients
* Many other good things
Here's a more comprehensive reading on why you want to do it: https://github.com/readme/guides/publishing-your-work
## Learning in Public for Zoomcamps
When you submit your homework or project, you can also submit
learning in public posts:
\n",
"\n",
"\n",
" \n",
"
\n",
"
\n",
" \n",
" | \n", " | cover_edition_key | \n", "cover_i | \n", "ebook_access | \n", "edition_count | \n", "first_publish_year | \n", "has_fulltext | \n", "key | \n", "lending_edition_s | \n", "lending_identifier_s | \n", "public_scan_b | \n", "title | \n", "_dlt_load_id | \n", "_dlt_id | \n", "subtitle | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "OL61027601M | \n", "15155833 | \n", "borrowable | \n", "396 | \n", "1997 | \n", "True | \n", "/works/OL82563W | \n", "OL38565767M | \n", "harrypotterylapi0000rowl_q5r6 | \n", "False | \n", "Harry Potter and the Philosopher's Stone | \n", "1770819876.9353185 | \n", "lGJrV2BS8Z9qJQ | \n", "None | \n", "
| 1 | \n", "OL26378158M | \n", "15158660 | \n", "printdisabled | \n", "144 | \n", "2007 | \n", "True | \n", "/works/OL82586W | \n", "None | \n", "None | \n", "False | \n", "Harry Potter and the Deathly Hallows | \n", "1770819876.9353185 | \n", "F9W0WQlLwgvsFw | \n", "None | \n", "
| 2 | \n", "OL26234270M | \n", "10580435 | \n", "borrowable | \n", "278 | \n", "1999 | \n", "True | \n", "/works/OL82536W | \n", "OL48101764M | \n", "bdrc-W8LS66814 | \n", "False | \n", "Harry Potter and the Prisoner of Azkaban | \n", "1770819876.9353185 | \n", "kSdfO1XbBVAjmQ | \n", "None | \n", "
You can watch this video to see how your learning in public posts may look like:
## Daily Documentation
- **Post Daily Diaries**: Document what you learn each day, including the challenges faced and the methods used to overcome them.
- **Create Quick Videos**: Make short videos showcasing your work and upload them to GitHub.
Send a PR if you want to suggest improvements for this document
================================================
FILE: projects/README.md
================================================
## Course Project
[🎥 Projects how-to (watch it!)](https://www.youtube.com/watch?v=BL0E8xO8OnE)
### Objective
The goal of this project is to apply everything we have learned
in this course to build an end-to-end data pipeline.
### Problem statement
Develop a dashboard with two tiles by:
* Selecting a dataset of interest (see [Datasets](#datasets))
* Creating a pipeline for processing this dataset and putting it to a datalake
* Creating a pipeline for moving the data from the lake to a data warehouse
* Transforming the data in the data warehouse: prepare it for the dashboard
* Building a dashboard to visualize the data
## Data Pipeline
The pipeline could be **stream** or **batch**: this is the first thing you'll need to decide
* **Stream**: If you want to consume data in real-time and put them to data lake
* **Batch**: If you want to run things periodically (e.g. hourly/daily)
## Technologies
You don't have to limit yourself to technologies covered in the course. You can use alternatives as well:
* **Cloud**: AWS, GCP, Azure, ...
* **Infrastructure as code (IaC)**: Terraform, Pulumi, Cloud Formation, ...
* **Workflow orchestration**: Airflow, Prefect, Luigi, ...
* **Data Warehouse**: BigQuery, Snowflake, Redshift, ...
* **Batch processing**: Spark, Flink, AWS Batch, ...
* **Stream processing**: Kafka, Pulsar, Kinesis, ...
If you use a tool that wasn't covered in the course, be sure to explain what that tool does.
If you're not certain about some tools, ask in Slack.
## Dashboard
You can use any of the tools shown in the course (Looker Studio or Streamlit) or any other BI tool of your choice to build a dashboard. If you do use another tool, please specify and make sure that the dashboard is somehow accessible to your peers.
Your dashboard should contain at least two tiles, we suggest you include:
- 1 graph that shows the distribution of some categorical data
- 1 graph that shows the distribution of the data across a temporal line
Ensure that your graph is easy to understand by adding references and titles.
Example dashboard: 
## Peer reviewing
> [!IMPORTANT]
> To evaluate the projects, we'll use peer reviewing. This is a great opportunity for you to learn from each other.
> * To get points for your project, you need to evaluate 3 projects of your peers
> * You get 3 extra points for each evaluation
## Evaluation Criteria
* Problem description
* 0 points: Problem is not described
* 2 points: Problem is described but shortly or not clearly
* 4 points: Problem is well described and it's clear what the problem the project solves
* Cloud
* 0 points: Cloud is not used, things run only locally
* 2 points: The project is developed in the cloud
* 4 points: The project is developed in the cloud and IaC tools are used
* Data ingestion (choose either batch or stream)
* Batch / Workflow orchestration
* 0 points: No workflow orchestration
* 2 points: Partial workflow orchestration: some steps are orchestrated, some run manually
* 4 points: End-to-end pipeline: multiple steps in the DAG, uploading data to data lake
* Stream
* 0 points: No streaming system (like Kafka, Pulsar, etc)
* 2 points: A simple pipeline with one consumer and one producer
* 4 points: Using consumer/producers and streaming technologies (like Kafka streaming, Spark streaming, Flink, etc)
* Data warehouse
* 0 points: No DWH is used
* 2 points: Tables are created in DWH, but not optimized
* 4 points: Tables are partitioned and clustered in a way that makes sense for the upstream queries (with explanation)
* Transformations (dbt, spark, etc)
* 0 points: No tranformations
* 2 points: Simple SQL transformation (no dbt or similar tools)
* 4 points: Tranformations are defined with dbt, Spark or similar technologies
* Dashboard
* 0 points: No dashboard
* 2 points: A dashboard with 1 tile
* 4 points: A dashboard with 2 tiles
* Reproducibility
* 0 points: No instructions how to run the code at all
* 2 points: Some instructions are there, but they are not complete
* 4 points: Instructions are clear, it's easy to run the code, and the code works
> [!NOTE]
> It's highly recommended to create a new repository for your project (not inside an existing repo) with a meaningful title, such as
> "Quake Analytics Dashboard" or "Bike Data Insights" and include as many details as possible in the README file. ChatGPT can assist you with this. Doing so will not only make it easier to showcase your project for potential job opportunities but also have it featured on the [Projects Gallery App](#projects-gallery).
> If you leave the README file empty or with minimal details, there may be point deductions as per the [Evaluation Criteria](#evaluation-criteria).
## Going the extra mile (Optional)
> [!NOTE]
> The following things are not covered in the course, are entirely optional and they will not be graded.
However, implementing these could significantly enhance the quality of your project:
* Add tests
* Use make
* Add CI/CD pipeline
If you intend to include this project in your portfolio, adding these additional features will definitely help you to stand out from others.
## Cheating and plagiarism
Plagiarism in any form is not allowed. Examples of plagiarism:
* Taking somebody's else notebooks and projects (in full or partly) and using it for the capstone project
* Re-using your own projects (in full or partly) from other courses and bootcamps
* Re-using your midterm project from ML Zoomcamp in capstone
* Re-using your ML Zoomcamp from previous iterations of the course
Violating any of this will result in 0 points for this project.
## Resources
### Datasets
Refer to the provided [datasets](datasets.md) for possible selection.
### Helpful Links
* [Unit Tests + CI for Airflow](https://www.astronomer.io/events/recaps/testing-airflow-to-bulletproof-your-code/)
* [CI/CD for Airflow (with Gitlab & GCP state file)](https://engineering.ripple.com/building-ci-cd-with-airflow-gitlab-and-terraform-in-gcp)
* [CI/CD for Airflow (with GitHub and S3 state file)](https://programmaticponderings.com/2021/12/14/devops-for-dataops-building-a-ci-cd-pipeline-for-apache-airflow-dags/)
* [CD for Terraform](https://medium.com/towards-data-science/git-actions-terraform-for-data-engineers-scientists-gcp-aws-azure-448dc7c60fcc)
* [Spark + Airflow](https://medium.com/doubtnut/github-actions-airflow-for-automating-your-spark-pipeline-c9dff32686b)
### Projects Gallery
Explore a collection of projects completed by members of our community. The projects cover a wide range of topics and utilize different tools and techniques. Feel free to delve into any project and see how others have tackled real-world problems with data, structured their code, and presented their findings. It's a great resource to learn and get ideas for your own projects.
[](https://datatalksclub-projects.streamlit.app/)
### DE Zoomcamp 2023
* [2023 Projects](../cohorts/2023/project.md)
### DE Zoomcamp 2022
* [2022 Projects](../cohorts/2022/project.md)
================================================
FILE: projects/datasets.md
================================================
## Datasets
Here are some datasets that you could use for the project:
* [Kaggle](https://www.kaggle.com/datasets)
* [AWS datasets](https://registry.opendata.aws/)
* [UK government open data](https://data.gov.uk/)
* [Github archive](https://www.gharchive.org)
* [Awesome public datasets](https://github.com/awesomedata/awesome-public-datasets)
* [Million songs dataset](http://millionsongdataset.com)
* [Some random datasets](https://components.one/datasets/)
* [COVID Datasets](https://www.reddit.com/r/datasets/comments/n3ph2d/coronavirus_datsets/)
* [Datasets from Azure](https://docs.microsoft.com/en-us/azure/azure-sql/public-data-sets)
* [Datasets from BigQuery](https://cloud.google.com/bigquery/public-data/)
* [Dataset search engine from Google](https://datasetsearch.research.google.com/)
* [Public datasets offered by different GCP services](https://cloud.google.com/solutions/datasets)
* [European statistics datasets](https://ec.europa.eu/eurostat/data/database)
* [Datasets for streaming](https://github.com/ColinEberhardt/awesome-public-streaming-datasets)
* [Dataset for Santander bicycle rentals in London](https://cycling.data.tfl.gov.uk/)
* [Common crawl data](https://commoncrawl.org/) (copy of the internet)
* [NASA's EarthData](https://search.earthdata.nasa.gov/search) (May require introductory geospatial analysis)
* Collection Of Data Repositories
* [part 1](https://www.kdnuggets.com/2022/04/complete-collection-data-repositories-part-1.html) (from agriculture and finance to government)
* [part 2](https://www.kdnuggets.com/2022/04/complete-collection-data-repositories-part-2.html) (from healthcare to transportation)
* [Data For Good by Meta](https://dataforgood.facebook.com/dfg/tools)
PRs with more datasets are welcome!
It's not mandatory that you use a dataset from this list. You can use any dataset you want.
================================================
FILE: workshop-best-practices.md
================================================
# Workshop Best Practices
Preferences and patterns learned from building the PyFlink streaming workshop.
## Structure and Pacing
- Introduce services one at a time, not all at once. Start with one container
(e.g., Redpanda), explain it, use it. Then add the next (PostgreSQL), etc.
- Start with the simplest version that works (plain Python consumer), then
motivate the more complex tool (Flink) by showing what's missing.
- Use `docker compose up