Fill the waitlist form to enroll into this course
Glue Catalog & Crawler for Lakehouse Engineering
Build the Metadata Foundation of a Modern Lakehouse (ELT, Athena & S3)
Course Summary
This course is a foundational Lakehouse mastery course designed specifically for Data Engineers, not analysts and not ETL-only developers.
Modern data engineering is no longer just about moving data into warehouses.
It starts with metadata-first design, data lakes, and serverless discovery and querying — and that foundation is built using AWS Glue Catalog and Glue Crawler.
This course teaches you how modern data platforms actually expose data to analysts in days instead of months, using Glue Catalog, Glue Crawler, and Amazon Athena, following real-world ELT and producer-driven patterns.
This course is part of the RADE Diamond Membership – Applied Data Engineering Mastery Program and serves as a core building block of the Lakehouse Mastery track.
What This Course Is Really About
This is not a “how to click around Glue” course.
It teaches:
-
Why Glue Catalog exists
-
Why metadata matters more than ETL early on
-
How companies enable analytics directly on S3
-
How Data Engineers reduce time-to-insight from months to days
You’ll learn to think like a modern data platform engineer, not just a pipeline builder.
What You’ll Learn
1 Traditional ETL vs Modern ELT (Context That Interviews Expect)
-
Why traditional ETL pipelines took 4–5 months
-
Why OLTP databases cannot serve analytics
-
How cloud + data lakes changed the architecture
-
Why ELT is the default starting point, not ETL
2 AWS Glue Catalog — Metadata, Not Data
-
What Glue Catalog really is (serverless metadata repository)
-
Databases vs tables in Glue Catalog
-
External tables and how they point to S3
-
Metadata vs actual data (critical interview distinction)
-
How Athena and BI tools rely entirely on Glue Catalog
3 Glue Crawler — Automated Schema Discovery
-
How Glue Crawler works internally
-
Correct S3 path configuration (folder vs file — common mistake)
-
Built-in classifiers (CSV, Parquet, JSON, Avro, ORC)
-
When (and when not) to use custom classifiers
-
Sampling, exclude patterns, multiple schema handling
-
Crawler logs and troubleshooting
You’ll also learn why crawlers are used in development but not blindly scheduled in production.
4 Querying the Data Lake with Athena
-
How Athena queries S3 using Glue Catalog metadata
-
Athena execution flow (metadata → S3 → results)
-
Query results bucket configuration (often missed)
-
Athena pricing model ($5/TB scanned) and cost awareness
-
Using Athena with Tableau / Power BI
-
When Athena is perfect — and when it is not
5 Real-World Lakehouse Workflow
-
Producer-driven data ingestion (source teams push to S3)
-
Day-1 querying without ETL
-
Development vs production practices:
-
Crawlers for discovery
-
CloudFormation for production deployment
-
-
How this fits before Redshift, not instead of it
How This Fits in Lakehouse Mastery
This course answers one critical question:
“How does data become queryable in a data lake before any warehouse exists?”
It naturally precedes:
-
Glue ETL / Spark processing
-
Redshift / Warehouse modeling
-
Lakehouse optimization & governance
Without this knowledge, Lakehouse architecture doesn’t make sense.
Who This Course Is For
✔ Data Engineers moving into modern Lakehouse architectures
✔ AWS Data Engineers working with S3, Athena, Glue, Redshift
✔ Engineers preparing for senior DE interviews
Who This Course Is NOT For
✖ Engineers expecting only ETL coding
Outcomes
By the end of this course, you will be able to:
1 Design metadata-first Lakehouse architectures
2 Use Glue Catalog correctly and confidently
3 Automate schema discovery using Glue Crawlers
4 Enable fast analytics directly on S3 using Athena
5 Explain ELT vs ETL clearly in interviews
6 Position yourself as a modern data platform engineer
Course Curriculum
Sachin Chandrashekhar
Lead Data Engineer @ World's #1 Airline
