Get Your CertGuidesPlayCompete
‹ All study guides

What Is AWS Glue? Data Catalog, Crawlers and ETL Jobs

Updated 19 October 2026 · 18 min read

AWS Glue is a serverless data integration service: it finds your data, records what it looks like in a central catalog, and runs the jobs that clean it, reshape it and move it somewhere it can be queried. You don't run any servers or clusters for it. You describe the work, and Glue provides the compute while the work runs and bills you for that time.

This guide is AWS Glue explained piece by piece. It covers the Data Catalog, crawlers, ETL jobs and the workers they run on, Glue Studio, triggers and workflows, and streaming jobs. Then it covers who uses Glue, the services it is normally paired with, the features worth remembering, and the traps the Data Engineer Associate (DEA-C01), Cloud Practitioner (CLF-C02) and Solutions Architect Associate (SAA-C03) exams set around it. The same material is drawn on a whiteboard in the video linked at the end.

What is AWS Glue?

Picture a company whose data is everywhere: CSV exports in Amazon S3, rows in a relational database, application logs piling up by the gigabyte. Someone asks a simple question and nobody can answer it, because nobody knows what data exists, where it is, or what shape it's in. Glue exists for that problem. AWS describes it as a service to discover, prepare, move and integrate data from many sources, and it can connect to more than 70 of them.

What makes Glue different from a plain Spark cluster is that several jobs live in one service: discovering data, a central catalog of it, cleaning, transforming, and running the pipelines. There is no infrastructure to manage, and it handles classic extract-transform-load (ETL), extract-load-transform (ELT) and streaming workloads.

An analogy helps. Imagine a huge library with no card index. The books are all on the shelves, but nobody can find anything. In Glue terms:

  • The Data Catalog is the card index.
  • Crawlers are the librarians who walk the shelves and fill in the cards.
  • ETL jobs are the staff who repair books and reshelve them where readers can use them.

How AWS Glue works, step by step

A typical Glue pipeline follows the same five steps, and they are a good map for the rest of this article:

  1. A crawler scans your data. It connects to a data store, such as an S3 prefix or a database.
  2. The crawler writes table definitions into the Data Catalog. Each table records a schema and where the data lives.
  3. You define an ETL job. Glue can generate the script for you, or you write it yourself in PySpark or Scala.
  4. A trigger starts the job. That can be a schedule, an event, or a run you start on demand.
  5. The job extracts, transforms and loads. It reads from the source, transforms the data in an Apache Spark environment, and writes the result to a target.

After that, you watch the job runs in the console's job run dashboard and in Amazon CloudWatch.

The Glue Data Catalog

The Data Catalog is the heart of Glue. It holds metadata only. Its two main objects are:

  • Databases: a logical group of related tables.
  • Tables: a schema plus the location of the data it describes.

The data itself never moves into the catalog. A table that points at an S3 prefix describes files that stay in S3; a table for an Amazon RDS database describes rows that stay in RDS. This point turns up in exam distractors: the catalog is not a storage service and not a warehouse.

Each AWS account has one Data Catalog per Region. Many beginner posts still say "one per account", so remember the Region part. The catalog is also compatible with the Apache Hive metastore, so any application built for a Hive metastore can use it. Amazon EMR, for example, can point Hive and Spark at the Glue Data Catalog instead of keeping its own metastore on a cluster.

Other ways into the catalog

Crawlers are the best-known way to fill the catalog, but not the only one:

  • Console, API and CLI: you can create databases and tables by hand.
  • Glue jobs: a job can create or update catalog tables as it writes.
  • Streaming sources: a Kinesis or Kafka source needs a Data Catalog table created manually for the streaming job.

To reach a database, Glue uses a connection: another catalog object that holds the properties needed to connect to a data store.

Storage is cheap at the start: the first million objects stored in the catalog, and the first million requests each month, are free.

Glue crawlers and classifiers

A crawler connects to a data store and works out what is in it using classifiers. It tries them in a fixed order:

  1. Your custom classifiers first, in the order you listed them. These use grok patterns, XML tags or JSON paths.
  2. Then the built-in classifiers, which recognise formats such as CSV, JSON, Parquet, Avro, ORC, XML, common log formats and JDBC databases.

The first classifier that is fully certain of a match wins. If none is certain, the one with the highest certainty is used, and if none matches at all, the table is classified as UNKNOWN.

Partitions

Crawlers also recognise how data is laid out in folders. A Hive-style path such as s3://sales/year=2026/month=10/ becomes a table with partition columns named year and month. Without the key=value form, the partitions still get detected but are named partition_0, partition_1 and so on, which is a reason to name your folders the Hive way.

Crawler options worth knowing

  • Incremental crawls add only new partitions instead of rescanning everything.
  • S3 event-driven crawls read S3 event notifications from an Amazon SQS queue, so a recrawl lists only the folders where objects changed, not the whole target.
  • Table level tells the crawler at which folder depth a table starts, counting from the bucket.
  • Schema change controls decide whether a changed schema updates the table or only gets logged.
  • Partition indexes are created by default for S3 and Delta Lake targets.

The grouping trap

By default, a crawler decides whether files under one path belong in one table by looking at two things: whether the data is compatible and whether the schemas are similar. If the schemas drift apart, it creates separate tables. AWS's own example has year=2017 and year=2018 files with different schemas turning into two tables, year_2017 and year_2018, instead of one partitioned table. The fix is the crawler setting "Create a single schema for each S3 path" (CombineCompatibleSchemas), which merges compatible data into one table.

Glue ETL jobs

A job is where the work happens. There are three job types to know:

  • Spark (glueetl): batch ETL on Apache Spark. This is the default and the one exams mean by "a Glue job".
  • Spark Streaming (gluestreaming): continuous jobs that read from streams.
  • Python shell (pythonshell): a plain Python script without Spark, for small tasks.

A fourth type, Glue for Ray, closed to new customers on April 30, 2026, so don't plan around it.

DynamicFrames

Glue's own data structure is the DynamicFrame, a distributed table that describes itself. Unlike a Spark DataFrame, it doesn't need a schema up front, which suits messy semi-structured data. When a field holds conflicting types, for example a number in some records and a string in others, the DynamicFrame keeps both as a "choice" type, and resolveChoice settles it by casting, splitting into columns, or building a struct. You can convert a DynamicFrame to a Spark DataFrame and back in the same script, so you can use whichever API fits each step.

Workers and DPUs

Jobs run on workers, and Glue measures capacity in DPUs (data processing units). One DPU is 4 vCPUs and 16 GB of memory. The worker types map onto DPUs:

  • G.1X: 1 DPU, 4 vCPUs and 16 GB.
  • G.2X: 2 DPUs, 8 vCPUs and 32 GB.
  • G.4X and G.8X: 4 and 8 DPUs, for heavier jobs.
  • G.12X, G.16X and R.1X to R.8X: larger and memory-optimized types.
  • G.025X: a quarter DPU, for low-volume streaming jobs only.

AWS recommends G.1X and G.2X for most transforms, joins and queries. A Spark job needs at least two workers, which is why a small script is often better as a Python shell job or an AWS Lambda function. A Python shell job runs on 0.0625 DPU by default, or 1 DPU.

AWS Glue pricing

You pay per DPU-hour, billed by the second. At the time of writing, AWS's pricing page lists $0.44 per DPU-hour for ETL jobs, and crawlers are charged at the same DPU-hour rate. A job that runs 10 G.1X workers for 30 minutes uses 5 DPU-hours. Prices vary by Region, so check the AWS Glue pricing page for yours.

Glue Studio and interactive sessions

You don't have to write code at all. Glue Studio is a visual editor: you drag in a source, add transforms, pick a target, and see the schema and a preview of the data at each step while Glue generates the script. If you prefer notebooks, interactive sessions let you explore and prepare data from the IDE or notebook of your choice, without managing any servers. The older development endpoints work only with Glue versions before 2.0, which have reached end of life.

Triggers and workflows

A single job is rarely the whole pipeline. Glue gives you two ways to chain steps:

  • Triggers start a job on demand, on a schedule, or in response to an event. Inside a workflow, a trigger can also fire when earlier jobs or crawlers finish.
  • Workflows wire crawlers, jobs and triggers into one graph that you can view and monitor as a unit. A workflow starts on a schedule, on demand, or from an Amazon EventBridge event.

A common workflow is crawl, transform, then crawl again: the first crawler catalogs new raw data, the job writes cleaned output, and the second crawler catalogs that output so it can be queried. Starting it from EventBridge when a new object lands in S3 makes it event-driven. The trigger can also batch events, waiting up to 15 minutes, so a burst of files can start one run; by default each event starts its own. AWS advises keeping a workflow to 100 or fewer jobs, crawlers and triggers combined.

For pipelines that reach beyond Glue, AWS Step Functions and Amazon Managed Workflows for Apache Airflow (MWAA) can orchestrate Glue jobs too; more on that below.

Glue streaming ETL

A streaming ETL job reads from Amazon Kinesis Data Streams, Apache Kafka or Amazon MSK, cleans and transforms the records, and loads them into S3 or a JDBC data store. Two details matter:

  • It processes data in 100-second windows by default. You can change the window size.
  • It tracks progress with checkpoints, not job bookmarks. When a streaming job restarts, it resumes from its checkpoint.

A Kafka source needs a Glue connection, and the stream needs a Data Catalog table defined by hand. Streaming jobs are billed for every hour they run, because they run continuously.

Who uses AWS Glue

AWS publishes a few customer stories that name Glue directly:

  • Merck: in an AWS video case study, Merck uses AWS Glue and Amazon Kinesis to ingest, transform and contextualize near real-time data from its manufacturing processes, and loads the harmonized data into Amazon Redshift.
  • FinAccel: on AWS's Glue customers page, the Indonesian fintech says it uses Glue to power its day-to-day ETL and pays only for the compute it uses.
  • Ontraport: a CRM and marketing software company, described its move on the AWS Big Data Blog in 2023. It had 100 TB of log data to process. Glue jobs read the raw files from S3 and wrote Parquet for Amazon Athena. Ten G.2X workers handled about 500 million records in under an hour, and the cost of processing fell by 80%, from about $500 to about $100 per terabyte.

Services AWS Glue works with

Glue rarely appears alone, in real architectures or in exam questions. The usual picture is a data lake on Amazon S3, which the S3 storage classes guide covers from the storage side.

Reading through the catalog

Raw data lands in S3. Glue crawls it, catalogs it, and transforms it, writing cleaned data back to S3. Then the readers come in through the catalog:

  • Amazon Athena: serverless SQL directly on S3. The databases and tables you see in the Athena editor are the Glue Data Catalog's.
  • Amazon Redshift Spectrum: queries the same S3 files from Redshift without loading them, through an external schema that points at a catalog database.
  • Amazon EMR: can use the catalog as its Hive metastore, which keeps table definitions alive when transient clusters are terminated.

Getting data in

  • AWS Database Migration Service (DMS): copies a database into S3 as a full load plus ongoing changes (change data capture).
  • Amazon Kinesis Data Streams: feeds Glue streaming jobs.
  • Amazon Data Firehose: can convert JSON records to Parquet or ORC before writing them to S3, using a schema from a Glue Data Catalog table.

Running it all

  • Amazon EventBridge: starts a Glue workflow when an event arrives, such as a new S3 object. The SQS vs SNS vs EventBridge guide explains EventBridge rules in more depth.
  • AWS Step Functions: a Task state with the glue:startJobRun.sync integration starts a Glue job and waits for it to finish before moving on, with retries and error handling around it.
  • Amazon MWAA: runs Glue jobs as steps in an Airflow DAG.

Governing and showing it

  • AWS Lake Formation: the permissions layer on top of the catalog. It grants fine-grained access to databases, tables, columns, rows and cells, and Athena, Redshift Spectrum, EMR and Glue all enforce it. It works alongside IAM, not instead of it.
  • Amazon CloudWatch: receives Glue's job metrics automatically, plus continuous logs and alarms.
  • Amazon Quick Sight (formerly QuickSight): builds dashboards on the results, reading through Athena or Redshift.

Glue features to remember

These eight features come up again and again, in production and in exam questions.

1. Job bookmarks

A job bookmark is state that Glue keeps for each job, recording what the job has already processed so the next run reads only new data. For S3 sources it goes by each object's last-modified time. Two catches:

  • Bookmarks are disabled by default. A new job reprocesses the whole dataset on every run until you set the bookmark option to Enable.
  • The script has to cooperate. It must call job.init() at the start and job.commit() at the end, and each source you want tracked needs a transformation_ctx.

Bookmarks track sources only. Rewinding or resetting a bookmark doesn't delete what earlier runs wrote to the target, so reprocessing can duplicate output. Streaming jobs and Python shell jobs don't support bookmarks at all.

2. Flex execution class

The Flex execution class is for jobs with no deadline: testing, pre-production and one-time loads. It costs less; AWS's pricing example uses $0.29 per DPU-hour against $0.44 for Standard. Use Standard when a job is time-sensitive and needs fast startup. Flex needs Glue 3.0 or later and runs only on G.1X or G.2X workers.

3. Auto scaling

With auto scaling on (Glue 3.0 and later), Glue adds and removes workers as the job's parallelism changes from stage to stage. The number of workers you set becomes the maximum. It suits jobs whose input volume varies a lot from run to run, where a fixed worker count is either too small on busy nights or mostly idle on quiet ones.

4. Glue Data Quality

Glue Data Quality is built on the open-source Deequ framework. You write rules in DQDL, the Data Quality Definition Language, or have Glue recommend rules from your data. It comes with 25 or more built-in rule types, and each run gets a score: the percentage of rules that pass. It works in two places:

  • At rest: on tables in the Data Catalog.
  • In transit: inside an ETL job, with the Evaluate Data Quality transform, so bad records can be stopped before they reach the target.

5. Sensitive data detection

The Detect PII transform scans data for personal information and can redact it, partially redact it, or replace it with a SHA-256 hash. Hashing keeps a column usable for joins, which redaction does not. You can scan every cell, or sample rows to find which columns hold personal data, which costs less.

6. FindMatches

FindMatches is a machine learning transform that finds duplicate records even when there is no common ID and the fields don't match exactly, such as "J. Smith, 12 Oak St" and "John Smith, 12 Oak Street". You teach it with labelled examples, and then call it from an ETL job.

7. Grouping small files

Thousands of tiny files slow a Spark job down: every file means more S3 requests and more tasks for the driver to track. AWS suggests files of around 128 MB. For text formats such as CSV and JSON, the groupFiles: inPartition option reads small files in bigger groups, with an optional groupSize. Glue turns grouping on by itself above 50,000 input files. It doesn't apply to Parquet, ORC or Avro.

8. Pushdown predicates

A pushdown predicate filters on partition columns from the Data Catalog, so Glue lists and reads only the matching partitions instead of reading everything and filtering afterwards. It only works when you read with from_catalog, because the partition columns come from the catalog. Don't partition on a high-cardinality column such as a customer ID: too many partitions slow catalog lookups and create more small files.

Defaults and limits

  • A job can run for at most 7 days. For Spark jobs on Glue 5.0 and later, the default timeout is 8 hours (480 minutes); on 4.0 and earlier it is 48 hours.
  • Retries can be set from 0 to 10, but a job that hits its timeout isn't retried.
  • A job allows one concurrent run by default.
  • The first million Data Catalog objects are free.

What the AWS exams ask about Glue

The Data Engineer Associate exam goes deepest. Glue runs through its domains: ingesting and transforming data, scheduling jobs and crawlers, orchestrating pipelines with Glue workflows, and using crawlers to populate the Data Catalog. On the DEA-C01 practice tests, about half the questions mention Glue somewhere. Glue is also on the in-scope service lists for Cloud Practitioner and Solutions Architect Associate, where questions stay at the "which service fits" level.

A job that reprocesses everything

The classic Glue question: a nightly job reads files from S3, the target's row count keeps doubling, and every run reads every file. The answer is to enable job bookmarks, because they are off by default. The distractors are a crawler before the job (it updates metadata, not job state), a streaming job (streams read Kinesis or Kafka, not an S3 prefix) and a lifecycle rule that deletes source files. This is the question from the video, a practice question on job bookmarks for incremental loads.

The streaming version flips the answer: a restarted streaming job resumes from checkpoints, and bookmarks are the wrong tool. See a practice question on streaming job checkpoints.

A crawler that makes dozens of tables

When a crawler suddenly produces tables named after partitions, such as dt_2026_08_14, the schemas have drifted too far apart to group. The fix is "Create a single schema for each S3 path", not a custom classifier or a different schema change policy. Work through it in a practice question on crawler table grouping.

Cost and performance tuning

Read the scenario's constraint carefully, because each tuning feature answers a different problem:

Data quality, privacy and duplicates

Look-alikes

Three names get confused on purpose:

  • Glue Data Catalog: metadata, not data.
  • AWS Lake Formation: permissions on that data and metadata.
  • AWS Glue DataBrew: visual, no-code data preparation for analysts who don't write code, with more than 250 prebuilt transformations. See a practice question on no-code data preparation.

Two more pairings show up often. Transient EMR clusters that lose their Hive tables need the Glue Data Catalog as the metastore, in a practice question on the Data Catalog as a Hive metastore. And a Glue job that must wait before the next step runs belongs in Step Functions with .sync, in a practice question on Step Functions and Glue.

At Cloud Practitioner and Solutions Architect level

The entry-level questions describe the job and ask for the service. "Discover schemas, keep a central catalog, run ETL, no servers" is Glue, as in a Cloud Practitioner question on serverless data integration. "Query logs in S3 with SQL, pay per query, no cluster" is the Glue Data Catalog plus Athena, as in a Solutions Architect question on querying logs in place.

More questions are on the full lists for DEA-C01, CLF-C02 and SAA-C03.

Frequently asked questions

Is AWS Glue an ETL tool?

Yes. Glue runs ETL jobs on Apache Spark, and it also supports ELT and streaming. It is wider than an ETL engine, though: the Data Catalog, crawlers, Data Quality and workflows are part of the same service.

Is AWS Glue serverless?

Yes. You choose a worker type and a number of workers, and Glue provides the compute while the job runs. There are no clusters or instances to manage, and you pay only while jobs and crawlers run.

How much does AWS Glue cost?

ETL jobs and crawlers are billed per DPU-hour, by the second; the pricing page lists $0.44 per DPU-hour, and lower for Flex jobs. The first million Data Catalog objects and requests each month are free. Prices vary by Region.

What is the difference between a Glue crawler and the Data Catalog?

The crawler is the process that scans a data store and works out its schema. The Data Catalog is where the result is stored, as databases and tables of metadata. You can fill the catalog without crawlers, but a crawler always writes to a catalog.

When should I use Glue instead of Amazon EMR or AWS Lambda?

AWS's guidance puts Glue at Spark ETL that runs now and then, such as daily, weekly or monthly jobs, with nothing to manage. EMR suits jobs that run frequently, and supports engines such as Hive and Presto as well as Spark. Lambda suits small pieces of work that finish within its 15-minute limit. "Serverless Spark" alone doesn't point to Glue, because EMR Serverless exists too.

Watch the whiteboard video

The video draws a Glue pipeline from the first crawler to the readers on one board, so you can see how the catalog, jobs, triggers and the surrounding services connect, and it works through the job bookmark question at the end: AWS Glue Explained: Data Catalog, Crawlers, ETL Jobs and More on YouTube.

Practice questions

AWS Certified Data Engineer – Associate (DEA-C01)100 questions with explanations
AWS Certified Cloud Practitioner (CLF-C02)130 questions with explanations
AWS Certified Solutions Architect – Associate (SAA-C03)100 questions with explanations