DEA-C01 Practice Question 91: Glue Pushdown Predicate Partitions
AWS Certified Data Engineer – Associate (DEA-C01) · Test 10 · question 1 of 10
A daily AWS Glue job processes only yesterday's data from a Data Catalog table that is partitioned by year, month and day. The table has grown to several years of data, and the job now takes hours because it reads the whole dataset into a DynamicFrame and only then filters it. The team wants the job to list and read only the needed partitions. What should they do?
Show answer and explanation
Answer:
AWS explains that *"you can use a pushdown predicate to filter on partitions without having to list and read all the files in your dataset. Instead of reading the entire dataset and then filtering in a DynamicFrame, you can apply the filter directly on the partition metadata in the Data Catalog."* With create_dynamic_frame.from_catalog(..., push_down_predicate = ...), *"only the partitions in the Data Catalog that satisfy the predicate expression"* are loaded, which can save a great deal of processing time (option C).
- A is wrong because a Filter transform runs after the entire dataset has been listed and read, which is the behavior the team wants to avoid. - B is wrong because more workers speed up reading unneeded data but do not stop the job from reading it, so cost rises while the waste stays. - D is wrong because from_options reads a path directly and does not use the catalog's partition information. AWS notes that from_catalog is the call that includes partition columns.