AZ-305 Practice Question 94: Data Factory Incremental Load

Microsoft Azure Solutions Architect Expert (AZ-305) · Test 10 · question 4 of 10

Data Storage Solutionsmedium

A retailer runs a nightly Azure Data Factory pipeline that copies the entire 800-GB orders table from an on-premises SQL Server instance into Azure Data Lake Storage Gen2 for reporting. The table grows by only about 2 GB of new and updated rows per day, but the full copy now takes over five hours and regularly misses its overnight window. Every row is expected to have a reliably increasing LastModified column. You need to redesign the pipeline to greatly reduce runtime and data transferred each night without changing the reporting queries or losing any historical rows already loaded.

What should you include in the design?

Select one answer
Show answer and explanation

Answer:

  • D. Change the pipeline to a watermark-based incremental copy that reads only rows with LastModified greater than the last recorded watermark value and appends them to the existing data

A watermark-based incremental copy pattern in Azure Data Factory uses a lookup activity to read the last stored watermark, a copy activity that filters the source query to rows newer than that watermark, and a final activity that updates the stored watermark, so each run transfers only the rows that changed since the previous run instead of the full table. This directly reduces both runtime and data transferred while preserving all previously loaded history.

Option A is wrong because throwing more compute and parallelism at a full copy still moves the entire 800 GB every night; it reduces wall-clock time somewhat but not the underlying data transferred, and it does not address the growth trend. Option B is wrong because switching integration runtimes changes where the copy engine runs, not how much data the copy activity reads from the source. Option C is wrong because migrating the source database is a large, unrelated change that does not by itself change the pipeline from a full load to an incremental one, and introduces its own migration risk that the requirement does not ask for.

Reference: Incrementally copy data from a source data store to a destination data store

Practice Test 10 ›

Study guides for AZ-305