Databricks Generative AI Engineer Associate Practice Question 2: Chunk Size Embedding Limit
Databricks Certified Generative AI Engineer Associate · Test 1 · question 2 of 10
A RAG application over long engineering manuals splits each manual into 2,000-token chunks with no overlap. The chunks are embedded by a model whose maximum input length is 512 tokens. In testing, questions about content near the *start* of a section retrieve the right chunk, but questions about details that appear later in a section almost never retrieve it, even though that chunk contains the answer.
What change most directly addresses the root cause?
Show answer and explanation
Answer:
When a chunk is longer than the embedding model's maximum token limit, the input is truncated before it is embedded. Here, only the first ~512 tokens of each 2,000-token chunk are represented in its vector, so text later in the chunk cannot be found by similarity search. That is exactly the "start of section works, later details don't" pattern. Sizing chunks to the embedding model's limit, with modest overlap so facts near a boundary are not split apart, makes every part of the text searchable.
Option A treats the symptom. The missing details are not in any chunk's vector, so returning more results mostly adds noise to the prompt and raises cost and latency. It does not reliably surface the right chunk.
Option C changes the generator, but the failure happens earlier, at retrieval. A bigger context window does not help when the relevant chunk is never retrieved.
Option D keeps every chunk at 2,000 tokens, so each one is still truncated at 512 tokens when embedded. Heavy overlap also greatly increases the number of vectors to store and search.
Reference: Build an unstructured data pipeline for RAG