ETD

• Dense Retrieval
• IR
• Neural Information Retrieval
• NIR

Modern search systems increasingly rely on dense retrieval, but collections keep expanding while the cost of storing and scanning
high-dimensional representations (memory, latency, and energy) becomes harder to sustain. At the same time, corpora contain substantial
amounts of redundant or weakly informative passages that often contribute little to retrieval quality, yet can still matter for a subset
of queries. This creates a persistent effectiveness–efficiency trade-off: static pruning lowers cost but can harm coverage in the long tail,
where less frequent information needs still require access to otherwise low-utility content.

This thesis studies a quality-aware alternative that sits between indexing the entire collection in a single dense index and static pruning.
Each passage is assigned a scalar quality score (QualT5) and the corpus is partitioned into high- and low-quality pools via controlled,
percentile-based splits. Retrieval is organized as a selective pipeline: a first pass searches the high-quality pool, while a query-level
fallback selectively accesses the low-quality pool when needed. We compare two policies that instantiate this idea: (i) Two-Tier retrieval
with two physical FAISS IVF indexes and explicit fallback, and (ii) Virtual Partitioning, which uses a single index and enforces partition
membership through ID-based filtering and an optional second pass. Experiments on MSMARCO evaluate effectiveness (NDCG@10, MRR@10) alongside
efficiency metrics under systematic sweeps of the high-tier budget and ANN search effort (nprobe), and analyze how QualT5 score distributions
(tiny vs base) affect the resulting splits and the attainable trade-offs.

Overall, the thesis provides a reproducible assessment of when quality-based partitioning with selective fallback can reduce average retrieval
cost while preserving robust effectiveness, and how system design choices influence this trade-off.