The short version

  • Redundancy keeps a system available. It does not prove the data is correct.
  • The dangerous failures are silent: every component reports success.
  • Rebuilds and re-indexing are when systems are most fragile.
  • Check integrity end to end, from the source to the answer.
  • Keep permissions attached to the data, all the way through.

Availability is not integrity

RAID protects against a disk failing. It does not protect against the wrong data being written, a controller corrupting blocks or a file being deleted, which is why RAID has never been a backup. The array stays healthy while the data inside it quietly goes wrong.

Retrieval systems for AI share the same blind spot. An index can be complete, fast and available while returning passages that are out of date, superseded or simply from the wrong document. The model then writes a fluent answer on top, and every component reports success.

The fragile moment is the rebuild

With large disks, rebuilding an array after a failure takes hours, and every remaining disk is read end to end. That is exactly when a second fault is most likely to surface. Storage engineers learned to plan for the rebuild, not just the failure.

Re-indexing is the retrieval equivalent. Changing the chunking, the embedding model or the source connectors can silently change what comes back. Treat a re-index like a rebuild: run it alongside the old index, compare results on a fixed set of questions, and switch only when the answers hold up.

Scrub the data, not just the hardware

Modern storage checks itself. Checksums and scheduled scrubbing find silent corruption before anyone needs the data. Retrieval needs the same habits:

  • Freshness checks that flag passages whose source has changed since they were indexed.
  • Duplicate and conflict detection, so two versions of the same policy never both answer.
  • A standing evaluation set: real questions with agreed answers, run after every change.
  • Visible sources on every answer, so a person can check where it came from.

Keep permissions with the data

A storage migration that drops file permissions is a breach waiting to happen. An index built with broader access than the people querying it is the same mistake in a new form. Trim results to what the person asking is allowed to see, at the moment they ask, using the source system's own permissions.

What we would do differently

Build the evaluation set before the index. Twenty real questions with agreed answers will tell you more about a retrieval system than any dashboard, in the same way a restore test tells you more than a backup report.

Building retrieval over your own documents?

Talk to an engineer