Persistence

Tachyon persists everything under a single --data-dir (TACHYON_DATA_DIR, default ./data). Two mechanisms work together: a write-ahead log for durability, and immutable, memory-mapped segments for the actual index.

Write-ahead log

Every write (upsert or delete) is appended to a write-ahead log before it's acknowledged. On startup, Tachyon replays the WAL to reconstruct any data that hadn't yet been flushed into a segment. If the log ends mid-record — say, a crash during a write — replay stops at the first incomplete or checksum-failing record and truncates there, rather than failing to start.

How often the WAL is fsynced to disk is controlled by --sync-interval-ms/TACHYON_SYNC_INTERVAL_MS:

• 0 (default) — fsync before acknowledging every write. Safest, slower under heavy write load.
• a positive value — fsync at most once per that many milliseconds, trading a bounded window of potential data loss for higher ingest throughput.

Segments

New documents accumulate in an in-memory memtable. Once it holds --max-memtable-docs/TACHYON_MAX_MEMTABLE_DOCS documents (default 100,000), it's flushed into a new immutable, memory-mapped segment on disk — postings, columnar field values, and document bodies are all read lazily and decoded only for what a given query actually touches, which is what keeps memory usage bounded by how much of the corpus is "hot" rather than by total corpus size.

A search fans out across the current memtable plus every committed segment, so segment count drives query latency up over time independent of corpus size. To bound that, Tachyon merges segments in the background: once a collection holds more than --merge-trigger-segments/TACHYON_MERGE_TRIGGER_SEGMENTS (default 8), the smallest --merge-fan-in/TACHYON_MERGE_FAN_IN (default 4) segments by document count are folded into one, right after the flush that crossed the threshold.

Recovery

Because the WAL is replayed on startup and segments are immutable once written, a crash mid-write loses at most the writes that hadn't yet been fsynced under the configured --sync-interval-ms — never a partially corrupted index.