$ cat case-studies/high-scale-search.md
Building a High-Scale Search Orchestrator in Go
An architecture deep-dive into high-scale-search — a search orchestration service designed for 10B+ records at sub-200ms p99, unifying Elasticsearch, ClickHouse, Firestore, Redis, and Kafka behind a fault-tolerant pipeline.
Why build this
Most side projects that claim to be "high-scale" stop at a README description. The goal here was the opposite: pick one hard, narrow problem — serving search over a dataset too large and too varied for any single datastore to handle well — and implement the actual resilience machinery a real search platform needs, not just the happy path.
Architecture
The 5-level fallback chain
Search should never return a hard error. Instead of a single retry-then-fail path, every request falls through a graduated chain of degradation:
| Level | Source | When |
|---|---|---|
| 0 | Redis cache | Cache hit on query fingerprint |
| 1 | Elasticsearch (primary) | Normal operation, behind a circuit breaker + retry |
| 2 | Redis stale cache | Primary ES fails; serve slightly stale results |
| 3 | ClickHouse (degraded) | Both ES and cache unavailable; basic text search |
| 4 | Static popular results | All backends down; pre-loaded popular results |
Resilience mechanisms
- Circuit breaker — opens after 5 consecutive failures, half-opens after 30s to probe recovery
- Retry with exponential backoff — 2 attempts, 50ms initial wait, 2x multiplier, capped at 500ms
- Dead letter queue — failed Kafka messages route to a DLQ with error metadata for replay, instead of being dropped
- Graceful degradation — ClickHouse, Firestore, and Kafka are each optional; the service starts and serves search even if they're unavailable
- Rate limiting — token-bucket concurrency limiter capped at 1,000 in-flight requests
- Graceful shutdown — SIGINT/SIGTERM handling drains in-flight requests and flushes the indexing buffer before exit
Cache strategy
TTLs are tuned per query type rather than applied uniformly — high-churn data (trending) expires fast, stable data (autocomplete) is cached longer:
| Query type | TTL | Key pattern |
|---|---|---|
| Autocomplete | 10 min | ac:{prefix_hash} |
| Search results | 2 min | sr:{query_hash} |
| Facet counts | 5 min | fc:{category}:{filters_hash} |
| Trending | 60 sec | trend:{region} |
| Stale fallback | 1 hour | sr:stale:{query_hash} |
Observability
Every layer emits Prometheus metrics — request latency histograms by intent/source/status, cache hit/miss counters, per-backend query duration, circuit breaker state, indexing lag, and Kafka consumer lag. Queries over 200ms are logged as slow; over 500ms are marked critical and written to a ClickHouse query_performance table for trend analysis. OpenTelemetry traces carry a consistent trace_id across every backend hop, so a single slow request can be followed end-to-end instead of guessed at.
Elasticsearch index strategy
- Time-partitioned index naming:
search-{type}-{region}-{yyyy.MM} - Tiered storage — hot (NVMe, 3 months) → warm (SSD, 3–12 months) → cold (object storage)
- Minimal
_source— only searchable fields are indexed; full documents are hydrated from Firestore on read, keeping the index lean - Score blending:
_score * (1 + log1p(popularity_score))so relevance and popularity aren't fought over in separate passes
What I'd do differently
The fallback chain and circuit breaker were the most valuable parts to build for real rather than describe abstractly — the interesting decisions (backoff caps, half-open probe timing, what "degraded" should even mean for a given query type) only show up once you implement it. If I extended this further, the next real gap is load-testing the fallback transitions themselves — proving level 1→2→3 handoffs stay correct under actual concurrent failure, not just in isolation.