Pierre Zemb's Blog

Building Index-Backed Query Plans in DataFusion

Pierre Zemb — Wed, 25 Feb 2026 00:00:00 +0000

When you build a system on top of a key-value store like FoundationDB, you eventually need secondary indexes. You create them, you maintain them, and then one day you need to query them. Not just scan a single index, but combine results from multiple indexes: intersect them for AND conditions, union them for OR conditions, and fetch the actual records at the end. That's a query engine's job. I didn't want to write a query engine. But I had to learn how one thinks.</p>

Last year</a>, I wrote about integrating DataFusion and mentioned a PoC library for index-backed queries. That PoC has grown into datafusion-index-provider</code></a>, a real library in datafusion-contrib</code> running in production. Building it meant learning how to construct physical query plans by hand, assembling them from existing DataFusion operators instead of writing execution logic from scratch.</p>

🔗</a>The PostgreSQL Pattern</h2>
Every database with secondary indexes follows the same two-phase pattern. Take SELECT * FROM employees WHERE age > 30</code>. PostgreSQL doesn't scan every row. It walks the B-tree index on age</code>, collecting TIDs</strong> (tuple identifiers) pointing to matching rows. Then it fetches the actual data using those TIDs. Find where, then fetch what.</strong></p>
For multi-index queries like WHERE age < 25 OR department = 'Sales'</code>, PostgreSQL uses BitmapIndexScan</strong>: each index produces a bitmap of matching TIDs, bitmaps get combined (OR for union, AND for intersection), and one pass fetches the results. No duplicates, no wasted reads.</p>
This is the pattern I wanted to bring to DataFusion. But DataFusion's existing index support (ParquetAccessPlan, zone maps) works at planning time</strong> on row groups</strong>. What I needed was OLTP-style indexes that resolve at execution time</strong> and produce specific row identifiers</strong>. The DataFusion community has discussed both approaches</a>. datafusion-index-provider</code> implements the OLTP path.</p>
🔗</a>Primary Keys as the Universal Glue</h2>
In PostgreSQL, TIDs connect everything. In my system, that role falls to the primary key schema</strong>. Every index declares an index_schema()</code> defining the columns that form the row's primary key. Could be a single id</code> column, could be a composite (tenant_id, employee_id)</code>. Every index scan produces batches of these primary key columns, every join operates on them, every record fetch consumes them. Because every operator in the pipeline agrees on what a "row identifier" looks like, you can wire up standard DataFusion joins, unions, and aggregations without any custom glue.</p>
🔗</a>From Filters to Execution Plans</h2>
The library first converts SQL filters into an intermediate IndexFilter</code> enum: Single</code> (one index handles one filter), And</code> (intersection), or Or</code> (union). Then it recursively builds the physical plan from that intermediate representation.</p>
The library introduces only two custom ExecutionPlan</code> nodes: IndexScanExec</code> (which wraps your index) and RecordFetchExec</code> (which wraps your storage). No custom join logic, no custom dedup, no custom union. Everything in between is standard DataFusion operators wired together.</p>
🔗</a>Single Index</h3>
SELECT </span>* </span>FROM</span> employees </span>WHERE</span> age > </span>29
</span></code></pre>
The simplest case needs just these two custom nodes wired together. IndexScanExec</code> calls index.scan(filters, limit)</code> and streams primary key batches. RecordFetchExec</code> consumes those batches and calls a RecordFetcher</code> to look up complete records.</p>

            flowchart BT
    A[IndexScanExec] --> B[RecordFetchExec]
    </pre>
</div>🔗</a>AND: Intersection Through Joins</h3>
SELECT </span>* </span>FROM</span> employees </span>WHERE</span> age > </span>25 </span>AND department = '</span>Engineering</span>'
</span></code></pre>
Both conditions must hold. Each index produces a separate stream of primary keys, and we need their intersection: only keys that appear in both streams. How do you compute an intersection of two streams? That's exactly what an INNER JOIN</strong> does when both sides share the same key columns.</p>

            flowchart BT
    A[IndexScanExec<br/>age index] --> C[HashJoinExec<br/>INNER on PK columns]
    B[IndexScanExec<br/>department index] --> C
    C --> P[ProjectionExec<br/>PK columns]
    P --> D[RecordFetchExec]
    </pre>
</div>
DataFusion ships two join implementations that we can pick from. HashJoinExec</strong> works in two phases: it reads the entire left (build) side into memory, constructs a hash table keyed on the join columns, then streams the right (probe) side through, looking up each row's key in that table. Matches produce output rows. Memory cost is proportional to the build side, but the probe side streams through with no buffering. The library uses PartitionMode::CollectLeft</code>, which collects the left input into a single partition before building the hash table.</p>
SortMergeJoinExec</strong> takes a different approach. When both inputs are already sorted on the join keys, it walks both streams in lockstep, comparing keys as it goes. When keys match, it buffers rows sharing that key value and outputs all combinations. For unique primary keys (the common case with index scans), this means constant memory: one row buffered from each side at a time. No hash table, no bulk memory allocation, just two cursors advancing together.</p>
How does the library choose? If both indexes report sorted output via is_ordered()</code>, it picks SortMergeJoin. Otherwise, HashJoin. For ordered key-value stores like FoundationDB, indexes naturally return sorted keys, so SortMergeJoin is the common path in practice.</p>
There's a wrinkle after the join. An inner join on column id</code> from both sides produces output with columns (id_left, id_right)</code>, but downstream operators expect just (id)</code>. A ProjectionExec</code> after the join strips the duplicates back to the primary key schema. This matters because when three or more indexes are involved, the library builds a left-deep join tree</strong>: join the first two, project back to the primary key schema, then join that result with the third, and so on. Each join progressively narrows the result set, and the projection keeps the schema clean between steps.</p>
🔗</a>OR: Union with Deduplication</h3>
SELECT </span>* </span>FROM</span> employees </span>WHERE</span> age < </span>25 </span>OR department = '</span>Sales</span>'
</span></code></pre>
A row matches if either condition is true, but a row satisfying both should appear exactly once. The plan needs to combine both index results and deduplicate before fetching.</p>

            flowchart BT
    A[IndexScanExec<br/>age index] --> C[UnionExec]
    B[IndexScanExec<br/>department index] --> C
    C --> D[AggregateExec<br/>GROUP BY PK columns]
    D --> E[RecordFetchExec]
    </pre>
</div>
Each index scan feeds into DataFusion's UnionExec</code>, which concatenates streams with zero-copy partition pass-through. But a row matching both conditions appears twice, once from each index. The deduplication step uses DataFusion's AggregateExec</code> with a GROUP BY</code> on all primary key columns. AggregateExec maintains a hash table mapping group key values to group indices. For pure dedup (no aggregate functions, just GROUP BY), it's essentially a hash set of seen primary keys. When memory pressure exceeds limits, it spills groups to disk in Arrow IPC format and merges them back later.</p>
Why not write a custom dedup node? Because AggregateExec</code> already handles hash-based grouping, memory tracking against DataFusion's memory pool, and spill-to-disk. Writing a custom dedup operator would mean reimplementing all of that. The library's philosophy is to construct a query plan that DataFusion already knows how to execute, not to reinvent execution primitives.</p>
🔗</a>Combining AND and OR</h3>
SELECT </span>* </span>FROM</span> employees
</span>WHERE</span> (age > </span>30 </span>AND department = '</span>Engineering</span>')
</span>   OR (age < </span>25 </span>AND department = '</span>Sales</span>')
</span></code></pre>
The IndexFilter</code> tree for this query is an Or</code> of two And</code> branches. Each And</code> branch becomes a join subtree (two IndexScanExec nodes joined on primary key columns), and the two subtrees feed into a UnionExec + AggregateExec for deduplication, just like a simple OR.</p>

            flowchart BT
    A1[IndexScanExec<br/>age > 30] --> J1[HashJoinExec<br/>INNER on PK]
    B1[IndexScanExec<br/>dept = Engineering] --> J1
    J1 --> P1[ProjectionExec]
    A2[IndexScanExec<br/>age < 25] --> J2[HashJoinExec<br/>INNER on PK]
    B2[IndexScanExec<br/>dept = Sales] --> J2
    J2 --> P2[ProjectionExec]
    P1 --> U[UnionExec]
    P2 --> U
    U --> AG[AggregateExec<br/>GROUP BY PK columns]
    AG --> RF[RecordFetchExec]
    </pre>
</div>
Joins for AND and union + dedup for OR compose naturally into nested plans. The library doesn't need special handling for nested expressions. It recurses down the IndexFilter</code> tree, builds the appropriate subtree for each node, and DataFusion executes the whole thing as one pipeline.</p>
🔗</a>Limitations and What's Next</h2>
The filter analysis has one important simplification: if any part of an AND/OR expression can't be handled by an index, the entire expression falls back to a regular scan. Consider WHERE age > 30 AND color = 'blue'</code> with an index on age</code> but not color</code>. A smarter approach would use the age index then scan-filter for color, but mixing index-backed and scan-based execution paths complicates plan construction, especially when AND/OR expressions are nested. For v1, the clean boundary keeps things correct. Partial index usage is on the roadmap, along with projection pushdown</strong> into the fetch phase and multi-partition execution</strong> for parallelism. Each is an opportunity for contribution</a>.</p>
🔗</a>Try It</h2>
If you're building a system that needs secondary index queries over your own storage, give datafusion-index-provider</code></a> a try. The tests directory</a> has reference implementations for both single-column and composite primary keys.</p>
This library only works because DataFusion's architecture is genuinely composable. Special thanks to Andrew Lamb</strong>, whose work on DataFusion and the architecture paper</a> has been instrumental. HashJoinExec, SortMergeJoinExec, AggregateExec, UnionExec, ProjectionExec: all ready to be wired up into whatever query plan your system needs. Do you have a custom storage layer that could benefit from secondary index queries?</p>

Feel free to reach out with any questions or to share your experiences with DataFusion. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>



Simulating Leader Election on top of FoundationDB
Pierre Zemb — Thu, 05 Feb 2026 00:00:00 +0000
People are right to fear LLM-generated code. The models hallucinate APIs, miss edge cases, and produce code that looks correct but fails under pressure. For distributed systems, where bugs hide behind race conditions and network partitions, the stakes are even higher. A subtle leader election bug can cause split-brain, data corruption, or cascading failures across your cluster.</p>
But what if you could give an LLM the right feedback loop</a>? Not just "write me leader election" but "write me leader election, and here's how I'll prove it correct." That feedback loop is simulation. I decided to test this idea by building a leader election recipe for foundationdb-rs</a>, with invariants generated by Claude Code.</p>
🔗</a>The Algorithm</h2>
The recipe uses a ballot-based approach, similar to Raft's term concept but simpler. The algorithm is based on "Leader Election Using NewSQL Database Systems"</a> (Ismail et al., DAIS 2015), which uses a database as distributed shared memory. Our implementation differs by leveraging FDB's strictly serializable transactions</a> and versionstamps instead of timestamps. Leaders hold time-bounded leases. Ballots are monotonically increasing fencing tokens that prevent stale leaders from corrupting state. FDB's serializable transactions handle mutual exclusion without explicit locking.</p>
The key insight is storing the leader state at a single key</strong>, making all operations O(1) instead of O(N) candidate scanning. Most leader election implementations require scanning all candidates to determine who's in charge. With N candidates, that's N reads per query. Our approach stores the current leader explicitly, so checking who's leader is a single read. Claiming leadership is a single write with a conflict check. No quorum is needed because FDB already provides the coordination guarantees we need through its serializable transactions.</p>

            sequenceDiagram
    participant A as Process A
    participant FDB as FoundationDB
    participant B as Process B

    A->>FDB: read leader key
    FDB-->>A: ballot=5, lease expired
    B->>FDB: read leader key
    FDB-->>B: ballot=5, lease expired

    A->>FDB: write ballot=6
    B->>FDB: write ballot=6

    FDB-->>A: COMMIT OK
    FDB-->>B: CONFLICT!

    Note over A: Becomes Leader
    Note over B: Retries, sees ballot=6, backs off
    </pre>
</div>
The diagram shows what happens when two processes try to claim leadership simultaneously. Both read the same expired leader state. Both try to write ballot=6. FDB's serializable transactions guarantee only one succeeds. The loser gets a conflict error, retries, sees ballot=6 with an active lease, and backs off. No split-brain possible. This is the fundamental safety property that the entire algorithm rests on.</p>
When a process wants to become leader, it first registers as a candidate. FDB assigns a versionstamp at commit time using the SetVersionstampedValue</code> atomic operation. This versionstamp is assigned by FDB itself when the transaction commits, providing a globally-ordered identity that no other process can have. The versionstamp never changes throughout the process's lifetime, even as the process sends heartbeats and refreshes its registration. This immutability is important: it means the process's identity is stable and can be used reliably in log replay during verification.</p>
Then the process enters a loop: send heartbeats to prove liveness, try to claim leadership if the current lease expired, and optionally resign if graceful handoff is needed. Each leadership claim increments the ballot number. The new leader stores its versionstamp in the leader state, so followers can verify they're talking to the actual leader. The full implementation lives in the foundationdb-rs recipes module</a>.</p>
🔗</a>The Simulation Workload</h2>
Simulation on top of FoundationDB may feel redundant. FDB itself has been validated through a trillion CPU-hours of simulation</a>. But FDB's resiliency doesn't mean your code is resilient. Your layer sits on top of FDB. Your transaction logic, your conflict handling, your retry behavior. How does your code respond when FDB is unhealthy?</p>
Simulation introduces chaos to answer that question. Network partitions, process crashes, clock skew up to ±1 second. Same seed, same execution, same bugs. Deterministic replay.</p>
Phase</th> Who</th> What</th></tr></thead>

Setup</strong></td> Client 0</td> Initialize election, register all candidates</td></tr>
Chaos</strong></td> All clients</td> Loop: heartbeat → try claim → maybe resign (10%)</td></tr>
Check</strong></td> Client 0</td> Read logs, snapshot state, run 13 invariants</td></tr>
</tbody></table>
FDB's simulator runs built-in chaos workloads alongside ours: RandomClogging</strong> injects network partitions, Attrition</strong> kills and reboots processes. Our workload adds clock skew simulation up to ±1 second.</p>
Each operation logs its intent atomically in the same transaction, using the pattern from FDB's own AtomicOps workload</a>. The SetVersionstampedKey</code> atomic operation</a> writes both the leader election mutation and its log entry in a single transaction. If the transaction commits, both succeed. If it aborts, neither does. This gives us a proper write-ahead log without the complexity of two-phase commit. The log key uses a versionstamp prefix for true FDB commit ordering. No clock skew ambiguity.</p>
🔗</a>How Atomic Logging Works</h3>
Each operation writes a log entry in the same transaction as the operation itself. The log key uses a versionstamp prefix, so entries sort by true FDB commit order, not by wall clock (which can drift ±1 second under simulation).</p>
Log Key Structure:
</span>┌─────────────────────┬───────────┬────────┐
</span>│ versionstamp (10B)  │ client_id │ op_num │
</span>└─────────────────────┴───────────┴────────┘
</span>         ↑
</span>   Assigned by FDB at commit time
</span></code></pre>
Here's what a typical log might look like after chaos:</p>
Versionstamp</th> Client</th> Op</th> Result</th></tr></thead>

0x0001...</td> 0</td> Register</td> ✓</td></tr>
0x0002...</td> 1</td> Register</td> ✓</td></tr>
0x0003...</td> 2</td> Register</td> ✓</td></tr>
0x0004...</td> 0</td> TryBecomeLeader</td> ✓ became leader, ballot=1</td></tr>
0x0005...</td> 1</td> TryBecomeLeader</td> ✗ conflict</td></tr>
0x0006...</td> 0</td> Heartbeat</td> ✓</td></tr>
0x0007...</td> 2</td> TryBecomeLeader</td> ✓ became leader, ballot=2</td></tr>
0x0008...</td> 2</td> Resign</td> ✓</td></tr>
</tbody></table>
Client 0 became leader with ballot 1. Client 1's claim failed (conflict). Client 2 later claimed ballot 2 when client 0's lease expired, then resigned. The versionstamp ordering is ground truth: no matter how skewed each client's clock was, the log shows exactly what committed and in what order.</p>
During check</strong>, client 0 reads all logs and database state in a single snapshot, then runs the invariants. Same seed, same execution, same bugs. Deterministic replay.</p>
🔗</a>The Invariants</h2>
Here's where Claude Code enters the story. I asked it to generate invariants for leader election validation. I didn't give it a detailed specification. I pointed it at my previous post about designing workloads that find bugs</a> and said "apply these patterns to leader election."</p>
It generated 13 invariants</a>. Here are the seven most important ones:</p>
Invariant</th> What</th> Why</th> How</th></tr></thead>

DualPathValidation</strong></td> Expected state from log replay must match actual FDB snapshot</td> The keystone check. If logs say process A is leader with ballot 7, but FDB shows process B with ballot 6, something corrupted state</td> Replay all logged operations in versionstamp order to compute expected leader/ballot, then compare with actual database state</td></tr>
FencingTokenMonotonicity</strong></td> Each successful claim must have a strictly higher ballot than the previous</td> When a network partition heals, an old leader might try to act with a stale ballot. This catches that write</td> For each claim in the log, verify ballot > previous_ballot from the same transaction</td></tr>
OneValuePerBallot</strong></td> Each ballot number maps to exactly one client</td> Two clients claiming the same ballot means either conflict detection failed or ballot increment is broken. Classic split-brain symptom</td> Scan all claims, verify no two different clients ever claimed the same ballot number</td></tr>
LeaderIsCandidate</strong></td> Current leader must exist in the candidates registry</td> Edge cases like crash-during-registration or eviction-while-claiming can leave orphaned leaders</td> Read leader state, verify a matching candidate entry exists</td></tr>
NoOverlappingLeadership</strong></td> Every leadership claim has a globally unique versionstamp</td> FDB serializes commits globally. Duplicate versionstamps mean either a logging bug or actual split-brain</td> Collect all claim versionstamps, verify no duplicates</td></tr>
GlobalBallotSuccession</strong></td> Each new leader must have ballot > previous leader's ballot</td> Catches state regression after partition heals. An old leader can't "go back" to a stale ballot</td> Track previous_ballot in log entries, verify new_ballot > previous_ballot for every transition</td></tr>
LeaseExpiryAfterClaim</strong></td> Lease must expire after the claim timestamp</td> Clock skew can cause a leader to claim with an already-expired lease. This catches incorrect lease calculation or extreme clock drift</td> For each claim, verify lease_expiry_nanos > claim_timestamp_nanos</td></tr>
</tbody></table>
🔗</a>What's Next</h2>
The leader election recipe has just been merged into foundationdb-rs as experimental. I haven't run it in production yet. The entire development was simulation-driven: write code, run simulation, fix what breaks, repeat. The simulation runs in CI on every PR.</p>
Simulation gives us guarantees that the code behaves correctly under the right rules, even when FDB is tortured. Network partitions, clock skew, process crashes. The invariants verify that safety properties hold through all of it. Not a proof of correctness, but confidence earned through chaos.</p>
The invariants that Claude Code generated encode patterns from prior simulation work: dual-path validation from AtomicOps, fencing tokens from the literature, lease checks tied to clock skew. What the LLM provided was speed: weeks of invariant development compressed into hours of review. The LLM proposes, simulation disposes.</p>
If you have ideas for new tortures or invariants, the simulation code</a> is open. Merge requests welcome.</p>

Feel free to reach out with questions or to share your experiences with leader election and simulation testing. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


FoundationDB's Transaction Model for Layer Engineers
Pierre Zemb — Fri, 30 Jan 2026 00:00:00 +0000
FoundationDB gives you serializable transactions with external consistency, automatic sharding, and fault tolerance. But once your first layer hits production under real load, you start seeing transaction conflicts you don't understand. The logic looks correct: read a key, check a condition, write the result. Under load, conflicts pile up and throughput collapses.</p>
🔗</a>How OCC Works</h2>
FoundationDB implements these guarantees using Optimistic Concurrency Control</strong> (OCC). Your transaction runs without holding any locks. It reads from a consistent snapshot, does its work, and at commit time the system checks whether anything you read was modified by another transaction since you started. If yes, your transaction is aborted and retried. If no, it commits atomically.</p>
All writes are buffered locally in the client until commit. Nothing goes to the cluster while your transaction is running. At commit time, the client sends the buffered writes and the read/write conflict sets to the Resolver in a single request. A read-only transaction that calls commit is mostly a no-op: the network thread checks there are no writes to send and skips the round-trip.</p>
No locks means no waiting, but it also means your reads and writes play different roles in conflict detection. Your reads determine whether YOU can conflict. Your writes determine what OTHER transactions will conflict with.</strong></p>
A read-only transaction never conflicts. It observes a snapshot and goes away. A write-only transaction also never conflicts. It blindly sets keys and commits. Only transactions that both read and write can fail. When they do, your writes don't cause your conflicts. Your reads do. The writes cause problems for future transactions, but your transaction was doomed the moment you issued reads on keys that someone else was modifying. Every time you add a read to a transaction, ask yourself: do I actually need to conflict on this?</strong></p>
🔗</a>Read Version, Commit Version, and the Window of Vulnerability</h2>
When your transaction starts, it obtains a read version</strong> from the cluster. All your reads see a consistent snapshot frozen at that version. When you commit, your transaction gets a commit version</strong>, guaranteed to be higher. Between these two versions lies what I call the Window of Vulnerability</strong>: any key you read that was modified by another committed transaction within this window will cause your transaction to abort.</p>
read version                                          commit version
</span>     │                                                      │
</span>     ▼                                                      ▼
</span>─────┼──────────────────────────────────────────────────────┼────── time
</span>     │              Window of Vulnerability                 │
</span>     │◄────────────────────────────────────────────────────►│
</span>     │                                                      │
</span>     │   your reads see          other transactions         │
</span>     │   a frozen snapshot       may commit writes here     │
</span></code></pre>
The longer your transaction runs, the wider this window grows. FoundationDB enforces a strict 5-second transaction limit</strong>, which is exactly 5 million versions</strong> (MAX_WRITE_TRANSACTION_LIFE_VERSIONS = 5 * VERSIONS_PER_SECOND</code>). The Resolver tracks conflict history in memory up to this age; transactions older than currentVersion - 5,000,000</code> are rejected as "transaction too old."</p>
A transaction that completes in 50 milliseconds has almost no exposure. A transaction that takes 4.5 seconds is exposed to every concurrent write on every key it read.</p>
This is why long transactions are one of the most common sources of production trouble. More work means more time, wider window, more conflicts. The fix is parallelizing your reads so the transaction completes faster, splitting work into smaller transactions when full atomicity isn't required, or using continuations</a> to checkpoint progress across transaction boundaries.</p>
🔗</a>How Conflicts Actually Work</h2>
Every read your transaction performs adds a read-conflict range</strong> to your transaction. Every write adds a write-conflict range</strong>. At commit time, the Resolver checks: does your read-conflict set intersect any committed write-conflict set since your read version? If yes, your transaction is aborted. The Resolver uses a version-aware skiplist to make this check efficient, pruning entire subtrees of committed writes that predate your read version.</p>
 Your Transaction                Resolver               Another Transaction
</span>┌─────────────────┐                                    ┌─────────────────┐
</span>│ get(key_A)      │─► read conflict: {key_A}           │                 │
</span>│ get_range(B, D) │─► read conflict: {B..D}            │ set(key_C)      │─► write conflict: {key_C}
</span>│ set(key_X)      │─► write conflict: {key_X}          │                 │
</span>└─────────────────┘                                    └─────────────────┘
</span>                              │
</span>                     at commit time:
</span>                     read conflicts ∩ write conflicts
</span>                     from txns committed since read version?
</span>                              │
</span>                     key_C ∈ {B..D}? → YES → ABORT
</span></code></pre>
get</code> and get_range</code> create read conflicts. set</code>, clear</code>, and clear_range</code> create write conflicts.</p>
The simplest conflict pattern is the hot key</strong>: a single key read and written by many concurrent transactions. A naive global counter, a "last updated" timestamp, a configuration value everyone checks. The read-modify-write creates a read conflict, and under concurrent updates, all but one transaction fails.</p>
🔗</a>The Phantom Conflict Problem</h2>
When you call get(key)</code>, FDB adds that single key to your read conflict set. Straightforward. But when you call get_range(start, end)</code>, FDB adds the entire range</strong> to your conflict set, not just the keys that happened to exist, not just the keys your code iterated over. The mathematical range from start to end, including every possible key that could exist within it. The SIGMOD 2021 paper calls this phantom read prevention</strong>: "The read set is checked against the modified key ranges of concurrent committed transactions, which prevents phantom reads." You can conflict on keys you never saw.</strong></p>
Your range read: get_range("order/user1/", "order/user1/\xff")
</span>
</span>Keyspace:
</span>  order/user1/001  ◄── exists, returned
</span>  order/user1/002  ◄── exists, returned
</span>  order/user1/003  ◄── exists, returned
</span>  order/user1/004  ◄── DOES NOT EXIST YET
</span>  ···
</span>
</span>Read conflict range: [ "order/user1/" , "order/user1/\xff" )
</span>                       ◄──────── covers EVERYTHING ────────►
</span>
</span>Another transaction: set("order/user1/004", ...)
</span>  → write conflict on "order/user1/004"
</span>  → inside your read conflict range
</span>  → YOUR transaction aborts (you never saw this key)
</span></code></pre>
Imagine you're scanning a user's orders to check if they have any pending shipments. Your range read returns 3 orders. You check each one, they're all shipped, great. You decide to update a status flag. Meanwhile, another transaction inserts a brand new order for that same user. The key for that new order falls within your scanned range. Your transaction conflicts and aborts, even though you never touched that key, never saw it, and your business logic doesn't care about it at all. Full table scans are the extreme version of this problem: the wider your range, the more phantom writes can abort you. The fix requires either narrowing your reads to touch less keyspace, or using snapshot reads with selective conflicts.</p>
🔗</a>Snapshot Reads</h2>
A snapshot read returns the same data as a regular read from the same consistent snapshot, but it does not add any read conflicts to your transaction. The operation is tr.snapshot().get(key)</code> or tr.snapshot().get_range(start, end)</code>. The data you get back is identical. The only difference is what happens at commit time: the Resolver won't check whether those keys changed.</p>
When would you want this? Whenever you need to read data for your logic but don't need the transaction to abort if that data changes concurrently. A common case is reading configuration or metadata that rarely changes and where a slightly stale value is acceptable within the transaction's own snapshot.</p>
The trade-off is that you're accepting your decision might be based on data that changed concurrently. This is safe for read-mostly metadata or filtering logic. It's dangerous for business-critical checks like balance verification or uniqueness constraints. If your code path is "read X, decide based on X, write Y", and the decision must hold at commit time, you need the read conflict.</p>
But what if you need to read broadly and conflict narrowly? FDB exposes manual conflict APIs</strong> that complement snapshot reads. add_read_conflict_key</code> and add_read_conflict_range</code> let you inject read conflicts explicitly: you read without conflicts, then selectively add conflicts on exactly the keys you care about. On the write side, add_write_conflict_key</code> and add_write_conflict_range</code> let you inject write conflicts without actually writing data. This is useful for implementing locks or coordination primitives where your transaction claims a key to block others without storing anything there.</p>
Go back to the phantom conflict problem: you need to scan a user's orders to check for pending shipments, but you don't want inserts of new orders to abort your transaction. With a regular get_range</code>, any write within that range kills you. With a snapshot range read plus manual conflicts, you read the entire range via tr.snapshot().get_range()</code> without adding any read conflicts, then call add_read_conflict_key</code> only on the specific pending orders your logic depends on. If another transaction inserts a new order, your transaction doesn't care. If another transaction modifies a pending order you're acting on, your transaction correctly conflicts. You went from conflicting on the entire keyspace range to conflicting on exactly the keys that matter.</p>
🔗</a>Atomic Operations: Writing Without Reading</h2>
When you need to increment a counter, the obvious approach is to read the current value, add one, and write the result back. This creates a read conflict on that key, and under concurrent updates, transactions start failing because they all race to write their incremented value. Atomic operations</strong> take a different approach: they send an instruction to the storage server ("add this delta to whatever value is there") without your transaction ever knowing the current value. No read, no read conflict. Instead of tr.get(key)</code> followed by tr.set(key, value + 1)</code>, you call tr.atomic_add(key, 1)</code> and concurrent updates all succeed.</p>
The Record Layer</a> exploits this for aggregate indexes. A COUNT</code> index issues atomic_add(count_key, 1)</code> on every record insertion and atomic_add(count_key, -1)</code> on deletion. A SUM</code> index adds the field's value. MAX_EVER</code> and MIN_EVER</code> use atomic_max</code> and atomic_min</code>. Unlimited concurrent updates to the same aggregate, zero conflicts between writers.</p>
But there's a trap: if you read a key and also atomically modify it in the same transaction, you lose all the benefits. The FoundationDB documentation is explicit: "If a transaction uses both an atomic operation and a strictly serializable read on the same key, the benefits of using the atomic operation (for both conflict checking and performance) are lost." The read already poisoned the transaction. The pattern only works when you genuinely don't need to see the current value.</p>
🔗</a>Versionstamps: Conflict-Free Ordering</h2>
Generating sequential IDs the obvious way means reading the current maximum, incrementing it, and writing the new value. That's a read-modify-write on a single key, which is exactly the conflict pattern we've been trying to avoid. Every concurrent transaction reads the same max ID, and all but one will abort.</p>
Versionstamps</strong> solve this by deferring ID assignment to commit time. Instead of your transaction deciding what the next ID is, FoundationDB fills it in at the moment of commit. A versionstamp is a 12-byte value</strong>: 8 bytes of commit version (assigned by the Sequencer), 2 bytes of batch ordering, and 2 bytes of user version. The result is globally unique and monotonically increasing across the entire cluster. You write a key containing a placeholder that FDB replaces with the actual versionstamp at commit. Your transaction doesn't know the final key until it commits, but multiple concurrent appends generate different versionstamps and write to different keys. Zero conflicts. Versionstamps also spread writes across shards, avoiding the hot spots that monotonic keys create. For more key design patterns, see crafting keys in FoundationDB</a>.</p>
The Record Layer uses this for its VERSION</code> index, which powers CloudKit's sync protocol. Each record stores its commit version, and a secondary index maps versions to primary keys. When a mobile device syncs, it scans the version index starting from its last-known version. Writers don't coordinate at all.</p>
One limitation: you cannot read a versionstamped key within the same transaction that creates it. The final key doesn't exist until commit. Versionstamps work beautifully for append-only structures where you write and walk away.</p>
🔗</a>Cross-Cluster Ordering</h3>
Versionstamps work within a single cluster. But what happens when data moves between clusters? Versions assigned by different FoundationDB clusters are uncorrelated. This creates a problem when migrating data between clusters for load balancing or locality. A sync index based purely on versionstamps would break: updates committed after the move might sort before updates committed before the move.</p>
The Record Layer solves this with an incarnation</strong> counter. Each user starts with incarnation 1, incremented every time their data moves to a different cluster. On every record update, the current incarnation is written to the record's header. The VERSION sync index maps (incarnation, version)</code> pairs to changed records, sorting first by incarnation, then by version. Updates after a move have a higher incarnation and correctly sort after pre-move updates, even if the new cluster's version numbers are lower.</p>
🔗</a>Conclusion</h2>
The next time you see a conflict error, ask yourself: what did I read that I didn't need to? The answer is usually hiding in a range read that could have been narrower, a read-modify-write that could have been an atomic operation, or a check that could have used a snapshot read.</p>
None of these techniques require changing FoundationDB itself. They're all about how you design your key schema and structure your transactions. As always, data-modeling in ordered key-value stores is the hard part of the job. What's the most surprising conflict you've debugged?</p>

Feel free to reach out with any questions or to share your experiences with FDB transaction debugging. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


What I Tell Colleagues About Using LLMs for Engineering
Pierre Zemb — Thu, 15 Jan 2026 00:00:00 +0000
In a few months, I went from skeptic to heavy user. Claude Code is now part of my daily workflow, both for my personal projects and at Clever Cloud where I help teams adopt these tools. I keep having the same conversation: colleagues ask how I use it, what works, what doesn't. This post captures what I tell them.</p>
The shift matters because code is becoming cheap</a>. What used to take hours now takes minutes. But this doesn't diminish the craft.</p>
Tobi Lütke</a> got MRI data on a USB stick that required commercial Windows software to view. He asked Claude to build an HTML viewer instead. It looked better than the commercial tool. That's the superpower: not just using software, but making</strong> software for your exact problem. As antirez wrote</a>, the fire that kept us coding until night was never about typing. It was about building. LLMs let us build more and better. The fun is still there, untouched.</p>
The first months were honestly frustrating. Code that looked right but broke in subtle ways. APIs that no longer existed. Patterns that didn't match my codebase. It took experimentation to find what actually works. These are the patterns that survived.</p>
🔗</a>What Becomes Reachable</h2>
I keep hearing that LLMs unlock velocity. We can ship faster! While that may be true, I think it misses the main benefit. LLMs are about reaching work that would never get done otherwise.</p>
Every engineering team has a backlog of things that matter but never happen: comprehensive doc tests, database migrations, dependency updates, technical debt. These tasks sit in dream lists because the return on investment is too low given the effort required. LLMs change that equation.</p>
moonpool</a> is my concrete example. Backporting FoundationDB's low-level internals to Rust was always a dream project. I had operated distributed systems for years and understood the concepts, but the sheer volume of translation work kept it out of reach. I could throw multiple codebases at Claude for analysis, create recap files summarizing key patterns, and nourish my own implementation plan in hours instead of days. The project exists because LLMs made it reachable.</p>
This is the shift worth paying attention to: LLMs amplify expertise, they do not replace it. The knowledge of what to build and why remains the bottleneck. The execution barrier just got lower.</p>
🔗</a>Plan First, Always</h2>
Here is the paradox: when code becomes cheap, design becomes more valuable. Not less. You can now afford to spend time on architecture, discuss tradeoffs, commit to an approach before writing a single line of code. Specs are coming back</a>, and the judgment to write good ones still requires years of building systems.</p>
Every significant task now starts in Plan Mode with ultrathink</code>. Boris Cherny says thinking is on by default now</a> and the command does not do much anymore, but old habits die hard. The practical goal is breaking work into chunks small enough that the AI can digest the context without hallucinating. This is not about limiting ambition. It is about matching task scope to context window.</p>
For large tasks, I produce a spec file</strong> that Claude and I iterate on together. Claude Code has an AskUserQuestion</code> tool that lets Claude ask clarifying questions mid-task. Combined with a spec file, this becomes powerful: Claude asks about edge cases, I refine the requirements, we converge on an approach before writing code. The collaboration happens in the spec, not scattered across conversation turns. As a bonus, the spec survives context compaction and remains the source of truth when Claude summarizes the conversation.</p>
Instead of generating a spec from assumptions, I tell Claude to clarify first. Here is an example prompt I use:</p>

ultrathink. Generate a spec.md for adding a new API endpoint to this codebase. Before writing anything, ask me about the endpoint's purpose, request/response schema, authentication requirements, and edge cases. Then produce a comprehensive spec covering motivation, technical design, error handling, and testing strategy.</p>
</blockquote>
The result is a spec that matches what I actually need, not what Claude guessed I might want. Fewer iterations, better alignment.</p>
A plan is only as good as the context it is built on.</p>
🔗</a>Context is Everything</h2>
The output quality depends entirely on the context you provide. This sounds obvious, but the implications took me a while to internalize. I now create context files with domain knowledge, code patterns, and project summaries. Writing down the hidden coding style rules that exist only in your head is surprisingly valuable. The conventions you enforce in code review but never documented? Write them down. The LLM will follow them, and so will newcomers on your team. I am currently experimenting with Claude skills to make this context reusable across sessions.</p>
The difference between vibe coding and vibe engineering</a>, as Simon Willison puts it, is whether you stay accountable for what the LLM produces. Accountability requires understanding, and understanding requires context.</p>
Without enough context framing the problem, Claude over-engineers. I have seen it add abstraction layers, configuration options, and patterns I never asked for. The cure is constraints: explicit context about what simplicity looks like in this codebase. The LLM can generate code faster than I ever could, but knowing what context matters is expertise that cannot be delegated.</p>
Context works when it is accurate. Documentation often is not.</p>
🔗</a>Clone Your Dependencies</h2>
MCP tools exist to fetch documentation, but I find git clone more powerful. I clone the dependencies I care about and checkout the version I actually use</strong>. Claude browses the real source code, not cached docs or outdated training data. When I ask about an API, the answer comes from the actual implementation in my lock file. This simple habit prevents entire categories of frustrating debugging sessions where the model confidently generates code for an API that no longer exists.</p>
This also works for understanding unfamiliar code</strong>. Clone a dependency, check out the version you use, and ask specific questions. The LLM handles breadth, you handle depth.</p>
Good context helps Claude generate better code. But how does it know when the code is wrong?</p>
🔗</a>Feedback Loops</h2>
Boris Cherny</a>, creator of Claude Code, calls this the most important thing: give Claude a way to verify its work</strong>. If Claude has that feedback loop, it will 2-3x the quality of the final result. The pattern is simple: generate code, get feedback, fix, repeat. The faster and clearer the feedback, the better the results.</p>
This is why Rust and Claude work so well together. The compiler gives actionable error messages</strong>. The type system catches bugs before runtime. Clippy suggests improvements. Claude reads the feedback and fixes issues immediately. The compiler output is isolated, textual, actionable. The model does not have to guess what went wrong. Any language or tool that provides clear, structured feedback enables this same cycle.</p>
TDD</strong> fits perfectly here. Tests are easy for you to read and verify, and they give fast feedback to the LLM. Write the test first, let Claude implement until it passes. You stay in control of the specification while delegating the implementation.</p>
For software that needs to be correct, the feedback must be exhaustive. I maintain the FoundationDB Rust crate</a>. Over 11 million downloads, used by real companies. The binding tester</a> generates operation sequences and compares our implementation against the reference. We run the equivalent of 219 days of continuous testing each month</strong> across our CI runners. When Claude contributes code, the binding tester tells it exactly where behavior diverges. This kind of feedback gives confidence to change things in a database driver that you would never touch otherwise.</p>
🔗</a>Simulation: Feedback for Distributed Systems</h3>
Compiler feedback catches syntax and types. Tests catch logic errors. But what about bugs that hide in timing and network partitions?</p>
Distributed systems fail in ways that only manifest under specific conditions. A network partition once disrupted a 70-node Hadoop cluster</a> and left it unable to restart due to corrupted state. That incident shaped how I think about testing. This is why I love FoundationDB: after years of on-call, it has never woken me up</a>.</p>
Distributed systems need feedback loops that inject failures before</strong> production. This is what deterministic simulation</a> provides. Same seed, same execution, same bugs. When every run is reproducible, the LLM can methodically explore the state space, find a failure, and debug it step by step.</p>
In moonpool, Claude discovered a bug I did not know existed</a> through active exploration of edge cases I had not considered. Armin Ronacher</a> recently noted that agents can now port entire codebases to new languages with all tests passing. The combination of simulation and LLMs makes this possible.</p>
The most awful bugs are the unknown unknowns</strong>. You cannot write a test for a bug you do not know exists. Simulation and state exploration are the cheatsheet. If the code survives exhaustive exploration of edge cases, failures, and adversarial conditions, it behaves correctly. It does not matter whether an LLM wrote it or you did.</p>
What dream project has been sitting on your list, waiting for the execution barrier to drop?</p>

Feel free to reach out with any questions or to share your experiences with LLM-assisted development. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


2025: A Year in Review
Pierre Zemb — Fri, 26 Dec 2025 00:00:00 +0000
2025 was the year I stopped managing and started shipping software again. After nearly two years of context-switching between fires and people issues, I returned to the keyboard. As the year closes, it feels like the right time to look back. It was about going deeper</strong>: into code, into writing, and into understanding why simulation testing is a superpower.</p>
🔗</a>Back in Engineering</h2>
🔗</a>The Transition</h3>
In January, I went back to engineering</a> after nearly two years in management. It felt like coming home.</p>
But I will be honest: the transition was harder than I expected. There was real imposter syndrome. Had I lost my edge? Was I still the technical person I used to be? The hardest part was not the code itself but giving myself permission to focus</strong>. Having another engineering manager handle the team and another SRE team handling the fires while I dove into low-level work was the perfect setup. I am also grateful</strong> to the whole team for making that transition possible. But after three years of context-switching between fires and people issues, sitting down to write code without interruption felt almost wrong. It took months to fully allow myself to focus without afterthoughts.</p>
🔗</a>Building the Toolbox Behind Materia, the Long Way Around</h3>
Helping put Clever Cloud's etcd shim</a> into production was meaningful because of the long arc behind it.</p>
At OVHcloud, I got paged too often when etcd hit its performance ceiling. We were adding hundreds of customers per etcd cluster, each with their own Kubernetes control plane. I talked about this at KubeCon</a>. Spawning three etcd nodes per customer is not a valid approach at scale, whether in the cloud or on-premise. You need to mutualize, but you cannot scale etcd horizontally because the whole keyspace must fit on every member and the recommended storage limit is 8GB</strong>. When you outgrow one cluster, you boot another, split your keys, and now you operate two, three, or many clusters. We had to balance customers manually across clusters. After contributing a custom balancer to HBase</a> to solve exactly this problem, doing it by hand for etcd felt like going backward.</p>
Then I discovered Apple's FDB Record Layer</a>. It was an eye-opener: here was a way to virtualize database-like systems</strong> on top of FoundationDB, building any storage abstraction you want on a rock-solid distributed foundation. During France's first lockdown, I prototyped an etcd layer</a> using the Record Layer. The prototype worked, but more importantly, the Record Layer showed me what was important: a reusable toolbox to encapsulate FoundationDB knowledge</strong>.</p>
I moved to Clever Cloud to build exactly that: serverless systems based on FoundationDB. We started building the toolbox in Rust, piece by piece, driven by what our layers actually needed. Our first layer was Materia KV</a>, exposing the Redis protocol. That forced us to build the foundational primitives.</p>
One highlight was building the query engine for Materia. I wrote datafusion-index-provider</a>, a library that extends Apache DataFusion with index-based query acceleration. I had a lot of fun digging into how a query plan might look when fetching indexes: a two-phase model where you first scan indexes to identify matching row IDs, then fetch complete records. The interesting part was combining AND</strong> and OR</strong> operations. AND predicates build a left-deep tree of joins to intersect row IDs across indexes. OR predicates use unions with deduplication to merge results without fetching the same record twice. The first time DataFusion, FoundationDB, and our indexes all connected and a SELECT query returned real data, I remembered why I write software</a>.</p>
Then came etcd, which required a lot</strong> more work: watches, leases, revision tracking. I debugged the watch cache</a> along the way. We are not alone in this approach: AWS</a> and GKE</a> also run custom storage layers for Kubernetes at scale. No more splitting clusters, no more manual data balancing, no more operational nightmares. FoundationDB handles the hard distributed systems parts. Years of frustration with etcd turned into an etcd-compatible API backing Kubernetes control planes.</p>
🔗</a>The Simulation Year</h2>
🔗</a>The Awakening</h3>
Then came BugBash 2025</a> in early April.</p>
The conference in Washington D.C., organized by Antithesis, brought together people like Kyle Kingsbury, Ankush Desai, and Mitchell Hashimoto to discuss software reliability. The highlight was meeting some of the original FoundationDB creators. Hearing their war stories and seeing how deeply simulation shaped FDB's legendary reliability reignited something in me. I had been using FDB's simulation for years, but I had never fully internalized that this could be how I write all software</a></strong>.</p>
BugBash reminded me of my painful on-call years at OVHcloud. Before etcd, I operated a massive HBase cluster: 255 machines, 2 million writes per second, 6 million reads. HBase was weak to network issues, and every incident triggered region split inconsistencies. We ran hbck in brutal ways just to keep things running. HBase led me to FDB, a system built to handle network chaos</strong>, and that robustness comes from simulation.</p>
🔗</a>From Understanding to Building</h3>
2025 was about turning that understanding into something real. At work, we added simulation workloads to critical software running on top of Materia. I spoke at Devoxx France</a> about embracing simulation-driven development. But I wanted to go deeper: to understand how FoundationDB created its robustness. So I dug into the implementation</a>, then started backporting it to Rust.</p>
That became moonpool</a>.</p>
I split moonpool into four crates, each a foundational block for distributed systems. moonpool-core</a></strong> defines provider traits for time, networking, task spawning, and randomness. Same code runs in production and simulation. moonpool-sim</a></strong> is the simulation engine: virtual time, event queues, and fault injection. moonpool-transport</a></strong> provides FDB-style RPC: connection management, checksummed wire formats, automatic reconnection. Claude accelerated this work significantly. I used it to translate C++ patterns into Rust. At some point, Claude started fixing bugs on its own</a>.</p>
At the beginning of 2025, I had basic knowledge of deterministic simulation. By the end, I had built a framework. In 2026, I will start a blogpost series</strong> explaining moonpool's internals. Maybe it becomes something others can use. Maybe it stays a hobby project. Either way, I am convinced: simulation is the future</strong>.</p>
🔗</a>Sharing</h2>
I set a goal to write one or two posts per month, and I published 20. You can trace my monthly focus just by looking at what I published. The results surprised me: traffic multiplied by 2.5x according to Plausible. This was also the first year where people actually reached out to say thank you</strong> for sharing. I always assumed no one was reading.</p>
The top posts by visitors: NixOS: The Good, The Bad, and The Ugly</a>, Unlocking Tokio's Hidden Gems</a>, Distributed Systems Resources</a>, and What if we embraced simulation-driven development?</a>. Strangely enough, my most shared post was not about distributed systems or FoundationDB but about NixOS. I think people appreciated the honest take: the good, the bad, and</strong> the ugly. The Tokio post being #2 was also unexpected. Sometimes the posts you almost do not publish are the ones that resonate.</p>
I love making presentations. In 2025, I gave two talks at Devoxx France: one about simulation-driven development</a> and another about prototyping distributed systems with Maelstrom</a>. I also presented my fdb-rs journey</a> at FinistDevs</a>, which I help organize in Brest.</p>
The FoundationDB Rust crate</a> keeps growing: 11 million downloads, used in production by real companies. But I was not a great maintainer this year. Development followed Clever Cloud's requirements. People asked for documentation about simulation testing and a roadmap, and I did not deliver. That is not a complaint about open source, just honesty. Being a solo maintainer without foundation backing means priorities get driven by the day job. Last week I finally wrote a roadmap and flushed my brain into GitHub issues so contributors can pick up work. I hope to do better in 2026.</p>
🔗</a>How LLMs Changed My Work</h2>
I cannot write about 2025 without talking about LLMs. I spent a lot</strong> of time learning how to use them in my work.</p>
For years, I had a weekly habit: two hours dedicated to reading codebases I depend on, understanding the internals of libraries, frameworks, and databases. Then I stopped. Life got busy, management took over, and diving into unfamiliar code took too long to justify. With LLMs, I picked the habit back up. What used to take hours now takes minutes. I can explore a codebase conversationally, asking questions, jumping to relevant sections, building mental models faster than ever. I learn more now than I did before.</p>
They handle peripheral code well: glue code, boilerplate, scaffolding. Geoffrey Litt calls this "coding like a surgeon</a>": delegate the prep work, focus on what matters. Moonpool is a good example. Claude translated C++ Flow patterns into Rust and generated test scaffolding. But knowing what</strong> to translate required years of operating distributed systems: why FDB uses interface swapping, why buggify has two-phase activation, why virtual time compresses hours into seconds. As João Alves wrote, when "software becomes fast food</a>", expertise becomes the scarce resource.</p>
But what I did not expect is that working with LLMs forces me to flesh out invariants and hidden rules somewhere explicit. You need to write things down for the LLM to understand, and that documentation ends up being useful for humans too. Context is everything</strong>: given the right context, LLMs generate the right code. So I spent a lot of time (and tokens) generating project recaps and summaries to feed them. When working with libraries, I make local git clones so the LLM can browse the actual source code instead of relying on potentially outdated training data. I have been using Claude extensively, and I found spec-kit</a> helpful for framing my prompts. It is a toolkit for "spec-driven development" that helps you focus on product scenarios instead of what Simon Willison calls "vibe coding</a>". But we are still missing the tools</a> to make this workflow seamless.</p>
Some posts became unexpectedly useful as LLM context. My practical guide to application metrics</a> and my guidelines for FDB workloads</a> now live in project contexts. When I ask Claude to add instrumentation or write a simulation workload, it already knows my patterns.</p>
🔗</a>Looking Ahead</h2>
2025 reminded me why I love this work: building systems, learning in public, watching years of investment pay off.</p>
For 2026, the habits stay: writing one or two posts per month, reading codebases with LLM assistance, speaking at conferences. I want to push moonpool toward something others can actually use. We have ambitious plans for Materia at Clever Cloud, and some of them should lead me to contribute to FoundationDB directly. I will keep helping organize FinistDevs</a> in Brest. And I will definitely be back at BugBash 2026.</p>
The theme is the same as 2025: go deeper, share what I learn, build things that last. What did your 2025 look like?</strong></p>

Feel free to reach out to share your own 2025 reflections. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Specs Are Back, But We're Missing the Tools
Pierre Zemb — Fri, 19 Dec 2025 00:00:00 +0000
I truly think LLMs are changing how we write software. For me, it's been a massive productivity boost. I can ask Claude to read some piece of code and explain it to me, or make a quick PoC of something, or refactor stuff that would take me hours. I even used it to help me backport features from FoundationDB in Rust</a>, and it worked surprisingly well 🤯</p>
João Alves made a great point recently</a>: code is becoming like fast food. Cheap, fast, everywhere. You ask an LLM to generate something, it compiles, the tests pass, and you ship it. For critical systems, simulation testing</a> can validate that code actually survives production chaos.</p>
But here's what I keep running into: we're still using natural language</strong> to prompt, correct, and guide LLMs. Vague instructions produce vague code. The bottleneck isn't writing code anymore, it's knowing what</strong> to write in the first place.</p>
🔗</a>Why specs died in the first place</h2>
Ask any engineering team "where's the spec for this service?" and you'll probably get one of three answers: blank stares, a link to some 3-year-old Google doc that's completely outdated, or my personal favorite, "the code is the spec."</p>
I think the problem was simple: specs had no feedback loop</strong>. Code compiles, tests pass, but specs? They just sit there. Nobody validates them, nobody updates them. Six months later, the spec has become archaeology, and new team members learn to ignore it because they can't trust it anyway.</p>
What changed is that LLMs can actually read</strong> specifications now. And suddenly, specs aren't dead documents anymore. They're instructions that can be executed. I've found two modes that actually work:</p>

Generation</strong>: you give an LLM a structured spec, and it gives you an implementation</li>
Validation</strong>: you give an LLM some existing code and a spec, and ask "does this implementation actually respect the specification?"</li>
</ul>
🔗</a>spec-kit and the right prompt chain</h2>
I tried spec-kit</a> a while ago and found it pretty useful. What it does well is guide you through a structured chain of prompts: you start with a Constitution (your project principles), then you write Specifications (requirements with acceptance criteria), then Technical Plans, then Tasks, and finally Implementation.</p>
It sounds obvious when I write it like that, but it's surprisingly effective. This isn't scattered TODO comments. It's a queryable structure that builds context progressively, and the LLM can use all of it.</p>
The generated code was actually good, because spec-kit forced me to build the right context first. And here's what surprised me: the LLM kept challenging my vague requirements. Every time I wrote something like "handle edge cases," it would ask "what happens when X? what about Y?" until the spec was actually implementable.</p>
I think that's the trick. Context is everything</strong>. Build the right context, and the LLM produces the right code.</p>
🔗</a>The limits of English</h2>
Here's where I hit a wall though. English-based specs work great for user stories and acceptance criteria, the kind of stuff product managers care about. But for algorithms and system behavior? Natural language gets ambiguous really fast.</p>
"Handle concurrent access" means different things to different people. "Ensure consistency" is even worse. When you're designing distributed algorithms with subtle timing constraints, you need precision. English just doesn't cut it.</p>
I needed something more engineering-driven. Not formal verification for academic purposes, but practical precision that the whole team could read and reason about.</p>
🔗</a>Finding an engineering-driven approach</h2>
I started looking at formal methods. TLA+</a> is the classic choice, but the notation felt like another language to maintain. I didn't want to be the only one on the team who could read the specs. I've been there before with other technologies, and it's not a great place to be 😅</p>
Then a friend</a> suggested Fizzbee</a>. It's based on Starlark, a Python dialect. Model checking without the TLA+ notation. The whole team can contribute.</p>
Learning new languages with LLMs works well. The trick is to find or generate a spec of the language first, then ask for a tutorial tailored to your specific problem. I asked Claude to write a Starlark reference and a Fizzbee concepts recap. Now we share vocabulary, and the conversations are productive.</p>
🔗</a>What we're still missing</h2>
Fizzbee is great for what it does. For algorithms and concurrency, model checking feels like the Rust compiler but for higher-level design. It explores all possible states and finds bugs before any code exists. Here's what an invariant looks like:</p>
# Safety: no duplicate completions
</span>always assertion NoDuplicateCompletions:
</span>    </span>return </span>len</span>(completed) == </span>len</span>(</span>set</span>(completed))
</span></code></pre>
Readable by anyone who knows Python.</p>
But most software isn't distributed algorithms. Most of what we build is about storing data in databases, sending messages to queues, calling other services, transforming inputs into outputs.</p>
And we describe this behavior in a dozen different places: C4 diagrams for architecture, OpenAPI for HTTP endpoints, protobuf for message schemas, ADRs for decisions, markdown for everything else. No single notation captures the full picture.</p>
I keep thinking about what this tool would need to be. It should be compact</strong>, short enough to fit in an LLM's context window without eating thousands of tokens. It should work at both levels: service interactions ("UserService stores in Postgres, publishes to Kafka") and function behavior ("validateUser checks format, queries DB, returns DTO"). It should be the common language</strong> that both the team and the LLM can read, write, and reason about.</p>
Most importantly, it should be verifiable</strong>. Not just documentation that sits there, but something that can actually validate whether an implementation matches what we said it would do. The feedback loop that specs never had.</p>
OpenAPI gets close for HTTP APIs. You can validate requests, generate clients, catch breaking changes. But for the rest? For business logic, for service contracts that aren't just endpoints, for the behavior that actually matters? The tooling doesn't exist yet.</p>
As the old joke goes, a spec precise enough to generate code is just called code</a>. But there has to be something in between prose and implementation. And yes, I'm aware that proposing a new format makes me the 15th competing standard</a>. But if you've found something that fills this gap, or have ideas about what it should look like, I'd love to hear about it.</p>

Feel free to reach out to share your thoughts on spec tooling. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Designing Rust FDB Workloads That Actually Find Bugs
Pierre Zemb — Tue, 09 Dec 2025 00:00:00 +0000
After one trillion CPU-hours of simulation testing</a>, FoundationDB has been stress-tested under conditions far worse than any production environment. Network partitions, disk failures, Byzantine faults. FDB handles them all. But what about your code?</strong> Your layer sits on top of FDB. Your indexes, your transaction logic, your retry handling. How do you know it survives chaos?</p>
At Clever Cloud, we are building Materia</a>, our serverless database product. The question haunted us: how do you ship layer code with the same confidence FDB has in its own? Our answer was to hack our way into FDB's simulator using foundationdb-simulation</a>, a crate that compiles Rust to run inside FDB's deterministic simulator. We're the only language besides Flow that can pull this off.</p>
The first seed triggered commit_unknown_result</code></a>, one of the most feared edge cases for FDB layer developers. When a connection drops, the client can't know if the transaction committed. Our atomic counters were incrementing twice. In production, this surfaces once every few months under heavy load and during failures. In simulation? Almost immediately.</strong></p>
This post won't walk you through the code mechanics. The foundationdb-simulation crate</a> and its README</a> cover that. Instead, this teaches you how to design</strong> workloads that catch real bugs. Whether you're a junior engineer or an LLM helping write tests, these principles will guide you.</p>
🔗</a>Why Autonomous Testing Works</h2>
Traditional testing has you write specific tests for scenarios you imagined. But as Will Wilson put it at Bug Bash 2025</a>: "The most dangerous bugs occur in states you never imagined possible."</strong> The key insight of autonomous testing (what FDB's simulation embodies) is that instead of writing tests, you write a test generator</strong>. If you ran it for infinite time, it would eventually produce all possible tests you could have written. You don't have infinite time, so instead you get a probability distribution over all possible tests. And probability distributions are leaky: they cover cases you never would have thought to test.</p>
This is why simulation finds bugs so fast. You're not testing what you thought to test. You're testing what the probability distribution happens to generate, which includes edge cases you'd never have written explicitly. Add fault injection (a probability distribution over all possible ways the world can conspire to screw you) and now you're finding bugs that would take months or years to surface in production.</p>
This is what got me interested in simulation in the first place: how do you test the things you see during on-call shifts? Those weird transient bugs at 3 AM, the race conditions that happen once a month, the edge cases you only discover when production is on fire. Simulation shifts that complexity from SRE time to SWE time. What was a 3 AM page becomes a daytime debugging session. What was a high-pressure incident becomes a reproducible test case you can bisect, rewind, and experiment with freely.</p>
🔗</a>The Sequential Luck Problem</h2>
Here's why rare bugs are so hard to find: imagine a bug that requires three unlikely events in sequence. Each event has a 1/1000 probability. Finding that bug requires 1/1,000,000,000 attempts, roughly a billion tries with random testing. Research confirms this: a study of network partition failures</a> found that 83% require 3+ events to manifest, 80% have catastrophic impact, and 21% cause permanent damage that persists after the partition heals. But here's the good news for Rust workloads</strong>: you don't solve this problem yourself. FDB's simulation handles fault injection. BUGGIFY injects failures at arbitrary code points. Network partitions appear and disappear. Disks fail. Machines crash and restart. The simulator explores failure combinations that would take years to encounter in production.</p>
Your job is different. You need to design operations that exercise interesting code paths. Not just reads and writes, but the edge cases your users will inevitably trigger. And you need to write invariants that CATCH bugs when simulation surfaces them. After a million injected faults, how do you prove your data is still correct? This division of labor is the key insight: FDB injects chaos, you verify correctness.</p>
🔗</a>Designing Your Operation Alphabet</h2>
The operation alphabet</strong> is the complete set of operations your workload can perform. This is where most workloads fail: they test happy paths with uniform distribution and miss the edge cases that break production. Think about three categories:</p>
Normal operations</strong> with realistic weights. In production, maybe 80% of your traffic is reads, 15% is simple writes, 5% is complex updates. Your workload should reflect this, because bugs often hide in the interactions between operation types. A workload that runs 50% reads and 50% writes tests different code paths than one that runs 95% reads and 5% writes. Both might be valid, but they'll find different bugs.</p>
Adversarial inputs</strong> that customers will inevitably send. Empty strings. Maximum-length values. Null bytes in the middle of strings. Unicode edge cases. Boundary integers (0, -1, MAX_INT). Customers never respect your API specs, so model the chaos they create.</p>
Nemesis operations</strong> that break things on purpose. Delete random data mid-test. Clear ranges that "shouldn't" be cleared. Crash batch jobs mid-execution to test recovery. Run compaction every operation instead of daily. Create conflict storms where multiple clients hammer the same key. Approach the 10MB transaction limit. These operations stress your error handling and recovery paths. The rare operations are where bugs hide. That batch job running once a day in production? In simulation, you'll hit its partial-failure edge case in minutes, but only if your operation alphabet includes it.</p>
🔗</a>Designing Invariants</h2>
After simulation runs thousands of operations with injected faults, network partitions, and machine crashes, how do you know your data is still correct? Unlike FDB's internal testing, Rust workloads can't inject assertions at arbitrary code points. You verify correctness in the check()</code> phase, after the chaos ends. The key question: "After all this, how do I PROVE my data is still correct?"</strong></p>
One critical tip: validate during start()</code>, not just in check()</code>.</strong> Don't wait until the end to discover corruption. After each operation (or batch of operations), read back the data and verify it matches expectations. If you're maintaining a counter, read it and check the bounds. If you're building an index, query it immediately after insertion. Early validation catches bugs closer to their source, making debugging far easier. The check()</code> phase is your final safety net, but continuous validation during execution is where you'll catch most issues.</p>
An invariant is just a property that must always hold, no matter what operations ran. If you've seen property-based testing, it's the same idea: instead of assertFalse(new User(GUEST).canUse(SAVED_CARD))</code>, you write assertEquals(user.isAuthenticated(), user.canUse(SAVED_CARD))</code>. The first tests one case. The second tests a rule that holds for all cases.</p>
Four patterns dominate invariant design:</p>
Reference Models</strong> maintain an in-memory copy of expected state. Every operation updates both the database and the reference model. In check()</code>, you compare them. If they diverge, something went wrong. Use BTreeMap</code> (not HashMap</code>) for deterministic iteration. This pattern works best for single-client workloads where you can track state locally.</p>
Conservation Laws</strong> track quantities that must stay constant. Inventory transfers between warehouses shouldn't change total inventory. Money transfers between accounts shouldn't create or destroy money. Sum everything up and verify the conservation law holds. This pattern is elegant because it doesn't require tracking individual operations, just the aggregate property.</p>
Structural Integrity</strong> verifies data structures remain valid. If you maintain a secondary index, verify every index entry points to an existing record and every record appears in the index exactly once. If you maintain a linked list in FDB, traverse it and confirm every node is reachable. The cycle validation pattern (creating a circular list where nodes point to each other) is a classic technique from FDB's own Cycle workload</a>. After chaos, traverse the cycle and verify you visit exactly N nodes.</p>
Operation Logging</strong> solves two problems at once: maybe_committed</code> uncertainty and multi-client coordination. The trick from FDB's own AtomicOps workload</a>: log the intent alongside the operation in the same transaction</strong>. Write both your operation AND a log entry recording what you intended. Since they're in the same transaction, they either both commit or neither does. No uncertainty. For multi-client workloads, each client logs under its own prefix (e.g., log/{client_id}/</code>). In check()</code>, client 0 reads all logs from all clients, replays them to compute expected state, and compares against actual state. If they diverge, something went wrong, and you'll know exactly which operations succeeded. See the Rust atomic workload example</a> for a complete implementation.</p>
🔗</a>The Determinism Rules</h2>
FDB's simulation is deterministic. Same seed, same execution path, same bugs. This is the superpower that lets you reproduce failures. But determinism is fragile. Break it, and you lose reproducibility. Five rules to remember:</p>

BTreeMap, not HashMap</strong>: HashMap iteration order is non-deterministic</li>
context.rnd(), not rand::random()</strong>: All randomness must come from the seeded PRNG</li>
context.now(), not SystemTime::now()</strong>: Use simulation time, not wall clock</li>
db.run(), not manual retry loops</strong>: The framework handles retries and maybe_committed</code> correctly</li>
No tokio::spawn()</strong>: The simulation runs on a custom executor, spawning breaks it</li>
</ol>
If you take nothing else from this post, memorize these. Break any of them and your failures become unreproducible. You'll see a bug once and never find it again.</p>
🔗</a>Architecture: The Three-Crate Pattern</h2>
Real production systems use tokio, gRPC, REST frameworks, all of which break simulation determinism. You can't just drop your production binary into the simulator. The solution is separating your FDB operations into a simulation-friendly crate:</p>
my-project/
</span>├── my-fdb-service/      # Core FDB operations - NO tokio
</span>├── my-grpc-server/      # Production layer (tokio + tonic)
</span>└── my-fdb-workloads/    # Simulation tests
</span></code></pre>
The service crate contains pure FDB transaction logic with no async runtime dependency. The server crate wraps it for production. The workloads crate tests the actual service logic under simulation chaos. This lets you test your real production code, not a reimplementation that might have different bugs.</p>
🔗</a>Common Pitfalls</h2>
Beyond the determinism rules above, these mistakes will bite you:</p>
Running setup or check on all clients.</strong> The framework runs multiple clients concurrently. If every client initializes data in setup()</code>, you get duplicate initialization. If every client validates in check()</code>, you get inconsistent results. Use if self.client_id == 0</code> to ensure only one client handles initialization and validation.</p>
Forgetting maybe_committed.</strong> The db.run()</code> closure receives a maybe_committed</code> flag indicating the previous attempt might have succeeded. If you're doing non-idempotent operations like atomic increments, you need either truly idempotent transactions or automatic idempotency</a> in FDB 7.3+. Ignoring this flag means your workload might count operations twice.</p>
Storing SimDatabase between phases.</strong> Each phase (setup</code>, start</code>, check</code>) gets a fresh database reference. Storing the old one leads to undefined behavior. Always use the db</code> parameter passed to each method.</p>
Wrapping FdbError in custom error types.</strong> The db.run()</code> retry mechanism checks if errors are retryable via FdbError::is_retryable()</code>. If you wrap FdbError</code> in your own error type (like anyhow::Error</code> or a custom enum), the retry logic can't see the underlying error and won't retry. Keep FdbError</code> unwrapped in your transaction closures, or ensure your error type preserves retryability information.</p>
Assuming setup is safe from failures.</strong> BUGGIFY is disabled during setup()</code>, so you might think transactions can't fail. But simulation randomizes FDB knobs, which can still cause transaction failures. Always use db.run()</code> with retry logic even in setup, or wrap your setup in a retry loop.</p>
🔗</a>The Real Value</h2>
That commit_unknown_result</code> edge case appeared on our first simulation seed. In production, we'd still be hunting it months later. 30 minutes of simulation covers what would take 24 hours of chaos testing. But the real value of simulation testing isn't just finding bugs, it's forcing you to think about correctness.</strong> When you design a workload, you're forced to ask: "What happens when this retries during a partition?" "How do I verify correctness when transactions can commit in any order?" "What invariants must hold no matter what chaos occurs?" Designing for chaos becomes natural. And if it survives simulation, it survives production.</p>

Feel free to reach out with questions or to share your simulation workloads. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Diving into Kubernetes' Watch Cache
Pierre Zemb — Wed, 12 Nov 2025 00:00:00 +0000

Diving Into</a> is a blogpost series where we dig into specific parts of a project's codebase. In this episode, we dig into Kubernetes' watch cache implementation.</p>

While debugging an etcd-shim on FoundationDB, I kept hitting "Timeout: Too large resource version"</code> errors. The cache was stuck at revision 3044, but clients requested 3047. Three seconds later: timeout. This led me into the watch cache internals: specifically the 3-second timeout in waitUntilFreshAndBlock()</code> and how progress notifications solve the problem. Let's dig into how it actually works.</p>

Note:</strong> Yes, Clever Cloud</a> runs an etcd-shim on top of FoundationDB for Kubernetes. Truth is, we're not alone: AWS</a> and GKE</a> have custom storage layers too. After operating etcd at OVHcloud</a>, we chose a different path. I actually wrote a naive PoC during COVID (fdb-etcd</a>) without testing it against a real apiserver 😅 it was mostly an excuse to discover the Record-Layer</a>. You can read more about the technical challenges in this FoundationDB forum discussion</a>.</p>
</blockquote>
🔗</a>Overview of the Watch Cache</h2>
When I first looked at the watch cache implementation, I expected a single monolithic cache sitting between the apiserver and etcd. It took compiling my own apiserver with additional logging to realize the architecture is more interesting: each resource type gets its own independent Cacher instance</strong>. Pods have one. Services have another. Deployments get their own. Every resource group runs an isolated LIST+WATCH loop, maintaining its own in-memory cache.</p>
As the Kubernetes 1.34 blog post</a> explains, this enhancement allows the API server to serve consistent read requests directly from the watch cache, significantly reducing the load on etcd and improving overall cluster performance.</p>
🔗</a>Architecture</h2>
Client Requests (kubectl, controllers)
</span>          ↓
</span>    Cacher (per resource)
</span>          ↓ In-memory watch cache
</span>          ↓ (on cache miss/delegate)
</span>    etcd3/Store
</span>          ↓
</span>    etcd / etcd-shim
</span></code></pre>
The main components:</p>

cacher.go</a> - The in-memory watch cache</li>
store.go</a> - Direct etcd</a> communication layer</li>
</ul>
🔗</a>How The Cache Gets Fed</h2>
🔗</a>Initialization: The LIST Phase</h3>
Nothing works until the cache initializes</strong>. When a Cacher starts, every read for that resource blocks until initialization completes. This matters because initialization isn't instant: it's a paginated LIST operation fetching 10,000 items per page. For a large cluster with thousands of pods, this takes time.</p>
Here's the sequence: The Reflector pattern kicks off with a complete LIST operation. Each resource cache fetches all existing objects through paginated requests. Once the LIST completes, watchCache.Replace()</code> populates the in-memory cache with these objects. The critical moment</strong> happens when the SetOnReplace()</code> callback fires (cacher.go:468-478</a>), marking the cache as READY. Until that callback fires, every request for that resource waits.</p>
🔗</a>Continuous Sync: The WATCH Phase</h3>
After initialization, the real trick begins: the cache maintains synchronization through a Watch stream that starts at LIST revision + 1. This guarantees no events are missed</strong> between the LIST and WATCH operations. The watch picks up exactly where the list left off. Events flow from etcd through a buffered channel (capacity: 100 events) and are processed by the dispatchEvents()</code> goroutine, which runs continuously, matching events to interested watchers.</p>
This pattern depends on continuous event flow. When events stop arriving, when resources go quiet, that's when progress notifications become essential. See Reflector documentation</a> for the complete pattern.</p>
🔗</a>The Problem: "Timeout: Too large resource version"</h2>
While debugging our etcd-shim, we kept hitting this error:</p>
Error getting keys: err="Timeout: Too large resource version: 3047, current: 3044"
</span></code></pre>
A client was requesting ResourceVersion 3047, but the cache only knew about revision 3044. The cache would wait... and timeout after 3 seconds.</p>
🔗</a>Understanding Cache Freshness</h2>
🔗</a>The Freshness Check</h3>
When a client requests a consistent read at a specific ResourceVersion, Kubernetes needs to ensure the cache is "fresh enough" to serve that request. Here's the check: is my current revision at least as high as the requested revision? If not, it calls waitUntilFreshAndBlock()</code> with a 3-second timeout, waiting for Watch events to bring the cache up to date.</p>
From cacher.go:1257-1261</a>:</p>
if </span>c</span>.</span>watchCache</span>.</span>notFresh</span>(</span>requestedWatchRV</span>) {
</span>    </span>c</span>.</span>watchCache</span>.</span>waitingUntilFresh</span>.</span>Add</span>()
</span>    </span>defer </span>c</span>.</span>watchCache</span>.</span>waitingUntilFresh</span>.</span>Remove</span>()
</span>}
</span>err </span>:= </span>c</span>.</span>watchCache</span>.</span>waitUntilFreshAndBlock</span>(</span>ctx</span>, </span>requestedWatchRV</span>)
</span></code></pre>
The actual timeout implementation (watch_cache.go:448-488</a>):</p>
func </span>(</span>w </span>*</span>watchCache</span>) </span>waitUntilFreshAndBlock</span>(</span>ctx context</span>.</span>Context</span>, </span>resourceVersion </span>uint64</span>) </span>error </span>{
</span>    </span>startTime </span>:= </span>w</span>.</span>clock</span>.</span>Now</span>()
</span>    </span>defer func</span>() {
</span>        </span>if </span>resourceVersion </span>> </span>0 </span>{
</span>            </span>metrics</span>.</span>WatchCacheReadWait</span>.</span>WithContext</span>(</span>ctx</span>).</span>WithLabelValues</span>(</span>w</span>.</span>groupResource</span>.</span>Group</span>, </span>w</span>.</span>groupResource</span>.</span>Resource</span>).</span>Observe</span>(</span>w</span>.</span>clock</span>.</span>Since</span>(</span>startTime</span>).</span>Seconds</span>())
</span>        }
</span>    }()
</span>
</span>    </span>// In case resourceVersion is 0, we accept arbitrarily stale result.
</span>    </span>// As a result, the condition in the below for loop will never be
</span>    </span>// satisfied (w.resourceVersion is never negative), this call will
</span>    </span>// never hit the w.cond.Wait().
</span>    </span>// As a result - we can optimize the code by not firing the wakeup
</span>    </span>// function (and avoid starting a gorotuine), especially given that
</span>    </span>// resourceVersion=0 is the most common case.
</span>    </span>if </span>resourceVersion </span>> </span>0 </span>{
</span>        </span>go func</span>() {
</span>            </span>// Wake us up when the time limit has expired.  The docs
</span>            </span>// promise that time.After (well, NewTimer, which it calls)
</span>            </span>// will wait *at least* the duration given. Since this go
</span>            </span>// routine starts sometime after we record the start time, and
</span>            </span>// it will wake up the loop below sometime after the broadcast,
</span>            </span>// we don't need to worry about waking it up before the time
</span>            </span>// has expired accidentally.
</span>            <-</span>w</span>.</span>clock</span>.</span>After</span>(</span>blockTimeout</span>)
</span>            </span>w</span>.</span>cond</span>.</span>Broadcast</span>()
</span>        }()
</span>    }
</span>
</span>    </span>w</span>.</span>RLock</span>()
</span>    </span>span </span>:= </span>tracing</span>.</span>SpanFromContext</span>(</span>ctx</span>)
</span>    </span>span</span>.</span>AddEvent</span>("</span>watchCache locked acquired</span>")
</span>    </span>for </span>w</span>.</span>resourceVersion </span>< </span>resourceVersion </span>{
</span>        </span>if </span>w</span>.</span>clock</span>.</span>Since</span>(</span>startTime</span>) >= </span>blockTimeout </span>{
</span>            </span>// Request that the client retry after 'resourceVersionTooHighRetrySeconds' seconds.
</span>            </span>return </span>storage</span>.</span>NewTooLargeResourceVersionError</span>(</span>resourceVersion</span>, </span>w</span>.</span>resourceVersion</span>, </span>resourceVersionTooHighRetrySeconds</span>)
</span>        }
</span>        </span>w</span>.</span>cond</span>.</span>Wait</span>()
</span>    }
</span>    </span>span</span>.</span>AddEvent</span>("</span>watchCache fresh enough</span>")
</span>    </span>return </span>nil
</span>}
</span></code></pre>
If the cache can't catch up within those 3 seconds, the request times out.</p>
If you've ever seen kubectl commands hang for exactly 3 seconds before returning data, this is why. The cache is waiting for events that will never come.</p>
🔗</a>The Problem with Quiet Resources</h3>
This is where things get tricky. For infrequently-updated resources (namespaces, configmaps, etc.):</p>
Time</th> Component</th> Event</th> Cache RV</th> etcd RV</th> Notes</th></tr></thead>

T0</td> Namespace cache</td> Idle, no changes</td> 3044</td> 3044</td> No namespace changes for 5 minutes</td></tr>
T1</td> Pod/Service caches</td> Resources changing</td> -</td> 3047</td> Global etcd revision advances</td></tr>
T2</td> Namespace watch</td> Receives nothing</td> 3044</td> 3047</td> No namespace events to process</td></tr>
T3</td> Namespace cache</td> Still waiting</td> 3044</td> 3047</td> Cache stuck, unaware of global progress</td></tr>
T4</td> Client</td> Lists pods successfully</td> -</td> 3047</td> Response includes current RV 3047</td></tr>
T5</td> Client</td> Requests namespace read at RV ≥ 3047</td> -</td> 3047</td> Consistent read requirement</td></tr>
T6</td> Namespace cache</td> waitUntilFreshAndBlock()</code></td> 3044</td> 3047</td> "I'm at 3044, need 3047... waiting"</td></tr>
T7</td> Namespace cache</td> Timeout!</td> 3044</td> 3047</td> 3 seconds elapsed, returns error</td></tr>
</tbody></table>
The cache has no way to know if etcd has moved forward. Is the system healthy? Is something broken? It just sees... nothing.</p>
🔗</a>Timeout Behavior Summary</h3>
Scenario</th> Cache RV</th> Requested RV</th> Result</th></tr></thead>

Fresh cache</td> 3047</td> 3045</td> ✓ Serve immediately</td></tr>
Stale cache</td> 3044</td> 3047</td> ⏱ Wait 3s → timeout</td></tr>
With progress</td> 3044</td> 3047</td> ✓ RequestProgress → serve</td></tr>
</tbody></table>
🔗</a>Progress Notifications: Keeping Quiet Resources Fresh</h2>
🔗</a>What Are Progress Notifications?</h3>
Here's the trick: progress notifications are empty Watch responses</strong> that only update the revision:</p>
WatchResponse </span>{
</span>    </span>Header</span>: { </span>Revision</span>: </span>3047 </span>},  </span>// Current etcd revision
</span>    </span>Events</span>: []                     </span>// No actual data changes
</span>}
</span></code></pre>
They solve the quiet resource problem by telling the cache: "etcd is now at revision X, even though your resource hasn't changed."</p>
This is exactly what we had forgotten to implement in our etcd-shim. We handled regular Watch events perfectly, but didn't support progress notifications. The result? Kubernetes' watch cache would timeout waiting for revisions that would never arrive through normal events. Once we added RequestProgress</code> support and started sending these empty bookmark responses, the timeouts disappeared.</p>
🔗</a>Two Mechanisms</h3>
🔗</a>1. On-Demand: RequestWatchProgress()</h4>
When the cache needs to catch up, it can explicitly request a progress notification. See store.go:99-103</a>:</p>
func </span>(</span>s </span>*</span>store</span>) </span>RequestWatchProgress</span>(</span>ctx context</span>.</span>Context</span>) </span>error </span>{
</span>    </span>return </span>s</span>.</span>client</span>.</span>RequestProgress</span>(</span>s</span>.</span>watchContext</span>(</span>ctx</span>))
</span>}
</span></code></pre>
When called, etcd responds with a bookmark (also called a progress notification) containing the current revision. The cache at revision 3044 calls RequestProgress()</code>, receives { Revision: 3047, Events: [] }</code>, and immediately updates its internal state to 3047.</p>
The progress notification is detected in the watch stream (watcher.go:401-404</a>):</p>
// Handle progress notifications (bookmarks)
</span>if </span>wres</span>.</span>IsProgressNotify</span>() {
</span>    </span>wc</span>.</span>queueEvent</span>(</span>progressNotifyEvent</span>(</span>wres</span>.</span>Header</span>.</span>GetRevision</span>()))
</span>    </span>metrics</span>.</span>RecordEtcdBookmark</span>(</span>wc</span>.</span>watcher</span>.</span>groupResource</span>)
</span>    </span>continue
</span>}
</span></code></pre>
🔗</a>2. Proactive: Periodic Progress Requests</h4>
Kubernetes also runs a background component called progressRequester</a> that monitors quiet watches. This component detects when watches haven't received events for a while and periodically calls RequestProgress()</code> to ensure even completely idle resources stay fresh. This proactive approach prevents timeout errors before they happen.</p>
The progress requester is initialized when the Cacher is created (cacher.go:425-428</a>):</p>
progressRequester </span>:= </span>progress</span>.</span>NewConditionalProgressRequester</span>(
</span>    </span>config</span>.</span>Storage</span>.</span>RequestWatchProgress</span>,  </span>// The function to call
</span>    </span>config</span>.</span>Clock</span>,
</span>    </span>contextMetadata
</span>)
</span></code></pre>
🔗</a>The Complete Flow</h2>
Timeline showing how progress notifications solve the timeout:</strong></p>
Time</th> Component</th> Action</th> Cache RV</th> etcd RV</th> Details</th></tr></thead>

T0</td> Namespace watch</td> Established</td> 3044</td> 3044</td> No namespace changes happening</td></tr>
T1</td> Pod resources</td> Creates/updates</td> 3044</td> 3047</td> Namespace watch: silent, cache stuck at 3044</td></tr>
T2</td> Client</td> Requests namespace LIST at RV 3047</td> 3044</td> 3047</td> notFresh(3047)</code> → true, starts waitUntilFreshAndBlock()</code></td></tr>
T3</td> progressRequester</td> Detects quiet watch</td> 3044</td> 3047</td> Calls RequestProgress()</code> on namespace watch stream</td></tr>
T4</td> etcd</td> Sends progress notification</td> 3044</td> 3047</td> WatchResponse { Header: { Revision: 3047 }, Events: [] }</code></td></tr>
T5</td> Namespace cache</td> Processes bookmark</td> 3047</td> 3047</td> Updates internal revision 3044 → 3047, signals waiters</td></tr>
T6</td> Namespace cache</td> Returns successfully</td> 3047</td> 3047</td> waitUntilFreshAndBlock()</code> completes, request served from cache</td></tr>
</tbody></table>
🔗</a>Key Takeaways</h2>
Here's what you need to know: Kubernetes runs a separate watch cache for each resource type (pods, services, deployments, etc.), and each one maintains its own LIST+WATCH loop. When you request a consistent read, the cache performs a freshness check with a 3-second timeout</strong> via waitUntilFreshAndBlock()</code>. Without this mechanism, you'd see 3-second hangs on every consistent read to quiet resources.</p>
Progress notifications</strong> solve the critical problem of quiet resources: those that don't receive updates for extended periods. These empty Watch responses update the cache's revision without transferring data. Kubernetes implements this through two mechanisms: on-demand</strong> (explicit RequestProgress calls when the cache needs to catch up) and proactive</strong> (periodic monitoring by the progressRequester component).</p>
Without progress notifications, consistent reads must bypass the cache entirely and go directly to etcd, significantly increasing load on the storage layer. This is the difference between a responsive cluster and one where every kubectl command feels sluggish.</p>
🔗</a>Related Posts</h2>
If you enjoyed this deep dive into Kubernetes watch caching, you might also be interested in:</p>

Notes about ETCD</a> - An overview and collection of resources about etcd, the distributed key-value store that powers Kubernetes</li>
Diving into ETCD's linearizable reads</a> - A deep dive into how etcd implements linearizable reads using Raft consensus</li>
</ul>

Feel free to reach out with any questions or to share your experiences with Kubernetes watch caching. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Diving into FoundationDB's Simulation Framework
Pierre Zemb — Thu, 30 Oct 2025 00:00:00 +0000

Diving Into</a> is a blogpost series where we are digging a specific part of the project's codebase. In this episode, we will dig into the implementation behind FoundationDB's simulation framework.</p>
</blockquote>
After years of on-call shifts running FoundationDB at Clever Cloud, here's what I've learned: I've never been woken up by FDB</strong>. Every production incident traced back to our code, our infrastructure, our mistakes. Never FDB itself. That kind of reliability doesn't happen by accident.</p>
The secret? Deterministic simulation testing</strong>. FoundationDB runs the real database software (not mocks, not stubs) in a discrete-event simulator alongside randomized workloads and aggressive fault injection. All sources of nondeterminism are abstracted: network, disk, time, and random number generation. Multiple FDB servers communicate through a simulated network in a single-threaded process. The simulator injects machine crashes, rack failures, network partitions, disk corruption, bit flips. Every failure mode you can imagine, happening in rapid succession, deterministically. Same seed, same execution path, same bugs, every single time.</p>
After roughly one trillion CPU-hours of simulation testing</strong>, FoundationDB has been stress-tested under conditions far worse than any production environment will ever encounter. The development environment is deliberately harsher than production: network partitions every few seconds, machine crashes mid-transaction, disks randomly swapped between nodes on reboot. If your code survives the simulator, production is easy.</p>
I've written before about FoundationDB</a>, simulation-driven development</a>, and testing prevention vs discovery</a>. Those posts cover the concepts and benefits. This post is different: this is how FoundationDB actually implements deterministic simulation</strong>. Interface swapping, deterministic event loops, BUGGIFY chaos injection, Flow actors, and the architecture that makes it all work. We're going deep into the implementation.</p>

  
  FoundationDB's simulation architecture: the same FDB server code runs in both the simulator process (using simulated I/O) and the real world (using real I/O)</em></p>
</div>
🔗</a>The Trick: Interface Swapping</h2>
The genius of FDB's simulation is surprisingly simple: the same code runs in both production and simulation by swapping interface implementations</strong>. The global g_network</code> pointer holds an INetwork</code> interface. In production, this points to Net2</code>, which creates real TCP connections using Boost.ASIO. In simulation, it points to Sim2</code> (sim2.actor.cpp</a>), which creates Sim2Conn</code> objects (fake connections that write to in-memory buffers).</p>
When code needs to send data, it gets a Reference<IConnection></code> from the network layer. In production, that's a real socket. In simulation, it's Sim2Conn</code> (sim2.actor.cpp</a>) with a std::deque<uint8_t></code> buffer. Network latency? The simulator adds delay()</code> calls with values from deterministicRandom()</code>. Packet loss? Just throw connection_failed()</code>. Network partition? Sim2Conn</code> checks g_clogging.disconnected()</code> and refuses delivery. It's all just memory operations with delays</strong>, running single-threaded and completely deterministic.</p>
What makes this truly deterministic is deterministicRandom()</code>, a seeded PRNG that replaces all randomness. Every network latency value, every backoff delay (like the Peer</code>'s exponential reconnection timing), every process crash timing goes through the same deterministic stream. Same seed, same execution path, every single time. When a test fails after 1 trillion simulated operations, you can reproduce the exact failure by running with the same seed.</p>
🔗</a>Biasing the Simulator: BUGGIFY</h3>
Most deep bugs need a rare combination of events. A network partition and</strong> a slow disk and</strong> a coordinator crash happening at the exact same moment. The probability of all three aligning randomly? Astronomical. You'd burn CPU-centuries waiting.</p>
FoundationDB solves this with BUGGIFY</code>, spread throughout the codebase. Each BUGGIFY</code> point fires 25% of the time, deterministically, so every test explores a different corner of the state space (Alex Miller's excellent post on BUGGIFY</a> covers the implementation details).</p>
Let's take timeout handling in data distribution as an example:</p>
// DDShardTracker.actor.cpp (fdbserver/DDShardTracker.actor.cpp:1508)
</span>choose {
</span>    </span>when</span>(</span>wait</span>(g_network-></span>isSimulated</span>() && </span>BUGGIFY_WITH_PROB</span>(</span>0.01</span>) ? </span>Never</span>()
</span>                                                          : </span>fetchTopKShardMetrics_impl</span>(self, req))) {}
</span>    </span>when</span>(</span>wait</span>(</span>delay</span>(SERVER_KNOBS-></span>DD_SHARD_METRICS_TIMEOUT</span>))) {
</span>        </span>// Timeout path
</span>    }
</span>}
</span></code></pre>
The Never()</code> future never completes. Literally hangs forever. This happens only in simulation (g_network->isSimulated()</code>) and with 1% probability (BUGGIFY_WITH_PROB(0.01)</code>). When it fires, the operation gets stuck, forcing the timeout branch to execute. Simple, elegant failure injection.</p>
But here's the trick: the timeout value itself is also buggified</strong>:</p>
// ServerKnobs.cpp
</span>init</span>( DD_SHARD_METRICS_TIMEOUT, </span>60.0 </span>);  </span>// Production: 60 seconds
</span>if</span>( randomize && BUGGIFY ) DD_SHARD_METRICS_TIMEOUT = </span>0.1</span>;  </span>// Simulation: 0.1 seconds!
</span></code></pre>
Production timeout: 60 seconds. BUGGIFY timeout: 0.1 seconds (600x shorter). The shrinking timeout window means legitimate operations are far more likely to hit timeout paths. Even without Never()</code> forcing a hang, simulated network delays and slow operations will trigger timeouts constantly. When Never()</code> does fire, you get guaranteed timeout execution. Every knob marked if (randomize && BUGGIFY)</code> becomes a chaos variable. Timeouts shrink, cache sizes drop, I/O patterns randomize.</p>
This creates combinatorial explosion</strong>. FoundationDB has hundreds of randomized knobs. Each BUGGIFY-enabled test run picks a different configuration: maybe connection monitors are 4x slower, but file I/O is using 32KB blocks, and cache size is 1000 entries, and reconnection delays are doubled. The next run? Completely different knob values. Same code, thousands of different operating environments. After one trillion simulated operations across countless test runs, you've stress-tested your code under scenarios that would take decades to encounter in production.</p>
🔗</a>The Developer Workflow: Simulation as CI/CD</h2>
Here's the FoundationDB developer experience: write code, run a few local simulation tests to catch obvious bugs, submit your merge request, then let the machines do the hard work</strong>.</p>
Every pull request triggers hundreds of thousands of simulation tests</strong> running on hundreds of cores for hours before a human even begins code review. Different seeds explore different execution paths, different failure timings, different BUGGIFY configurations. Nightly testing runs tens of thousands more simulations, crawling through edge cases you'd never think to test manually.</p>
In the early days when FoundationDB was still a company, they took this philosophy to its logical extreme: merge requests were automatically merged if simulation passed</strong>. No human approval needed. The simulation was so trusted that passing tests meant the code was production-ready. (You can hear more about FoundationDB's early development culture on The BugBash Podcast</a>).</p>
This changes how you think about distributed systems development. Instead of spending hours debugging race conditions or trying to mentally model all possible failure scenarios, you focus on building features. The simulation finds the edge cases. It discovers the bugs you'd never anticipate. It stress-tests your code under conditions that would take years to encounter in production.</p>
The scale ramps up through the development cycle: thousands of seeds during merge request testing, tens of thousands in nightly runs, potentially millions during major release cycles. Each seed represents a completely different execution path through your code. By the time your change reaches production, it's survived more chaos than most distributed systems see in their entire lifetime.</p>
The confidence this gives developers is extraordinary</strong>: if your code survives hundreds of thousands of simulated disasters, production feels easy in comparison.</p>
🔗</a>Flow: Actors and Cooperative Multitasking</h2>
FoundationDB doesn't use traditional threads. It uses Flow, a custom actor model built on C++. Here's a simple example:</p>
ACTOR Future<</span>int</span>> </span>asyncAdd</span>(Future<</span>int</span>> </span>f</span>, </span>int </span>offset</span>) {
</span>    </span>int</span> value = </span>wait</span>(f);  </span>// Suspend until f completes, then resume with its value
</span>    </span>return</span> value + offset;
</span>}
</span></code></pre>
The ACTOR</code> keyword marks functions that can use wait()</code>. When you call wait(f)</code>, the actor suspends</strong>. It returns control to the event loop and resumes later when the Future</code> completes, continuing with the result. No blocking. All asynchronous. Use the state</code> keyword for variables that need to persist across multiple wait()</code> calls.</p>
If you know Rust's async/await, Flow is the same concept. ACTOR</code> functions are like async fn</code>, wait()</code> is like .await</code>, and Future<T></code> is like Rust's Future</code>. The difference? Flow was built in 2009 for C++, and gets compiled by actorcompiler.h</code> into state machines rather than relying on language support.</p>
The same Flow code runs in both production and simulation. An actor waiting for network I/O gets a real socket in production, a simulated buffer in simulation. The code doesn't know the difference. The Flow documentation at apple.github.io/foundationdb/flow.html</a> covers the full programming model.</p>
🔗</a>Single-Threaded Time Travel: The Event Loop</h2>
Hundreds of actors running concurrently. Coordinators electing leaders, transaction logs replicating commits, storage servers handling reads. All happening in one thread</strong>.</p>
The trick is cooperative multitasking. Actors yield control with wait()</code>. When all actors are waiting, the event loop can advance simulated time</strong>:</p>

            flowchart TD
    Start([Event Loop]) --> CheckReady{Any actors<br/>ready to run?}
    CheckReady -->|Yes| RunActor[Run next ready actor<br/>until it hits wait]
    RunActor --> CheckReady
    CheckReady -->|No, all waiting| CheckPending{Any pending<br/>futures?}
    CheckPending -->|Yes| AdvanceTime[Advance simulated clock<br/>to next event]
    AdvanceTime --> CheckReady
    CheckPending -->|No| Done([Simulation complete])
    </pre>
</div>
Here's the key insight: when all actors are blocked waiting on futures, the event loop finds the next scheduled event (the earliest timestamp) and jumps the simulated clock forward</strong> to that time. Then it wakes the actors waiting for that event and runs them until they wait()</code> again.</p>
Example: 100 storage servers each execute wait(delay(deterministicRandom()->random01() * 60.0))</code>. In wall-clock time, this takes microseconds. In simulated time, these delays are spread across 60 seconds. The event loop processes them in order, advancing time as it goes. Zero wall-clock time has passed. 60 simulated seconds have passed.</strong></p>
This gives you:</p>

Compressed time</strong>: Years of uptime in seconds of testing. wait(delay(86400.0))</code> simulates 24 hours instantly.</li>
Perfect determinism</strong>: Single-threaded execution means no race conditions. Same seed, same event ordering, exact same execution path.</li>
Reproducibility</strong>: Test fails after 1 trillion simulated operations? Run again with the same seed, get the exact same failure at the exact same point.</li>
</ul>
No actor ever blocks. They all cooperate, yielding control back to the event loop. This is the foundation that makes realistic cluster simulation possible.</p>
🔗</a>Building the Simulated Cluster</h2>
Now that we understand Flow actors and the event loop, let's see what runs on it. SimulatedCluster builds an entire distributed cluster in memory</strong>.</p>
SimulatedCluster</code> starts by generating a random cluster configuration: 1-5 datacenters, 1-10+ machines per DC, different storage engines (memory, ssd, redwood-1), different replication modes (single, double, triple). Every test run gets a different topology.</p>
The actor hierarchy looks like this: SimulatedCluster creates machine actors (simulatedMachine</code>). Each machine actor creates process actors (simulatedFDBDRebooter</code>). Each process actor runs actual fdbserver code</strong>. The machine actor sits in an infinite loop: wait for all processes to die, delay 10 simulated seconds, reboot.</p>
The same fdbserver code that runs in production runs here</strong>. No mocks. No stubs. Real transaction logs writing to simulated disk. Real storage engines (RocksDB, Redwood). Real Paxos consensus. The only difference? Sim2</code> network instead of Net2</code>.</p>
And of course, BUGGIFY shows up here too. Remember how BUGGIFY shrinks timeouts and injects failures? It also does something completely insane</strong> during machine reboots. When a machine reboots, the simulator can swap its disks</strong>:</p>
// SimulatedCluster.actor.cpp - machine reboot
</span>state </span>bool</span> swap = killType == ISimulator::KillType::Reboot &&
</span>                  </span>BUGGIFY_WITH_PROB</span>(</span>0.75</span>) &&
</span>                  g_simulator-></span>canSwapToMachine</span>(localities.</span>zoneId</span>());
</span>if </span>(swap) {
</span>    availableFolders[localities.</span>dcId</span>()].</span>push_back</span>(myFolders);  </span>// Return my disks to pool
</span>    myFolders = availableFolders[localities.</span>dcId</span>()][randomIndex];  </span>// Get random disks from pool
</span>}
</span></code></pre>
75% of the time when BUGGIFY is enabled, a rebooting machine gets random disks from the datacenter pool</strong>. Maybe it gets its own disks back. Maybe it gets another machine's disks with completely different data. Maybe it gets the disks from a machine that was destroyed 10 minutes ago. Your storage server just woke up with someone else's data (or no data at all). Can the cluster handle this? Can it detect the mismatch and rebuild correctly?</p>
For extra chaos, there's also RebootAndDelete</code> which gives the machine brand new empty folders</strong>. No data. Fresh disks. This tests the actual failure mode of replacing a dead drive or provisioning a new machine.</p>
Read that again. During testing, FoundationDB randomly swaps or deletes storage server data on reboot</strong>. If your distributed database doesn't assume storage servers occasionally come back with amnesia or someone else's memories, you're not testing the real world. Because surely, no one has ever accidentally mounted the wrong volume in a Kubernetes deployment, right?</p>
What you get from all this:</p>

Real cluster behavior</strong>: Coordinators elect leaders, transaction logs replicate commits, storage servers handle reads/writes, backup agents run</li>
Real failure modes</strong>: Process crashes, machine reboots, network partitions (via g_clogging</code>), slow disks (via AsyncFileNonDurable</code>), disk swaps, data loss</li>
Realistic topologies</strong>: Multi-region configurations, different storage engines, different replication modes, different machine counts</li>
</ul>
When you run a simulation test, SimulatedCluster boots this entire virtual cluster, lets it stabilize, runs workloads against it while injecting chaos, then validates correctness.</p>
🔗</a>Workloads: Stress Testing Under Chaos</h2>
30 seconds. 2500 transactions per second. Concurrent machines swapping edges in a distributed data structure while chaos engines inject failures. Let's see if the database survives.</p>
Simulation Overview
</span>┌────────────┬──────────────────┬────────────────┬─────────────────┬────────────────┐
</span>│ Seed       ┆ Replication      ┆ Simulated Time ┆ Real Time       ┆ Storage Engine │
</span>╞════════════╪══════════════════╪════════════════╪═════════════════╪════════════════╡
</span>│ 1876983470 ┆ triple           ┆ 5m 47s         ┆ 18s 891ms       ┆ ssd-2          │
</span>└────────────┴──────────────────┴────────────────┴─────────────────┴────────────────┘
</span>
</span>Timeline of Chaos Events
</span>┌──────────┬────────────────────┬──────────────────────────────────────┐
</span>│ Time (s) ┆ Event Type         ┆ Details                              │
</span>╞══════════╪════════════════════╪══════════════════════════════════════╡
</span>│ 87.234   ┆ Coordinator Change ┆ Triggering leader election           │
</span>│ 92.156   ┆ Process Reboot     ┆ KillInstantly process at 10.0.4.2:3  │
</span>│ 92.156   ┆ Process Reboot     ┆ KillInstantly process at 10.0.4.2:1  │
</span>│ 95.871   ┆ Coordinator Change ┆ Triggering leader election           │
</span>│ 103.445  ┆ Process Reboot     ┆ RebootAndDelete process at 10.0.2.1:4│
</span>│ 103.445  ┆ Process Reboot     ┆ RebootAndDelete process at 10.0.2.1:2│
</span>└──────────┴────────────────────┴──────────────────────────────────────┘
</span>
</span>Chaos Summary
</span>  Network Partitions: 187 events (max duration: 5.2s)
</span>  Process Kills: 2 KillInstantly, 2 RebootAndDelete
</span>  Coordinator Changes: 2
</span></code></pre>
The cluster survived.</strong> 187 network partitions. 4 process kills. 2 coordinator changes. 5 minutes of simulated time compressed into 18 seconds of wall-clock time. Every transaction completed correctly. The cycle invariant never broke.</p>
How did we unleash this chaos? Here's the test configuration:</p>
[configuration]
</span>buggify </span>= </span>true
</span>minimumReplication </span>= </span>3
</span>
</span>[[test]]
</span>testTitle </span>= '</span>CycleWithAttrition</span>'
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>Cycle</span>'
</span>    </span>testDuration </span>= </span>30.0
</span>    </span>transactionsPerSecond </span>= </span>2500.0
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>RandomClogging</span>'
</span>    </span>testDuration </span>= </span>30.0
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>Attrition</span>'
</span>    </span>testDuration </span>= </span>30.0
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>Rollback</span>'
</span>    </span>testDuration </span>= </span>30.0
</span></code></pre>
🔗</a>What Just Happened?</h3>
Four concurrent workloads ran on the same simulated cluster for 30 seconds. Workloads</strong> are reusable scenario templates (180+ built-in in fdbserver/workloads/</a>) that either generate transactions or inject chaos.</p>
The application workload</strong> we ran:</p>

Cycle</strong> (Cycle.actor.cpp</a>): Hammered the database with 2500 transactions/second, each one swapping edges in a distributed graph. Tests SERIALIZABLE isolation by maintaining a cycle invariant. If isolation breaks, the cycle splits or nodes vanish. We'll dive deep into how this works below.</li>
</ul>
The chaos workloads</strong> that tried to break it:</p>

RandomClogging</strong> (RandomClogging.actor.cpp</a>): Calls g_simulator->clogInterface(ip, duration)</code> to partition machines. Those 187 network partitions</strong> we saw? This workload. Some lasted over 5 seconds.</li>
Attrition</strong> (MachineAttrition.actor.cpp</a>): Calls g_simulator->killMachine()</code> and g_simulator->rebootMachine()</code>. The 4 process kills</strong> (2 instant, 2 with deleted data)? This workload.</li>
Rollback</strong> (Rollback.actor.cpp</a>): Forces proxy-to-TLog failures, triggering coordinator recovery. The 2 coordinator changes</strong>? This workload.</li>
</ul>
Workloads are composable. The TOML format lets you stack them: [configuration]</code> sets global parameters (BUGGIFY, replication), each [[test.workload]]</code> adds another concurrent workload. Want to test atomic operations under network partitions? Stack AtomicOps</code> + RandomClogging</code>. Want to test backup during machine failures? Combine BackupToBlob</code> + Attrition</code>. Test files live in tests/</a>.</p>
🔗</a>How Does Cycle Work?</h3>
Remember that test we just ran? Let's break down how the Cycle</code> workload actually works. It creates a directed graph where every node points to exactly one other node, forming a single cycle: 0→1→2→...→N→0</code>. Then it runs 2500 concurrent transactions per second, each one randomly swapping edges in the graph. Meanwhile, chaos workloads kill machines, partition the network, and force coordinator changes. If SERIALIZABLE isolation works correctly, the cycle never breaks</strong>. You always have exactly N nodes in one ring, never split cycles or dangling pointers.</p>
Every workload implements four phases (workloads.actor.h</a>):</p>
SETUP</strong> (Cycle.actor.cpp</a>): Creates nodeCount</code> nodes. Each key stores the index of the next node in the cycle. Key 0 → value 1, key 1 → value 2, ..., key N-1 → value 0.</p>
EXECUTION</strong>: Multiple concurrent cycleClient</code> actors run this loop:</p>

Pick random node r</code></li>
Read three hops: r→r2→r3→r4</code></li>
Swap the middle two edges: make r→r3</code> and r2→r4</code></li>
Commit</li>
</ol>
This transaction reads 3 keys and writes 2. If isolation breaks, you could create cycles of the wrong length or lose nodes entirely.</p>
CHECK</strong>: One client reads the entire graph in a single transaction. Starting from node 0, follow pointers: 0→next→next→next. Count the hops. After exactly nodeCount</code> hops, you must be back at node 0. If you get there earlier (cycle too short) or can't get there (broken chain), the test fails. Also verifies transaction throughput met the expected rate.</p>
METRICS</strong>: Reports transactions completed, retry counts, latency percentiles.</p>
This is the pattern all workloads follow: SETUP initializes data, EXECUTION generates load, CHECK verifies correctness, METRICS reports results. When you execute a test, SimulatedCluster boots the cluster, runs SETUP phases sequentially, then runs all EXECUTION phases concurrently (they're Flow actors on the same event loop). After testDuration</code>, CHECK phases verify correctness.</p>
This is what runs before every FoundationDB commit.</strong> Not once. Not a few times. Thousands of test runs with different seeds, different cluster configurations, different workload combinations. Application workloads generate realistic transactions. Chaos workloads inject failures. The CHECK phases prove correctness survived the chaos. This is why FoundationDB doesn't fail in production. The simulator has already broken it every possible way, and every bug got fixed before shipping.</p>
I generated that simulation output using fdb-sim-visualizer</a>, a tool I wrote to parse simulation trace logs and understand what happened during testing.</p>
🔗</a>Verifying Correctness: Building Reliable Workloads</h2>
But here's the hard part: proving correctness when everything is randomized.</strong> The cluster survived. Transactions completed. The cycle invariant never broke... or did it? When you're running 2500 transactions per second with random edge swaps under 187 network partitions, how do you prove</strong> nothing went wrong? You can't just check if the database "looks okay." You need proof</strong> the invariants held.</p>
FoundationDB's approach: track during EXECUTION, verify in CHECK.</strong> Three patterns emerge across the codebase:</p>
🔗</a>Pattern 1: Reference Implementation</h3>
The challenge</strong>: How do you verify complex API behavior under chaos?</p>
The solution</strong>: Run every operation twice. ApiCorrectness</code> (ApiCorrectness.actor.cpp</a>) mirrors all operations in a simple MemoryKeyValueStore</code> (just a std::map<Key, Value></code>). Every transaction->set(k, v)</code> also executes store.set(k, v)</code> in memory. The CHECK phase reads from FDB and compares with the memory model. Mismatch = bug found.</p>
🔗</a>Pattern 2: Operation Logging</h3>
The challenge</strong>: How do you verify atomic operations executed in the right order?</p>
The solution</strong>: Log everything. AtomicOps</code> (AtomicOps.actor.cpp</a>) logs every operation to a separate keyspace. During EXECUTION: atomicOp(ops_key, value)</code> on real data, set(log_key, value)</code> to track what happened. During CHECK: replay all logged operations, compute what the final state should be, compare with actual database state.</p>
🔗</a>Pattern 3: Invariant Tracking</h3>
The challenge</strong>: How do you prove SERIALIZABLE isolation worked during chaos?</p>
The solution</strong>: Maintain a mathematical invariant that breaks if isolation fails. Cycle</code> (from our test earlier) maintains "exactly N nodes in one ring." During EXECUTION, random edge swaps must preserve the invariant. During CHECK, walk the graph: 0→next→next→next. After exactly N hops, you must be back at node 0. If you arrive earlier (cycle split) or can't arrive (broken chain), isolation failed. The CHECK phase catches this immediately.</p>
🔗</a>Using clientId for Work Distribution</h3>
Every workload gets clientId</code> (0, 1, 2...) and clientCount</code> (total clients). Three patterns:</p>
Client 0 only</strong> (AtomicOps.actor.cpp</a>):</p>
if </span>(clientId != </span>0</span>) </span>return </span>true</span>;  </span>// Common for CHECK phases
</span></code></pre>
Partition keyspace</strong> (WatchAndWait.actor.cpp</a>):</p>
uint64_t startNode = (nodeCount * clientId) / clientCount;
</span>uint64_t endNode = (nodeCount * (clientId + </span>1</span>)) / clientCount;
</span>// Client 0: nodes 0-33, Client 1: nodes 34-66, Client 2: nodes 67-99
</span></code></pre>
Round-robin</strong> (Watches.actor.cpp</a>):</p>
if </span>(i % clientCount == clientId)
</span>// Client 0: keys 0,3,6,9... Client 1: keys 1,4,7,10...
</span></code></pre>
Use clientId</code> to create concurrency (multiple clients hitting different keys) or coordinate work (one client checks, others generate load).</p>
🔗</a>Randomize Everything</h3>
The key to finding bugs: randomize every decision</strong>. Which keys to read? Random. How many operations per transaction? Random. Which atomic operation type? Random. Order of operations? Random. When to inject chaos? Random.</p>
But use deterministicRandom()</code> for all randomness. It's a seeded PRNG. Same seed = same random choices = reproducible failures. When a test fails after 10 million operations, rerun with the same seed, get the exact same failure at the exact same point.</p>
🔗</a>Pattern Selection Guide</h3>
Testing</th> Use Pattern</th> Example Workload</th></tr></thead>

API correctness</td> Reference implementation</td> ApiCorrectness</td></tr>
Atomic operations</td> Operation logging</td> AtomicOps</td></tr>
ACID guarantees</td> Invariant tracking</td> Cycle</td></tr>
Backup/restore</td> Absence checking</td> BackupCorrectness</td></tr>
</tbody></table>
Chaos workloads (RandomClogging</code>, Attrition</code>, Rollback</code>) don't need CHECK phases. They just return true</code>. They inject failures. Application workloads verify that correctness survived the chaos.</p>
🔗</a>Writing Workloads in Rust</h2>
Remember those chaos workloads hammering the Cycle test? RandomClogging</code>, Attrition</code>, Rollback</code>. All written in C++ Flow. But you can write workloads in Rust</strong> and compile them directly into the simulator. At Clever Cloud, we open-sourced foundationdb-simulation</a>, which lets you implement the RustWorkload</code> trait with setup()</code>, start()</code>, and check()</code> methods using Rust's async/await:</p>
#[</span>async_trait</span>]
</span>impl </span>RustWorkload </span>for </span>MyWorkload {
</span>    async </span>fn </span>setup</span>(&</span>mut </span>self</span>, </span>db</span>: Database, </span>_ctx</span>: Context) -> Result<()> {
</span>        </span>// Initialize test data
</span>        db.</span>run</span>(|</span>tx</span>, _| async </span>move </span>{
</span>            tx.</span>set</span>(</span>b</span>"</span>key</span>", </span>b</span>"</span>value</span>");
</span>            Ok(())
</span>        }).await
</span>    }
</span>
</span>    async </span>fn </span>start</span>(&</span>mut </span>self</span>, </span>db</span>: Database, </span>ctx</span>: Context) -> Result<()> {
</span>        </span>// Generate load under simulation
</span>        </span>for </span>_ in </span>0</span>..ctx.</span>get_option</span>("</span>nodeCount</span>", </span>100</span>) {
</span>            db.</span>run</span>(|</span>tx</span>, _| async </span>move </span>{
</span>                </span>let</span> value = tx.</span>get</span>(</span>b</span>"</span>key</span>", </span>false</span>).await?;
</span>                </span>// Your workload logic here
</span>                Ok(())
</span>            }).await?;
</span>        }
</span>        Ok(())
</span>    }
</span>
</span>    async </span>fn </span>check</span>(&</span>mut </span>self</span>, </span>db</span>: Database, </span>_ctx</span>: Context) -> Result<()> {
</span>        </span>// Verify correctness after chaos
</span>        Ok(())
</span>    }
</span>}
</span></code></pre>
Your Rust code compiles to a shared library, FDB's ExternalWorkload</code> loads it at runtime via FFI, and your Rust async functions run on the same Flow event loop as the C++ cluster. The FFI boundary is managed by the foundationdb-simulation</code> crate, which handles marshaling between Flow's event loop and Rust futures. Same determinism, same reproducibility, same chaos injection. But you're writing async fn</code> instead of ACTOR Future<Void></code>.</p>
Here's a complete example workload</a> testing atomic operations in ~100 lines of Rust.</p>
🔗</a>Simulation at Clever Cloud: Building Materia</h3>
At Clever Cloud, we use simulation to build Materia</a>, our serverless database products. I'm the lead engineer behind Materia and the main maintainer of foundationdb-rs</a>.</p>
We built a Rust SDK on top of FDB, similar to Apple's Record Layer</a>. It provides structured records, secondary indexes, query planning, and multi-tenant isolation. The result: a distributed transactional database built on FoundationDB's guarantees.</p>
FoundationDB is a hidden technology: you don't use it directly, you build layers on top. But here's the trick: patterns like index design, quota management, and schema management should be written once and consumed, not reimplemented in every product</strong>. Our SDK abstracts these patterns. Need secondary indexes? The SDK handles keyspace layout, index updates, and query planning. Need multi-tenant isolation? The SDK provides it. Need quota enforcement or permission management? We built a common control plane that works across all products</strong> built on the SDK. Write the hard distributed systems logic once, simulate it until it's bulletproof, then reuse it everywhere.</p>
Every merge request runs simulation tests in CI. We test SDK scenarios (indexing, query planning) and full product workloads under chaos. Multi-tenant isolation, concurrent queries, network partitions, machine crashes. The bugs we catch vary by layer. SDK changes catch nasty bugs like duplicated indexes during maybe_committed</code> transactions. Product changes catch simpler errors like accidentally blocking FDB's retry logic or breaking atomicity.</p>
But the real value isn't just bug detection. Instead of writing hundreds of unit tests, we write workloads that fuzz our code under chaos.</strong> One workload with randomized operations and deterministic chaos replaces dozens of hand-crafted test cases. When engineers write workloads for their features, they're forced to think: "What happens when this retries during a partition?" "How do I verify correctness when transactions can commit in any order?" Designing for chaos</strong> becomes natural. The act of writing simulation workloads improves the design itself.</p>
The confidence this gives a small team is extraordinary. When you can prove your code survives hundreds of network partitions and machine crashes before shipping, you sleep better at night. Our latest layer, an etcd-compatible API for managed Kubernetes, was built from the ground up with simulation in mind. We're even contributing features back to FoundationDB</a> to better support layers like ours.</p>
If it survives simulation, it survives production.</p>
🔗</a>Running Simulations Yourself</h2>
Think you can break FoundationDB? You don't need to build from source or set up a cluster. Download a prebuilt fdbserver</code> binary from the releases page</a>, create a test file, and unleash chaos:</p>
# Download fdbserver (Linux example, adjust for your platform)
</span>wget</span> https://github.com/apple/foundationdb/releases/download/7.3.27/fdbserver.x86_64
</span>chmod</span> +x fdbserver.x86_64
</span>
</span># Create the folder for traces
</span>mkdir</span> events
</span>
</span># Run a simulation test with JSON trace output
</span>./fdbserver.x86_64 -r</span> simulation</span> -f</span> Attritions.toml</span> --trace-format</span> json</span> -L</span> ./events</span> --logsize</span> 1GiB
</span></code></pre>
Here are two test files to get you started. Save either as a .toml</code> file and run with the command above.</p>
Attritions.toml</strong> - Network partitions + machine crashes + database reconfigurations (the NemesisTest shown earlier):</p>
[configuration]
</span>buggify </span>= </span>true
</span>minimumReplication </span>= </span>3
</span>
</span>[[test]]
</span>testTitle </span>= '</span>NemesisTest</span>'
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>ReadWrite</span>'
</span>    </span>testDuration </span>= </span>30.0
</span>    </span>transactionsPerSecond </span>= </span>1000.0
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>RandomClogging</span>'  </span># Network partitions
</span>    </span>testDuration </span>= </span>30.0
</span>    </span>swizzle </span>= </span>1  </span># Unclog in reversed order
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>Attrition</span>'  </span># Machine crashes
</span>    </span>testDuration </span>= </span>30.0
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>Rollback</span>'  </span># Proxy-to-TLog errors
</span>    </span>testDuration </span>= </span>30
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>ChangeConfig</span>'  </span># Database reconfigurations
</span>    </span>coordinators </span>= '</span>auto</span>'
</span></code></pre>
DiskFailureCycle.toml</strong> - Disk failures + bit flips during the Cycle workload:</p>
[configuration]
</span>minimumReplication </span>= </span>3
</span>minimumRegions </span>= </span>3
</span>
</span>[[test]]
</span>testTitle </span>= '</span>DiskFailureCycle</span>'
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>Cycle</span>'
</span>    </span>transactionsPerSecond </span>= </span>2500.0
</span>    </span>testDuration </span>= </span>30.0
</span>
</span>    [[test.workload]]
</span>    </span>testName </span>= '</span>DiskFailureInjection</span>'
</span>    </span>testDuration </span>= </span>120.0
</span>    </span>stallInterval </span>= </span>5.0
</span>    </span>stallPeriod </span>= </span>5.0
</span>    </span>throttlePeriod </span>= </span>30.0
</span>    </span>corruptFile </span>= </span>true
</span>    </span>percentBitFlips </span>= </span>10
</span></code></pre>
The simulation generates JSON trace logs in ./events/</code>. Parse them with fdb-sim-visualizer</a>.</p>
For more test examples, check FoundationDB's tests/</a> directory. Hundreds of workload combinations testing every corner of the system.</p>
🔗</a>Why I've Never Been Woken Up by FDB</h2>
After years of on-call and one trillion CPU-hours of simulation, I've never been woken up by FoundationDB. Now you know why.</p>
Interface swapping lets the same code run in both production and simulation. Flow actors enable single-threaded determinism. The event loop compresses years into seconds. BUGGIFY injects chaos into every corner of the codebase. SimulatedCluster builds entire distributed systems in memory. Workloads generate realistic transactions while chaos engines try to break everything. And deterministic randomness guarantees every bug can be reproduced, diagnosed, and fixed before shipping.</p>
The simulator has already broken FoundationDB in every possible way. Network partitions during coordinator elections. Machine crashes mid-transaction. Disks swapped between nodes on reboot. Bit flips. Slow I/O. Every edge case, every race condition, every distributed systems nightmare. Found, fixed, and verified before production ever sees it.</p>
Want to try breaking FoundationDB yourself?</strong> Grab a test config from above, run the simulator, inject chaos, and see if you can find a bug that survived one trillion CPU-hours. If you do, the FDB team would love to hear about it.</p>

Feel free to reach out with any questions or to share your simulation testing experiences or FDB workloads. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


From Arc to Box: One Deref Bound to Rule Them All
Pierre Zemb — Thu, 02 Oct 2025 00:00:00 +0000
While working on FoundationDB-rs</a>, I hit a design problem that seemed like it would require complex trait gymnastics. I had two transaction types with identical APIs but different ownership semantics, and I needed functions to accept both. The solution turned out to be embarrassingly simple. It was already implemented.</p>
🔗</a>The Problem: Two Transaction Types, One API</h2>
FoundationDB-rs has two transaction types that do exactly the same thing but with different ownership models:</p>
pub struct </span>Transaction {
</span>    </span>inner</span>: NonNull<fdb_sys::FDBTransaction>,
</span>    </span>metrics</span>: Option<TransactionMetrics>,
</span>}
</span>
</span>pub struct </span>RetryableTransaction {
</span>    </span>inner</span>: Arc<Transaction>,  </span>// Arc needed for retry loop ownership
</span>}
</span></code></pre>
Why two types?</strong> FoundationDB requires retry loops for handling conflicts and retriable errors. The Transaction</code> is perfect when you're managing retries manually or doing single-shot operations. The RetryableTransaction</code> wraps it in an Arc</code> so the automatic retry machinery in Database::run()</code> can clone references across async boundaries and exponential backoff delays.</p>
The challenge: users need to write code that works with both. Real FoundationDB applications mix both patterns depending on the use case.</p>
🔗</a>The Obvious Solutions Didn't Work</h2>
My first instinct was creating a trait. But FoundationDB-rs operates directly on raw C pointers (NonNull<fdb_sys::FDBTransaction></code>) with custom Future</code> implementations that handle FFI complexity and error mapping. Writing a trait with async methods that return these custom futures means associated types, lifetime bounds, and complex error handling. The resulting trait becomes painful to use and understand.</p>
I considered an enum wrapper:</p>
enum </span>AnyTransaction<'a> {
</span>    Regular(&</span>'a</span> Transaction),
</span>    Retryable(&</span>'a</span> RetryableTransaction),
</span>}
</span></code></pre>
But this felt wrong. Users would need to match everywhere, and it adds runtime overhead for what should be a compile-time decision. Plus it doesn't feel natural to use.</p>
🔗</a>The Accidental Solution</h2>
The RetryableTransaction</code> already had this implementation for convenience:</p>
impl </span>Deref </span>for </span>RetryableTransaction {
</span>    </span>type </span>Target = Transaction;
</span>    </span>fn </span>deref</span>(&</span>self</span>) -> &Transaction {
</span>        </span>self</span>.inner.</span>deref</span>()
</span>    }
</span>}
</span></code></pre>
I'd added this so users could call transaction methods directly on RetryableTransaction</code> instances. But this accidentally solved the entire design problem.</strong></p>
Functions can accept both types through a simple generic bound:</p>
async </span>fn </span>perform_operations</span><T>(</span>tx</span>: &T) -> FdbResult<()>
</span>where
</span>    T: Deref<Target = Transaction>,
</span>{
</span>    tx.</span>set</span>(</span>b</span>"</span>key</span>", </span>b</span>"</span>value</span>");
</span>    </span>let</span> value = tx.</span>get</span>(</span>b</span>"</span>key</span>", </span>false</span>).await?;
</span>    tx.</span>clear_range</span>(</span>b</span>"</span>start</span>", </span>b</span>"</span>end</span>");
</span>    Ok(())
</span>}
</span></code></pre>
Now the same function works seamlessly with both transaction types:</p>
// Direct transaction usage
</span>let</span> tx = db.</span>create_transaction</span>()?;
</span>perform_operations</span>(&tx).await?;
</span>
</span>// Automatic retry loop usage
</span>db.</span>run</span>(|</span>rtx</span>| async </span>move </span>{
</span>    </span>perform_operations</span>(&rtx).await?;  </span>// Same function, no changes needed!
</span>    Ok(())
</span>}).await?;
</span></code></pre>
The compiler handles everything through deref coercion. All methods of Transaction</code> remain directly accessible on both types, and there's zero runtime overhead.</p>
🔗</a>The Pattern: Arc + Deref = Universal APIs</h2>
This pattern works whenever you have a type T</code> and a wrapper containing Arc<T></code> (or Box<T></code>, Rc<T></code>, etc.). As long as the wrapper implements Deref<Target = T></code>, you can write generic functions that accept both:</p>
// Any function with this signature accepts:
</span>// - &T directly  
</span>// - &WrapperType where WrapperType: Deref<Target = T>
</span>// - &Arc<T>, &Box<T>, &Rc<T> (stdlib types already implement Deref)
</span>fn </span>use_any_version</span><D>(</span>val</span>: &D)
</span>where 
</span>    D: Deref<Target = T>,
</span>{
</span>    val.</span>some_method</span>();  </span>// All methods of T available through deref coercion
</span>}
</span></code></pre>
The key insight: when you're designing APIs that need to work with both T</code> and Arc<T></code>, don't reach for traits or enums. The standard library already solved this. Arc<T></code> implements Deref<Target = T></code>, and your custom wrapper types should do the same.</p>
Once you implement Deref</code>, any function that accepts &D where D: Deref<Target = T></code> automatically works with your owned type, your wrapper type, and any smart pointer containing your type. The compiler handles everything through deref coercion, and you get zero-cost abstraction that feels completely natural to use.</p>

Feel free to reach out with any questions or to share your experiences with Deref patterns in Rust. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


A Practical Guide to Application Metrics: Where to Put Your Instrumentation
Pierre Zemb — Wed, 24 Sep 2025 00:00:00 +0000
I keep having the same conversation with junior developers. They're building their first production service, and they ask: "Where should I put metrics in my application?" Then, inevitably: "What should I actually measure?"</p>
After mentoring dozens of engineers and running distributed systems for years, I've learned these aren't just beginner questions. Even experienced developers struggle with metrics placement because most of us learned observability as an afterthought, not as a core design principle.</p>
I've been on both sides: deploying services with no metrics and scrambling at 3 AM to understand what broke, and also building comprehensive monitoring that caught issues before users noticed. The difference isn't just about sleep quality; it's about building systems you can actually operate with confidence.</p>
This post gives you a practical framework for where to instrument your applications. No theory, just patterns I've learned from years of production incidents.</p>
🔗</a>The Five Essential Metric Types</h2>

Quick note on naming:</strong> Throughout this post, I use dots (.</code>) as metric separators like api.requests.total</code>. This works perfectly for us because we're heavy Warp 10</a> users, and Warp 10 handles dots beautifully. If you're using Prometheus or other systems that prefer underscores, just replace the dots with underscores (api_requests_total</code>). The patterns remain the same!</p>
</blockquote>
All useful application metrics fall into five categories. Understanding these helps you decide what to instrument and where:</p>
1. Operational Counters</strong> track discrete events in your system. Every time something happens (a request arrives, a job finishes, an error occurs), you increment a counter. The most critical insight here is measuring both success and failure paths. Most developers remember to count successful operations but forget the errors, leaving them blind when things break. Examples include api.requests.total</code>, db.queries.executed</code>, auth.failures.count</code>, payments.declined.count</code>, jobs.started</code>, and cache.evictions</code>. Always include labels like method</code>, endpoint</code>, error_type</code> to provide context.</p>
2. Resource Utilization</strong> answers "how much of X am I using right now?" These are your early warning system for capacity problems. Track current values with gauges, cumulative usage with counters. The key is monitoring resources before they're completely exhausted. A connection pool might support 100 connections, but if 95 are active, you're in trouble. Monitor memory.used.bytes</code>, db.connections.active</code>, cache.size.entries</code>, thread_pool.active_threads</code>, and disk.space.available.bytes</code>. Watch for patterns like steadily increasing memory usage or connection counts approaching pool limits.</p>
3. Performance and Latency</strong> shows how fast (or slow) things are running. Users feel latency immediately, making these often your most-watched dashboards. Always include units in metric names (.ms</code>, .seconds</code>, .bytes</code>) to make dashboards self-documenting. Track api.response_time.ms</code>, db.query.duration.ms</code>, jobs.processing_time.seconds</code>, and external_api.call.duration.ms</code>. Monitor percentiles (p50, p95, p99) not just averages: a 1ms average with a 5-second p99 indicates serious problems.</p>
4. Data Volume and Throughput</strong> tracks data flow through your system. These metrics are crucial for capacity planning and spotting bottlenecks before they cause user-visible problems. Monitor both input and output rates to understand processing efficiency. Focus on queue.messages.consumed</code>, network.bytes.sent</code>, database.rows.processed</code>, file_processor.files.completed</code>, and batch_processor.records.per_batch</code>. Compare input vs output rates to identify accumulating backlogs.</p>
5. Business Logic</strong> captures domain-specific metrics that relate to your actual business value. These are often the most valuable metrics for understanding how your application is really being used and whether technical problems are affecting business outcomes. Track orders.placed</code>, users.registered</code>, searches.executed</code>, documents.uploaded</code>, and subscriptions.activated</code>. Don't underestimate these: they're what your executives care about and often reveal problems that technical metrics miss.</p>
Type</th> Examples</th> Key Insight</th></tr></thead>

Operational</td> api.requests.total</code>, auth.failures</code></td> Track success AND failure paths</td></tr>
Resource</td> memory.used.bytes</code>, db.connections.active</code></td> Early warning for capacity issues</td></tr>
Performance</td> api.response_time.ms</code>, db.query.duration.ms</code></td> Users feel latency immediately</td></tr>
Throughput</td> queue.messages.consumed</code>, network.bytes.sent</code></td> Understand data flow patterns</td></tr>
Business</td> orders.placed</code>, users.login</code></td> What executives actually care about</td></tr>
</tbody></table>
🔗</a>Where to Instrument: A Component Guide</h2>
🔗</a>API Endpoints and HTTP Requests</h3>
Your application's front door deserves comprehensive monitoring. Every HTTP request tells a story from arrival to completion, and you want to capture that entire narrative, not just the happy path. Track api.requests.total</code> with labels for method</code>, endpoint</code>, and status_code</code> to understand usage patterns. Monitor api.response_time.ms</code> to show user experience, and api.errors.count</code> with error_type</code> labels to reveal reliability issues. Include auth.failures.count</code> with failure_reason</code> to catch security problems, and api.concurrent_requests</code> to identify when you're approaching capacity limits. The common mistake is only instrumenting successful requests; the real value comes from measuring what happens when things go wrong: network timeouts, validation errors, service dependencies failing.</p>
🔗</a>Database Layer</h3>
Database calls are often your biggest bottleneck and cause more production incidents than any other component. For connection management, track db.connections.active</code> (critical for pool management), db.connections.idle</code> (available connections), and db.connections.wait_time.ms</code> (time threads wait for connections). Monitor query performance with db.queries.executed</code> (including operation_type</code> and table</code> labels), db.query.duration.ms</code> (with percentile tracking), db.slow_queries.count</code> (queries exceeding thresholds), and db.query.rows_affected</code> (rows returned or modified). For error monitoring, track db.errors.count</code> by error_type</code> (timeout, deadlock, constraint violation) and db.connection_errors.count</code> for connection failures. Connection pool exhaustion is a classic way to kill your entire application: if you support 100 connections and 95 are active, you're in danger. Start alerting when you hit 85% utilization.</p>
🔗</a>Message Queues and Background Processing</h3>
Message queues often hide subtle bugs that manifest as slowly growing delays or stuck processing. Track data flow in both directions to catch issues early. For producers, monitor queue.messages.produced</code> (with topic</code> and producer_id</code> labels), queue.messages.failed</code> (with detailed error_type</code> labels), queue.producer.wait_time.ms</code> (time waiting for producer availability), and queue.batch_size</code> (messages sent per batch). For consumers, track queue.messages.consumed</code> (successfully processed), queue.processing.time.ms</code> (per-message duration), queue.processing.errors</code> (failures with error_type</code> and recovery action), jobs.queue.depth</code> (messages waiting), and consumer.lag.ms</code> (how far behind real-time). Background jobs need additional metrics: jobs.started</code>, jobs.completed</code>, jobs.failed</code> (with failure reason), jobs.retry.count</code>, and jobs.execution_time.seconds</code>. Growing queue depth usually means you're processing jobs slower than they're being created, leading to increasing delays and eventual system overload. Consumer lag helps you understand if you're keeping up with real-time processing needs.</p>
🔗</a>Caching and Locks</h3>
For cache performance, track cache.requests.total</code> (with operation</code> labels for get, set, delete), cache.hits</code> and cache.misses</code> (for calculating hit ratio), cache.size.entries</code> (current cached items), cache.size.bytes</code> (memory usage), cache.evictions</code> (items removed with eviction_reason</code>), and cache.operation.duration.ms</code> (time for operations). Hit ratio below 80% usually indicates problems: either you're caching the wrong things, cache TTL is too short, or your working set exceeds cache capacity.</p>
For lock and synchronization, monitor locks.acquire.duration.ms</code> (time from requesting to getting lock), locks.held.duration.ms</code> (how long locks are held), locks.contention.count</code> (threads waiting), and locks.timeouts.count</code> (failed acquisitions within timeout). Lock contention can kill your entire application, but it stays invisible without metrics. I've debugged more performance issues with lock metrics than almost any other single type. High acquisition times mean contention; long hold times suggest you're doing too much work while holding the lock.</p>
🔗</a>Real-World Instrumentation Patterns</h2>
🔗</a>The Request Lifecycle Pattern</h3>
For every user-facing operation, track the complete journey from entry to exit. This means instrumenting not just the success path, but every branch your code can take. Increment api.requests.received</code> the moment a request hits your service, track auth.attempts.count</code> and auth.failures.count</code> separately to show both volume and failure rate, monitor authorization.decisions.count</code> with labels for granted</code> vs denied</code>, measure business_logic.duration.ms</code> to isolate your application logic performance, and record final response.status_code</code> distribution to understand your error patterns. Most developers instrument the happy path but forget edge cases. A request that fails authentication never reaches your business logic, but it still uses resources and affects user experience. The real value comes from measuring what happens when things go wrong: network timeouts, validation errors, service dependencies failing.</p>
🔗</a>The Resource Exhaustion Pattern</h3>
Systems fail when they run out of resources. The trick is measuring resources before they're completely exhausted, giving you time to react. For connection pools, track db.connections.active</code> vs db.connections.max</code> and don't wait until you hit 100% utilization: start alerting at 85%. Monitor db.connections.wait_time.ms</code> because long waits indicate you're close to exhaustion even if you haven't hit the limit. For memory pressure, monitor both memory.heap.used.bytes</code> and gc.frequency.per_minute</code> since high GC frequency often predicts memory pressure before OutOfMemory errors occur. Track memory.allocation.rate.bytes_per_second</code> to understand if your allocation rate is sustainable. For queue management, a growing jobs.queue.depth</code> indicates you're processing work slower than it arrives, eventually leading to timeouts and system overload. Track queue.processing.rate.per_second</code> and queue.arrival.rate.per_second</code>: the relationship between these rates tells you if you're keeping up. For disk space, track disk.available.bytes</code> and disk.usage.rate.bytes_per_hour</code>. Linear growth can be predicted and prevented, while sudden spikes indicate immediate problems.</p>
🔗</a>The Business Context Pattern</h3>
Technical metrics tell you what</em> is happening; business metrics tell you why</em> it matters. Always pair technical instrumentation with business context to understand the real impact of technical problems. Track api.errors.count</code> alongside orders.lost.count</code> to understand how technical problems affect sales, monitor payment_service.response_time.ms</code> alongside checkout.abandonment.rate</code> to see if slow payments drive users away, and measure search.response_time.ms</code> alongside search.result_clicks.count</code> to understand if slow search reduces engagement. For user experience correlation, pair cache.misses.count</code> with page.load.time.ms</code> to quantify cache performance impact, track db.slow_queries.count</code> alongside user.session.duration.minutes</code> to see if database performance affects user retention, and monitor auth.failures.count</code> with support.tickets.count</code> to predict support load from technical issues. For capacity planning, correlate server.cpu.usage.percent</code> with concurrent.users.count</code> to understand scaling requirements, track memory.usage.bytes</code> alongside active.sessions.count</code> to predict memory needs, and monitor network.bandwidth.used.mbps</code> with file.uploads.count</code> to plan infrastructure scaling. This pairing helps you understand the business impact of technical problems and prioritize fixes based on actual user and revenue impact.</p>
🔗</a>The Error Classification Pattern</h3>
Not all errors are created equal. Classify errors by their impact and actionability to build appropriate response strategies. User errors (4xx) like auth.invalid_credentials</code>, validation.missing_field</code>, or resource.not_found</code> are usually not your fault, but track patterns to identify UX issues. High rates might indicate confusing interfaces or inadequate client-side validation; alert on unusual spikes that might indicate attacks or system confusion. System errors (5xx) like db.connection_timeout</code>, service.unavailable</code>, or memory.exhausted</code> are your responsibility to fix immediately. They're always actionable and usually indicate infrastructure or code problems that should trigger immediate alerts and investigation. External dependency errors like payment_gateway.timeout</code>, third_party_api.rate_limited</code>, or cdn.unavailable</code> are outside your direct control but affect users. They require fallback strategies and user communication, and help predict when to escalate with external providers. Distinguish transient errors (network.timeout</code>, rate_limit.exceeded</code> that often resolve themselves) from persistent errors (config.invalid</code>, database.schema_mismatch</code> that require immediate intervention). Each category needs different alerting strategies, escalation procedures, and response timeframes: user errors might warrant daily review, system errors need immediate alerts, external errors require monitoring trends and fallback activation.</p>
🔗</a>Best Practices for Production Metrics</h2>
🔗</a>Naming Conventions That Scale</h3>
Consistent naming prevents the confusion that kills metrics adoption. Use a clear hierarchy: <system>.<component>.<operation>.<metric_type></code>. Examples include api.auth.requests.count</code>, db.user_queries.duration.ms</code>, cache.metadata.hits.total</code>, and queue.order_processing.messages.consumed</code>. Standardize your suffixes: .count/.total</code> for event counters, .current/.active</code> for current gauge values, .duration/.ms/.seconds</code> for time measurements, .bytes/.mb/.gb</code> for data volume, .errors/.failures</code> for error counters, and .ratio/.rate</code> for ratios and rates. Avoid mixing naming styles (requestCount</code> vs request_total</code>), ambiguous units (response_time</code> without units), and inconsistent hierarchies (api_requests</code> vs requests.api</code>). This consistency pays off during 3 AM troubleshooting when you don't want to waste mental energy remembering naming schemes.</p>
🔗</a>Critical Mistakes to Avoid</h3>
The biggest mistake is using unbounded values as labels. Don't tag metrics with user IDs, session tokens, IP addresses, or other unlimited values; your metrics system will eventually explode from too many unique series. Use api.requests{user_type="premium", region="us-west"}</code> instead of api.requests{user_id="12345", session="abc123xyz"}</code>. Always instrument failure cases, not just success paths. Track both payments.succeeded</code> AND payments.failed</code> with error type labels, monitor auth.attempts</code> alongside auth.failures</code> to understand failure rates, and count file.uploads.completed</code> and file.uploads.failed</code> to see processing reliability. Every metric has a cost in storage, network bandwidth, and cognitive load. If you can't explain why a metric matters for operations or business decisions, skip it. Ask yourself: "Would this metric help me during an incident?" Metrics that stop updating can be worse than no metrics at all. Always include heartbeat or health check metrics to verify your instrumentation is working; track metrics.last_updated.timestamp</code> to detect collection failures.</p>
🔗</a>A Note on Histograms (Or: Why They're Not Here)</h2>
I know what some of you are thinking: "Where are the histograms?" After all, this is a comprehensive guide to application metrics, and histograms are everywhere in monitoring discussions. Well, I deliberately left them out, and here's why.</p>
Prometheus histograms are fundamentally broken in ways that make them more dangerous than useful. The core problem is what I call the bucket pre-configuration paradox: you must define bucket boundaries before you know your data distribution. As LinuxCzar eloquently put it in his "tale of woe"</a>, this creates an impossible choice between accuracy (many buckets) and operability (few buckets). Get it wrong, and you either lose precision or crash your Prometheus server with cardinality explosion</a>.</p>
But the problems run deeper. You mathematically cannot aggregate percentiles</a> across instances because the underlying event data is lost. The linear interpolation algorithm produces significant estimation errors</a>, and Prometheus's scraping architecture introduces data corruption</a> where histogram buckets update inconsistently. The operational burden</a> never ends: every performance improvement potentially invalidates your bucket choices, forcing constant manual reconfiguration.</p>

Pierre Chapuis</a> pointed out</a> the root cause I missed: Prometheus implements an outdated 2005 algorithm from Cormode et al. for histogram summaries and quantiles. There are much better algorithms available now, including improved versions from the same authors. Check out this paper</a> for a good overview of modern sketch algorithms. The estimation errors and operational problems I described are symptoms of using this old algorithm.</p>
</blockquote>
Instead, I prefer the combination of simple counters and gauges paired with distributed tracing. Trace-derived global metrics give you actual data distributions without guessing bucket boundaries, eliminate the aggregation problem by preserving request context, and adapt automatically as your system evolves. You get better insights with less operational overhead, which seems like a better deal to me.</p>
🔗</a>Quick Reference: Metrics by Component</h2>
Component</th> Essential Metrics</th> Purpose</th> Key Labels</th></tr></thead>

API Endpoints</strong></td> api.requests.total</code>
api.response_time.ms</code>
api.errors.count</code>
auth.failures.count</code></td> Track every request lifecycle
Monitor user-facing performance
Catch errors before users complain
Security monitoring</td> method</code>, endpoint</code>, status_code</code>
endpoint</code>, user_type</code>
error_type</code>, endpoint</code>
failure_reason</code></td></tr>
Database</strong></td> db.connections.active</code>
db.queries.executed</code>
db.query.duration.ms</code>
db.errors.count</code>
db.slow_queries.count</code></td> Prevent connection exhaustion
Track database usage patterns
Identify performance bottlenecks
Monitor database health
Catch expensive queries</td> pool_name</code>
operation_type</code>, table</code>
operation_type</code>, table</code>
error_type</code>, operation</code>
table</code>, query_type</code></td></tr>
Message Queues</strong></td> queue.messages.produced</code>
queue.messages.consumed</code>
queue.processing.time.ms</code>
queue.processing.errors</code>
jobs.queue.depth</code></td> Track producer health
Monitor consumer throughput
Identify processing bottlenecks
Catch processing failures
Detect backlog buildup</td> topic</code>, producer_id</code>
topic</code>, consumer_group</code>
topic</code>, message_type</code>
error_type</code>, topic</code>
queue_name</code>, priority</code></td></tr>
Cache</strong></td> cache.requests.total</code>
cache.hits</code>
cache.misses</code>
cache.size.entries</code>
cache.evictions</code></td> Monitor cache usage
Track cache effectiveness
Identify cache problems
Monitor memory usage
Understand eviction patterns</td> cache_type</code>, operation</code>
cache_type</code>, key_prefix</code>
cache_type</code>, miss_reason</code>
cache_type</code>
cache_type</code>, eviction_reason</code></td></tr>
Locks</strong></td> locks.acquire.duration.ms</code>
locks.held.duration.ms</code>
locks.contention.count</code></td> Detect lock contention
Find locks held too long
Monitor thread blocking</td> lock_name</code>, thread_type</code>
lock_name</code>, operation</code>
lock_name</code></td></tr>
Business</strong></td> orders.placed</code>
users.login</code>
payments.processed</code>
feature.usage.count</code>
workflow.state_changes</code></td> Business KPI tracking
User activity monitoring
Revenue stream health
Feature adoption metrics
Process flow monitoring</td> user_type</code>, order_value_range</code>
user_type</code>, login_method</code>
payment_method</code>, amount_range</code>
feature_name</code>, user_segment</code>
workflow_name</code>, from_state</code>, to_state</code></td></tr>
</tbody></table>
🔗</a>Conclusion</h2>
Effective metrics instrumentation isn't about collecting everything; it's about collecting the right things in the right places. Start with the five essential metric types, instrument your critical components, and build from there.</p>
Think about metrics as part of your application design, not an afterthought. When writing code, ask yourself: "How will I know if this is working correctly in production?" The answer guides your instrumentation decisions.</p>
The best observability system helps you sleep better at night. If your metrics aren't giving you confidence in your system's health, you're measuring the wrong things.</p>

Feel free to reach out with any questions or to share your experiences with application metrics. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Testing: prevention vs discovery
Pierre Zemb — Mon, 08 Sep 2025 00:00:00 +0000
While working on moonpool</a>, my hobby project for studying and backporting FoundationDB's low-level engineering concepts (actor model, deterministic simulation, network fault injection), Claude Code did something remarkable: it found a bug I didn't know existed on its own. Not through traditional testing, but through active exploration of edge cases I hadn't considered.</p>

</p>
</div>
Claude identified a faulty seed triggering an edge case, debugged it locally using deterministic replay, and added it to the test suite. All by itself. 🤯 This wasn't prevention but discovery.</strong> It's time to shift our testing paradigm from preventing regressions to actively discovering unknown bugs.</p>
🔗</a>Building for Discovery</h2>
The difference between prevention and discovery isn't just philosophical but requires completely different system design. Moonpool was built from day one around three principles that enable active bug discovery:</p>
Deterministic simulation</strong>: Every execution is completely reproducible. Given the same seed, the system makes identical decisions every time. This changes debugging from "I can't reproduce this" to "let me replay exactly what happened." More importantly, it lets LLMs explore the state space step by step without getting lost in non-deterministic noise.</p>
Controlled failure injection</strong>: Built-in mechanisms intentionally introduce failures in controlled, reproducible ways. This includes timed failures like network delays and disconnects, plus "buggify" mechanisms</a> that inject faulty internal state at strategic points in the code. Each buggify point is either enabled or disabled for an entire simulation run, creating consistent failure scenarios instead of random chaos. Instead of waiting for production to reveal edge cases, we force the system to encounter dangerous, bug-finding behaviors during development.</p>
Observability through sometimes assertions</strong>: Borrowed from Antithesis</a>, these verify we're actually discovering the edge cases we think we're testing. Here's what they look like:</p>
// Verify that server binds sometimes fail during chaos testing
</span>sometimes_assert!(
</span>    server_bind_fails,
</span>    </span>self</span>.bind_result.</span>is_err</span>(),
</span>    "</span>Server bind should sometimes fail during chaos testing</span>"
</span>);
</span>
</span>// Ensure message queues sometimes approach capacity under load
</span>sometimes_assert!(
</span>    peer_queue_near_capacity,
</span>    state.send_queue.</span>len</span>() >= (</span>self</span>.config.max_queue_size as </span>f64 </span>* </span>0.8</span>) as </span>usize</span>,
</span>    "</span>Message queue should sometimes approach capacity limit</span>"
</span>);
</span></code></pre>
Traditional code coverage only tells you "this line was reached." Sometimes assertions verify "this interesting scenario actually happened." If a sometimes assertion never triggers across thousands of test runs, you know you're not discovering the edge cases that matter.</p>
These three elements shift testing from prevention to discovery. Instead of developers writing tests for scenarios they already know about, the system forces them to hit failure modes they haven't thought of. For Claude, this meant it could explore the state space step by step, understanding not just what the code does, but what breaks it.</p>
🔗</a>The Chaos Environment</h2>
Moonpool is currently limited to simulating TCP connections through its Peer abstraction, but even this narrow scope creates a surprisingly rich failure environment. Here's what the chaos testing configuration looks like (borrowed from TigerBeetle's approach):</p>
impl </span>NetworkRandomizationRanges {
</span>    </span>/// Create chaos testing ranges with connection cutting enabled for distributed systems testing
</span>    </span>pub fn </span>chaos_testing</span>() -> </span>Self </span>{
</span>        </span>Self </span>{
</span>            bind_base_range: </span>10</span>..</span>200</span>,                       </span>// 10-200µs
</span>            bind_jitter_range: </span>10</span>..</span>100</span>,                     </span>// 10-100µs
</span>            accept_base_range: </span>1000</span>..</span>10000</span>,                 </span>// 1-10ms in µs
</span>            accept_jitter_range: </span>1000</span>..</span>15000</span>,               </span>// 1-15ms in µs
</span>            connect_base_range: </span>1000</span>..</span>50000</span>,                </span>// 1-50ms in µs
</span>            connect_jitter_range: </span>5000</span>..</span>100000</span>,             </span>// 5-100ms in µs
</span>            read_base_range: </span>5</span>..</span>100</span>,                        </span>// 5-100µs
</span>            read_jitter_range: </span>10</span>..</span>200</span>,                     </span>// 10-200µs
</span>            write_base_range: </span>50</span>..</span>1000</span>,                     </span>// 50-1000µs
</span>            write_jitter_range: </span>100</span>..</span>2000</span>,                  </span>// 100-2000µs
</span>            clogging_probability_range: </span>0.1</span>..</span>0.3</span>,           </span>// 10-30% chance of temporary network congestion
</span>            clogging_base_duration_range: </span>50000</span>..</span>300000</span>,    </span>// 50-300ms congestion duration in µs
</span>            clogging_jitter_duration_range: </span>100000</span>..</span>400000</span>, </span>// 100-400ms additional congestion variance in µs
</span>            cutting_probability_range: </span>0.10</span>..</span>0.20</span>,          </span>// 10-20% cutting chance per tick
</span>            cutting_reconnect_base_range: </span>200000</span>..</span>800000</span>,   </span>// 200-800ms in µs
</span>            cutting_reconnect_jitter_range: </span>100000</span>..</span>500000</span>, </span>// 100-500ms in µs
</span>            cutting_max_cuts_range: </span>1</span>..</span>3</span>,                   </span>// 1-2 cuts per connection max (exclusive upper bound)
</span>        }
</span>    }
</span>}
</span></code></pre>
Even with just TCP simulation, this creates a hostile environment where connections randomly fail, messages get delayed, and network operations experience unpredictable latencies. Each seed represents a different combination of timing and probability, creating unique failure scenarios.</p>
🔗</a>Why Even Simple Network Code Needs Chaos</h3>
You might think testing a simple peer implementation with fault injection is overkill, but production experience and research show otherwise. "An Analysis of Network-Partitioning Failures in Cloud Systems"</a> (OSDI '18) studied real-world failures and found:</p>

80%</strong> of network partition failures have catastrophic impact</li>
27%</strong> lead to data loss (the most common consequence)</li>
90%</strong> of these failures are silent</li>
21%</strong> cause permanent damage that persists even after the partition heals</li>
83%</strong> need three additional events to manifest</li>
</ul>
That last point is crucial; exactly the kind of complex interaction that deterministic simulation with fault injection helps uncover.</p>
My peer implementation only does simple ping-pong communication, yet it still took some work to make it robust enough to pass all the checks and assertions. It's enough complexity for Claude to discover edge cases in connection handling, retry logic, and recovery mechanisms.</p>
The breakthrough wasn't that Claude wrote perfect code but that Claude could discover and explore failure scenarios I hadn't thought to test, then use deterministic replay to debug and fix what went wrong.</strong></p>
🔗</a>The Paradigm Shift</h2>
The difference between prevention and discovery completely changes how we think about software quality. Prevention testing asks "did we break what used to work?" Discovery testing asks "what else is broken that we haven't found yet?"</strong></p>
This shift creates a powerful feedback loop for young engineers and LLMs alike. Both developers and LLMs learn what production failure really looks like, not the sanitized version we imagine. When Claude can explore failure scenarios step by step and immediately see the results through sometimes assertions, it becomes a discovery partner that finds edge cases human intuition misses.</p>
This isn't theoretical. It's working in my hobby project today. Moonpool is definitely hobby-grade, but if a side project can enable LLM-assisted bug discovery, imagine what's possible with production systems designed from the ground up for deterministic testing.</p>
The FoundationDB</a>, TigerBeetle</a>, and Antithesis</a> communities have been practicing discovery-oriented testing for years. FoundationDB's legendary reliability comes from exactly this approach; deterministic simulation that actively hunts for bugs rather than just preventing regressions. After operating FoundationDB in production for 3 years, I can confirm it's by far the most robust and predictable distributed system I've encountered. Everything behaves exactly as documented, with none of the usual distributed systems surprises. I've written more about these ideas in my posts on simulation-driven development</a> and notes about FoundationDB</a>.</p>
What's new is that LLMs can now participate in this process.</strong> Through deterministic simulation and sometimes assertions, we're not just telling the LLM "write good code" but showing it exactly what production failure looks like. If you're curious about production-grade implementations of these ideas, check out Antithesis</a>; their best hidden feature is that it works on any existing system without requiring a rewrite.</p>
The tools exist. The techniques are proven. Testing must evolve from prevention to discovery.</strong> The future isn't about writing better test cases but about building systems that actively reveal their own bugs.</p>

Feel free to reach out with any questions or to share your experiences with deterministic testing. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Shipped vs. Operated, or How Many Bash Scripts Does It Take?
Pierre Zemb — Mon, 18 Aug 2025 00:00:00 +0000

Summary:</strong> The difference between shipped and operated software is the difference between something you can run and forget, and something that demands ongoing, hands-on care. Choosing the former protects your team’s focus and sanity.</p>
</blockquote>
🔗</a>The Shipped vs. Operated Spectrum</h2>
Some technologies arrive as complete systems: you deploy them, give them minimal care, and they quietly do their job. Others arrive like complex machines: powerful, but demanding regular attention and maintenance. That’s the difference between shipped</em> and operated</em>.</p>
The distinction isn’t just about features; it’s about the level of operational effort the system will demand over its lifetime. Operated</strong> technologies require continuous human care to stay healthy. They age, drift, and accumulate operational quirks. They often have sharp edges you only discover at 2 a.m., and when something goes wrong, you need people who already know the failure modes by heart. Think of a self-managed HBase</strong> or a ZooKeeper ensemble that you really</em> hope never splits brain.</p>
Shipped</strong> technologies are built to reduce that constant overhead. They can still fail, but they tend to fail in ways that are predictable, recoverable, and not existential. You can learn them as you go. Your outages will be frustrating, but they won’t demand a dedicated handler on payroll. FoundationDB</strong> is a good example: it’s not magic, but its operational surface area is small enough to fit in a single human brain.</p>
For contrast, I’ve also spent years with the other kind: HBase</strong> clusters spread over 250+ nodes, Ceph</strong>, Kafka</strong> and ZooKeeper</strong> in various configurations, Pulsar</strong>, Warp10</strong>, etcd</strong>, Kubernetes</strong>, Flink</strong>, and RabbitMQ</strong>, each with its own set of operational “adventures.”</p>
🔗</a>Identifying Operated Systems</h2>
Some systems live in both worlds depending on how you use them. PostgreSQL</strong> in standalone mode is usually shipped: it’s simple to run, predictable, and rarely causes surprises. But under certain conditions, like fighting vacuum performance at scale or running it in HA mode under sustained heavy load, it shifts into operated territory. The difference isn’t in the codebase, but in the demands your use case puts on it.</p>
A quick way to tell which camp your system belongs to is the Bash Script Test</strong>: ask how many bash scripts or home-grown tools are required to survive an on-call shift. If the answer includes a collection of automation to clean up data, shuffle it between nodes, or probe the cluster’s health, you’re probably in operated territory. I’ve been there: running hbck</code> and manually moving regions in HBase</strong>, shuffling partitions around in Kafka</strong> to balance load, or triggering repairs in Ceph</strong> after failed scrub errors. Many distributed systems quietly rely on these manual interventions, often run weekly, to stay healthy, and that’s an operational cost you can’t ignore.</p>
By contrast, we have no</strong> such scripts for FoundationDB</strong>, and that’s exactly why it feels shipped.</p>
🔗</a>The Strategic Cost of Operations</h2>
Each operated system consumes a slice of your team’s focus. Add too many, and you’ll spend more time keeping the lights on than moving forward. The more you can choose robust, low-maintenance software, the more space you keep for actually building new things.</p>
I’m not a fan of Kubernetes from an operational perspective. But it does something important for end users: it gives them a standard way to write software that reacts to the state of the infrastructure through Operators</a>. Operators turn that into continuous automation, with a reconciliation loop that keeps drifting systems aligned with the desired state. It’s a way to bake SRE knowledge into code, so even complex systems can be run and handed over without months of hand-holding.</p>
The stakes are only going to get higher as LLMs become a common tool for software engineers. We’ll inevitably build more advanced and complex systems, but that complexity doesn’t disappear; it gets pushed to the people on call. LLMs are good at fixing failures that are reproducible and deterministic, because they can alter the system freely, but most on-call incidents aren’t like that. The only way to keep operational load sustainable is to change how we design and test: building for robustness from the start, and using techniques like simulation-driven development</a> to expose failure modes before they reach production.</p>
If you can, choose the system you can deploy and leave alone, not the complex machine that demands your weekends.</p>

Feel free to reach out with any questions or to share your experiences with shipped/operated software. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Two Podcast Episodes on Topics Developers Rarely Talk About
Pierre Zemb — Mon, 11 Aug 2025 00:00:00 +0000
I was listening to a couple of podcasts the other day and stumbled across two episodes that were so compelling I had to stop my chores and listen. They dive into corners of software engineering that most developers barely think about; not because they’re unimportant, but because they appear in the hard corners of engineering:</p>

catastrophic data corruption,</li>
correctness work done before a single line is shipped.</li>
</ul>
The first is Adventures in Data Corruption</a> from Oxide and Friends</em>. Two years ago, the Oxide team ran into data corruption during what should have been a routine network transfer. The debugging journey that followed went from packet traces to CPU speculation quirks, peeling back the stack layer by layer, hardware, kernel, network, application, asking hard questions at each step. What I love here is the combination of clear storytelling and the rapid-fire hypotheses: they make an assumption, test it, discard it, and immediately move to the next, pulling you along in the investigation until the root cause finally clicks into place.</p>
The second is Scaling Correctness: Marc Brooker on a Decade of Formal Methods at AWS</a> of The BugBash Podcast</em> by Antithesis. Marc Brooker, who has spent nearly 17 years building core AWS services like S3 and Lambda, shares the company’s decade-long journey with formal methods, from heavyweight tools like TLA+ to the lightweight</em> approaches that any team can adopt like simulation-based testing</a>. At AWS, they’ve learned that investing in correctness up front not only improves reliability but actually speeds up delivery. They also touch on deterministic simulation testing, the challenge of verifying UIs and control planes, and the role AI might play in the future of verification.</p>
I’ve been paged way too many times for metastable failures, data corruption, network meltdowns, or NTP drift in production. These days, I’d rather tackle the correctness part before</em> those alarms go off. Every new layer I build is designed to be simulated to explore failure modes in a controlled environment before they can hurt real users.</p>
But when things fall apart anyway, and spoilers they will</strong>, developers have the opportunity to truly understand their software. Being responsible for the systems you build means you’re the one getting paged, and it’s in those moments of crisis that the sharpest debugging skills are forged.</p>
So don’t just bookmark them. Put them at the top of your queue. Listen. And maybe, the next time your system misbehaves, you’ll be ready.</p>

Feel free to reach out with any questions or to share your experiences with debugging and correctness. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Three Years of Nix and NixOS: The Good, the Bad, and the Ugly
Pierre Zemb — Wed, 02 Jul 2025 00:37:27 +0100
For years, I was a serial distro-hopper, working my way through Ubuntu, Arch, Gentoo, Exherbo, Void Linux, Fedora, Pop!_OS, and Manjaro. Every few months, a new Linux distribution would catch my eye, and I’d spend a weekend migrating my setup, hoping to find the perfect fit. That cycle broke three years ago when I switched to NixOS. It has since become the foundation for all my Linux machines, not because it’s perfect, but because it fundamentally changes the contract between the user and the operating system.</p>
It's important to distinguish between Nix</strong>, the powerful package manager that can run on any Linux distro (and even macOS), and NixOS</strong>, the full immutable operating system built around it. This post is a review of my three years with both—the good, the bad, and the ugly.</p>
🔗</a>The Good</h2>
🔗</a>Declarative and Atomic System Management on NixOS</h3>
The core promise of NixOS is that your entire system is configured from a set of files, which you can store in a Git repository. Every change is a commit, giving you a complete, auditable history of your system's state. This makes setting up a new machine trivial: I clone my repository, run one command, and my entire setup is replicated perfectly. No more manually copying dotfiles or running install scripts.</p>
This declarative approach also makes the system incredibly robust. I once broke a laptop running Exherbo right before an on-call shift, and it was a nightmare to fix. With NixOS, that fear is gone. Every nixos-rebuild switch</code> creates a new "generation" of the system. If an update breaks something, you simply reboot and select the previous generation from the boot menu. This atomic update mechanism makes you fearless about making and testing changes.</p>
🔗</a>System Crafting as a First-Class Citizen</h3>
On NixOS, customizing your system is not an afterthought—it's a core feature. While the Nix package manager gives you fine-grained control over packages, NixOS uses this power to make deep system modifications simple. For example, building a custom ISO with your SSH keys pre-installed is just a few lines of configuration. This philosophy extends to packages: you can use pre-built binaries for most things, but easily build a package from source with your own patches when you need to.</p>
🔗</a>Sandboxed Development Environments</h3>
A powerful feature of Nix</strong> (the package manager) is the ability to define per-project development environments using a flake.nix</code> file. When you enter the project directory, direnv</code> can automatically load a shell with all the specific tools and libraries you need for that project—a specific version of Rust, Node.js, or any other dependency. This completely solves the problem of conflicting dependencies between projects. Each project is perfectly isolated, and you can be sure that you and your colleagues are using the exact same environment.</p>
My favorite tip is to add if has nix; then use nix; fi</code> to the .envrc</code> file, so the environment is only loaded for team members who have Nix installed, avoiding errors for everyone else.</p>
🔗</a>Built-in VM-Based Testing</h3>
A great, underrated NixOS</strong> feature is the built-in testing framework. You can write tests that spin up lightweight virtual machines with their own configurations to test your setup. I saw this firsthand when I recently packaged fdbserver</code>. It took me about 30 minutes to get a test running that spins up a full FoundationDB cluster across multiple VMs. The setup is still basic—it doesn't even use systemd—but it was more than enough to validate the packaging. You can see the test definition here</a>. Being able to build that kind of complex integration test so quickly is something I've only found in NixOS.</p>
🔗</a>The Bad</h2>
🔗</a>The Friction of Simple Changes on NixOS</h3>
On a normal system, if you want to add a shell alias, you edit .bashrc</code> and you're done. In NixOS, there are no quick edits. You have to find the right option in your configuration, add the line, and then rebuild your system. This is great for keeping your configuration tracked, but it adds a lot of friction to simple tasks.</p>
🔗</a>A Steep and Isolated Learning Curve</h3>
Learning the Nix ecosystem is a big commitment. The ideas are very different from other Linux systems, so your existing knowledge doesn't help much. You have to learn the Nix language, how derivations work, and now Flakes. It takes a few months before you feel productive.</p>
🔗</a>Incompatibility with the Wider Ecosystem</h3>
Because NixOS doesn't use the standard Filesystem Hierarchy Standard (FHS), you can't just download a pre-compiled binary and expect it to work. It will fail to run because it can't find its shared libraries in places like /lib</code> or /usr/lib</code>. The Nix way to solve this is to use patchelf</code> to modify the binary and tell it where to find its dependencies inside the /nix/store</code>.</p>
A similar problem occurs with "impure" build tools. For example, the standard Protobuf plugin for Gradle tries to download the protoc</code> compiler during the build. To make this work in a pure Nix environment, you have to disable this feature and instead provide protoc</code> through the Nix derivation.</p>
While these tools provide a solution, they are another hurdle to overcome. For a deep dive on patching binaries, Sander van der Burg's post on deploying prebuilt binaries with Nix</a> is an excellent resource.</p>
🔗</a>Handling Hardcoded Build Environments</h3>
Sometimes, you can't override impure behavior. Certain libraries, particularly in the cryptography space, might have build scripts that are hardcoded to look for dependencies in standard locations like /usr/lib</code>. In these cases, your only option is to fall back on buildFHSUserEnv</code></a> to create a sandboxed environment that simulates a traditional filesystem. It's a powerful tool, but it feels like a workaround and highlights the gap between the pure world of Nix and how many other tools work.</p>
🔗</a>The Ugly</h2>
🔗</a>The Nix Language Barrier</h3>
The Nix language itself is the hardest part. It’s a functional language that feels very different from most programming languages. Simple things can be hard to figure out, and you often have to look up how to do basic operations.</p>
LLMs have made this much easier. Before they were widely available, I spent countless hours searching for similar packages on GitHub to figure out how to solve a specific problem. Now, you can ask for a code snippet and get something that works. But needing an AI to help with basic packaging shows how hard the language is to learn.</p>
🔗</a>Conclusion</h2>
So, what's the verdict? The scales may seem evenly balanced between praise and frustration, yet I wouldn't switch away from NixOS. The learning curve is a mountain, and the daily friction can be grating. But the payoff—the absolute, ironclad guarantee of reproducibility—is a superpower.</p>
As someone who builds and tests complex distributed systems, I spend my days fighting entropy. NixOS provides a sane foundation where the environment is a solved problem. The fear of a broken update before an on-call shift is gone. The hours spent debugging "works on my machine" issues have vanished. Setting up a new machine is a 15-minute, one-command affair.</p>
NixOS demands a significant upfront investment for long-term peace of mind. It trades short-term convenience for long-term stability and control. It's not for everyone, but if you're a developer or systems engineer who sees your OS as a critical part of your toolkit—one that should be as reliable and version-controlled as your code—then the tough road of NixOS is absolutely worth it.</p>
🔗</a>A Gentler Start: Try Nix First</h3>
If this article makes you curious but wary of diving headfirst into a full OS migration, there’s good news: you don’t have to. You can get a taste of Nix’s power on your existing macOS or Linux setup.</p>
By installing just the Nix package manager, you can start creating reproducible development environments using nix-shell</code> or Nix Flakes. This lets you manage project-specific dependencies without conflicts and share a consistent setup with your team. It's a fantastic way to learn the Nix language and experience its benefits in a familiar environment before committing to NixOS.</p>
I’ve found it incredibly useful to have dependencies managed the same way between Linux and macOS. This website, for example, is built using the same Flake to pull Zola, and it works identically on my Linux laptop and my Mac.</p>

Feel free to reach out with any questions or to share your experiences with NixOS. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Thank You, DataFusion: Queries in Rust, Without the Pain
Pierre Zemb — Wed, 04 Jun 2025 00:00:00 +0000
🔗</a>That “YATTA!” Moment, Rebooted</h2>
We just merged at work our first successful data retrieval using DataFusion</a> — a real SQL query, over real data, flowing through a system we built. And I’ll be honest: I haven’t had a “YATTA!” moment like this in years. This wasn't just a feature shipped; it felt like unlocking a new superpower for our entire system, a complex vision finally materializing.</p>
Not a silent nod. Not “huh, that works.” A real</em>, physical, joyful reaction. The kind that makes you want to run a lap around the office (or, in my remote-first case, the living room).</p>
Because plugging a query engine into your software isn’t supposed to feel this smooth. It's usually a battle. But this one did. This one felt like an invitation.</p>
🔗</a>You Don’t Just Add a Query Engine</h2>
Adding a query engine to a codebase isn’t something you do lightly. It’s a foundational piece of infrastructure, the kind of decision that usually ends in regret, or at least a lot</em> of rewriting. Most engines assume they own the world: they want to dictate your storage, your execution model, your schema, your optimizer, often forcing you to contort your application around their idiosyncrasies. It's a path often paved with impedance mismatches, performance bottlenecks, and the haunting feeling that you’ve just bolted an opinionated, unyielding black box onto your carefully crafted system.</p>
But then there’s DataFusion. A SQL engine written in Rust, and — against all odds — one you can actually use</em>. Drop-in? Not quite. But close enough to be kind of magical, offering a set of powerful, composable tools rather than a rigid framework.</p>
🔗</a>I’ve Been Watching From Day One</h2>
I’ve been following DataFusion since it was a weekend project. I still remember the early blog posts, the prototypes, the potential. And more importantly, I read Andy Grove’s book How Query Engines Work</em></a>. That book unlocked it for me.</p>
It demystified concepts like logical plans, physical plans, and execution trees — enough to give me the confidence to experiment. I first played with Apache Calcite, then circled back to DataFusion. Eventually, I contributed a small example: a custom TableProvider</code>, added to DataFusion in this issue</a> to demonstrate how to integrate custom datasources.</p>
And then... it only took me three years</strong> to actually write the code that used</em> it. Why so long? Well, let's just say a gazillion other things, the never-ending sagas of on-call, and a brief-but-eventful detour into management</a> kept my dance card impressively full. But hey, it still felt amazing when it finally clicked.</p>
More recently, I was genuinely happy to see that Andrew Lamb</strong> co-authored an academic paper describing DataFusion’s architecture</a>. There’s something really validating about seeing a project you’ve followed for years get formalized in research — it’s a sign that the internals are solid and the ideas are worth sharing. And they are.</p>
That moment was big. Because here was a Rust-native query engine where I could plug in my own data</em>, and get real queries</em> back. No layers of JVM glue, no corroded abstractions. Just composable, hackable Rust.</p>
🔗</a>Modular, Composable, Respectful</h2>
What I love about DataFusion is that it doesn’t try to control your application. It’s a query engine that knows it’s a library — not a database.</p>
It lets you:</p>

Plug in your own data sources</li>
Register logical tables dynamically</li>
Push down filters, projections, even partitions</li>
Swap in or extend physical execution nodes</li>
Keep your own runtime, threading, and lifecycle</li>
</ul>
And all that without feeling like you’re stepping into “internal” code. It’s all open, cleanly layered, and welcoming.</p>
🔗</a>My Goal: Join Indexes Without Going Insane</h2>
From the beginning, my goal was never to just scan data — it was to query it properly</strong>, with indexes, joins, and all the things a real engine should do. I never had any intention of writing a join execution engine myself. That’s not the kind of wheel I want to reinvent.</p>
It's no secret that at work, we're building a system on top of FoundationDB that draws inspiration from Apple's FDB Record Layer</a> (you can learn more about its concepts in this talk</a>). We offer a similar programmatic API for constructing queries</a>, which naturally leads to similar requirements. For example, developers need to express sophisticated data retrieval logic, much like this FDB Record Layer example for querying orders:</p>
RecordQuery</span> query = </span>RecordQuery</span>.</span>newBuilder</span>()
</span>        .</span>setRecordType</span>("</span>Order</span>")
</span>        .</span>setFilter</span>(</span>Query</span>.</span>and</span>(
</span>                </span>Query</span>.</span>field</span>("</span>price</span>").</span>lessThan</span>(</span>50</span>),
</span>                </span>Query</span>.</span>field</span>("</span>flower</span>").</span>matches</span>(</span>Query</span>.</span>field</span>("</span>type</span>").</span>equalsValue</span>(</span>FlowerType</span>.</span>ROSE</span>.</span>name</span>()))))
</span>        .</span>build</span>();
</span></code></pre>
The challenge then becomes translating such programmatic queries into efficient, index-backed scans and, crucially, leveraging a robust engine for complex operations like joins—without rebuilding that engine from scratch.</p>
What I wanted was the ability to:</p>

Fetch rows efficiently through custom index-backed scans</li>
Join them using HashJoinExec</code> or MergeJoinExec</code></li>
Let the planner and execution engine figure out the hard parts</li>
</ul>
This vision is what spurred me to start working on datafusion-index-provider</code></a>, a library hosted in the datafusion-contrib</code> GitHub organization — part of the growing ecosystem around DataFusion. At the time of writing, I’ve built a PoC — you can find it on this branch</a> — and I’m integrating it into our internal stack before opening a proper PR upstream.</p>
The architecture makes it feel possible. The abstractions are ready. And I still don’t have to write a join engine. Victory.</p>
🔗</a>The Joy of Real Libraries</h2>
There’s a special joy in finding a library that slots in</em> — that doesn’t just solve a problem, but fits the shape of your system. DataFusion was that for me.</p>
It didn’t just let me query data; it gave me a better way to think about the data I already had, and how I wanted to work with it. Instead of manually stitching together filters and projections, I could describe my intent, and let the engine handle the rest.</p>
What’s even more exciting is that this isn’t happening in a vacuum.</p>
We’re seeing a quiet shift in how query engines are built and used. Projects like DuckDB</a> have shown just how powerful it is to have SQL as a library</strong>, not a service. No server to deploy. No socket to connect to. Just an API, embedded right in your code.</p>
DataFusion follows that same philosophy — Rust-native, embeddable, and unapologetically library-first.</p>
🔗</a>To the DataFusion Team: Thank You</h2>
To Andy Grove, to all the contributors, to everyone filing issues and refining abstractions: thank you. Your work is enabling a new generation of Rust systems to think like databases — without becoming one.</p>
I don’t know if you realize how rare that is. I just know it changed what I thought was possible in my software.</p>
And I’m having a lot more fun because of it.</p>

Feel free to reach out with any questions or to share your experiences with DataFusion. You can find me on Twitter</a>, Bluesky</a> or through my website</a>.</p>


Bypassing FoundationDB's Transaction Limits with Record Layer Continuations
Pierre Zemb — Tue, 03 Jun 2025 00:30:00 +0200
🔗</a>Introducing the FoundationDB Record Layer</h2>
Before we dive into the specifics of handling large operations with continuations (the main topic of this post), let's briefly introduce the FoundationDB Record Layer</strong></a>. It's a powerful open-source library built atop FoundationDB that brings a structured, record-oriented data model to FDB's highly scalable key-value store. Think of it as adding schema management, rich indexing capabilities, and a sophisticated query engine, making it easier to build complex applications.</p>
The Record Layer is versatile and has been adopted for demanding use-cases, most notably by Apple as the core of CloudKit, powering services for millions of users. It allows developers to define their data models using Protocol Buffers and then query them in a flexible manner.</p>
For instance, you can express queries like finding all 'Order' records for roses costing less than $50 with a declarative API (example in Java):</p>
RecordQuery</span> query = </span>RecordQuery</span>.</span>newBuilder</span>()
</span>        .</span>setRecordType</span>("</span>Order</span>")
</span>        .</span>setFilter</span>(</span>Query</span>.</span>and</span>(
</span>                </span>Query</span>.</span>field</span>("</span>price</span>").</span>lessThan</span>(</span>50</span>),
</span>                </span>Query</span>.</span>field</span>("</span>flower</span>").</span>matches</span>(</span>Query</span>.</span>field</span>("</span>type</span>").</span>equalsValue</span>(</span>FlowerType</span>.</span>ROSE</span>.</span>name</span>()))))
</span>        .</span>build</span>();
</span></code></pre>
To get started and explore its capabilities further, the official Getting Started Guide</a> is an excellent resource. You can also watch these talks for a deeper understanding:</p>

Using FoundationDB and the FDB Record Layer to Build CloudKit - Scott Gray, Apple</a></li>
FoundationDB Record Layer: Open Source Structured Storage on FoundationDB - Nicholas Schiefer, Apple</a></li>
</ul>

For a detailed academic perspective on its design and how CloudKit uses it, refer to the SIGMOD'19 paper: FoundationDB Record Layer: A Multi-Tenant Structured Datastore</a>.</p>
</blockquote>
🔗</a>The Challenge: FDB's Transaction Constraints</h2>
FoundationDB (FDB) imposes strict constraints on its transactions: they must complete within 5 seconds and are limited to 10MB of manipulated data, either writes or reads. These constraints are fundamental to FDB's design, ensuring high performance and serializable isolation. However, they pose a significant challenge for operations that inherently require processing large datasets or executing complex queries that cannot complete within these tight boundaries, such as full table scans, large analytical queries, or bulk data exports.</p>
The FoundationDB Record Layer</strong> addresses this challenge through a mechanism known as continuations</strong>. Continuations allow a single logical operation to be broken down into a sequence of smaller, independent FDB transactions. Each transaction processes a segment of the total workload and, if more work remains, yields a continuation token</strong>. This opaque token encapsulates the state required to resume the operation precisely where the previous transaction left off.</p>
This article delves into the technical details of Record Layer continuations, exploring how they function and how to leverage them effectively to build robust, scalable applications on FDB.</p>
🔗</a>Bridging Transactions: The Role of Continuations</h2>
Consider a query to retrieve all records matching a specific filter from a large dataset. Executing this as a single FDB transaction would likely violate the 5-second or 10MB limit. The Record Layer employs continuations to serialize this operation across multiple transactions:</p>

Initial Request:</strong> The application initiates a query against the Record Layer.</li>
Segmented Execution:</strong> The Record Layer's query planner executes the query, but with built-in scan limiters. It processes records until a predefined limit (e.g., row count, time duration, or byte size) is approached, or it nears FDB's intrinsic transaction limits.</li>
State Serialization:</strong> Before the current FDB transaction commits, if the logical operation is incomplete, the Record Layer serializes the execution state of the query plan into a continuation token.</li>
Partial Result & Token:</strong> The application receives the processed segment of data and the continuation token. The FDB transaction for this segment commits successfully.</li>
Resumption:</strong> To fetch the next segment, the application submits a new request, providing the previously received continuation token.</li>
State Deserialization & Continued Execution:</strong> The Record Layer deserializes the token, restores the query plan's state, and resumes execution from the exact point it paused. This typically involves adjusting scan boundaries (e.g., starting a key-range scan from the key after the last one processed).</li>
</ol>
This cycle repeats until the entire logical operation is complete. The continuation token acts as the critical link, enabling a series of short, FDB-compliant transactions to collectively achieve the effect of a single, long-running operation without violating FDB's core constraints.</p>
🔗</a>Dissecting the Continuation Token</h2>
While the continuation token is opaque</strong> to the application (it's a byte[]</code> that should not be introspected or modified), it internally contains structured information vital for resuming query execution. The exact format is an implementation detail of the Record Layer and can evolve, but conceptually, it must capture:</p>

Scan Boundaries:</strong> The key (or keys, for multi-dimensional indexes) where the next scan segment should begin. This ensures no data is missed or re-processed unnecessarily.</li>
Query Plan State:</strong> For complex query plans involving joins, filters, aggregations, or in-memory sorting, the token may need to store intermediate state specific to those operators. For instance, a UnionPlan</code> or IntersectionPlan</code> might need to remember which child plan was active and its respective continuation.</li>
Scan Limiter State:</strong> Information about accumulated counts or sizes if the scan was paused due to application-defined limits rather than FDB limits.</li>
Version Information:</strong> To ensure compatibility if the token format changes across Record Layer versions.</li>
</ul>
The opacity of the token is a deliberate design choice. It decouples the application from the internal mechanics of the Record Layer, allowing the latter to evolve its continuation strategies (e.g., for efficiency or new features) without breaking client applications. The application's responsibility is solely to store and return this token verbatim.</p>
🔗</a>Resuming Query Execution via Continuations</h2>
When a continuation token is provided to a RecordCursor</code> (the Record Layer's abstraction for iterating over query results), the underlying RecordQueryPlan</code> uses it to reconstruct its state.</p>

Plan Identification:</strong> The token typically identifies the specific query plan or sub-plan it pertains to.</li>
State Restoration:</strong> Each operator in the query plan (e.g., IndexScanPlan</code>, FilterPlan</code>, SortPlan</code>) that can be stateful across transaction boundaries implements logic to initialize itself from the continuation. For an IndexScanPlan</code>, this primarily means setting the ScanComparisons</code> for the next range read. For a UnionPlan</code>, it might mean restoring the continuation for one of its child plans and indicating which child to resume.</li>
Execution Resumption:</strong> Once the plan's state is restored, the RecordCursor</code> can proceed to fetch the next batch of records. The execution effectively "jumps" to the point encoded in the continuation.</li>
</ol>
This mechanism allows the Record Layer to transparently manage the complexities of distributed, stateful iteration over potentially vast datasets, all while adhering to FDB's transactional model.</p>
🔗</a>Implications of Non-Atomicity</h2>
It's important to understand a key implication of this multi-transaction approach: while each individual FDB transaction executed as part of a continued operation is atomic and isolated (typically providing serializable isolation), the overall logical operation spanning multiple continuations is not atomic</strong> in the same way. Mutations to the data by other concurrent transactions can occur between</em> the FDB transactions of a continued scan. As a result, a long-running operation that uses continuations doesn't see the entire dataset at a single, frozen moment in time. Instead, it might see some data that was present or changed after</em> the operation began but before</em> it completed. This is a natural consequence of breaking the work into smaller pieces to fit within FDB's transaction limits. Applications should be aware of this behavior, particularly if they need all the data to reflect its state from one specific instant.</p>
🔗</a>Conclusion</h2>
The Record Layer's continuation feature is a powerful tool for handling large datasets and complex queries in FoundationDB, but it's important to understand the implications of non-atomicity. By breaking operations into smaller, FDB-compliant transactions, the Record Layer provides a flexible and scalable solution while maintaining the core principles of FDB's transactional model.</p>

Feel free to reach out with any questions or to share your thoughts. You can find me on Bluesky</a>, Twitter</a> or through my website</a>.</p>


Unlocking Tokio's Hidden Gems: Determinism, Paused Time, and Local Execution
Pierre Zemb — Sun, 18 May 2025 18:13:02 +0200
Tokio is the powerhouse of asynchronous Rust, celebrated for its blazing speed and robust concurrency primitives. Many of us interact with its core components daily—spawn</code>, select!</code>, async fn</code>, and the rich ecosystem of I/O utilities. But beyond these well-trodden paths lie some incredibly potent, albeit less-publicized, features that can dramatically elevate your testing strategies, offer more nuanced task management, and grant you surgical control over your runtime's execution.</p>
Today, let's pull back the curtain on a few of these invaluable tools: current-thread runtimes for embracing single-threaded flexibility with !Send</code> types, seeded runtimes for taming non-determinism, and the paused clock for mastering time in your tests.</p>
🔗</a>Effortless !Send</code> Futures with Current-Thread Runtimes</h2>
While Tokio's multi-threaded scheduler is a marvel for CPU-bound and parallel I/O tasks, there are scenarios where a single-threaded execution model is simpler or even necessary. This is particularly true when dealing with types that are not Send</code> (i.e., cannot be safely transferred across threads), such as Rc<T></code> or RefCell<T></code>, or when you want to avoid the overhead and complexity of synchronization primitives like Arc<Mutex<T>></code> for state shared only within a single thread of execution.</p>
Tokio's Builder::new_current_thread()</code></a> followed by build_local()</code></a> (part of the same Builder</code></a> API) provides a streamlined way to create a runtime that executes tasks on the thread that created it. This setup inherently supports spawning !Send</code> futures using tokio::task::spawn_local</code></a> without needing to manually manage a LocalSet</code> for basic cases. This approach aligns well with ongoing discussions in the Tokio community aimed at simplifying !Send</code> task management.</p>
This build_local()</code> method not only simplifies handling !Send</code> types today but also reflects the direction Tokio is heading. The Tokio team is exploring ways to make this even more integrated and ergonomic through a proposed LocalRuntime</code></strong> type (#6739</a>). The vision for LocalRuntime</code> is a runtime that is inherently !Send</code> (making !Send</code> task management seamless within its context), where tokio::spawn</code> and tokio::task::spawn_local</code> would effectively behave identically.</p>
This proposed enhancement is linked to a discussion about potentially deprecating the existing tokio::task::LocalSet</code></a></strong> (#6741</a>). While LocalSet</code> currently offers fine-grained control for running !Send</code> tasks (e.g., within specific parts of larger, multi-threaded applications), it comes with complexities, performance overhead, and integration challenges that LocalRuntime</code> aims to resolve.</p>
So, what's the takeaway for you?</strong></p>

For most scenarios requiring !Send</code> tasks on a single thread</strong> (like entire applications, test suites, or dedicated utility threads): Using Builder::new_current_thread().build_local()</code> is the recommended, simpler, and more future-proof path. It embodies the principles of the proposed LocalRuntime</code>.</li>
If you need to embed !Send</code> task execution within a specific scope of a larger, multi-threaded application</strong>: LocalSet</code> is the current tool. However, be mindful of its potential deprecation and associated complexities. For new projects, evaluate if a dedicated thread using a build_local()</code> runtime (or a future LocalRuntime</code>) might offer a cleaner solution.</li>
</ul>
Essentially, Tokio is moving towards making single-threaded !Send</code> execution more straightforward and deeply integrated. The build_local()</code></a> method is a current gem that aligns you with this forward-looking approach.</p>
Here's how you typically set one up (the build_local()</code> way):</p>
use </span>tokio::runtime::Builder;
</span>
</span>fn </span>main</span>() {
</span>    </span>let mut</span> rt = Builder::new_current_thread()
</span>        .</span>enable_all</span>() </span>// Enable I/O, time, etc.
</span>        .</span>build_local</span>(&</span>mut </span>Default::default()) </span>// Builds a runtime on the current thread
</span>        .</span>unwrap</span>();
</span>
</span>    </span>// The runtime itself is the 'LocalSet' in this context
</span>    rt.</span>block_on</span>(async {
</span>        </span>// Spawn !Send futures here using tokio::task::spawn_local(...)
</span>        </span>// For example:
</span>        </span>let</span> rc_value = std::rc::Rc::new(</span>5</span>);
</span>        tokio::task::spawn_local(async </span>move </span>{
</span>            println!("</span>RC value: </span>{}</span>", *rc_value);
</span>        }).await.</span>unwrap</span>();
</span>
</span>        println!("</span>Running !Send futures on a current-thread runtime!</span>");
</span>    });
</span>}
</span></code></pre>
This approach simplifies designs where tasks don't need to cross thread boundaries, allowing for more straightforward state management.</p>
🔗</a>Taming Non-Determinism: Seeded Runtimes</h2>
One of the challenges in testing concurrent systems is non-determinism. When multiple futures are ready to make progress simultaneously, such as in a tokio::select!</code></a> macro, the order in which they are polled can vary between runs. This can make reproducing and debugging race conditions or specific interleavings tricky.</p>
Tokio offers a solution: seeded runtimes</strong>. By providing a specific RngSeed</code></a> when building the runtime, you can make certain scheduler behaviors deterministic. This is particularly useful for select!</code> statements involving multiple futures that become ready around the same time.</p>
Consider this example, which demonstrates how a seed can influence which future 'wins' a select!</code> race:</p>
use </span>tokio::runtime::{Builder, RngSeed};
</span>use </span>tokio::time::{sleep, Duration};
</span>
</span>// Example function to show deterministic select!
</span>fn </span>demo_deterministic_select</span>() {
</span>    </span>// Try changing this seed to see the select! behavior change (but consistently per seed).
</span>    </span>let</span> seed = RngSeed::from_bytes(</span>b</span>"</span>my_fixed_seed_001</span>");
</span>    </span>// e.g., let seed = RngSeed::from_bytes(b"another_seed_002");
</span>
</span>    </span>let mut</span> rt = Builder::new_current_thread()
</span>        .</span>enable_time</span>()
</span>        </span>// Pausing the clock is crucial here to ensure both tasks become ready 
</span>        </span>// at the *exact same logical time* after we call `tokio::time::advance`.
</span>        </span>// This makes the seed's role in tie-breaking very clear.
</span>        .</span>start_paused</span>(</span>true</span>)
</span>        .</span>rng_seed</span>(seed)     </span>// Apply the seed for deterministic polling order
</span>        .</span>build_local</span>(&</span>mut </span>Default::default())
</span>        .</span>unwrap</span>();
</span>
</span>    </span>// Now, let's run some tasks and see select! in action.
</span>    rt.</span>block_on</span>(async {
</span>        </span>let</span> task_a = async {
</span>            </span>sleep</span>(Duration::from_millis(</span>50</span>)).await;
</span>            println!("</span>Task A finished.</span>");
</span>            "</span>Result from A</span>"
</span>        };
</span>
</span>        </span>let</span> task_b = async {
</span>            </span>sleep</span>(Duration::from_millis(</span>50</span>)).await;
</span>            println!("</span>Task B finished.</span>");
</span>            "</span>Result from B</span>"
</span>        };
</span>
</span>        </span>// Advance time so both sleeps complete and both tasks become ready.
</span>        tokio::time::advance(Duration::from_millis(</span>50</span>)).await;
</span>
</span>        </span>// With the same seed, the select! macro will consistently pick the same
</span>        </span>// branch if both are ready. Change the seed to see if the other branch gets picked.
</span>        tokio::select! {
</span>            res_a = task_a => {
</span>                println!("</span>Select chose Task A, result: '</span>{}</span>'</span>", res_a);
</span>            }
</span>            res_b = task_b => {
</span>                println!("</span>Select chose Task B, result: '</span>{}</span>'</span>", res_b);
</span>            }
</span>        }
</span>    });
</span>}
</span>
</span>fn </span>main</span>() {
</span>    </span>demo_deterministic_select</span>();
</span>}
</span></code></pre>
🔗</a>Mastering Time: Paused Clock and Auto-Advancement</h2>
Testing time-dependent behavior (timeouts, retries, scheduled tasks) can be slow and flaky. Waiting for real seconds or minutes to pass during tests is inefficient. Tokio's time facilities can be paused</strong> and manually advanced</strong>, giving you precise control over the flow of time within your tests.</p>
When you initialize a runtime with start_paused(true)</code></a>, the runtime's clock will not advance automatically based on wall-clock time. Instead, you use tokio::time::advance(Duration)</code> to move time forward explicitly.</p>
What's particularly neat is Tokio's auto-advance</strong> feature when the runtime is paused and idle. This works because Tokio's runtime separates the executor</strong> (which polls your async code until it's blocked) from the reactor</strong> (which wakes tasks based on I/O or timer events). If all tasks are sleeping, the executor is idle. The reactor can then identify the next scheduled timer, allowing Tokio to automatically advance its clock to that point. This prevents tests from hanging indefinitely while still allowing for controlled time progression.</p>
Here's your example illustrating this:</p>
use </span>tokio::time::{Duration, Instant, sleep};
</span>
</span>async </span>fn </span>auto_advance_kicks_in_when_idle_example</span>() {
</span>    </span>let</span> start = Instant::now();
</span>
</span>    </span>// Sleep for 5 seconds. Since the runtime is paused, this would normally hang.
</span>    </span>// However, if no other tasks are active, Tokio auto-advances time.
</span>    </span>sleep</span>(Duration::from_secs(</span>5</span>)).await;
</span>
</span>    </span>let</span> elapsed = start.</span>elapsed</span>();
</span>
</span>    </span>// This will be exactly 5 seconds (simulated time)
</span>    assert_eq!(elapsed, Duration::from_secs(</span>5</span>));
</span>
</span>    println!("</span>Elapsed (simulated): </span>{:?}</span>", elapsed);
</span>}
</span></code></pre>
In this scenario, sleep(Duration::from_secs(5)).await</code> doesn't cause your test to wait for 5 real seconds. Because the clock is paused and this sleep</code> is the only pending timed event, Tokio advances its internal clock by 5 seconds, allowing the sleep to complete almost instantaneously in real time. This makes testing timeouts, scheduled events, and other time-sensitive logic fast and reliable.</p>
🔗</a>Conclusion</h2>
Tokio offers more than just speed; it's a powerful toolkit. Features like current-thread runtimes for !Send</code> tasks, seeded runtimes for deterministic tests, and a controllable clock for time-based logic help build robust and debuggable async Rust applications. These 'hidden gems' allow you to confidently handle complex concurrency and testing. So, explore Tokio's depth—the right tool for your challenge might be closer than you think.</p>

Feel free to reach out with any questions or to share your thoughts. You can find me on Bluesky</a>, Twitter</a> or through my website</a>.</p>


What if we embraced simulation-driven development?
Pierre Zemb — Fri, 18 Apr 2025 11:12:12 +0200
This article has been translated from my original French presentation at the upcoming Devoxx France 2025, titled "What if we embraced simulation-driven development?</a>".</p>
🔗</a>The Tale of a Bug</h2>
As a software engineer, my responsibilities include debugging distributed systems during on-call shifts. My tendency to attract peculiar issues during these shifts earned me the nickname "Black Cat". Let me share a particularly memorable incident:</p>
One of the most memorable incidents happened when a network partition</strong> completely disrupted a 70+ node Apache Hadoop cluster. The system was in disarray, with nodes confused about block replication</strong> and management</strong>. After the network issue was resolved, we decided to restart the cluster</strong>...</p>
But it wouldn't come back online.</p>
The reason? The system was encountering a NullPointerException</code> during startup due to its faulty state. The cluster was too slow to restart properly because of how severely degraded it had become after the network partition. This bug had actually been fixed in newer versions of HDFS</strong>, but we were running an older release.</p>
The solution required patching the Hadoop codebase</strong> by backporting the fix</strong>, recompiling</strong>, and distributing the new jar</strong> across all nodes—not exactly what you want to be doing during an active incident. Rolling out patches to a distributed system while it's already "on fire" is rarely recommended, but we had no choice.</p>
This is exactly the type of code that feels disconnected from production requirements—the bug appeared at the worst possible moment, during recovery, when the system was most vulnerable.</p>
🔗</a>The Development-Production Gap</h2>
This incident highlights a fundamental truth in software engineering: production environments are vastly different from development environments</strong>. The gap between them is comparable to the difference between passing a written driving test and actually driving on a busy highway during rush hour.</p>

            flowchart LR
    S["Your System"] 
    U["Your Users"]
    W["The World"]
    
    U --> S
    W --> S
    </pre>
</div>
In development, everything is controlled</strong>, clean</strong>, and predictable</strong>. In production:</p>

Users do unexpected things</strong></li>
Systems operate under pressure</strong></li>
Components fail in complex ways</strong></li>
Edge cases</strong> occur regularly</li>
</ul>
Being on-call forces you to confront this reality. The pager is an unforgiving teacher, but is there a better way to instill a production mindset without throwing engineers into the deep end of incident response?</p>
🔗</a>The Testing Problem</h2>
Let's consider a standard e-commerce API with multiple dimensions of variability:</p>

User Types: Guest, Logged-in, Premium, Business (4)</li>
Payment Methods: Credit Card, PayPal, Apple Pay, Gift Card, Bank Transfer (5)</li>
Delivery Options: Standard, Express, In-Store Pickup, Same-Day (4)</li>
Promotions: Yes, No, Expired (3)</li>
Inventory Status: In Stock, Low Stock, Out of Stock, Preorder (4)</li>
Currency: USD, EUR, GBP, JPY (4)</li>
</ul>
Testing all possible combinations requires 4×5×4×3×4×4 = 3,840 unique test cases—and that's just for the happy path! Add error conditions, network failures, and other edge cases, and this number explodes exponentially.</p>
This is why comprehensive end-to-end testing is so difficult. Every new feature multiplies the complexity, and bugs often hide in rare combinations of conditions that we never thought to test.</p>
🔗</a>The World Is Harsh</h2>
Meanwhile, the real world is even more chaotic than our test cases. Research papers like "An Analysis of Network-Partitioning Failures in Cloud Systems</a>" (OSDI '18) and "Metastable Failures in Distributed Systems</a>" (HotOS '21) document just how complex failure modes can be in production.</p>
In a presentation by John Wilkes (Google) at QCon London 2015</a>, a 2,000-machine service will experience more than 10 machine crashes per day—and this is considered normal, not exceptional. When you operate at scale, failures become a constant background noise rather than exceptional events.</p>
And yes, your microservices architecture</strong> is absolutely a distributed system susceptible to these issues.</p>
🔗</a>SRE vs. SWE Perspectives</h2>
There's often a gap between the Software Engineer (SWE) perspective and the Site Reliability Engineer (SRE) perspective:</p>
SWEs tend to focus on:</p>

Development environments (which are completely different from production)</li>
Feature implementations</li>
Code that passes tests (but may not account for real-world complexity)</li>
</ul>
SREs worry about:</p>

System interactions in production under pressure</li>
Complex, unpredictable failure modes</li>
Recovery mechanisms when things are already broken</li>
Being paged at 3 AM to fix critical issues alone</li>
</ul>
The question then becomes: How can we help developers gain a better understanding of production realities without subjecting them to the trial-by-fire of on-call rotations?</strong> How might we bridge this gap between development and operations, creating environments where engineers can experience production-like conditions safely, learn from failures, and build more resilient systems from the beginning?</p>
We need to test not just our expected use cases, but the "worse" versions of both our users and the world</strong>. How do we accomplish this comprehensively?</p>

            flowchart LR
    S["Your System"] 
    U["Your worst Users"]
    W["The worst World"]
    
    U --> S
    W --> S
    </pre>
</div>🔗</a>Deterministic Simulation Testing</h2>
The solution lies in a strategy that's both robust and practical: Deterministic Simulation Testing (DST)</strong>.</p>
For effective testing of complex distributed systems, we need an approach that satisfies three key requirements:</p>


Fast and debuggable testing</strong> → We achieve this with a single-threaded approach that uses a deterministic event loop, making issues perfectly reproducible</p>
</li>

Testing the entire system at once</strong> → By packaging everything into a single binary with simulated network interactions, we can test complex distributed behaviors without actual network infrastructure</p>
</li>

Robust against unknown issues</strong> → Through randomized testing with controlled entropy injection, we discover edge cases that we wouldn't think to test explicitly</p>
</li>
</ol>
These three elements work together to create a powerful testing methodology that's both practical to implement and effective at finding real-world issues.</p>
Let's see how we can simulate both our users and the world?</p>
🔗</a>How to simulate?</h2>
🔗</a>Simulating Users: Randomized Input and Property-Based Testing</h3>
Instead of writing thousands of individual test cases, we can use property-based testing</strong> to generate randomized inputs and verify system properties. This approach is not new and is well-known for unit tests but is relatively new for integration tests:</p>
enum </span>UserType </span>{ </span>GUEST</span>, LOGGED_IN, PREMIUM, BUSINESS }
</span>enum </span>PaymentMethod </span>{ </span>CARD</span>, PAYPAL, APPLE_PAY, GIFT_CARD, BANK_TRANSFER }
</span>// ...
</span>
</span>Random</span> rand = </span>new </span>Random</span>(); </span>// random seed
</span>
</span>UserType</span> user = </span>pickRandom</span>(rand, </span>UserType</span>.</span>values</span>());
</span>PaymentMethod</span> paymentMethod = </span>pickRandom</span>(rand, </span>PaymentMethod</span>.</span>values</span>());
</span></code></pre>
Rather than hardcoding test cases like:</p>
assertFalse</span>(</span>new </span>User</span>(</span>GUEST</span>).</span>canUse</span>(</span>SAVED_CARD</span>));
</span></code></pre>
We can write property-based assertions:</p>
assertEquals</span>(user.</span>isAuthenticated</span>(), user.</span>canUse</span>(</span>SAVED_CARD</span>));
</span></code></pre>
This approach is implemented in libraries like:</p>

Python: Hypothesis</strong></li>
Java: jqwik</strong></li>
Rust: proptest</strong></li>
</ul>
🔗</a>Simulating the World: Injecting Chaos</h3>
We also need to simulate the chaotic nature of production environments by injecting failures into:</p>

Time (delays, timeouts, retries, race conditions)</li>
Network (latency, failure, disconnection)</li>
Infrastructure (disk full, service crash, replica lag)</li>
External dependencies (slow APIs, rate limiting)</li>
Load (varying numbers of concurrent users)</li>
</ul>
It's important to note that implementing full deterministic simulation requires control over every aspect of your system, from task scheduling to I/O operations. This is significantly easier if your system is built with simulation in mind from day one. Some languages offer advantages in this area—for example, Rust's ecosystem makes it relatively straightforward to implement custom virtual threading executors compared to modifying the JVM.</p>
For existing codebases where a full rewrite isn't practical, you can still benefit from simulation testing by adding layers of indirection. Even simple mocks like the HTTP client example below can help you discover how your system behaves under various failure conditions:</p>
class </span>HttpClientMock </span>{
</span>    </span>private final </span>Random </span>random </span>= </span>new </span>Random</span>(); </span>// random seed
</span>
</span>    </span>String </span>get</span>(</span>String </span>url</span>) {
</span>        </span>// Simulate random chance of returning an error
</span>        </span>if </span>(random.</span>nextDouble</span>() </span>< </span>0.2</span>) {
</span>            </span>return </span>"</span>HTTP 500 Internal Server Error</span>"</span>;
</span>        }
</span>
</span>        </span>int</span> delay </span>=</span> random.</span>nextInt</span>(</span>500</span>); </span>// Simulate 0–499ms latency
</span>        </span>Thread</span>.</span>sleep</span>(delay);
</span>        </span>return </span>"</span>HTTP 200 OK</span>"</span>;
</span>    }
</span>}
</span></code></pre>
🔗</a>Who Uses DST?</h2>
Not many companies are using DST, but we are starting to have a nice list:</p>

Clever Cloud</li>
TigerBeetle</li>
Resonate</li>
RisingWave</li>
Sync @ Dropbox</li>
sled.rs</li>
Kafka’s KRaft</li>
Astradot</li>
Polar Signals</li>
AWS</li>
Antithesis</li>
</ul>
🔗</a>DST at Clever Cloud</h3>
At Clever Cloud, we're implementing a multi-tenant, multi-model distributed database heavily relying on FoundationDB. While we haven't developed our own deterministic simulation testing framework yet, we leverage FoundationDB</a>'s built-in simulation by injecting custom workloads.</strong> This approach is core to developing our first serverless product, Materia KV</a>. The simulations FoundationDB provides include:</p>

Random network partitions</li>
Machine reboots</li>
Concurrent chaos events, like shuffling the actual data disk between 2 nodes</li>
</ul>
Our simulation-driven development workflow runs simulations:</p>

In CI loops</li>
Continuously in the cloud</li>
With 30 minutes of simulation equating to roughly 24 hours of chaos testing</li>
</ul>
When we find a faulty seed, we can replay it locally, providing a superpower for debugging complex distributed systems issues.</p>
🔗</a>Benefits for Developer Education</h3>
Deterministic simulation testing doesn't just help find bugs—it helps developers grow. By working with simulated but realistic failure scenarios, developers build intuition for how distributed systems behave under stress without having to experience painful on-call incidents.</p>
Moreover, deterministic simulation testing has instilled a deep trust in our software</strong>, as it is tested under conditions even more challenging than those encountered in production. This confidence has been crucial for us.</p>
🔗</a>Conclusion</h2>
The gap between development and production is real and significant. Traditional testing approaches can't scale to cover all the possible combinations of user behavior and world events that our systems will encounter.</p>
Deterministic simulation testing offers a powerful alternative that allows us to test complex distributed systems more thoroughly, find bugs before they impact users, and train developers to build more resilient systems.</p>
By embracing simulation-driven development, we can build software that better handles the chaotic reality of production environments—and maybe reduce those 3 AM pages that give engineers like me unfortunate nicknames.</p>

Want to learn more? Check out my curated list of resources on deterministic simulation testing</a>, which includes articles, talks, and implementation examples.</p>
Feel free to reach out with any questions or to share your experiences with simulation testing. You can find me on Twitter</a> or through my website</a>.</p>


So, You Want to Learn More About Deterministic Simulation Testing?
Pierre Zemb — Fri, 11 Apr 2025 00:00:00 +0200
I recently attended BugBash 2025</a>, a software reliability conference organized by Antithesis</a> in Washington, D.C. during April 3-4, 2025. The conference brought together industry experts like Kyle Kingsbury, Ankush Desai, and Mitchell Hashimoto to discuss various aspects of building reliable software, with deterministic simulation testing being a significant focus throughout many of the sessions and discussions.</p>
One of the highlights for me was having the chance to talk with the Antithesis team and meet some of the original creators of FoundationDB.</p>
🔗</a>What is Deterministic Simulation Testing?</h2>

Note:</strong> For a deeper dive into this concept and its practical applications, check out my article on What if we embraced simulation-driven development?</a>.</p>
</blockquote>
The best description of DST I've found is described in FoundationDB's testing page</a>:</p>

The major goal of Simulation is to make sure that we find and diagnose issues in simulation rather than the real world. Simulation runs tens of thousands of simulations every night, each one simulating large numbers of component failures. Based on the volume of tests that we run and the increased intensity of the failures in our scenarios, we estimate that we have run the equivalent of roughly one trillion CPU-hours of simulation on FoundationDB.</p>
</blockquote>

Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-threaded process. Determinism is crucial in that it allows perfect repeatability of a simulated run, facilitating controlled experiments to home in on issues. The simulation steps through time, synchronized across the system, representing a larger amount of real time in a smaller amount of simulated time. In practice, our simulations usually have about a 10-1 factor of real-to-simulated time, which is advantageous for the efficiency of testing.</p>
</blockquote>

We use Simulation to simulate failures modes at the network, machine, and datacenter levels, including connection failures, degradation of machine performance, machine shutdowns or reboots, machines coming back from the dead, etc. We stress-test all of these failure modes, failing machines at very short intervals, inducing unusually severe loads, and delaying communications channels.</p>
</blockquote>

Simulation's success has surpassed our expectation and has been vital to our engineering team. It seems unlikely that we would have been able to build FoundationDB without this technology.</p>
</blockquote>
After years of operating many Apache-oriented distributed systems, I can confidently say that FoundationDB stands apart in its remarkable robustness—I've rarely been paged for it, which speaks volumes about its stability in production. At Clever Cloud</a>, we've even leveraged FoundationDB's simulation framework during our application development by embedding Rust code inside FDB's simulation environment</a>, allowing us to inherit the same reliability guarantees for our custom applications.</p>
🔗</a>TL;DR</h2>
If you only have limited time, here are the four must-watch videos that will give you the best introduction to deterministic simulation testing:</p>

Will Wilson: Testing Distributed Systems with Deterministic Simulation</a></li>
Will Wilson: Autonomous Testing and the Future of Software Development</a></li>
Will Wilson: Testing a Single-Node, Single Threaded, Distributed System Written in 1985</a></li>
Will Wilson: Let's all write good software</a></li>
</ul>
(Yes, it seems Will Wilson has a monopoly on great introductory talks on the topic. Having had the chance to meet him, I can personally vouch that he is not a deterministic algorithm for generating insightful presentations, though the sheer quality of his talks might make you wonder.)</p>
A curated feed of recent articles and blog posts about DST can be found at Planet DST</a>.</p>
🔗</a>Essential Reading</h2>
🔗</a>Foundations & Concepts</h3>

CockroachLabs: Demonic Nondeterminism</a></li>
Alex Miller: BUGGIFY</a></li>
TigerBeetle: Building Reliable Systems</a></li>
Dominik Tornow: Deterministic Simulation Testing</a></li>
Poorly Defined Behaviour: Deterministic Simulation Testing</a></li>
Phil Eaton: What's the big deal about Deterministic Simulation Testing?</a></li>
AWS: Systems Correctness Practices</a></li>
</ul>
🔗</a>Language-Specific Implementations</h3>

Turmoil: Network Simulation Framework for Rust</a></li>
MadSim: Deterministic Simulation Testing Library for Rust</a></li>
Sled: Simulation Testing</a></li>
S2: Deterministic simulation testing for async Rust</a></li>
Polar Signals: Mostly-DST in Go</a></li>
</ul>
🔗</a>Real-World Case Studies</h3>

TigerBeetle: A Friendly Abstraction Over io_uring and kqueue</a></li>
Dropbox: Testing Our New Sync Engine</a></li>
TigerBeetle: We Put a Distributed Database in the Browser</a></li>
Antithesis: Case Studies</a></li>
RisingWave: A New Era of Distributed System Testing</a></li>
RisingWave: The RisingWave Story</a></li>
WarpStream: DST for Our Entire SaaS</a></li>
Antithesis: How Antithesis finds bugs (with help from the Super Mario Bros.)</a></li>
</ul>
🔗</a>Talks</h2>

Ben Collins: FoundationDB Testing: Past & Present</a></li>
Marc Brooker: AWS re:Invent 2024 - Try again: The tools and techniques behind resilient systems (ARC403)</a></li>
TigerBeetle: Episode 064: Two In One, New Request Protocol and VOPR Tutorial</a></li>
FOSDEM 2025: Squashing the Heisenbug with Deterministic Simulation Testing</a></li>
BugBash 2025: Lawrie Green - How to succeed in software testing without really trying</a></li>
</ul>

Have I missed any important resources on Deterministic Simulation Testing? This field is rapidly evolving, and I'm always looking to expand this collection. If you know of any articles, talks, or tools related to DST that should be included here, please reach out! I'd love to hear about your experiences with deterministic testing as well.</p>
Please, feel free to react to this article, you can reach me on Twitter</a>, or have a look on my website</a>.</p>


Key design tip: reverse number scanning in ordered key-value stores
Pierre Zemb — Thu, 27 Mar 2025 05:24:27 +0100
Ordered key-value stores like HBase, FoundationDB or RocksDB store keys in lexicographical order. When getting the latest version or most recent events, this ordering often requires scanning through all values in reverse order. While this works, it can become a performance bottleneck, especially in distributed systems. Let's explore a simple yet powerful optimization technique that I've been using recently 🚀</p>
🔗</a>Key design in Key-value stores</h2>
Let's look at this using a tuple structure of (key, number)</code>. This could represent a document version, a timestamp, or any numeric identifier:</p>
("my-key-1", 1)
</span>("my-key-1", 2)
</span>("my-key-2", 1)
</span></code></pre>
In ordered key-value stores, keys are stored in lexicographical order</code>. This works well when you want to scan from lowest to highest values, but becomes inefficient when you need the opposite order. For example, to find the highest number for a key, you need to scan through all values:</p>
("my-key-1", 1)
</span>("my-key-1", 2)
</span>("my-key-1", 3)
</span>...
</span>("my-key-1", 99)
</span></code></pre>
You could scan in reverse mode, but you would lose the order of your first prefix(the "my-key-1").</p>
🔗</a>Reverse Number Scanning</h2>
By reversing the numbers using a simple subtraction from the maximum possible value (e.g., Long.MAX_VALUE</code> in Java), we can optimize the scanning process:</p>
long</span> reversedNumber = </span>Long</span>.</span>MAX_VALUE </span>- number;
</span></code></pre>
This transforms our data into:</p>
("my-key-1", 9223372036854775804) // number 3
</span>("my-key-1", 9223372036854775805) // number 2
</span>("my-key-1", 9223372036854775806) // number 1
</span></code></pre>
Now, the highest number (which appears first in the reversed order) can be found efficiently, allowing us to stop after finding the first match.</p>
This technique is particularly useful in systems dealing with time-series data, versioned documents, or any scenario requiring efficient retrieval of the most recent or highest-valued items.</p>
number 1: 9223372036854775806
</span>number 2: 9223372036854775805
</span>number 3: 9223372036854775804
</span>
</span>// Reversing back is straightforward
</span>Long.MAX_VALUE - 9223372036854775806 = 1
</span></code></pre>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> or Bluesky</a> if needed.</p>


Debugging FoundationDB's Data Distributor
Pierre Zemb — Fri, 07 Mar 2025 00:00:00 +0100
FoundationDB is a powerful, distributed database designed to handle massive workloads with high consistency guarantees. At its core, the Data Distributor</strong> plays a critical role in determining how shards are distributed across the cluster to maintain load balance and resilience.</p>
In this post, we dive into the Data Distributor's</strong> internals, along with practical lessons we learned during a outage.</p>
🔗</a>What is the Data Distributor?</h2>
The Data Distributor (DD)</strong> is a subsystem</a> responsible for efficiently placing and relocating shards (range of keys) in a FoundationDB cluster. Its key goals are:</p>

Balancing load across servers</li>
Handling failures by redistributing data</li>
Ensuring optimal data placement for performance reliability</li>
</ul>
🔗</a>Data Distributor wording</h2>
The architecture and behavior of the Data Distributor</strong> are documented in the official design document</a>, and introduce the following concepts:</p>

Machine</strong>: A failure domain in FoundationDB, often considered equivalent to a rack.</li>
Shard</strong>: A range of key-values—essentially a contiguous block of the database keyspace.</li>
Server Team</strong>: A group of k</code> processes (where k</code> is the replication factor) hosting the same shard.</li>
Machine Team</strong>: A collection of k</code> machines, ensuring fault isolation for redundancy.</li>
</ul>
The term "machine" in FoundationDB’s documentation often translates better as "rack"</strong> in many practical cases. Using racks makes the Machine Team's role clearer: it ensures fault isolation by storing copies of data in different racks.</p>
🔗</a>Debug DD with status json</code></h2>
Your first input point should be to have a look at the team_trackers</code> key in the status json</code>. The JSON should contain enough information for basic monitoring:</p>
"</span>team_trackers</span>": [
</span>  {
</span>    "</span>primary</span>": </span>true</span>,
</span>    "</span>unhealthy_servers</span>": </span>0</span>,
</span>    "</span>state</span>": {
</span>      "</span>healthy</span>": </span>true</span>,
</span>      "</span>name</span>": "</span>healthy_rebalancing</span>"
</span>    }
</span>  }
</span></code></pre>
🔗</a>Debug DD with Trace events</h2>
FoundationDB provides a robust tracing system where each process generates detailed events in either XML or JSON formats. To troubleshoot the Data Distributor</strong>, you first need to locate the process it has been elected to. From there, trace events can be analyzed to understand shard movements, priorities, and failures.</p>
One particularly important attribute in these events is the Priority</code> field. This field determines the precedence of shard placement or redistribution tasks:</p>
init</span>( PRIORITY_RECOVER_MOVE, </span>110 </span>);
</span>init</span>( PRIORITY_REBALANCE_UNDERUTILIZED_TEAM, </span>120 </span>);
</span>init</span>( PRIORITY_REBALANCE_OVERUTILIZED_TEAM, </span>122 </span>);
</span>init</span>( PRIORITY_TEAM_UNHEALTHY, </span>700</span>);
</span>init</span>( PRIORITY_SPLIT_SHARD, </span>950 </span>);
</span></code></pre>
A full list of defined priorities can be found in the Knobs file</a>, providing useful insights into how tasks are scheduled.</p>
EDIT: Yes, SPLIT_SHARD</code> has an higher priority! See https://bsky.app/profile/alexmillerdb.bsky.social/post/3ljsqqvfslc24</a>.</p>
🔗</a>ServerTeamInfo</code> Event</h3>
Understanding the state of server teams is essential since the Data Distributor schedules data movements based on real-time metrics. The fdbcli</code> command triggerddteaminfolog</code> triggers informative logs by invoking printSnapshotTeamsInfo</a>.</p>
{
</span>  "</span>Type</span>": "</span>ServerTeamInfo</span>",
</span>  "</span>Priority</span>": "</span>709</span>",
</span>  "</span>Healthy</span>": "</span>0</span>",
</span>  "</span>TeamSize</span>": "</span>3</span>",
</span>  "</span>MemberIDs</span>": "</span>5a69... 5fc1... 8718...</span>",
</span>  "</span>LoadBytes</span>": "</span>1135157527</span>",
</span>  "</span>MinAvailableSpaceRatio</span>": "</span>0.94108</span>"
</span>}
</span></code></pre>
🔗</a>ServerTeamPriorityChange</code> Event</h3>
This event is logged when server team priorities change, often indicating server failures or rebalancing actions.</p>
{
</span>  "</span>Type</span>": "</span>ServerTeamPriorityChange</span>",
</span>  "</span>Priority</span>": "</span>950</span>",
</span>  "</span>TeamID</span>": "</span>e9b362decbafbd81</span>"
</span>}
</span></code></pre>
🔗</a>RelocateShard</code> Event</h3>
This event tracks shard movement between teams to maintain balance.</p>
{
</span>  "</span>Type</span>": "</span>RelocateShard</span>",
</span>  "</span>Priority</span>": "</span>120</span>", </span>// PRIORITY_REBALANCE_UNDERUTILIZED_TEAM
</span>  "</span>RelocationID</span>": "</span>3f1290654949771e</span>"
</span>}
</span></code></pre>
Again, the most useful field is the priority, indicating why it is relocated.</p>
🔗</a>"ValleyFiller" and "MountainChopper" Mechanisms</h3>
To optimize shard placement, FoundationDB employs two balancing strategies:</p>

ValleyFiller</strong>: Fills underutilized servers (the valleys</strong>) with data to balance the load.</li>
MountainChopper</strong>: Redistributes shards from overutilized servers (the mountains</strong>) to spread the load evenly.</li>
</ul>
Both logs will have a SourceTeam</code> and DestTeam</code> to use in other traces:</p>
{
</span>  "</span>Type</span>": "</span>BgDDValleyFiller</span>",
</span>  "</span>QueuedRelocations</span>": "</span>0</span>",
</span>  "</span>SourceTeam</span>": "</span>TeamID 95819f0d3d7ea40d</span>",
</span>  "</span>DestTeam</span>": "</span>TeamID 0817e6fe3135e6f6</span>",
</span>  "</span>ShardBytes</span>": "</span>398281250</span>"
</span>}
</span></code></pre>
{
</span>  "</span>Type</span>": "</span>BgDDMountainChopper</span>",
</span>  "</span>QueuedRelocations</span>": "</span>0</span>",
</span>  "</span>SourceTeam</span>": "</span>TeamID 95819f0d3d7ea40d</span>",
</span>  "</span>DestTeam</span>": "</span>TeamID e17dcecd86547e09</span>",
</span>  "</span>ShardBytes</span>": "</span>308000000</span>"
</span>}
</span></code></pre>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Ensuring Safety in FoundationDB's Rust Crate
Pierre Zemb — Tue, 11 Feb 2025 00:00:00 +0100
As we approach 5 million downloads of the FoundationDB Rust crate</a> (4,998,185 at the time of writing), I wanted to share some insights into how I ensure the safety of the crate. Being the primary maintainer of a database driver comes with responsibility, but I sleep well at night knowing that we have robust safety measures in place.</p>
🔗</a>Crate Overview</h2>
The Rust crate, foundationdb-rs</code>, provides bindings to interact with FoundationDB's C API (libfdb</code>). It has around 13k lines of code and is used by companies (like Clever Cloud) and projects (such as Apache OpenDAL, SurrealDB). Having experienced numerous outages and issues with drivers and distributed systems, I understand the importance of safety. To ensure the safety of the crate, we need to focus on three layers:</p>

The underlying client, libfdb</code>,</li>
The crate itself,</li>
The code that uses the crate.</li>
</ul>
Let's dig into each of these areas.</p>
🔗</a>libfdb Safety</h2>
This is the simplest part. libfdb</code>'s safety is guaranteed by FoundationDB's simulation framework</a>. Therefore, we can consider it safe.</p>
🔗</a>Classic testing suite</h3>
Since we are using a C library, we need to use FFI (Foreign Function Interface) and unsafe code blocks. With around 130 unsafe blocks, we must be extra careful when calling C code, ensuring all preconditions are met. Naturally, we conduct extensive testing, but most importantly, we run tests in high-variety environments:</p>

On multiple operating systems (Ubuntu, macOS)</li>
On multiple FoundationDB versions (from FDB 6.1 to 7.3)</li>
On multiple Rust compiler versions (Minimum Supported Rust Version or MSRV, stable, beta, nightly)</li>
</ul>
The most useful tests are run on the nightly Rust compiler, as we can catch new behaviors in the Rust compiler early</a>.</p>
While these testing practices provide significant coverage, the most powerful tool we utilize comes from FoundationDB’s maintainers: the BindingTester</code>.</p>
🔗</a>The BindingTester</h3>
FoundationDB is renowned for its simulation and testing</a> frameworks. Bindings are no exception. They developed the BindingTester, a cross-language validation suite ensuring that all bindings behave correctly and consistently across different languages.</p>
The BindingTester uses a stack-based machine</a> to queue operations for FoundationDB. A program then reads the stack and performs the operations. These operations are run twice: once in the target environment and once against a reference implementation. Any differences are reported by the BindingTester.</p>
It looks like this:</p>
./bindings/bindingtester/bindingtester.py --num-ops</span> 1000</span> --api-version</span> 730</span> --test-name</span> api</span> --compare</span> python rust
</span>
</span>Creating</span> test at API version 730
</span>Generating</span> api test at seed 3208032894 with 1000 op(s) </span>and</span> 1 concurrent tester(s)</span>...
</span>
</span># Inserting Rust tests
</span>Inserting</span> test into database...
</span>Running</span> tester '</span>/home/runner/work/foundationdb-rs/foundationdb-rs/target/debug/bindingtester test_spec 730</span>'...
</span>
</span>Reading</span> results from '</span>(</span>'tester_output'</span>, </span>'workspace'</span>)</span>'...
</span>Reading</span> results from '</span>(</span>'tester_output'</span>, </span>'stack'</span>)</span>'...
</span>
</span># Inserting Python tests
</span>Inserting</span> test into database...
</span>Running</span> tester '</span>python /home/runner/work/foundationdb-rs/foundationdb-rs/target/foundationdb_build/foundationdb/bindings/bindingtester/../python/tests/tester.py test_spec 730</span>'...
</span>
</span>Reading</span> results from '</span>(</span>'tester_output'</span>, </span>'workspace'</span>)</span>'...
</span>Reading</span> results from '</span>(</span>'tester_output'</span>, </span>'stack'</span>)</span>'...
</span>
</span># Comparing the results
</span>Comparing</span> results from '</span>(</span>'tester_output'</span>, </span>'workspace'</span>)</span>'...
</span>Comparing</span> results from '</span>(</span>'tester_output'</span>, </span>'stack'</span>)</span>'...
</span>Test</span> with seed 3208032894 and concurrency 1 had 0 incorrect result(s) </span>and</span> 0 error(s) </span>at</span> API version 730
</span>Completed</span> api test with random seed 3208032894 and 1000 operations
</span></code></pre>
The great advantage of this method is that the tests are seeded, meaning the operations are:</p>

randomly selected to cover all binding usages,</li>
deterministic, so a failing seed can be replayed locally.</li>
</ul>
Combined with code coverage, this gives us a good idea of what has been tested (though code coverage may vary).</p>
We run the BindingTester</code> every hour</strong> on our GitHub actions, amounting to around 219 days of continuous testing each month</strong> (316,335 minutes of correctness last month according to Github).</p>
🔗</a>User Safety</h2>
Thanks to libfdb</code> and the BindingTester</code>, we can ensure that the library is quite safe. But what about the user's code? How can we help users ensure their code can handle all of FoundationDB's caveats, such as commit_unknown_result</a>? We added a great feature: the ability to include Rust code within FDB's simulation framework</strong>.</p>
We can implement an Rust workload with the following Trait:</p>
pub trait </span>RustWorkload {
</span>    </span>fn </span>description</span>(&</span>self</span>) -> String;
</span>    </span>fn </span>setup</span>(&</span>'static mut </span>self</span>, </span>db</span>: SimDatabase, </span>done</span>: Promise);
</span>    </span>fn </span>start</span>(&</span>'static mut </span>self</span>, </span>db</span>: SimDatabase, </span>done</span>: Promise);
</span>    </span>fn </span>check</span>(&</span>'static mut </span>self</span>, </span>db</span>: SimDatabase, </span>done</span>: Promise);
</span>    </span>fn </span>get_metrics</span>(&</span>self</span>) -> Vec<Metric>;
</span>    </span>fn </span>get_check_timeout</span>(&</span>self</span>) -> </span>f64</span>;
</span>}
</span></code></pre>
Which can be runned inside the simulation while injecting some faults:</p>
fdbserver -r</span> simulation</span> -f</span> /root/atomic.toml</span> -b</span> on</span> --trace-format</span> json
</span>
</span># Choosing a random seed
</span>Random</span> seed is 394378360...
</span>
</span># Then, everything is derived from the seed, including:
</span># * cluster topology,
</span># * cluster configuration,
</span># * timing to inject faults,
</span># * operations to run
</span># * ...
</span>Datacenter</span> 0: 3/12 machines, 1/1 coordinators
</span>Datacenter</span> 1: 3/12 machines, 0/1 coordinators
</span>Datacenter</span> 2: 3/12 machines, 0/1 coordinators
</span>Datacenter</span> 3: 3/12 machines, 0/1 coordinators
</span>
</span># Starting the Atomic workload
</span>Run</span> test:AtomicWorkload start
</span>
</span>AtomicWorkload</span> complete
</span>checking</span> test (AtomicWorkload)</span>...
</span>
</span>5</span> test clients passed; </span>0</span> test clients failed
</span>Run</span> test:AtomicWorkload Done.
</span>
</span>1</span> tests passed; </span>0</span> tests failed.
</span>
</span>Unseed:</span> 66324
</span>Elapsed:</span> 405.055622 simsec, 30.342000 real seconds
</span></code></pre>
This has been a major keypoint</strong> for us to develop and operate Materia, Clever Cloud's serverless database offer</a>, as we can enjoy the same Simulation framework used by FDB's core engineers for layer engineering 🤯</p>
🔗</a>Closing words</h2>
As with any open-source project, there is always more to accomplish, but I am quite satisfied with the current level of safety provided by the crate. I would like to express my gratitude to the FoundationDB community for developing the BindingTester, and former contributors to the crate.</p>
I also would like to encourage everyone to explore the simulation framework. Integrating Rust code within this framework has allowed us to harness the full potential of simulation without the need to build our own, and it has forever changed my perspective on testing and software engineering.</p>
There is a strong likelihood that future blog posts will focus on simulation, so feel free to explore the simulation tags</a>.</p>


Back in engineering!
Pierre Zemb — Wed, 15 Jan 2025 00:37:27 +0100
Time flies—it’s already 2025! Looking back, 2024 was an incredibly fast-paced year for me professionally as an Engineering Manager. This year, I’ve decided to take a new direction and return to a more engineering-focused role.</p>
I moved in a management position early 2023. It was a time where my company was growing fast (from 20-ish to 60-ish), and we needed coordination to ship things out in parallel, but also to migrate customers (and ourselves!) to new datacenters</a>. With my on-call experience, I helped our CTO, Steven, bootstrap two data-oriented teams and later led the Materia team, shaping its technology.</p>
During this time, we successfully launched Materia KV in its Alpha version</a> and built strong internal trust in FoundationDB. Our largest cluster effortlessly handles hundreds of thousands of writes per second, and the ability to simulate Rust code within FDB’s simulation framework</a> has significantly boosted our developers' confidence.</p>
Management is a rewarding challenge, like unlocking a new skill tree to walk through. Resources like The Manager's Path</a> or Engineering Management for the Rest of Us</a> can give you a head-start and should be read by anyone. Blogs like Charity Majors</a> are also useful to read, especially The Engineer/Manager Pendulum</a> and its follow-up</a>.</p>
After nearly two years, I feel it's the right time to return to core engineering. I just miss it.  From the start, my manager and I had an understanding that I could transition back whenever I wanted to—and that time is now. I've found an excellent manager to lead the team, giving me the space to focus on the technical side of Materia. I also should have more time to work around open-source, and I'm looking forward to it.</p>


Redwood’s memory tuning in FoundationDB
Pierre Zemb — Mon, 22 Apr 2024 00:37:27 +0100
While FoundationDB allows you to obtain sub-milliseconds transactions’s latency without any knob-tuning, we had to bump a bit memory usage for Redwood under certain usage and workload. The following configuration has been tested on clusters from 7.1 to 7.3.</p>
🔗</a>BTree page cache</h2>
We discovered the issue when we saw a performance decrease on our cluster storing time-series data. Our cluster was reporting some high disk-business, causing outages:</p>
10.0.3.23:4501 ( 65% cpu; 61% machine; 0.010 Gbps; 93% disk IO; 7.5 GB / 7.4 GB RAM  )
</span>10.0.3.24:4501 ( 61% cpu; 61% machine; 0.010 Gbps; 87% disk IO; 9.7 GB / 7.4 GB RAM  )
</span>10.0.3.25:4501 ( 69% cpu; 61% machine; 0.010 Gbps; 93% disk IO; 5.4 GB / 7.4 GB RAM  )
</span></code></pre>
This was our first «we need to dig into this» moment with FDB. We couldn’t find the root-cause and we asked the community. Turns out we had a classic page-cache issue which was spotted by Markus Pilman</a> and William Dowling</a>. While the trace files are pretty verbose, they are containing a lot of information like this one:</p>
"PagerCacheHit": "39852",
</span>"PagerCacheMiss": "25903",
</span></code></pre>
Yep, that’s a 40% cache-miss ratio over 5s 😱 This is why the disk was so busy, spending his time moving pages back and forth. We need to bump the memory, but how much? The general recommandation that worked for us is to target around 1-2% of the kvstore_used_bytes</code> metrics. As we have around 1TiB of data per StorageServer, we can add the following config key:</p>
cache_memory = 10GiB
</span></code></pre>
Which fixed our cache-miss issue 🎉</p>
"PagerCacheHit": "51968",
</span>"PagerCacheMiss": "432",
</span></code></pre>
 </p>
🔗</a>Byte Sample memory usage</h2>
But our problems are still unresolved, as we are still seeing some OOM 😭 Because this cluster is storing time-series data, each StorageServers is holding around 1TiB of data. As we were holding more and more data, we saw more and more OOM errors on our fdbmonitor</code> logs. Something was growing linearly with our usage and needed tuning. This time, we had help from Steve Atherton</a> which pointed us towards the direction of the Byte Sample</a>:</p>

There is a data structure that storage servers have called the Byte Sample which stores a deterministic random sample of keys. This data is persisted on disk in the storage engine and is loaded immediately upon storage server startup. Unfortunately, its size is not tracked or reported, but grows linearly with KV size and I suspect yours is somewhere around 4GB-6GB based on the memory usage I’ve seen for smaller storage KV sizes.</p>
</blockquote>
So, we need to add around 4GB more in the memory, but there is no config for that parameter. It needs to be embedded in the global memory</code> parameter. Let’s compute the right value!</p>
🔗</a>The global memory formula</h2>
By testing things on our clusters, we ended up with this formula:</p>
# Default is 2
</span>cache_memory = (1-2% of kvstore_used_bytes)GiB
</span># Default is 8
</span>memory = (8 + cache_memory + 4-6GB per TB of kvstore_used_bytes)GiB
</span></code></pre>
Which fixed all our memory issues with FoundationDB 🎉 And to be fair, this is the only things we needed to tune on our clusters, which is quite impressive 👀</p>
🔗</a>Special thanks</h2>
I would like to thank Markus, William and Steve from the FoundationDB community for their help 🤝</p>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


True idempotent transactions with FoundationDB 7.3
Pierre Zemb — Tue, 12 Mar 2024 00:37:27 +0100
I have been working around FoundationDB</a> for several years now, and the new upcoming version is fixing one of the most evil and painful caveats you can deal with when writing layers: commit_unknown_result</code>.</p>
🔗</a>Transactions with unknown results</h2>
When you start writing code with FDB, you may be under the assertions that given the database’s robustness, you will not experience some strange behavior under certain failure scenarios. Turns out, there is one scenario that is possible to reach, and quickly explained in the official documentation</a>:</p>

As with other client/server databases, in some failure scenarios a client may be unable to determine whether a transaction succeeded. In these cases, commit() will raise a commit_unknown_result</code></a> exception. The on_error() function treats this exception as retriable, so retry loops that don’t check for commit_unknown_result</code></a> could execute the transaction twice. In these cases, you must consider the idempotency of the transaction.</p>
</blockquote>
While having idempotent retry loops is possible, sometimes it is not possible, for example when using atomic operations to keep track of statistics.</p>

Is this problem worth fixing? Seems a really edgy case 🤔</p>
</blockquote>
It truly depends whether it is acceptable for your transaction to be committed twice. For most of the case, it is not, but sometimes developers are not aware of this behavior, leading to errors. This is one of the reasons why we worked and open-sourced a way to embed rust-code within FoundationDB’s simulation framework. Using the simulation crate, your layer can be tested like FDB, and I can assure you: you will see</strong> those transactions in simulation 🙈.</p>
This behavior has given headache to my colleagues, as we tried to bypass correctness and validation code in simulation when transactions state are unknown, and who could blame us? Validate the correctness of your code is hard when certains transactions (for example, one that could clean everything) are “maybe committed”. Fortunately, the community has released a workaround for this: automatic idempotency</code></a>.</p>
🔗</a>Automatic idempotency</h2>
The documentation is fairly explicit:</p>

Use the automatic_idempotency transaction option to prevent commits from failing with commit_unknown_result</code> at a small performance cost.</p>
</blockquote>
The option appeared in FoundationDB 7.3, and could solve our issue. I decided to give it a try and modify the foundationdb-simulation crate example. The example is trying to use a atomic increment under simulation. Before 7.1, during validation, we had to write some code</a> that looks like this:</p>
// We don't know how much maybe_committed transactions has succeeded,
</span>// so we are checking the possible range
</span>if </span>self</span>.success_count <= count
</span>   && count <= </span>self</span>.expected_count + </span>self</span>.maybe_committed_count {
</span>// ...
</span></code></pre>
As I was adding 7.3 support in the crate, I decided to update the example and try the new option:</p>
// Enable idempotent txn
</span> trx.</span>set_option</span>(TransactionOption::AutomaticIdempotency)?;
</span></code></pre>
If the behavior is correct, I can simplify my consistency checks:</p>
if </span>self</span>.success_count == count {
</span>    </span>self</span>.context.</span>trace</span>(
</span>        Severity::Info,
</span>        "</span>Atomic count match</span>",
</span>        details![],
</span>     );
</span>}
</span>// ...
</span></code></pre>
I’ve been running hundreds of seeds on my machine and everything works great: no more maybe-committed transactions! Now that 7.3 support is merged in the rust bindings, we will be able to expands our testing thanks to our simulation farm. I'm also looking to see the performance impact of the feature, even if I'm pretty sure that it will outperform any layer-work.</p>
This is truly a very useful feature and I hope this option will be turned on by default on the next major release. The initial PR can be found here</a>.</p>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


The unseen treasures of Infrastructure Engineering: Academic Papers
Pierre Zemb — Mon, 22 Jan 2024 15:37:27 +0100
</p>
I really like using RSS feeds. My Feedly account has more than 190 feeds, all neatly organized by categories. They help me keep up with new ideas and interesting blog posts about engineering. But there's another source of information I've been using for a long time that not many people know about: academic papers</strong>.</p>
You can discover details about infrastructure that you might not find in regular blog posts. Academic papers, unlike typical blog content, often dive deeper</strong> into specific aspects of infrastructure. They provide more in-depth information, uncovering details that are not commonly discussed. So, if you're interested in gaining a more comprehensive understanding of infrastructure-related topics, exploring academic papers can be really worthwile.</p>

Sounds a bit too academic, doesn't it? 🤔</p>
</blockquote>
I don't think so!  It's true that academic research can sometimes seem distant from everyday industry needs, but following both academic and industry tracks is beneficial. R&D from academia often lead to new ideas and technologies that eventually find their way into practical use.</p>
Moreover, numerous academic conferences feature a "industry track"</strong> that is essential to monitor.</p>

Aren't they too complex to read? 🤔</p>
</blockquote>
If you don't get everything right away, that's okay. Reading these smart papers might be a bit hard, but it's a skill that gets better with practice. And who knows, maybe you'll be inspired to write your own paper someday! 😉</p>

I'm intrigued! Where should I start?</p>
</blockquote>
Here's a short list of my go-to academic papers and conferences that you can follow for infrastructure engineering. Please note that many conferences exists on other subjects, like security and so.</p>
🔗</a>The USENIX community</h2>
🔗</a>OSDI</h3>
As part of the USENIX Association, the Operating Systems Design and Implementation</a> is an annual computer science conference that you shouldn't miss. You can catch most of the sessions online along with some useful slides.</p>
🔗</a>NSDI</h3>
In a similar fashion, the Networked Systems Design and Implementation</a> focuses on the design principles, implementation, and practical evaluation of networked and distributed systems.</p>
🔗</a>Usenix ATC</h3>
The Usenix Annual Technical Conference</a> is another classic to follow.</p>
🔗</a>The ACM family</h2>
🔗</a>SIGMOD</h3>
SIGMOD, or the Special Interest Group on Management of Data</a>, is an essential conference under the ACM umbrella, focusing on the management and organization of data.</p>
🔗</a>DaMoN</h3>
Held with ACM SIGMOD/PODS, you can also find the Data Management on New Hardware </a>.</p>
🔗</a>SoCC</h3>
The Symposium on Cloud Computing</a> or SoCC for short belongs to ACM. It has a bit less content, as videos are not published, but you should keep it in your watchlist.</p>
🔗</a>SOSP</h3>
The Symposium on Operating Systems Principles</a> is another noteworthy conference in the ACM family. It's a top-tier venue for discussing operating systems research. Stay tuned for updates on the latest breakthroughs and innovative ideas.</p>
🔗</a>Others</h2>
🔗</a>VLDB</h3>
Not belonging to USENIX or ACM, the Very Large Data Bases</a> (VLDB) conference is a key event in the database community. It provides a platform for researchers and professionals to exchange ideas on managing and analyzing large-scale datasets.</p>
🔗</a>CIDR</h3>
The Conference on Innovative Data Systems Research</a> (CIDR) is a systems-oriented conference, complementary in its mission to the mainstream database conferences like SIGMOD and VLDB, emphasizing the systems architecture perspective.</p>
🔗</a>Cool papers examples</h2>

This is a nice list, but how about some paper examples that you</strong> like?🤔</p>
</blockquote>
Sure! Here's a quick list with some infrastructure-related informations:</p>

On-demand Container Loading in AWS Lambda</a></li>
Scalable OLTP in the Cloud: What's the BIG DEAL?</a></li>
Kora: A Cloud-Native Event Streaming Platform For Kafka</a></li>
Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3</a></li>
FoundationDB: A Distributed, Unbundled, Transactional Key Value Store</a></li>
Virtual consensus with Delos</a></li>
</ul>
I'm also trying to organize them into my Zotero's library</a>.</p>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Best resources to learn about data and distributed systems
Pierre Zemb — Mon, 17 Jan 2022 01:37:27 +0100
Learning distributed systems is tough. You need to go through a lot of academic papers, concepts, code review, before being able to have a global pictures. Thankfully, there is a lot of resources out there that can help you to get started.  Here's a list of resources I used to learn distributed systems. I will keep this blogpost up-to-date with books, conferences, and so on.</p>
</p>

A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.</p>
-Lamport, 1987</p>
</blockquote>
🔗</a>Reading 📚</h2>
🔗</a>Designing Data-Intensive Applications</h3>
Let's start by one of my favorite book, Designing Data-Intensive Applications</a>, written by Martin Kleppmann</a>. This is by far the most practical book you will ever find about distributed systems. It covers:</p>

Data models, query languages and encoding,</li>
Replication, partitioning, the associated troubles, consistency, consensus,</li>
batch and stream processing.</li>
</ul>

NoSQL… Big Data… Scalability… CAP Theorem… Eventual Consistency… Sharding…</p>
Nice buzzwords, but how does the stuff actually work?</p>
As software engineers, we need to build applications that are reliable, scalable and maintainable in the long run. We need to understand the range of available tools and their trade-offs. For that, we have to dig deeper than buzzwords.</p>
This book will help you navigate the diverse and fast-changing landscape of technologies for storing and processing data. We compare a broad variety of tools and approaches, so that you can see the strengths and weaknesses of each, and decide what’s best for your application.</p>
</blockquote>
🔗</a>Database Internals</h3>
Database Internals</a>, written by Alex Petrov</a>, is a fantastic book for anyone wondering how a database works. I recommend reading it after Designing Data-Intensive Applications</code>, as the author dives in more details compared to Martin's book.</p>

Have you ever wanted to learn more about Databases but did not know where to start? This is a book just for you.</p>
We can treat databases and other infrastructure components as black boxes, but it doesn’t have to be that way. Sometimes we have to take a closer look at what’s going on because of performance issues. Sometimes databases misbehave, and we need to find out what exactly is going on. Some of us want to work in infrastructure and develop databases. This book’s main intention is to introduce you to the cornerstone concepts and help you understand how databases work.</p>
The book consists of two parts: Storage Engines and Distributed Systems since that’s where most of the differences between the vast majority of databases is coming from.</p>
</blockquote>
🔗</a>Distributed Systems</h3>
Maarten van Steen</a> wrote a book called Distributed Systems 3rd edition</a>. It is a nice book which you can get a digital copy of this book for free.</p>

Distributed systems are like 3D brain teasers: easy to disassemble; hard to put together.</p>
</blockquote>
🔗</a>Understanding Distributed Systems</h3>
If you are not a backend engineer but still curious about distributed systems, I highly recommend Understanding Distributed Systems</a>. Roberto Vitillo</a> is doing an insane job to vulgarize the subject.</p>

Want to learn how to build scalable and fault-tolerant cloud applications?</p>
This book will teach you the core principles of distributed systems so that you don’t have to spend countless hours trying to understand how everything fits together.</p>
</blockquote>
🔗</a>The Internals of PostgreSQL</h3>
PostgreSQL is getting a lot of love and traction these years, and Hironobu Suzuki</a> wrote a terrific book the about the The Internals of PostgreSQL</a>.</p>

PostgreSQL is a well-designed open-source multi-purpose relational database system which is widely used throughout the world. It is one huge system with the integrated subsystems, each of which has a particular complex feature and works with each other cooperatively. Although understanding of the internal mechanism is crucial for both administration and integration using PostgreSQL, its hugeness and complexity prevent it. The main purposes of this document are to explain how each subsystem works, and to provide the whole picture of PostgreSQL.</p>
</blockquote>
🔗</a>Jepsen blog</h3>
We are often using databases as a source of truth, but they are also pieces of software with bugs in it. Kyle Kingsbury is the most famous database-breaker with Jepsen</a>:</p>

Jepsen is an effort to improve the safety of distributed databases, queues, consensus systems, etc. We maintain an open source software library for systems testing, as well as blog posts and conference talks exploring particular systems’ failure modes. In each analysis we explore whether the system lives up to its documentation’s claims, file new bugs, and suggest recommendations for operators.</p>
</blockquote>
You will find analysis on many databases, such as CockroachDB, etcd, Kafka, MongoDB, and so on.</p>
🔗</a>Aphyr distsys class notes</h3>
Following Jepsen, here's a great bonus: Kyle is also teaching distributed systems, and his notes are available</a>.</p>
🔗</a>Distributed systems for fun and profit</h3>
Despite being free, Distributed systems for fun and profit</a> is an awesome book. The author, Mikito Takada</a> has done a terrific work to vulgarize distributed systems.</p>

I wanted a text that would bring together the ideas behind many of the more recent distributed systems - systems such as Amazon's Dynamo, Google's BigTable and MapReduce, Apache's Hadoop and so on.</p>
</blockquote>

In this text I've tried to provide a more accessible introduction to distributed systems. To me, that means two things: introducing the key concepts that you will need in order to have a good time reading more serious texts, and providing a narrative that covers things in enough detail that you get a gist of what's going on without getting stuck on details.</p>
</blockquote>
🔗</a>Translucent Databases</h3>
I really like the pitch of the book:</p>

Do you have personal information in your database?</p>
Do you keep files on your customers, your employees, or anyone else?</p>
Do you need to worry about European laws restricting the information you keep?</p>
Do you keep copies of credit card numbers, social security numbers, or other information that might be useful to identity thieves or insurance fraudsters?</p>
Do you deal with medical records or personal secrets?</p>
Most database administrators have some of these worries. Some have all of them. That's why database security is so important.</p>
This new book, Translucent Databases, describes a different attitude toward protecting the information.</p>
</blockquote>
Translucent Databases</a> is a short book, focus on how to store sensitive data. You will find several dozen examples of interesting case studies on how to efficiently and privately store sensitive data. A must-have.</p>
🔗</a>The Art of PostgreSQL</h3>
The Art of PostgreSQL</a> is all about showing the power of both SQL and PostgreSQL. It explains the how's and why's of using Postgres's many feature, and how you, as a developers, can take advantages of it. A brilliant book that should be read by every developer.</p>

This book is for developers, covering advanced SQL techniques for data processing. Learn how to get exactly the result set you need in your application’s code!</p>
Learn advanced SQL with practical examples and datasets that help you get the most of the book! Every query solves a practical use case and is given in context.</p>
The book covers (de-)normalisation with simple practical examples to dive into this seemingly complex topic, including Caching and Indexing Strategy.</p>
Writing efficient SQL is easier than it looks, and begins with database modeling and writing clear code. The book teaches you how to write fast queries!</p>
</blockquote>
🔗</a>Readings in Database Systems</h3>
Another free book, Readings in Database Systems</a> is a great read if you are looking for an opinionated and short review on subject like architecture, engines, analytics and so on.</p>

Readings in Database Systems (commonly known as the "Red Book") has offered readers an opinionated take on both classic and cutting-edge research in the field of data management since 1988. Here, we present the Fifth Edition of the Red Book — the first in over ten years.</p>
</blockquote>
🔗</a>Watching 📺</h2>
🔗</a>CMU Database Group</h3>
The Database Group at Carnegie Mellon University have been publishing a lot of contents, including:</p>

Intro to Database Systems lecture</a></li>
Advanced Database Systems lecture</a></li>
</ul>
which are the best lectures about database in my opinion.</p>
I also recommend their Quarantine database talks playlists:</p>

the "Quarantine Database Tech Talks" is a on-line seminar series at Carnegie Mellon University with leading developers and researchers of database systems. Each speaker will present the implementation details of their respective systems and examples of the technical challenges that they faced when working with real-world customers.</p>
</blockquote>

Vaccination Database Tech Talks First Dose</a></li>
Vaccination Database Tech Talks Second Dose</a></li>
</ul>
🔗</a>Distributed Systems lecture series</h3>
Martin Kleppmann</a>(Designing Data Intensive applications</code>'s author) published an 8-lecture series on distributed systems</a>:</p>

This video is part of an 8-lecture series on distributed systems, given as part of the undergraduate computer science course at the University of Cambridge.</p>
</blockquote>
🔗</a>Academic conferences</h3>
Keeping track of the academic world is not easy, but thankfully, we can keep track of several academic conferences which are data-related, including:</p>

CIDR</a></li>
SIGMOD/PODS</a></li>
VLDB</a></li>
PaPoC</a></li>
</ul>
🔗</a>Industrial conference</h3>
There is not much database-focused conferences, but you will be interested to see talks from:</p>

HydraConf</a></li>
HYTRADBOI</a></li>
</ul>
🔗</a>DistSys Reading Group sessions</h3>
If you are looking for explanations about a distributed systems paper, you may be interested in the DistSys Reading Group</a>:</p>

Every week we present and discuss one distributed systems paper. We try to focus on relatively new papers, although we occasionally break this rule for some important older publications. The main objective of this group is to share knowledge through the discussion. Our participants come from academia and industry and often carry a unique perspective and expertise on the subject matter.</p>
</blockquote>
Every session can be found on their YouTube channel</a>.</p>
🔗</a>Coding 🧑‍💻</h2>
🔗</a>Maelstrom</h3>
Ever wonder to develop your own toy distributed systems? Fear no more, you can use Maelstrom</a> for that!</p>

Maelstrom is a workbench for learning distributed systems by writing your own. It uses the Jepsen testing library to test toy implementations of distributed systems. Maelstrom provides standardized tests for things like "a commutative set" or "a transactional key-value store", and lets you learn by writing implementations which those test suites can exercise. It's used as a part of a distributed systems workshop by Jepsen.</p>
</blockquote>

Maelstrom provides a range of tests for different kinds of distributed systems, built on top of a simple JSON protocol via STDIN and STDOUT. Users write servers in any language. Maelstrom runs those servers, sends them requests, routes messages via a simulated network, and checks that clients observe expected behavior. You want to write Plumtree in Bash? Byzantine Paxos in Intercal? Maelstrom is for you.</p>
</blockquote>
🔗</a>PingCAP's Talent Plan</h3>
PingCAP is the company behind the tidb/tikv stack, a new distributed systems. They developed their own open source training program</a>:</p>

Talent Plan is an open source training program initiated by PingCAP. It aims to create or combine some open source learning materials for people interested in open source, distributed systems, Rust, Golang, and other infrastructure knowledge. As such, it provides a series of courses focused on open source collaboration, rust programming, distributed database and systems.</p>
</blockquote>
I went through the Raft project in Rust and I learned a lot!</p>
🔗</a>Patterns of Distributed Systems</h3>
Unmesh Joshi</a> is writing an on-going serie called Patterns of Distributed Systems</a>:</p>

Distributed systems provide a particular challenge to program. They often require us to have multiple copies of data, which need to keep synchronized. Yet we cannot rely on processing nodes working reliably, and network delays can easily lead to inconsistencies. Despite this, many organizations rely on a range of core distributed software handling data storage, messaging, system management, and compute capability. These systems face common problems which they solve with similar solutions. This article recognizes and develops these solutions as patterns, with which we can build up an understanding of how to better understand, communicate and teach distributed system design.</p>
</blockquote>
🔗</a>Reading lists 👀</h2>
🔗</a>Dan Creswell's reading List</h3>
If you want more contents, Dan Creswell</a> has a nice Distributed Systems Reading List</a> 🚀</p>

Thank you</strong> for reading my post! Feel free to react to this article, you can find me on Twitter</a>.</p>


Crafting row keys in FoundationDB
Pierre Zemb — Sun, 21 Feb 2021 00:24:27 +0100
</p>
As I'm working on my latest contribution around FoundationDB and Rust</a>, I had the chance to dig a bit into how FoundationDB's bindings are offering helpers to generate keys. Their approach is interesting enough to deserve a blogpost 😎</p>
🔗</a>Row key?</h2>
When you are using a key/value store, the design of the row key</code> is extremely important, as this will define how well:</p>

your scans will be optimized,</li>
your puts will be spread,</li>
you will avoid hot-spotting</code> a shard/region.</li>
</ul>
If you need more information on row keys</code>, I recommend going through these links before moving on:</p>

"Designing your schema" BigTable documentation</a></li>
"Rowkey Design" HBase documentation</a></li>
</ul>
🔗</a>Hand-crafting row keys</h2>
Most of the time, you will need to craft the row key</code> "by hand", like this for an HBase's app</a>:</p>
// Prefix + classId + labelsId + timestamp
</span>// 128 bits
</span>byte[]</span> rowkey = </span>new byte</span>[</span>Constants</span>.</span>HBASE_RAW_DATA_KEY_PREFIX</span>.length + </span>8 </span>+ </span>8 </span>+ </span>8</span>];
</span>
</span>System</span>.</span>arraycopy</span>(</span>Constants</span>.</span>HBASE_RAW_DATA_KEY_PREFIX</span>, </span>0</span>, rowkey, </span>0</span>, </span>Constants</span>.</span>HBASE_RAW_DATA_KEY_PREFIX</span>.length);
</span>// Copy classId/labelsId
</span>System</span>.</span>arraycopy</span>(</span>Longs</span>.</span>toByteArray</span>(msg.</span>getClassId</span>()), </span>0</span>, rowkey, </span>Constants</span>.</span>HBASE_RAW_DATA_KEY_PREFIX</span>.length, </span>8</span>);
</span>System</span>.</span>arraycopy</span>(</span>Longs</span>.</span>toByteArray</span>(msg.</span>getLabelsId</span>()), </span>0</span>, rowkey, </span>Constants</span>.</span>HBASE_RAW_DATA_KEY_PREFIX</span>.length + </span>8</span>, </span>8</span>);
</span></code></pre>
Or maybe you will wrap things in a function like this in Go</a>:</p>
// EncodeRowKey encodes the table id and record handle into a kv.Key
</span>func </span>EncodeRowKey</span>(</span>tableID </span>int64</span>, </span>encodedHandle </span>[]</span>byte</span>) </span>kv</span>.</span>Key </span>{
</span> </span>buf </span>:= </span>make</span>([]</span>byte</span>, </span>0</span>, </span>prefixLen</span>+</span>len</span>(</span>encodedHandle</span>))
</span> </span>buf </span>= </span>appendTableRecordPrefix</span>(</span>buf</span>, </span>tableID</span>)
</span> </span>buf </span>= </span>append</span>(</span>buf</span>, </span>encodedHandle</span>...)
</span> </span>return </span>buf
</span>}
</span></code></pre>
Each time, you need to wrap the complexity of converting your objects to a row-key, by creating a buffer and write stuff in it.</p>
In our Java example, there is an interesting comment:</p>
// Prefix + classId + labelsId + timestamp
</span></code></pre>
If we are replacing some characters, we are not really far from:</p>
// (Prefix, classId, labelsId, timestamp)
</span></code></pre>
Which looks like a Tuple</code>(a collection of values of different types) and this is what FoundationDB is using as an abstraction to create keys 😍</p>
🔗</a>FDB's abstractions and helpers</h2>
🔗</a>Tuple</h3>
Instead of crafting bytes by hand, we are packing</code> a Tuple:</p>
// create a Tuple<String, i64> with ("tenant-42", 1)
</span>let</span> tuple = (String::from("</span>tenant-42</span>"), </span>1</span>);
</span>
</span>// and compute a row-key from the Tuple
</span>let</span> row_key = foundationdb::tuple::pack::<(String, </span>i64</span>)>(&tuple);
</span></code></pre>
The generated row-key will be readable from any bindings, as it's construction is standardized. Let's print it:</p>
// and print-it in hexa
</span>println!("</span>{:#04X?}</span>", row_key);
</span></code></pre>
// can be verified with https://www.utf8-chartable.de/unicode-utf8-table.pl
</span>[
</span>    0x02,
</span>    0x74, // t
</span>    0x65, // e 
</span>    0x6E, // n
</span>    0x61, // a
</span>    0x6E, // n
</span>    0x74, // t
</span>    0x2D, // -
</span>    0x31, // 1
</span>    0x00, 
</span>    0x15,
</span>    0x2A, // 42
</span>]
</span></code></pre>
As you can see, pack</code> added some extra-characters. There are used to recognized the next type, a bit like when you are encoding/decoding some wire protocols. You can find the relevant documentation here</a>.</p>
Having this kind of standard means that we can easily decompose/unpack</code> it:</p>
// retrieve the user and the magic number In a Tuple (String, i64)
</span>let</span> from_row_key = foundationdb::tuple::unpack::<(String, </span>i64</span>)>(&row_key)?;
</span>
</span>println!("</span>user='</span>{}</span>', magic_number=</span>{}</span>", from_row_key.</span>0</span>, from_row_key.</span>1</span>);
</span>// user='tenant-42', magic_number=42
</span></code></pre>
Now that we saw Tuples</code>, let's dig in the next abstraction: subspaces</code></p>
🔗</a>Subspace</h3>
When you are working with key-values store, we are often playing with what we call keyspaces</code>, by dedicating a portion of the key to an usage, like this for example:</p>
/users/tenant-1/...
</span>/users/tenant-2/...
</span>/users/tenant-3/...
</span></code></pre>
Here, /users/tenant-1/</code> can be view like a prefix where we will put all the relevant keys. Instead of passing a simple prefix, FoundationDB is offering a dedicated structure called a Subspace</code>:</p>

A Subspace represents a well-defined region of keyspace in a FoundationDB database</p>
</blockquote>

It provides a convenient way to use FoundationDB tuples to define namespaces for different categories of data. The namespace is specified by a prefix tuple which is prepended to all tuples packed by the subspace. When unpacking a key with the subspace, the prefix tuple will be removed from the result.</p>
</blockquote>
As you can see, the Subspace</code> is heavily relying on FoundationDB's tuples, as we can pack</code> and unpack</code> it.</p>

As a best practice, API clients should use at least one subspace for application data.</p>
</blockquote>
Well, as we have now the tools to handle keyspaces easily, it is now futile to craft keys by hand 🙃 Let's create a subspace!</p>

</span>// create a subspace from the Tuple ("tenant-1", 42)
</span>let</span> subspace = Subspace::from((String::from("</span>tenant-1</span>"), </span>42</span>));
</span>
</span>// let's print the range
</span>println!("</span>start: {:#04X?}</span>\n</span> end: {:#04X?}</span>", subspace.</span>range</span>().</span>0</span>, subspace.</span>range</span>().</span>1</span>);
</span></code></pre>
We can see observe this:</p>
// can be verified with https://www.utf8-chartable.de/unicode-utf8-table.pl
</span>start: [
</span>    0x02,
</span>    0x74, // t
</span>    0x65, // e 
</span>    0x6E, // n
</span>    0x61, // a
</span>    0x6E, // n
</span>    0x74, // t
</span>    0x2D, // -
</span>    0x31, // 1
</span>    0x00, 
</span>    0x15,
</span>    0x2A, // 42
</span>    0x00,
</span>    0x00, // smallest possible byte
</span>]
</span>end: [
</span>    0x02,
</span>    0x74, // t
</span>    0x65, // e 
</span>    0x6E, // n
</span>    0x61, // a
</span>    0x6E, // n
</span>    0x74, // t
</span>    0x2D, // -
</span>    0x31, // 1
</span>    0x00, 
</span>    0x15,
</span>    0x2A, // 42
</span>    0x00,
</span>    0xFF, // biggest possible byte
</span>]
</span></code></pre>
Which make sens, if we take ("tenant-1", 42)</code> as a prefix, then the range for this subspace will be between ("tenant-1", 42, 0x00)</code> and ("tenant-1", 42, 0xFF)</code></p>
🔗</a>Directory</h3>
Now that we know our way around Tuples</code> and Subspaces</code>, we can now talk about what I'm working on, which is the Directory</code>. Let's have a look at the relevant documentation</a>:</p>

FoundationDB provides directories (available in each language binding) as a tool for managing related subspaces.</p>
</blockquote>

Directories are a recommended approach for administering applications. Each application should create or open at least one directory to manage its subspaces.</p>
</blockquote>
Okay, let's see the API(in Go, as I'm working on the Rust API):</p>
subspace</span>, </span>err </span>:= </span>directory</span>.</span>CreateOrOpen</span>(</span>db</span>, []</span>string</span>{"</span>application</span>", "</span>my-app</span>", "</span>tenant</span>", "</span>tenant-42</span>"}, </span>nil</span>)
</span>if </span>err </span>!= </span>nil </span>{
</span> </span>log</span>.</span>Fatal</span>(</span>err</span>)
</span>}
</span>
</span>fmt</span>.</span>Printf</span>("</span>%+v</span>\n</span>", </span>subspace</span>.</span>Bytes</span>())
</span>// [21 18]
</span></code></pre>
We can see that we have a shorter subspace! The directory</code> allows you to generate some integer that will be bind to a path, like here "application", "my-app", "tenant", "tenant-42"</code>.</p>
There are two advantages to this:</p>

shorter keys,</li>
cheap metadata operations like List</code> or Move</code>:</li>
</ul>
// list all tenant in "application", "my-app":
</span>tenants</span>, </span>err </span>:= </span>directory</span>.</span>List</span>(</span>db</span>, []</span>string</span>{"</span>application</span>", "</span>my-app</span>", "</span>tenant</span>"})
</span>if </span>err </span>!= </span>nil </span>{
</span> </span>log</span>.</span>Fatal</span>(</span>err</span>)
</span>}
</span>fmt</span>.</span>Printf</span>("</span>%+v</span>\n</span>", </span>tenants</span>)
</span>// [tenant-42]
</span>
</span>// renaming 'tenant-42' in 'tenant-142'
</span>// This will NOT move the data, only the metadata is modified
</span>directorySubspace</span>, </span>err </span>= </span>directory</span>.</span>Move</span>(</span>db</span>, 
</span> []</span>string</span>{"</span>application</span>", "</span>my-app</span>", "</span>tenant</span>", "</span>tenant-42</span>"},  </span>// old path
</span> []</span>string</span>{"</span>application</span>", "</span>my-app</span>", "</span>tenant</span>", "</span>tenant-142</span>"}) </span>// new path
</span>if </span>err </span>!= </span>nil </span>{
</span> </span>log</span>.</span>Fatal</span>(</span>err</span>)
</span>}
</span>fmt</span>.</span>Printf</span>("</span>%+v</span>\n</span>", </span>directorySubspace</span>.</span>Bytes</span>())
</span>// still [21 18]
</span></code></pre>
The returned object is actually a DirectorySubspace</code>, which implements both Directory</code> and Subspace</code>, which means that you can use it to recreate many directories and subspaces at will 👌</p>

If you are wondering about how this integer is generated, I recommend going through this awesome blogpost on how high contention allocator works in FoundationDB.</a></p>
</blockquote>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Notes about ETCD
Pierre Zemb — Mon, 11 Jan 2021 00:24:27 +0100
</p>
Notes About</a> is a blogpost serie  you will find a lot of links, videos, quotes, podcasts to click on</strong> about a specific topic. Today we will discover ETCD.</p>
🔗</a>Overview of ETCD</h2>
As stated in the official documentation</a>:</p>

etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It gracefully handles leader elections during network partitions and can tolerate machine failure, even in the leader node.</p>
</blockquote>
🔗</a>History</h2>
ETCD was initially developed by CoreOS:</p>

CoreOS built etcd to solve the problem of shared configuration and service discovery.</p>
</blockquote>

July 23, 2013 - announcement</li>
December 27, 2013 - etcd 0.2.0 - new API, new modules and tons of improvements</li>
February 07, 2014 - etcd 0.3.0 - Improved Cluster Discovery, API Enhancements and Windows Support</li>
January 28, 2015 - etcd 2.0 - First Major Stable Release</li>
June 30, 2016 - etcd3 - A New Version of etcd from CoreOS</li>
June 09, 2017 - etcd 3.2 - etcd 3.2 now with massive watch scaling and easy locks</li>
February 01, 2018 - etcd 3.3 - Announcing etcd 3.3, with improvements to stability, performance, and more</li>
August 30, 2019 - etcd 3.4 - Better Storage Backend, concurrent Read, Improved Raft Voting Process, Raft Learner Member</li>
</ul>
🔗</a>Overall architecture</h2>

The etcd key-value store is a distributed system intended for use as a coordination primitive. Like Zookeeper and Consul, etcd stores a small volume of infrequently-updated state (by default, up to 8 GB) in a key-value map, and offers strict-serializable reads, writes and micro-transactions across the entire datastore, plus coordination primitives like locks, watches, and leader election. Many distributed systems, such as Kubernetes and OpenStack, use etcd to store cluster metadata, to coordinate consistent views over data, to choose leaders, and so on.</p>
</blockquote>
ETCD is:</p>

using the raft consensus algorithm</a>,</li>
a single group raft,</li>
using gRPC</a> for communication,</li>
using a self-made WAL implementation,</li>
storing key-values into bbolt,</li>
optimized for consistency over latency in normal situations and consistency over availability in the case of a partition (in terms of the PACELC theorem</a>).</li>
</ul>
🔗</a>Consensus? Raft?</h3>

Raft is a consensus algorithm for managing a replicated log.</li>
consensus involves multiple servers agreeing on values.</li>
two common consensus algorithm are Paxos and Raft</li>
</ul>

Paxos is quite difficult to understand, inspite of numerous attempts to make it more approachable. Furthermore, its architecture requires complex changes to support practical systems. As a result, both system builders and students struggle with Paxos.</p>
</blockquote>

A common alternative to Paxos/Raft is a non-consensus (aka peer-to-peer) replication protocol.</li>
</ul>

Raft separates the key elements of consensus, such as leader election, log replication, and safety</p>
</blockquote>
ETCD contains several raft optimizations:</p>

Read Index,</li>
Follower reads,</li>
Transfer leader,</li>
Learner role,</li>
Client-side load-balancing.</li>
</ul>
🔗</a>Exposed API</h3>
ETCD is exposing several APIs through different gRPC services:</p>

Put(key, value),</li>
Delete(key, Optional(keyRangeEnd)),</li>
Get(key, Optional(keyRangeEnd)),</li>
Watch(key, Optional(keyRangeEnd)),</li>
Transaction(if/then/else ops),</li>
Compact(revision),</li>
Lease:

Grant,</li>
Revoke,</li>
KeepAlive</li>
</ul>
</li>
</ul>
Key and values are bytes-oriented but ordered.</p>
🔗</a>Transactions</h3>
// From google paxosdb paper:
</span>// Our implementation hinges around a powerful primitive which we call MultiOp. All other database
</span>// operations except for iteration are implemented as a single call to MultiOp. A MultiOp is applied atomically
</span>// and consists of three components:
</span>// 1. A list of tests called guard. Each test in guard checks a single entry in the database. It may check
</span>// for the absence or presence of a value, or compare with a given value. Two different tests in the guard
</span>// may apply to the same or different entries in the database. All tests in the guard are applied and
</span>// MultiOp returns the results. If all tests are true, MultiOp executes t op (see item 2 below), otherwise
</span>// it executes f op (see item 3 below).
</span>// 2. A list of database operations called t op. Each operation in the list is either an insert, delete, or
</span>// lookup operation, and applies to a single database entry. Two different operations in the list may apply
</span>// to the same or different entries in the database. These operations are executed
</span>// if guard evaluates to
</span>// true.
</span>// 3. A list of database operations called f op. Like t op, but executed if guard evaluates to false.
</span>message </span>TxnRequest </span>{
</span>  </span>// compare is a list of predicates representing a conjunction of terms.
</span>  </span>// If the comparisons succeed, then the success requests will be processed in order,
</span>  </span>// and the response will contain their respective responses in order.
</span>  </span>// If the comparisons fail, then the failure requests will be processed in order,
</span>  </span>// and the response will contain their respective responses in order.
</span>  </span>repeated </span>Compare compare </span>= </span>1</span>;
</span>  </span>// success is a list of requests which will be applied when compare evaluates to true.
</span>  </span>repeated </span>RequestOp success </span>= </span>2</span>;
</span>  </span>// failure is a list of requests which will be applied when compare evaluates to false.
</span>  </span>repeated </span>RequestOp failure </span>= </span>3</span>;
</span>}
</span></code></pre>
🔗</a>Versioned data</h3>
Each Key/Value has a revision. When creating a new key, revision starts at 1, and then will be incremented each time the key is updated.</p>
In order to avoid having a growing keySpace, one can issue the Compact</code> gRPC service:</p>

Compacting the keyspace history drops all information about keys superseded prior to a given keyspace revision</p>
</blockquote>
🔗</a>Lease</h3>
// this message represent a Lease
</span>message </span>Lease </span>{
</span>  </span>// TTL is the advisory time-to-live in seconds. Expired lease will return -1.
</span>  int64 </span>TTL </span>= </span>1</span>;
</span>  </span>// ID is the requested ID for the lease. If ID is set to 0, the lessor chooses an ID.
</span>  int64 </span>ID </span>= </span>2</span>;
</span>
</span>  int64 </span>insert_timestamp </span>= </span>3</span>;
</span>}
</span></code></pre>
🔗</a>Watches</h3>
message </span>Watch </span>{
</span>  </span>// key is the key to register for watching.
</span>  bytes </span>key </span>= </span>1</span>;
</span>
</span>  </span>// range_end is the end of the range [key, range_end) to watch. If range_end is not given,
</span>  </span>// only the key argument is watched. If range_end is equal to '\0', all keys greater than
</span>  </span>// or equal to the key argument are watched.
</span>  </span>// If the range_end is one bit larger than the given key,
</span>  </span>// then all keys with the prefix (the given key) will be watched.
</span>  bytes </span>range_end </span>= </span>2</span>;
</span>
</span>  </span>// If watch_id is provided and non-zero, it will be assigned to this watcher.
</span>  </span>// Since creating a watcher in etcd is not a synchronous operation,
</span>  </span>// this can be used ensure that ordering is correct when creating multiple
</span>  </span>// watchers on the same stream. Creating a watcher with an ID already in
</span>  </span>// use on the stream will cause an error to be returned.
</span>  int64 </span>watch_id </span>= </span>7</span>;
</span>}
</span></code></pre>
🔗</a>Linearizable reads</h3>
Section 8 of the raft paper explains the issue:</p>

Read-only operations can be handled without writing anything into the log. However, with no additional measures, this would run the risk of returning stale data, since the leader responding to the request might have been superseded by a newer leader of which it is unaware. Linearizable reads must not return stale data, and Raft needs two extra precautions to guarantee this without using the log. First, a leader must have the latest information on which entries are committed. The Leader Completeness Property guarantees that a leader has all committed entries, but at the start of its term, it may not know which those are. To find out, it needs to commit an entry from its term. Raft handles this by having each leader commit a blank no-op entry into the log at the start of its term. Second,a leader must check whether it has been deposed before processing a read-only request (its information may be stale if a more recent leader has been elected). Raft handles this by having the leader exchange heartbeat messages with a majority of the cluster before responding to read-only requests.</p>
</blockquote>
ETCD implements ReadIndex</code> read(more info on Diving into ETCD’s linearizable reads</a>).</p>
🔗</a>How ETCD is using bbolt</h3>
bbolt</a> is the underlying kv used in etcd. A bucket called key</code> is used to store data, and the key is the revision</a>. Then, to find keys, a B-Tree is used</a>.</p>


Bolt allows only one read-write transaction at a time but allows as many read-only transactions as you want at a time.</li>
Each transaction has a consistent view of the data as it existed when the transaction started.</li>
Bolt uses a B+tree internally and only a single file. Both approaches have trade-offs.</li>
If you require a high random write throughput (>10,000 w/sec) or you need to use spinning disks then LevelDB could be a good choice. If your application is read-heavy or does a lot of range scans then Bolt could be a good choice.</li>
Try to avoid long running read transactions. Bolt uses copy-on-write so old pages cannot be reclaimed while an old transaction is using them.</li>
Bolt uses a memory-mapped file so the underlying operating system handles the caching of the data. Typically, the OS will cache as much of the file as it can in memory and will release memory as needed to other processes. This means that Bolt can show very high memory usage when working with large databases.</li>
Etcd implements multi-version-concurrency-control (MVCC) on top of Boltdb</li>
</ul>
</blockquote>
From an Github issue</a>:</p>

Note that the underlying bbolt</code> mmap its file in memory. For better performance, usually it is a good idea to ensure the physical memory available to etcd is larger than its data size.</p>
</blockquote>
🔗</a>ETCD in K8S</h2>
The interface can be found here</a>.</p>

Create use TTL and Txn</li>
Get use KV.Get</li>
Delete use Get and then for with a Txn</li>
GuaranteedUpdate uses Txn</li>
List uses Get</li>
Watch uses Watch with a channel</li>
</ul>
🔗</a>Jepsen</h2>
The Jepsen team tested etcd-3.4.3</a>, here's some quotes:</p>

In our tests, etcd 3.4.3 lived up to its claims for key-value operations: we observed nothing but strict-serializable consistency for reads, writes, and even multi-key transactions, during process pauses, crashes, clock skew, network partitions, and membership changes.</p>
</blockquote>

Watches appear correct, at least over single keys. So long as compaction does not destroy historical data while a watch isn’t running, watches appear to deliver every update to a key in order.</p>
</blockquote>

However, etcd locks (like all distributed locks) do not provide mutual exclusion. Multiple processes can hold an etcd lock concurrently, even in healthy clusters with perfectly synchronized clocks.</p>
</blockquote>

If you use etcd locks, consider whether those locks are used to ensure safety, or simply to improve performance by probabilistically limiting concurrency. It’s fine to use etcd locks for performance, but using them for safety might be risky.</p>
</blockquote>
🔗</a>Operation notes</h2>
🔗</a>Deployements tips</h3>
From the official documentation</a>:</p>

Since etcd writes data to disk, SSD is highly recommended.
To prevent performance degradation or unintentionally overloading the key-value store, etcd enforces a configurable storage size quota set to 2GB by default.
To avoid swapping or running out of memory, the machine should have at least as much RAM to cover the quota.
8GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.</p>
</blockquote>
🔗</a>Defrag</h3>

After compacting the keyspace, the backend database may exhibit internal fragmentation.
Defragmentation is issued on a per-member so that cluster-wide latency spikes may be avoided.</p>
</blockquote>
Defrag is basically dumping the bbolt tree on disk and reopening it</a>.</p>
🔗</a>Snapshot</h3>
An ETCD snapshot is related to Raft's snapshot:</p>

Snapshotting is the simplest approach to compaction. In snapshotting, the entire current system state is written to a snapshot on stable storage, then the entire log up to that point is discarded</p>
</blockquote>
Snapshot can be saved using etcdctl</code>:</p>
etcdctl</span> snapshot save backup.db
</span></code></pre>
🔗</a>Lease</h3>
Be careful on Leader's change and lease, this can create some issues</a>:</p>

The new leader extends timeouts automatically for all leases. This mechanism ensures no lease expires due to server side unavailability.</p>
</blockquote>
🔗</a>War stories</h3>

An analysis of the Cloudflare API availability incident on 2020-11-02</a></li>
How a production outage in Grafana Cloud's Hosted Prometheus service was caused by a bad etcd client setup</a></li>
Random performance issue on etcd 3.4</a></li>
Impact of etcd deployment on Kubernetes, Istio, and application performance</a></li>
</ul>


10 years of programming and counting 🚀
Pierre Zemb — Wed, 30 Sep 2020 00:24:27 +0100
I’ve just realized that I’ve spent the last decade programming 🤯 While 2020 feels like a strange year, I thought it would be nice to write down a retrospective of the last 10 years 🗓</p>
🔗</a>Learning to program 👨🏻‍💻</h2>
I wrote my first Hello, world</em> program somewhere around September 2010, when I started my engineering school to do some electronics, but that C language got me. I spent 6 months struggling to understand pointers and memory. I remember spending nights trying to find a memory leak with valgrind. Of course there were multiples mistakes, but it felt good to dig that far.</p>
I also discovered Linux around that time, and spent many nights playing with Linux commands. I started my journey to Linux with Centos and then Ubuntu 11.04. I think this started the loop I’m (still!) stuck in:
for {tryNewDistro()}</code>
I’m pretty sure that if I wanted to go away from distributed systems, I would try to land a job around operating systems. So many things to learn 🤩</p>
After learning C, we started to learn web-based technologies like HTML/CSS/JS/PHP. I remember struggling to generate a calendar with PHP 🐘 I learned about APIs the week after the project 😅 I remember digging into cookies, and network calls from popular websites to see how they were using it.</p>
🔗</a>Java and Hadoop 🐘</h2>
I had the chance to land a part-time internship during the third year (out of five) of my engineering school. I joined the Systems team @ Arkea, a french bank.
I remember spending a lot of time with my coworkers, learning things from them, from Hadoop to mainframes and Linux. It was my first time grasping the work around “system programming”.</p>
My first task was around writing an installer for a java app on windows, but my tutor tried to push me further. He saw my interest around some specific layers of their perimeter, such as Hadoop and Kafka. He gave to me a chance to work directly on those. A small API that was could load old monitoring data stored in HDFS and expose them back into the “real-time” visualization tool. I also used Kafka and even deployed a small HBase cluster for testing.</p>
I can't thank my tutor enough for giving me this chance, and for allowing me to discover what will become my focus: distributed systems.</p>
🔗</a>Let’s meet other people 👋</h2>
Around the same time, I discovered tech meetups and conferences. At that time, Google I/O was a major event with people jumping from a plane and streaming it through Google Glass. I found out there was a group of people watching the live together. And this is how I discovered my local GDG/JUG 🥳 I learned so many things by watching local talks, even if it was difficult to grasp everything at first. I remember taking 📝 about what I didn’t understand, to learn about it later.</p>
I also met amazing persons, that are now friends and/or mentors. I remember feeling humble to be able to learn from them.</p>
I also discovered more global tech conferences. I asked as a birthday 🎁 to go to Devoxx France and DotScale, in 2014. It was awesome 😎</p>
By dint of watching talks, I wanted to give some. I started small, giving talks at my engineering school, then moved to the JUG itself. I learned a lot</strong> by making a lot of mistakes, but I’m pretty happy how things turned out, as I’m now speaking at tech conferences as part of my current work.</p>
</p>
I also started to be involved in events and organizations such as:</p>

The JUG/GDG</li>
A coworking place</li>
Startup Weekend</li>
Devoxx4kids</li>
DevFest du bout du monde</li>
</ul>
🔗</a>Learning big data 💾</h2>
After my graduation and a(nother) part-time internship at OVH, I started working on something called Metrics Data Platform. It is the platform massively used internally to store, query and alert on timeseries data. We avoid the Borgmon approach (deploying Prometheus’s like database for every team), instead we created a unique platform to ingest all OVHcloud’s datapoints using a big-data approach. Here’s the key point of Metrics:</p>

multi-tenant</strong>: as we said before, a single metrics cluster is handling all telemetry, from servers to applications and smart data centers from OVHcloud.</li>
scalable</strong>: today we are receiving around 1.8 million datapoints per second/s 🙈 for about 450 million timeseries 🙉. During European daytime, we are reading around 4.5 millions datapoints per seconds thank to Grafana’s auto-refresh mode 🙊</li>
multi-protocol support</strong>: we didn't want to reflect our infrastructure choice to our users, so we wrote some proxies that can translate known protocols to our query language, so users can query and push data using OpenTSDB, Prometheus, InfluxDB and so on.</li>
based on open source</strong> we are using Warp10 as the core of our infrastructure with Kafka and HBase. Alerting was built with Apache flink. We open sourced many software, from agent to our proxies. We also gave many talks about what we learnt.</li>
</ul>
I had the chance to built Metrics from the ground. I started working on the management layer and proxies. Then I wanted to learn operations, so I learned it by deploying Hadoop clusters 🤯 it took me a while to be able to start doing on-calls. I cannot count how many nights I was up, trying to fix some buggy softwares, or yelling at HBase for an inconsistent hbck</code>, or trying to find a way to handle a side effect of a loosing multiple racks.</p>
Our work was highly technical, and I loved it:</p>

We optimized a lot of things, from HBase to our Go’s based proxies. optimize HBase's data balancer</code> or fix issues with Go’s gc</code>  was almost a normal task to do</li>
We saw Metrics’s growth, from hundred to millions of datapoints 😎 we saw systems breaking at scale, causing us to rewrite software or change architecture. Production became the final test.</li>
Every software we developed had a keep it simple, yet scalable</code> policy, and doing on-calls was a good way to ensure software quality. We all learned it the hard way I guess 🤣</li>
We were only 4 to 6 to handle ~800 servers, 3 Hadoop clusters, and thousands of lines of Java/Go/Rust/Ansible codes.</li>
</ul>
As always, things were not always magical, and i struggled more time than I can count. I learned that personal struggle is more difficult than technical, as you can always drill-down your tech problems by reading the code. The team was amazing 🚀, and we were helping each other a lot 🤝</p>
🔗</a>Searching for planets 🔭 🪐</h2>
When I started working on Metrics, we did a lot of internal on boarding. At his core, metrics is usine Warp10, which is coming with his own language to analyze timeseries. This provides heavy query-capabilities, but as it is stack-based, getting started was difficult. I needed a project to dive into timeseries analysis.</p>
I love astronomy 🔭, but there’s too much ☁️ (not the servers) in my city. I decided to look for astronomical timeseries. Turns out there is a lot, but one use case triggered my interest: exoplanet’s search. Almost everything from NASA is Opendata, so we decided to create HelloExoWorld</a>.</p>
We imported the 25TB dataset into a Warp10 instance</strong> and start writing some WarpScript to search for transits. We wrote a hands-on about it</a>. We also did several labs in french conferences like Devoxx and many others.</p>
🔗</a>IO timeout 🚧</h2>
Around 2018, OVHcloud started Managed Kubernetes, a free K8S control-plane. With this product we saw more developers coming to OVHcloud. We started thinking about how we could help them. Running stateful systems is hard</strong>, so maybe we could offer them some databases or queues in a As-a-Service fashion. We started to design such products from our Metrics experience. We started the IO Vision to offer popular Storage APIs in front of a scalable storage</code>. Does it sound familiar? 😇 I had a lot of fun working on that vision as a Technical Leader.</p>
We started with queuing with ioStream. We wanted something that was:</p>

Multi-tenant</li>
Multi-protocol</li>
Geo-replicated natively</li>
Less operation burden at scale than Kafka</li>
</ul>
We built ioStream around Apache Pulsar, and opened the beta around September 2019. As the same time we were working on Kafka’s support as a proxy in Rust. Writing such a software capable of translating Kafka’s TCP frames to Pulsar with a state-machine was a fun and challenging work</strong>. Rust is really a nice language to write such software.</p>
Then we worked with Apache Pulsar’s PMC to introduce a Kafka protocol handler on Pulsar brokers. I had the chance to work closely to two PMCs, it was an amazing experience for me 🚀 You can read about our collaboration here</a>.</p>
Unfortunately as stated by the official communication, the project has been shut down:</p>
However, the limited success of the beta service and other strategic focuses,
</span>have resulted in us taking the very difficult decision to close it.
</span></code></pre>
I learned a lot of things, both technically and on the product-side, especially considering the fact that it was shutdown.</p>
🔗</a>Today</h2>
After ioStream’s shutdown, most of the team moved to create a new LBaaS. I helped them wrote an operator to schedule HAProxy’s containers on a Kubernetes cluster. It was a nice introduction to operators.</p>
Then I decided to join the Managed Kubernetes ☸️ team. This is my current team now, where I’m having a lot of fun working around ETCD.</p>
I really hope the next 10 years will be as fun as the last 10 years 😇</p>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Announcing Record-Store, a new (experimental) place for your data
Pierre Zemb — Wed, 23 Sep 2020 10:24:27 +0100
TL;DR: I'm really happy to announce my latest open-source project called Record-Store 🚀 Please check it out on https://pierrez.github.io/record-store</a>.</p>
🔗</a>What?</h2>
Record-Store</code> is a layer</a> running on top of FoundationDB</a>. It provides abstractions to create, load and deletes customer-defined data called records</code>, which are hold into a RecordSpace</code>. We would like to have this kind of flow for developers:</p>

Opening RecordSpace, for example prod/users</code></li>
Create a protobuf definition which will be used as schema</li>
Upsert schema</li>
Push records</li>
Query records</li>
delete records</li>
</ol>
You need another KeySpace</code> to store another type of data, or maybe a KeySpace</code> dedicated to production env? Juste create it and you are good to go!</p>
🔗</a>Features</h2>
It is currently an experiment, but it already has some strong features:</p>


Multi-tenant</strong> A tenant</code> can create as many RecordSpace</code> as we want, and we can have many tenants</code>.</p>
</li>

Standard API</strong> We are exposing the record-store with standard technologies:</p>

gRPC</a></li>
very experimental</em> GraphQL</a></li>
</ul>
</li>

Scalable</strong> We are based on the same tech behind CloudKit</a> called the Record Layer</a>,</p>
</li>

Transactional</strong> We are running on top of FoundationDB</a>. FoundationDB gives you the power of ACID transactions in a distributed database.</p>
</li>

Encrypted</strong> Data are encrypted by default.</p>
</li>

Multi-model</strong> For each RecordSpace</code>, you can define a schema</code>, which is in-fact only a Protobuf</code> definition. You need to store some users</code>, or a more complicated structure? If you can represent it as Protobuf</a>, you are good to go!</p>
</li>

Index-defined queries</strong> Your queries's capabilities are defined by the indexes you put on your schema.</p>
</li>

Secured</strong> We are using Biscuit</a>, a mix of JWT</code> and Macaroons</code> to ensure auth{entication, orization}.</p>
</li>
</ul>
🔗</a>Why?</h2>
Lately, I have been playing a lot with my ETCD-Layer</a> that is using the Record-Layer</a>. Thanks to it, I was able to bootstrap my ETCD-layer very quickly, but I was not using a tenth of the capacities of this library. So I decided to go deeper. What would a gRPC abstraction of the Record-Layer look like?</strong></p>
The name of this project itself is a tribute to the Record Layer as we are exposing the layer within a gRPC interface.</p>
🔗</a>Try it out</h2>
Record-Store is open sourced under Apache License V2 in https://github.com/PierreZ/record-store</a> and the documentation can be found https://pierrez.github.io/record-store</a>.</p>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Diving into ETCD's linearizable reads
Pierre Zemb — Fri, 18 Sep 2020 05:24:27 +0100
</p>
Diving Into</a> is a blogpost serie where we are digging a specific part of the project's basecode. In this episode, we will digg into the implementation behind ETCD's Linearizable reads.</p>

🔗</a>What is ETCD?</h2>
From the official website</a>:</p>

etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It gracefully handles leader elections during network partitions and can tolerate machine failure, even in the leader node.</p>
</blockquote>
ETCD is well-known to be Kubernetes's datastore, and a CNCF incubating project.</p>
🔗</a>Linea-what?</h2>
Let's quote Kyle Kingsbury, a.k.a "Aphyr"</a>, for this one:</p>

Linearizability is one of the strongest single-object consistency models, and implies that every operation appears to take place atomically, in some order, consistent with the real-time ordering of those operations: e.g., if operation A completes before operation B begins, then B should logically take effect after A.</p>
</blockquote>
🔗</a>Why?</h2>
ETCD is using Raft</a>, a consensus algorithm at his core. As always, the devil is hidden in the details, or when things are going wrong. Here's an example:</p>

node1</code> is leader</code> and heartbeating properly to node2</code> and node3</code>,</li>
network partition is happening, and node1</code> is isolated from the others.</li>
</ol>
At this moment, all the actions are depending on timeouts and settings. In a (close) future, all nodes will go into election mode</strong> and node 2 and 3 will be able to create a quorum. This can lead to this situation:</p>

node1</code> thinks he is a leader as heartbeat timeouts and retry are not yet reached, so he can serve reads 😱</li>
node2</code> and node3</code> have elected a new leader and are working again, accepting writes.</li>
</ul>
This situation is violating Linearizable reads, as reads going through node1</code> will not see the last updates from the current leader.</p>
How can we solve this? One way is to use ReadIndex</code>!</p>
🔗</a>ReadIndex</h2>
The basic idea behind this is to confirm that the leader is true leader or not</strong> by sending a message to the followers. If a majority of responses are healthy, then the leader can safely serve the reads. Let's dive into the implementation!</p>
All codes are from the current latest release v3.4.13</a>.</p>
Let's take a Range operation</a>:</p>
 </span>if </span>!</span>r</span>.</span>Serializable </span>{
</span>  </span>err </span>= </span>s</span>.</span>linearizableReadNotify</span>(</span>ctx</span>)
</span>  </span>trace</span>.</span>Step</span>("</span>agreement among raft nodes before linearized reading</span>")
</span>  </span>if </span>err </span>!= </span>nil </span>{
</span>   </span>return </span>nil</span>, </span>err
</span>  }
</span> }
</span></code></pre>

</span>func </span>(</span>s </span>*</span>EtcdServer</span>) </span>linearizableReadNotify</span>(</span>ctx context</span>.</span>Context</span>) </span>error </span>{
</span> </span>s</span>.</span>readMu</span>.</span>RLock</span>()
</span> </span>nc </span>:= </span>s</span>.</span>readNotifier
</span> </span>s</span>.</span>readMu</span>.</span>RUnlock</span>()
</span>
</span> </span>// signal linearizable loop for current notify if it hasn't been already
</span> </span>select </span>{
</span> </span>case </span>s</span>.</span>readwaitc </span><- </span>struct</span>{}{}:
</span> </span>default</span>:
</span> }
</span>
</span> </span>// wait for read state notification
</span> </span>select </span>{
</span> </span>case </span><-</span>nc</span>.</span>c</span>:
</span>  </span>return </span>nc</span>.</span>err
</span> </span>case </span><-</span>ctx</span>.</span>Done</span>():
</span>  </span>return </span>ctx</span>.</span>Err</span>()
</span> </span>case </span><-</span>s</span>.</span>done</span>:
</span>  </span>return </span>ErrStopped
</span> }
</span>}
</span></code></pre>
So in linearizableReadNotify</a>, we are waiting for a signal. readwaitc</code> is used in another goroutine called linearizableReadLoop</a>. This goroutines will call this:</p>
func </span>(</span>n </span>*</span>node</span>) </span>ReadIndex</span>(</span>ctx context</span>.</span>Context</span>, </span>rctx </span>[]</span>byte</span>) </span>error </span>{
</span> </span>return </span>n</span>.</span>step</span>(</span>ctx</span>, </span>pb</span>.</span>Message</span>{</span>Type</span>: </span>pb</span>.</span>MsgReadIndex</span>, </span>Entries</span>: []</span>pb</span>.</span>Entry</span>{{</span>Data</span>: </span>rctx</span>}}})
</span>}
</span>
</span></code></pre>
that will create a MsgReadIndex</code> message that will be handled in stepLeader</a>, who will send the message to the followers, like this:</p>
 </span>case </span>pb</span>.</span>MsgReadIndex</span>:
</span>  </span>// If more than the local vote is needed, go through a full broadcast,
</span>  </span>// otherwise optimize.
</span>  </span>if </span>!</span>r</span>.</span>prs</span>.</span>IsSingleton</span>() {
</span>      </span>// PZ: omitting some code here
</span>   </span>switch </span>r</span>.</span>readOnly</span>.</span>option </span>{
</span>   </span>case </span>ReadOnlySafe</span>:
</span>    </span>r</span>.</span>readOnly</span>.</span>addRequest</span>(</span>r</span>.</span>raftLog</span>.</span>committed</span>, </span>m</span>)
</span>    </span>// The local node automatically acks the request.
</span>    </span>r</span>.</span>readOnly</span>.</span>recvAck</span>(</span>r</span>.</span>id</span>, </span>m</span>.</span>Entries</span>[</span>0</span>].</span>Data</span>)
</span>    </span>r</span>.</span>bcastHeartbeatWithCtx</span>(</span>m</span>.</span>Entries</span>[</span>0</span>].</span>Data</span>)
</span>   </span>case </span>ReadOnlyLeaseBased</span>:
</span>    </span>ri </span>:= </span>r</span>.</span>raftLog</span>.</span>committed
</span>    </span>if </span>m</span>.</span>From </span>== </span>None </span>|| </span>m</span>.</span>From </span>== </span>r</span>.</span>id </span>{ </span>// from local member
</span>     </span>r</span>.</span>readStates </span>= </span>append</span>(</span>r</span>.</span>readStates</span>, </span>ReadState</span>{</span>Index</span>: </span>ri</span>, </span>RequestCtx</span>: </span>m</span>.</span>Entries</span>[</span>0</span>].</span>Data</span>})
</span>    } </span>else </span>{
</span>     </span>r</span>.</span>send</span>(</span>pb</span>.</span>Message</span>{</span>To</span>: </span>m</span>.</span>From</span>, </span>Type</span>: </span>pb</span>.</span>MsgReadIndexResp</span>, </span>Index</span>: </span>ri</span>, </span>Entries</span>: </span>m</span>.</span>Entries</span>})
</span>    }
</span>   }
</span></code></pre>
So, the leader</code> is sending a heartbeat in ReadOnlySafe</code> mode. Turns out there is two modes:</p>
const </span>(
</span> </span>// ReadOnlySafe guarantees the linearizability of the read only request by
</span> </span>// communicating with the quorum. It is the default and suggested option.
</span> </span>ReadOnlySafe </span>ReadOnlyOption </span>= </span>iota
</span> </span>// ReadOnlyLeaseBased ensures linearizability of the read only request by
</span> </span>// relying on the leader lease. It can be affected by clock drift.
</span> </span>// If the clock drift is unbounded, leader might keep the lease longer than it
</span> </span>// should (clock can move backward/pause without any bound). ReadIndex is not safe
</span> </span>// in that case.
</span> </span>ReadOnlyLeaseBased
</span>)
</span></code></pre>
Responses from the followers will be handled here:</p>
 </span>case </span>pb</span>.</span>MsgHeartbeatResp</span>:
</span>  </span>// PZ: omitting some code here
</span>  </span>rss </span>:= </span>r</span>.</span>readOnly</span>.</span>advance</span>(</span>m</span>)
</span>  </span>for </span>_</span>, </span>rs </span>:= </span>range </span>rss </span>{
</span>   </span>req </span>:= </span>rs</span>.</span>req
</span>   </span>if </span>req</span>.</span>From </span>== </span>None </span>|| </span>req</span>.</span>From </span>== </span>r</span>.</span>id </span>{ </span>// from local member
</span>    </span>r</span>.</span>readStates </span>= </span>append</span>(</span>r</span>.</span>readStates</span>, </span>ReadState</span>{</span>Index</span>: </span>rs</span>.</span>index</span>, </span>RequestCtx</span>: </span>req</span>.</span>Entries</span>[</span>0</span>].</span>Data</span>})
</span>   } </span>else </span>{
</span>    </span>r</span>.</span>send</span>(</span>pb</span>.</span>Message</span>{</span>To</span>: </span>req</span>.</span>From</span>, </span>Type</span>: </span>pb</span>.</span>MsgReadIndexResp</span>, </span>Index</span>: </span>rs</span>.</span>index</span>, </span>Entries</span>: </span>req</span>.</span>Entries</span>})
</span>   }
</span>  }
</span></code></pre>
We are storing things into a ReadState</code>:</p>
// ReadState provides state for read only query.
</span>// It's caller's responsibility to call ReadIndex first before getting
</span>// this state from ready, it's also caller's duty to differentiate if this
</span>// state is what it requests through RequestCtx, eg. given a unique id as
</span>// RequestCtx
</span>type </span>ReadState </span>struct </span>{
</span> </span>Index      </span>uint64
</span> </span>RequestCtx </span>[]</span>byte
</span>}
</span></code></pre>
Now that the state has been updated, we need to unblock our linearizableReadLoop</a>:</p>
  </span>for </span>!</span>timeout </span>&& !</span>done </span>{
</span>   </span>select </span>{
</span>   </span>case </span>rs </span>= <-</span>s</span>.</span>r</span>.</span>readStateC</span>:
</span></code></pre>
Cool, another channel! Turns out, readStateC</code> is updated in one of the main goroutine</a>:</p>
// start prepares and starts raftNode in a new goroutine. It is no longer safe
</span>// to modify the fields after it has been started.
</span>func </span>(</span>r </span>*</span>raftNode</span>) </span>start</span>(</span>rh </span>*</span>raftReadyHandler</span>) {
</span> </span>internalTimeout </span>:= </span>time</span>.</span>Second
</span>
</span> </span>go func</span>() {
</span>  </span>defer </span>r</span>.</span>onStop</span>()
</span>  </span>islead </span>:= </span>false
</span>
</span>  </span>for </span>{
</span>   </span>select </span>{
</span>   </span>case </span><-</span>r</span>.</span>ticker</span>.</span>C</span>:
</span>    </span>r</span>.</span>tick</span>()
</span>   </span>case </span>rd </span>:= <-</span>r</span>.</span>Ready</span>():
</span>    </span>// PZ: omitting some code here
</span>    </span>if </span>len</span>(</span>rd</span>.</span>ReadStates</span>) != </span>0 </span>{
</span>     </span>select </span>{
</span>     </span>case </span>r</span>.</span>readStateC </span><- </span>rd</span>.</span>ReadStates</span>[</span>len</span>(</span>rd</span>.</span>ReadStates</span>)-</span>1</span>]:
</span>    }
</span></code></pre>
Perfect, now readStateC</code> is notified, and we can continue on linearizableReadLoop</a>:</p>
  </span>if </span>ai </span>:= </span>s</span>.</span>getAppliedIndex</span>(); </span>ai </span>< </span>rs</span>.</span>Index </span>{
</span>   </span>select </span>{
</span>   </span>case </span><-</span>s</span>.</span>applyWait</span>.</span>Wait</span>(</span>rs</span>.</span>Index</span>):
</span>   </span>case </span><-</span>s</span>.</span>stopping</span>:
</span>    </span>return
</span>   }
</span>  }
</span>  </span>// unblock all l-reads requested at indices before rs.Index
</span>  </span>nr</span>.</span>notify</span>(</span>nil</span>)
</span></code></pre>
The first part is a safety measure to makes sure the applied index is lower that the index stored in ReadState</code>. And then finally we are unlocking all pending reads 🤩</p>
🔗</a>One more thing: Follower read</h2>
We went through stepLeader</code> a lot, be there is something interesting in stepFollower</code></a>:</p>
 </span>case </span>pb</span>.</span>MsgReadIndex</span>:
</span>  </span>if </span>r</span>.</span>lead </span>== </span>None </span>{
</span>   </span>r</span>.</span>logger</span>.</span>Infof</span>("</span>%x</span> no leader at term </span>%d</span>; dropping index reading msg</span>", </span>r</span>.</span>id</span>, </span>r</span>.</span>Term</span>)
</span>   </span>return </span>nil
</span>  }
</span>  </span>m</span>.</span>To </span>= </span>r</span>.</span>lead
</span>  </span>r</span>.</span>send</span>(</span>m</span>)
</span></code></pre>
This means that a follower can send a MsgReadIndex</code> message to perform the same kind of checks than a leader. This small features is in fact enabling follower-reads</strong> on ETCD 🤩 That is why you can see Range</code> requests from a follower</code>.</p>
🔗</a>operational tips</h2>

If you are running etcd <= 3.4, make sure logger=zap</strong> is set. Like this, you will be able to see some tracing logs, and I trully hope you will not witness this one:</li>
</ul>
{
</span>  "</span>level</span>": "</span>info</span>",
</span>  "</span>ts</span>": "</span>2020-08-12T08:24:56.181Z</span>",
</span>  "</span>caller</span>": "</span>traceutil/trace.go:145</span>",
</span>  "</span>msg</span>": "</span>trace[677217921] range</span>",
</span>  "</span>detail</span>": "</span>{range_begin:/...redacted...; range_end:; response_count:1; response_revision:2725080604; }</span>",
</span>  "</span>duration</span>": "</span>1.553047811s</span>",
</span>  "</span>start</span>": "</span>2020-08-12T08:24:54.628Z</span>",
</span>  "</span>end</span>": "</span>2020-08-12T08:24:56.181Z</span>",
</span>  "</span>steps</span>": [
</span>    "</span>trace[677217921] 'agreement among raft nodes before linearized reading'  (duration: 1.534322015s)</span>" 
</span>  ]
</span>}
</span></code></pre>

there is a random performance issue on etcd 3.4</a></li>
there is some metrics than you can watch for ReadIndex issues:

etcd_server_read_indexes_failed_total</code></li>
etcd_server_slow_read_indexes_total</code></li>
</ul>
</li>
</ul>

Thank you</strong> for reading my post! feel free to react to this article, I'm also available on Twitter</a> if needed.</p>


Notes about Raft's paper
Pierre Zemb — Thu, 30 Jul 2020 07:24:27 +0100
</p>
Notes About</a> is a blogpost serie  you will find a lot of links, videos, quotes, podcasts to click on</strong> about a specific topic. Today we will discover Raft's paper called 'In Search of an Understandable Consensus Algorithm'.</p>

As I'm digging into ETCD, I needed to refresh my memory about Raft. I started by reading the paper located here</a> and I'm also playing with the amazing Raft labs made by PingCAP</a>.</p>

These labs are derived from the lab2:raft</a> and lab3:kvraft</a> from the famous MIT 6.824</a> course but rewritten in Rust.</p>
</blockquote>
🔗</a>Abstract</h2>

Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to (multi-)Paxos, andit is as efficient as Paxos, but its structure is differentfrom Paxos; this makes Raft more understandable thanPaxos and also provides a better foundation for build-ing practical systems.</p>
</blockquote>

Raft separates the key elements of consensus, such asleader election, log replication, and safety, and it enforcesa stronger degree of coherency to reduce the number ofstates that must be considered.</p>
</blockquote>
🔗</a>Introduction</h2>

Consensus algorithms allow a collection of machines to work as a coherent group that can survive the failures of some of its members.</p>
</blockquote>

Paxos has dominated the discussion of consensus algorithms over the last decade.</p>
</blockquote>

Unfortunately, Paxos is quite difficult to understand, inspite of numerous attempts to make it more approachable.Furthermore, its architecture requires complex changes to support practical systems. As a result, both systembuilders and students struggle with Paxos.</p>
</blockquote>

Our approach was unusual in that our primary goal was understandability</strong>.</p>
</blockquote>

We believe that Raft is superior to Paxos and other consensus algorithms, both for educational purposes and as a foundation for implementation.</p>
</blockquote>
🔗</a>Replicated state machines</h2>
The main idea is to compute identical copies of the same state (i.e x:3, y:9</code>) in case of machines's failure. Most of the time, an ordered wal</code> (write-ahead log) is used in the implementation, to hold the mutation (x:4</code>). Keeping the replicated log consistent is the job of the consensus algorithm, here Raft.</p>
Raft creates a true split between:</p>

the consensus module,</li>
the wal,</li>
the state machine.</li>
</ul>

🔗</a>What’s wrong with Paxos?</h2>
The paper is listing the drawbacks of Paxos:</p>

difficult to understand, and I can't blame them</a></li>
many details are missing from the paper to implement Multi-Paxos</code> as the paper is mainly describing single-decree Paxos</code></li>
</ul>

It is simpler and more efficient to design a system around a log, where new entries are appended sequentially in a constrained order.</p>
</blockquote>

As a result, practical systems bear little resemblance to Paxos. Each implementation begins with Paxos, discovers the difficulties in implementing it, and then develops a significantly different architecture. This is time-consuming and error-prone, and the difficulties of understanding Paxos exacerbate the problem. The following com-ment from the Chubby</a> implementers is typical:</p>
</blockquote>


There are significant gaps between the description of the Paxos algorithm and the needs of a real-world system
the final system will be based on an un-proven protocol [4].</p>
</blockquote>
</blockquote>
🔗</a>Designing for understandability</h2>
Beside all the others goals of Raft:</p>

a complete and practical foundation for system building,</li>
must be safe under all conditions and available under typical operating conditions,</li>
must be efficient for common operations,</li>
</ul>
understandability</strong> was the most difficult challenge:</p>

It must be possible for a large audience to understand the algorithm comfortably. In addition, it must be possible to develop intuitions about the algorithm, so that system builders can make the extensions that are inevitable in real-world implementations.</p>
</blockquote>

we divided problems into separate pieces that could be solved, explained, and understood relatively independently. For example, in Raft we separated leader election, log replication, safety, and membership changes.</p>
</blockquote>

Our second approach was to simplify the state spaceby reducing the number of states to consider, making thesystem more coherent and eliminating nondeterminism where possible.</p>
</blockquote>
🔗</a>The Raft consensus algorithm</h2>
Raft is heavily relying on the leader</code> pattern:</p>

Raft implements consensus by first electing a distinguished leader, then giving the leader complete responsibility for managing the replicated log.</p>
</blockquote>

The leader accepts log entries from clients, replicates them on other servers, and tells servers when it is safe to apply log entries to their state machines.</p>
</blockquote>
Thanks to this pattern, Raft is splitting the consensus problem into 3:</p>

Leader election</li>
Log replication</li>
Safety</li>
</ul>
🔗</a>Raft basics</h3>
Each server can be in one of the three states:</p>

Leader</strong> handle all requests,</li>
Follower</strong> passive member, they issue no requests on their own but simply respond to requests from leaders and candidates,</li>
Candidate</strong> is used to elect a new leader.</li>
</ul>
Leader is elected through election</code>: Each term (interval of time of arbitrary length packed with an number) begins with an election, in which one or more candidates attempt to become leader. If a candidate wins the election, then it serves as leader for the rest of the term. In the case of a split vote, the term will end with no leader; a new term (with a new election) will begin.</p>

Terms act as a logical clock [14] in Raft.</p>
</blockquote>

Each server stores a current term number, which increases monotonically over time. Current terms are exchanged whenever servers communicate; if one server’s current term is smaller than the other’s, then it updates its current term to the larger value. If a candidate or leader discovers that its term is out of date, it immediately reverts to fol-lower state. If a server receives a request with a stale term number, it rejects the request.</p>
</blockquote>
RPC</code> is used for communications:</p>

RequestVote RPCs</strong> are initiated by candidates during elections,</li>
Append-Entries RPCs</strong> are initiated by leaders to replicate log en-tries and to provide a form of heartbeat.</li>
</ul>
🔗</a>Leader election</h3>
A good vizualization is available here</a>.</p>
The key-point of the election are the fact that:</p>

nodes vote for themselves,</li>
the term number is used to recover from failure,</li>
election timeouts are randomized.</li>
</ul>

To begin an election, a follower increments its current term and transitions to candidate state. It then votes for itself and issues RequestVote RPCs in parallel to each of the other servers in the cluster. A candidate continues in this state until one of three things happens:</p>

(a) it wins the election,</li>
(b) another server establishes itself as leader,</li>
(c) a period of time goes by with no winner.</li>
</ul>
</blockquote>

Raft uses randomized election timeouts to ensure that split votes are rare and that they are resolved quickly. To prevent split votes in the first place, election timeouts are chosen randomly from a fixed interval (e.g., 150–300ms).</p>
</blockquote>
🔗</a>Log replication</h3>
A good vizualization is available here</a>.</p>

Once a leader has been elected, it begins servicing client requests. Each client request contains a command to be executed by the replicated state machines. The leader appends the command to its log as a new entry, then issues AppendEntries RPCs in parallel to each of the other servers to replicate the entry. When the entry has been safely replicated (as described below), the leader applies the entry to its state machine and returns the result of that execution to the client.</p>
</blockquote>

The term numbers in log entries are used to detect inconsistencies between logs</p>
</blockquote>

The leader decides when it is safe to apply a log entry to the state machines; such an entry is called committed. Raft guarantees that committed entries are durable and will eventually be executed by all of the available state machines.</p>
</blockquote>
Raft is implementing a lot of safety inside the log:</p>

When sending an AppendEntries RPC, the leader includes the index and term of the entry in its log that immediately precedes the new entries. If the follower does not find an entry in its log with the same index and term, then it refuses the new entries</p>
</blockquote>
This is really interesting to be leader-failure proof. And for follower's failure:</p>

In Raft, the leader handles inconsistencies by forcing the followers’ logs to duplicate its own.</p>
</blockquote>

To bring a follower’s log into consistency with its own,the leader must find the latest log entry where the two logs agree, delete any entries in the follower’s log after that point, and send the follower all of the leader’s entries after that point.</p>
</blockquote>
🔗</a>Safety</h2>
🔗</a>Leader election</h3>
As Raft guarantees that all the committed entries are available on all followers, log entries only flow in one di-rection, from leaders to followers, and leaders never over-write existing entries in their logs.</p>

Raft uses the voting process to prevent a candidate from winning an election unless its log contains all committed entries. A candidate must contact a majority of the cluster in order to be elected, which means that every committed entry must be present in at least one of those servers.</p>
</blockquote>

Raft determines which of two logs is more up-to-date by comparing the index and term of the last entries in the logs. If the logs have last entries with different terms, then the log with the later term is more up-to-date. If the log send with the same term, then whichever log is longer is more up-to-date.</p>
</blockquote>
🔗</a>Committing entries from previous terms</h3>

Raft never commits log entries from previous terms by counting replicas. Only log entries from the leader’s current term are committed by counting replicas.</p>
</blockquote>
This behavior avoids future leaders to attempt to finish replicating an entry where the leader crashes before committing an entry.</p>
🔗</a>Follower and candidate crashes</h3>

If a follower or candidate crashes, then future RequestVote and AppendEntries RPCs sent to it will fail. Raft handles these failures by retrying indefinitely.</p>
</blockquote>
🔗</a>Cluster membership changes</h2>
This section presents how to do cluster configuration(the set of servers participating in the consensus algorithm). Raft implements a two-phase approach:</p>

In Raft the cluster first switches to a transitional configuration we call joint consensus; once the joint consensus has been committed,the system then transitions to the new configuration. The joint consensus combines both the old and new configurations:</p>

Log entries are replicated to all servers in both con-figurations,</li>
Any server from either configuration may serve asleader,</li>
Agreement (for elections and entry commitment) requires separate majorities from both the old and new configurations.</li>
</ul>
</blockquote>
🔗</a>Log compaction</h2>
As the WAL holds the commands, we need to compact it. Raft is using snapshots as describe here:</p>


the leader must occasionally send snapshots to followers that lag behind.</p>
</blockquote>
This is useful for slow follower or a new server joining the cluster.</p>

The leader uses a new RPC called InstallSnapshot to send snapshots to followers that are too far behind.</p>
</blockquote>
🔗</a>Client interaction</h2>

Clients of Raft send all of their requests to the leader. When a client first starts up, it connects to a randomly-chosen server. If the client’s first choice is not the leader,that server will reject the client’s request and supply information about the most recent leader it has heard from.</p>
</blockquote>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Announcing Kafka-on-Pulsar: bring native Kafka protocol support to Apache Pulsar
Pierre Zemb — Tue, 24 Mar 2020 10:24:27 +0100

This is a repost from OVHcloud's official blogpost.</a>, please read it there to support my company. Thanks Horacio Gonzalez</a> for the awesome drawings!</p>
</blockquote>
This post has been published on both the StreamNative and OVHcloud blogs and was co-authored by Sijie Guo</a>, Jia Zhai</a> and Pierre Zemb</a>. Thanks Horacio Gonzalez</a> for the illustrations!</p>
</p>
We are excited to announce that StreamNative and OVHcloud are open-sourcing "Kafka on Pulsar" (KoP). KoP brings the native Apache Kafka protocol support to Apache Pulsar by introducing a Kafka protocol handler on Pulsar brokers. By adding the KoP protocol handler to your existing Pulsar cluster, you can now migrate your existing Kafka applications and services to Pulsar without modifying the code. This enables Kafka applications to leverage Pulsar's powerful features, such as:</p>

Streamlined operations with enterprise-grade multi-tenancy</li>
Simplified operations with a rebalance-free architecture</li>
Infinite event stream retention with Apache BookKeeper and tiered storage</li>
Serverless event processing with Pulsar Functions</li>
</ul>
🔗</a>What is Apache Pulsar?</h2>
Apache Pulsar is an event streaming platform designed from the ground up to be cloud-native- deploying a multi-layer and segment-centric architecture. The architecture separates serving and storage into different layers, making the system container-friendly. The cloud-native architecture provides scalability, availability and resiliency and enables companies to expand their offerings with real-time data-enabled solutions. Pulsar has gained wide adoption since it was open-sourced in 2016 and was designated an Apache Top-Level project in 2018.</p>
🔗</a>The need behind KoP</h2>
Pulsar provides a unified messaging model for both queueing and streaming workloads. Pulsar implemented its own protobuf-based binary protocol to provide high performance and low latency. This choice of protobuf makes it convenient to implement Pulsar clients</a> and the project already supports Java, Go, Python and C++ languages alongside thirdparty clients</a> provided by the community. However, existing applications written using other messaging protocols had to be rewritten to adopt Pulsar's new unified messaging protocol.</p>
To address this, the Pulsar community developed applications to facilitate the migration to Pulsar from other messaging systems. For example, Pulsar provides a Kafka wrapper</a> on Kafka Java API, which allows existing applications that already use Kafka Java client switching from Kafka to Pulsar without code change</a>. Pulsar also has a rich connector ecosystem, connecting Pulsar with other data systems. Yet, there was still a strong demand from those looking to switch from other Kafka applications to Pulsar.</p>
🔗</a>StreamNative and OVHcloud's collaboration</h2>
StreamNative was receiving a lot of inbound requests for help migrating from other messaging systems to Pulsar and recognized the need to support other messaging protocols (such as AMQP and Kafka) natively on Pulsar. StreamNative began working on introducing a general protocol handler framework in Pulsar that would allow developers using other messaging protocols to use Pulsar.</p>
Internally, OVHcloud had been running Apache Kafka for years, but despite their experience operating multiple clusters with millions of messages per second on Kafka, there were painful operational challenges. For example, putting thousands of topics from thousands of users into a single cluster was difficult without multi-tenancy.</p>
As a result, OVHcloud decided to shift and build the foundation of their topic-as-a-service product, called ioStream, on Pulsar instead of Kafka. Pulsar's multi-tenancy and the overall architecture with Apache Bookkeeper simplified operations compared to Kafka.</p>
After spawning the first region, OVHcloud decided to implement it as a proof-of-concept proxy capable of transforming the Kafka protocol to Pulsar on the fly. During this process, OVHcloud discovered that StreamNative was working on bringing the Kafka protocol natively to Pulsar, and they joined forces to develop KoP.</p>
</p>
KoP was developed to provide a streamlined and comprehensive solution leveraging Pulsar and BookKeeper's event stream storage infrastructure and Pulsar's pluggable protocol handler framework. KoP is implemented as a protocol handler plugin with protocol name "kafka". It can be installed and configured to run as part of Pulsar brokers.</p>
🔗</a>The distributed log</h2>
Both Pulsar and Kafka share a very similar data model around log</strong> for both pub/sub messaging and event streaming. For example, both are built on top of a distributed log. Kafka implements the distributed log in a partition-basis architecture, where a distributed log (a partition in Kafka) is designated to store in a set of brokers, while Pulsar deploys a segment</strong>-based architecture to implement its distributed log by leveraging Apache BookKeeper as its scale-out segment storage layer. Pulsar's segment</em> based architecture provides benefits such as rebalance-free, instant scalability, and infinite event stream storage. You can learn more about the key differences between Pulsar and Kafka in this Splunk blog</a> and in this blog from the Bookkeeper project</a>.</p>
Since both of the systems are built on a similar data model, a distributed log, it is very simple to implement a Kafka-compatible protocol handler by leveraging Pulsar's distributed log storage and its pluggable protocol handler framework (introduced in the 2.5.0 release).</p>
🔗</a>Implementations</h2>
The implementation is done by comparing the protocols between Pulsar and Kafka. We found that there are a lot of similarities between these two protocols. Both protocols are comprised of the following operations:</p>

Topic Lookup</strong>: All the clients connect to any broker to lookup the metadata (i.e. the owner broker) of the topics. After fetching the metadata, the clients establish persistent TCP connections to the owner brokers.</li>
Produce</strong>: The clients talk to the owner</strong> broker of a topic partition to append the messages to a distributed log.</li>
Consume</strong>: The clients talk to the owner</strong> broker of a topic partition to read the messages from a distributed log.</li>
Offset</strong>: The messages produced to a topic partition are assigned with an offset. The offset in Pulsar is called MessageId. Consumers can use offsets</strong> to seek to a given position within the log to read messages.</li>
Consumption State</strong>: Both systems maintain the consumption state for consumers within a subscription (or a consumer group in Kafka). The consumption state is stored in __offsets topic in Kafka, while the consumption state is stored as cursors in Pulsar.</li>
</ul>
As you can see, these are all the primitive operations provided by a scale-out distributed log storage such as Apache BookKeeper. The core capabilities of Pulsar are implemented on top of Apache BookKeeper. Thus it is pretty easy and straightforward to implement the Kafka concepts by using the existing components that Pulsar has developed on BookKeeper.

The following figure illustrates how we add the Kafka protocol support within Pulsar. We are introducing a new Protocol Handler</strong>which implements the Kafka wire protocol by leveraging the existing components (such as topic discovery, the distributed log library – ManagedLedger, cursors and etc) that Pulsar already has.</p>
</p>
🔗</a>Topics</h3>
In Kafka, all the topics are stored in one flat namespace. But in Pulsar, topics are organized in hierarchical multi-tenant namespaces. We introduce a setting kafkaNamespace</em> in broker configuration to allow the administrator configuring to map Kafka topics to Pulsar topics.</p>
In order to let Kafka users leverage the multi-tenancy feature of Apache Pulsar, a Kafka user can specify a Pulsar tenant and namespace as its SASL username when it uses SASL authentication mechanism to authenticate a Kafka client.</p>
🔗</a>Message ID and offset</h3>
In Kafka, each message is assigned with an offset once it is successfully produced to a topic partition. In Pulsar, each message is assigned with a MessageID</code>. The message id consists of 3 components, ledger-id</em>, entry-id</em>, and batch-index</em>. We are using the same approach in Pulsar-Kafka wrapper to convert a Pulsar MessageID to an offset and vice versa.</p>
🔗</a>Messages</h3>
Both a Kafka message and a Pulsar message have key, value, timestamp, and headers (note: this is called 'properties' in Pulsar). We convert these fields automatically between Kafka messages and Pulsar messages.</p>
🔗</a>Topic lookup</h3>
We use the same topic lookup approach for the Kafka request handler as the Pulsar request handler. The request handler does topic discovery to lookup all the ownerships for the requested topic partitions and responds with the ownership information as part of Kafka TopicMetadata back to Kafka clients.</p>
🔗</a>Produce Messages</h3>
When the Kafka request handler receives produced messages from a Kafka client, it converts Kafka messages to Pulsar messages by mapping the fields (i.e. key, value, timestamp and headers) one by one, and uses the ManagedLedger append API to append those converted Pulsar messages to BookKeeper. Converting Kafka messages to Pulsar messages allows existing Pulsar applications to consume messages produced by Kafka clients.</p>
🔗</a>Consume Messages</h3>
When the Kafka request handler receives a consumer request from a Kafka client, it opens a non-durable cursor to read the entries starting from the requested offset. The Kafka request handler converts the Pulsar messages back to Kafka messages to allow existing Kafka applications to consume the messages produced by Pulsar clients.</p>
🔗</a>Group coordinator & offsets management</h3>
The most challenging part is to implement the group coordinator and offsets management. Because Pulsar doesn't have a centralized group coordinator for assigning partitions to consumers of a consumer group and managing offsets for each consumer group. In Pulsar, the partition assignment is managed by broker on a per-partition basis, and the offset management is done by storing the acknowledgements in cursors by the owner broker of that partition.</p>
It is difficult to align the Pulsar model with the Kafka model. Hence, for the sake of providing full compatibility with Kafka clients, we implemented the Kafka group coordinator by storing the coordinator group changes and offsets in a system topic called *public/kafka/*offsets</em> in Pulsar.</p>
This allows us to bridge the gap between Pulsar and Kafka and allows people to use existing Pulsar tools and policies to manage subscriptions and monitor Kafka consumers. We add a background thread in the implemented group coordinator to periodically sync offset updates from the system topic to Pulsar cursors. Hence a Kafka consumer group is effectively treated as a Pulsar subscription. All the existing Pulsar toolings can be used for managing Kafka consumer groups as well.</p>
🔗</a>Bridge two popular messaging ecosystems</h2>
At both companies, we value customer success. We believe that providing a native Kafka protocol on Apache Pulsar will reduce the barriers for people adopting Pulsar to achieve their business success. By integrating two popular event streaming ecosystems, KoP unlocks new use cases. Customers can leverage advantages from each ecosystem and build a truly unified event streaming platform with Apache Pulsar to accelerate the development of real-time applications and services.</p>
With KoP, a log collector can continue collecting log data from its sources and producing messages to Apache Pulsar using existing Kafka integrations. The downstream applications can use Pulsar Functions to process the events arriving in the system to do serverless event streaming.</p>
🔗</a>Try it out</h2>
KoP is open sourced under Apache License V2 in https://github.com/streamnative/kop</a>.</p>
We are looking forward to your issues, and PRs. You can also join #kop channel in Pulsar Slack</a> to discuss all things about Kafka-on-Pulsar.</p>
StreamNative and OVHcloud are also hosting a webinar about KoP on March 31. If you are interested in learning more details about KoP,please sign up</a>. Looking forward to meeting you online.</p>
</p>
🔗</a>Thanks</h2>
The KoP project was originally initiated by StreamNative. The OVHcloud team joined the project to collaborate on the development of the KoP project. Many thanks to Pierre Zemb and Steven Le Roux from OVHcloud for their contributions to this project!</p>


Contributing to Apache HBase: custom data balancing
Pierre Zemb — Fri, 14 Feb 2020 10:24:27 +0100

This is a repost from OVHcloud's official blogpost.</a>, please read it there to support my company. Thanks Horacio Gonzalez</a> for the awesome drawings!</p>
</blockquote>
In today's blogpost, we're going to take a look at our upstream
contribution to Apache HBase's stochastic load balancer, based on our
experience of running HBase clusters to support OVHcloud's monitoring.</p>
</p>
🔗</a>The context</h2>
Have you ever wondered how:</p>

we generate the graphs for your OVHcloud server or web hosting package?</li>
our internal teams monitor their own servers and applications?</li>
</ul>
All internal teams are constantly gathering telemetry and monitoring data</strong> and sending them to a dedicated team,</strong> who are responsible for handling all the metrics and logs generated by OVHcloud's infrastructure</strong>: the Observability team.</p>
We tried a lot of different Time Series databases</strong>, and eventually chose Warp10</a> to handle our workloads. Warp10</strong> can be integrated with the various big-data solutions</strong> provided by the Apache Foundation.</a> In our case, we use Apache HBase</a> as the long-term storage datastore for our metrics.</p>
Apache HBase</a>, a datastore built on top of Apache Hadoop</a>, provides an elastic, distributed, key-ordered map.</strong> As such, one of the key features of Apache HBase for us is the ability to scan</strong>, i.e. retrieve a range of keys. Thanks to this feature, we can fetch thousands of datapoints in an optimised way</strong>.</p>
We have our own dedicated clusters, the biggest of which has more than 270 nodes to spread our workloads:</p>

between 1.6 and 2 million writes per second, 24/7</li>
between 4 and 6 million reads per second</li>
around 300TB of telemetry, stored within Apache HBase</li>
</ul>
As you can probably imagine, storing 300TB of data in 270 nodes comes with some challenges regarding repartition, as every</strong> bit is hot data, and should be accessible at any time</strong>. Let's dive in!</p>
🔗</a>How does balancing work in Apache HBase?</h2>
Before diving into the balancer, let's take a look at how it works. In Apache HBase, data is split into shards called Regions</code>, and distributed through RegionServers</code>. The number of regions will increase as the data is coming in, and regions will be split as a result. This is where the Balancer</code> comes in. It will move regions</strong> to avoid hotspotting a single RegionServer</code> and effectively distribute the load.</p>
</p>
The actual implementation, called StochasticBalancer</a>, uses a cost-based approach:</strong></p>

It first computes the overall cost</strong> of the cluster, by looping through cost functions</code>. Every cost function returns a number between 0 and 1 inclusive</strong>, where 0 is the lowest cost-best solution, and 1 is the highest possible cost and worst solution. Apache Hbase is coming with several cost functions, which are measuring things like region load, table load, data locality, number of regions per RegionServers... The computed costs are scaled by their respective coefficients, defined in the configuration</strong>.</li>
Now that the initial cost is computed, we can try to Mutate</code> our cluster. For this, the Balancer creates a random nextAction</code>, which could be something like swapping two regions</strong>, or moving one region to another RegionServer</strong>. The action is applied</strong> virtually</strong> , and then the new cost is calculated</strong>. If the new cost is lower than our previous one, the action is stored. If not, it is skipped. This operation is repeated thousands of times</code>, hence the Stochastic</code>.</li>
At the end, the list of valid actions is applied to the actual cluster.</strong></li>
</ol>
🔗</a>What was not working for us?</h2>
We found out that for our specific use case</strong>, which involved:</p>

Single table</li>
Dedicated Apache HBase and Apache Hadoop, tailored for our requirements</strong></li>
Good key distribution</li>
</ul>
the number of regions per RegionServer was the real limit for us</strong>.</p>
Even if the balancing strategy seems simple, we do think that being able to run an Apache HBase cluster on heterogeneous hardware is vital</strong>, especially in cloud environments, because you may not be able to buy the same server specs again in the future.</strong>
In our earlier example, our cluster grew from 80 to ~250 machines in
four years. Throughout that time, we bought new dedicated server
references, and even tested some special internal references.</p>
We ended-up with differents groups of hardware: some servers can handle only 180 regions, whereas the biggest can handle more than 900</strong>. Because of this disparity, we had to disable the Load Balancer to avoid the RegionCountSkewCostFunction</a>, which would try to bring all RegionServers to the same number of regions.</p>
</p>
Two years ago we developed some internal tools, which are responsible
for load balancing regions across RegionServers. The tooling worked
really good for our use case, simplifying the day-to-day operation of
our cluster.</p>
Open source is at the DNA of OVHcloud</strong>, and that means that we build our tools on open source software, but also that we contribute</strong>
and give it back to the community. When we talked around, we saw that
we weren't the only one concerned by the heterogenous cluster problem.
We decided to rewrite our tooling to make it more general, and to contribute</strong> it directly upstream</strong> to the HBase project .</strong></p>
🔗</a>Our contributions</h2>
The first contribution was pretty simple, the cost function list was a constant</a>. We added the possibility to load custom cost functions</a>.</p>
The second contribution was about adding an optional costFunction to balance regions according to a capacity rule</a>.</p>
🔗</a>How does it works?</h2>
The balancer will load a file containing lines of rules. A rule is composed of a regexp for hostname, and a limit.</strong> For example, we could have:</p>
rs[0-9] 200
</span>rs1[0-9] 50
</span></code></pre>
RegionServers with hostnames matching the first rules will have a limit of 200</strong>, and the others 50</strong>. If there's no match, a default is set.</p>
Thanks to these rule, we have two key pieces of information:</p>

the max number of regions for this cluster</strong></li>
the *rules for each servers</em></li>
</ul>
The HeterogeneousRegionCountCostFunction</code> will try to balance regions, according to their capacity.</strong></p>
Let's take an example... Imagine that we have 20 RS:</p>

10 RS, named rs0</code> to rs9</code>, loaded with 60 regions each, which can each handle 200 regions.</li>
10 RS, named rs10</code> to rs19</code>, loaded with 60 regions each, which can each handle 50 regions.</li>
</ul>
So, based on the following rules:</p>
rs[0-9] 200
</span>rs1[0-9] 50
</span></code></pre>
... we can see that the second group is overloaded</strong>, whereas the first group has plenty of space.</p>
We know that we can handle a maximum of 2,500 regions</strong> (200×10 + 50×10), and we have currently 1,200 regions</strong> (60×20). As such, the HeterogeneousRegionCountCostFunction</code> will understand that the cluster is full at 48.0%</strong> (1200/2500). Based on this information, we will then try to put all the RegionServers at ~48% of the load, according to the rules.</strong></p>
</p>
🔗</a>Where to next?</h2>
Thanks to Apache HBase's contributors, our patches are now merged</strong> into the master branch. As soon as Apache HBase maintainers publish a new release, we will deploy and use it at scale. This will allow more automation on our side, and ease operations for the Observability Team.</strong></p>
Contributing was an awesome journey. What I love most about open
source is the opportunity ability to contribute back, and build stronger
software. We had an opinion</strong> about how a particular issue should addressed, but the discussions with the community helped us to refine it</strong>. We spoke with e ngineers from other companies, who were struggling with Apache HBase's cloud deployments, just as we were</strong>, and thanks to those exchanges, our contribution became more and more relevant.</strong></p>


Notes about FoundationDB
Pierre Zemb — Thu, 30 Jan 2020 10:24:27 +0100
</p>
Notes About</a> is a blogpost serie  you will find a lot of links, videos, quotes, podcasts to click on</strong> about a specific topic. Today we will discover FoundationDB.</p>

🔗</a>Overview of FoundationDB</h2>
As stated in the official documentation</a>:</p>

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads but also has excellent performance for write-intensive workloads.</p>
</blockquote>
It has strong key points:</p>

Multi-model data store</li>
Easily scalable and fault tolerant</li>
Industry-leading performance</li>
Open source.</li>
</ul>
From a database dialect, it provides:</p>

strict serializability</a>(operations appear to have occurred in some order),</li>
external consistency</a>(For any two transactions, T1 and T2, if T2 starts to commit after T1 finishes committing, then the timestamp for T2 is greater than the timestamp for T1).</li>
</ul>
🔗</a>The story</h2>
FoundationDB started as a company in 2009, and then has been acquired in 2015 by Apple</a>. It was a bad public publicity for the database as the download were removed.</a></p>
On April 19, 2018, Apple open sourced the software, releasing it under the Apache 2.0 license</a>.</p>
🔗</a>Tooling before coding</h2>
🔗</a>Flow</h3>
From the Engineering page</a>:</p>

FoundationDB began with ambitious goals for both high performance per node and scalability. We knew that to achieve these goals we would face serious engineering challenges that would require tool breakthroughs. We’d need efficient asynchronous communicating processes like in Erlang or the Async in .NET, but we’d also need the raw speed, I/O efficiency, and control of C++. To meet these challenges, we developed several new tools, the most important of which is Flow</strong>, a new programming language that brings actor-based concurrency to C++11.</p>
</blockquote>
Flow is more of a stateful distributed system framework</strong> than an asynchronous library. It takes a number of highly opinionated stances on how the overall distributed system should be written, and isn’t trying to be a widely reusable building block.</p>

Flow adds about 10 keywords to C++11 and is technically a trans-compiler: the Flow compiler reads Flow code and compiles it down to raw C++11, which is then compiled to a native binary with a traditional toolchain.</p>
</blockquote>
Flow was developed before FDB, as stated in this 2013's post</a>:</p>

FoundationDB founder here. Flow sounds crazy. What hubris to think that you need a new programming language for your project? Three years later: Best decision we ever made.</p>
</blockquote>

We knew this was going to be a long project so we invested heavily in tools at the beginning. The first two weeks of FoundationDB were building this new programming language to give us the speed of C++ with high level tools for actor-model concurrency. But, the real magic is how Flow enables us to use our real code to do deterministic simulations of a cluster in a single thread. We have a white paper upcoming on this.</p>
</blockquote>

We've had quite a bit of interest in Flow over the years and I've given several talks on it at meetups/conferences. We've always thought about open-sourcing it... It's not as elegant as some other actor-model languages like Scala or Erlang (see: C++) but it's nice and fast at run-time and really helps productivity vs. writing callbacks, etc.</p>
</blockquote>

(Fun fact: We've only ever found two bugs in Flow. After the first, we decided that we never wanted a bug again in our programming language. So, we built a program in Python that generates random Flow code and independently-executes it to validate Flow's behavior. This fuzz tester found one more bug, and we've never found another.)</p>
</blockquote>
A very good overview of Flow is available here</a> and some details here</a>.</p>
🔗</a>Simulation-Driven development</h3>
One of Flow’s most important job is enabling Simulation</strong>:</p>

We wanted FoundationDB to survive failures of machines, networks, disks, clocks, racks, data centers, file systems, etc., so we created a simulation framework closely tied to Flow. By replacing physical interfaces with shims, replacing the main epoll-based run loop with a time-based simulation, and running multiple logical processes as concurrent Flow Actors, Simulation is able to conduct a deterministic simulation of an entire FoundationDB cluster within a single-thread! Even better, we are able to execute this simulation in a deterministic way, enabling us to reproduce problems and add instrumentation ex post facto. This incredible capability enabled us to build FoundationDB exclusively in simulation for the first 18 months and ensure exceptional fault tolerance long before it sent its first real network packet. For a database with as strong a contract as the FoundationDB, testing is crucial, and over the years we have run the equivalent of a trillion CPU-hours of simulated stress testing.</p>
</blockquote>
A good overview of the simulation can be found here</a>. You can also have a look at this awesome talk!</p>
</div>
    
    </iframe>
</div>
Simulation has been made possible by combining:</p>

Single-threaded pseudo-concurrency,</li>
Simulated implementation of all external communication,</li>
determinism.</li>
</ul>
Here's an example of a testfile</a>:</p>
testTitle=SwizzledCycleTest
</span>    testName=Cycle
</span>    transactionsPerSecond=5000.0
</span>    testDuration=30.0
</span>    expectedRate=0.01
</span>
</span>    testName=RandomClogging
</span>    testDuration=30.0
</span>    swizzle = 1
</span>
</span>    testName=Attrition
</span>    machinesToKill=10
</span>    machinesToLeave=3
</span>    reboot=true
</span>    testDuration=30.0
</span>
</span>    testName=Attrition
</span>    machinesToKill=10
</span>    machinesToLeave=3
</span>    reboot=true
</span>    testDuration=30.0
</span>
</span>    testName=ChangeConfig
</span>    maxDelayBeforeChange=30.0
</span>    coordinators=auto
</span></code></pre>
The test is splitted into two parts:</p>


The goal</strong>, for example doing transaction pointing to another with thousands of transactions per sec and there should be only 0.01% of success.</p>
</li>

What will be done to try to prevent the test to succeed</strong>. In this example it will at the same time</strong>:</p>

do random clogging. Which means that network connections will be stopped</strong> (preventing actors to send and receive packets). Swizzle flag means that a subset of network connections will be stopped and bring back in reverse order, 😳</li>
will poweroff/reboot machines</strong> (attritions) pseudo-randomly while keeping a minimal of three machines, 🤯</li>
change configuration</strong>, which means a coordination changes through multi-paxos for the whole cluster. 😱</li>
</ul>
</li>
</ul>
Keep in mind that all these failures will appears at the same time!</strong> Do you think that your current datastore has gone through the same test on a daily basis?</strong> I think not</a>.</p>
Applications written using the FoundationDB simulator have hierarchy: DataCenter -> Machine -> Process -> Interface</code>. Each of these can be killed/freezed/nuked</strong>. Even faulty admin commands fired by some DevOps are tested!</p>
🔗</a>Known limitations</h3>
Limitations are well described in the official documentation</a>.</p>
🔗</a>Recap</h3>
An awesome recap is available on the Software Engineering Daily podcast</a>:</p>

FoundationDB is tested in a very rigorous way using what's called a deterministic simulation</strong>. The reason they needed a new programming language to do this, is that to get a deterministic simulation, you have to make something that is deterministic. It's kind of obvious, but it's hard to do.</p>
</blockquote>

For example, if your process interacts with the network, or disks, or clocks, it's not deterministic. If you have multiple threads, not deterministic. So, they needed a way to write a concurrent program that could talk with networks and disks and that type of thing. They needed a way to write a concurrent program that does all of those things that you would think are non-deterministic in a deterministic way.</p>
</blockquote>

So, all FoundationDB processes, and FoundationDB, it's basically all written in Flow except a very small amount of it from the SQLite B-tree. The reason why that was useful is that when you use Flow, you get all of these higher level abstraction that let what you do what feels to you like asynchronous stuff, but under the hood, it's all implemented using callbacks in C++, which you can make deterministic by running it in a single thread. So, there's a scheduler that just calls these callbacks one after another and it's very crazy looking C++ code, like you wouldn't want to read it, but it's because of Flow they were able to implement that deterministic simulation.</p>
</blockquote>
🔗</a>The Architecture</h2>
According to the fdbmonitor and fdbserver</a>:</p>

The core FoundationDB server process is fdbserver</code>. Each fdbserver</code> process uses up to one full CPU core, so a production FoundationDB cluster will usually run N such processes on an N-core system.</p>
</blockquote>

To make configuring, starting, stopping, and restarting fdbserver processes easy, FoundationDB also comes with a singleton daemon process, fdbmonitor</code>, which is started automatically on boot. fdbmonitor</code> reads the foundationdb.conf</code> file and starts the configured set of fdbserver processes. It is also responsible for starting backup-agent.</p>
</blockquote>
The whole architecture is designed to automatically:</p>

load-balanced data and traffic,</li>
self-healing.</li>
</ul>
🔗</a>Microservices</h3>
A typical FDB cluster is composed of different actors which are describe here</a>.</p>
The most important role in FDB is the Coordinator</code>, it uses Paxos</code> to manage membership on a quorum to do writes. The Coordinator</code> is mostly only used to elect some peers and during recovery. You can view it as a Zookeeper-like stack.</p>
The Coordinator starts by electing a Cluster Controller</code>. It provides administratives informations about the cluster(I have 4 storage processes). Every process needs to register to the Cluster Controller</code> and then it will assign roles to them. It is the one that will heart-beat all the processes.</p>
Then a Master</code> is elected. The Master</code> process is reponsible for the data distribution</code> algorithms. Fun fact, the mapping between keys and storage servers is stored within FDB, which is you can actually move data by running transactions like any other application. He is also the one providing read versions</code> and version number</code> internally. He is also acting as the RateKeeper</code>.</p>
The Proxies</code> are responsible for providing read versions, committing transactions, and tracking the storage servers responsible for each range of keys.</p>
The Transaction Resolvers</code> are responsible determining conflicts between transactions. A transaction conflicts if it reads a key that has been written between the transaction’s read version and commit version. The resolver does this by holding the last 5 seconds of committed writes in memory, and comparing a new transaction’s reads against this set of commits.</p>
</p>
🔗</a>Read and Write Path</h3>
</div>
    
    </iframe>
</div>🔗</a>Read Path</h4>

Retrieve a consistend read version for the transaction</li>
Do reads from a consistent MVCC snapshot at that read version on the storage node</li>
</ol>
🔗</a>Write Path</h4>

client is sending a bundle to the proxy</code> containing:

read version for the transaction</li>
every readen key</li>
every mutation that you want to do</li>
</ul>
</li>
The proxy will assign a Commit version</code> to a batch of transactions. Commit version</code> is generated by the Master</code></li>
Proxy is sending to the resolver. This will check if the data that you want to mutate has been changed between your read Version</code> and your Commit version</code>. They are sharded by key-range.</li>
Transaction is made durable within the Transaction Logs</code> by fsync</code>ing the data. Before the data is even written to disk it is forwarded to the storage servers</code> responsible for that mutation. Internally, Transactions Logs</code> are creating a stream per Storage Server</code></strong>. Once the storage servers</code> have made the mutation durable, they pop it from the log. This generally happens roughly 6 seconds after the mutation was originally committed to the log.</li>
Storage servers</code> are lazily updating data on disk from the Transaction logs</code>. They are keeping new write in-memory.</li>
Transaction Logs</code> is responding OK to the Proxy and then the proxy is replying OK to the client.</li>
</ol>
You can find more diagrams about transactions here</a>.</p>
🔗</a>Recovery</h3>
Recovery processes are detailled at around 25min.</p>
During failure of a process (Except storage servers), the systems will try to create a new generation</code>, so new Master</code>, proxies</code>, resolvers</code> and transactions logs</code>. New master will get a read version from transactions logs, and commit with Paxos</code> the fact that starting from Read version</code>, the new generation is the one in charge.</p>
Storage servers</code> are replicating data on failures.</p>
🔗</a>The 5-second transaction limit</h3>
FoundationDB currently does not support transactions running for over five seconds. More details around 16min but the tl;dr</code> is:</p>

Storage servers are caching latest read in-memory,</li>
Resolvers are caching the last 5 seconds transactions.</li>
</ul>
🔗</a>Ratekeeper</h3>
More details around 31min but the tl;dr</code> is that when system is saturated, retrieving the Read version</code> is slowed down.</p>
🔗</a>Storage</h3>
A lot of information are available in this talk:</p>
</div>
    
    </iframe>
</div>

memory</code> is optimized for small databases. Data is stored in memory and logged to disk. In this storage engine, all data must be resident in memory at all times, and all reads are satisfied from memory.</li>
SSD</code> Storage Engine is based on SQLite B-Tree</li>
Redwood</code> will be a new storage engine based on Versioned B+Tree</li>
</ul>
🔗</a>Developer experience</h2>
FoundationDB’s keys are ordered, making tuples</code> a particularly useful tool for data modeling. FoundationDB provides a tuple layer</strong> (available in each language binding) that encodes tuples into keys. This layer lets you store data using a tuple like (state, county)</code> as a key. Later, you can perform reads using a prefix like (state,)</code>. The layer works by preserving the natural ordering of the tuples.</p>
Everything is wrapped into a transaction in FDB.</p>
You can have a nice overview by reading the README of tsdb-layer</a>, an experiment combining Time Series and FoundationDB: Millions of writes/s and 10x compression in under 2,000 lines of Go.</p>
🔗</a>FDB One more things: Layers</h2>
🔗</a>Concept of layers</h3>
</div>
    
    </iframe>
</div>
FDB is resolving many distributed problems, but you still need things like security, multi-tenancy, query optimizations, schema, indexing</strong>.</p>

</p>

Layers are designed to develop features above FDB.</strong> The record-layer provided by Apple is a good starting point to build things above it, as it provides structured schema, indexes, and (async) query planner.</strong></p>

</p>

The record-layer provided by Apple is a good starting point to build things above it, as it provides structured schema, indexes, and (async) query planner.</strong></p>

</p>
🔗</a>Apple's Record Layer</h3>
The paper is located FoundationDB Record Layer:A Multi-Tenant Structured Datastore</a></p>
</div>
    
    </iframe>
</div>
Record Layer was designed to solve CloudKit problem.</p>

</p>

Record allow multi-tenancy with schema above FDB</p>

</p>
</p>

Record Layers is providing stateless compute</p>

</p>

And streaming queries!</p>

</p>

🔗</a>Kubernetes Operators</h2>
🔗</a>Overview of the operator</h3>
</div>
    
    </iframe>
</div>

</p>

</p>

Upgrade is done by bumping all processes at once</strong> 😱</p>

</p>

</p>
🔗</a>Combining chaos-mesh and the operator</h3>
I played a bit with the operator by combining:</p>

FoundationDB/fdb-kubernetes-operator</a></li>
pingcap/go-ycsb</a></li>
pingcap/chaos-mesh</a></li>
PierreZ/fdb-prometheus-exporter</a></li>
</ul>
The experiment is available here</a>.</p>
🔗</a>Roadmap</h2>
FoundationDB Release 7.0 Planning</a></p>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Diving into Kafka's Protocol
Pierre Zemb — Sun, 08 Dec 2019 15:00:00 +0100
</p>
Diving Into</a> is a blogpost serie where we are digging a specific part of of the project's basecode. In this episode, we will digg into Kafka's protocol.</p>

🔗</a>The protocol reference</h2>
For the last few months, I worked a lot around Kafka's protocols, first by creating a fully async Kafka to Pulsar Proxy in Rust, and now by contributing directly to KoP (Kafka On Pulsar)</a>. The full Kafka Protocol documentation is available here</a>, but it does not offer a global view of what is happening for a classic Producer and Consumer exchange. Let's dive in!</p>
🔗</a>Common handshake</h3>
After a client established the TCP connection, there is a few common requests and responses that are almost always here.</p>
The common handhake can be divided in three parts:</p>

Being able to understand each other. For this, we are using API_VERSIONS</a></strong> to know which versions of which TCP frames can be uses,</li>
Establish Auth using SASL</strong> if needed, thanks to SASL_HANDSHAKE</a></strong> and SASL_AUTHENTICATE</a></strong>,</li>
Retrieve the topology of the cluster using METADATA</a></strong>.</li>
</ul>

All exchange are based between a Kafka 2.0 cluster and client.</p>
</blockquote>

All the following diagrams are generated with MermaidJS</a>.</p>
</blockquote>

            sequenceDiagram

    Note left of KafkaClient: I'm speaking Kafka <br/> 2.3,but can the <br/> broker understand <br/> me?

    KafkaClient ->>+ Broker0: API_VERSIONS request

    Note right of Broker0: I can handle theses <br/> structures in theses <br/>versions: ...
    Broker0 ->>- KafkaClient: 

    Note left of KafkaClient: Thanks!<br/> I see you can handle <br/> SASL, let's auth! <br/> can you handle <br/> SASL_PLAIN?
    KafkaClient ->>+ Broker0: SASL_HANDSHAKE request

    Note right of Broker0: Yes I can handle <br/> SASL_PLAIN <br/> among others
    Broker0 ->>- KafkaClient: 

    Note left of KafkaClient: Awesome, here's <br/> my credentials!
    KafkaClient ->>+ Broker0: SASL_AUTHENTICATE request

    Note right of Broker0: Checking...
    Note right of Broker0: You are <br/>authenticated!
    Broker0 ->>- KafkaClient: 

    Note left of KafkaClient: Cool! <br/> Can you give <br/> the cluster topology?<br/> I want to <br/> use 'my-topic'
    KafkaClient ->>+ Broker0: METADATA request

    Note right of Broker0: There is one topic <br/> with one partition<br/> called 'my-topic'<br/>The partition's leader <br/> is Broker0
    Broker0 ->>- KafkaClient: 

Note left of KafkaClient: That is you, I don't <br/> need to handshake <br/> again with <br/> another broker
    </pre>
</div>🔗</a>Producing</h3>
The PRODUCE</a></strong> API is used to send message sets to the server. For efficiency it allows sending message sets intended for many topic partitions in a single request.</p>

            sequenceDiagram

    Note over KafkaClient,Broker0: ...handshaking, see above...

    loop pull msg
        Note left of KafkaClient: I have a batch <br/> containing one <br/> message for the <br/> partition-0 <br/> of 'my-topic'
        KafkaClient ->>+ Broker0: PRODUCE request

        Note right of Broker0: Processing...<br/>
        Note right of Broker0: Done!
        Broker0 ->>- KafkaClient: 
        
        Note left of KafkaClient: Thanks
    end
    </pre>
</div>🔗</a>Consuming</h3>
Consuming is more complicated than producing. You can learn more in The Magical Group Coordination Protocol of Apache Kafka</a> By Gwen Shapira, Principal Data Architect @ Confluent and also in the Kafka Client-side Assignment Proposal</a>.</p>
Consuming can be divided in three parts:</p>

coordinating the consumers to assign them partitions, using:

FIND_COORDINATOR</a></strong>,</li>
JOIN_GROUP</a></strong>,</li>
SYNC_GROUP</a></strong>,</li>
</ul>
</li>
then fetch messages using:

OFFSET_FETCH</a></strong>,</li>
LIST_OFFSETS</a></strong>,</li>
FETCH</a></strong>,</li>
OFFSET_COMMIT</a></strong>,</li>
</ul>
</li>
Send lifeproof to the coordinator using HEARTBEAT</a></strong>.</li>
</ul>
For the sake of the explanation, we have now another Broker1 which is holding the coordinator for topic 'my-topic'. In real-life, it would be the same.</p>

            sequenceDiagram

    Note over KafkaClient,Broker0: ...handshaking, see above...

    Note left of KafkaClient: Who is the <br/> coordinator for<br/> 'my-topic'?
    KafkaClient ->>+ Broker0: FIND_COORDINATOR request

    Note right of Broker0: It is Broker1!
    Broker0 ->>- KafkaClient: 

    Note left of KafkaClient: OK, let's connect<br/> to Broker1
    Note over KafkaClient,Broker1: ...handshaking, see above...

    Note left of KafkaClient: Hi, I want to join a <br/> consumption group <br/>for 'my-topic'
    KafkaClient ->>+ Broker1: JOIN_GROUP request

    Note right of Broker1: Welcome! I will be <br/> waiting a bit for any <br/>of your friends.
    Note right of Broker1: You are now leader. <br/>Your group contains <br/> only one member.<br/> You now  need to <br/> assign partitions to <br/> them. 
    Broker1 ->>- KafkaClient: 

    Note left of KafkaClient: Computing <br/>the assigment...
    Note left of KafkaClient: Done! I will be <br/> in charge of handling <br/> partition-0 of <br/>'my-topic'
    KafkaClient ->>+ Broker1: SYNC_GROUP request

    Note right of Broker1: Thanks, I will <br/>broadcast the <br/>assigmnents to <br/>everyone
    Broker1 ->>- KafkaClient: 

    Note left of KafkaClient: Can I get the <br/> committed offsets <br/> for partition-0<br/>for my consumer<br/>group?
    KafkaClient ->>+ Broker1: OFFSET_FETCH request

    Note right of Broker1: Found no <br/>committed offset<br/> for partition-0
    Broker1 ->>- KafkaClient: 

    Note left of KafkaClient: Thanks, I will now <br/>connect to Broker0

    Note over KafkaClient,Broker0: ...handshaking again...

    opt if new consumer-group
        Note left of KafkaClient: Can you give me<br/> the earliest position<br/> for partition-0?
        KafkaClient ->>+ Broker0: LIST_OFFSETS request
        
        Note right of Broker0: Here's the earliest <br/> position: ...
        Broker0 ->>- KafkaClient: 
    end 
    loop pull msg

        opt Consume
            Note left of KafkaClient: Can you give me<br/> some messages <br/> starting  at offset X?
            KafkaClient ->>+ Broker0: FETCH request

            Note right of Broker0: Here some records...
            Broker0 ->>- KafkaClient: 

            Note left of KafkaClient: Processing...
            Note left of KafkaClient: Can you commit <br/>offset X?
            KafkaClient ->>+ Broker1: OFFSET_COMMIT request

            Note right of Broker1: Committing...
            Note right of Broker1: Done!
            Broker1 ->>- KafkaClient: 
        end

        Note left of KafkaClient: I need to send <br/> some lifeness proof <br/> to the coordinator           
        opt Healthcheck
            Note left of KafkaClient: I am still alive!  
            KafkaClient ->>+ Broker1: HEARTBEAT request
            Note right of Broker1: I hear you
            Broker1 ->>- KafkaClient: 
        end
    end
    </pre>
</div>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Diving into Hbase's MemStore
Pierre Zemb — Sun, 17 Nov 2019 10:24:27 +0100
</p>
Diving Into</a> is a blogpost serie where we are digging a specific part of of the project's basecode. In this episode, we will digg into the implementation behind Hbase's MemStore.</p>

tl;dr:</code> Hbase is using the ConcurrentSkipListMap</a>.</p>
🔗</a>What is the MemStore?</h2>

The memtable</code> from the official BigTable paper</a> is the equivalent of the MemStore</code> in Hbase.</p>
</blockquote>
As rows are sorted lexicographically</strong> in Hbase, when data comes in, you need to have some kind of a in-memory buffer</strong> to order those keys. This is where the MemStore</code> comes in. It absorbs the recent write (or put in Hbase semantics) operations. All the rest are immutable files called HFile</code> stored in HDFS. There is one MemStore</code> per column family</code>.</p>
Let's dig into how the MemStore internally works in Hbase 1.X.</p>
🔗</a>Hbase 1</h2>
All extract of code for this section are taken from rel/1.4.9</a> tag.</p>
🔗</a>in-memory storage</h3>
The MemStore interface</a> is giving us insight on how it is working internally.</p>
  </span>/**
</span>   * Write an update
</span>   * </span>@param </span>cell
</span>   * </span>@return</span> approximate size of the passed cell.
</span>   */
</span>long add(final Cell cell);
</span></code></pre>
-- add function on the MemStore</a></p>
The implementation is hold by DefaultMemStore</a>. add</code> is wrapped by several functions, but in the end, we are arriving here:</p>
  </span>private boolean </span>addToCellSet</span>(</span>Cell</span> e) {
</span>    </span>boolean</span> b = </span>this</span>.activeSection.</span>getCellSkipListSet</span>().</span>add</span>(e);
</span></code></pre>
-- addToCellSet on the DefaultMemStore</a></p>
CellSkipListSet class</a> is built on top of ConcurrentSkipListMap</a>, which provide nice features:</p>

concurrency</li>
sorted elements</li>
</ul>
🔗</a>Flush on HDFS</h3>
As we seen above, the MemStore</code> is supporting all the puts. When asked to flush, the current memstore is moved to snapshot and is cleared</strong>. Flushed file are called (HFiles</a>) and they are similar to SSTables</code> introduced by the official BigTable paper</a>. HFiles are flushed on the Hadoop Distributed File System called HDFS</code>.</p>

If you want deeper insight about SSTables, I recommend reading Table Format from the awesome RocksDB wiki</a></p>
</blockquote>
🔗</a>Compaction</h3>
Compaction are only run on HFiles. It means that if hot data is continuously updated, we are overusing memory due to duplicate entries per row per MemStore</strong>. Accordion tends to solve this problem through in-memory compactions</em>. Let's have a look to Hbase 2.X!</p>
🔗</a>Hbase 2</h2>
🔗</a>storing data</h3>
All extract of code starting from here are taken from rel/2.1.2</a> tag.</strong></p>
Does MemStore</code> interface changed?</p>
  </span>/**
</span>   * Write an update
</span>   * </span>@param </span>cell
</span>   * </span>@param </span>memstoreSizing</span> The delta in memstore size will be passed back via this.
</span>   *        This will include both data size and heap overhead delta.
</span>   */
</span>  void add(final Cell cell, MemStoreSizing memstoreSizing);
</span></code></pre>
-- add function in MemStore interface</a></p>
The signature changed a bit, to include passing a object instead of returning a long. Moving on.</p>
The new structure implementing MemStore is called AbstractMemStore</a>. Again, we have some layers, where AbstractMemStore is writing to a MutableSegment</code>, which itsef is wrapping Segment</code>. If you dig far enough, you will find that data are stored into the CellSet class</a> which is also things built on top of ConcurrentSkipListMap</strong>!</p>
🔗</a>in-memory Compactions</h3>
Hbase 2.0 introduces a big change to the original memstore called Accordion which is a codename for in-memory compactions. An awesome blogpost is available here: Accordion: HBase Breathes with In-Memory Compaction</a> and the document design</a> is also available.</p>

Thank you</strong> for reading my post! feel free to react to this article, I'm also available on Twitter</a> if needed.</p>


What can be gleaned about GFS successor codenamed Colossus?
Pierre Zemb — Sun, 04 Aug 2019 15:07:11 +0200
In the last few months, there has been numerous blogposts about the end of the Hadoop-era. It is true that:</p>

Health of Hadoop-based companies are publicly bad</a></li>
Hadoop has a bad publicity with headlines like 'What does the death of Hadoop mean for big data?'</a></li>
</ul>
Hadoop, as a distributed-system, is hard to operate, but can be essential for some type of workload</strong>. As Hadoop is based on GFS, we can wonder how GFS evolved inside Google.</p>
🔗</a>Hadoop's story</h2>
Hadoop is based on a Google's paper called The Google File System</a> published in 2003. There are some key-elements on this paper:</p>

It was designed to be deployed with Borg</a>,</li>
to "simplify the overall design problem</a>", they:

implemented a single master architecture</li>
dropped the idea of a full POSIX-compliant file system</li>
</ul>
</li>
Metadatas are stored in RAM in the master,</li>
Datas are stored within chunkservers,</li>
There is no YARN or Map/Reduce or any kind of compute capabilities.</li>
</ul>
🔗</a>Is Hadoop still revelant?</h2>
Google with GFS and the rest of the world with Hadoop hit some issues:</p>

One (Metadata) machine is not large enough for large FS,</li>
Single bottleneck for metadata operations,</li>
Not appropriate for latency sensitive applications,</li>
Fault tolerant not HA,</li>
Unpredictable performance,</li>
Replication's cost,</li>
HDFS Write-path pipelining,</li>
fixed-size of blocks,</li>
cost of operations,</li>
...</li>
</ul>
Despite all the issues, Hadoop is still relevant for some usecases, such as Map/Reduce, or if you need Hbase as a main datastore. There is stories available online about the scalability of Hadoop:</p>

Twitter has multiple clusters storing over 500 PB (2017)</a></li>
whereas Google prefered to "Scaled to approximately 50M files, 10P" to avoid "added management overhead" brought by the scaling.</a></li>
</ul>
Nowadays, Hadoop is mostly used for Business Intelligence or to create a datalake, but at first, GFS was designed to provide a distributed file-system on top of commodity servers.</p>
Google's developers were/are deploying applications into "containers", meaning that any process could be spawned somewhere into the cloud</strong>. Developers are used to work with the file-system abstraction, which provide a layer of durability and security. To mimic that process, they developed GFS, so that processes don't need to worry about replication</strong> (like Bigtable/HBase).</p>
This is a promise that, I think, was forgotten. In a world where Kubernetes seems</em> to be the standard, the need of a global distributed file-system is now higher than before</strong>. By providing a "file-system" abstraction for applications deployed in Kubernetes, we may be solving many problems Kubernetes-adopters are hitting, such as:</p>

How can I retrieve that particular file for my applications deployed on the other side of the Kubernetes cluster?</li>
Should I be moving that persistent volume over my slow network?</li>
What is happening when Kubernetes killed an alpha pod in the middle of retrieving snapshot</a>?</li>
</ul>
🔗</a>Well, let's put Hadoop in Kubernetes</h2>
Putting a distributed systems inside Kubernetes is currently a unpleasant experience because of the current tooling:</p>

Helm is not helping me expressing my needs as a distributed-system operator. Even worse, the official Helm chart for Hadoop is limited to YARN and Map/Reduce and "Data should be read from cloud based datastores such as Google Cloud Storage, S3 or Swift."</a></li>
Kubernetes Operators has no access to key-metrics, so they cannot watch over your applications correctly. It is only providing a "day-zero to day-two" good experience,</li>
Google seems to not be using the Operators design internally</a>.</li>
CouchDB developers</a> are saying that:

"For certain workloads, the technology isn’t quite there yet"</li>
"In certain scenarios that are getting smaller and smaller, both Kubernetes and Docker get in the way of that. At that point, CouchDB gets slow, or you get timeout errors, that you can’t explain."</li>
</ul>
</li>
</ul>
🔗</a>How GFS evolved within Google</h2>
🔗</a>Technical overview</h3>
As GFS's paper was published in 2003, we can ask ourselves if GFS has evolved. And it did! The sad part is that there is only a few informations about this project codenamed Colossus</code>. There is no papers, and not a lot informations available, here's what can be found online:</p>


From Storage Architecture and Challenges(2010)</a>:</p>

They moved from full-replication to Reed-Salomon</a>. This feature is acually in Hadoop 3</a>,</li>
replication is handled by the client, instead of the pipelining,</li>
the metadata layer is automatically sharded. We can find more informations about that in the next ressource!</li>
</ul>
</li>

From Cluster-Level Storage @ Google(2017)</a>:</p>

GFS master replaced by Colossus</li>
GFS chunkserver replaced by D</li>
Colossus rebalances old, cold data</li>
distributes newly written data evenly across disks</li>
Metadatas are stored into BigTable. each Bigtable row corresponds to a single file.</li>
</ul>
</li>
</ul>
The "all in RAM" GFS master design was a severe single-point-of-failure, so getting rid of it was a priority. They didn't had a lof of options for a scalable and rock-solid datastore beside BigTable</strong>. When you think about it, a key/value datastore is a great replacement for a distributed file-system master:</p>

automatic sharding of regions,</li>
scan capabilities for files in the same "directory",</li>
lexical ordering,</li>
...</li>
</ul>
The funny part is that they now need a Colossus for Colossus. The only things saving them is that storing the metametametadata (the metadata of the metadata of the metadata) can be hold in Chubby.</p>


From GFS: Evolution on Fast-forward(2009)</a></p>

they moved to chunks of 1MB of files, as the limitations of the master disappeared. This is also allowing Colossus to support latency sensitive applications,</li>
</ul>
</li>

From a Github comment on Colossus</a>:</p>

File reconstruction from Reed-Salomnon was performed on both client-side and server-side</li>
on-the-fly recovery of data is greatly enhanced by this data layout(Reed Salomon)</li>
</ul>
</li>

From a Hacker News comment</a>:</p>

Colossus and D are two separate things.</li>
</ul>
</li>

From Colossus under the hood: a peek into Google’s scalable storage system</a>:</p>

Colossus's Control Plane is a scalable metadata service, which consists of many Curators. Clients talk directly to curators for control operations, such as file creation, and can scale horizontally.</li>
background storage managers called Custodians, there are handling tasks like disk space balancing and RAID reconstruction.</li>
Applications needs to specifies I/O, availability, and durability requirements</li>
</ul>
</li>
</ul>
What is that "D"?</p>


From The Production Environment at Google, from the Viewpoint of an SRE</a>:</p>

D stands for Disk</em>,</li>
D is a fileserver running on almost all machines in a cluster.</li>
</ul>
</li>

From The Production Environment at Google</a>:</p>

D is more of a block server than a file server</li>
It provides nothing apart from checksums.</li>
</ul>
</li>
</ul>
🔗</a>Deployments</h3>

How everything is deployed? Using Borg!</p>
🔗</a>Migration</h3>
The migration process is described in the now free Case Studies in Infrastructure Change Management</a> book.</p>
🔗</a>Is there an open-source effort to create a Colossus-like DFS?</h2>
I did not found any point towards a open-source version of Colossus, beside some work made for The Baidu File System</a> in which the Nameserver is implemented as a raft group.</p>
There is some work to add colossus's features in Hadoop</a> but based on the bad publicity Hadoop has now, I don't think there will be a lot of money to power those efforts.</p>
I do think that rewriting an distributed file-system based on Colossus would be a huge benefit for the community:</p>

Reimplement D may be easy, my current question is how far can we use modern FS such as OpenZFS</strong> to facilitate the work? FS capabilities such as OpenZFS checksums</a> seems pretty interesting.</li>
To resolve the distributed master issue, we could use Tikv</a> as a building block to provide an "BigTable experience" without the need of a distributed file-system underneath.</li>
</ul>
But remember:</p>

Like crypto, Do not roll your own DFS!</p>
</blockquote>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Playing with TTL in HBase
Pierre Zemb — Mon, 27 May 2019 22:07:11 +0200

   
</header>
Among all features provided by HBase, there is one that is pretty handy to deal with your data's lifecyle: the fact that every cell version can have Time to Live</strong> or TTL. Let's dive into the feature!</p>
🔗</a>Time To Live (TTL)</h1>
Let's read the doc first!</p>

ColumnFamilies can set a TTL length in seconds, and HBase will automatically delete rows once the expiration time is reached</strong>.</p>
</blockquote>
HBase Book: Time To Live (TTL)</a></p>
Let's play with it! You can easily start an standalone HBase by following the HBase Book</a>. Once your standalone cluster is started, we can get started:</p>
./bin/hbase</span> shell
</span>
</span>hbase</span>(main)</span>:001:0</span>> create '</span>test_table</span>', {'</span>NAME</span>' => '</span>cf1</span>','</span>TTL</span>' => 30} </span># 30 sec
</span></code></pre>
Now that our test_table is created, we can put</code> some data on it:</p>
hbase</span>(main)</span>:002:0</span>> put '</span>test_table</span>','</span>row123</span>','</span>cf1:desc</span>', '</span>TTL Demo</span>'
</span></code></pre>
And you can get</code> it with:</p>
hbase</span>(main)</span>:003:0</span>> get '</span>test_table</span>','</span>row123</span>','</span>cf1:desc</span>'
</span>COLUMN</span>                             CELL
</span> </span>cf1:desc</span>                          timestamp=1558366581134, value=TTL Demo
</span>1</span> row(s) </span>in</span> 0.0080 seconds
</span></code></pre>
Here's our row! But if you wait a bit, it will disappear</strong> thanks to the TTL:</p>
hbase</span>(main)</span>:004:0</span>> get '</span>test_table</span>','</span>row123</span>','</span>cf1:desc</span>'
</span>COLUMN</span>                             CELL
</span>0</span> row(s) </span>in</span> 0.0220 seconds
</span></code></pre>
It has been filtered from the result, but the data is still here.  You can trigger a raw</strong> scan to check:</p>
hbase</span>(main)</span>:002:0</span>> scan '</span>test_table</span>', {RAW => true}
</span>ROW</span>                                COLUMN+CELL
</span> </span>row123</span>                            column=cf1:desc, timestamp=1558366581134, value=TTL Demo
</span>1</span> row(s) </span>in</span> 0.3280 seconds
</span></code></pre>
It will be removed only when a major-compaction</strong> will occur. As we are playing, we can:</p>

force the memstore to be flushed as HFiles</strong></li>
force the compaction</strong>:</li>
</ul>

You may have heard about Accordion</a></b>, the new feature in HBase 2. If you are playing with HBase 2, you can enable it by following this link</a> and run compactions directly in the MemStores.</b>
</div>
hbase</span>(main)</span>:014:0</span>> flush '</span>test_table</span>'
</span>Took</span> 0.4456 seconds    
</span>hbase</span>(main)</span>:015:0</span>> compact '</span>test_table</span>'
</span>Took</span> 0.0468 seconds
</span># wait a bit
</span>hbase</span>(main)</span>:016:0</span>> scan '</span>test_table</span>', {RAW => true}
</span>ROW</span>                            COLUMN+CELL
</span>0</span> row(s)
</span>Took</span> 0.0060 seconds
</span></code></pre>
🔗</a>How does it works?</h1>
As always, the truth is held by the documentation:</p>

A {row, column, version} tuple exactly specifies a cell in HBase. It’s possible to have an unbounded number of cells where the row and column are the same but the cell address differs only in its version dimension.</p>
</blockquote>

While rows and column keys are expressed as bytes, the version is specified using a long integer</strong>. Typically this long contains time instances</strong> such as those returned by java.util.Date.getTime() or System.currentTimeMillis()</strong>,</p>
</blockquote>
HBase Book: Versions</a></p>
You may have seen it during our scan earlier, there is a timestamp associated</strong> with the version of the cell:</p>
hbase</span>(main)</span>:003:0</span>> get '</span>test_table</span>','</span>row123</span>','</span>cf1:desc</span>'
</span>COLUMN</span>                             CELL
</span> </span>cf1:desc</span>                          timestamp=1558366581134, value=TTL Demo
</span> </span>#                           here  ^^^^^^^^^^^^^^^^^^^^^^^ 
</span></code></pre>
Hbase used the System.currentTimeMillis()</code> at ingest time to add it. During scanner and compaction, as time went by, there was more than TTL seconds between the cell version and now, so the row was discarded</strong>.</p>
Now the real question is: can you set it by yourself and be real Time-Lord</strong> (of HBase)?</p>
The reponse is yes!</em> There is also a bit of a warning a bit below:</a></p>

Caution:</em> the version timestamp is used internally by HBase for things like time-to-live calculations</strong>. It’s usually best to avoid setting this timestamp yourself. Prefer using a separate timestamp attribute of the row, or have the timestamp as a part of the row key, or both.</p>
</blockquote>
Let's try it:</p>
date</span> +%</span>s -d </span>"</span>+2 min</span>"
</span>1558472441  </span># don't forget to add 3 zeroes as the time need to be in millisecond!
</span>
</span>./bin/hbase</span> shell
</span>hbase</span>(main)</span>:001:0</span>> put '</span>test_table</span>','</span>row1234</span>','</span>cf1:desc</span>', '</span>timestamp Demo</span>', 1558472441000  
</span>hbase</span>(main)</span>:044:0</span>> scan '</span>test_table</span>'
</span>ROW</span>                            COLUMN+CELL
</span> </span>row1234</span>                       column=cf1:desc, timestamp=1558473315, value=timestamp Demo
</span>1</span> row(s)
</span>Took</span> 0.0031 seconds
</span></code></pre>
Notice that we are using a timestamp at the end of the put</code> method? This will add the desired timestamp to the version</strong>. Which means that your application can control when your version will be removed, even with a TTL on your column-qualifier.</strong> You just need to compute a timestamp like this:</p>

ts = now - ttlCF + desiredTTL</code>.</p>
</blockquote>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Handling OVH's alerts with Apache Flink
Pierre Zemb — Sun, 03 Feb 2019 15:37:27 +0100
This is a repost from OVH's official blogpost.</a>. Thanks Horacio Gonzalez</a> for the awesome drawings!</p>
🔗</a>Handling OVH's alerts with Apache Flink</h1>
</p>
OVH relies extensively on metrics</strong> to effectively monitor its entire stack. Whether they are low-level</strong> or business</strong> centric, they allow teams to gain insight</strong> into how our services are operating on a daily basis. The need to store millions of datapoints per second</strong> has produced the need to create a dedicated team to build a operate a product to handle that load: **Metrics Data Platform</a>.By relying on **Apache Hbase</a>, Apache Kafka</a></strong> and Warp 10</strong></a>, we succeeded in creating a fully distributed platform that is handling all our metrics… and yours!</p>
After building the platform to deal with all those metrics, our next challenge was to build one of the most needed feature for Metrics: the Alerting.</strong></p>
🔗</a>Meet OMNI, our alerting layer</h2>
OMNI is our code name for a fully distributed</strong>, as-code</strong>, alerting</strong> system that we developed on top of Metrics. It is split into components:</p>

The management part</strong>, taking your alerts definitions defined in a Git repository, and represent them as continuous queries,</li>
The query executor</strong>, scheduling your queries in a distributed way.</li>
</ul>
The query executor is pushing the query results into Kafka, ready to be handled! We now need to perform all the tasks that an alerting system does:</p>

Handling alerts deduplication</strong> and grouping</strong>, to avoid alert fatigue.</a></li>
Handling escalation</strong> steps, acknowledgement</strong>or snooze</strong>.</li>
Notify</strong> the end user, through differents channels</strong>: SMS, mail, Push notifications, …</li>
</ul>
To handle that, we looked at open-source projects, such as Prometheus AlertManager,</a> LinkedIn Iris,</a> we discovered the hidden</em> truth:</p>

Handling alerts as streams of data,

moving from operators to another.</p>
</blockquote>
We embraced it, and decided to leverage Apache Flink</a> to create Beacon</strong>. In the next section we are going to describe the architecture of Beacon, and how we built and operate it.</p>
If you want some more information on Apache Flink, we suggest to read the introduction article on the official website: What is Apache Flink?</a></p>
🔗</a>Beacon architecture</strong></h2>
At his core, Beacon is reading events from Kafka</strong>. Everything is represented as a message</strong>, from alerts to aggregations rules, snooze orders and so on. The pipeline is divided into two branches:</p>

One that is running the aggregations</strong>, and triggering notifications based on customer's rules.</li>
One that is handling the escalation steps</strong>.</li>
</ul>
Then everything is merged to generate</strong> a</strong> notification</strong>, that is going to be forward to the right person. A notification message is pushed into Kafka, that will be consumed by another component called beacon-notifier.</strong></p>
🔗</a>Handling States</h2>
If you are new to streaming architecture, I recommend reading Dataflow Programming Model</a> from Flink official documentation.</p>
Everything is merged into a dataStream, partitionned</strong> (keyed by</a>in Flink API) by users. Here's an example:</p>
    </span>final </span>DataStream</span>> alertStream =
</span>    
</span>      </span>// Partitioning Stream per AlertIdentifier
</span>      cleanedAlertsStream.</span>keyBy</span>(</span>0</span>)
</span>      </span>// Applying a Map Operation which is setting since when an alert is triggered
</span>      .</span>map</span>(</span>new </span>SetSinceOnSelector</span>())
</span>      .</span>name</span>("</span>setting-since-on-selector</span>").</span>uid</span>("</span>setting-since-on-selector</span>")
</span>    
</span>      </span>// Partitioning again Stream per AlertIdentifier
</span>      .</span>keyBy</span>(</span>0</span>)
</span>      </span>// Applying another Map Operation which is setting State and Trend
</span>      .</span>map</span>(</span>new </span>SetStateAndTrend</span>())
</span>      .</span>name</span>("</span>setting-state</span>").</span>uid</span>("</span>setting-state</span>");
</span></code></pre>
In the example above, we are chaining two keyed operations:</p>

SetSinceOnSelector</strong>, which is setting since</strong> when the alert is triggered</li>
SetStateAndTrend</strong>, which is setting the state</strong>(ONGOING, RECOVERY or OK) and the trend</strong>(do we have more or less metrics in errors).</li>
</ul>
Each of this class is under 120 lines of codes because Flink is handling all the difficulties</strong>. Most of the pipeline are only composed</strong> of classic transformations</strong> such as Map, FlatMap, Reduce</a>, including their Rich</a> and Keyed</a> version. We have a few Process Functions</a>, which are very handy</strong> to develop, for example, the escalation timer.</p>
🔗</a>Integration tests</h2>
As the number of classes was growing, we needed to test our pipeline. Because it is only wired to Kafka, we wrapped consumer and producer to create what we call **scenari:**a series of integration tests running different scenarios.</p>
🔗</a>Queryable state</h2>
One killer feature of Apache Flink is the capabilities of **querying the internal state</strong></a></strong> of an operator**. Even if it is a beta feature, it allows us the get the current state of the different parts of the job:</p>

at which escalation steps are we on</li>
is it snoozed or ack</em>-ed</li>
Which alert is ongoing</li>
and so on.</li>
</ul>
Queryable state overview</p>
Thanks to this, we easily developed an API</strong> over the queryable state, that is powering our alerting view</strong> in Metrics Studio,</a> our codename for the Web UI of the Metrics Data Platform.</p>
🔗</a>Apache Flink deployment</h3>
We deployed the latest version of Flink (1.7.1</strong> at the time of writing) directly on bare metal servers with a dedicated Zookeeper's cluster using Ansible. Operating Flink has been a really nice surprise for us, with clear documentation and configuration</strong>, and an impressive resilience</strong>. We are capable of rebooting</strong> the whole Flink cluster, and the job is restarting at his last saved state</strong>, like nothing happened.</p>
We are using RockDB</strong> as a state backend, backed by OpenStack Swift storage</strong>provided by OVH Public Cloud.</p>
For monitoring, we are relying on Prometheus Exporter</a> with Beamium</a> to gain observability</strong> over job's health.</p>
🔗</a>In short, we love Apache Flink</h3>
If you are used to work with stream related software, you may have realized that we did not used any rocket science or tricks. We may be relying on basics streaming features offered by Apache Flink, but they allowed us to tackle many business and scalability problems with ease.</p>
</p>
As such, we highly recommend that any developers should have a look to Apache Flink. I encourage you to go through Apache Flink Training</a>, written by Data Artisans. Furthermore, the community has put a lot of effort to easily deploy Apache Flink to Kubernetes, so you can easily try Flink using our Managed Kubernetes!</p>


What are ACID transactions?
Pierre Zemb — Sun, 03 Feb 2019 00:00:00 +0000
🔗</a>Transaction?</h1>
"Programming should be about transforming data"
</span></code></pre>
--- Programming Elixir 1.3 by Dave Thomas</p>

As developers, we are interacting oftenly with data, whenever handling it from an API or a messaging consumer. To store it, we started to create softwares called relational database management system</strong> or RDBMS</a>. Thanks to them, we, as developers, can develop applications pretty easily, without the need to implement our own storage solution</strong>. Interacting with mySQL</a> or PostgreSQL</a> have now become a commodity</strong>. Handling a database is not that easy though, because anything can happen, from failures to concurrency isssues:</p>

How can we interact with datastores that can fail?</strong></li>
What is happening if two users are  updating a value at the same time?</strong></li>
</ul>
As a database user, we are using transactions</code> to answer these questions. As a developer, a transaction is a single unit of logic or work</strong>, sometimes made up of multiple operations. It is mainly an abstraction</strong> that we are using to hide underlying problems</strong>, such as concurrency or hardware faults.</p>
ACID</code> appears in a paper published in 1983 called "Principles of transaction-oriented database recovery"</a> written by Theo Haerder</em> and Andreas Reuter</em>. This paper introduce a terminology of properties for a transaction:</p>

A</strong>tomic, C</strong>onsistency, I</strong>solation, D</strong>urability</p>
</blockquote>
🔗</a>Atomic</h2>
Atomic, as you may have guessed, atomic</code> represents something that cannot be splitted</strong>. In the database transaction world, it means for example that if a transaction with several writes is started and failed</strong> at some point, none of the write will be committed</strong>. As stated by many, the word atomic</code> could be reword as abortability</code>.</p>

🔗</a>Consistency</h2>
You will hear about consistency</code> a lot of this serie. Unfortunately, this word can be used in a lot of context. In the ACID definition, it refers to the fact that a transaction will bring the database from one valid state to another.</strong></p>

🔗</a>Isolation</h2>
Think back to your database. Were you the only user on it? I don't think so. Maybe they were concurrent transactions at the same time, beside yours. Isolation while keeping good performance is the most difficult item on the list.</strong> There's a lot of litterature and papers about it, and we will only scratch the surface. There is different transaction isolation levels, depending on the number of guarantees provided.</p>
🔗</a>Isolation by the theory</h3>
The SQL standard defines four isolation levels: Serializable</code>, Repeatable Read</code>, Read Commited</code> and Read Uncommited</code>. The strongest isolation is Serializable</code> where transaction are not runned in parallel</strong>. As you may have guessed, it is also the slowest. Weaker isolation level are trading speed against anomalies</strong> that can be sum-up like this:</p>
Isolation level</th> dirty reads</a></th> Non-repeatable reads</a></th> Phantom reads</a></th> Performance</th></tr></thead>

Serializable</td> 😎</td> 😎</td> 😎</td> 👍</td></tr>
Repeatable Read</td> 😎</td> 😎</td> 😱</td> 👍👍</td></tr>
Read Commited</td> 😎</td> 😱</td> 😱</td> 👍👍👍</td></tr>
Read uncommited</td> 😱</td> 😱</td> 😱</td> 👍👍👍👍</td></tr>
</tbody></table>

I encourage you to click on all the links within the table to see everything that could go wrong in a weak database!</strong></p>
</blockquote>
🔗</a>Isolation in Real Databases</h3>
Now that we saw some theory, let's have a look on a particular well-known database: PostgreSQL. What kind of isolation PostgreSQL is offering?</p>

PostgreSQL provides a rich set of tools for developers to manage concurrent access to data. Internally, data consistency is maintained by using a multiversion model (Multiversion Concurrency Control, MVCC</strong>).</p>
</blockquote>
--- Concurrency Control introduction</a></p>
Wait what? What is MVCC? Well, turns out that after the SQL standards came another type of Isolation: Snapshot Isolation</strong>. Instead of locking that row for reading when somebody starts working on it, it ensures that any transaction will see a version of the data that is corresponding to the start of the query</strong>. As it is providing a good balance between performance and consistency</strong>, it became a standard used by the industry</a>.</p>

🔗</a>Durability</h2>
Durability</code> ensure that your database is a safe place</strong> where data can be stored without fear of losing it. If a transaction has commited successfully, any written data will not be forgotten.</p>
🔗</a>That's it?</h1>
All these properties may seems obvious to you</strong> but each of the item is involving a lot of engineering and researchs. And this is only valid for a single machine, the distributed transaction field</strong> is even more complicated, but we will get to it in another blogpost!</p>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Hbase Data Model
Pierre Zemb — Sun, 27 Jan 2019 20:24:27 +0100
🔗</a>HBase?</h2>
</p>

Apache HBase™</a> is a type of "NoSQL" database. "NoSQL" is a general term meaning that the database isn’t an RDBMS which supports SQL as its primary access language. Technically speaking, HBase is really more a "Data Store" than "Data Base" because it lacks many of the features you find in an RDBMS, such as typed columns, secondary indexes, triggers, and advanced query languages, etc.</p>
</blockquote>
-- Hbase architecture overview</a></p>
🔗</a>Hbase data model</h2>
The data model is simple: it's like a multi-dimensional map:</p>

Elements are stored as rows</strong> in a table</strong>.</li>
Each table has only one index, the row key</strong>. There are no secondary indices.</li>
Rows are sorted lexicographically by row key</strong>.</li>
A range of rows is called a region</strong>. It is similar to a shard.</li>
A row in HBase consists of a row key</strong> and one or more columns</strong>, which are holding the cells.</li>
Values are stored into what we call a cell</strong> and are versioned with a timestamp.</li>
A column is divided between a Column Family</strong> and a Column Qualifier</strong>. Long story short, a Column Family is kind of like a column in classic SQL, and a qualifier is a sub-structure inside a Colum family. A column Family is static</strong>, you need to create it during table creation, whereas Column Qualifiers can be created on the fly.</li>
</ul>
Not as easy as you thought? Here's an example! Let's say that we're trying to save the whole internet</strong>. To do this, we need to store the content of each pages, and versioned it. We can use the page address as the row key</strong> and store the contents in a column called "Contents"</strong>. Nowadays, website contents can be anything</strong>, from a HTML file to a binary such as a PDF. To handle that, we can create as many qualifiers</strong> as we want, such as "content:html" or "content:video".</p>
{
</span>  "</span>fr.pierrezemb.www</span>": {          </span>// Row key
</span>    "</span>contents</span>": {                 </span>// Column family
</span>      "</span>content:html</span>": {           </span>// Column qualifier
</span>        "</span>2017-01-01</span>":             </span>// A timestamp
</span>          "</span><html>...</span>",            </span>// The actual value
</span>        "</span>2016-01-01</span>":             </span>// Another timestamp
</span>          "</span><html>...</span>"             </span>// Another cell
</span>      },
</span>      "</span>content:pdf</span>": {            </span>// Another Column qualifier
</span>        "</span>2015-01-01</span>": 
</span>          "</span><pdf>...</span>"  </span>// my website may only contained a pdf in 2015
</span>      }
</span>    }
</span>  }
</span>}
</span></code></pre>
🔗</a>Key design</h2>
Hbase is most efficient at queries when we're getting a single row key</strong>, or during row range</strong>, ie. getting a block of contiguous data because keys are sorted lexicographically by row key</strong>. For example, my website fr.pierrezemb.www</code> and org.pierrezemb.www</code> would not be "near".</p>
As such, the key design</strong> is really important:</p>

If your data are too spread, you will have poor performance.</li>
If your data are too much collocate, you will also have poor performance.</li>
</ul>
As stated by the official documentation</a>:</p>

Hotspotting occurs when a large amount of client traffic is directed at one node, or only a few nodes, of a cluster</strong>. This traffic may represent reads, writes, or other operations. The traffic overwhelms the single machine responsible for hosting that region, causing performance degradation and potentially leading to region unavailability.</p>
</blockquote>
As you may have guessed, this is why we are using the reverse address name</strong> in my example, because www</code> is too generic, we would have hotspot the poor region holding data for www</code>.</p>
If you are curious about Hbase schema, you should have a look on Designing Your BigTable Schema</a>, as BigTable is kind of the proprietary version of Hbase.</p>
🔗</a>Be warned</h2>
I have been working with Hbase for the past three years, including operation and on-call duty.</strong> It is a really nice data store, but it diverges from classical RDBMS. Here's some warnings extracted from the well-written documentation:</p>

HBase is really more a "Data Store" than "Data Base" because it lacks many of the features you find in an RDBMS, such as typed columns, secondary indexes, triggers, and advanced query languages, etc. However, HBase has many features which supports both linear and modular scaling.</p>
</blockquote>
-- NoSQL?</a></p>

If you have hundreds of millions or billions of rows, then HBase is a good candidate. If you only have a few thousand/million rows, then using a traditional RDBMS might be a better choice due to the fact that all of your data might wind up on a single node (or two) and the rest of the cluster may be sitting idle.</p>
</blockquote>
-- When Should I Use HBase?</a></p>

Thank you</strong> for reading my post! Feel free to react to this article, I am also available on Twitter</a> if needed.</p>


Introducing HelloExoWorld: The quest to discover exoplanets with Warp10 and Tensorflow
Pierre Zemb — Wed, 11 Oct 2017 10:23:11 +0000
update 2019:</strong> this is a repost on my own blog. original article can be read on medium</a>.</p>


Artist’s impression of the super-Earth exoplanet LHS 1140b By ESO/spaceengine.org</a> — CC BY 4.0</a></em></p>
My passion for programming was kind of late, I typed my first line of code at my engineering school. It then became a passion</strong>, something I’m willing to do at work, on my free-time, at night or the week-end. But before discovering C and other languages, I had another passion: astronomy</strong>. Every summer, I was participating at the Nuit des Etoiles</strong></a>, a global french event</strong> organized by numerous clubs of astronomers offering several hundreds (between 300 and 500 depending on the year) of free animation sites for the general public.</p>

As you can see below, I was kind of young at the time</strong>!</em></p>
But the sad truth is that I didn’t do any astronomy during my studies. But now, I want to get back to it and look at the sky again</strong>. There were two obstacles:</p>

The price of equipments</li>
The local weather</li>
</ul>
I was looking for something that would unit my two passions: computer and astronomy</strong>. So I started googling:</p>
</p>
I found a lot of amazing projects using Raspberry pis, but I didn’t find something that would motivate me</strong> over the time. So I started typing over keywords, more work-related, such as time series</strong></em> or analytics</strong></em>. I found many papers related to astrophysics, but there was two keywords that were coming back: exoplanet detection</strong>.</p>
🔗</a>What is an exoplanet and how to detect it?</h3>
Let’s quote our good old friend Wikipedia</strong></a>:</p>

An exoplanet or extrasolar planet is a planet outside of our solar system that orbits a star.</em></p>
</blockquote>
do you know how many exoplanets that have been discovered? 3,529 confirmed planets</strong> as of 10/09/2017</a>. I was amazed by the number of them. I started digging into the detection methods</strong></a>. Turns out there is one method heavily used, called the transit method</strong>. It’s like a eclipse: when the exoplanet is passing in front of the star, the photometry is varying during the transit, as shown below:</p>
</p>
animation illustrating how a dip in the observed brightness of a star may indicate the presence of an exoplanet. Credits: NASA’s Goddard Space Flight Center</strong></em></p>
To recap, exoplanet detection using the transit method are in reality a time series analysis problem</strong>. As I’m starting to be familiar with that type of analytics thanks to my current work at OVH in Metrics Data Platform</strong></a>, I wanted to give it a try.</p>
🔗</a>Kepler/K2 mission</h3>
</p>
Image Credit: NASA Ames/W. Stenzel</em></p>
Kepler is a space observatory</strong> launched by NASA in March 2009 to discover Earth-sized planets orbiting other stars</strong>. The loss of a second of the four reaction wheels during May 2013</a> put an end to the original mission. Fortunately, scientists decided to create an entirely community-driven mission</strong> called K2, to reuse the Kepler spacecraft and its assets</strong>. But furthermore, the community is also encouraged to exploit the mission’s unique open</strong> data archive. Every image taken by the satellite can be downloaded and analyzed by anyone</strong>.</p>
More information about the telescope itself can be found here</strong></a>.</p>
🔗</a>Where I’m going</h3>
The goal of my project is to see if I can contribute to the exoplanets search</strong> using new tools such as Warp10</strong></a> and TensorFlow</strong></a>. Using Deep Learning to search for anomalies could be much more effective</strong> than writing WarpScript, because it is the neural network's job to learn</strong> by itself how</strong> to detect the exoplanets.</p>
As I’m currently following Andrew Ng courses about Deep Learning</strong></a>, it is also a great opportunity for me to play with Tensorflow</strong> in a personal project. The project can be divided into several steps:</p>

Import</strong> the data</li>
Analyze</strong> the data using WarpScript</li>
Build</strong> a neural network to search for exoplanets</li>
</ul>
Let's see how the import was done!</p>
🔗</a>Importing Kepler and K2 dataset</h3>
🔗</a>Step 0: Find the data</h4>
As mentioned previously, data are available from The Mikulski Archive for Space Telescopes or MAST</a>. It’s a NASA funded project</strong> to support and provide the astronomical community with a variety of astronomical data archives. Both Kepler and K2 dataset are available</strong> through campaigns</strong>. Each campaign has a collection of tar files, which are containing the FITS files associated. A FITS</strong></a> file is an open format</strong> for images which is also containing scientific data</strong>.</p>
</p>
FITS file representation.</em> Image Credit: KEPLER & K2 Science Center</em></a></p>
🔗</a>Step 1: ETL (Extract, Transform and Load) into Warp10</h4>
To speed-up acquisition, I developed kepler-lens</strong></a> to automatically</strong> download Kepler/K2 datasets and extract the needed time series</strong> into a CSV format. Kepler-lens</strong> is using two awesome libraries:</p>

pyKe</strong></a> to export the data from the FITS</strong></a> files to CSV (#PR69</strong></a> and #PR76</strong></a>  have been merged).</li>
kplr</strong></a> is used to tag</strong> the dataset. With it, I can easily find stars</strong> with confirmed</strong> exoplanets or candidates</strong>.</li>
</ul>
Then Kepler2Warp10</strong></a> is used to push the CSV files generated by kepler-lens to Warp10</strong>.</p>
To ease importation, an Ansible role</strong></a>  has been made, to spread the work across multiples small virtual machines</strong>.</p>

550k distincts stars</strong></li>
around 50k datapoints per star</strong></li>
</ul>
That's around 27,5 billions of measures</strong> (300GB of LevelDB files), imported on a standalone</strong> instance. The Warp10 instance is self-hosted</strong> on a dedicated Kimsufi</strong></a> server at OVH. Here’s the full specifications for the curious ones:</p>
</p>
Now that the data are available</strong>, we are ready to dive into the dataset</strong> and look for exoplanets</strong>! Let's use WarpScript</p>
!### Let's see a transit using WarpScript</p>
</p>
WarpScript logo</p>
For those who don’t know WarpScript, I recommend reading my previous blogpost “Engage maximum warp speed in time series analysis with WarpScript</strong></a>”.</p>
Let’s first plot the data! We are going to take a well-known star called Kepler-11</strong></a>. It has (at least) 6 confirmed exoplanets. Let's write our first WarpScript:</p>
The FETCH</a> function retrieves raw datapoints</strong> from Warp10. Let’s plot the result of our script:</p>
</p>
Mmmmh, the straight lines are representing empties period with no datapoints</strong>; they correspond to different observations</strong>. Let's divide the data</strong> and generate one time series per observation</strong> using TIMESPLIT</a>:</p>
To ease the display, 0 GET is used to get only the first observation</strong>. Let's see the result:</p>
</p>
Much better. Do you see the dropouts? Those are transiting exoplanets!</strong> Now we’ll need to write a WarpScript to automatically detect transits.</strong> But that was enough for today, so we’ll cover this **in the next blogpost!**Thank you for reading! Feel free to comment</strong> and to subscribe</strong> to the twitter account</a>!</p>
</p>
Artist’s impression of the ultracool dwarf star TRAPPIST-1 from close to one of its planets</strong>. Image Credit: By ESO/M. Kornmesser</a> — CC BY-SA 4.0</a></p>


Engage maximum warp speed in time series analysis with WarpScript
Pierre Zemb — Sun, 08 Oct 2017 20:43:05 +0000
update 2019:</strong> this is a repost on my own blog. original article can be read on medium</a>.</p>

</p>
We, at Metrics Data Platform</a>, are working everyday with Warp10 Platform</a>, an open source Time Series database. You may not know it because it’s not as famous as Prometheus</a> or InfluxDB</a> but Warp10 is the most powerful and generic solution</strong> to store and analyze sensor data. It’s the core</strong> of Metrics, and many internal teams from OVH are using Metrics Data Platform</a> to monitor their infrastructure. As a result, we are handling a pretty nice traffic 24/7/365, as you can see below:</p>
</p>
Not only Warp10 allows us to reach an unbelievable scalability but it also comes with his own language called WarpScript</strong>, to manipulate and perform heavy time series analysis. Before digging into the need of a new language, let’s talk a bit about the need of time series analysis.### What is a time serie ?</p>
A time serie, or sensor data, is simply a sequence of measurements over time</strong>. The definition is quite generic, because many things can be represented as a time serie:</p>

the evolution of the stock exchange or a bank account</li>
the number of calls on a webserver</li>
the fuel consumption of a car</li>
the time to insert a value into a database</li>
the time a customer is taking to register on your website</li>
the heart rate of a person measured through a smartwatch</li>
</ul>
From an historical point of view, time series appeared shortly after the creation of the Web, to help engineers monitor the networks</strong>. It quickly expands to also monitors servers. With the right monitoring system, you can have insights</strong> and KPIs</strong> about your service:</p>
Analysis of long-term trend</strong></p>

How fast is my database growing?</li>
At what speed my number of active user accounts grows?</li>
</ul>
The comparison over time</strong></p>

My queries run faster with the new version of my library? Is my site slower than last week?</li>
</ul>
Alerts</strong></p>

Trigger alerts based on advanced queries</li>
</ul>
Displaying data through dashboards</strong></p>

Dashboards help answer basic questions on the service, and in particular the 4 indispensable metrics: latency, traffic, errors and service saturation</strong></li>
</ul>
The possibility of designing retrospective</strong></p>

Our latency is doubling, what’s going on?### Time series are complicated to handle</li>
</ul>
Storage, retrieval and analysis of time series cannot be done through standard relational databases. Generally, highly scalable databases are used to support volumetry. For example, the 300,000 Airbus A380 sensors on board can generate an average of 16 TB of data per flight</strong>. On a smaller scale, a single sensor that measures every second generates 31.5 million values per year</strong>. Handling time series at scale is difficult, because you’re running into advanced distributed systems issues, such as:</p>

ingestion scalability</strong>, i.e. how to absorb all the datapoints 24⁄7</li>
query scalability</strong>, i.e. how to query in a raisonnable amount of time</li>
delete capability</strong>, i.e. how to handle deletes without stopping ingestion and query</li>
</ul>
Frustration with existing open source monitoring tools like Nagios</strong> and Ganglia</strong> is why the giants created their own tools — Google has Borgmon</strong> and Facebook has</strong> Gorilla</strong></a>, just to name two. They are closed sources but the idea of treating time-series data as a data source for generating alerts is now accessible to everyone, thanks to the former Googlers who decided to rewrite Borgmon</strong> outside Google.### Why another time series database?</p>
Now the time series ecosystem is bigger than ever, here’s a short list of what you can find to handle time series data:</p>

InfluxDB.</li>
Prometheus.</li>
Riak TS.</li>
OpenTSDB.</li>
</ul>
Then there’s Warp10</strong>. The difference is quite simple, Warp10 is a platform</strong> whereas all the time series listed above are stores</strong>. This is game changing, for multiples reasons.</p>
🔗</a>Security-first design</h4>
Security is mandatory for data access and sharing job’s results, but in most of the above databases, security access is not handled by default. With Warp10, security is handled with crypto tokens similar to Macaroons</a>.</p>
🔗</a>High level analysis capabilities</h4>
Using classical time series database, high level analysis must be done elsewhere</strong>, with R, Spark, Flink, Python, or whatever languages or frameworks that you want to use. Using Warp10, you can just submit your script</strong> and voilà</em>!</p>
🔗</a>Server-side calculation</h4>
Algorithms are resource heavy. Whatever they’re using CPU, ram, disk and network, you’ll hit limitations</strong> on your personal computer. Can you really aggregate and analyze one year of data from thousands of sensors on your laptop? Maybe, but what if you’re submitting the job from a mobile? To be scalable</strong>, analysis must be done server-side</strong>.### Meet WarpScript</p>
</p>
Warp10 folks created WarpScript, an extensible</strong> stack oriented programming language</strong></a> which offers more than 800 functions</strong> and several high level frameworks</strong> to ease and speed your data analysis. Simply create scripts</strong> containing your data analysis code and submit them to the platform</strong>, they will execute close to where the data resides</strong> and you will get the result of that analysis as a JSON object</strong> that you can integrate into your application</strong>.</p>
Yes, you’ll be able to run that awesome query that is fetching millions of datapoints</strong> and only get the result. You need all the data, or just the timestamp of a weird datapoint? The result of the script is simply what’s left on the stack</strong>.</p>
🔗</a>Dataflow language</h4>
WarpScript is really easy to code, because of the stack design</strong>. You’ll be pushing elements into the stack and consume them</strong>. Coding became logical. First you need to fetch</strong> your points, then applying some downsampling</strong> and then aggregate</strong>. These 3 steps are translated into 3 lines of WarpScript</strong>:</p>

FETCH</strong> will push the needed Geo Time Series into the stack</li>
BUCKETIZE</strong> will take the Geo Time Series from the stack, apply some downsampling, and push the result into the stack</li>
REDUCE</strong> will take the Geo Time Series from the stack, aggregate them, and push them back into the stack</li>
</ul>
Debugguing as never be that easy, just use the keyword STOP</strong> to see the stack at any moment.</p>
🔗</a>Rich programming capabilities</h4>
WarpScript is coming with more than 800 functions</strong>, ready to use. Things like Patterns and outliers detections, rolling average, FFT, IDWT</strong> are built-in.</p>
🔗</a>Geo-Fencing capabilities</h4>
Both space</strong> (location) and time</strong> are considered first class citizens</strong>. Complex searches like “find all the sensors active during last Monday in the perimeter delimited by this geo-fencing polygon</strong>” can be done without involving expensive joins between separate time series for the same source.</p>
🔗</a>Unified Language</h4>
WarpScript can be used in batch</strong> mode, or in real-time</strong>, because you need both of them in the real world.</p>
🔗</a>Geez, give me an example</h3>
Here’s an example of a simple but advanced query:</p>
// Fetching all values  
</span>[ $token ‘temperature’ {} NOW 1 h ] FETCH // Get max value for each minute  
</span>[ SWAP bucketizer.max 0 1 m 0 ] BUCKETIZE // Round to nearest long  
</span>[ SWAP mapper.round 0 0 0 ] MAP // reduce the data by keeping the max, grouping by 'buildingID'  
</span>[ SWAP [ 'buildingID' ] reducer.max ] REDUCE
</span></code></pre>
Have you guessed the goal? The result will display the temperature from now to 1 hour of the hottest room per buildingID</strong>.</p>
🔗</a>What about a more complex example?</h3>
You’re still here? Good, let’s have a more complex example. Let’s say that I want to do some patterns recognition. Let’s take an example. Here’s a cosinus with an increasing amplitude:</p>
</p>
I want to detect the green part</strong> of the time series, because I know that my service is crashing when I have that kind of load. With WarpScript, it’s only a 2 functions calls</strong>:</p>

PATTERNS</strong> is generating a list of motifs.</li>
PATTERNDETECTION</strong> is running the list of motifs on all the time series you have.</li>
</ul>
Here’s the code</p>
// defining some variables  
</span>32 'windowSize' STORE  
</span>8 'patternLength' STORE  
</span>16 'quantizationScale' STORE  
</span>
</span>// Generate patterns   
</span>$pattern.to.detect 0 GET   
</span>$windowSize $patternLength $quantizationScale PATTERNS  
</span>VALUES 'patterns' STORE  
</span>
</span>// Running the patterns through a list of GTS (Geo Time Series)  
</span>$list.of.gts $patterns   
</span>$windowSize $patternLength $quantizationScale  PATTERNDETECTION
</span></code></pre>
Here’s the result:</p>
</p>
As you can see, PATTERNDETECTION</strong> is working even with the increasing amplitude! You can discover this example by yourself by using Quantum</a>, the official web-based IDE for WarpScript. You need to switch X-axis scale to Timestamp in order to see the courbe</strong>.Thanks for reading, here’s a nice list of additionnals informations about the time series subject and Warp10:</p>

Metrics Data Platform</a>, our product</li>
Warp10 official documentation</a></li>
Warp10 tour</a>, similar to “The Go Tour”</li>
Presentation of the Warp 10 Time Series Platform at the 42 US school in Fremont</a></li>
Warp10 Google Groups</a></li>
</ul>


Event-driven architecture 101
Pierre Zemb — Fri, 13 May 2016 17:19:23 +0000
update 2019:</strong> this is a repost on my own blog. original article can be read on medium</a>.</p>

</p>
Do your own cover on http://dev.to/rly</a></p>
I’m still a student, so my point of view could be far from reality, be gentle ;)</em></p>
**tl;dr: Queue messaging are cool. Use them at the core of your architecture.</em>**I’m currently playing a lot around Kafka</a> and Flink</a> at work. I also discovered Vert.x</a> at my local JUG. All three have a common word: events</strong>. Event-driven architecture is not something that I learned at school, and I think that’s a shame. It’s really powerful and useful, especially in a world where we speak more and more about “serverless” and “micro services” stuff. So here’s my attempt to make a big sum-up.</p>
🔗</a>the Unix philosophy</h1>
</p>
I’m a huge fan of GNU/Linux. I just love my terminal. It’s been difficult at the beginning, but now, I consider myself fluent with it. My favorite feature ? Pipes or |</strong>. For those who don’t know, it’s the ability to pass the result of the command to another command. For example, to count how many files you have in a folder, you’ll find yourself doing something like this:</p>

list files</strong> in a folder</li>
From this list, manipulate/filter</strong> it. One line must correspond to one file, things like folder are omitted</li>
And then count</strong> the line!</li>
</ul>
In the UNIX world, it should give you something like “ls -l | grep ^- | wc -l”.</em></strong> it might feels like chinese. For me, it’s just feels logical. 3 operations mapped into 3 commands.</strong> You declare a set a commands that, in the end, give you the result. It’s simple and also very fast (in fact, you can find funny articles like this one: Command-line tools can be 235x faster than your Hadoop cluster</a>). This is only possible thanks to the UNIX philosophy</strong>, greatly describe by Doug McIlroy, Elliot Pinson and Berk Tague in 1978:</p>

Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”.> Expect the output of every program to become the input to another, as yet unknown, program.</p>
</blockquote>
Why should I care? It’s 2016, not 1978! Well…</p>
🔗</a>Back in 2016</h1>
</p>
Cloud changed everything in terms of software engineering. We can now deploy applications without thinking about the underlying server</strong>. How cool is that? Let’s take some steps back. Now that you can easily deploy a huge application, what can be accomplished? Well, if I can deploy one app with ease, Why should I deploy only one huge app ?</strong> why can’t I deploy multiples applications instead of one? Let’s call theses applications micro services</strong> because we are in 2016.</p>
</p>
OK, so now I’m applying the first rule of the UNIX Philosophy, because I have multiples programs that are doing one job each. But about the second rule? How can they communicate? How can we simulate UNIX pipes?</strong> Before answering, let’s answer to another question first: What do we really need to send through our network?</strong> Don’t forget the  Fallacies of distributed computing</strong></a>…</strong></p>
Let’s take an example. We are a new startup, and we are building our plateform. We’ll certainly need to handle our customers. Let’s say that for each new customer, we need to make two actions</strong>: add it to our database, and then to our mailing-list. A simple and classical way would be to just call two functions</strong> (whether on the same applications or not), and then say to the customer: “You’re successfully registered”. Like this:</p>
</p>
Classic approach</p>
Is there another approach? Let’s use an event-based architecture</strong>:</p>
🔗</a>Let’s talk events</strong></h1>
Let’s ask Google, what’s an event?</p>

a thing that happens, especially one of importance.</p>
</blockquote>
Well, handling a new customer is a thing that happens (hopefully). For this, we’ll be using a Queue messaging system or Broker</strong>. It’s a middleware</strong> that will receive events, and making them available for another application or groups of applications.</strong></p>
</p>
Queue messaging architecture with 2 producers and 4 consumers</p>
So let’s rethink our architecture. Pay attention to the words: our Register page will produce</strong> an event that will contains all the information about our client. This event will be queued</strong>, waiting to be consumed</strong> by the associated micro services.</p>
</p>
Simple event-driven architecture</p>
We didn’t changed much, but we enable many things over here:</p>

Simplicity</strong>. Remember, the first rule ! “Make each program do one thing well”. Like this, your code base for each app will be simple</strong> as hell</strong>, and you’ll be able to easily replace your software if needed.</li>
Modularity</strong>. You need to add another action to the event, for example CreateProfile ? Easy, just plug another app on the same queue</strong>. You need to test a new version of your program? Easy, just plug it on the same queue</strong>.</li>
Scalability</strong>. One of your micro services is taking too much time? Just start a new instance of it</strong>. Huge traffic? Add new instances. With this approach, you can start really small and become giant.</li>
Big-data friendly.</strong> This type of architecture is often used to handle a lot of data. With plateform like Apache Flink</a>, you can do some stream processing directly</strong>. Look how easy it is</a>.</li>
Polyglotism.</strong> Most messaging system are offering libraries for many languages.Like this, you can use whatever language you want</strong> . But be aware, With great power comes great responsibility</em>.</li>
</ul>
🔗</a>What about serverless?</strong></h1>
Serverless is the “new” buzz word. Ignited by Amazon with their product AWS Lambda</a> and quickly followed by Google</a>, Microsoft</a>, IBM</a> and Iron.io</a>, the goal is to offer to developers a new way of building apps</strong>. Instead of writing apps, you’ll just write a function that will respond to an event</strong>. In fact, you’ll be paying only for the time it’s running. It’s a interesting point-of-view, because you’ll be deploying an architecture built only using events</strong>. I must admit that I didn’t try it yet, but I think it’s a great idea to force developers to split their apps and really think about events,</strong> but you could just build the same thing with any cloud provider.</p>
🔗</a>Additional links and talks about this topic</h1>

Apache Kafka, Samza, and the Unix Philosophy of Distributed Data</a> by Martin Kleppmann</a></li>
Apache Kafka for Beginners</a> by Cloudera Engineering Blog</li>
Introduction to Apache Kafka</a> by Guglielmo Iozza</li>
[Apache Flink Training] (http://dataartisans.github.io/flink-training/)by</a> data-artisans</li>
Meetup LeboncoinTech — AMQP 101 by Quentin ADAM</a> (French sorry)</li>
vert.x 3 — be reactive on the JVM but not only in Java by Clement Escoffier/Paulo Lopes DEVOXX 2015</li>
</ul>
Please, Feel free to react to this article, you can reach me on Twitter</a>, or have a look on my website</a>.</p>


Let’s talk about containers
Pierre Zemb — Mon, 04 Jan 2016 18:52:19 +0000
update 2019:</strong> this is a repost on my own blog. original article can be read on medium</a>.</p>

English is not my first language, so the whole story may have some mistakes… corrections and fixes will be greatly appreciated. I’m also still a student, so my point of view could be far from “production ready”, be gentle ;-)</em></p>
In the last two years, there’s been a technology that became really hype. It was the graal for easy deployments, easy applications management. Let’s talk about containers.</p>
🔗</a>“Write once, run everywhere”</h3>
</p>
When I first heard about containers, I was working as a part-time internship for a french bank as a developer in a Ops team. I was working around Hadoop</a> and monitoring systems, and I was wondering “How should I properly deploy my work?”. It was a java app, running into the official Java version provided by my company. I couldn’t just give it to my colleagues</strong> and leave them do some vaudou stuff because they are the Ops team</strong>. I remembered saying to myself ”fortunately, all the features that I need are in this official java version, I don’t need the latest JRE. I just need to bundle everything into a jar and done”. But what if it wasn’t? What if I had to explain to my colleagues that I need the new JRE for a really small app written by an intern? Or I needed another non-standard library during runtime?</p>
The important thing here at the time was that, at any time, I could deploy it on another server that had Java, because everything is bundled into that big fat jar file</strong>. After all, “write once, run everywhere</strong>” was the slogan created by Sun Microsystems to illustrate the cross-platform benefits of the Java language. That is a real commodity, and this is the first thing that strike me with Docker.</p>
🔗</a>Docker hype</h3>
I will always remember my chat with my colleagues about it. I was like this:</p>
</p>
🔗</a>And they were more like</h2>
</p>
Ops knew about containers since the dawn of time, so why such hype now? I think that “write once, run everywhere” is the true slogan of Docker, because you can run docker containers in any environments that has Docker. You want to try the latest datastore/SaaS app that you found on Hacker News or Reddit? There’s a Dockerfile for that</strong>. And that is super cool. So everyone started to get interested in Docker, myself included. But the real benefit is that many huge companies like Google admits that containers are the way they are deploying apps. They don’t care what type of applications they are deploying or where it’s running, it’s just running somewhere.</strong> That’s all that matters. By unifying the packages, you can automatize and deliver whatever you want somewhere. Do you really care if it’s on a specific machine? No you don’t. That’s a powerful way to think infrastructure more like a bunch of compute or storage power, and not individual machines.</p>
🔗</a>Let’s create a container</h3>
That’s not a secret: I love Go</a>. It’s in my opinion a very nice programming language that you should really try</a>. So let’s say that I’m creating a go app, and then ship it with Docker. So I’ll use the officiel Docker image right? Then I end up with a 700MB container to ship a 10MB app</strong>… I thought that containers were supposed to be small… Why? because it’s based on a full OS, with go compiler and so on. To run a single binary, there’s no need to have the whole Go compiler stack.</p>
That was really bothering me. At this point, if the container is holding everything, why not use a VM? Why do we need to bundle Ubuntu into the container? From a outside point-of-view, running a container in interactive mode is much like a virtual machines right? At the time of writing, Docker’s official image for Ubuntu was pulled more than 36,000,000 time</strong>. That’s huge! And disturbing. Do you really need for example “ls, chmod, chown, sudo” into a container?</p>
There is another huge impact on having a full distribution on a container: Security. You now have to watch not only for CVEs (Common Vulnerabilities and Exposures) on the packages in your host distribution, but also in your container</strong>! After all, based on this presentation</a>, 66.6% of analyzed images on Quay.io are vulnerable to Ghost</a>, and 80% to Heartbleed</a>. That is quite scary… So adding this nightmare doesn’t seems the solution.</p>
🔗</a>So what should I put into my container?</h3>
I looked a lot around the internet, I saw things like docker-alpine</a> or [baseimage-docker] (https://github.com/phusion/baseimage-docker)which</a> are cool, but in fact, the answer was on Docker’s website… Here’s the [official sentence] (https://www.docker.com/what-docker)that</a> explains the difference between containers and virtual machines:</p>

“Containers include the application and all of its dependencies, but share the kernel with other containers.”</p>
</blockquote>
This specific sentence triggers something in my head. When you execute a program on your UNIX system, the system creates a special environment for that program. This environment contains everything needed for the system to run the program as if no other program were running on the system. It’s exactly the same! So a container should be abstract not as a Virtual machines, but as a UNIX process!</strong></p>

application + dependencies represent the image</li>
Runtime environment like token/password will be passed through env vars for example</li>
</ul>
🔗</a>Static compilation</h3>
</p>
Meet Go</p>
Here’s an interesting fact: Go, the open-source programming language pushed by Google supports statically apps</strong>, what a coincidence! That means that this statically app will be directly talking to the kernel. Our Docker image can be empty</strong>, except for the binary and needed files like configuration. There’s a strange image on Docker that you might have seen, which is called “scratch”:</p>

You can use Docker’s reserved, minimal image, scratch, as a starting point for building containers. Using the scratch “image” signals to the build process that you want the next command in the Dockerfile to be the first filesystem layer in your image. While scratch appears in Docker’s repository on the hub, you can’t pull it, run it, or tag any image with the name scratch. Instead, you can refer to it in your Dockerfile.</p>
</blockquote>
That means that our Dockerfile now looks like this:</p>
FROM</span> scratch  
</span>ADD </span>hello /  
</span>CMD [/hello]
</span></code></pre>
So now, I have finally (I think) the right abstraction for a container! We have a container containing only our app</strong>. Can we go even further? The most interesting thing that I learned from (quickly) reading Large-scale cluster management at Google with Borg</em></a> is this:</p>

Borg programs are statically linked to reduce dependencies on their runtime environment, and structured as packages of binaries and data files, whose installation is orchestrated by Borg.</p>
</blockquote>
Here’s the (final) answer! By trully coming back to the UNIX process point-of-view, we can abstract containers as Unix processes. Bu we still need to handle them. So the role of Docker would be more like a Operating System builder</strong> (nice name found by Quentin ADAM</a>).As a conclusion, I think that Docker true success was to show developers that they can sandbox their apps easily, and now it’s our work to build better software, and learning new design patterns.</p>
Please, Feel free to react to this article, you can reach me on Twitter</a>, Or visite my website</a>.</p>