CheckProof Scheduling Tool

02 / Decisions

1. DEC-01 Took the 1.69× speedup. Turned down the 4.4× one.

   Writes took most of each monitor run, so the write path decided how much work the monitor could handle.

   ✕ Turned down

   A raw row writer, measured 4.4–4.9× faster

   ✓ Chose

   Wrote each status change and its history row in one statement, so the two can't come apart. That cut 4 database round trips to 1.

   Cost

   Left the bigger speedup unused. The raw writer needed a second copy of the transition rules, and two copies drift apart. If one worker is not enough, the monitor can run once per status instead.

   Proof

   Measured both writers on the same 1M-task backlog before choosing, then stress-tested the result: zero failures or deadlocks. The decision is on record with the condition that would reopen it.

2. DEC-02 One broken customer can't stall the rest

   A failed task stayed at the front of the queue and was retried on every run. One customer's bad data could hold up maintenance tasks for every other customer.

   ✕ Turned down

   One failure limit for the whole run

   ✓ Chose

   Gave each company a failure budget per run, and made a failed task wait longer after each failure, up to 60 minutes.

   Cost

   A broken backlog is held back, not cleared. It waits until its cause is fixed.

   Proof

   In a test, 1M broken tasks from two companies sat at the front of the queue. The monitor tried 100 from each, set both companies aside, and cleared a third company's 1,000 healthy tasks in 3 seconds of a 50-second run. The old limit stopped the whole run after 100 failures, so those healthy tasks would have waited.

3. DEC-03 Left a known race in. On purpose.

   A user can complete a task while the monitor writes it, and the monitor's older write wins.

   ✕ Turned down

   A status guard on every write

   ✓ Chose

   Kept the write path as it is. The guard would add a cost to every status change to protect a rare one.

   Cost

   A rare overwrite is still possible until production shows it matters.

   Proof

   Characterization tests pin the current behavior, so the race is a decision on record, not a bug waiting to be found.

03 / What I did

1. 01

   Built the Laravel backend of CheckProof's new Scheduling Tool, automating maintenance task scheduling, expiry deviations, and push alerts on a SaaS platform used in 35+ countries.

2. 02

   Built the task lifecycle monitor that moves tasks through their statuses on a schedule, and recorded each design decision with the condition that would reopen it.

3. 03

   Built the cutover from the legacy scheduler: captured its behavior in characterization tests first, then moved customers' live maintenance schedules to the new system with one-switch rollback at any point.

04 / Stack

• PHP
• Laravel
• PostgreSQL
• GraphQL