[{"content":"In my previous post, I wrote about building an executor from scratch and how cooperative async tasks actually interact with OS threads. After getting cooperative multitasking working, the next logical milestone was implementing timers: giving tasks the ability to sleep(duration).await.\nDeceived by how effortless tokio::time::sleep looks from the outside, I challenged myself to build timers for my own runtime, thinking it would be straightforward. My approach was to store deadlines in a BinaryHeap, park a background thread until the earliest deadline at the head of the heap, and wake up the corresponding task as soon as the thread woke up.\n// Naive approach: O(log N) insertion // (Ord / Eq implemented by comparing `deadline`) struct TimerEntry { deadline: Instant, waker: Option\u0026lt;Waker\u0026gt;, } let mut heap: BinaryHeap\u0026lt;Reverse\u0026lt;TimerEntry\u0026gt;\u0026gt; = BinaryHeap::new(); For a small-scale runtime, a BinaryHeap is more than fast enough. Still, getting it working sparked my curiosity: how does a production runtime like Tokio handle millions of concurrent timers under massive throughput?\nI peeked inside Tokio\u0026rsquo;s timer subsystem, and the implementation completely blew my mind. Instead of a heap, what I found looked less like typical Rust and more like dark systems magic: bitwise odometers, raw pointer manipulation, hardware bit rotations, and PhantomPinned.\nThis post serves as a guided tour of Tokio\u0026rsquo;s timer internals, tracing a single sleep(duration).await from registration to wakeup.\nOverall Architecture: The sleep() Lifecycle Before diving into the dark systems magic of bitwise math and raw pointers, here is how all the moving pieces of Tokio\u0026rsquo;s timer subsystem fit together from setup to task wakeup:\nflowchart TD subgraph Setup[\"1. Runtime \u0026 Driver Setup\"] RT[\"Builder::build()\"] --\u003e TD[\"time::Driver::new()\"] TD --\u003e TH[\"time::Handle (Context / Thread Local Storage)\"] TH --\u003e Inner[\"Mutex#lt;InnerState#gt;- Wheel (6 levels x 64 slots)- next_wake: Option#lt;NonZeroU64#gt;\"] end subgraph Ingest[\"2. Timer Ingestion (sleep.await)\"] Sleep[\"tokio::time::sleep(duration).await\"] --\u003e Tick[\"when = Convert Instant to wheel tick(ms since runtime start)\"] Tick --\u003e Locate[\"Locate Level \u0026 Slotlevel = level_for(elapsed, when)slot = slot_for(when, level)\"] Locate --\u003e Slot[\"Insert TimerShared into levels[level].slot[slot]Set occupied bitmask |= (1 #lt;#lt; slot)\"] Slot --\u003e CheckWake{\"when #lt; next_wake?\"} CheckWake -- \"Yes (Earlier deadline)\" --\u003e Unpark[\"IoHandle::unpark()Wake sleeping driver early\"] CheckWake -- \"No\" --\u003e Pending[\"Return Poll::Pending\"] end subgraph Consume[\"3. Driving Time \u0026 Waking (process_at_time)\"] Drive[\"Driver::park_internal() or 61-tick poll\"] --\u003e PollWheel[\"Wheel::poll(now)\"] PollWheel --\u003e NextExp[\"next_expiration()Find earliest occupied slot across levels\"] NextExp --\u003e SlotDue{\"expiration.deadline #lt;= now?\"} SlotDue -- \"Yes\" --\u003e TakeSlot[\"process_expiration()Take entries from expired slot\"] SlotDue -- \"No (Not due yet)\" --\u003e AdvanceNow[\"set_elapsed(now)\"] TakeSlot --\u003e EntryDue{\"entry.true_when() #lt;= deadline?\"} EntryDue -- \"No (Time remaining)\" --\u003e Reinsert[\"Cascade entry down to lower level\"] Reinsert --\u003e NextExp EntryDue -- \"Yes (Expired)\" --\u003e PushPending[\"Move expired entries to Wheel::pending\"] PushPending --\u003e Batch[\"Try pop pending into WakeList(local array buffer up to 32)\"] AdvanceNow --\u003e Batch Batch --\u003e DropLock[\"Drop Mutex#lt;InnerState#gt;\"] DropLock --\u003e WakeAll[\"WakeList::wake_all()Push tasks to scheduler run queue\"] end Setup --\u003e Ingest Ingest --\u003e Consume Component Breakdown (Ordered by Lifecycle) You can read straight through to follow a timer\u0026rsquo;s journey, or jump directly to any component below:\nThe Core Data Structure: The 6-Level Timer Wheel: Instead of sorting deadlines in a BinaryHeap, Tokio buckets timers into 6 circular arrays (levels) of 64 slots each, achieving O(1) insertion and O(1) cancellation across an unbounded time horizon (using Level 5 as a 2.17-year pseudo-ring buffer). Ingesting a Timer: The Bitwise Odometer: When sleep(duration).await is polled, Tokio converts the deadline into a millisecond tick, uses an XOR odometer to find the level in 4 CPU instructions (level_for), and shifts the bits to index the exact slot (slot_for). Storing and Resetting Timers: Intrusive Linked Lists: To eliminate memory allocations on the heap every time a timer is inserted or moved, each Sleep future embeds its own prev/next pointers (PhantomPinned) and tracks registered_when vs true_when() for O(1) resets. Finding the Next Deadline: Hardware Bit Rotation: Each level tracks non-empty slots inside a single u64 bitmask (occupied), using rotate_right and trailing_zeros() to locate the next due timer in O(1) time without looping. Who Turns the Wheel? TryLock\u0026lt;Driver\u0026gt;: Tokio has no dedicated timer thread; instead worker threads take turns acquiring a non-blocking TryLock\u0026lt;Driver\u0026gt; so at most one worker sleeps on the OS poller or advances the wheel at a time. Preempting a Sleeping Driver: next_wake and unpark: Tokio caches the currently parked wakeup tick inside next_wake. If another worker inserts a timer that expires earlier than next_wake, it calls unpark.unpark() (mio::Waker) to interrupt the sleeping driver holder immediately. Waking Tasks Safely: WakeList and DropGuard: To avoid deadlocks, Tokio never wakes tasks while holding the wheel lock. Instead, it buffers up to 32 expired Wakers in a local array (WakeList), drops the mutex, and wakes them in panic-safe FIFO order without heap allocations. The Core Data Structure: The 6-Level Timer Wheel In a high-throughput network server, almost every incoming request sets a timeout that gets cancelled milliseconds later when a response arrives. While my naive BinaryHeap approach requires O(log N) to insert and cannot cheaply remove cancelled timers from the middle (O(N)), Tokio achieves both insertion and cancellation in O(1) time.\nIt does this using a Hierarchical Timing Wheel, an algorithm introduced in a 1987 paper by George Varghese and Tony Lauck titled \u0026ldquo;Hashed and Hierarchical Timing Wheels: Data Structures for the Efficient Implementation of a Timer Facility\u0026rdquo;.\nThe idea is modeled after a mechanical clock:\nInstead of keeping all deadlines in a sorted queue, you organize time into circular arrays (wheels) divided into fixed \u0026ldquo;slots\u0026rdquo;. Just like a clock has a second hand, a minute hand, and an hour hand, a hierarchical timer wheel has multiple levels: Level 0 (The second hand): Moves rapidly, handling near-term deadlines with 1ms resolution. Higher Levels (The minute/hour hands): Move slowly, handling distant deadlines with coarser resolution. As time advances, timers on higher levels \u0026ldquo;cascade\u0026rdquo; down into lower levels until they finally land in Level 0 and fire. This same architecture powers both the Linux kernel\u0026rsquo;s timer wheel and Apache Kafka\u0026rsquo;s Request Purgatory.\nTokio\u0026rsquo;s 6-Level Clock Gearbox In Tokio\u0026rsquo;s timer wheel, time is structured into 6 levels, with each level containing 64 slots (2^6):\nLevel 0: 64 slots × 1 ms = 64 ms total span Level 1: 64 slots × 64 ms ≈ 4.1 seconds total span Level 2: 64 slots × 4.1 sec ≈ 4.3 minutes total span Level 3: 64 slots × 4.3 min ≈ 4.6 hours total span Level 4: 64 slots × 4.6 hr ≈ 12.4 days total span Level 5: 64 slots × 12.4 day ≈ 2.17 years total span The time represented by a single slot at level N is:\nslot_range(N) = 64^N milliseconds Notice that a single full rotation of Level 5 spans 2.17 years (MAX_DURATION = 1 \u0026lt;\u0026lt; 36 ms, or roughly 795 days).\nDoes this mean Tokio cannot handle timers beyond two years? Not at all. Timers in Tokio are unbounded (up to u64::MAX milliseconds, or roughly 584 million years). Instead of adding infinite levels to the wheel, Tokio treats Level 5 as a pseudo-ring buffer:\nAny timer scheduled further than 2.17 years into the future (exceeding one full Level 5 rotation) is placed directly into Level 5. When Level 5 completes a 2.17-year revolution and visits that slot, process_expiration checks the timer\u0026rsquo;s exact target deadline. If the deadline is still years away, Tokio sees the timer has not expired yet and reinserts it right back into Level 5 for another lap. Once the remaining duration finally falls within 2.17 years of current time, the timer cascades down to lower levels just like any other. By recycling Level 5 as an unbounded circular buffer, Tokio supports arbitrarily distant deadlines while keeping the entire wheel fixed at 6 levels of 64 slots (384 slots, or roughly 6 KB total), which is compact enough to live permanently inside the CPU\u0026rsquo;s L1 cache.\nIngesting a Timer: The Bitwise Odometer When a task calls tokio::time::sleep(duration).await, Tokio converts its Instant deadline into a u64 tick (when, measured in milliseconds since the runtime started) and compares it against the wheel\u0026rsquo;s current tick (elapsed).\nNow Tokio must answer: which of the 6 levels should hold this timer?\nThe naive approach to determine the level is to check whether elapsed and when fall within the same wheel boundary using an if/else chain:\n// Naive approach: chained divisions and branches fn level_for_naive(elapsed: u64, when: u64) -\u0026gt; usize { if (elapsed / 64) == (when / 64) { 0 // Same 64ms epoch } else if (elapsed / 4_096) == (when / 4_096) { 1 // Same 4.1-second epoch (64^2) } else if (elapsed / 262_144) == (when / 262_144) { 2 // Same 4.3-minute epoch (64^3) } else if (elapsed / 16_777_216) == (when / 16_777_216) { 3 // Same 4.6-hour epoch (64^4) } else if (elapsed / 1_073_741_824) == (when / 1_073_741_824) { 4 // Same 12.4-day epoch (64^5) } else { 5 } } In a high-throughput runtime, executing repeated 64-bit integer divisions and unpredictable if/else branches on every timer registration is expensive. Because each level is a power of 64 (2^6), Tokio replaces the entire if/else ladder by treating time as a binary odometer:\nfn level_for(elapsed: u64, when: u64) -\u0026gt; usize { const SLOT_MASK: u64 = (1 \u0026lt;\u0026lt; 6) - 1; let masked = elapsed ^ when | SLOT_MASK; if masked \u0026gt;= MAX_DURATION { // Fudge distant timers into the top level pseudo-ring buffer return NUM_LEVELS - 1; } masked.ilog2() as usize / 6 } Three simple bitwise steps make this work in roughly 4 CPU instructions:\nA. The XOR (^) Odometer Trick On a 4-dial car odometer, when the mileage ticks from 0123 to 0128, only the rightmost dial (3 → 8, Level 0) moves. When it rolls over from 0129 to 0130, the second dial (2 → 3, Level 1) clicks forward.\nIn binary, each 6-bit chunk represents a level (2^6 = 64):\nBit position: [63 .. 36] [35 .. 30] [29 .. 24] [23 .. 18] [17 .. 12] [11 .. 6] [5 .. 0] ────────── ────────── ────────── ────────── ────────── ───────── ──────── Target level: Level 5* Level 5 Level 4 Level 3 Level 2 Level 1 Level 0 (*clamped by masked \u0026gt;= MAX_DURATION) The XOR operator (elapsed ^ when) outputs a 0 for every bit that hasn\u0026rsquo;t changed and a 1 for every bit that has. Taking the base-2 logarithm via masked.ilog2() finds the bit position of the highest 1 bit, and dividing by 6 maps that bit directly to the highest dial on the odometer that differs between elapsed and when.\nB. The SLOT_MASK Safety Floor Level 0 handles differences between 0 and 63 milliseconds (the bottom 6 bits). By OR-ing with SLOT_MASK (0b111111), Tokio forces the lowest 6 bits to always be 1.\nWhy? If a timer is scheduled for the exact current millisecond (when == elapsed), elapsed ^ when is 0. Calculating 0.ilog2() would panic. SLOT_MASK sets a hard floor so masked is always at least 63, keeping same-tick timers safely at Level 0.\nC. The MAX_DURATION Ring-Buffer Cap If the time difference exceeds 36 bits (masked \u0026gt;= MAX_DURATION, or roughly 2.17 years), Tokio avoids computing the logarithm and immediately returns NUM_LEVELS - 1 (Level 5), routing distant timers straight into the top-level pseudo-ring buffer.\nD. Reading the Dial: slot_for(when, level) Once level_for identifies which 6-bit gear on the odometer changed, Tokio still needs to know which of the 64 slots (0..63) on that gear to drop the timer into.\nThink about how you would isolate a specific digit on a regular base-10 odometer, such as extracting the hundreds digit (3) from 1,357. First, you divide by 100 (10^2) to chop off the last two digits (1357 / 100 = 13). Then, you take the remainder after dividing by 10 (13 % 10 = 3).\nIn binary, shifting bits to the right (\u0026gt;\u0026gt;) is the exact same thing as integer division by a power of two: shifting right by 6 bits divides by 2^6 (64), shifting right by 12 bits divides by 2^12 (4,096), and in general shifting right by level * 6 bits divides when by 2^(level * 6) (that level\u0026rsquo;s slot width) to chop off all the lower odometer dials. Taking % 64 then grabs the remainder (0..63), giving us the exact slot on that dial:\nfn slot_for(duration: u64, level: usize) -\u0026gt; usize { ((duration \u0026gt;\u0026gt; (level * 6)) % 64) as usize } Storing and Resetting Timers: Intrusive Linked Lists Once level_for and slot_for pick the target level and slot index idx, Tokio needs a way to store the timer inside levels[level].slot[idx]. Because each level only has 64 slots, many timers will inevitably land on the same slot, so Tokio chains multiple timers together in a linked list under each slot.\nIn standard Rust collections, pushing to a LinkedList\u0026lt;T\u0026gt; allocates a heap Node containing prev, next, and value. Allocating and freeing millions of wrapper nodes as timeouts are created and cancelled would thrash the memory allocator.\nTokio avoids heap allocation on insertion by using an intrusive linked list:\npub(crate) struct TimerShared { pointers: linked_list::Pointers\u0026lt;TimerShared\u0026gt;, // Intrusive prev/next pointers stored inline registered_when: AtomicU64, // Tick where this node is currently slotted in the wheel state: StateCell, // Holds the true target deadline (retrieved via true_when()) and Waker _p: PhantomPinned, } impl TimerShared { pub(super) fn true_when(\u0026amp;self) -\u0026gt; u64 { self.state.when().expect(\u0026#34;Timer already fired\u0026#34;) } } Instead of wrapping TimerShared in a separate node, the prev and next pointers live inside TimerShared itself. Inserting or cancelling a timer simply rewires raw NonNull\u0026lt;TimerShared\u0026gt; pointers in O(1) time with zero heap allocations.\nWhy TimerShared Needs PhantomPinned Intrusive linked lists break two assumptions that Rust normally makes about structs:\nStructs can be moved in memory: In Rust, moving a value copies its bytes to a new memory address. If a TimerShared node moved while linked into a wheel slot, the prev and next pointers on neighboring timers would instantly become dangling pointers. Mutable references are exclusive (noalias): Normally, when a task polls a future through a mutable reference, Rust tells its code-generation backend (LLVM) that no other pointer in the program can access that struct at the same time. In an intrusive list, that isn\u0026rsquo;t true, because the timer wheel holds raw prev and next pointers to TimerShared at the exact same time. If the compiler assumes exclusive access, it can reorder or optimize away pointer updates and silently corrupt the list. Adding _p: PhantomPinned marks TimerShared as !Unpin, solving both problems at once: it locks the struct to a fixed memory address once pinned, and it turns off the exclusive-aliasing (noalias) optimization so every pointer read and write is emitted faithfully.\nWhy Timers Track Two Deadlines: registered_when vs true_when() Looking at TimerShared above, you might wonder why Tokio stores a registered_when timestamp alongside the actual target deadline inside state: StateCell (retrieved via true_when()).\nThe reason comes down to the difference between coarse slot buckets and exact expiration times:\nCoarse Wheel Buckets (true_when()): Higher-level slots represent coarse time ranges rather than exact milliseconds. For example, a timer for tick 95ms sits in Level 1\u0026rsquo;s [64ms..128ms) bucket. When that bucket expires at tick 64ms, the slot only knows its base timestamp (64ms), so the timer must remember its exact target inside StateCell (true_when() == 95ms) to know it still needs to cascade down rather than fire immediately.\nO(1) Resets and Cancellations (registered_when): If a task calls sleep.reset(new_deadline) or cancels the timer, Tokio may need to unlink the node from its current slot. Updating StateCell overwrites the target deadline, so Tokio preserves registered_when (the timestamp used when the node was placed into the wheel) to locate and unlink the node from its old slot in O(1) time before re-inserting it.\nFinding the Next Deadline: Hardware Bit Rotation Once timers are sitting in the wheel, the driver needs to know how long it can go to sleep. To compute that timeout, each level must answer: \u0026ldquo;Starting from now, what is the next slot that actually contains a timer?\u0026rdquo;\nBecause each level is a circular ring of 64 slots, a naive search would loop from now_slot to 63 and wrap around from 0 to now_slot. Instead, Tokio packs each level\u0026rsquo;s occupancy into a single u64 bitmask (self.occupied, where Bit i = Slot i) and finds the next occupied slot in O(1) without loops:\nfn next_occupied_slot(\u0026amp;self, now: u64) -\u0026gt; Option\u0026lt;usize\u0026gt; { if self.occupied == 0 { return None; } let now_slot: usize = (now / slot_range(self.level)) as usize; let occupied: u64 = self.occupied.rotate_right(now_slot as u32); let zeros: usize = occupied.trailing_zeros() as usize; let slot: usize = (zeros + now_slot) % LEVEL_MULT; Some(slot) } rotate_right(now_slot): Rotates the bitmask so our current position (now_slot) becomes the starting line at Bit 0. All upcoming slots now line up in clockwise order after it, with lower-indexed slots (0..now_slot) wrapping around to the upper bits. trailing_zeros(): Uses a single hardware instruction (tzcnt) to count how many zeros sit at the bottom of the register. This directly tells us how many slots forward we need to travel to reach the next timer. (zeros + now_slot) % 64: Adds that distance to now_slot to get the final slot index. To visualize this, imagine an 8-slot wheel where our clock hand is at Slot 5, and the only active timer is at Slot 1:\nBefore rotation (self.occupied = 0b0000_0010): Bit position: Bit 7 Bit 6 Bit 5 Bit 4 Bit 3 Bit 2 Bit 1 Bit 0 Slot index: Slot 7 Slot 6 Slot 5 Slot 4 Slot 3 Slot 2 Slot 1 Slot 0 Value: 0 0 0 0 0 0 1 0 [ NOW ] [TIMER] 1. rotate_right(5) shifts bits right by 5 (bringing Slot 5 to Bit 0 on the right): Bit position: Bit 7 Bit 6 Bit 5 Bit 4 Bit 3 Bit 2 Bit 1 Bit 0 Mapped slot: Slot 4 Slot 3 Slot 2 Slot 1 Slot 0 Slot 7 Slot 6 Slot 5 Value: 0 0 0 1 0 0 0 0 [TIMER] \u0026lt;--- 4 trailing zeros -\u0026gt; [ NOW ] 2. trailing_zeros(): 4 zeros on the right -\u0026gt; the timer is 4 steps ahead. 3. (5 + 4) % 8: 5 + 4 = 9 -\u0026gt; 9 % 8 = Slot 1. Even though Slot 1 sat \u0026ldquo;behind\u0026rdquo; Slot 5 in array order, rotating the clock face so \u0026ldquo;now\u0026rdquo; sits at index 0 allows hardware trailing-zero detection to resolve circular wrap-around in a single instruction.\nWho Turns the Wheel? TryLock\u0026lt;Driver\u0026gt; Unlike my naive mini-rt timer, Tokio does not spawn a dedicated background timer thread, and there is no permanent \u0026ldquo;main thread\u0026rdquo; in its multi-threaded scheduler. All worker threads are equal peers sharing the timer wheel behind Mutex\u0026lt;InnerState\u0026gt;.\nSo who actually calls Wheel::poll(now) to advance time and cascade timers? Tokio splits this across two regimes:\nRegime A: When Workers Are Idle (TryLock\u0026lt;Driver\u0026gt;) In Tokio\u0026rsquo;s multi-threaded scheduler, the I/O and timer driver is guarded by a non-blocking TryLock\u0026lt;Driver\u0026gt;:\nThe Winner (The Temporary Driver Thread): The first idle worker thread to park acquires shared.driver.try_lock(). It queries wheel.next_expiration() and blocks inside the OS poller (epoll_wait / kevent) for that duration. The Losers (Condvar Workers): Any other workers that run out of tasks fail try_lock() and sleep on a standard std::sync::Condvar, waiting only for new tasks to be spawned. Because TryLock\u0026lt;Driver\u0026gt; is exclusive, at most one worker thread in the entire runtime is ever sleeping on the timer wheel deadline at any time.\nRegime B: When Workers Are Busy (Every 61 Ticks) What if all worker threads are 100% busy polling CPU-heavy tasks and never park? To keep timers and I/O from starving, every worker increments a tick counter on each scheduler step. Every 61 ticks (event_interval = 61), a busy worker tries to lock the driver and calls driver.park_timeout(Duration::ZERO), performing a non-blocking poll of the OS selector and advancing the timer wheel up to now before immediately resuming task execution.\nCascading Timers When the driver turns the wheel via Wheel::poll(now), it calls next_expiration() to find the earliest occupied slot across all levels (Expiration { level, slot, deadline }). Once that slot\u0026rsquo;s deadline is reached (expiration.deadline \u0026lt;= now), process_expiration() drains all entries from the slot and checks each timer\u0026rsquo;s exact target deadline:\nIf entry.true_when() \u0026lt;= deadline: The timer is due now, so Tokio marks it as pending (item.mark_pending(deadline)) and pushes it into self.pending to be fired. If entry.true_when() \u0026gt; deadline: The timer is not due yet (for example, a timer for tick 95ms sitting in Level 1\u0026rsquo;s [64ms..128ms) slot when tick 64ms arrives). Tokio recomputes level_for(deadline, entry.true_when()) and cascades the entry down into a lower, finer-grained level. Preempting a Sleeping Driver: next_wake and unpark When the driver-holding thread parks in the OS poller (epoll_wait / kevent) for the next timer deadline, another active worker thread might register a shorter timer while it is asleep. For example, if Worker 1 blocks on a 100ms deadline and 2ms later Worker 2 registers a 10ms timer, Worker 2 must interrupt Worker 1\u0026rsquo;s sleep so the 10ms timer does not fire 90ms late:\nsequenceDiagram autonumber participant W1 as Worker 1 (Driver Holder) participant D as OS Poller (epoll / kqueue) participant W2 as Worker 2 (Active Task) participant T as Timer Wheel \u0026 next_wake Note over W1,D: Worker 1 is idle and acquires driver.try_lock() W1-\u003e\u003eT: next_expiration() -\u003e 100ms Note over T: Stores next_wake = Some(100) W1-\u003e\u003eD: Blocks in epoll_wait(timeout = 100ms) Note over W2: Worker 2 spawns sleep(10ms) W2-\u003e\u003eT: Inserts 10ms timer into Wheel Note over T: 10ms \u003c next_wake (100ms) W2-\u003e\u003eD: Calls unpark.unpark() (mio::Waker) D--\u003e\u003eW1: Interrupts epoll_wait immediately W1-\u003e\u003eT: Re-polls wheel \u0026 checks next_expiration() -\u003e 10ms Note over T: Updates next_wake = Some(10) W1-\u003e\u003eD: Re-enters epoll_wait(timeout = 10ms) Worker 1 caches its wakeup tick inside lock.next_wake before blocking in the OS poller. Whenever another worker inserts a timer into the wheel, it compares the new timer\u0026rsquo;s tick when against lock.next_wake:\nmatch unsafe { lock.wheel.insert(entry) } { Ok(when) =\u0026gt; { if lock .next_wake .is_none_or(|next_wake| when \u0026lt; next_wake.get()) { unpark.unpark(); } None } // ... } Only when when \u0026lt; next_wake (or when next_wake is None, meaning the driver parked indefinitely with an empty wheel) does Worker 2 call unpark.unpark() (mio::Waker, backed by eventfd on Linux and EVFILT_USER on macOS) to wake the sleeping driver thread. If when \u0026gt;= next_wake, Worker 2 skips the syscall entirely because Worker 1 is already scheduled to wake up earlier.\nWaking Tasks Safely: WakeList and DropGuard Once expired timers are popped from Wheel::poll(now), Tokio extracts their Wakers and wakes the sleeping tasks. However, calling waker.wake() while holding Mutex\u0026lt;InnerState\u0026gt; is dangerous: a custom Waker could immediately register or cancel another timer on the same thread, deadlocking on InnerState.\nTo wake tasks outside the mutex with zero heap allocations, Tokio collects wakers into a local buffer called WakeList:\nconst NUM_WAKERS: usize = 32; pub(crate) struct WakeList { inner: [MaybeUninit\u0026lt;Waker\u0026gt;; NUM_WAKERS], curr: usize, } Whenever waker_list fills up (32 wakers), the driver temporarily drops the mutex, fires the batch, and re-acquires the lock to continue polling the wheel:\nlet mut waker_list = WakeList::new(); let mut lock = self.inner.lock(); while let Some(entry) = lock.wheel.poll(now) { if let Some(waker) = unsafe { entry.fire(Ok(())) } { waker_list.push(waker); if !waker_list.can_push() { // Wake a batch of wakers with the lock temporarily dropped to avoid deadlock. drop(lock); waker_list.wake_all(); lock = self.inner.lock(); } } } drop(lock); waker_list.wake_all(); Why wake_all() Uses Raw Pointers and DropGuard Inside WakeList::wake_all(), instead of a simple for i in 0..self.curr loop, Tokio uses a raw-pointer DropGuard:\npub(crate) fn wake_all(\u0026amp;mut self) { struct DropGuard { start: *mut Waker, end: *mut Waker, } impl Drop for DropGuard { fn drop(\u0026amp;mut self) { let len = unsafe { self.end.offset_from(self.start) } as usize; let slice = ptr::slice_from_raw_parts_mut(self.start, len); unsafe { ptr::drop_in_place(slice) }; } } let mut guard = { let start = self.inner.as_mut_ptr().cast::\u0026lt;Waker\u0026gt;(); let end = unsafe { start.add(self.curr) }; // Transfer drop responsibility of the wakers to DropGuard self.curr = 0; DropGuard { start, end } }; while !ptr::eq(guard.start, guard.end) { let waker = unsafe { ptr::read(guard.start) }; guard.start = unsafe { guard.start.add(1) }; // If this panics, `guard` will clean up the remaining wakers waker.wake(); } } At first glance, DropGuard looks like overkill for iterating over a 32-element array, until we trace what happens if a waker panics:\nWhy a naive for loop causes a double-free: waker.wake() consumes the Waker by value and can panic. If a naive for i in 0..self.curr loop panicked at index 2 (out of 5), stack unwinding would run WakeList::drop on 0..self.curr, causing a double-free on indices 0 and 1 after .wake() already consumed them. Why not just pop from the back (LIFO vs. FIFO fairness): We could avoid that double-free with a single counter by popping from the back (self.curr -= 1 before each .wake()), since 0..self.curr would always track the remaining unconsumed items. However, LIFO order is unfair because the earliest-expiring timer would be woken last, so we need to consume the array from front to back (FIFO). DropGuard gives us FIFO wake order without risking a double-free by transferring the drop obligation out of WakeList. Setting self.curr = 0 upfront resets WakeList for reuse and ensures WakeList::drop does nothing if a panic unwinds the stack. Meanwhile, guard.start advances from front to back before each waker.wake() call so DropGuard::drop only ever cleans up the remaining unconsumed slice [guard.start..guard.end].\nConclusion Tracing tokio::time::sleep from end to end turned what initially looked like cryptic bit-twiddling into a cohesive engineering story:\nA 6-level × 64-slot hierarchical wheel gives O(1) timer insertion and cancellation across an unbounded time horizon by turning Level 5 into a pseudo-ring buffer and cascading coarse buckets down to 1ms precision as deadlines approach. Bitwise odometers (elapsed ^ when | SLOT_MASK) and bit rotations (rotate_right + tzcnt) replace loops and branches with constant-time bit math. Intrusive linked lists (TimerShared) and fixed-capacity wake buffers (WakeList and DropGuard) eliminate heap allocations on the hot path while keeping wakeups FIFO-fair and panic-safe. Cooperative driver locking (TryLock\u0026lt;Driver\u0026gt;, next_wake, and unpark) keeps timers accurate across worker threads without spawning a dedicated timer thread. Unpacking how Tokio\u0026rsquo;s timer wheel works under the hood got me excited to try building a simplified version of it myself. My next experiment is to replace the naive BinaryHeap timer in mini-rt with a hierarchical timer wheel of my own.\nReferences Varghese, G., \u0026amp; Lauck, T. (1987): \u0026ldquo;Hashed and Hierarchical Timing Wheels: Data Structures for the Efficient Implementation of a Timer Facility\u0026rdquo;, Proceedings of the Eleventh ACM Symposium on Operating Systems Principles (SOSP \u0026lsquo;87), 25-38 (DOI: 10.1145/41457.37504). Linux Kernel: The hierarchical timer wheel subsystem for low-resolution kernel timeouts. Apache Kafka: The hierarchical timing wheel implementation in Request Purgatory. Tokio Runtime Source: The time driver, hierarchical wheel, and WakeList modules in the Tokio repository. ","permalink":"https://tayts.me/posts/demystifying-tokio-timer-wheel/","summary":"A deep-dive learning journal tracing a single tokio::time::sleep() from registration to wakeup, and how Tokio achieves O(1) timer management with bitwise math, intrusive lists, and cooperative drivers.","title":"Demystifying Tokio's Timer Wheel: How Async Rust Keeps Time at Scale"},{"content":"When I first started writing async Rust, I was honestly pretty overwhelmed. Coming from languages where async just works out of the box, I wasn\u0026rsquo;t used to the \u0026ldquo;batteries-not-included\u0026rdquo; approach. Instead of a built-in runtime, Rust just hands you a Future trait and tells you to go pick a external runtime yourself, such as tokio, async-std, or smol.\nI defaulted to tokio for my past projects, but it always felt like a black box. I knew how to use it, but I never really looked under the hood. Recently, I was toying with a custom task scheduler project and struggling with fairness algorithms. Hitting this roadblock made me wonder how production-grade systems handle task scheduling. Knowing that Rust is famous for its performance, I started wondering how its async runtimes manage and schedule the tasks across a limited number of OS threads. I figured that diving into those mechanics might spark some new ideas for my own scheduler.\nTo strip away the magic and learn how it\u0026rsquo;s done, I decided to build my own minimal runtime from scratch. The initial goal was simple: get block_on and spawn working, build a basic RunQueue, and get two tasks to cooperatively ping-pong back and forth.\nAfter reading the Rust Async Book and skimming tokio source code, I naively thought I had Wakers and threads figured out. My first implementation quickly proved me wrong.\nEveryone\u0026rsquo;s learning path is different, but hopefully, walking through my two biggest mistakes can help you understand async Rust a little better. Here is what I got wrong about Wakers, and how I fixed it.\n1. Cooperative Async Requires Actual Cooperation To test my executor, I spawned two futures: one to print \u0026ldquo;ping\u0026rdquo; and one to print \u0026ldquo;pong\u0026rdquo;, sharing a Mutex\u0026lt;Option\u0026lt;String\u0026gt;\u0026gt;. My expectation was that they would take turns automatically. Instead, the first task printed \u0026ldquo;ping\u0026rdquo; exactly once, and then the entire program froze permanently.\nI had written a standard while count \u0026lt; 5 loop inside the async block:\n// Naive approach: assuming the executor will pause this loop let fut1 = spawn(async move { let mut count = 0; while count \u0026lt; 5 { let mut guard = shared_state.lock().unwrap(); if guard.as_deref() == Some(\u0026#34;pong\u0026#34;) || guard.is_none() { *guard = Some(\u0026#34;ping\u0026#34;.to_string()); count += 1; println!(\u0026#34;ping\u0026#34;); } // BUG: No .await point here! // The loop spins synchronously forever, starving the executor. } }); I was still thinking in terms of OS threads (preemptive multitasking), assuming the executor would magically pause my loop to let the other task run.\nRealization: Async Rust is cooperative. The executor cannot force a task to pause. A task will hold the CPU hostage and run synchronously until it hits an .await point that explicitly returns Poll::Pending. Because my while loop had no .await points, the first task refused to cooperate. It starved the executor, spinning infinitely in a deadlock because it never yielded the CPU to let the other task update the shared state.\nTo fix this without busy-spinning, I used std::future::poll_fn (a helper to yield Poll) to check if it was the task\u0026rsquo;s turn. If it wasn\u0026rsquo;t, the task would cooperatively save its Waker, return Poll::Pending, and hand the CPU back to the executor.\nuse std::future::poll_fn; use std::task::Poll; // Cooperative approach: yielding control while waiting let fut1 = spawn(async move { let mut count = 0; while count \u0026lt; 5 { poll_fn(|cx| { let mut guard = shared_state.lock().unwrap(); if guard.data.as_deref() == Some(\u0026#34;pong\u0026#34;) || guard.data.is_none() { // It is my turn! Take the turn. guard.data = Some(\u0026#34;ping\u0026#34;.to_string()); // Wake the other task using the waker stored by other \u0026#34;pong\u0026#34; task. if let Some(waker) = guard.waiting_task.take() { waker.wake(); } Poll::Ready(()) } else { // It is NOT my turn. // Save my waker and cooperatively yield the CPU. guard.waiting_task = Some(cx.waker().clone()); Poll::Pending } }).await; // \u0026lt;-- The executor takes back control here! count += 1; println!(\u0026#34;ping\u0026#34;); } }); 2. Wakers Don\u0026rsquo;t Automatically Know Who to Wake Once my tasks could cooperatively go to sleep, I needed to test waking them up from a background OS thread. I implemented my Waker\u0026rsquo;s wake_by_ref function like this:\nunsafe fn wake_by_ref(data: *const ()) { let arc_data = unsafe { Arc::from_raw(data as *const Task) }; let cloned = arc_data.clone(); // Push the task back to the run queue arc_data.task_sender.push(cloned); // BUG: unpark() wakes the thread that called it! std::thread::current().unpark(); std::mem::forget(arc_data); } My background thread successfully grabbed the Waker, called .wake_by_ref(), the task was pushed to the queue\u0026hellip; and then the executor timed out and died.\nRealization: std::thread::current() returns a handle to the thread currently executing the code. Because the background thread is the one calling .wake_by_ref(), it ended up unparking itself instead of the main thread. Meanwhile, the main thread running my block_on loop stayed asleep in std::thread::park() forever.\nWakers are just passive data structs. They don\u0026rsquo;t magically know which thread they belong to. To fix this, the RunQueue must explicitly capture the main thread\u0026rsquo;s handle upon creation, store it, and automatically ring that specific \u0026ldquo;phone number\u0026rdquo; whenever a new task is pushed.\nstruct RunQueue { queue: Arc\u0026lt;Mutex\u0026lt;VecDeque\u0026lt;Arc\u0026lt;Task\u0026gt;\u0026gt;\u0026gt;\u0026gt;, // Store the specific thread handle of the executor executor_thread: std::thread::Thread, } impl RunQueue { fn push(\u0026amp;self, task: Arc\u0026lt;Task\u0026gt;) { if let Ok(mut queue) = self.queue.lock() { queue.push_back(task); } // Fix: The queue rings the specific phone number // of the executor thread whenever a new task arrives! self.executor_thread.unpark(); } } Conclusion Building mini-rt taught me that \u0026ldquo;single-threaded executor\u0026rdquo; is a bit of a misnomer. The polling happens on a single thread, but the wake-ups come from across the OS. Understanding how those background threads safely throw wake-up signals across the boundary is the true secret of async Rust.\nIf you want to poke around the actual code for this first phase, you can find it in the mini-rt repository (Tag: M1).\n","permalink":"https://tayts.me/posts/what-i-got-wrong-about-wakers/","summary":"Building a custom Rust async runtime from scratch reveals why preemptive mindsets break cooperative executors and how Wakers actually target the correct OS thread.","title":"What I Got Wrong About Wakers: Building an Async Executor in Rust"},{"content":"I recently needed to build a low volume internal workflow: a user uploads a CSV, a backend processes it asynchronously, and the user downloads the results.\nThe standard approach job processing stack would be a database for task metadata, a queue for dispatch, and worker deployments. But for an occasional internal tool, that\u0026rsquo;s often more infrastructure than the problem deserves.\nSince the workflow was entirely file-oriented (large input and output CSVs), S3 was already the natural fit.\nSo the question became: could S3 also hold the task state and handle concurrent workers safely for a workflow that only runs occasionally?\n🥁 \u0026hellip; Yes, while everyone knows S3 is great for storing files, what makes this pattern interesting is how far you can push it. Using a tiny metadata file status.json as the task record, S3 conditional writes for safe concurrency, and an inverted ULID for newest-first listing, you can get a lightweight task queue for free.\nDisclaimer: this isn\u0026rsquo;t \u0026ldquo;S3 as a replacement for queue\u0026rdquo;. It\u0026rsquo;s a highly targeted pattern for low-volume, file-oriented workflows where the main goal is keeping your infrastructure footprint minimal while keeping the workflow durable and recoverable.\nWhen Does This Pattern Make Sense? \u0026ldquo;Low volume\u0026rdquo; is subjective, so let\u0026rsquo;s put numbers on it. This design is a great fit when:\nYou process roughly 1 to 100 tasks per day. Seconds-level latency for task pickup is acceptable. You only need to list the latest tasks (no complex querying, filtering or reporting). If your workload requires high throughput, complex filtering, or strict ordering, you absolutely should reach for a database and queue.\nBut for small, file-oriented workflows, the standard infrastructure overhead (database schemas, queue configuration, worker deployments, and monitoring runbooks) often becomes larger than the feature itself.\nHow It Fits Together Before the details, here is the whole system in one view. It is four moving parts on top of S3:\nArtifacts live in a per-task prefix - the input and output files. State lives in a tiny status.json next to them - the task record. Workers find work by listing a small set of marker objects, then claim tasks using S3 conditional writes (compare and set - CAS). Ordering comes for free from the task ID (inverted ULID), so the newest tasks list first. sequenceDiagram actor C as Client participant API as Control API participant S3 participant W as Worker Note over C,S3: Submit C-\u003e\u003eAPI: POST /tasks API-\u003e\u003eS3: write status.json (created) API--\u003e\u003eC: presigned upload URL C-\u003e\u003eS3: upload input.csv C-\u003e\u003eAPI: POST /tasks/{id}/submit API-\u003e\u003eS3: status.json (queued) + active marker Note over W,S3: Process W-\u003e\u003eS3: list active/ W-\u003e\u003eS3: claim via If-Match ETag W-\u003e\u003eS3: write output + status.json (completed) Note over C,S3: Download C-\u003e\u003eAPI: GET /tasks/{id}/output API--\u003e\u003eC: presigned download URL C-\u003e\u003eS3: download output.csv The rest of this post walks through each part: the layout, the state record, the worker, the ordering trick, the API, and where the pattern stops fitting.\nS3 Layout Each task gets its own prefix:\nbatch-tasks/ \u0026lt;task-id\u0026gt;/ input.csv status.json output-attempt-1.csv active/ \u0026lt;task-id\u0026gt; Per-Task Artifacts The task prefix holds the durable artifacts:\ninput.csv — uploaded by the user via a presigned URL. status.json — the task state and progress. output-attempt-N.csv — written by the worker, one per attempt. Active Marker active/\u0026lt;task-id\u0026gt; is a zero-byte object that acts as a worker index: its presence means \u0026ldquo;this task may need attention\u0026rdquo;. It is not the source of truth for the task\u0026rsquo;s state - status.json is. Markers are cheap to list for polling, but correctness always comes from the state document.\nTask Lifecycle A task moves through a small set of states:\nstateDiagram-v2 [*] --\u003e created created --\u003e queued: submit queued --\u003e running: worker claims lease running --\u003e completed: output written running --\u003e failed: error failed --\u003e queued: retry if attempt \u003c max completed --\u003e [*] created --\u003e abandoned: cancel queued --\u003e abandoned: cancel running --\u003e abandoned: cancel failed --\u003e abandoned: cancel abandoned --\u003e [*] failed --\u003e [*]: max attempt reached created: record exists and an upload URL was issued but the input is not there yet. queued: input uploaded, ready to be processed. running: a worker holds the lease and is processing it. completed: output written. failed: an attempt failed; retryable until attempt_id reaches the max, at which point the active marker is removed and the task stays failed permanently. abandoned: terminal - cancelled by an operator. The created state exists because uploads go directly to S3: the API creates the task and returns a presigned URL before the file exists, then a separate submit step confirms the upload and moves the task to queued.\nTask Record status.json is the entire task record:\n{ \u0026#34;schema_version\u0026#34;: 1, \u0026#34;task_id\u0026#34;: \u0026#34;7ZZZZZZZY1F801XTH00MY7XF09K\u0026#34;, \u0026#34;task_name\u0026#34;: \u0026#34;June batch import\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;queued\u0026#34;, \u0026#34;attempt_id\u0026#34;: 0, \u0026#34;row_count\u0026#34;: 1000, \u0026#34;processed_count\u0026#34;: 250, \u0026#34;success_count\u0026#34;: 245, \u0026#34;error_count\u0026#34;: 5, \u0026#34;input_object_path\u0026#34;: \u0026#34;batch-tasks/7ZZZZZZZY1F801XTH00MY7XF09K/input.csv\u0026#34;, \u0026#34;output_object_path\u0026#34;: null, \u0026#34;worker_id\u0026#34;: \u0026#34;worker-8\u0026#34;, \u0026#34;lease_expires_at\u0026#34;: \u0026#34;2026-06-21T10:15:00Z\u0026#34;, \u0026#34;last_error\u0026#34;: null, \u0026#34;created_at\u0026#34;: \u0026#34;2026-06-21T10:00:00Z\u0026#34;, \u0026#34;updated_at\u0026#34;: \u0026#34;2026-06-21T10:05:00Z\u0026#34;, \u0026#34;created_by\u0026#34;: \u0026#34;john.doe\u0026#34; } The exact fields depend on the workflow, but the useful categories are stable:\nCategory Fields Notes identity task_id, task_name, created_by task_id is the inverted ULID (see ordering).\ncreated_by is whoever submitted it, useful for audit and access checks. lifecycle schema_version, status, attempt_id schema_version lets you evolve the record format safely.\nattempt_id increments on each retry and names the output file (output-attempt-2.csv). progress row_count, processed_count, success_count, error_count Drives the operator UI\u0026rsquo;s progress bar.\nThe worker updates these as it streams through the input. storage input_object_path, output_object_path Full S3 keys.\noutput_object_path stays null until the run completes. ownership worker_id, lease_expires_at worker_id identifies the current holder - a pod name, hostname, or instance ID is fine.\nlease_expires_at is how long that worker\u0026rsquo;s claim is valid. The worker extends it while making progress; if it lapses, the task is treated as not picked up (the worker likely crashed) and becomes available for another worker to retry. audit created_at, updated_at, last_error last_error captures the most recent failure message so an operator can see why a task failed without digging through logs. Treat the categories as the skeleton of any task record. The workflow-specific parts will vary - progress counters can be different per job, and the lifecycle can carry a task_type and payload that the worker dispatches on (more on that later). But identity, lifecycle, ownership, and audit are what make a task trackable, claimable, and debuggable regardless of what it actually does.\nWorker Polling for Work The worker can run inside the existing service if the workload is small enough. It periodically lists batch-tasks/active/, reads each listed task\u0026rsquo;s status.json, and decides whether the task can be acquired. Listing the active markers keeps polling cheap, the worker scans only the handful of tasks that may need attention, not every task ever submitted.\nWhen a task reaches a terminal state (completed or abandoned), the worker removes its active marker. The task\u0026rsquo;s status.json and output stay in S3 for audit but drop out of the polling set. Because the marker is only an index, neither failure mode corrupts the task: a missing marker on a pending task is recreated the next time it is retried or submitted, and a stale marker on a terminal task is harmless - the worker ignores or removes it on the next poll.\nAcquiring a Task Safely The hardest part is making sure two workers never process the same task. S3 conditional writes give us a compare and set (CAS) on a single object:\nRead status.json and capture its ETag Write the update only If-Match the same ETag Worker A reads status.json → queued, ETag \u0026#34;abc\u0026#34; Worker B reads status.json → queued, ETag \u0026#34;abc\u0026#34; Worker A writes running, If-Match \u0026#34;abc\u0026#34; → success, new ETag \u0026#34;def\u0026#34; Worker B writes running, If-Match \u0026#34;abc\u0026#34; → conflict Only one worker wins the lease. The same primitive protects heartbeats, retries, cancellation, and terminal writes: a stale worker\u0026rsquo;s ETag no longer matches, so it cannot clobber newer state.\nLeases and Retries On acquisition the worker writes status=running, its worker_id, and a lease_expires_at, extending the lease as it makes progress. If it crashes, the lease expires and a retry path moves the task back to queued.\nRetries write to attempt-specific paths so earlier output is never overwritten:\noutput-attempt-1.csv output-attempt-2.csv Cap the number of attempts. Without a limit, a task that fails deterministically due to bad input, a permanent downstream error will be re-queued forever, and workers will keep burning cycles re-running work that can never succeed. Once attempt_id hits the max, remove the active marker and leave the task in failed, it stays in S3 for audit but drops out of the polling set. Surface last_error so an operator can see why without digging through logs.\nNewest-First Listing With Inverted ULIDs S3 lists object keys lexicographically in ascending order, but an operator UI almost always wants \u0026ldquo;show me the newest tasks\u0026rdquo; — the equivalent of ORDER BY created_at DESC LIMIT 50. We get that for free from the task ID.\nA ULID is a 26-character identifier: a 48-bit millisecond timestamp followed by 80 bits of randomness.\n0JWHDX3P8Q KYGTVF3NE2RFMB |----------| |--------------| timestamp randomness (10 chars) (16 chars) 48-bit ms 80 bits ULID uses Crockford base32 (5 bits per character), so 10 chars hold 50 bits, the timestamp only occupies 48 of them. That makes normal ULIDs sortable by creation time (oldest first), which is the opposite of what we want. So we invert the timestamp:\ninverted_timestamp = (2^48 - 1) - current_unix_ms Then keep the normal random suffix.\nThe result is still a fixed-width, sortable, collision-resistant ID, but newer tasks produce smaller key prefixes. Since S3 lists keys ascending, listing the task prefix naturally returns newest-first.\nExample:\nbatch-tasks/ 7ZZZZZZZY1F801XTH00MY7XF09K/ ← newest 7ZZZZZZZY0HR51M5BZ9QY1Q8P0C/ 7ZZZZZZZ08S8B2Y2HXEBYB0C3PR/ ← oldest The tradeoff: the key format now encodes an ordering decision. That is fine when documented clearly, but it is not a substitute for rich filtering or arbitrary sort orders, that is where a database wins.\nAPI Layer Control API Shape A small control API is enough:\nPOST /tasks POST /tasks/{id}/submit GET /tasks GET /tasks/{id} POST /tasks/{id}/retry POST /tasks/{id}/cancel GET /tasks/{id}/output The key boundary is that the API owns state transitions. The UI can display a backend-computed retryable flag, but the retry endpoint re-checks state authoritatively, clients never decide transitions from their own clock or assumptions.\nPresigned URLs Keep the Service Thin The service never proxies file bytes; uploads and downloads go direct to S3.\nUpload: POST /tasks writes status.json (created) and returns a presigned upload URL → the client uploads to S3 → POST /tasks/{id}/submit verifies the object exists, marks the task queued, and writes the active marker.\nDownload: GET /tasks/{id}/output verifies the task is completed and returns a short-lived presigned download URL.\nThis keeps the service focused on task control and authorization; S3 handles transfer.\nPagination If history outgrows one screen, GET /tasks takes an opaque cursor:\nGET /tasks?limit=50\u0026amp;cursor=... Internally the cursor maps to an S3 start-after key (last task ID seen), which is passed to the next ListObjectsV2 call to resume from that position. Keep it opaque to clients, it is a pagination mechanism, not a query language.\nTradeoffs and Limits Where It Fits This pattern is a good fit when:\nWorkflow is file-oriented and artifacts already belong in S3 Task volume is low and not high-throughput Seconds-level pickup latency is acceptable UI only needs recent task history Tasks fail cleanly: errors are retryable or cancellable, not partial/unrecoverable Keeping infrastructure footprint minimal matters more than query flexibility Where It Breaks Down S3 is not a queue: there is no built-in message locking, no dead-letter queue, no consumer groups, and no dispatch policy, you build the state transitions yourself. Listing is cheap for small pages but it is not a query engine, and rendering a list costs one list call plus one status.json read per task shown (fine for 20–50 rows, bad for thousands).\nReach for the right tool once the workflow earns it:\nTool When Database Filtering, joins, arbitrary ordering, or pagination across many records Queue High throughput, backpressure, fair distribution, delayed retries, or dead-lettering Workflow engine Multi-step orchestration: dependencies, approvals, branching, cross-step recovery Final Thought: Typed Tasks As A Future Extension Start with one concrete workflow to keep the model honest and avoids prematurely building a generic task platform. But if the service later owns several similar low-volume jobs, status.json can grow a task_type and a type-specific payload:\n{ \u0026#34;task_id\u0026#34;: \u0026#34;7ZZZZZZZY1F801XTH00MY7XF09K\u0026#34;, \u0026#34;task_type\u0026#34;: \u0026#34;email_enrichment\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;queued\u0026#34;, \u0026#34;payload\u0026#34;: { \u0026#34;input_object\u0026#34;: \u0026#34;batch-tasks/7ZZZZZZZY1F801XTH00MY7XF09K/input.csv\u0026#34;, \u0026#34;output_format\u0026#34;: \u0026#34;csv\u0026#34; } } The worker can then dispatch by type:\nemail_enrichment → read CSV, enrich rows, write output report_export → generate report artifact bulk_reconciliation → run reconciliation and emit result file This works only while the types share one lifecycle, retry model, permissions, and operational profile. Once they need different SLAs, worker pools, retention, or query patterns, stop stretching the pattern, that is the signal for a database, queue, or workflow engine.\nThe goal was never to avoid infrastructure forever. It is to add it once the workflow has earned it.\n","permalink":"https://tayts.me/posts/s3-task-store-queue/","summary":"ETags for compare-and-set, an inverted ULID for newest-first ordering, a zero-byte marker as a polling index — turns out that\u0026rsquo;s enough for a real task queue, no database or queue service needed.","title":"Building a Low-Volume Batch Worker Without a Database: Using S3 as Task Storage"},{"content":"Hi, I’m Tzu Shieh (TS, and NO it doesn\u0026rsquo;t stand for Typescript). I’m a software engineer who is just trying to understand how things work under the hood and get better at building reliable, scalable software in the midst of the AI hype.\nI am a strong believer that the foundations of software engineering like algorithms, data structures, design patterns, and clean code are still relevant in the AI era. AI may change the way we build software, but it doesn\u0026rsquo;t change the fundamental principles of software engineering. I hope to share my thoughts and experiences on these topics, as well as my journey in the world of software engineering.\nP.S. I am fueled by cat and kitten videos. Please say hello with a meow~\n⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⢀⣴⠖⠚⠛⠒⠶⠦⣤⡾⠋⠙⣦⣤⣤⣤⡴⠚⢳⡀⠀ ⠀⠀⢠⠟⠀⠀⠀⠀⠀⠀⠀⠘⠃⠀⠀⠀⠀⠀⠀⠀⠀⠘⣇⠀ ⠀⣠⡟⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⡆ ⣸⠋⠟⡀⠀⠀⢀⡀⠀⠀⠀⠀⠀⠀⢧⣤⠆⢀⡀⢠⣤⡤⢸⡇ ⠹⣄⠀⠙⠛⠛⠉⠉⠹⣧⠀⠚⠓⢦⡀⠀⠀⠈⠉⠀⠀⢠⣿⡄ ⠀⠙⢳⢦⣤⣤⣤⢴⡿⡳⠶⢤⣤⡼⣷⠦⢤⣤⣤⡤⡴⣿⠞⠃ ⠀⠀⠁⠊⠁⠈⠈⠁⠀⠀⠈⠁⠉⠀⠁⠀⠉⠐⠁⠉⠀⠀⠀⠀ ","permalink":"https://tayts.me/about/","summary":"About me","title":"About"}]