Handle<T> — Generational Handle Types

v0.54.5 — Lifecycle, stale detection, and safety tests added as part of the Uncovered Types Hardening series (UTH-011..016).

1. Overview

Handle<T> is Nitpick's generational handle type. It provides a safe alternative to raw pointers for managing heap-allocated resources:

• A handle is a 64-bit integer packed as [generation:32 | arena_id:16 | slot_index:16]
• Generation counting detects use-after-free at deref time — no dangling pointers, no UB
• Handles are arena-scoped — all memory backing the handle is owned by a HandleArena
• The type parameter T provides compile-time type checking to prevent cross-type aliasing

use "handle.npk".*;

int64:arena = raw HandleArena.create();
Handle<int32>:h = raw HandleArena.alloc(arena, 4i64);   // typed binding

int64:ptr = raw HandleArena.deref(h);    // 0 = stale or null; non-zero = live
if (ptr == 0i64) { exit 1; }

raw HandleArena.free(h);
int32:_ = raw HandleArena.destroy(arena);

2. Handle Layout

64-bit handle value:
┌─────────────────────────────┬───────────────┬───────────────┐
│      generation (32 bits)   │ arena_id (16) │  slot_idx (16)│
└─────────────────────────────┴───────────────┴───────────────┘
         starts at 1                                          
         bumped on free/destroy                               
         0 = NPK_HANDLE_NULL                                  

• Generation 0 (NPK_HANDLE_NULL) is the null handle — HandleArena.deref(0i64) always returns 0.
• Generations start at 1 on first alloc and increment on every free or destroy.
• When the generation counter saturates at UINT32_MAX, the slot is permanently retired (never reused).
• Arena IDs are in [1..65535]; slot indices are in [0..65534].

3. Generational Safety — How It Works

The runtime stores the current generation for each slot alongside the slot's data. When you call HandleArena.deref(h):

1. Extract arena_id and slot_index from h
2. Look up the slot's current generation
3. Compare it to the generation embedded in h
4. Match → return the slot's data pointer
5. Mismatch → return NULL (0i64)

This means that even if the physical memory slot is reused by a later alloc, the old handle's generation check fails first. There is no UAF window.

Timeline: single slot, three allocations

Alloc h1 → gen=1, slot=0   deref(h1) → ptr     free(h1) → slot gen bumped to 2
Alloc h2 → gen=2, slot=0   deref(h2) → ptr     deref(h1) → 0 (gen mismatch!)
Alloc h3 → gen=3, slot=0   deref(h3) → ptr     deref(h2) → 0

4. Allocation

Create an arena, then allocate slots of a given byte size:

use "handle.npk".*;

int64:arena = raw HandleArena.create();        // arena id (>= 1)
Handle<int32>:h = raw HandleArena.alloc(arena, 4i64);  // 4-byte slot

HandleArena.alloc returns an int64 handle. Because Handle<T> lowers to int64, you can assign the result directly to a typed binding. The T parameter is not carried at runtime — it exists solely for the type checker.

Failure cases

HandleArena.alloc returns 0 (null handle) if: - Arena is invalid or already destroyed - Negative size - Allocator is out of memory - Per-arena slot limit (65,534) exhausted - Generation saturation on a recycled slot

Always check before use:

int64:h = raw HandleArena.alloc(arena, 8i64);
if (h == 0i64) { exit 1; }   // alloc failed

[!NOTE] Handle<T> cannot be directly compared to 0i64. Use an int64 intermediate: nitpick Handle<int32>:h = raw HandleArena.alloc(arena, 4i64); int64:hv = h; if (hv == 0i64) { exit 1; }

5. Dereference

HandleArena.deref(h) returns the slot's buffer pointer as int64:

int64:ptr = raw HandleArena.deref(h);
if (ptr == 0i64) {
    // stale, freed, or null handle — do NOT use ptr
    exit 1;
}
// ptr is valid for the arena's lifetime

The pointer is owned by the arena — do not call free() on it. The runtime guarantees: - A live handle always returns the same pointer (handles don't move) - A freed handle always returns 0 - A handle from a destroyed arena always returns 0 - deref(0i64) always returns 0

6. Free

HandleArena.free(h) releases the slot and bumps its generation:

raw HandleArena.free(h);

int64:ptr = raw HandleArena.deref(h);   // Returns 0 — slot is stale

Free is safe on stale handles. Calling free on a null handle, stale handle, or a handle from a destroyed arena is a no-op. This prevents double-free crashes.

You do not need to free every handle before destroying the arena — HandleArena.destroy handles bulk invalidation.

7. Destroy Arena

HandleArena.destroy(arena) tears down the entire arena in a single operation:

int32:_ = raw HandleArena.destroy(arena);

After destroy: - Every handle that referred to this arena returns 0 on deref - The arena id is reclaimed

int64:arena = raw HandleArena.create();
int64:h = raw HandleArena.alloc(arena, 8i64);
// ... use h ...
int32:_ = raw HandleArena.destroy(arena);
// h is now stale — any deref returns 0

8. Cross-Type Safety

The T parameter in Handle<T> is enforced by the type checker. A Handle<int32> cannot be assigned to a Handle<int64> binding:

Handle<int32>:hi = raw HandleArena.alloc(arena, 4i64);
Handle<int64>:hl = hi;   // ❌ Error: Handle<int32> → Handle<int64> type mismatch

Handles can be freely assigned to and from raw int64. This is the escape hatch for FFI and storage:

Handle<int32>:h  = raw HandleArena.alloc(arena, 4i64);
int64:stored     = h;             // store as raw int64 (e.g. in a struct field)
Handle<int32>:h2 = stored;        // recover the typed binding

9. nodrop — Manual Lifecycle Management

By default, Nitpick's RAII system auto-destructs handle arenas when bindings go out of scope. Use nodrop to opt out and manage lifecycle explicitly:

use "handle.npk".*;

int64:arena = nodrop raw HandleArena.create();
Handle<int64>:h = nodrop raw HandleArena.alloc(arena, 8i64);

// ... use h ...

raw HandleArena.free(h);
raw HandleArena.destroy(arena);

nodrop is necessary when: - You need a deterministic cleanup order - Handles outlive their declaring scope (e.g., stored in a struct) - You're interfacing with FFI that requires manual resource management

[!WARNING] With nodrop, you are responsible for calling HandleArena.destroy(arena) exactly once. Forgetting to destroy an arena leaks all its memory.

10. Self-Referential Data Structures

Handle<T> is the idiomatic way to build linked data structures without raw pointers:

struct:Node = {
    int32:value;
    int64:next_handle;   // int64 = raw Handle<Node> value; 0 = null/end
};

int64:arena = raw HandleArena.create();

int64:h1 = raw HandleArena.alloc(arena, 12i64);   // 12 = sizeof(Node)
int64:h2 = raw HandleArena.alloc(arena, 12i64);
int64:h3 = raw HandleArena.alloc(arena, 12i64);

// Link: h1 → h2 → h3 → null
// (Values written via FFI/extern or struct initializers in future versions)

int64:_ = raw HandleArena.destroy(arena);
// All three handles are now stale — no dangling pointers

Why this is safe: When you destroy the arena, the generation of every slot is bumped. Any handle stored as a next_handle field in another node immediately becomes stale on deref. There are no dangling raw pointers in the residue.

11. Complete API Reference

Underlying C ABI (for FFI)

These functions are exposed directly if you prefer not to use the HandleArena wrapper:

12. Handle<T> vs Raw Pointers (any / buffer)

13. Advanced Patterns

Handle<T> in Arrays

Because Handle<T> lowers to int64 and has value semantics, you can store arrays of handles:

Handle<Node>[10]:nodes;
nodes[0i64] = raw HandleArena.alloc(arena, 32i64);
// Uninitialized slots hold 0i64 (null handle)

Async and Thread Safety

The handle generation metadata is stored centrally in the HandleArena. - Deref and Free are Thread-Safe: The runtime uses atomic loads/stores for the generation counter, making HandleArena.deref and HandleArena.free safe to call concurrently from multiple threads. - Alloc is Locked: HandleArena.alloc takes a lightweight spinlock to find a free slot. In highly contended multithreaded systems, use thread-local arenas rather than a single global arena.

14. Debugging Stale Handles

If your program is unexpectedly exiting or failing because a handle is returning 0i64 on dereference, here are some debugging tips:

1. Verify the Arena Lifecycle: Are you using RAII or nodrop? If the scope defining the HandleArena.create() call has exited, the arena was destroyed, making all handles stale.
2. Check for Unintended free Calls: Ensure you did not double-free the handle or free it in an earlier subsystem.
3. Check Saturation: Did the slot recycle 4,294,967,295 times? (Rare, but possible in extreme long-running systems on a small arena).
4. Use dbug.check(h): You can inspect the raw int64 value. A value of 0 means the handle itself is null, meaning it was never successfully allocated. If it's non-zero but deref fails, the slot generation bumped.

15. Worked Example: Object Pool with Handle

use "handle.npk".*;

struct:Particle = {
    flt64:x;
    flt64:y;
    flt64:vx;
    flt64:vy;
};

func:failsafe = int32(tbb32:err) { exit 1; };

func:main = int32() {
    int64:pool = raw HandleArena.create();
    if (pool == 0i64) { exit 1; }

    // Allocate 3 particles (4 × 8 bytes = 32 bytes each)
    Handle<Particle>:p1 = raw HandleArena.alloc(pool, 32i64);
    Handle<Particle>:p2 = raw HandleArena.alloc(pool, 32i64);
    Handle<Particle>:p3 = raw HandleArena.alloc(pool, 32i64);

    int64:p1v = p1;
    if (p1v == 0i64) { exit 2; }   // alloc failed

    // Deref to get buffer pointers (write via FFI or future field access)
    int64:buf1 = raw HandleArena.deref(p1);
    int64:buf2 = raw HandleArena.deref(p2);
    int64:buf3 = raw HandleArena.deref(p3);

    // Simulate physics step: free a dead particle
    raw HandleArena.free(p2);
    // p2 is stale — any future deref returns 0 (safe)

    // Tear down pool — all remaining handles become stale
    int32:_ = raw HandleArena.destroy(pool);

    exit 0;
};

16. Known Limitations

• No nodrop auto-reminder — The compiler does not warn if you forget to call HandleArena.destroy after nodrop. This will be addressed in a future nodrop auditing pass.
• T parameter is advisory — Handle<T> carries no runtime information about T. The arena slot holds raw bytes; type safety is enforced only by the type checker, not the runtime.
• No arena.grow() — Arena capacity is currently fixed at creation. Exceeding the per-arena slot limit (65,534) returns a null handle. Dynamic growing is planned.
• No K-semantics — Handle<T> is not formally modeled in k-semantics/nitpick.k. Generational safety is a runtime property verified by tests, not by the formal model.
• No use-after-free compile-time detection (UTH-014) — The NITPICK-032 diagnostic detects static arena-lifetime violations within the same function scope, but cross-function or cross-scope UAF is not currently detected at compile time.

17. Error Reference

NITPICK-032: Handle outlives arena

error: [NITPICK-032] Handle 'h' outlives its arena 'arena'.
The arena was destroyed earlier in this function; the handle is no longer valid.

Cause: You called HandleArena.deref(h) or HandleArena.free(h) after calling HandleArena.destroy(arena) in the same function scope.

Fix: Either restructure so the deref happens before the destroy, or use nodrop raw HandleArena.destroy(arena) to opt out of the static check and rely on the runtime generational detection instead.

See also

• Arenas — arena design and capacity
• Lifetimes — when the compiler catches misuse
• Diagnostics — NITPICK-032 and handle-related errors