Transactions & Recovery

This page explains how Managed mode prevents a live pointer from naming missing content and how it coordinates SQLite state with filesystem changes.

The invariant

On a supported local POSIX filesystem, a client following a Managed protocol pointer reads a complete old view or a complete new view, including after interruption.

Everything below exists to hold that line: metadata is fully staged and verified before anything public moves, the pointer swap is the commit decision, and every operation records enough durable evidence that the next command can finish it or undo it without guessing.

Plain sow create is intentionally outside this transaction model. Its package directory is the authority and its metadata is a disposable projection: it performs one content pass, a final stat check, and overwrite publication. An interruption is handled by running sow create again, not by replaying a journal. See Plain Flat Repositories.

Note what this does not claim. dirty does not mean a half-written index — it means the Desired state is ahead of the Built Generation while the old Built view remains complete. SOW also does not promise that two Dists flip at the same instant; it promises that each protocol view is self-consistent and that, when a write returns, every Dist included in that Operation is on its recorded Built Generation.

Two durable journals

Managed lifecycle and repository mutation use two durability substrates, each with a narrow scope:

The split is not arbitrary. Workspace lifecycle operations run when the target repository’s database does not exist yet or is about to be deleted, so they cannot use it. Repository mutations have a database available and use it. Plain has neither because its recovery unit is a fresh rebuild from packages.

The workspace journal stores the operation kind, a random 64-hex id, the repository name, and both the old and new raw sow.yml bytes with their SHA-256. The workspace lock guarantees at most one active operation. The atomic rename of sow.yml is the commit decision: if the current config still hashes to the old value, the planned journal is cleaned up and rolled back; if it hashes to the new value, SOW idempotently finishes creating the repository shell or moves the removed objects into recovery. If it matches neither, SOW refuses to guess.

The repository operation journal commits a planned operation into SQLite before any public file side effect, then records each state transition. Its payload binds the repository, the config SHA-256, the exact selected Dist set, the exact build_dists, the --skip decision, and a manifest hash covering the new object facts, the complete Desired set, the per-Dist policy outcomes, the RPM public certificate snapshot, and the target generation.

This is not SQLite’s WAL. WAL handles SQLite’s own page transactions; it cannot atomically coordinate the pool, the staging area, and dists/. The application-level journal is what spans the database and the POSIX file actions.

The operation lifecycle

planned → staged → applied → built → done
                       └──────────────→ done_dirty
   any nonterminal → recovering → built / rolled_back
   pre-apply error → failed

sow log <OPERATION> shows the transitions with timestamps:

"events":[
  {"sequence":0,"state":"planned","occurred_at":"2026-08-04T04:06:32.907704Z"},
  {"sequence":1,"state":"staged","occurred_at":"2026-08-04T04:06:33.067824Z"},
  {"sequence":2,"state":"applied","occurred_at":"2026-08-04T04:06:33.253073Z"},
  {"sequence":3,"state":"built","occurred_at":"2026-08-04T04:06:34.074916Z"},
  {"sequence":4,"state":"done","occurred_at":"2026-08-04T04:06:34.077441Z"}
]

done_dirty is reachable only when you explicitly pass --skip. A default add that fails after applied returns an error, keeps the old built view serving, and leaves the operation recoverable — it does not quietly settle as dirty.

An operation that fails before applied becomes failed. This matters for a subtle case in the contract: add must record a planned operation before parsing packages, so a package with a disallowed architecture does produce an audit record. But apart from that terminal failed record, nothing is written — no package object, no membership, no pending bytes, no public tree change, no generation. You keep the audit trail without letting an invalid architecture reach any product projection.

The lock model

Locks are POSIX advisory flock on the local machine. The product contract is single-writer, local POSIX, cooperative locking — network filesystems are neither detected nor supported.

When both are needed, the order is fixed: workspace first, then repository, released in reverse. The repository lock’s inode lives at a stable path and never moves with the private state directory, so removing a repository can withdraw the lock path while another process still holds an old descriptor, without a second writer forming on a new inode.

sow create locks the target directory and its stable parent. The parent lock is what stops another cooperating writer from replacing the directory by rename and then acquiring an independent lock on the substitute.

Read-only commands never take a write lock and do not accept lock flags. The ones that combine config, SQLite, and live metadata (config check, repo ls/show, dist ls/show) take shared locks for the duration of their snapshot. status is deliberately lighter: it probes the repository lock so it can report recovering or locked while a write is in flight, without blocking on it.

Two flags control waiting, on every command that takes a write lock:

Both failure paths exit 4. Combining --no-wait with a non-zero --timeout is a usage error, exit 2.

$ sow add ./build/*.rpm -r pgsql -d el9 -N
lock unavailable

Use -N in cron jobs where a skipped run is better than a pile-up, and -T 30s in CI where a short queue is fine but a hang is not.

The commit order

Every generation is written in the same four phases, and the order is what makes the invariant hold:

payload  →  metadata  →  pointer  →  delete

1. payload — canonical package bytes into pool/. Nothing references them yet.
2. metadata — checksum-named RPM metadata, Packages, Packages.gz, and by-hash index copies. Still nothing points at them.
3. pointer — the client entry points: repomd.xml (plus .asc if configured) for RPM; for Managed APT, Release (plus InRelease and Release.gpg) after every per-architecture direct and by-hash index is in place. This is the commit.
4. delete — expired metadata from generations that have aged out.

Pending payload promotion is batched under the single writer: at most 512 objects or 1 GiB per group commit. Pool directory entries are persisted before pending names are removed, so recovery can bind a pending-only, exact dual-link, or Pool-only state back to the Operation without risking loss of both names.

Read it forward: a package always exists before an index names it, and an index always exists before a pointer names it. Read it backward: nothing is deleted until a pointer that no longer references it is durable. There is no window in which a client can follow a live pointer to a missing file.

All of this happens through a staging area on the same filesystem as the target, verified at initialization by comparing st_dev. A different mount or device is an explicit failure, never a degraded copy. Files are written, fsynced, validated by SOW’s own parser and closure validator, and only then moved in with atomic renames. Public files do not inherit your umask: repodata/ is 0755, index files and pointers are 0644.

sow changes describes the generation delta for audit and delivery planning. It is not a safe substitute for the publication protocol: use sow publish, or copy the complete tree into offline staging and switch it into service atomically. See Observability & Audit.

Crash recovery

Every Managed write command recovers before it does its own work. There is no separate repair command and no daemon watching for stale state; recovery is a precondition of mutation. If a nonterminal operation exists, the next add, rm, build, dist new/rm, or log prune completes or rolls it back first, then proceeds.

Global recovery order is fixed: workspace lifecycle first under the workspace lock, then — if that was not a repository removal — repository operations in repository-name order under each stable repository lock. A workspace operation that has already passed the repository-removal commit decision takes precedence and forbids any nested repository recovery, since recovering state inside a repository that is being deleted would be meaningless.

Recovery is evidence-driven, not optimistic. Each phase has a defined rule:

This was validated by sending SIGKILL to sow add at many different moments. In every case status reported recovering, the next write command recovered that operation before executing its own, the final check passed all layers, and the public tree was never torn.

$ sow status
repository=pigsty status=recovering ready_to_copy=false ...

sow build is the one explicit forward-recovery entry point: it attempts to complete or roll back any decidable nonterminal operation before converging. If you see recovering, running sow build is the normal response.

error is reserved for the case where the journal, database, and file evidence contradict each other and no automatic choice is safe. Build refuses to overwrite; the last completed view keeps serving; you restore from backup and then run check and build. There is deliberately no repair --force, because a repair that guesses wrong is worse than a repair that refuses.

Fail-closed path safety

Managed paths are never assembled from user-supplied strings. Every create, rename, and delete follows the same sequence:

1. resolve the workspace root to an absolute real path;
2. reconstruct the target from a fixed relative fragment and verify the relative path contains no escape;
3. Lstat every existing controlled component and reject symlinks and unexpected file types;
4. delete only objects that were first atomically moved into .sow/.../recovery;
5. before deleting, prove again that the recovery target sits inside the corresponding private state directory.

Names must match [a-z0-9][a-z0-9._-]*, and ., .., .sow, pool, dists and workspace-reserved names are rejected outright.

The same posture applies to file handles. SQLite is opened with O_NOFOLLOW and bound to a regular-file inode, re-verified by path after the connection is established; a database, WAL, shm, or rollback journal that is a symlink, a non-regular file, multiply hardlinked, or rebound during the open is rejected. log export refuses to overwrite an existing file and refuses a symlinked parent directory — which is why exporting into /tmp on macOS fails, since /tmp is a symlink there.

Journals are bounded by size: 32 MiB for the workspace, 16 MiB for a repository operation payload, and 64 MiB each for the external mutation and base manifests. An oversized journal is never truncated and never degraded — it fails outside the commit window, so a writer can never produce an operation record that a recovery reader would be unable to read back.

None of this claims to defend against a malicious process running as the same user with unlimited privileges. It defends against the realistic failure modes: crashes, races between cooperating processes, and paths that changed shape between the check and the use.

Next

• Observability & Audit — reading the state these mechanisms maintain
• Exit Codes — what 4, 5, and 6 mean and when you see them
• build, status, check, and changes