Features

• 1: Plain Flat Repositories
• 2: Managed Workspaces
• 3: Pool & Metadata Views
• 4: Membership Policy
• 5: Signing Model
• 6: Transactions & Recovery
• 7: Observability & Audit

1 - Plain Flat Repositories

sow create takes a directory that already contains .rpm and .deb files and writes flat repository indexes beside them. Plain mode has no workspace, configuration file, database, desired state, or operation journal. The package directory is the authority; every index is a disposable projection of its current contents.

That distinction is intentional. Managed repositories retain state and recover transactions. Plain repositories are cheap to recreate: if a run fails or is interrupted, run the same command again and overwrite the derived metadata.

Contract

Four rules define Plain mode:

1. Packages are authoritative. Default create does not modify package bytes. It replaces only repodata/, Packages, and Packages.gz; --pigsty and explicit RPM signing are the documented exceptions.
2. One package-content pass. On the normal unsigned path, every selected package is opened once, SHA-256 hashed once, and parsed during that same pass. Complete parsed RPM/DEB metadata is retained for rendering; render and validation do not reopen package payloads.
3. One cheap final check. Immediately before publication, SOW relists the top-level package set and compares stat facts with the scan snapshot. It does not compute a second package SHA-256.
4. Failure means rebuild. There is no Plain transaction journal, pre-image, roll-forward, or rollback. A failure may leave partially replaced derived metadata. The next sow create discards owned temporary residue and rebuilds from the package directory.

For identical input bytes, output remains deterministic and a repeat reports noop=true.

The one-pass pipeline

--jobs defaults to the logical CPU count and controls the only package-content pass:

lock directory
  -> list and sort top-level RPM/DEB candidates
  -> parallel open + SHA-256 + parse (once per package)
  -> resolve coordinates and Pigsty filtering
  -> render RPM/DEB metadata from retained parsed facts
  -> validate generated metadata only
  -> relist and compare package stat snapshots
  -> replace derived outputs; repo_complete last with --pigsty

Worker completion order never affects bytes: facts are consumed in canonical basename/index order. RPM XML is written from the parsed package object retained by the worker. DEB Packages paragraphs are written from the retained control paragraph and the SHA-256 already computed by that worker.

Output self-validation still reads generated XML, repomd.xml, Packages, and Packages.gz. Those files are small derived metadata; it does not read package bodies again.

What the final stat check proves

The final check requires:

• the same sorted set of top-level regular .rpm/.deb basenames;
• the same file identity/inode;
• unchanged file type and mode;
• unchanged size and modification time.

If any fact differs, publication is rejected with integrity exit code 5. This catches normal add, remove, replace, truncate, and rewrite races at directory-scan cost.

It is deliberately not cryptographic validation. An external writer that changes bytes in place while preserving inode, size, and mtime can evade it. Plain mode accepts that tradeoff because it targets one local cooperating writer and a rebuildable result. Use a Managed repository when hostile/concurrent mutation evidence or durable recovery is required.

Scan and output rules

• Only regular files at the directory top level are considered; scanning never recurses and never follows symlinks.
• Only .rpm and .deb suffixes are selected.
• Package identity and architecture come from the RPM header or DEB control member, never the filename. RPM src and nosrc are rejected.
• Every valid version is indexed. Two different byte streams with the same logical coordinate are rejected.
• Default mode rejects a directory with no packages. --pigsty may converge an interrupted cleanup whose authoritative package set is now empty.

RPM input produces repodata/; DEB input produces Packages and Packages.gz:

/srv/repo/
├── pev2-1.23.0-1.noarch.rpm
├── xray_26.2.6-1_amd64.deb
├── Packages
├── Packages.gz
└── repodata/
    ├── <sha256>-primary.xml.gz
    ├── <sha256>-filelists.xml.gz
    ├── <sha256>-other.xml.gz
    └── repomd.xml

Flat locations are relative: RPM uses the bare basename and DEB uses ./<basename>. Public directories are mode 0755; generated files and repo_complete are 0644, independent of umask.

When one format disappears, SOW removes its known derived outputs. A rerun also replaces incomplete pairs such as a lone Packages left by an interruption, and removes SOW-shaped checksum RPM metadata no longer referenced by the new generation. Unknown files remain untouched.

Determinism and no-op

repomd.xml always uses revision and timestamps 0; gzip headers are fixed; ordering is canonical. A given package set therefore produces byte-identical metadata. Before publication SOW compares the staged metadata with live metadata. If package cleanup/signing is unnecessary and every output already matches, it removes the private stage without replacing public inodes and returns noop=true.

The JSON field recovered is always false because Plain create never performs journal recovery.

Publication and interruption semantics

All metadata is generated and validated in a same-directory private stage before publication starts. Individual file replacements use same-filesystem rename, and RPM publishes checksum-named metadata before repomd.xml.

This is not a multi-file transaction. A process killed during publication may leave new RPM metadata with old DEB metadata, one member of the DEB pair, or extra old checksum-named RPM metadata. That state is not evidence to reconcile; it is disposable output. The next run renders the complete current projection and overwrites/removes the residue.

The implementation creates no durable journal or recovery trash. On startup it discards SOW-owned residue in the reserved Plain staging namespace before starting a fresh scan.

The --pigsty marker gate

--pigsty additionally removes parsed package facts matching its compatibility rules (DEB i386 and Patroni 3.0.4) and writes repo_complete as <sha256><two spaces><basename>, sorted by basename. An RPM is not removed merely because its architecture is i386/i486/i586/i686.

Its publication order is:

stage + validate
  -> final stat check
  -> withdraw old repo_complete
  -> install signed RPMs, if explicitly requested
  -> install RPM and DEB metadata
  -> delete matched packages
  -> write repo_complete last

A missing marker means “not complete”; consumers must not use the directory until the marker reappears. If a run stops after withdrawing the marker, rerun sow create --pigsty. The rerun scans the packages that now exist, overwrites metadata, finishes cleanup, and writes a fresh marker. No action log is required.

Default mode refuses to run while repo_complete exists, preventing an un-gated command from leaving a stale readiness claim.

Explicit RPM signing

--sign-with authorizes package mutation and is a separate slow path. SOW signs private stage copies, validates the embedded signature and signature-neutral digest, reparses the resulting RPM, then installs the signed bytes before their metadata. These necessary signing/copy/verification reads are outside the unsigned one-pass guarantee. If signing is interrupted, rerun from the package directory; no signing transaction is replayed from a journal.

Locking and scope

sow create locks the target directory and its stable parent for one run. --timeout and --no-wait control cooperative lock acquisition. The lock prevents another cooperating SOW process from writing concurrently; it does not turn arbitrary external package mutation into a supported workload.

Use Plain for a local, single-process flat-directory build that can be regenerated. Use Managed Workspaces when desired state, audit history, atomic generation switching, or evidence-driven crash recovery is part of the requirement.

Next

• sow create reference — flags, output, and failure contract
• Transactions & Recovery — the Managed durability boundary
• Quick Start — build one in five minutes

2 - Managed Workspaces

Managed mode is what you use when the same repository will be updated for months: packages arrive in batches, policy decides what stays, and you need to prove afterwards what changed and when. This page explains the three-tier model, the layout it produces, and how commands figure out which repository and which Dist you meant.

The three tiers

Workspace                          discovery and configuration boundary
└── Repository                     ownership boundary: pool, dists, SQLite, lock, generations
    └── Dist                       a named membership set in exactly one format
        └── Architecture View      a rendered projection — not a membership

Each tier has one job, and the boundaries are strict:

Workspace owns exactly two things: the root sow.yml and the .sow/ state directory. Nothing else at the workspace root belongs to SOW. It is the unit of discovery — commands find a workspace by walking up from a starting directory — and it is where the architecture permit list lives.

Repository is fixed at <workspace>/<name>. You cannot point it somewhere else; there is no path option. A Repository owns its pool/, dists/, SQLite database, lock, recovery state, generations, retained-generation roots, publication checkpoints, and GC evidence. Two Repositories never deduplicate against each other — the same package added to both is stored twice, on purpose, so that removing one Repository can never damage the other.

Dist is an ordinary named set of memberships in exactly one format, rpm or deb. The name is an opaque string to SOW. el9, trixie, el9-beta, customer-acme — none of these create a state machine, a promotion workflow, or a snapshot. If you want a beta channel, make a Dist called el9-beta; the meaning lives in your head and your .repo files, not in SOW.

Architecture View is what build renders. It creates no second membership. A noarch RPM has exactly one package object and exactly one membership, and gets projected into every applicable view. See Pool & Metadata Views.

One Repository can hold an RPM Dist and a DEB Dist at the same time, sharing one pool/.

The layout

Managed paths are never assembled from user input. They are derived from the resolved real workspace root, a validated name, and a fixed relative fragment — which is why symlink substitution and path escape have no surface to attack.

<workspace>/
├── sow.yml                       # the only configuration file
├── .sow/                         # private state; never serve this
│   ├── workspace.lock
│   ├── workspace-ops/            # workspace lifecycle journal
│   ├── repo-locks/<repo>.lock
│   ├── <repo>.db                 # one SQLite per repository
│   └── <repo>/
│       ├── stage/                # staging on the same filesystem
│       ├── recovery/             # atomic-move target for deletions
│       └── pending/              # durable payload for --skip
└── <repo>/                       # the servable tree
    ├── pool/                     # immutable package bytes
    └── dists/
        └── <dist>/               # architecture views rendered here

A real workspace after two dist new and two add commands:

$ find .sow | sort
.sow
.sow/pigsty
.sow/pigsty.db
.sow/pigsty.db-shm
.sow/pigsty.db-wal
.sow/pigsty/pending
.sow/pigsty/recovery
.sow/pigsty/stage
.sow/repo-locks
.sow/repo-locks/pigsty.lock
.sow/workspace-ops
.sow/workspace.lock

Everything under <repo>/ is the public delivery tree. Serve it directly, publish it with SOW, or copy the whole tree through offline staging and an atomic switch. Everything under .sow/ is private and must not be exposed; see the serving guide.

Names must match [a-z0-9][a-z0-9._-]*, and ., .., .sow, pool, dists and workspace-reserved names are rejected outright.

State database and package facts

The private SQLite database indexes Desired and Built Membership by package_sha256, so queries and builds expand a complete membership projection in bulk instead of issuing one query per package. It also keeps a rebuildable package-facts cache keyed by immutable package SHA-256. Ingest authenticates and parses a new package once; production builds load only the selected digest rows in bounded, deterministic batches and lazily rebuild a missing or corrupt row from authenticated package bytes. Unrelated or oversized facts rows are not read.

For unchanged Pool files, warm builds use device, inode, size, mtime, and ctime fingerprints to avoid rereading payload bodies. Fingerprint drift and a missing facts row share one authoritative SHA-256 pass and self-heal; sow check remains the explicit full cryptographic audit and hashes every unique physical payload once regardless of fingerprints.

The cache and its fingerprints are private implementation state; they do not change the public pool/ + dists/ layout. Never edit the database or its PRAGMA user_version by hand. A v0.3 Repository must be backed up and explicitly upgraded with sow repo migrate before ordinary 0.4 reads or writes. Fresh 0.4 Repositories already use the current schema; other maintenance is performed only when a SOW diagnostic names it.

sow.yml drives everything

There is one configuration file, parsed with a strict decoder. Unknown fields do not get ignored — they fail. So do duplicate normalized architectures, illegal names or formats, a Dist architecture that is not a subset of the workspace permit list, an invalid glob or category, and an incomplete signing block.

schema: sow/v3
architectures:
  - x86_64
  - aarch64
repos:
  pigsty:
    signing:
      rpm:
        packages:
          mode: never
    dists:
      el9:
        format: rpm
      trixie:
        format: deb
targets:
  local:
    repository: pigsty
    provider: filesystem
    endpoint: file:///srv/mirror
    prefix: pigsty
    public_endpoint: file:///srv/mirror/pigsty/
    max_cache_ttl: 0s
    authoritative_workspace: true
    single_writer: true
    exclusive_write_authority: true

config show --all expands every default and normalized alias so you can see what SOW actually decided:

$ sow config show --all
schema: sow/v3
architectures:
  - x86_64
  - aarch64
repos:
  pigsty:
    protected: false
    signing:
      rpm:
        packages:
          mode: never
    dists:
      el9:
        format: rpm
        architectures:
          - x86_64
          - aarch64
        limit: 0
        exclude: []
      trixie:
        format: deb
        architectures:
          - x86_64
          - aarch64
        limit: 0
        exclude: []

Architecture aliases are normalized once, at the parse boundary: amd64 → x86_64, arm64 → aarch64. Output is always the canonical family. The ecosystem names come back only in the rendered DEB view directory names (binary-amd64, binary-arm64).

config check is not a YAML linter. It opens each initialized Repository’s SQLite and compares the candidate configuration against the live Dists, architectures, memberships, built state, and signing availability. Removing an architecture family that memberships or built state still reference is an expected rejection (exit 6); corrupt database or protocol evidence is an integrity error (exit 5). Every write command runs the same preflight before it journals anything, so config check tells you in advance whether the next add would be refused.

$ sow config check
configuration valid: /data/ws repositories=1 dists=2

The full schema, including filesystem and r2 publication targets, is in the sow.yml reference.

Discovery: which workspace?

Managed commands look for the nearest ancestor sow.yml, in this order:

1. If -C/--workdir DIR is given, search upward from DIR; this suppresses the current-directory candidate.
2. Otherwise search upward from the current directory.
3. If that finds nothing, search upward from $SOW_DIR, including after an explicit -C search fails.
4. Still nothing: fail, with a hint about sow init, --workdir, and SOW_DIR.

The first sow.yml found wins; SOW does not keep climbing past it looking for a better one.

--workdir is not chdir. It changes only where discovery starts. A relative PATH argument in sow add is still resolved against your real current directory, which is what you want when you run sow add ./build/*.rpm -C /srv/ws.

sow create does not participate in any of this.

Selection: which repository, which Dist?

Repository selection, in order:

1. Explicit -r/--repo NAME.
2. The command’s starting directory is inside <workspace>/<repo>/.
3. The workspace has exactly one Repository.
4. Otherwise fail and list the candidates.

Dist selection, in order:

1. One or more explicit -d/--dist NAME (repeatable).
2. The starting directory is inside <workspace>/<repo>/dists/<dist>/.
3. The selected Repository has exactly one Dist.
4. Otherwise fail and list the candidates.

The important asymmetry: build, check, and status default to all Dists of the selected Repository when -d is absent, because operating on everything is the safe reading of “no filter”. add, rm, and ls require a definite Dist set, because guessing where a package should land is not safe:

$ sow ls
workspace discovery error: repository "pigsty" has multiple Dists (el9, trixie); select one or more with --dist

That is exit 2. Every inference happens only after path type and symlink validation.

init converges, it does not reset

sow init is idempotent by design, and its rules are an architectural invariant rather than a convenience:

• No sow.yml: create one with schema: sow/v3 and architectures: [x86_64, aarch64], plus .sow/. No Repository is created automatically.
• Valid config already present: in stable name order, fill in whatever is not initialized yet — a missing Repository shell, a missing SQLite, an entire missing Dist. A newly created Dist immediately gets all empty views for its effective architectures.
• Valid database state or a valid protocol pointer already present: verify only. Never overwrite, never zero a generation, never rewrite bytes.
• An architecture was added to the config after a Dist was already initialized: init does not render the new view and does not advance the generation. The Dist stays dirty and waits for an explicit build. Removing a family that memberships or built state still use is a failure.

$ sow init .
initialized /data/ws: config_created=false repositories_initialized=0 dists_initialized=0

The point of the third and fourth rules is that init must be safe to run on a repository holding real content. It converges toward the declared configuration; it never uses “not initialized yet” as an excuse to rebuild something that already works.

Objects are processed in stable order. If an early config, Repository, or Dist commits durably and a later object then fails, the committed count is preserved, the human output reports what did commit, --json keeps the structured result, and the command exits 3 (partial success). If nothing had committed yet, it exits with the original error class instead.

Empty has a valid protocol surface

A Dist created by dist new has complete protocol entry points before you add a single package. An RPM Dist gets valid empty repodata per architecture family; a DEB Dist gets Packages, Packages.gz, by-hash entries, and a Release. If the Repository has a metadata key configured, the empty Dist is signed too.

Removing the last package therefore leaves a valid signed-if-configured empty index rather than a missing or broken protocol entry point. Actual package-manager acceptance remains a separate compatibility gate.

Protected repositories

repos:
  pigsty:
    protected: true

protected: true refuses repo rm even with -f, and returns exit 6. It does not restrict anything else: add, rm, build, and normal Dist maintenance all work. To actually delete the Repository you must edit sow.yml, pass config check, and only then remove it — which is precisely the friction the flag exists to create.

Next

• Pool & Metadata Views — what build actually writes
• Publication Model — how a generation reaches a target
• Membership Policy — how exclude and limit decide what stays
• First Workspace — a ten-minute hands-on version of this page
• sow.yml reference — the complete schema

3 - Pool & Metadata Views

The invariant

Within one Repository, every live Package Object has one canonical payload path under pool/. Dists and architecture views own metadata, not aliases of package bytes:

<repo>/pool/...                              canonical package payloads
<repo>/dists/<rpm-dist>/<arch>/repodata/... RPM metadata only
<repo>/dists/<deb-dist>/main/binary-*/...   APT metadata only

The same digest in another Repository or publication prefix is a separately owned object. SOW deliberately does not turn local deduplication into shared distributed ownership.

What a built Repository looks like

An RPM Dist with one x86_64 package and one noarch package has this shape:

demo/
├── pool/
│   ├── c/centos-release/centos-release-6-0.el6.centos.5.x86_64.rpm
│   └── e/epel-release/epel-release-7-5.noarch.rpm
└── dists/el9/
    ├── aarch64/repodata/
    │   ├── <sha256>-primary.xml.gz
    │   ├── <sha256>-filelists.xml.gz
    │   ├── <sha256>-other.xml.gz
    │   └── repomd.xml
    └── x86_64/repodata/
        ├── <sha256>-primary.xml.gz
        ├── <sha256>-filelists.xml.gz
        ├── <sha256>-other.xml.gz
        └── repomd.xml

There is no dists/.../pool/ subtree. Content-addressed metadata from a retained live generation may coexist with the current files; repomd.xml is the pointer that selects the active set.

RPM views use computed parent-relative hrefs

rpm-md resolves each package <location href> relative to the architecture view. SOW computes the path from that view to the canonical Pool object:

<location href="../../../pool/c/centos-release/centos-release-6-0.el6.centos.5.x86_64.rpm"/>
<location href="../../../pool/e/epel-release/epel-release-7-5.noarch.rpm"/>

The depth is derived from the actual view root, not copied from a hostname or hard-coded deployment path. sow check resolves and normalizes each href, rejects escapes outside the Repository, and proves that it reaches the expected Pool object.

The complete Repository root is therefore the client and delivery boundary. Point DNF at dists/el9/x86_64/, but serve or copy the parent Repository that also contains pool/.

Why views contain metadata only

Copying each package into every architecture view would create extra object keys and uploads on storage systems without inode identity. SOW therefore gives payload ownership to the Repository Pool and lets indexes project membership. A complete copy, archive, or publication preserves that contract without depending on hardlinks.

The one-copy boundary is one Repository or one publication prefix—not a Workspace, bucket, account, or fleet. Identical packages in separate Repositories or targets retain separate owners.

Neutral packages are selected, not duplicated

An x86_64 view selects x86_64 + noarch; an aarch64 view selects aarch64 + noarch. The neutral package remains one Pool object. Each view gets its own metadata record whose location resolves to that same object.

DEB works the same way at the archive-root level: all packages appear in each applicable Packages index, while Filename: pool/... points to one canonical payload.

APT views

APT already defines Filename relative to the archive root:

Filename: pool/p/postgresql-18/libpq5_18.3-1_amd64.deb

SOW renders Packages, Packages.gz, and by-hash entries under dists/<dist>/main/binary-<arch>/. Release, InRelease, and Release.gpg are the protocol pointers and signatures. There is no per-view package alias and no per-architecture Release stub.

Ordinary clients and reposync are different contracts

The canonical layout is designed for package clients that consume the complete Repository and honor relative protocol paths. Default EL dnf reposync has a different contract: its safe-write check rejects a package location that normalizes above the per-repository download directory. This is an explicit unsupported combination; use an exported leaf for that workflow.

When a self-contained RPM leaf is required, create it outside the Repository and every configured filesystem publication root:

sow export rpm-leaf el9 x86_64 /srv/exports/el9-x86_64

The export contains its own package tree, repodata, manifest, and .sow-export.json completion marker. Copy is the default. --hardlink is an explicit same-filesystem, trusted read-only optimization. The export is not Membership, Generation, publication input, or a garbage-collection root.

Copy and publication

Canonical correctness does not depend on inode identity. Prefer a configured publication target. If another transport is required, copy the complete settled pool/ + dists/ tree with rsync, cp, or tar into an offline staging location, verify it, and switch it into service atomically. Never update the live tree file by file. Copying only one RPM architecture leaf is not supported because its metadata intentionally references the sibling root Pool.

sow changes lists each payload once under pool/, followed by metadata and pointers:

add  payload   pool/c/centos-release/centos-release-6-0.el6.centos.5.x86_64.rpm  ...
add  payload   pool/e/epel-release/epel-release-7-5.noarch.rpm                  ...
add  metadata  dists/el9/x86_64/repodata/<sha256>-primary.xml.gz               ...
add  pointer   dists/el9/x86_64/repodata/repomd.xml                            ...

There are no package-payload entries under dists/.

Next

• Managed Workspaces — ownership and Generation state
• Platforms & Integrations — tested and unsupported combinations
• Serve Repositories — HTTP, copies, and publication targets
• Repository Layout — exact public and private paths

4 - Membership Policy

Policy is the answer to “I dumped a build directory into this Dist and I do not want the debuginfo packages, and I only want the latest version of each package.” Two rules do that work, they run in a fixed order, and they run over the whole candidate set — not just the packages you happened to add this time.

The two rules and their order

candidate set  →  exclude  →  limit  →  Desired Membership

exclude drops packages that match a rule. limit then caps how many versions survive per package name and architecture. The order is fixed and never configurable, because the reverse order would let an excluded package consume a version slot on its way out.

Both rules are enforced on every add, every rm, and every build. That last one matters: editing limit or exclude in sow.yml marks the affected Dists dirty, and the next build re-applies the new policy to the existing membership. You do not have to re-add anything to make a tightened policy take effect.

dists:
  el9:
    format: rpm
    limit: 1
    exclude:
      - kind: [debuginfo, debugsource, llvmjit]

exclude

exclude is a list of rules. Within one rule, fields are combined with AND. Within one field, multiple patterns are combined with OR. Rules are combined with OR — any rule matching excludes the package. Field order and rule order never change the result.

exclude:
  - kind: [debuginfo, debugsource, dbgsym, dbg, llvmjit]
  - name: ["test-*", "*-experimental"]
    arch: [aarch64]

That reads as: drop every debug-ish package regardless of architecture, and drop aarch64 packages whose name starts with test- or ends with -experimental.

Five fields are allowed:

Patterns are case-sensitive exact strings or shell globs (*, ?, []). There is no regex, no version comparison, no negation, and no expression language. Unknown fields, empty rules, and invalid globs fail at config check rather than silently matching nothing.

kind is derived from the binary name, most specific suffix winning:

Classification comes from the package itself. It never depends on which directory the file came from or which host you are running on, so the same input always classifies the same way. sow show --json exposes the computed kind.

An excluded package is reported, not treated as a parse error, and it is not stored:

$ sow add pkg/blackbox_exporter-0.28.0-1.x86_64.rpm pkg/pev2-1.23.0-1.noarch.rpm -r demo -d el9
add repository=demo operation=7877233225745514469 accepted=1 failed=0 memberships=+1/-0 revision=3 generation=3 dirty=false
item input="pkg/blackbox_exporter-0.28.0-1.x86_64.rpm" status=excluded format=rpm coordinate="blackbox_exporter-0:0.28.0-1.x86_64" sha256:5759c643… dists=el9:excluded
item input="pkg/pev2-1.23.0-1.noarch.rpm" status=accepted format=rpm coordinate="pev2-0:1.23.0-1.noarch" sha256:d06d7f23… dists=el9:accepted

The command exits 0. Nothing was wrong with the excluded package — it just does not belong in this Dist. If a package is accepted by no Dist at all, no ownerless pool object is written for it.

limit

limit groups memberships by (binary name, native architecture) and keeps the newest N:

• 0 — keep every version. This is the default.
• positive N — keep the N newest by native version ordering.
• negative — a configuration error.

Two details decide most real questions.

The grouping key includes architecture. limit: 1 does not mean “one version of this package in this Dist”; it means “one version per name and native architecture”. So pg_sample-1.13 for x86_64 and pg_sample-1.17 for noarch both survive in a limit: 1 Dist, because they are in different groups. Neutral (noarch/all) counts once as its own native architecture even though it renders into multiple views.

Ordering is native to the format. RPM uses EVR comparison — epoch, version, release, with the standard rpm segment rules. DEB uses Debian version comparison, where the version string already carries the epoch and revision. SOW does not invent a version scheme or compare strings lexically.

Here is limit: 1 deciding between two Debian versions of the same package and architecture:

$ sow add pkg/libpq5_18.4-1.bookworm_amd64.deb pkg/libpq5_18.4-1.trixie_amd64.deb -r demo -d trixielim
add repository=demo operation=2402398619981505515 accepted=1 failed=0 memberships=+1/-0 revision=4 generation=4 dirty=false
item input="pkg/libpq5_18.4-1.bookworm_amd64.deb" status=excluded format=deb coordinate="libpq5=3:18.4-1.bookworm:amd64" sha256:be8a2863… dists=trixielim:limited
item input="pkg/libpq5_18.4-1.trixie_amd64.deb" status=accepted format=deb coordinate="libpq5=3:18.4-1.trixie:amd64" sha256:0a7df397… dists=trixielim:accepted

Note the two levels of reporting: the item’s overall status is excluded (it did not become a member anywhere), while the per-Dist outcome is limited — telling you it lost on version, not on an exclude rule. When you have several Dists selected, each one reports its own outcome, so a package can be accepted in one and limited in another in a single command.

limit removing an older membership and adding the newer one happens inside the same operation, so the ledger shows one atomic decision rather than a delete followed by an unrelated insert.

Policy runs over the full candidate set

A common misreading is that add applies policy only to the packages on the command line. It does not. After merging your input into the target memberships, SOW evaluates exclude and then limit over the complete membership set of each selected Dist.

The practical consequence: adding version 3 to a limit: 2 Dist that already holds versions 1 and 2 removes version 1 in the same operation. You cannot smuggle a package past the version cap by adding it separately, and you never end up with N+1 members because the cap was only checked against the delta.

Loosening a policy never resurrects anything

This is the semantics people most often expect to work the other way, so it is worth showing directly. Continuing from the limit: 1 example above, remove the version that won:

$ sow rm 'deb:libpq5=3:18.4-1.trixie:amd64' -r demo -d trixielim
$ sow ls -d trixielim
repository=demo dists=trixielim dirty=false
SHA256	COORDINATE	DISTS	BUILT_DISTS	POOL_PATH

The Dist is empty. The bookworm build did not come back, even though its bytes are still sitting in the pool, and even though there is now a free slot under limit: 1.

The reason is that exclude and limit remove actual Desired Memberships. SOW does not maintain a shadow list of “candidates that policy suppressed but might return later”. Pool bytes are storage, not a candidate set. Raising a limit or relaxing an exclude therefore gives you room for future additions; it does not reach back into history and guess which of the packages you once had should reappear.

To get it back, add it again — explicitly:

$ sow add pkg/libpq5_18.4-1.bookworm_amd64.deb -r demo -d trixielim
add repository=demo operation=590501245267266669 accepted=1 failed=0 memberships=+1/-0 revision=6 generation=6 dirty=false
item input="pkg/libpq5_18.4-1.bookworm_amd64.deb" status=accepted format=deb coordinate="libpq5=3:18.4-1.bookworm:amd64" sha256:be8a2863… dists=trixielim:accepted

Convergence is one-directional and it is stated as an invariant: tightening policy can remove members; loosening policy never restores them. That asymmetry is what makes build safe to run at any time. If it were symmetric, editing sow.yml could silently republish a package you deliberately withdrew — which is exactly the failure you do not want in a security update.

Previewing a decision

sow rm -c computes the removals, the policy consequences, and the file changes a build would produce, and writes nothing:

sow rm patroni -r pgsql -d el9 -c

-c/--check takes no write lock and is mutually exclusive with --skip. Passing --timeout or --no-wait alongside it is a usage error, so that nobody mistakes a preview for something that waits on a write transaction.

Next

• sow.yml reference — the complete policy schema
• sow add reference — per-item statuses and the partial-success exit code
• Pool & Architecture Views — where the surviving members get rendered
• sow retain and sow gc — payload lifecycle controls

5 - Signing Model

There are two different questions a client can ask about a repository, and SOW answers them with two separate mechanisms. Confusing them is the most common source of “I signed it but dnf still complains”, so this page starts by pulling them apart.

Two independent trust chains

They are configured separately and can be used separately. Metadata signing alone is usually the right starting point: it authenticates the whole index in one place and requires no change to the packages you received from upstream.

Managed metadata signing is controlled entirely by sow.yml. There is no CLI override, no --sign flag on build, and no way to sign one build differently from the next. That is deliberate — a repository’s signing identity is a property of the repository, not of the command that happened to update it.

Configuration

repos:
  pigsty:
    signing:
      rpm:
        packages:
          mode: never              # never | fill | always
        metadata:
          key: "file:///secure/repo-signing.asc"
      deb:
        metadata:
          key: "file:///secure/repo-signing.asc"

RPM and DEB metadata keys are declared separately, so you can use the same key for both (as above) or split them. Each metadata block accepts an optional passphrase reference alongside key.

With a metadata key configured, every build produces the signature files, including for an empty Dist:

• RPM, per architecture view: repodata/repomd.xml plus an ASCII-armored repodata/repomd.xml.asc
• DEB, per Dist: Release plus a clearsigned InRelease and a detached armored Release.gpg

The clearsigned body of InRelease is identical to Release. Without a metadata key, neither signature file is generated at all — you get repomd.xml and Release and nothing else.

Four key-reference forms

A key reference is a URI, and the scheme decides who does the signing:

file:// and env:// need nothing installed — SOW signs metadata itself, which is why a repository with file:// metadata keys builds identically on macOS and inside a minimal container. agent:// delegates to your GPG agent, which is the right choice when the private key lives on a smartcard or must never touch a file. agent:// cannot be combined with a passphrase reference, because the agent owns that interaction.

A passphrase reference accepts a Workspace-relative path, file://, or env://; it does not accept agent://.

Nothing secret is ever persisted. Configuration, SQLite, logs, JSON output, and error text hold only the reference string, the fingerprint, and the public verifier certificate. config show --all prints references and fingerprints, never key material. If a key reference is unresolvable or unusable for signing, config check says so before you run a build.

RPM package signing

signing:
  rpm:
    packages:
      mode: fill
      key: agent://7F721C4AD40F4A9D8CA578BFAC7E4690B50CCF3B
      trusted_keys: [keys/pgdg.asc]

Three modes:

trusted_keys automatically includes the public half of the configured key. Without a key, never is the only legal mode. fill is the default when a key is present.

Trust rings are verified independently

For a signed RPM, SOW evaluates each retained single-key ring, the current policy ring, and the combined trusted_keys ring independently. Every recognized OpenPGP signature packet must verify inside the same candidate ring, and at least one verified path must authenticate the payload. SOW never combines a packet accepted by one key with another packet accepted by a different single-key ring to invent a retained signer.

This distinction matters during a deliberate dual-signing transition: the combined trusted ring may accept the package, while neither individual retained key is allowed to claim it alone. Historical CentOS OpenPGP v3/v4 signatures remain supported. All candidate rings share one signed-byte stream, so adding trusted keys changes authorization but does not multiply package reads.

Package signing always runs rpm --addsign or rpm --resign from your environment, on a private staged copy. Your input file is never modified in place. After signing, SOW re-parses the result and requires an embedded signature, unchanged signature-neutral digest and NEVRA, and the exact configured public-key identity. The rpm and gpg executables and a matching secret key in the GPG environment used by rpm are mandatory for fill and always. A key reference identifies and verifies the signer; it does not provision the secret key into that environment.

Because signatures embed a timestamp, signing is not reproducible — the same unsigned RPM signed twice gives different bytes. Re-adding a package you already added would therefore look like a content conflict. SOW avoids that with a signature-neutral payload digest: a SHA-256 over the immutable header and payload, excluding the RPM signature header. If the logical coordinate already exists and the neutral digest matches, and the existing object satisfies the current policy, SOW reuses the existing final bytes instead of signing again. Repeated add of the same package is a stable no-op.

That reuse is narrow on purpose. never requires a full byte match, since that mode promises to preserve input bytes. If the payload digest differs, or the stored object does not satisfy the current signing policy, it is a hard conflict — add will not quietly re-sign a package in place under an existing coordinate. There is no --replace; if re-signing changes the bytes, bump the release, or plan a proper key-rotation workflow.

Key changes make Dists dirty

A Dist’s Built configuration digest covers its format, canonical architectures, limit, exclude, and the frozen signing identity. Change a key reference or a fingerprint and the digest changes, so every affected Dist becomes dirty:

$ sow status
repository=pigsty status=dirty ready_to_copy=false revision=5 generation=4 dirty_dists=el9,trixie pending=0/0 locked=false

For a metadata key change, sow build signs the indexes with the new identity and produces a new Generation.

RPM package bodies are immutable Package Objects. build does not silently re-sign an existing object under the same coordinate. If current Desired RPMs do not satisfy the new package-signing policy, build rejects the change. A staged rollover normally uses fill, makes the new key current, and keeps the old public key in trusted_keys; existing old-key packages retain their bytes while newly ingested packages use the new key. Remove the old trust only after those package coordinates have been withdrawn or replaced by new releases. Switching directly to always with a new key is valid only when every Desired RPM is already signed by that key.

The exact public certificate identity of the current built metadata is recorded per Dist, and multiple certificate versions for the same primary fingerprint can coexist — so extending an expiry or adding a subkey does not invalidate what is already published.

Plain mode

sow create /srv/repo --sign-with 6D5C5A26C36B1F73
sow create /srv/repo --sign-with 6D5C5A26C36B1F73 --overwrite

Plain mode signs RPM package bodies only; it has no metadata signing. KEY is exactly 16, 40, or 64 hexadecimal characters, without an 0x prefix; it is normalized to uppercase and passed to rpm as the _gpg_name macro. Without --overwrite, only RPMs with no parseable embedded signature are signed. With it, every retained RPM is re-signed.

--sign-with requires at least one top-level RPM retained after --pigsty cleanup. A DEB-only directory, a missing rpm binary, or an unavailable key fails before anything public changes. Signing is an explicit slow path with necessary copy, signature-verification, and final-RPM parse reads; if interrupted, rerun from the current package directory rather than replaying a Plain journal. See Plain Flat Repositories.

What the client verifies

[pigsty-el9]
name=Pigsty EL9
baseurl=https://repo.example.com/pigsty/dists/el9/$basearch/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://repo.example.com/keys/repo-signing.asc

Types: deb
URIs: https://repo.example.com/pigsty
Suites: trixie
Components: main
Signed-By: /etc/apt/keyrings/repo-signing.asc

repo_gpgcheck=1 makes dnf verify repomd.xml.asc; gpgcheck=1 makes it verify each package’s embedded signature. On the APT side, Signed-By makes apt verify InRelease. Automated checks validate generated signatures directly. Complete signed Managed dnf/APT acceptance must run with real clients in the target environment; see Platforms & Integrations for the exact evidence.

sow check verifies every declared signature and file hash as part of its normal run, so a signing misconfiguration shows up before you ship rather than on a customer’s machine.

Next

• Sign Your Repository — generating a dedicated key and wiring up both chains
• sow.yml reference — the full signing schema and key reference grammar
• Observability & Audit — how check proves the signatures

6 - Transactions & Recovery

This page explains how Managed mode prevents a live pointer from naming missing content and how it coordinates SQLite state with filesystem changes.

The invariant

On a supported local POSIX filesystem, a client following a Managed protocol pointer reads a complete old view or a complete new view, including after interruption.

Everything below exists to hold that line: metadata is fully staged and verified before anything public moves, the pointer swap is the commit decision, and every operation records enough durable evidence that the next command can finish it or undo it without guessing.

Plain sow create is intentionally outside this transaction model. Its package directory is the authority and its metadata is a disposable projection: it performs one content pass, a final stat check, and overwrite publication. An interruption is handled by running sow create again, not by replaying a journal. See Plain Flat Repositories.

Note what this does not claim. dirty does not mean a half-written index — it means the Desired state is ahead of the Built Generation while the old Built view remains complete. SOW also does not promise that two Dists flip at the same instant; it promises that each protocol view is self-consistent and that, when a write returns, every Dist included in that Operation is on its recorded Built Generation.

Two durable journals

Managed lifecycle and repository mutation use two durability substrates, each with a narrow scope:

The split is not arbitrary. Workspace lifecycle operations run when the target repository’s database does not exist yet or is about to be deleted, so they cannot use it. Repository mutations have a database available and use it. Plain has neither because its recovery unit is a fresh rebuild from packages.

The workspace journal stores the operation kind, a random 64-hex id, the repository name, and both the old and new raw sow.yml bytes with their SHA-256. The workspace lock guarantees at most one active operation. The atomic rename of sow.yml is the commit decision: if the current config still hashes to the old value, the planned journal is cleaned up and rolled back; if it hashes to the new value, SOW idempotently finishes creating the repository shell or moves the removed objects into recovery. If it matches neither, SOW refuses to guess.

The repository operation journal commits a planned operation into SQLite before any public file side effect, then records each state transition. Its payload binds the repository, the config SHA-256, the exact selected Dist set, the exact build_dists, the --skip decision, and a manifest hash covering the new object facts, the complete Desired set, the per-Dist policy outcomes, the RPM public certificate snapshot, and the target generation.

This is not SQLite’s WAL. WAL handles SQLite’s own page transactions; it cannot atomically coordinate the pool, the staging area, and dists/. The application-level journal is what spans the database and the POSIX file actions.

The operation lifecycle

planned → staged → applied → built → done
                       └──────────────→ done_dirty
   any nonterminal → recovering → built / rolled_back
   pre-apply error → failed

sow log <OPERATION> shows the transitions with timestamps:

"events":[
  {"sequence":0,"state":"planned","occurred_at":"2026-08-04T04:06:32.907704Z"},
  {"sequence":1,"state":"staged","occurred_at":"2026-08-04T04:06:33.067824Z"},
  {"sequence":2,"state":"applied","occurred_at":"2026-08-04T04:06:33.253073Z"},
  {"sequence":3,"state":"built","occurred_at":"2026-08-04T04:06:34.074916Z"},
  {"sequence":4,"state":"done","occurred_at":"2026-08-04T04:06:34.077441Z"}
]

done_dirty is reachable only when you explicitly pass --skip. A default add that fails after applied returns an error, keeps the old built view serving, and leaves the operation recoverable — it does not quietly settle as dirty.

An operation that fails before applied becomes failed. This matters for a subtle case in the contract: add must record a planned operation before parsing packages, so a package with a disallowed architecture does produce an audit record. But apart from that terminal failed record, nothing is written — no package object, no membership, no pending bytes, no public tree change, no generation. You keep the audit trail without letting an invalid architecture reach any product projection.

The lock model

Locks are POSIX advisory flock on the local machine. The product contract is single-writer, local POSIX, cooperative locking — network filesystems are neither detected nor supported.

When both are needed, the order is fixed: workspace first, then repository, released in reverse. The repository lock’s inode lives at a stable path and never moves with the private state directory, so removing a repository can withdraw the lock path while another process still holds an old descriptor, without a second writer forming on a new inode.

sow create locks the target directory and its stable parent. The parent lock is what stops another cooperating writer from replacing the directory by rename and then acquiring an independent lock on the substitute.

Read-only commands never take a write lock and do not accept lock flags. The ones that combine config, SQLite, and live metadata (config check, repo ls/show, dist ls/show) take shared locks for the duration of their snapshot. status is deliberately lighter: it probes the repository lock so it can report recovering or locked while a write is in flight, without blocking on it.

Two flags control waiting, on every command that takes a write lock:

Both failure paths exit 4. Combining --no-wait with a non-zero --timeout is a usage error, exit 2.

$ sow add ./build/*.rpm -r pgsql -d el9 -N
lock unavailable

Use -N in cron jobs where a skipped run is better than a pile-up, and -T 30s in CI where a short queue is fine but a hang is not.

The commit order

Every generation is written in the same four phases, and the order is what makes the invariant hold:

payload  →  metadata  →  pointer  →  delete

1. payload — canonical package bytes into pool/. Nothing references them yet.
2. metadata — checksum-named RPM metadata, Packages, Packages.gz, and by-hash index copies. Still nothing points at them.
3. pointer — the client entry points: repomd.xml (plus .asc if configured) for RPM; for Managed APT, Release (plus InRelease and Release.gpg) after every per-architecture direct and by-hash index is in place. This is the commit.
4. delete — expired metadata from generations that have aged out.

Pending payload promotion is batched under the single writer: at most 512 objects or 1 GiB per group commit. Pool directory entries are persisted before pending names are removed, so recovery can bind a pending-only, exact dual-link, or Pool-only state back to the Operation without risking loss of both names.

Read it forward: a package always exists before an index names it, and an index always exists before a pointer names it. Read it backward: nothing is deleted until a pointer that no longer references it is durable. There is no window in which a client can follow a live pointer to a missing file.

All of this happens through a staging area on the same filesystem as the target, verified at initialization by comparing st_dev. A different mount or device is an explicit failure, never a degraded copy. Files are written, fsynced, validated by SOW’s own parser and closure validator, and only then moved in with atomic renames. Public files do not inherit your umask: repodata/ is 0755, index files and pointers are 0644.

sow changes describes the generation delta for audit and delivery planning. It is not a safe substitute for the publication protocol: use sow publish, or copy the complete tree into offline staging and switch it into service atomically. See Observability & Audit.

Crash recovery

Every Managed write command recovers before it does its own work. There is no separate repair command and no daemon watching for stale state; recovery is a precondition of mutation. If a nonterminal operation exists, the next add, rm, build, dist new/rm, or log prune completes or rolls it back first, then proceeds.

Global recovery order is fixed: workspace lifecycle first under the workspace lock, then — if that was not a repository removal — repository operations in repository-name order under each stable repository lock. A workspace operation that has already passed the repository-removal commit decision takes precedence and forbids any nested repository recovery, since recovering state inside a repository that is being deleted would be meaningless.

Recovery is evidence-driven, not optimistic. Each phase has a defined rule:

This was validated by sending SIGKILL to sow add at many different moments. In every case status reported recovering, the next write command recovered that operation before executing its own, the final check passed all layers, and the public tree was never torn.

$ sow status
repository=pigsty status=recovering ready_to_copy=false ...

sow build is the one explicit forward-recovery entry point: it attempts to complete or roll back any decidable nonterminal operation before converging. If you see recovering, running sow build is the normal response.

error is reserved for the case where the journal, database, and file evidence contradict each other and no automatic choice is safe. Build refuses to overwrite; the last completed view keeps serving; you restore from backup and then run check and build. There is deliberately no repair --force, because a repair that guesses wrong is worse than a repair that refuses.

Fail-closed path safety

Managed paths are never assembled from user-supplied strings. Every create, rename, and delete follows the same sequence:

1. resolve the workspace root to an absolute real path;
2. reconstruct the target from a fixed relative fragment and verify the relative path contains no escape;
3. Lstat every existing controlled component and reject symlinks and unexpected file types;
4. delete only objects that were first atomically moved into .sow/.../recovery;
5. before deleting, prove again that the recovery target sits inside the corresponding private state directory.

Names must match [a-z0-9][a-z0-9._-]*, and ., .., .sow, pool, dists and workspace-reserved names are rejected outright.

The same posture applies to file handles. SQLite is opened with O_NOFOLLOW and bound to a regular-file inode, re-verified by path after the connection is established; a database, WAL, shm, or rollback journal that is a symlink, a non-regular file, multiply hardlinked, or rebound during the open is rejected. log export refuses to overwrite an existing file and refuses a symlinked parent directory — which is why exporting into /tmp on macOS fails, since /tmp is a symlink there.

Journals are bounded by size: 32 MiB for the workspace, 16 MiB for a repository operation payload, and 64 MiB each for the external mutation and base manifests. An oversized journal is never truncated and never degraded — it fails outside the commit window, so a writer can never produce an operation record that a recovery reader would be unable to read back.

None of this claims to defend against a malicious process running as the same user with unlimited privileges. It defends against the realistic failure modes: crashes, races between cooperating processes, and paths that changed shape between the check and the use.

Next

• Observability & Audit — reading the state these mechanisms maintain
• Exit Codes — what 4, 5, and 6 mean and when you see them
• build, status, check, and changes

7 - Observability & Audit

Each read surface answers a different question.

status: cheap state

sow status -r local

It reports the Desired revision, Built Generation, dirty Dists, pending payload counts, lock state, and ready_to_copy. It does not hash the public tree, recover an operation, or build.

Use status to diagnose. Do not use it as a substitute for check.

check: delivery proof

sow check -r local

In steady state, the checker reports nine ordered layers:

A non-terminal layout transition has a shorter diagnostic surface: config, state, public-modes, then layout-transition. The check stops there and returns not-ready until the diagnosed maintenance operation completes or is safely aborted before commit.

check writes and repairs nothing. A dirty or recovering Repository is not deliverable even if its last committed tree remains readable. Put check in the release pipeline and stop on any nonzero exit.

Every unique physical package payload receives one authoritative SHA-256 pass per check, regardless of cached fingerprints. Descriptor-bound evidence is then shared across retained records, indexes, signatures, the final Generation manifest, and changes. Signed RPM verification uses one additional shared stream for all signature packets and trust rings; it is not multiplied by the number of Dists, keys, or retained Generations.

changes: Generation difference

sow changes -r local
sow changes 0 -r local
sow changes 42 -r local --json

• no base argument compares the current Built Generation with its predecessor;
• base 0 describes the complete current public tree;
• base N gives the net difference from recorded Generation N to current Built.

Rows include operation, phase, Repository-relative path, size, and SHA-256. Phases use the same vocabulary as local construction: payload, metadata, pointer, delete.

changes is a manifest/difference surface. It does not contact a destination, persist a remote checkpoint, enforce cache grace, or recover an interrupted transfer. Use sow publish TARGET for a configured live target. For an offline copy, stage a complete tree, verify it, and switch it into service atomically.

Generation retention and GC

sow retain add 42 -r local
sow retain ls -r local
sow retain rm 42 -r local
sow gc -r local

retain add verifies and freezes a Generation’s metadata and reference sets; it does not copy another package tree. The retained record is an explicit GC root. retain rm removes that root but deletes no package bytes itself.

Local sow gc deletes only payloads proven unreachable from the current state, explicit retentions, active recovery/publication state, and other recorded roots. Target GC is a separate operation: sow gc TARGET uses that Provider’s safety model.

During ordinary builds, SOW carries forward the immediately preceding RPM immutable metadata and APT by-hash objects so a reader of the previous pointer can finish. This bounded protocol window is separate from explicit retain.

Operation log

sow log -r local
sow log OPERATION -r local
sow log export operations.jsonl -r local
sow log prune 2026-01-01 -r local

The ledger records operation kind/state, timestamps, configuration/manifest identities, package dispositions, membership changes, and physical changes where applicable.

log export writes stable JSONL, refuses to overwrite an existing file, and validates its output path. log prune accepts a date or RFC 3339 timestamp and removes only eligible terminal audit records; it does not remove current state, recovery evidence, or Generation manifests still required elsewhere.

Operational pattern

sow build -r local
sow check -r local
sow publish public

Use status for monitoring, check for the gate, publish for target mutation, and log for later evidence.

See also

• sow build and sow check
• Log commands
• sow publish and sow gc