Skip to content

This is the multi-page printable view of this section. .

Return to the regular view of this page.

Features

Choose a repository mode, then add package policies, signing, publication, and audit.

SOW creates repository indexes in two modes. Choose Plain for a directory you can rebuild. Choose Managed for ongoing package updates, policy, signing, and publication.

Capabilities

Capability Plain Managed
RPM and DEB indexes One directory, either or both formats One repository, separate Dists
Stored membership and generations — Yes
Per-architecture views — Yes, including neutral packages
Package exclusions Fixed cleanup with --pigsty Configurable per Dist
Version limits sow prune removes superseded RPMs Configurable per Dist
Metadata signing — RPM and DEB
RPM package signing --sign-with never, fill, always
Recovery after interruption Rerun create or prune Recorded operations and publication attempts
Operation log and JSONL export — Yes
Publication targets — Filesystem and R2

SOW parses packages and generates indexes itself. It does not require createrepo_c, dpkg-scanpackages, or reprepro. RPM package signing uses the host rpm and GPG environment. Metadata signing uses gpg when a key reference resolves to a GPG key identifier.

Generated formats

Format Plain Managed additions
RPM primary, filelists, other XML and repomd.xml Architecture views; optional repomd.xml.asc
DEB Packages, Packages.gz Release, by-hash indexes; optional InRelease and Release.gpg

RPM XML is gzip-compressed. SOW does not generate SQLite rpm-md, zchunk, modulemd, or source-package indexes. Use your HTTP server or object-storage service to serve the output. See Platforms and integrations for client details.

Learn a feature

Task Read
Generate indexes beside existing packages Plain repositories
Manage repeated package updates Managed workspaces
Share package bytes across Dist views Pool and views
Exclude packages or keep recent versions Membership policy
Sign packages and indexes Signing
Continue after an interruption Transactions and recovery
Inspect state, validate output, and read history Observability and audit

1 - Plain Flat Repositories

Build flat RPM and DEB indexes, repeat interrupted builds, and use signing or the Pigsty marker.

Use sow create to build indexes for a directory of RPM or DEB packages. It writes the indexes beside the packages. It does not read sow.yml or create a state database.

sow create /srv/offline

Inputs and outputs

SOW scans top-level regular files with .rpm or .deb suffixes. It does not recurse or follow package symlinks. Package names, versions, and architectures come from package headers. SOW rejects source RPMs (src and nosrc).

Input Generated files
RPM repodata/repomd.xml and gzip-compressed primary, filelists, and other XML
DEB Packages and Packages.gz
Both formats Both sets of indexes

RPM locations use the package basename. DEB Filename values use ./<basename>. SOW includes all valid versions. Different bytes with the same package coordinate cause a conflict. SOW rejects a directory with no supported packages, except when --pigsty finishes an interrupted cleanup.

By default, package files and unrelated files stay unchanged. If a format disappears, SOW removes its old generated indexes. For RPMs, it keeps the current and immediately previous checksum-named metadata sets. It removes older SOW-shaped metadata and replaces incomplete output. If repodata/ contains non-SOW files such as SQLite databases, modulemd, comps, or .asc signatures, sow create removes those foreign files and reports them in dropped_metadata (--json) and as a stderr warning. Generated directories use 0755; generated files use 0644.

Build and repeat

The normal unsigned build reads each selected package once to calculate SHA-256 and parse its metadata. --jobs controls this work and defaults to the logical CPU count. SOW renders and validates the indexes from those parsed facts.

Before replacing the indexes, SOW checks the package list, file identity, mode, size, and modification time again. Keep other programs from changing packages during the build. A changed snapshot stops publication with exit code 5.

Identical package bytes and options produce identical metadata. If the generated files already match, the command reports noop=true in text output or noop: true in JSON and leaves them in place.

RPM metadata uses revision 0 and data timestamp 0 by default. Set a publication time when updating an existing repository used by EL7 YUM:

sow create /srv/offline --metadata-timestamp 1791849600

The timestamp changes repomd.xml only. It does not change the compressed XML, package bytes, or DEB indexes. Maintain a timestamp that advances from the existing repository; see YUM migration.

A small .sow-plain-rpm-history.json file tracks RPM metadata retention. Unchanged builds keep the same previous index. Keep this file when copying a directory that you will rebuild. Package retention remains the operator’s responsibility. See RPM index retention.

Prune superseded RPMs

Use sow prune to keep only the highest EVR per RPM (name, arch) in a Plain directory:

sow prune /srv/offline --check
sow prune /srv/offline

sow prune compares versions with native rpmvercmp rules, including ~ pre-release and ^ post-release caret sorting. It updates repodata/ before removing superseded .rpm files so clients never observe metadata that points to missing packages. If the directory already contains repo_complete, sow prune refreshes that marker automatically. Unlike sow create, sow prune refuses to run when repodata/ contains foreign non-SOW metadata files.

Interrupted builds

SOW stages and checks metadata before it replaces public files. The replacements do not form one transaction across all RPM and DEB files. An interruption can leave mixed or incomplete indexes.

Run the same sow create command again. It removes its stale temporary files and rebuilds from the packages currently in the directory. Plain mode has no operation journal; the JSON field recovered is always false.

For a live service, build in a separate directory and switch the complete directory into service after the command succeeds. Use Managed mode when you need recorded membership, generations, and transaction recovery.

Pigsty cleanup and completion marker

sow create /srv/offline --pigsty

--pigsty removes DEB i386 packages and Patroni 3.0.4 packages according to parsed package facts. It does not remove an RPM solely because its architecture is i386, i486, i586, or i686.

SOW retains documentation packages such as annobin-docs unless they match those cleanup rules.

After updating indexes and removing the selected packages, SOW writes repo_complete. Each line contains a SHA-256, two spaces, and a retained package basename, sorted by basename.

SOW removes an existing marker before replacing public files and writes the new marker last. Consumers that use this convention must wait for the marker. If the build stops without a marker, rerun with --pigsty. A plain invocation without --pigsty refuses to rebuild a directory that already contains repo_complete.

RPM package signing

sow create /srv/offline --sign-with 6D5C5A26C36B1F73

This option signs RPMs that have no parseable embedded signature. Add --overwrite to re-sign every retained RPM. It requires the host rpm command and a usable GPG key. SOW signs and checks staged copies, then replaces the package files before their indexes. This path reads package bytes more than once.

For a protected key, pass --passfile PATH or set SOW_GPG_PASSFILE. --sign-with is still required to enable signing. See Password files for the exact file format; Plain mode does not read the Managed sow.yml configuration.

Plain mode does not sign repository metadata. See Signing for Managed metadata signing and package-signing policy.

Concurrent commands

sow create locks the target directory and its parent. Use -N to fail immediately if another SOW writer holds the lock, or -T 30s to wait up to 30 seconds. The default waits until the lock becomes available.

See sow create for all options and result fields.

2 - Managed Workspaces

Workspace structure, package membership, configuration, discovery, and initialization.

Use Managed mode for repositories that receive regular updates. SOW records package membership, applies Dist policy, builds metadata, and keeps operation history.

Workspace, repository, and Dist

Workspace
└── Repository
    └── Dist
        └── Architecture view
Term Purpose
Workspace Holds sow.yml, private .sow/ state, and the allowed architecture families
Repository Owns one package pool, Dists, a SQLite database, generations, and publication state
Dist Selects a named set of packages in one format: rpm or deb
Architecture view Presents a Dist’s packages to clients of one architecture

A repository can contain RPM and DEB Dists. Dist names such as el9, trixie, and el9-beta are labels you choose; they do not inspect operating-system compatibility or create a promotion workflow.

One package can belong to several Dists in the same repository without another payload copy. An architecture-neutral package has one membership per Dist and appears in each applicable architecture view. Separate repositories keep separate package objects.

Public files and private state

  • <workspace>/
    • sow.yml
    • .sow/
      • workspace.lock
      • workspace-ops/
      • repo-locks/<repo>.lock
      • <repo>.db
      • <repo>/
        • stage/
        • recovery/
        • pending/
        • retained/
        • transitions/
    • <repo>/
      • pool/
      • dists/

Serve or publish the complete <workspace>/<repo>/ directory. Keep sow.yml and .sow/ private. Back up configuration, private state, and package files together when you need to preserve a working workspace. See Repository layout.

Create and configure

sow init /srv/repo
sow repo new demo -C /srv/repo
sow dist new el9 --format rpm -r demo -C /srv/repo
sow dist new trixie --format deb -r demo -C /srv/repo

The initial configuration uses schema: sow/v3 and architecture families [x86_64, aarch64]. init creates no repository unless one is already declared in the configuration. Repository and Dist lifecycle commands update sow.yml for you.

Edit sow.yml to set architecture restrictions, membership policy, signing, or publication targets. Then validate the configuration:

sow config check -C /srv/repo
sow config show --all -C /srv/repo

The parser rejects unknown fields. config check also compares configuration with initialized state and resolves signing references. config show --all expands defaults. Configuration accepts amd64 and arm64 as aliases for x86_64 and aarch64. DEB indexes use the native names amd64 and arm64.

To prevent removal of a repository, set protected: true on it. This blocks repo rm, including -f, while package and Dist changes remain available.

Desired state and built state

Desired state records which packages should belong to each Dist. Built state records the last completed metadata generation.

By default, add and rm update membership and build the affected output. With --skip, they update membership without building. New package bytes stay in private pending/ until a build promotes them to the pool.

sow add ./incoming/ -C /srv/repo -r demo -d el9 --skip
sow status -C /srv/repo -r demo
sow build -C /srv/repo -r demo
sow check -C /srv/repo -r demo

A dirty repository needs a build. Its public tree still represents the previous built generation. Policy, architecture, or signing changes can also make a Dist dirty. SOW rejects removing an architecture that membership or built state still references. If a Dist or architecture is removed after being published to a target, a full-repository sow build -r REPO must produce a new complete Built Generation before sow publish withdraws the removed target pointers.

Every new Dist starts with valid empty indexes for its architectures. Metadata signing, when configured, applies to empty Dists too. Removing the last member leaves empty indexes.

Find the workspace

Managed commands search for the nearest ancestor sow.yml:

  1. Start at -C/--workdir DIR, if supplied; otherwise start at the current directory.
  2. If no workspace is found, search from $SOW_DIR.
  3. If neither search succeeds, report a discovery error.

SOW uses the first configuration it finds, even if that file contains an error. --workdir changes workspace discovery, not the process directory. Relative input paths still refer to your current directory:

sow add ./build/example.rpm -C /srv/repo -r demo -d el9

A symlink can be a search starting point, but the search follows the supplied path’s ancestors. A symlink within the workspace prevents implicit repository or Dist selection. Use the physical workspace path and explicit -r and -d when needed.

Plain sow create does not use workspace discovery.

Select a repository and Dist

Repository selection uses the first available choice:

  1. Explicit -r/--repo NAME.
  2. The repository containing the discovery starting directory.
  3. The only repository in the workspace.

add, rm, and ls select Dists in this order:

  1. Explicit -d options; repeat the option to select several Dists.
  2. The Dist containing the discovery starting directory.
  3. The only Dist in the selected repository.

If more than one choice remains, the command requires an explicit selection.

build, check, and status default to all Dists in the selected repository. show searches that repository; where searches all repositories. Use -d to narrow commands that accept it. See Commands for each command’s selection rules.

Repeat initialization or upgrade

Rerunning init fills in missing declared repositories and Dists. It checks existing initialized state without resetting generations. After adding an architecture to an existing Dist, run build to create its new view.

If initialization commits some objects and then fails, it returns exit 3 and reports what completed. Resolve the error and rerun init.

SOW 0.7.1 uses database schema v13, as do 0.5.0 through 0.7.0. Upgrading from these versions needs no schema migration. For a 0.3 or 0.4 workspace, back up the complete workspace, then run sow repo migrate. Do not edit the SQLite schema version manually.

The database also caches parsed package facts. Warm builds can reuse those facts and file fingerprints; sow check reads and hashes package bytes for a full consistency check.

See Policy, Views, and Transactions and recovery for the next steps.

3 - Pool & Metadata Views

Share package files across RPM and APT views, export RPM leaves, and deliver complete repositories.

A Managed repository stores each package object once under pool/. Its Dists and architecture views contain indexes that refer to those package bytes.

<repo>/pool/                              package files
<repo>/dists/<rpm-dist>/<arch>/repodata/   RPM metadata
<repo>/dists/<deb-dist>/main/binary-*/     APT metadata

The storage boundary is one repository. The same package in another repository or publication target occupies that repository’s or target’s namespace.

RPM views

An RPM Dist has one metadata directory per configured architecture:

  • demo/
    • pool/
      • e/example/example-1.0-1.x86_64.rpm
      • d/demo-data/demo-data-1.0-1.noarch.rpm
    • dists/el9/
      • x86_64/repodata/
        • <sha256>-primary.xml.gz
        • <sha256>-filelists.xml.gz
        • <sha256>-other.xml.gz
        • repomd.xml
      • aarch64/repodata/
        • ...

The x86_64 view includes x86_64 and noarch packages. The aarch64 view includes aarch64 and noarch packages. Both views refer to the same noarch file.

A package location is relative to the architecture directory:

<location href="../../../pool/d/demo-data/demo-data-1.0-1.noarch.rpm"/>

Point the client at dists/el9/x86_64/, and serve the complete demo/ directory so that the relative package location resolves. sow check verifies that indexed locations stay inside the repository and resolve to the expected package objects.

APT views

APT indexes use paths relative to the repository root:

Filename: pool/e/example/example_1.0-1_amd64.deb

Each configured architecture has Packages, Packages.gz, and SHA-256 by-hash entries under dists/<dist>/main/binary-<arch>/. The Dist root contains Release and, when signed, InRelease and Release.gpg.

all packages appear in each applicable architecture index while their payload remains one pool object. SOW creates no per-view package copies or per-architecture Release files.

Export for dnf reposync

Default dnf reposync rejects Managed RPM locations that climb above its per-repository download directory. Create a standalone leaf for that workflow:

The RPM Dist must belong to the current Built Generation and be clean. A clean RPM Dist can be exported even when a later build updated only another Dist, because SOW carries its unchanged view into the current Built Generation. Check the export preconditions before running this command.

sow export rpm-leaf el9 x86_64 /srv/exports/el9-x86_64 -r demo

The export has local package paths, its own repodata, a manifest, and a .sow-export.json completion marker. Choose a destination outside the repository and all configured filesystem publication targets’ prospective roots (endpoint + prefix).

Warning

The default copies files. --hardlink requires the same filesystem and makes exported packages share the source inode; keep those files read-only. An export does not create Dist membership or a retained generation and is not input to sow publish.

Deliver the complete repository

Use a configured publication target for live updates. When a Dist or architecture is removed, run a full-repository sow build -r REPO before publishing so the current Built Generation covers the whole repository and allows sow publish to withdraw the removed target pointers safely. For another transport, copy the complete settled pool/ + dists/ tree into a separate staging directory, verify the copy, then switch it into service. Keep package URLs referenced by cached indexes reachable until those caches expire. See Serving for a complete transfer procedure. Copying only an RPM architecture directory leaves its package locations unresolved.

sow changes 0 lists each package under pool/ once, followed by metadata and client pointers. It lists local files; sow publish manages target progress and cache handling.

See Repository layout, Serving, and export.

4 - Membership Policy

Configure exclusions and version limits, read policy results, and restore removed members.

Use Dist policy to exclude unwanted packages and limit the number of versions in indexes. Set exclude and limit in sow.yml:

dists:
  el9:
    format: rpm
    limit: 2
    exclude:
      - kind: [debuginfo, debugsource, llvmjit]

SOW applies exclude first, then limit. add, rm, and build apply policy to the complete candidate membership of each selected Dist. Edit the policy, run sow config check, then run sow build to apply it to existing members.

Exclude packages

Each rule contains one or more fields. All fields in a rule must match. Any pattern within a field can match. SOW excludes a package when any rule matches.

exclude:
  - kind: [debuginfo, debugsource, dbgsym, dbg, llvmjit]
  - name: ["test-*", "*-experimental"]
    arch: [aarch64]

This excludes the listed package kinds on every architecture. It also excludes aarch64 packages whose names start with test- or end with -experimental.

Field Package value
name Binary package name
source Normalized source package name
arch x86_64, aarch64, or neutral
kind Classification in the table below
format rpm or deb

Patterns are case-sensitive exact strings or shell globs: *, ?, and [...]. SOW does not support regular expressions, version comparisons, or negation. Empty rules, empty or untrimmed patterns, duplicate patterns within a field, and invalid globs are errors.

SOW derives kind from the binary package name:

Format Name suffix kind
RPM -debuginfo debuginfo
RPM -debugsource debugsource
RPM -llvmjit llvmjit
DEB -dbgsym dbgsym
DEB -dbg dbg
Either No matching suffix main

Use sow show PACKAGE --json to inspect the computed kind. SOW does not check pattern values against this table: kind: [debug] is valid YAML, but it matches no package kind.

Limit versions

limit groups packages by binary name and canonical architecture family. Neutral packages (noarch or all) form their own group.

Value Result
0 Keep all versions; this is the default
Positive N Keep the newest N versions in each group
Negative value Configuration error

RPM versions use native rpmvercmp epoch-version-release ordering, including ~ pre-release and ^ post-release caret operators (2.0~rc1 < 2.0 < 2.0^20250611 < 2.0.1). DEB versions use native Debian version ordering. SOW never sorts versions as plain strings.

With limit: 2, adding version 3 to a Dist that contains versions 1 and 2 removes version 1 from membership in the same operation. An x86_64 package and a noarch package count in separate groups, even when they share a binary name.

Read policy results

An input excluded from every selected Dist has item status excluded. Its per-Dist result is excluded for an exclusion rule or limited for the version cap. Policy exclusion does not count as a failed input and does not make the command fail.

One Dist can accept a package while another Dist excludes it. If no Dist accepts a new package, SOW does not store an unreferenced package object for it.

Restore a removed version

Relaxing policy does not restore previous members. Raising limit, removing an exclusion, or deleting the newest version only changes the rules for current and future candidates. To restore a version, add its package file again:

sow add ./archive/example-1.0-1.x86_64.rpm -r demo -d el9

Policy removes membership, not package bytes. A package can remain in the pool because a built or retained generation still needs it. Use sow gc to reclaim unreferenced objects. Use sow gc TARGET for a published target.

Preview removal

sow rm example -r demo -d el9 --check

The preview reports membership removals and predicted file changes without changing repository state. Check prediction_complete in JSON before treating the file list as complete. Do not combine --check with --skip, --timeout, or --no-wait.

See Configuration for the full schema and sow add for input results.

5 - Signing Model

Configure metadata and RPM package signing, reference keys, and rotate signing identities.

Configure metadata signing to let clients verify repository indexes. Configure RPM package signing separately when clients also need to verify signatures embedded in RPM files.

What gets signed

Setting Signed content Output Client setting
signing.rpm.metadata RPM index manifest repodata/repomd.xml.asc DNF repo_gpgcheck=1
signing.deb.metadata APT release manifest InRelease, Release.gpg APT Signed-By
signing.rpm.packages RPM package Embedded OpenPGP signature DNF gpgcheck=1

Metadata signing leaves package bytes unchanged. RPM package signing changes package bytes and therefore their SHA-256. SOW does not add signatures to DEB package files.

Configure metadata signing

repos:
  demo:
    signing:
      rpm:
        metadata:
          key: file:///secure/repo-signing.asc
      deb:
        metadata:
          key: file:///secure/repo-signing.asc
    dists:
      el9:
        format: rpm
      trixie:
        format: deb

RPM and DEB can use the same key or different keys. For a protected key, add passfile: /run/secrets/gpg-password next to key, or set SOW_GPG_PASSFILE to the password-file path. Validate the settings, then build:

sow config check
sow build -r demo
sow check -r demo

Signed empty Dists receive the same signature files as populated Dists. The signed body of InRelease matches Release. Without a configured key, SOW produces unsigned indexes. Managed keys and signing policy come from sow.yml. --passfile PATH overrides the password source for one invocation, including build.

Key and passphrase references

Metadata key reference Meaning Signing implementation
keys/repo.asc OpenPGP key file relative to the workspace root In-process
file:///secure/repo.asc OpenPGP key file at an absolute path In-process
env://SOW_METADATA_KEY Armored key material, file path, or GPG key identifier Depends on the value
agent://KEY_ID_OR_FINGERPRINT Key available through the local GPG environment External gpg

Key files support ASCII-armored and binary OpenPGP data. A GPG key identifier is a 16-digit key ID or a 40- or 64-digit fingerprint. References that resolve to a GPG key identifier use external gpg for private-key access. An explicit password file enables noninteractive unlocking through GPG loopback mode.

A signing key reference must resolve to exactly one primary key. A trusted_keys file can contain several complete public keys, including consecutive armored blocks. SOW reads the complete bundle and rejects malformed or truncated input. It does not silently use only the first key.

Use --passfile PATH, per-key passfile, or SOW_GPG_PASSFILE, in that priority order. Legacy metadata passphrase references remain supported at the per-key priority; passfile and passphrase cannot both be nonempty on one key. All three signing settings accept passfile, including agent:// keys.

Password files contain one nonempty UTF-8 line. SOW removes exactly one final LF or CRLF and preserves spaces and tabs. It does not search a default path or fall back from an invalid selected source. See Password files for the full format, relative-path rules, permissions, and recovery command coverage.

config show displays references and resolved public identities, not private-key or passphrase contents. config check unlocks metadata key files and checks their suitability; external GPG metadata keys also perform a signing probe. RPM package signing checks credential format, tools, and secret-key availability, without signing a package or proving the password correct. See config check. For new metadata signatures, the selected signing key must be usable at the publication time and at the current time. Historical verification uses the retained certificate.

New APT signatures use SHA-256. A new publication requires SHA-256, SHA-384, or SHA-512 in both APT signature files. If upgrading a workspace with SHA-1 metadata signatures, run build before starting a new publication. An existing publication attempt resumes its recorded files.

Configure RPM package signing

signing:
  rpm:
    packages:
      mode: fill
      key: agent://7F721C4AD40F4A9D8CA578BFAC7E4690B50CCF3B
      trusted_keys: [keys/upstream.asc]
Mode Behavior
never Store the input bytes unchanged
fill Keep a package whose signature verifies with the configured key or trusted keys; sign other packages
always Keep a package already signed by the configured key; re-sign other packages

With no key, the default mode is never. With a key, the default is fill. Both fill and always require a key.

Package signing uses the host rpm --addsign or rpm --resign command on a private copy. Install rpm and gpg, and make the matching private key available in the GPG environment used by rpm. A key reference identifies the signer; it does not import a private key into that environment. SOW leaves the original input file unchanged.

SOW checks the result’s signature, package coordinate, and signature-neutral payload digest. If an existing object has the same coordinate and payload and satisfies the current policy, SOW can reuse its signed bytes. In never mode, reuse requires an exact byte match.

Use RPM format v4 for package-signing workflows. SOW verifies RSA, DSA, ECDSA, and EdDSA RPM signatures, including header-only signatures stored in RPM signature header tag 267 (RPMSIGTAG_DSA). SOW does not support the signature tag used by RPM format v6. This is separate from OpenPGP packet versions; SOW supports historical CentOS OpenPGP v3 and v4 signatures.

Change keys

Changing only a passfile path does not make Dists dirty. A changed signing identity does. For metadata keys, run build to create signatures with the new identity. Distribute the new public key to clients before publishing the new metadata.

Existing pool objects are immutable. Changing RPM package policy does not re-sign them in place. For a staged package-key rotation:

  1. Set the new key as current and use mode: fill.
  2. Keep the old public key in trusted_keys while old packages remain members.
  3. Add new package releases signed by the new key.
  4. Remove old trust after you withdraw or replace the old signed packages.

Switching directly to always with a new key requires every Desired RPM to satisfy that key policy. If changed signatures would replace an existing package URL with different bytes, publish a new package release and filename.

RPM verification evaluates certificate and signing-subkey validity at the package’s signature time. Renewing a certificate does not itself invalidate old signatures when the required historical self-certifications and subkey bindings remain in the certificate. Keep that historical public-key material when exporting renewed keys.

Plain signing

sow create /srv/offline --sign-with 6D5C5A26C36B1F73 --passfile /run/secrets/gpg-password
sow create /srv/offline --sign-with 6D5C5A26C36B1F73 --overwrite --passfile /run/secrets/gpg-password

Plain mode signs RPM package files only. Without --overwrite, it signs packages with no parseable embedded signature. With --overwrite, it re-signs all retained RPMs. The key is 16, 40, or 64 hexadecimal characters, with an optional 0x prefix. At least one RPM must remain after any --pigsty cleanup.

See Sign your repository for client setup and Configuration for all signing fields.

6 - Transactions & Recovery

Recover interrupted Managed operations and understand local build order and lock behavior.

Managed mode records changes before it applies them. After an interruption, SOW uses these records to finish or roll back the operation. The recovery command depends on the operation that stopped.

Recover an interrupted operation

Start with the recorded state:

sow status -r demo
sow log -r demo
State Next step
clean Run sow check before delivery
dirty Run sow build to apply pending membership or configuration changes
recovering Use the recovery command for the interrupted operation, listed below
error Inspect the reported file or operation in the diagnostic and log
Interrupted operation Recovery command
add, rm, Dist change, or build sow build -r demo
init, repo new, or repo rm Rerun the corresponding lifecycle command
Publication sow publish TARGET
Local or target garbage collection sow gc -r demo or sow gc TARGET
Retained generation removal sow retain rm GENERATION -r demo
Plain index generation or RPM pruning Rerun sow create or sow prune with the original options

SOW selects an active repository mutation by its pending state rather than its recorded timestamp, so system clock adjustments never hide an unfinished operation behind an older completed record. When local or target gc or retain rm stops after freezing its plan, SOW preserves that plan so rerunning the command finishes the interrupted cleanup.

Recovery that must regenerate signed metadata needs access to the signing key and password again. SOW does not store password values in journals. Use --passfile PATH, the per-key configuration, or SOW_GPG_PASSFILE; see Password files for availability and priority. For a pending Workspace lifecycle journal, supply a CLI/environment override without editing the recorded sow.yml, whose digest is part of recovery validation.

First correct any reported input, configuration, or storage error. Keep the workspace configuration, database, and private state together during recovery. After a local recovery, run sow check -r demo. See Troubleshooting for common errors and publish for publication recovery.

What SOW records

Record Location Purpose
Workspace lifecycle journal .sow/workspace-ops/active.json Coordinates configuration with repository creation or removal
Repository operation journal .sow/<repo>.db Records local membership changes, builds, and maintenance
Publication state .sow/<repo>.db Records target bindings, frozen publication plans, and progress

Workspace lifecycle recovery compares the current sow.yml digest with the journal’s old and new configurations. The configuration replacement decides whether to roll back or finish that lifecycle change. A digest that matches neither configuration causes an integrity error.

Repository mutations record their own state transitions:

planned → staged → applied → built → done
                       └────────────→ done_dirty
State Meaning
planned SOW recorded the operation and its intended scope
staged SOW prepared the inputs for the next stage
applied SOW committed the Desired state; the previous public generation can still be active
built SOW installed the generated public tree
done The operation is complete
done_dirty SOW committed the Desired state and left a build pending

Other terminal outcomes include rolled_back and failed. Read transitions and timestamps with sow log OPERATION -r demo.

--skip normally ends at done_dirty. A recognized signing-policy change, configuration change before a build plan freezes, or supported contract upgrade can also leave a committed operation dirty. The log records the reason. Correct the configuration if needed, then run build again.

Errors after a committed change

Database commit and later maintenance are separate steps. A checkpoint or cleanup error can occur after package membership or a build has committed. SOW keeps the committed result in command output. It does not report those items as rolled back.

Read the result and sow status, then inspect the recorded operation with sow log. Resume the command named in the diagnostic. Keep original input files until you have confirmed the result.

Local build order

SOW stages and validates output on the same filesystem, then applies it in this order:

package bytes → metadata → client pointers → obsolete metadata cleanup
  1. Put package bytes in the canonical pool/ path.
  2. Install checksum-named RPM metadata and APT direct and by-hash indexes.
  3. Install RPM repomd.xml and configured signatures, or APT Release, InRelease, and Release.gpg as applicable.
  4. Remove obsolete metadata after the replacement is durable.

The build keeps the immediately preceding RPM immutable metadata and APT by-hash objects available for readers of the previous pointer. Explicit generation retention is separate; see retain.

A dirty repository still serves its last built generation. A multi-Dist operation can switch views at different times. Remote HTTP requests and caches also have their own lifetimes; use sow publish for live target updates. sow changes lists local file changes and does not perform publication or target cleanup.

Locks and waiting

SOW uses advisory locks to coordinate its processes. Workspace lifecycle uses .sow/workspace.lock; repository writes use .sow/repo-locks/<repo>.lock. When a command needs both locks, it acquires the workspace lock first.

Read commands that need a consistent snapshot use shared locks. status probes the repository writer lock so it can report activity without waiting for that writer. Plain create and prune lock their target directory and parent.

Option Behavior
-T 30s Wait up to 30 seconds for a write lock
-T 0 Wait until the lock is available; default
-N Try once; exit immediately if another writer holds the lock

A failed lock attempt exits 4. Combining -N with a nonzero -T exits 2. Use only the lock options accepted by the command; read-only commands do not accept them.

Filesystem requirements

Use a local POSIX filesystem with advisory locks, fsync, and atomic rename. On macOS, use APFS. Keep public files and private staging on the same filesystem. SOW rejects symlinked control paths, unexpected file types, and path escapes. It checks file identity before replacing or removing controlled objects. Do not place a workspace on a network filesystem.

Repository and Dist names use [a-z0-9][a-z0-9._-]*; Repository layout lists reserved names. For script responses to interruption, lock contention, or integrity errors, see Exit codes.

7 - Observability & Audit

Inspect state, verify repository content, compare generations, and read operation history.

Use status to inspect state, check to verify a repository, changes to list file changes, and log to read operation history.

Command Answers
status Does the repository need a build or recovery? Is a writer active?
check Are state, package bytes, indexes, and signatures consistent and ready for delivery?
changes Which local files changed between built generations?
log Which operations ran, and what did they change?
retain ls Which generations are explicitly retained?

These commands do not modify repository contents.

Inspect state

sow status -r demo

The result includes Desired revision, built generation, dirty Dists, pending package count and bytes, writer-lock state, and ready_to_copy.

State Meaning
clean Desired and built state agree
dirty Membership or configuration needs a build
recovering Recorded work needs recovery or cleanup
error Recorded state or file evidence has an unresolved error

status reports state without hashing the public tree. For dirty, run build. For recovering or error, follow the diagnostic and recovery procedure. Run check before delivery.

Verify the repository

sow check -r demo

A normal check reports nine layers:

Layer Checks
config Configuration and effective Dist settings
state SQLite schema and relational state
public-modes Public file and directory permissions
retained Retained generation records and metadata
package-bytes Pool and pending bytes against recorded SHA-256
desired-membership Package identity, architecture, and membership
index Index contents and referenced files
signature Configured metadata and RPM package signatures
generation-manifest Built file manifest against the public tree

An unfinished layout transition instead reports config, state, public-modes, and layout-transition. Complete or abort that migration as directed by repo migrate.

check reads and hashes each unique physical package regardless of cached fingerprints. It does not repair files. A dirty or recovering repository fails readiness even if its previous built tree remains consistent.

Compare generations

sow changes -r demo
sow changes 0 -r demo
sow changes 42 -r demo --json

No base compares with the previous built generation. Base 0 lists the complete current tree. Base 42 compares recorded generation 42 with the current one. Rows contain the action, phase, repository-relative path, size, and SHA-256 as applicable.

This comparison does not inspect a publication target. Use sow publish TARGET to update that target and resume interrupted transfers. Use target GC to reclaim remote objects.

Retain generations and collect unused packages

retain add accepts the current built generation only. If status reports generation 42, retain it before a later build replaces it:

sow retain add 42 -r demo
sow retain ls -r demo

The command verifies the generation and saves its metadata and references. Package files remain in the shared pool. When you no longer need generation 42, remove its retention record with sow retain rm 42 -r demo. Then run sow gc -r demo to collect unused bytes. If retain rm or gc stops mid-execution, rerun the same command to resume its recorded plan. Local gc requires a clean repository; build pending changes first. It preserves packages needed by Desired, built, and previous-client membership records, retained generations, recovery, or publication state. See gc for the complete rules.

The previous-generation metadata kept during ordinary builds is separate from explicit retention. For target cleanup, run sow gc TARGET. Filesystem targets support conditional deletion after cache grace; R2 targets report candidates without deleting them.

Read and export the log

sow log -r demo
sow log OPERATION -r demo
sow log export operations.jsonl -r demo
sow log prune 2026-01-01 -r demo

The log records operation state, times, package results, membership changes, and file changes where applicable. An operation detail can include build-progress events.

log export writes JSON Lines to a new file or stdout. log prune accepts a date or RFC 3339 timestamp and removes eligible terminal audit records. Required recovery records and generation manifests remain available.

See JSON output for scripts and Transactions and recovery for interrupted operations.