Pull mirror of github.com/smilinTux/skstacks (canonical)
  • Python 62.5%
  • Jinja 26.9%
  • Shell 6.7%
  • HCL 2.5%
  • HTML 1.1%
  • Other 0.2%
Find a file
chefboyrdave2.1 b01a24fb50
chore: forward-merge release/v2.26.7 (skstacks-v2.26.7) into main (#184)
* [target v2.26.2] skgraph, skvector: opt-in local-disk data on a pinned node (#149)

* test(skgraph,skvector): local-disk data knobs, red first

Render, guard and directory-task tests for skgraph.DATA_NODE/DATA_PATH and
skvector.DATA_NODE/STORAGE_PATH/SNAPSHOTS_PATH, plus baselines of the
compose render with the render gate example vars, generated from the
skstacks-v2.26.1 templates before any change, so unset knobs are proven
byte-identical to the release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skgraph,skvector): opt-in local-disk data on a pinned node

Same design as skhub DATA_NODE (#139). skgraph.DATA_NODE/DATA_PATH and
skvector.DATA_NODE/STORAGE_PATH/SNAPSHOTS_PATH: a path outside /var/data
is local disk, created on DATA_NODE (not recursive, same owner/mode as the
shared-storage copy), the old /var/data path is no longer created, and the
play stops if a local path is set without DATA_NODE. DATA_NODE appends
node.hostname == <DATA_NODE> after the existing constraints (skvector does
not add it twice). skvector named volume devices follow the paths. Unset
knobs render byte-identical to v2.26.1. New READMEs with the key table and
the migration procedure; render gate set-pass vars for both stacks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* release: skstacks-v2.26.2 (changelog)

Date the v2.26.2 changelog section before the release-gate run, so the
commit the gate tests is the commit that gets tagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [target v2.26.3] skhub: post-deploy URLs use the router hosts; TURN_DOMAIN reaches talk-hpb (#151)

* fix(skhub): post-deploy URLs follow SKHUB_HOSTNAME / COLLABORA_HOSTNAME

notify_push:setup, notify_push base_endpoint, whiteboard collabBackendUrl,
Talk spreed signaling_servers and richdocuments wopi_url were hardcoded to
skhub./collabora.<CLUSTERNAME>.<DOMAIN>, while the Traefik routers and
OVERWRITEHOST use SKHUB_HOSTNAME / COLLABORA_HOSTNAME. On an instance
whose public hosts lack the cluster label, every deploy pointed all five
at names that did not resolve or hit another route (Talk HPB "Unknown
error"). They now use the same keys, falling back to the old pattern, so
with the keys unset every line renders byte-identical to v2.26.2. README
"Public hostnames" and an upgrade note: set the keys before the next
skhub deploy when the public host differs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(skhub): post-deploy URLs use the router hosts; TURN_DOMAIN reaches talk-hpb

Broaden the fix: the five post-deploy settings now use the exact host
expressions of the Traefik routers and OVERWRITEHOST (SKHUB_HOSTNAME,
default skhub[-dev|-staging].<base domain>; COLLABORA_HOSTNAME, default
collabora.<base domain>; base domain honours CLOUDFLARED), so a
CLOUDFLARED instance needs no new vault key. A prod instance without
CLOUDFLARED and without the keys still renders byte-identical to v2.26.2.

talk-hpb.env renders TURN_DOMAIN when set; #145 only fed it to Nextcloud
turn_servers, so each deploy rebuilt talk-hpb without it and Janus fell
back to NC_DOMAIN for TURN. Unset renders as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [target v2.26.3] skhub: default talk-hpb to a digest-pinned current aio-talk (#152)

aio-talk:20260122_105751 ignores TURN_DOMAIN and writes no TURN server
into janus.jcfg, so Janus offered only its container address and calls
failed ICE on an instance that fell back to the default. Default to the
pinned current image (its start.sh writes turn_server and still writes
the relay_ipv4_addr line the relay wrapper rewrites). It is dinit-based,
so TALK_HPB_CMD defaults to its own CMD when TALK_HPB_IMAGE is unset; a
pinned TALK_HPB_IMAGE keeps the old supervisord default.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* release: skstacks-v2.26.3 (changelog)

Date the v2.26.3 changelog section before the release-gate run, so the
commit the gate tests is the commit that gets tagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [target v2.26.3] skhub: talk-hpb relay CMD follows the image, not the pin (#153)

#152 chose dinit only when TALK_HPB_IMAGE was unset, so an instance that
pins TALK_HPB_IMAGE to the new (dinit-based) default digest without
TALK_HPB_CMD would render supervisord and crash-loop. Key the default on
which image runs: dinit for the default digest (unset or pinned to it),
supervisord for any other pinned image. An explicit TALK_HPB_CMD wins.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* [target v2.26.3] skhub: talk-recording one replica; ECH off in the recording Firefox (#154)

Several talk-recording replicas behind one service VIP split the start and
stop of a recording across replicas ("Trying to stop unknown recording",
orphaned recorder). One recording server runs concurrent recordings
itself and the image has no concurrency setting, so replicas is fixed at
1 and TALK_RECORDING_MAX_CONCURRENT is ignored with a deploy warning.

The recording Firefox intermittently sent the ECH outer SNI to the
instance's own Traefik on a Cloudflare-proxied host with an ech= HTTPS
record, got the default certificate and failed with InsecureCertificate.
Render a locked policies.json (ECH, HTTP/3 ECH and native HTTPS-record
lookups off) and bind-mount it read-only into the Firefox distribution
dir. Always on.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(sksso): postgres healthcheck as the configured user; empty data paths mean unset (#157)

The postgres healthcheck ran pg_isready -U postgres. It passed, but on an
instance whose database user is not postgres every probe made the server
log FATAL: role "postgres" does not exist every 10 seconds. It now probes
with the container's own POSTGRES_USER and POSTGRES_DB ($$, never rendered
into the compose file) when sksso.postgres_user is not postgres; with
postgres the render is byte-identical to skstacks-v2.26.3.

The local-disk knobs (DATA_NODE, POSTGRES_DATA_PATH, REDIS_DATA_PATH) have
existed since v2.23.0. An empty data path now means unset, as for skgraph
and skvector; before it rendered a bind with no source and the guard
stopped the play with a misleading message. Unset or empty knobs render
byte-identical to v2.26.3, tested against fixtures rendered from the
v2.26.3 template before this change.

README "Migrating a running instance" rewritten for moving a live instance
off NFS: stop order, rsync -aHAX --numeric-ids, verify by regular-file
count and byte sum (not du --apparent-size), deploy, rename the old dirs
to *.STALE-moved-to-<node>-local-<date>, never delete.

CHANGELOG: skstacks-v2.26.4 section, carrying the skbackup not-release-gated
disclosure forward.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skhealth): lightweight host-level health checks with alerts [target v2.26.5] (#159)

* feat(skhealth): lightweight host-level health checks with deduplicated alerts

New optional role v1/ansible/optional/skhealth: one stdlib Python script on
a systemd timer (every 5 min) on every swarm node, no daemon or container.
Checks swarm replicas and crash loops, nodes, filesystems and hung NFS
mounts, load/memory/PSI, OOM kills, failed units, SMART and NVMe wear,
ZFS/md RAID, remote read-only hosts over a forced-command SSH wrapper, and
TLS expiry. Telegram and/or webhook alerts on state change, re-alert,
RECOVERED. Verification pending (full suite, lab gate).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(single-manager): a documented every-node exemption for node-local agents

skhealth runs on every swarm node by design. The new `# skstacks: every-node`
marker exempts it from the one-selected-manager rule only with a reason and
only while the playbook stays node-local: no /var/data, no docker stack or
service commands (own test).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skpayment): optional role for the SKPayment Stripe gateway [target v2.26.5] (#161)

* feat(skpayment): framework optional role for the SKPayment Stripe gateway

New optional role v1/ansible/optional/skpayment reproducing the
hand-deployed gateway: digest-pinned Python base image, app code fetched on
the controller at a pinned APP_REF and verified against APP_SHA256 (fail
closed), Stripe keys and apps.json from the vault in 0600 files, data path
and data node knobs, Traefik router from the framework host expressions,
MIDDLEWARES knob defaulting to default-no-crowdsec@file (flagged in the
README). Verification pending (full suite, lab gate).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(skpayment): image pin visible to list_images; guarded restart form

The compose template writes the default image digest out (list_images.py and
the image canary resolve only inline defaults), kept equal to the playbook
skp_default_image by a test. The restart after a changed code/env/apps.json
uses the standard "preexisted <svc> && ... || true" form that the
first-install guard test checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* sksec: LAPI allowlist sync from ALLOWLIST_CIDRS; per-router TRAEFIK_MIDDLEWARES knobs (#160) (#162)

* feat(v1): per-router TRAEFIK_MIDDLEWARES knobs for ten optional services

skgit, skorch, skpdf, skpeek, skport, skseek, skstor, skvector (HTTP and
gRPC), skpulse and skboard hardcoded their router chain to
default-no-crowdsec@file (skpdf and skvector with their own upload or
buffering middleware). Putting one behind sksec's crowdsec-bouncer@file
meant hand-editing live labels, reverted by the next deploy.

Each router now reads <svc>.TRAEFIK_MIDDLEWARES (skvector's gRPC router
skvector.TRAEFIK_MIDDLEWARES_GRPC), the skgallery convention. The value
replaces the whole chain; unset or empty renders byte-identical to
v2.26.4 (baselines rendered from the v2.26.4 templates before the change).
Documented in docs/v1-service-catalog.md ("Router middleware overrides")
and in the service READMEs that exist; the render gate's set pass now
drives every knob down its set path.



* feat(sksec): sync ALLOWLIST_CIDRS into a CrowdSec LAPI allowlist

The s02-enrich whitelist parser only drops LOCAL events. Community
blocklist (CAPI), console blocklist and `cscli decisions add` decisions
still reached the bouncer and could ban the operator's own ranges.
CrowdSec 1.6.8+ applies a centralized LAPI allowlist to every decision.

After the stack deploy, all three playbooks now include
tasks/lapi_allowlist.yml: resolve the node running the crowdsec task
(docker service ps; TRAEFIK_HA_MODE pins it to the ACME master), delegate
there, wait (bounded by LAPI_WAIT_SECONDS, default 300) for `cscli lapi
status`, then make allowlist LAPI_ALLOWLIST_NAME (default sksec-operator)
hold exactly ALLOWLIST_CIDRS: create it, add missing entries, remove
unlisted ones, verify. Empty list empties it; other allowlists are never
touched; changed only when something changed. cscli runs via argv, no
shell. LAPI_ALLOWLIST_ENABLED (default true) turns it off.

Tests run the real task file through ansible-playbook against a docker
test double modelled on crowdsecurity/crowdsec:v1.6.11 (fresh, idempotent,
drift, empty, rename, LAPI wait, LAPI never ready, no running task,
non-list input). README "Allowlist" rewritten; render gate stubs added.



* docs(changelog): Unreleased entries for the sksec LAPI allowlist and router middleware knobs



* fix: skorch knob reads skorch_cfg; sksec allowlist parse via set_fact

skorch templates read the combined skorch_cfg fact (never skorch.*, see
test_skorch_extra_vars_precedence.py). The LAPI allowlist comparison no
longer uses task-level vars, which the static playbook walkers render
with plain Jinja (no from_json).



---------


(cherry picked from commit 39389d7e76)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(sksec): bouncer replicas, healthcheck and trusted-proxies knobs [target v2.26.5] (#163)

* test(sksec): byte-identity baselines rendered from the current templates (before the bouncer knobs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(sksec): bouncer resilience and client-IP knobs

BOUNCER_REPLICAS (default 1; >1 adds max_replicas_per_node: 1),
BOUNCER_HEALTHCHECK (default off; on renders the image's /healthchecker
explicitly with BOUNCER_HEALTH_CHECK_* timings) and BOUNCER_TRUSTED_PROXIES
(rendered as TRUSTED_PROXIES into sksec.env; unset the bouncer trusts every
hop and takes the leftmost, client-forgeable X-Forwarded-For). The play
refuses 0 replicas, a non-list, non-address entries, 0.0.0.0/0 and ::/0.
Unset knobs render byte-identical to the baselines rendered from the
templates before this change (all envs, base and set passes). README
"Bouncer resilience" and "Client IP".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skhealth): every run reports its own CPU and peak RSS; README cost measured (#164)

The summary line of run and check now carries "cpu Xs, peak RSS Y MiB
(largest command Z MiB)" from getrusage, so the README "Resource cost"
figures can be checked on any host from the journal. The README table now
holds figures measured on skstack06 lab nodes (2 vCPU): about 0.4 s wall and
0.35 s CPU per run on an ordinary node, 1.6 s and 0.6 s on the primary,
engine peak RSS about 22 MiB, largest child command about 45 MiB. Replaces
the placeholder figures that had not been measured.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(skpayment): TLS_OPTIONS "none" omits the tls.options label (#165)

Found by the lane-2 lab trial: the default intermediate@file is sniStrict, so
a host without a certificate (the lab has no ACME) fails the handshake; and
Traefik rejects "default@file" ("unknown TLS options"), so there was no way
to ask for Traefik's default options. "none" now leaves the label out. The
default (intermediate@file, as on the hand-deployed gateway) is unchanged.
Verified on the lab: with the label removed /health answers 200 through
skfenceha.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* chore(release): CHANGELOG for skstacks-v2.26.5 (dated before the gate) (#166)

Folds the cherry-picked #160 Unreleased section into the dated
skstacks-v2.26.5 section, adds skhealth, skpayment and the sksec bouncer
knobs, and carries forward the skbackup "not release-gated" disclosure plus
one for skhealth's hardware checks (unit-tested only).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skhub): talk-recording CPU and memory limits/reservations knobs (#168)

skhub.TALK_RECORDING_CPUS_LIMIT, _CPUS_RESERVATION, _MEMORY_LIMIT and
_MEMORY_RESERVATION set the talk-recording swarm resources. Defaults are
the historic values (2.0 / 1.0 CPUs, 4G / 1G), so the default render is
byte-identical. On older Xeons without AVX2 a 1080p recording at 2.0 CPUs
records at about 2 fps; operators can now raise the limit. README row.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(skhealth): a missing SMART device alerts "device not found" (#169)

A remote smart_devices path that does not exist (wrong by-id name, swapped
disk) made smartctl answer "No such device" with exit 2 and no health line;
parse_smart() found nothing, so the disk read as healthy. Now smartctl
"No such device"/"No such file", or exit bit 1 (open failed) without a
health line, is a PROBLEM "device not found"; exit bit 0 without a health
line (command line or sudo failure) is "could not read it". The message
names smartctl's reason line instead of its banner.

Local: optional disks.devices lists the /dev paths a host must have. A
missing (or dangling by-id) path alerts "device not found" and is not
queried; a present path the scan did not list is checked by its path. The
runtime config gets the key only when it is non-empty, so the default
skhealth.json is byte-identical. Deploy asserts the list is /dev paths.

Tests: test_skhealth_missing_device.py (stub binaries record argv), render
tests for the knob and its assert.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skmesh): NetBird 0.79.0 -> 0.80.0, dashboard v2.94.0, digest-pinned (#171)

management, signal and relay default to 0.80.0 (2026-10-01, latest 0.80.x)
and the dashboard to v2.94.0 (the release paired with it; netbirdio/dashboard
does publish semver tags now, so the stale "digest only" comment goes), all
as tag@sha256:<multi-arch OCI index digest> read from the registry.

0.80.0 release notes: no breaking change, no manual step. Management's
automatic schema migration adds an indexed public_id column to policies,
routes and network resources (additive). README "Upgrading from 0.79.0"
says: dump the DB first; a rollback leaves the columns, 0.79 ignores them.

The default render changes by these four image lines only (intended).
Tests: test_image_defaults pins all four; dashboard default test updated.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skhealth): optional HMAC-signed webhook with an idempotency request id (#170)

Receivers such as the Hermes webhook platform reject unsigned posts. With
the optional skhealth.alerts.webhook_hmac_secret (rendered only into the
root-only secrets.env, never printed) each webhook POST carries
X-Webhook-Signature (hex HMAC-SHA256 of the raw body) and X-Request-ID
(skhealth-<cluster>-<host>-<hash of the (kind, key) pairs and the saved
state's last run>): a retry of an undelivered change set repeats the id,
any later change set gets a new one. Unset: neither header, the request
and the rendered files are unchanged. Telegram is never signed. skhealth
status shows webhook_signed as a boolean. The deploy refuses a secret
without webhook_url.

Tests: test_skhealth_webhook_hmac.py (fake webhook records headers; the
signature verifies over the body, not with another secret; unset sends no
header; retry repeats the id; later change sets differ), render tests.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(skgallery): redeploy no longer force-restarts the database (#173)

The wrapper force-updated every pre-existing service after the stack
deploy, the database included. That restart killed an in-flight
postgres-db-backup dump (the sidecar's first run after an Immich upgrade
failed this way) and dropped every Immich DB connection on every deploy.

The force only helps a service whose bind-mounted config files changed.
The database has none: env_file values, image, resources and healthcheck
are spec fields, so docker stack deploy already restarts it exactly when
its image or config changed. The database leaves the forced loop (and the
preexisting list); server, ML and redis are unchanged.

skhub (forces nextcloud/collabora only) and sksso (server, worker,
capauth) do not force their databases: no change.

Tests: test_skgallery_db_not_force_restarted.py (redeploy against a fake
docker: database and backup sidecar not forced, app services still
forced, first install forces nothing; static check; premise check that
the database has no bind-mounted config). The skgallery case of
test_no_forced_restart_on_first_install.py drops the database.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skpayment): "Stripe not configured" mode; pin app v1.1.0 (no unverified webhooks) (#172)

The role required non-empty STRIPE_SECRET_KEY and STRIPE_WEBHOOK_SECRET,
so an instance without a Stripe account had to invent placeholder keys.
Both may now be empty (both or neither; one alone is refused).

The pinned app (smilinTux/skpayments 17b0080) ACCEPTED unsigned webhooks
when STRIPE_WEBHOOK_SECRET was empty: anyone could post
checkout.session.completed and be granted a tier. The default pin moves to
skpayments v1.1.0 (d3138a2, sha256 5a88f86b...), which answers 503 on
/checkout and /portal without a secret key, 503 to every webhook without a
webhook secret, 400 to bad or replayed (>300 s) signatures, and reports
stripe_configured in /health. The deploy refuses the empty-secrets mode on
the 17b0080 file hash. env template defaults the two values to empty.

Tests: test_skpayment_render.py (+8: both empty, both absent, one alone
refused both ways, empty mode on the old app hash refused, default pin,
old pin with secrets still deploys). Coord card 1c2347e9.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* chore(release): CHANGELOG for skstacks-v2.26.6 (dated before the gate) (#174)

Upgrade notes (skmesh 0.80 schema step, skpayment app pin, skhealth
missing-device alerts, skgallery DB), one entry per PR #168-#173, the new
release-gate checks, and the not-release-gated disclosures (skbackup
carried forward; skhealth hardware checks, skhub recording knob values,
real Stripe flows).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skhub): apply the vault mail settings to Nextcloud on every deploy [target v2.26.6] (#175)

* feat(skhub): apply the vault mail settings to Nextcloud on every deploy

The role rendered SMTP_* into skhub.env, but the Nextcloud image reads them
only on its first install: an existing install never got mail_* keys and
sent no mail. With skhub.smtp_enabled a post-deploy task
(tasks/mail_config.yml) now sets mail_smtpmode/sendmailmode=smtp,
mail_smtphost, mail_smtpport (int), mail_smtpsecure, mail_smtpauth (true),
mail_smtpauthtype, mail_smtpname, mail_smtppassword, mail_from_address and
mail_domain from the existing vault keys.

The settings go to the container on stdin (never argv); the applier
(files/nextcloud-mail-config.php, run with php -r as www-data) compares
them with the effective config (config.php + config/*.config.php; the
password by sha256) and only on a difference writes a 0600 temp file, runs
occ config:import from an argv array, overwrites and removes the file and
re-checks. Prints key names only; the task is no_log and reports changed
only when it wrote. The deploy validates the keys when enabled. Unset or
false: nothing runs and the render is byte-identical. README "Mail (SMTP)".

Tests: test_skhub_mail_config.py (16: the PHP applier against a fake
Nextcloud root and occ; the task through real ansible-playbook with a fake
docker that records argv: password never on argv or in the output, rerun
changed=0, disabled never calls docker; static checks). php is required
under CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(skhub): compare the mail settings with config.php itself, not the merged config

Values only in the image env-driven smtp.config.php vanish with that env
(the reason an existing install had no mail), so config.php must hold them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* chore(release): CHANGELOG for skstacks-v2.26.6: skhub mail settings (#175) (#176)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skgym): new optional role for openGym (gym tracker), opt-in (#177)

* feat(skgym): new optional role for openGym (verification pending)

openGym (AGPL-3.0 gym and body-weight tracker) from the upstream images,
pinned by digest: api (one replica, JSON files in /data), web (nginx app +
/api proxy, one origin for passkeys) and a backup sidecar that writes a
verified tar.gz of the data dir for skbackup. SESSION_SECRET is required
and written to the data dir; every other upstream setting is a vault key
with a default. TRAEFIK_MIDDLEWARES, SSO_MIDDLEWARE (forward auth on all but
/api/), TLS_OPTIONS, DATA_PATH/BACKUP_PATH/DATA_NODE with the local-disk
guard, placement, health check and resource knobs. Opt-in: no shared file
changes. Tests: test_skgym_render.py (real Ansible), test_skgym_backup.py.

Full v1 suite and the skstack06 lab stage are pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(skgym): backup script pipes tar into gzip so a failed run leaves no partial

With tar -czf, GNU tar opens the archive in a forked gzip child, which can
create the temp file after a failed tar exited and the cleanup ran: a stray
.partial next to the good archives (seen under parallel test load). The
explicit pipeline ends before any check, and tar's own status goes through
a status file (POSIX sh has no pipefail). Same behaviour in busybox.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(skgym): backup check lists ./secret with tar -t, no pipe into grep -q

test_no_pipe_to_grep_q_under_pipefail flagged `tar -tzf | grep -qx` (the
file mentions pipefail). `tar -tzf ARCHIVE ./secret` exits non-zero when the
member is missing, in GNU tar and busybox alike, with no SIGPIPE shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: chefboyrdave21 <184315360+chefboyrdave21@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skpayment): CrowdSec on the site, Stripe webhook router exempt [target v2.26.7] (#179)

* feat(skpayment): CrowdSec on the site, Stripe webhook exempt (two routers)

A payment page belongs behind CrowdSec, but Stripe's webhook callers must
never be banned. The role now renders two routers on the same host and
service:

- site router (Host): MIDDLEWARES, default moves from
  default-no-crowdsec@file to default@file (CrowdSec when skfenceha has
  CROWDSEC_ENABLED);
- webhook router (Host && Path(/webhook/stripe), the app's handler):
  WEBHOOK_MIDDLEWARES, default default-no-crowdsec@file, explicit
  WEBHOOK_PRIORITY 1000 so it always outranks the site router (whose
  default priority is its rule length).

The exemption is safe because the app verifies every webhook's
Stripe-Signature (HMAC-SHA256 with STRIPE_WEBHOOK_SECRET; 400 on a bad
signature, v1.1.0 also refuses replays and refuses all webhooks without a
secret), and the deploy already refuses the one app release that accepted
unsigned webhooks without a secret.

Optional, off by default: WEBHOOK_IP_ALLOWLIST adds an ipAllowList of
Stripe's published webhook IPs (WEBHOOK_SOURCE_RANGES, default the list as
of 2026-10-13) in front of the webhook chain, with WEBHOOK_IP_STRATEGY_DEPTH
for a proxy in front of Traefik. WEBHOOK_PATH none restores one router.
MIDDLEWARES and the webhook knobs are validated.

Tests first: test_skpayment_render.py +17 (both routers, priority order
incl. a long hostname, every knob, non-prod names, invalid values, the IP
list identical in every env playbook): 43 passed. Full v1 suite pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(skpayment): IP allowlist and CrowdSec behind a Cloudflare tunnel

Measured on a tunnel-fronted instance: Traefik sees the tunnel's private
address for every public request and no X-Forwarded-For, so ipAllowList
(which cannot read Cf-Connecting-Ip) would refuse every webhook, and the
bouncer only ever sees the tunnel address. Say so in the allowlist section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* chore(release): CHANGELOG for skstacks-v2.26.7 (dated before the gate) (#181)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(sksec): portless LAPI host refused, middleware to the ACME Traefik, tunnel real-IP recipe; skfenceha DROP_FORWARDED_HEADERS [target v2.26.7] (#182)

* fix(sksec): refuse a portless LAPI host, middleware to the ACME Traefik, tunnel real-IP recipe [target v2.26.7]

Found on a production cluster whose public traffic arrives through a
Cloudflare tunnel:

- CROWDSEC_AGENT_HOST set to a bare service name made the bouncer dial
  the LAPI on port 80 (it listens on 8080). Every check failed and the
  bouncer, failing closed, answered 403 on every protected router. The
  play now refuses a value that is not <host>:<port>, and an empty value
  renders the default instead of an empty host.
- With skfenceha, sksec wrote the bouncer middleware only to dynamic/.
  The ACME-master Traefik reads dynamic_acme/, so its routers on the
  default chain (which names crowdsec-bouncer@file once skfenceha has
  CROWDSEC_ENABLED) were disabled. The same file now goes to both;
  skfence keeps its single dir.
- README "Behind a tunnel or CDN": the real client IP needs no new knob,
  only skfenceha TRUSTED_PROXIES (connector source) + sksec
  BOUNCER_TRUSTED_PROXIES (overlays + connector) + ALLOWLIST_CIDRS
  (connector + internal ranges, never banned). Traefik logs the trusted
  X-Forwarded-For as ClientHost and the stock traefik-logs parser takes
  its last entry, so a forged first entry does not move it.

Tests first: v1/tests/test_sksec_tunnel_real_ip.py (24, red before the
change); sksec + parity + shell-body subset 449 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(skfenceha): DROP_FORWARDED_HEADERS entrypoint middleware for TRUSTED_PROXIES behind a tunnel/CDN [target v2.26.7]

Verified live on a production cluster behind a Cloudflare tunnel: once the
worker trusts the connector (TRUSTED_PROXIES) to get the real client IP,
Traefik also trusts every X-Forwarded-* header from it, and Cloudflare
passes a client-sent X-Forwarded-Host through. Authentik then rendered
api.base "https://<forged host>/" into its flow page (password-reset link
poisoning setup). Of X-Forwarded-Host/-Port/-Prefix/-Proto/-Server and
Forwarded, only X-Forwarded-Host was reflected by any app probed.

skfenceha.DROP_FORWARDED_HEADERS (list of header names, validated after the
vault loads) renders a drop-forwarded-headers@file headers middleware
(customRequestHeaders: name -> "") and attaches it to the worker's
websecure entrypoint (http.middlewares), so it runs before every router.
Apps fall back to the Host header Traefik routed on; forwardAuth still
sends the real host. web (redirect only) and the ACME Traefik unchanged.
Unset or empty: byte-identical.

Tests first: v1/tests/test_skfenceha_drop_forwarded_headers.py (11, red
before). skfenceha/sksec/render/readme subset: 719 passed, 5 skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: chefboyrdave21 <184315360+chefboyrdave21@users.noreply.github.com>
2026-10-13 19:10:13 -04:00
.github/workflows release: skstacks-v2.26.0 (skfenceha, skhub, skstacks-config-template, skstacks-deploy) (#146) 2026-10-05 07:12:23 -04:00
.skred release: skstacks-v2.20.0 (first-install hardening, skgit admin keys, skmail relay secret at rest, skfenceha parity hooks 2) (#81) 2026-09-27 05:25:50 -04:00
assets SKStacks v2 framework: first public release 2026-09-24 06:09:15 -04:00
coturn SKStacks v2 framework: first public release 2026-09-24 06:09:15 -04:00
docs chore: forward-merge release/v2.26.6 (skstacks-v2.26.6) into main (#183) 2026-10-13 15:55:46 -04:00
scripts release: skstacks-v2.26.0 (skfenceha, skhub, skstacks-config-template, skstacks-deploy) (#146) 2026-10-05 07:12:23 -04:00
templates/config-repo release: skstacks-v2.26.0 (skfenceha, skhub, skstacks-config-template, skstacks-deploy) (#146) 2026-10-05 07:12:23 -04:00
tools release: skstacks-v2.26.0 (skfenceha, skhub, skstacks-config-template, skstacks-deploy) (#146) 2026-10-05 07:12:23 -04:00
v1 chore: forward-merge release/v2.26.7 (skstacks-v2.26.7) into main (#184) 2026-10-13 19:10:13 -04:00
v2 feat(skbackup): host-level restic engine with recursive dataset trees, replaces Duplicati (#109) 2026-10-04 23:22:53 -04:00
v3 SKStacks v2 framework: first public release 2026-09-24 06:09:15 -04:00
.gitignore SKStacks v2 framework: first public release 2026-09-24 06:09:15 -04:00
AGENTS.md release: skstacks-v2.25.0 (talk-hpb dinit command, Talk recording, local AI) (#133) 2026-10-04 03:39:12 -04:00
CHANGELOG.md chore: forward-merge release/v2.26.7 (skstacks-v2.26.7) into main (#184) 2026-10-13 19:10:13 -04:00
CLAUDE.md release: skstacks-v2.25.0 (talk-hpb dinit command, Talk recording, local AI) (#133) 2026-10-04 03:39:12 -04:00
LICENSE SKStacks v2 framework: first public release 2026-09-24 06:09:15 -04:00
README.md release: skstacks-v2.19.0 (skfenceha, skmail, skbook, skboard, skdesk, skseek, sksec, skmesh, skbackup, skport, skwhoami; parity hooks; deploy hardening) (#69) 2026-09-26 22:22:57 -04:00
renovate.json release: skstacks-v2.19.0 (skfenceha, skmail, skbook, skboard, skdesk, skseek, sksec, skmesh, skbackup, skport, skwhoami; parity hooks; deploy hardening) (#69) 2026-09-26 22:22:57 -04:00

SKStacks — Sovereign Infrastructure Configs

Infrastructure-as-code configurations for the SKWorld sovereign stack.


Versions

Version Status Description
v2/ 🟢 Active development Multi-platform (Swarm + K8s + RKE2), pluggable secret backends
v1/ 🟢 Published, one service at a time Docker Swarm + Ansible, the framework current production clusters run on

See v2/README.md for the full v2 architecture and quick start, or v1/README.md + the v1 service catalog for v1.


Framework / instance model (v1 and v2 alike)

This repo is the framework: the Ansible roles/playbooks (v1) or Helm/Kustomize/Terraform (v2) that define each service, with no estate-specific values baked in. A separate, private instance repo pins this framework as a framework/ git submodule at a released tag (skstacks-vMAJOR.MINOR.PATCH), and supplies everything specific to that estate: the inventory, the vault tree with real secrets, and which optional services it deploys. Upgrading an instance means bumping the submodule to a newer tag, not editing framework code in place. See v2/docs/INSTANCE-MODEL.md for the full model (it predates v1's publication but applies the same way).


v2 Highlights

  • 3 secret backends: vault-file (Ansible) · HashiCorp Vault · CapAuth/PGP sovereign
  • 3 platforms: Docker Swarm · Kubernetes · RKE2 (Rancher, CIS-hardened)
  • 4 CI/CD targets: Forgejo Actions · GitHub Actions · GitLab CI · ArgoCD GitOps
  • Sanitized public templates — no real secrets, all CHANGEME_* placeholders

Standalone Services

Directory Service Purpose
coturn/ coturn TURN/STUN server WebRTC NAT traversal relay for turn.skworld.io

coturn — Sovereign TURN Server

STUN/TURN server for WebRTC ICE negotiation. Used by SKComm's WebRTC transport and SKChat voice/data channels as the NAT traversal fallback when direct P2P fails.

Domain: turn.skworld.io

Architecture

WebRTC Peer A                coturn                WebRTC Peer B
     │                          │                         │
     │── STUN binding req ──────→                         │
     │←── mapped address ────────                         │
     │                          │                         │
     │  (direct ICE fails)       │                         │
     │── TURN Allocate ──────────→                         │
     │←── relay address ─────────                         │
     │── TURN CreatePermission ──→ (for Peer B's IP)       │
     │                          │                         │
     │══ encrypted DTLS-SRTP through relay ════════════════│

For Tailscale-connected peers, the Tailscale 100.x IP is used as a host ICE candidate and Tailscale DERP handles relay — coturn is only needed for non-tailnet peers.

Quick Setup

1. Install coturn:

# Debian/Ubuntu
sudo apt install coturn

# Arch/Manjaro
sudo pacman -S coturn

2. Generate a shared secret:

openssl rand -hex 32
# → copy this into turnserver.conf static-auth-secret AND
#   into ~/.skcomm/config.yml transports.webrtc.settings.turn_secret
#   (or set SKCOMM_TURN_SECRET env var)

3. Deploy the config:

sudo cp coturn/turnserver.conf /etc/coturn/turnserver.conf
# Edit: set static-auth-secret, uncomment external-ip=YOUR_PUBLIC_IP

4. Set up TLS (recommended):

certbot certonly --standalone -d turn.skworld.io
# Then uncomment the cert/pkey lines in turnserver.conf

5. Open firewall ports:

ufw allow 3478/udp    # STUN/TURN
ufw allow 3478/tcp    # TURN over TCP
ufw allow 5349/tcp    # TURN over TLS
ufw allow 49152:65535/udp  # UDP relay range

6. Start and enable:

sudo systemctl enable --now coturn
sudo systemctl status coturn

Configuration Summary

Setting Value
STUN/TURN port 3478 (UDP + TCP)
TLS TURN port 5349
UDP relay range 49152–65535
Realm turn.skworld.io
Auth HMAC-SHA1 time-limited credentials
Credential TTL 86400s (24h)
Private IP relay Denied (SSRF protection)

Client Integration (SKComm)

# ~/.skcomm/config.yml
transports:
  webrtc:
    enabled: true
    priority: 1
    settings:
      turn_server: "turn:turn.skworld.io:3478"
      turn_secret: "${SKCOMM_TURN_SECRET}"   # same secret as static-auth-secret

The SKComm WebRTC transport generates HMAC-SHA1 time-limited credentials automatically:

# credential = HMAC-SHA1(secret, f"{expiry}:{username}")
# username    = f"{expiry}:{agent_name}"

Testing

# Test STUN binding (should return your public IP)
stun turn.skworld.io

# Test TURN allocation (requires valid credentials)
turnutils_uclient -u "<expiry>:<username>" -w "<credential>" turn.skworld.io

# Check logs
journalctl -u coturn -f
tail -f /var/log/coturn/turnserver.log

License

AGPL-3.0-or-later. Part of the smilinTux sovereign stack.