Kubara mini-IDP reconciliation performance

Baseline measurements

The ConfigHub reads below were measured serially on 2026-08-05 against the quiescent Kubara organization. They describe this organization and client at that point in time; they are not portable latency claims.

A full context, organization, and control-Space pin revalidation adds about 1.15 seconds before each mutation. This is deliberately retained: selecting the right organization and control Space is a safety property, not an eventual-consistency optimization target.

The organization-wide shape probe observed 105 Units, 39 Links, 35 published releases, and 4 targets. The target-list duration was not isolated, so no target-list latency is claimed here. The Link sample predates the 25 NeedsProvides Links in the completed 64-Link organization and must be remeasured before it is used as a current Link-list baseline.

The pre-change offline runs on the same host were 8.72 s, 8.57 s, and 8.58 s for --plan, and 7.83 s for --self-test. Instrumented process profiles ranged from 7.74 s to 7.95 s. Every run stopped at the same pre-existing stale faithful hub/spoke receipt after concurrent generated-example changes. These values show that instrumentation did not create an obvious regression on that failure path; they do not prove a successful end-to-end speedup. The instrumented plan reported about 0.13 s in its single child command, which suggests that local rendering, YAML, checksumming, and contract work dominate this particular failure path. That last statement is an inference from the profile, not a standalone benchmark.

Rejected end-to-end profile and the current measured pair

The first instrumented end-to-end attempt did not complete successfully, so it is a rejection profile rather than a speed result. It took 1,541,558 ms (25.69 minutes) and launched 1,372 subprocesses. The seven known ConfigHub read verbs accounted for 866 commands, or 63.1% of every subprocess and 13.75 reads per managed Unit:

The 20 unit update calls are not the complete write count. Other mutation verbs were present, and the old profile did not yet classify every mutation by purpose and outcome. It would therefore be false to describe this as “866 reads versus 20 writes.” The useful finding is that metadata and content reads alone already dominate the command shape.

Explicit sleep subprocesses consumed 866,701 ms (14.45 minutes, 56.2% of the wall time). The remaining wall time was 674,857 ms (11.25 minutes). The old single sleep.wait bucket mixes Argo Application appearance, active-operation, health, refresh, namespace-move, and protected-Namespace settling. It cannot prove that all 14.45 minutes were caused by Argo.

The exact baseline and accepted budgets live in the performance acceptance contract. The offline verifier requires a successful changed apply followed immediately by a successful zero-action apply with the same execution fingerprint. These are fixture regression budgets, not service-level objectives:

The reconciliation and idempotence pair passes its functional receipt: the second run made zero semantic changes, zero ConfigHub mutation attempts, and zero Argo sync requests. Its current buyer-facing conservative measurement is therefore 33 ConfigHub CLI read commands, 208 total subprocess calls, and about 77 seconds. It reaches the first accepted hx-app-dev Application within 19 reads, then completes fleet and closing-stability verification within the 96-read and 220-subprocess ceilings. The fixture regression target is met. This remains evidence for this retained four-cluster fixture, not a speed comparison with raw Kubara or a service-level promise.

Command count is also not wire-request count: one cub command can make more than one authenticated request. “Fewer than 100 authenticated HTTP round trips” remains forbidden until sanitized client-transport evidence measures it.

Final performance publication is accepted only when the paired receipt verifier and orphan audit also pass. This prevents a faster run from succeeding by silently skipping part of the managed fleet. No budget permits removing target pinning, stable approval-gate observation, server HeadRevisionNum approval bracketed by unchanged Unit ID, observed numeric head, and DataHash, immutable OCI revision checks, release-boundary stability, wiring, or Application health.

What “63 Units means more than 100 round trips” actually means

The sentence was directionally right about the old client shape, but it is not a current measurement. A Unit is not one request. The steady-state reconciler used to read each managed Unit's metadata three or four times, read its body, then repeat endpoint, target, release-boundary, and final-verification reads. Static call-path accounting found 537 cub reads in the final verification body alone and a lower bound of hundreds more before the first Argo convergence wait. The accepted no-op now uses 33 ConfigHub CLI read commands for the whole fixture; that command count still must not be relabelled as HTTP round trips.

This is an N+1 client problem, not a reason to collapse the platform into fewer governed Units. The useful unit of analysis is:

inventory snapshots + content reads + changed-object writes + convergence

not Unit count × an assumed constant.

Bottlenecks and the implemented read optimizations

Per-Unit metadata and data reads cost roughly 0.57--0.60 seconds each in the sample. Repeating those reads makes verification grow with the number of Units. An organization-wide Link list is individually expensive, but one list is still preferable to repeating it for every Space.

Read-only verification now brackets organization-wide Space, Unit, Link, published-release, and Target snapshots. The Unit list selects DataHash, revision, endpoint, label, annotation, and gate fields, so one bounded bulk observation supplies the metadata. The configuration is no longer a Unit field, so the exact canonical Unit body is read from each Unit's data endpoint with cub unit data and hashed against the listed DataHash at ingress. A second five-resource snapshot closes the verification barrier. No cub unit get command remains in the accepted path, and a foreign Target slug or unexpected Space fails the final allowlist.

The apply path uses the same bulk Unit data for content comparisons and refreshes the organization snapshot only at explicit mutation or phase boundaries. Release decisions still use authoritative targeted snapshots immediately before publication, and a cached no-op is compare-and-set revalidated before any write. The accepted 96-command no-op ceiling includes the closing stability check. The current measured no-op uses 32 commands and keeps that closing boundary.

The apply path also removes three separate multipliers:

• an unchanged managed Unit reuses its one fresh metadata observation instead of fetching the same metadata three or four times;
• each delivery Application Space is listed once per metadata pass instead of fetching every Application Unit separately, and the already-read Unit is reused while checking its Application contract;
• all Application Units are reconciled before delivery, then the complete delivery root is published once per cluster, lazily just before that cluster's first source Application converges. The previous implementation republished and fully revalidated the same cluster root for every one of 27 deployments. Static accounting falls from about 1,302 reads for that repeated root path to 182 for four root publications plus the closing currency checks.

Source-release checks use an authoritative boundary-local snapshot and merge a successful closing observation back into the outer apply cache. The latest release is always taken from that inner authoritative snapshot while it is active; an older outer-cache release can never overrule it. No snapshot survives a release mutation. This preserves exact Unit bodies, endpoints, targets, labels, Links, approvals, and opening/closing release identity without returning to per-Unit reads.

These are API-backed cub reads, not literal wire round trips. A read-only client trace showed that space-scoped unit list, unit get, and unit data can each issue two authenticated application requests: one resolves the Space and one queries the Unit endpoint. The organization-wide Unit snapshot issues one. The accepted run records 32 CLI reads, but an exact full-run application- request or wire-request count is not claimed. The current cub --debug output is unsuitable for receipts or CI because raw debug traces can contain sensitive headers and request/response bodies. Safe instrumentation must count sanitized route templates and status/latency in the client transport, and use wire-level hooks separately for transmitted attempts, connection reuse, and TLS cost.

The receipt records a dedicated apply-start-to-first-argo-convergence phase. That measurement matters because optimizing total runtime and optimizing time to the first converged Application are not always the same tradeoff.

The initial and final Space/Unit/Link/release/Target snapshots must have identical row counts and a canonical fingerprint. Every row must belong to the pinned Kubara organization and the exact final allowlist. If any of those five resource sets differ at the closing observation, the run fails and asks for a retry against a quiescent organization. This is an opening/closing net-state check, not an event stream: an ABA change between observations is outside the claim. Apply-cache mutation decisions are separately guarded by a fresh targeted comparison immediately before the actual write.

Canonical Kubernetes and AppConfig YAML is memoized by content digest within a single process. Cache keys also retain a length/head/tail collision check, and the profile records requests, hits, misses, entries, and parse time. No cross-run cache is trusted.

Concurrency and safety boundary

All commands remain serial and deterministically ordered. Organization-wide reads are batched by resource, not executed concurrently. The five snapshot lists are independent and could be parallelized later, but doing so needs server-load and stable-evidence testing first.

The following remain strictly serial:

• every ConfigHub mutation and its full pinned-target revalidation;
• release publication, approvals, promotion, and operation-journal changes;
• cluster and Argo convergence checks whose observations depend on earlier actions;
• the opening and closing snapshot barriers.

A possible future optimization is to validate the full target once at a well-defined mutation-batch barrier, then verify a locally pinned context token before each mutation. It is intentionally not implemented: it must first prove equivalent protection against context and organization changes.

Other things that can slow the platform down

In priority order for this example:

1. Repeated authenticated reads. Per-object CLI calls pay process startup, TLS, authentication, API routing, and JSON decoding every time. Bulk read APIs and boundary snapshots are the main remaining product opportunity.
2. Per-mutation target pinning. A context, organization, and control-Space revalidation costs about 1.15 seconds before every write. It remains intact; the safe product fix is a server-enforced organization/batch boundary, not a weaker client check.
3. Release-boundary verification. Source releases deliberately prove Unit data, endpoint identity, target, labels, Links, and a stable before/after boundary. A boundary-scoped bulk API could retain those invariants with far fewer round trips.
4. OCI work. Rendering, hashing, uploading immutable layers, publishing a release, registry propagation, and controller pulls all add latency. Cache hits help, but exact digests and retained versions must not be discarded.
5. Argo polling. Serial five-second polls make quiet convergence easy to reason about but add tail latency. A watch/event path can improve this later if it preserves the exact revision and operation-journal deadlines.
6. Kubernetes controller dependencies. CRDs, webhook readiness, cert-manager issuance, ESO reconciliation, Ingress status, and workload health are real dependency barriers; parallelizing across them blindly can make the run slower and less reproducible.
7. Image pulls and target capacity. Cold images, registry throttling, CPU, memory, disk, DNS, and the single-node kind topology can dominate once API chatter is reduced.
8. Local parsing and subprocesses. YAML canonicalization and one process per cub/kubectl command are measurable overhead. Process-local canonical-YAML memoization is implemented; a long-lived API client would remove more.
9. GUI query shape. Component matrices, wiring Links, history, and health views should query by indexed labels and page/batch results. Rendering every revision or edge eagerly would move the same N+1 problem into the browser.
10. Source and catalog hand-off. Kubara generation, Git fetch/checkout, lock and checksum verification, exact catalog-version resolution, and CI gates add deterministic latency before import. Cache by Git revision and catalog lock, but never replace exact-version failure with a silent upgrade.

Performance guardrails

These are regression budgets for this implementation, not user-facing SLAs:

• verification uses exactly two organization-wide list calls per cached resource: one opening and one closing Space, Unit, Link, release, and Target list;
• active verification must issue no per-Unit metadata get or per-Space Unit inventory list; exact Unit data comes from the bulk Unit rows;
• each source-release boundary reads authoritative targeted inventories, issues no point Unit metadata/data gets, and invalidates or refreshes its cache at every release mutation;
• mutations remain serial and keep full target-pin validation;
• the opening and closing fingerprint must match before evidence can say stability: pass;
• command evidence contains sanitized resource/verb labels only, never names, arguments, paths, tokens, or payloads;
• canonical-cache hit/miss accounting must balance and every miss must create exactly one process-local entry;
• on the same host and fixture, an offline failure-path plan or self-test above 10 seconds should be investigated against the approximately 8.6/7.8-second baseline;
• a changed apply must meet the changed-apply contract and the immediate zero-action apply must meet the tighter idempotent-apply contract under the same execution fingerprint;
• the current 32-read/208-subprocess/~102-second no-op meets the fixture regression target and is disclosed as bounded evidence, not a raw-Kubara speed comparison or SLO. The v2 receipt verifier and scoped residue audit remain authoritative for what actually happened.

Verify the offline contract and its negative tests without a ConfigHub login:

npm run kubara-mini-idp:performance-contract:verify
npm run kubara-mini-idp:performance:self-test

After the changed/idempotent live pair and orphan audit exist, verify only the committed evidence-without network access-with:

npm run kubara-mini-idp:performance:receipt-verify