Ci Monitor

CI Monitor Lifecycle

• The green observation only clears the failure signal by default: the step keeps polling until the PR is merged, closed, or its idle timeout elapses, which is what every run has always done. Under the opt-in ci.release_after_checks_pass (sctx.Config.CI.ReleaseAfterChecksPass, resolved from the operator's global ci block and overridable by a trusted repository config) CIStep.Execute returns readiness in StepOutcome.CIReadyNoCI instead of polling again; Executor.executeStep commits it with CI completion through db.CompleteCIStep, which also stamps runs.ci_released_at, then emits readiness after the step-completed event. Release readiness must never appear with a running CI step, which would make AXI promise a live monitor. Readiness-write and terminal-verdict read failures fail the run. The run ends there and releases its worktree. Executor.terminalRunStatus records types.RunChecksPassed only when that durable release marker is set (with pr_state open and the CI step completed); it deliberately does not infer the verdict from ci_ready_at, because a monitoring run also records readiness and can end with an open PR after a timeout gate is approved. db.RecoverStaleRunsExcept finalizes the durable state before generic stale-run failure if the daemon crashed before run completion, preserving the CI duration (TestRecoverStaleRunsCompletedCI, TestRecoverOnStartup_FinalizesCommittedCI); axi maps the status to outcome: checks-passed (outcomeFor) and renders the released contract, types.RunStatus.Terminal() includes it, and a merge observed afterwards writes only pr_state - never the run's recorded outcome, and never the CI step's status or duration (which is why a released run's recorded duration covers CI only). Regressions: TestCIStep_DefaultKeepsMonitoringAfterGreenChecks (the default), TestCIStep_GreenChecksReleaseWithoutWaitingForTheMerge, TestCIStep_TrustedNoCIReleasesWithoutWaiting, TestExecutor_GreenCIRunRecordsChecksPassedAndReleases, TestExecutor_MergedCIRunStillRecordsCompleted, TestRenderDriveResult_TerminalChecksPassed, TestDriveRun_ReleaseAndWatchGuidance, TestExecutor_CIReleaseReadinessFollowsStepCompletion, TestExecutor_CIReadinessWriteFailureFailsRun, TestExecutor_TerminalVerdictReadErrors, TestReleasedChecksPassedGuidance_SyncedAcrossSurfaces, TestCIReleaseAfterChecksPassDefaultsOffAndResolvesFromTrustedConfig.
• The CI step produces findings, not fixes: a settled observation returns one finding per issue with an action (ciObservationFindings, internal/pipeline/steps/ci_findings.go) and the executor's shared auto-fix loop drives fix rounds (repairFromFindings in ci_fix.go consumes sctx.PreviousFindings), so auto_fix.ci is counted by the executor from round history and a review-bot check (scm.ReviewBots, identified by scm.Check.App) parks as ask-user comment findings without spending a round. A published repair keeps monitoring inside the fix round and must call sctx.MarkRunning() so status returns to running for the readiness consumers. "CI Step Findings Model" below owns the contract and regression list.
• ci_timeout is an idle timeout, not an absolute deadline: only timeoutAnchor re-arms when the upstream default-branch tip advances, started stays fixed for poll pacing, and re-arm only ever extends the deadline (fail-safe on transient base-tip failures). It bounds the wait for checks to finish, and the merge watch after that unless ci.release_after_checks_pass ends the step at the verdict. Value semantics (0 unset, negative unlimited sentinel, keyword parsing) live in config.go; keep config.DefaultCITimeout and defaultConfigYAML in sync (TestDefaultConfigYAML_MatchesGoDefaults). User-facing semantics are owned by docs/src/content/docs/reference/global-config.md.
• GitHub readiness is the union of the exact current PR head commit's check rollup and every Actions workflow run returned by the Actions API for that same SHA. A workflow rejected before creating jobs/check-runs is absent from the commit rollup but still present in that API; run discovery errors and unknown run states fail closed instead of certifying a green rollup. Regressions: TestGetChecksIncludesFailedWorkflowRunMissingFromPRRollup, TestGetChecksBindsRollupAcrossABAHeadMovement, TestCIStep_FailedHeadWorkflowRunPreventsChecksPassed.
• GitHub's raw commit statusCheckRollup returns every check run a commit ever had, including a same-named run a later run at that same head has already superseded (e.g. a required-check job re-triggered by synchronize after a pipeline auto-fix push). Host.GetChecks (internal/scm/github/github.go) collapses same-name reruns of one workflow to the newest startedAt (using Actions run identity to break timestamp ties, comparing against every retained duplicate, and preserving unordered records, including ambiguous pending replacements, before a completedAt fallback) while preserving independent workflows, unidentified external checks, and same-named commit status contexts, restoring the semantics gh pr checks already applies and this repo's own gate verify.py already assumes (last-wins). Collapse must run AFTER appendUnrepresentedWorkflowRuns, never before: that call dedupes the Actions-run union against checks by run ID over the FULL uncollapsed rollup, so collapsing first drops a superseded run's ID out of the "represented" set and the union re-adds the same stale run under its own workflow run name. Without this collapse, a monitor that keeps synchronize in its gate trigger set can loop forever: each auto-fix push legitimately fails the gate at the old head, a later same-head rebind turns it green, but the stale FAILURE stays visible and re-triggers another auto-fix round. Regressions: TestGetChecksCollapsesSupersededSameNameCheckToLatestAtOneHead, TestGetChecksCollapseOrderingDoesNotLetWorkflowRunUnionResurrectSupersededCheck, TestGetChecksPreservesIndependentSameNameWorkflows, TestGetChecksPreservesIndependentSameNameExternalCheckRuns, TestGetChecksCollapseComparesNewestRunWithEverySameNameCandidate, TestGetChecksKeepsQueuedReplacementWithEqualStartTime, TestGetChecksPreservesUnorderedExternalPendingReplacement, TestGetChecksPreservesSameNameStatusContextAndCheckRun.
• A published CI repair rebinds an existing pipeline attestation in the PR body to the new head (restampPublishedAttestation after publishRepair) only when the host has an existing attestation and a PRContentReader. All supported providers implement raw reads; Bitbucket owned templates carry the same marker as visible text. Owned restamps enforce provider budgets before the adapter can clamp (especially Azure). Hosts without readers skip with a warning. A restamp that cannot settle after retries is unsettled, not success. The CI fixer prompt may conclude that a stale/attestation/infrastructure red is not a code defect; a genuine failing test or build must still be fixed. Regressions: TestCIStep_PublishRepairRebindsAttestationAcrossRepairPushes, TestCIStep_PublishRepairDoesNotMintAttestation, TestCIStep_PublishRepairSkipsRestampWithoutReader, TestRestampPRAttestation_MissingReaderIsSkipped, TestRebindPipelineAttestationHead_VerifyPyRoundTrip, TestCIStep_FixPromptPrefersSimplificationOverMachinery, TestCIStep_FixPromptDistinguishesCodeDefectFromExternalFailure.
• CI readiness never treats an unproven empty forge check list as green. Ready requires observed all-green checks, or trusted default-branch no_ci: true with zero registered checks (internal/pipeline/steps/ci.go decides whether the declaration applies; internal/cimonitor owns the agent-facing log vocabulary and Ready/DeclaredNoCI parse). Delayed registration, pending checks, failures, errors, unknowns, and stale-head evidence stay not-ready; registered checks on a declared no-CI repo are still honored. Regressions: TestChecksPassed_PR607RealLogSequence, TestCIStep_EmptyChecksWithoutNoCIStaysNotReadyPastOldGracePeriod, TestCIStep_EmptyChecksWithTrustedNoCIBecomesReady, TestCIStep_DelayedCheckRegistrationStaysNotReadyUntilGreen, TestCIStep_DeclaredNoCIWithUnexpectedChecksHonorsThem, TestEffectiveRepoConfig_NoCITrustedOnly.
• Persistent provider check-read failures are not an invisible spin: 6 consecutive GetChecks errors while the PR is still open park the CI step at an ask-user gate whose provider-neutral finding names the provider CLI/credentials support and includes the underlying error (with a GitHub-specific clause requiring gh >= 2.50 for the gh pr checks --json call), and the streak resets on any successful read. Regression: TestCIStep_PersistentCheckReadFailureParksAtAskUser.
• Reap an orphaned monitor from outside its worktree with no-mistakes axi abort --run <id>; it needs only NM_HOME and never starts a stopped daemon. A known run succeeds only with durable terminal truth, a recorded nonterminal run fails unconfirmed, and only an unknown id is an idempotent no-op. Bare axi abort stays worktree/branch-scoped.
• A merged or closed PR observation transactionally completes an active run and its CI step; PR lifecycle state is monotonic, so duplicate or delayed observations cannot reactivate or regress a terminal run. Startup reconciles legacy pending or running rows that already hold terminal PR state before parked-run planning and generic crash recovery. Regressions: TestUpdateRunPRStateFinalizesActiveTerminalOutcomes, TestUpdateRunPRStateIgnoresDuplicateAndDelayedRegressions, TestReconcileTerminalPRRunsFinalizesLegacyActiveRows, TestRecoverOnStartup_FinalizesLegacyTerminalPRRun, e2e TestTerminalPRRunDisappearsFromActiveListing.
• A provider-reported cancelled check is never a job verdict, so the deterministic rerun runs strictly before any CI fix round: cancellation and a GitHub pre-run infrastructure failure (a job that failed in setup/action resolution before any repository step ran, flagged Check.PreRunFailure by the opt-in scm.PreRunFailureDetector and re-bucketed to cancel so it never masks a real test/lint failure, which cleared setup and failed a later step) are the outcomes that earn a rerun, either outcome after its budget parks as ask-user instead of entering the auto_fix.ci loop, and any genuine or unrecognized failure or merge conflict in the same poll suppresses reruns so real failures still escalate on their first observation. The budget is per check name per run and spent on request rather than on success, and a rerun is never issued once the published branch head no longer equals runs.head_sha, because it would certify a commit this run never delivered. Each outstanding rerun records its verified pipeline head and the same-name provider links visible when it was requested; it retires durably when the run head advances or a new conclusive non-cancel link appears. Retirement keeps the spent budget and never changes check buckets. A delayed same-named green sibling can satisfy the link trigger on the same head, matching the default branch's existing name-keyed masking; removing that limitation requires provider truth outside this policy. Classification, the deliberate TIMED_OUT/STALE exclusions, rollup-lag grace, and retirement live in internal/pipeline/steps/ci_transient.go; provider support is the optional scm.CheckRerunner (GitHub only), pre-run infrastructure detection is the optional scm.PreRunFailureDetector (GitHub only, folded in by markPreRunInfraFailures), and user-facing semantics are owned by docs/src/content/docs/reference/repo-config.md. Regressions: TestCIStep_CancelledCheckIsRerunBeforeEscalating, TestCIStep_CancelledCheckStaysUnresolvedAfterItsBudget, TestCIStep_LaggingRerunRollupKeepsWaitingForTheRepublishedCheck, TestCIStep_SameHeadGreenRerunEmitsChecksPassed, TestCIStep_DelayedSameNameCheckRetainsLegacyNameBehavior, TestCIStep_ResolvedRerunDoesNotParkALaterGreenHead, TestRetireResolvedReruns, TestRetireResolvedRerunsRetriesAfterPersistenceFailure, TestCIStep_MovedPublishedHeadTerminatesInsteadOfRerunning, TestCIStep_MovedPublishedHeadClearsCIReadiness, TestClassifyCheckFailure, TestMarkPreRunInfraFailures_RetriesInfraButNotGenuine, TestMarkPreRunInfraFailures_OptInGated, TestPreRunFailures_FlagsSetupFailureNotGenuine, TestPreRunFailures_FailsClosedOnUnreadableRun.
• Terminal is not pending. Readiness must reject every non pass/fail/skip bucket (hasUnresolvedChecks), but only checks that can still finish on their own (hasExecutingPendingChecks) may defer escalation. A GitHub run held for maintainer approval (scm.Check.AwaitingApproval: action_required with no jobs) is pending for readiness but never defers another check's failure (internal/pipeline/steps/ci_awaiting_approval_test.go). A cancel bucket - GitHub CANCELLED, GitLab canceled, Bitbucket STOPPED, and how GitHub reports a job killed by its own timeout-minutes - is a published conclusion that nothing will replace, so with no rerun outstanding it is reported as an ask-user transient finding (unresolvedTransientFindings, selected by cancelledWithoutRerun) instead of waiting. Conflating the two is the #628 regression that hung real runs for their whole ci_timeout; an unrecognized bucket is deliberately still treated as waiting, because unknown is not evidence of terminal. Regressions: TestCIStep_CancelledCheckAmongPassingChecksEscalatesInsteadOfPollingForever, TestCIStep_ZeroRerunBudgetEscalatesCancelledCheckWithoutMakingItReady, TestCIStep_BitbucketStoppedCheckParksForADecision.
• A green GitHub PR reporting mergeStateStatus: BEHIND (scm.MergeableBehind, parsed by parseMergeableState in internal/scm/github/github.go; GitHub reports it only when the base requires up-to-date branches) reads as MERGEABLE and reports checks-passed unless trusted ci.rebase_behind opts in (off by default: each rebase is another full pass). Opted in, it is never checks-passed: once a second consecutive poll sees it green and BEHIND at the same resolved base tip (behindBaseTip in ci.go), it becomes one auto-fix ci-behind-base finding that parseCIFixTargets routes into the merge-conflict rebase repair (ciFixTargets.BehindBase), so it revalidates from Review like a conflict. Other providers never report BEHIND. Regressions: TestGetMergeableStateReadsMergeStateStatusBehind, internal/pipeline/steps/ci_behind_base_test.go (default off: TestCIStep_GreenBehindPRReportsChecksPassedWhenRebaseBehindIsOff), TestCIRebaseBehind_OffByDefaultAndTrustedOnly.
• A rebase repair (merge conflict or BehindBase) normally revalidates from Review because ancestry cannot prove continuity. Trusted opt-in ci.carry_review_on_replay lets recordRepair try recordReplayedRepair first: pureReplayGap (ci_replay.go) must prove the repaired head is the reviewed chain replayed onto exactly CIStep.replayBase (the resolved rebase target, bound only for that repair's recording) - equal-length linear chains, each reviewed commit git merge-tree-replaying conflict-free onto its new parent to the repaired commit's exact tree, and git patch-id --verbatim over the full binary diff equal. Any shape mismatch or git error fails closed to today's Review restart; the flag is ignored under ci.revalidate_repairs: true and when a gate is anchored after: review (Test restart would skip it). On success the approval moves via db.UpdateRunHeadSHAWithCarriedReview, a compare-and-swap on the approval the proof used (never resurrects a revoked one), and the outcome is RestartFrom: types.StepTest (ciRepairResult.restartFrom). Nothing publishes early. That outcome does not survive a parked CI gate (a Fix response replaces it, a daemon restart drops it), so the same write sets the durable runs.ci_test_restart_pending, which only UpdateRunPublication clears (CI never publishes while it is set, so only Push can). While set, recordRepair restarts the unchanged head at Test and sends a commit on top to recordLocalRepair, a fix round whose fixer failed records its worktree head through recordRepair instead of monitoring (a commit it left on top revalidates from Review, the unchanged head restarts at Test), and Resume restarts a recovered CI gate's approval at Test. Regressions: internal/pipeline/steps/ci_carry_review_on_replay_test.go, TestCICarryReviewOnReplay_OffByDefaultAndTrustedOnly, TestExecutor_RestartFromTestKeepsTheCompletedReview, TestExecutor_RecoveredCIGateApprovalRestartsAPendingCarryAtTest.
• CI readiness is read from the provider's live PR head check rollup on every poll, so it always describes the head the forge currently has for that PR; no recorded SHA gates it, and a run whose row still names a pre-advance commit must still recognize green at the head the pipeline last pushed. Regression: TestCIStep_GreenChecksAtAdvancedHeadAreRecognizedWhileRunTracksOlderHead.

CI Step Findings Model

• CIStep.Execute has the review step's shape: a fix round (sctx.Fixing) first repairs the findings the executor selected (repairFromFindings, internal/pipeline/steps/ci_fix.go), then the monitor runs; every settled observation (all checks finished, every authorized transient rerun spent, at least one issue) returns one finding per issue with its action (ciObservationFindings, ci_findings.go) and the executor's shared loop decides what follows. The step never counts or limits fix rounds; auto_fix.ci is enforced by the executor from the round history, which survives ResetStepsFrom and daemon restarts. step_results.ci_fix_attempts is retired (column kept, never read).
• Classification reads provider structure only: a job verdict (fail bucket) and a merge conflict are auto-fix errors; an unresolved transient is ask-user (unresolvedTransientFindings); a check whose scm.Check.App (GitHub checkSuite.app.slug) is in scm.ReviewBots (Greptile: greptile-apps) becomes one ask-user warning per unresolved bot comment, bounded (maxReviewBotCommentFindings), so it never spends a round. An empty App is never a bot. The comment half uses provider-scoped scm.ReviewBotForLogin: GitHub matches greptile-apps and greptile-apps[bot], while GitLab matches only greptileai; only GitHub can attribute a check to a bot, so a GitLab run reaches the comment read through the green/not-yet-registered path below and on_failure reads nothing there. scm.Capabilities.ReviewComments is the opt-in (GetReviewComments), true on GitHub (unresolved review threads) and GitLab (unresolved discussion notes: resolvable && !resolved, not a system note, bot-authored, diff position for the anchor); a GitLab host without the project path answers scm.ErrUnsupported rather than erroring, so it degrades instead of parking. Under opt-in ci.review_bot_comments: always (default on_failure), a bot check that completed green, or a registered bot with no check on the head yet (its earlier threads can still be unresolved), goes through the same path (greenReviewBotFindings -> ciObservationFindings with greenBots): its unresolved comments are the same ask-user findings and block checks-passed, a green bot with none produces nothing, an unreadable comment list withholds readiness and parks after consecutiveCheckErrorLimit, and VerifyApprovalOverride records an approval over them as an override. The fix agent, not metadata, decides whether an unknown red is about the code: its code_change_needed=false conclusion parks the selected findings as ask-user (ciRepairParkOutcome).
• types.Finding.Check (display name), types.Finding.CheckID (provider identity), and the FindingCategoryCI* constants are the routing keys the fix half reads (parseCIFixTargets); Finding.UnmarshalJSON and findingWire copy fields explicitly, so a new field must be added there or it drops on every parse. Findings ride the IPC event stream, so per-comment findings stay bounded.
• A published repair keeps monitoring inside the same fix round and must call sctx.MarkRunning() (executor-owned: durable status back to running plus a step_started event), or the step stays fixing and neither the TUI nor axi's checks-passed (ciReadyToMerge requires running) can surface. The post-repair stale-evidence guard is lastRepairStillUnverified: selected checks are snapshotted before a resumed repair, and targets that remain red with neither a newer completion/execution discriminator nor an observed pending state keep the monitor waiting instead of re-escalating. Findings deferred by a fix round survive pending observations and terminal monitor errors until a fresh settled issue set or an established clean result supersedes them.
• Regressions: TestCIStep_GreptileOnlyRedParksAsAskUserAndSpendsNoAutoFixRound, TestCIStep_MixedGreptileAndTestFailureRoutesOnlyTheTestToAutoFix, TestCIStep_PublishedRepairResumesMonitoringAndWaitsForTheRerun, TestCIResumedRepairSnapshotsSelectedCheckCompletion, TestTerminalFailureFreshnessUsesExecutionIDWithoutCompletionTime, TestCIStep_PersistentCheckReadFailureParksAtAskUser, TestCIStep_UserFixOfTransientFindingIsHonored, TestCIObservationFindings_ClassifiesEachIssueByProviderStructure, TestExecutor_MarkRunningReturnsAFixingStepToRunning, TestExecutor_RestartsValidationFromRequestedStep, TestGetChecksCarriesCheckSuiteAppSlug, TestParseFindingsJSON_RoundTripsCICheckFields, TestCIStep_GreenReviewBotCommentsParkUnderAlways, TestCIStep_GreenReviewBotCommentsIgnoredByDefault, TestCIStep_GreenReviewBotUnreadableCommentsParkUnderAlways, TestCIStep_JobFailureWithGreenReviewBotFollowsThePolicy, TestCIStep_VerifyApprovalOverride_GreenReviewBotComments, TestCIStep_UnregisteredReviewBotCommentsFollowThePolicy, TestReviewBotCommentsOnGitLabFollowThePolicy, TestCIStep_GitLabReviewBotDiscussionCommentsParkUnderAlways, TestGetReviewCommentsReadsUnresolvedBotDiscussions, TestReviewBotRegistry_ResolvesAppAndLoginToTheSameBot. Step tests drive the loop with stepstest.ExecuteWithAutoFix (the executor's auto-fix loop in miniature); a test that calls Execute once sees only the observation.