Post-Deployment Smoke Testing: Catch Regressions After Release

Authenticated workflow testing must prove that the right user can sign in and finish the protected task after a deployment. AlwaysQA tests password and six-digit email-code sign-in through a fresh browser session, giving the team visibility into the complete workflow without giving the decrypted password, QA Inbox, Email Code, or raw authentication message to a Coding Agent or browser agent. Most teams make the mistake of treating login as a checkpoint. The expensive failures happen across the whole chain: form, credential submission, identity-provider redirect, session, permissions, and the product action the customer came to complete.Reality check: A green login page proves that the page loaded. It doesn't prove that an authenticated customer can use the product.The use case: verify the job behind the login Consider a SaaS team that has just shipped a small permissions change. Public pages respond normally. The sign-in form loads. Infrastructure monitoring is green. But standard users now return to the login page after entering a valid password. Administrators are unaffected, so the problem survives an internal spot check. Customers discover it first. A useful Test Case doesn't stop at “login succeeded.” It verifies a visible product result:Starting URL https://app.example.com/login Instructions Sign in as the configured standard user. Create a test project named with today's date. Confirm that it appears in the project list, then delete it. Success Condition The project appears after creation and no longer appears after deletion.That Run covers the sign-in form, credential acceptance, redirect, authenticated session, role permissions, record creation, persistence, and cleanup. It answers a business question, not a transport-level one. For broader coverage across signup, onboarding, permissions, billing, and other release risks, use the web app regression testing checklist. This page focuses specifically on authentication and the workflows gated by it. Why URL checks and login-only tests miss real failures Authentication is a dependency chain. A failure at any layer can block the customer while shallow monitoring still reports success.Layer What a shallow check sees What the customer experiencesLogin page The URL returned 200 OK The submit action does nothingPassword submission The form accepted input A valid credential is rejectedEmail delivery The email service is available The current six-digit code never arrivesCode verification The form accepts six characters A fresh code is rejected or expiresHosted authentication The identity provider responded The redirect returns to the wrong originSession Authentication completed The next page asks the customer to sign in againAuthorization The dashboard opened The account has the wrong role or permissionsProduct workflow Login passed The customer still can't complete the taskCustomers don't buy access to a login screen. They buy what the authenticated product lets them do. Use dedicated QA Accounts, not real customer credentials AlwaysQA uses a QA Account: a named, Project-owned authentication resource for a dedicated identity in your application. Multiple Test Cases in the same Project can reuse it. Create separate QA Accounts for the roles you need to verify, such as standard user, administrator, or billing manager. Give each one only the permissions and test data required for its workflows. That reduces operational risk and makes a failure easier to interpret. It also follows the least-privilege principle described in NIST SP 800-53 Revision 5. Each QA Account has one explicit, immutable Authentication Method:Authentication Method What you configure When to use itPassword A dedicated identifier and password The application signs in with username or email plus passwordEmail Code A dedicated identity using an AlwaysQA-controlled QA Inbox The application sends an exact six-digit code by emailAlwaysQA doesn't infer the method from the page. If the product supports both methods, create two QA Accounts and assign each Test Case to the appropriate one. The QA Account feature overview explains how this setup supports role-specific and permission-specific workflows. The trust boundary: the agent navigates, trusted code authenticates The browser agent needs to understand the interface. It doesn't need the authentication secret. During a Run, the agent navigates to the matching sign-in form and requests one QA Account Sign-in action. Trusted code completes the configured Authentication Method and returns only a sanitized result. After successful sign-in, the browser agent continues with the Test Case instructions. The boundary is easier to see as a handoff: Browser agent Trusted sign-in action Application | | | | navigate to login | | |-------------------------->| | | | submit protected material | | |-------------------------->| | | sanitized result | |<---------------------------| | | continue product workflow | |------------------------------------------------------->|What stays outside agent context:Authentication material Handled by Given to Coding Agent or browser agent? Retained as Run Evidence?Decrypted password Trusted sign-in action No NoQA Inbox address Authenticated user and trusted code No NoRaw authentication message Trusted code No NoSix-digit Email Code Trusted code No NoSanitized sign-in result Trusted code and Run Yes Only sanitized observationsIt's the product's core security distinction: the agent can evaluate the authenticated experience without receiving the material used to enter it. It's still a dedicated identity, and it still needs operational discipline. Use synthetic data, minimum permissions, a cleanup strategy, and an owner who can rotate or remove the identity when the application changes. How password authentication works For a Password QA Account, configure the identifier, password, Login Page URL, and approved Login Origin directly through AlwaysQA. Every Run starts with fresh browser state. The browser agent reaches the visible sign-in interface, then trusted code submits the configured authentication material. The agent receives a sanitized result and continues only after the sign-in action completes. The password is write-only from the perspective of product read interfaces. It isn't returned to your Coding Agent, placed in its prompt, or exposed to the browser agent. That's different from instructing an AI agent to retrieve a secret from an environment variable and type it into a page. The agent controls the workflow; AlwaysQA controls the credential boundary. How six-digit email-code authentication works Passwordless testing is harder because “use the latest email” isn't a reliable rule. An inbox may contain a code from an earlier attempt, delayed mail, or multiple messages that look valid. Each Email Code QA Account receives one stable, opaque QA Inbox. Its address contains no User, Project, or QA Account name. The authenticated user copies that address when provisioning the dedicated identity in the application; the Coding Agent and browser agent never receive it. For each unattended Run, AlwaysQA:opens a fresh browser session; navigates to the matching sign-in form; opens one short-lived authentication attempt; requests one authentication message; accepts only a fresh message delivered after that request; extracts one exact six-digit Email Code through trusted code; submits that code on the approved Login Origin; and continues with the authenticated Test Case.The Run doesn't reuse an old code, request an automatic resend, or try several candidates after rejection. Runs using the same Email Code QA Account execute one at a time. Serialization prevents overlapping messages and product activity from being attributed to the wrong Run. Other Test Cases remain independent. Each Email Code QA Account receives an AlwaysQA-controlled QA Inbox. Your company mailbox isn't part of the workflow. When trusted processing ends, the provider copy of the raw authentication message is deleted. The payload isn't stored in the AlwaysQA database, AI context, logs, or Run Evidence; only sanitized diagnostics such as delivery timing and failure classification remain. Deep dive: approve the exact destination before submitting a secret Keeping credentials out of an AI prompt solves only half the problem. Trusted code must also know where it is allowed to submit them. An application may start sign-in here: https://app.example.com/loginand collect the password or Email Code here: https://auth.example-provider.comThey're different origins. The Login Page URL tells AlwaysQA where to begin navigating; it isn't permission to submit authentication material. During setup, AlwaysQA follows the visible flow and detects the exact public HTTPS Login Origin that would receive the identifier or credential. The authenticated user must review and approve that origin before the QA Account becomes Ready. If a later Run reaches a different origin, AlwaysQA stops before submitting authentication material. The QA Account moves to Needs Attention, dependent Schedules pause, and the team must approve the new origin before those Runs can continue. This protects against an unexpected redirect receiving stored authentication material. It doesn't replace an application-security review. Teams should separately assess authenticator lifecycle, session management, recovery, and abuse controls against guidance such as NIST SP 800-63B. Where AlwaysQA fits alongside Playwright or Cypress Keep your existing lower-level automated coverage and scripted browser tests. They're the right choice for fast deterministic assertions in a development pipeline. AlwaysQA covers a different operating need: post-deploy QA that a Coding Agent can create, run, inspect, and use during fix-and-rerun work without receiving the QA Account's authentication material. The AlwaysQA workflow overview shows how Test Cases, Runs, Outcomes, and Evidence connect.Keep a scripted browser test when… Use AlwaysQA when…The team needs deterministic assertions on implementation details The team needs to verify an observable customer outcomeThe check belongs on every commit The check should run after a change, on demand, Daily, or WeeklyThe team owns and maintains the test code and selectors The workflow is defined through instructions and a Success ConditionExisting CI infrastructure already protects its secrets The Coding Agent should operate QA without retrieving the credentialA machine-readable failure is enough The team needs observations, a timeline, replay context, and an Issue ReportThe strongest setup isn't either/or. Scripted tests catch known implementation failures quickly. AlwaysQA verifies that the complete critical flow still works after the change reaches the application. Separate a broken product from an inconclusive Run Every finished Run receives one of three Outcomes:Passed: the Success Condition was observed. Failed: valid Evidence showed that the expected application behavior was broken. Needs Attention: AlwaysQA couldn't reach a trustworthy conclusion.The third Outcome matters in authenticated testing. A missing email, delayed delivery, expired Email Code, bot protection, or unclear result shouldn't be reported as proof that the product failed. Some problems affect only the current Run. If CAPTCHA appears, for example, the Run ends as Needs Attention without asking AI or trusted code to bypass it, while the QA Account remains Ready. Other problems require account-level action. An explicitly rejected password or a changed Login Origin can move the QA Account to Needs Attention, make it unavailable for new Runs, and pause dependent Schedules until the user resolves it. AlwaysQA preserves structured Evidence: a summary, browser observations, a compact action timeline, the final Outcome, guidance when action is required, and a temporary browser replay when available. A Failed Run can also produce an editable Markdown Issue Report without including QA Account secrets, Email Codes, or raw authentication messages. Use deployment QA history to connect that Evidence to the affected check version, fix the problem, and rerun the same Test Case. Current authentication coverage The safe product promise is a precise one.Supported now Not supported nowUsername or email plus password Email sign-in linksExact six-digit codes delivered by email SMS codesHosted authentication pages with an approved Login Origin TOTP authenticator codesFresh sign-in during every Run Reusing stored authenticated browser stateObservable post-login browser workflows SSO or OAuth popupsOn-demand, Daily, or Weekly Runs PasskeysHuman approval during a RunCAPTCHA bypassThese boundaries are useful qualification criteria. If a workflow needs a magic link, SMS, passkey, OAuth popup, or human approval, use a different testing approach for that path today. Put one authenticated workflow under post-deploy QA Start with the protected task whose unnoticed failure would cost the team most. Login by itself is usually too narrow. Before scheduling the Test Case:create a dedicated identity with minimum permissions; select Password or Email Code explicitly; approve the exact HTTPS Login Origin; use synthetic data that the Run can safely change; choose a workflow with a visible, business-relevant result; define cleanup for records the Run creates; make one person responsible for Failed and Needs Attention Outcomes; and run the Test Case once on demand to review its Evidence.Then add a Daily or Weekly Schedule when the workflow and test data are stable. For deployment-specific verification, run the Test Case again after the relevant change ships. Create your AlwaysQA account and put the authenticated workflow your customers depend on under post-deploy QA. Frequently asked questions Does AlwaysQA expose passwords or Email Codes to AI agents? No. Trusted code handles the configured password, raw authentication message, and six-digit Email Code. The Coding Agent and browser agent receive only the non-secret workflow context and sanitized sign-in result they need. Does AlwaysQA need access to an existing company mailbox? No. Each Email Code QA Account uses an AlwaysQA-controlled QA Inbox instead of connecting to an existing company mailbox. Can one QA Account test password and email-code authentication? No. Each QA Account has one immutable Authentication Method. Create separate Password and Email Code QA Accounts and assign each Test Case explicitly. What happens when the hosted authentication origin changes? AlwaysQA stops before submitting authentication material, moves the QA Account to Needs Attention, and pauses dependent Schedules. The user must review and approve the new Login Origin before Runs can continue.

To diagnose browser test failures, we recommend this process: start with what the Run can prove, not with a guess about root cause. AlwaysQA gives your Coding Agent a clear Outcome and structured Evidence, then provides private browser replay when the text alone can't explain the observed behavior. Most teams make the mistake of treating a red result or a replay as the diagnosis. Neither is enough. Failed tells you the Success Condition wasn't met; replay shows what the browser displayed. The useful question is narrower: what did the Run observe, what happened immediately before the workflow diverged, and what should the developer inspect next? That distinction matters after a deployment, when every blind reproduction attempt delays the fix. AlwaysQA turns one completed Run into a shared investigation record that the Coding Agent, developer, and product owner can use without retelling the failure from memory. The use case: investigate a failed browser workflow without starting from zero Imagine a critical Test Case that uploads a CSV file, maps its columns, starts an import, and waits for a completion message. Its Success Condition is explicit: the import must finish and the newly imported records must appear in the application. After a deployment, the Run finishes with a Failed Outcome. The final screen is still showing “Processing,” and the expected records aren't visible. A generic alert leaves the team with unanswered questions:Did the file upload succeed? Did the browser click the intended control, or did another element intercept the interaction? Did the page navigate or stall? Was there a visible validation error? Did the workflow demonstrably fail? Or did the Run lack enough evidence to confirm success?AlwaysQA preserves the investigation trail as structured Evidence. The developer can start with the summary, inspect the observations and compact action timeline, and open the temporary replay only if the text doesn't explain the behavior. The outcome isn't an automatic root-cause claim. It's a tighter, evidence-backed starting point that cuts out blind reproduction work. When this use case fitsSituation Why the Evidence mattersA critical flow fails after deployment The team needs to separate a confirmed regression from an unresolved Run.A failure is difficult to reproduce locally Observations preserve the last confirmed state and the action immediately before it.A visual interaction looks suspicious Temporary replay can reveal an overlay, redirect, reset, or timing problem.A Coding Agent must hand work to a developer The Issue Report carries expected behavior, observed behavior, reproduction steps, and Run context.The first decision: wait, investigate, or correct the Test Case? Reliable diagnosis starts with precise language. A Run State describes where execution is in its lifecycle. An Outcome describes what a finished Run established.Field Possible values What it answersRun State Queued, Running, Finished Is the Run still in progress?Outcome Passed, Failed, Needs Attention What did the finished Run establish?Only a Finished Run has an Outcome. Use it to choose the next move:What you see What it establishes Next moveQueued or Running The Run hasn't reached a conclusion. Wait for it to finish; don't diagnose partial Evidence.Finished + Passed The Evidence confirms the Success Condition. Record the pass and move on.Finished + Failed The Evidence confirms the expected behavior didn't occur. Inspect the failure boundary and prepare an Issue Report.Finished + Needs Attention AlwaysQA couldn't responsibly confirm pass or failure. Follow the Attention Reason instead of assigning a product defect.Needs Attention may point the team to a QA Account, unclear Test Case, unusable Test File, or Success Condition that can't be observed reliably. That's useful information. It prevents an uncertain automation result from becoming a misleading bug ticket. You can see how this fits into the broader verification workflow in How AlwaysQA works. Use an evidence ladder, not a replay-first habit AlwaysQA organizes a Run so the fastest evidence is available first. That creates a practical investigation ladder:Outcome and summary: read the conclusion first. Observations: identify the meaningful browser states, including the last one that matched expectations. Action timeline: connect the divergence to the preceding interaction. Browser replay: use the visual record only when timing, layout, or interaction details could change the diagnosis.This order keeps diagnosis efficient. A concise final observation may reveal a validation message immediately. There's no reason to scrub through an entire replay when the relevant evidence already says, for example, that the import remained in “Processing” and the expected completion state never appeared. The principle is simple: preserve the evidence needed for diagnosis before asking someone to form a theory. Microsoft's Playwright Workspaces guidance uses the same broad pattern for browser-test diagnostics: retain reports and test artifacts, then inspect them after the Run (Microsoft Learn: advanced browser-test diagnostics). AlwaysQA applies that principle through its own Outcome, Evidence, replay, and Issue Report model. What a useful Evidence record looks like Here is an illustrative record for the CSV import scenario. It shows the level of specificity a developer needs without pretending to identify an unsupported root cause.Test Case: Import customers from a valid CSV file Success Condition: Import completes and the imported records appear in the customer list Outcome: Failed Summary: The file was accepted and the import started, but the completion state wasn't observed. Final observation: The import page continued to display “Processing”; no completion message or imported records were visible. Last action: Waited for the import result after selecting “Start import.” Final URL: https://qa.example.test/imports Replay: Available temporarilyThis example proves that the expected end state wasn't reached during the Run. It doesn't prove whether the cause was a background job, an application response, a frontend state update, or something else. The developer still owns causal diagnosis, but begins with a far narrower search area.Reality check: Browser automation can show what happened in the browser. It can't, by itself, prove why an internal service behaved that way. Treat Evidence as the bridge to engineering diagnosis, not as a substitute for application logs, traces, or code inspection.Read each Evidence layer for a different answer Summary: what did the Run establish? The summary should let a developer understand the observed result without watching the whole session. It connects the Outcome to the Success Condition and stays within what the Run can support. Good summary:Checkout reached the payment confirmation step, but no order confirmation appeared and the order wasn't visible in order history.Weak summary:Checkout is broken because the payment service timed out.The second statement may sound more useful, but it invents a cause unless the browser Evidence directly supports it. AlwaysQA doesn't label the summary as an “AI diagnosis.” That restraint makes the record safer to use in triage. Observations: what meaningful states appeared? Observations preserve the significant states encountered during the Run: a successful sign-in, an accepted upload, an error message, a missing confirmation, or a page that stopped changing. Read them in sequence. The last successful observation identifies how far the workflow got; the first unexpected observation marks where the investigation should begin. If the final screen looks wrong but all earlier milestones are intact, the likely search area is much smaller than the complete user flow. Action timeline: what happened immediately beforehand? The compact action timeline connects an observation to browser activity: navigation, typing, selecting a file, clicking, or waiting for a visible result. It helps answer whether the failure followed a specific interaction or whether the page never reached a usable state. This is especially useful for intermittent failures. Comparing the final meaningful action and observation across Runs can expose a repeatable boundary, even when the eventual cause lives outside the browser. Replay: what did the interaction look like? Open the browser replay when the visual sequence matters. It can help answer questions such as:Did a loading overlay block the next control? Did a menu open, accept the click, and then close before the selection registered? Did the page redirect before the confirmation appeared? Was the expected element below the fold or obscured? Did a multi-step form reset? If so, at which transition?Replay is most useful for animation, timing, layout, and interaction ambiguity. It's less efficient when the summary already names a clear validation message. Browser replay is temporary; structured Evidence is the durable record AlwaysQA retains the structured Evidence associated with the Run. Browser replay is a provider-hosted recording and is available only for the provider's retention period. Because that availability can change, don't treat replay as the permanent system of record. The practical rule is simple:Review replay promptly when a failed Run needs visual investigation. Capture the relevant finding in the Issue Report or engineering notes. Rely on the structured summary, observations, timeline, and Run context for durable history.Replay access is private. AlwaysQA exposes it through an authenticated, short-lived handoff rather than placing a permanent public recording URL in the Run record. If the replay has expired, the structured Evidence remains available. For a longer-term view across deployments and Test Case versions, use Deployment QA History. That page owns the historical comparison workflow; this use case focuses on investigating one Run well. Technical deep dive: where browser Evidence stops Structured Evidence and browser replay serve different consumers. The Coding Agent needs compact text it can retrieve through MCP and reason over. A User needs a private visual session only when the interaction itself is unclear. AlwaysQA therefore returns Evidence with the Run and handles replay through a separate, authenticated browser handoff. That separation matters when replay playback has latency or the recording has expired. The Coding Agent can still inspect the Outcome, observations, and timeline without waiting for video. Now consider an SSO-backed import. Browser Evidence may show that the SSO redirect completed, the file was accepted, and the page remained on “Processing.” That bounds the failure. It still can't tell you whether an internal API returned an error, a background API call never completed, or the frontend ignored a valid response. That next step needs application-side records; NIST's log-management guidance explains why retained events matter for operational analysis and reconstruction. This is the handoff point. Take the last confirmed browser state and compare it with application logs, service telemetry, and recent code or configuration changes. There is still maintenance work. Someone must review temporary replay before it expires, add the relevant visual finding to the ticket, and keep Test Cases and Success Conditions specific enough to produce useful Evidence. AlwaysQA reduces reproduction work; it doesn't remove ownership or engineering judgment. Failed is evidence of a missed Success Condition, not evidence of a particular root cause. Needs Attention is uncertainty made explicit, not a softer word for failure. Turn a Failed Run into a developer-ready Issue Report When a Run fails, AlwaysQA can generate an editable Markdown Issue Report. It's designed to move the relevant Evidence into the team's existing engineering workflow without forcing someone to reconstruct the browser session from memory. The report uses a predictable sequence: What happened → Expected result → Steps to reproduce → Key observations → Evidence → Run context. A developer can scan the difference between expected and observed behavior before opening the full Run. The report is intentionally compact. It includes up to five key observations, the first two and last three, and the last action. The authorized Run link remains the place to inspect the complete timeline and available replay.Included Deliberately limited or excludedOutcome and evidence-backed summary Unsupported root-cause claimsExpected result and reproduction steps The full action timeline in the pasted reportSelected key observations and last action Test File contentsSanitized URL and relevant Run context URL query strings, fragments, and embedded user informationTest File name, type, and size when relevant Known QA Account credentialsThe Issue Report is editable before it leaves AlwaysQA. Copying it is an explicit User action; AlwaysQA doesn't silently create an issue in an external tracker. A developer can add service logs, an owner, severity, or a suspected code path before filing it. If your team wants to convert verified defects into permanent coverage, follow the workflow in How to turn bug reports into regression tests. Handle replay and test data as sensitive operational material AlwaysQA sanitizes exported Run information and excludes known credentials and Test File contents from the Issue Report. That reduces accidental exposure, but it doesn't make every visual browser session inherently non-sensitive. A replay can show whatever the test browser was allowed to display. Your QA environment might contain names, records, account details, or uploaded values visible on screen. Use dedicated QA Accounts, synthetic data, and purpose-built Test Files. Don't use production credentials or confidential customer data just because the test runs in an automated browser. This is the right security promise: the product limits what it stores and exports, access to replay is private and temporary, and your team still controls what data the QA workflow can see. You can review the broader product capabilities on the AlwaysQA features page. A six-step browser failure investigation workflow Use this sequence when a Run doesn't pass:Confirm the Run is Finished. Partial Evidence isn't a conclusion.Branch on the Outcome. Investigate Failed. For Needs Attention, correct the stated blocker before involving a developer.Find the failure boundary. Compare the last successful observation with the first unexpected state. Stop reading once the relevant transition is clear.Check the preceding action. What changed immediately before the browser diverged?Use replay selectively. Open it for timing, layout, redirects, or interaction ambiguity. Review it before retention expires.Complete the handoff. Edit the Issue Report, add application-side evidence, and assign an owner. After the fix, rerun the same Test Case against the same Success Condition.For developers working through MCP, the Coding Agent can retrieve the Run details and Evidence without a separate manual QA handoff. When visual review is necessary, the User opens the authenticated replay in the browser. Text stays agent-readable; recording access stays private. What changes for the teamWithout structured Evidence With AlwaysQA“The test is red.” The Outcome states Passed, Failed, or Needs Attention.Someone reruns the whole workflow from memory. The last confirmed state and preceding action define where to start.A developer watches every recording. The team opens replay only when visual context can change the decision.An ambiguous result becomes a low-quality bug. Needs Attention routes the team to the unresolved input or condition.The ticket lacks reproducible context. The editable Issue Report carries expected behavior, observations, and Run context.That is the commercial value: less time reconstructing the failure, fewer unsupported defect claims, and a cleaner handoff from verification to engineering. AlwaysQA doesn't guess the root cause. It removes avoidable uncertainty before the root-cause investigation begins. Diagnose the next failed Run with evidence, not guesswork Connect AlwaysQA to your Coding Agent, run a critical Test Case, and inspect the Evidence returned with the Outcome. If a workflow fails, you'll have a structured record for triage and a temporary replay when the browser sequence needs closer inspection. Start using AlwaysQA and give every failed Run a clearer path to a decision. Frequently asked questions Does AlwaysQA automatically identify the root cause of a failure? No. AlwaysQA reports what the Run observed and whether the Success Condition was met. Its Evidence can narrow the investigation substantially, but a developer may still need application logs, traces, database records, or code inspection to establish the underlying cause. What is the difference between Failed and Needs Attention? Failed means the Evidence supports the conclusion that the Success Condition wasn't met. Needs Attention means the Run couldn't responsibly confirm pass or failure and includes a reason plus a recommended next action. How long is browser replay available? Replay availability follows the browser provider's retention period, so it should be treated as temporary. Review it promptly. AlwaysQA's structured Evidence remains with the Run after the recording is no longer available. Does AlwaysQA create an issue in our tracker automatically? No. A Failed Run can produce an editable Markdown Issue Report, but copying and filing it is an explicit action. Your team decides what to add and where to send it.

When intermittent browser test failures disappear on rerun, we recommend this process: investigate the original Run before calling the problem fixed or the test flaky. AlwaysQA preserves the Outcome and structured Evidence from that specific evaluation, with temporary browser replay when visual context matters. Most teams make the same mistake. They rerun the workflow immediately, see a pass, and treat the new result as an explanation for the old one. It isn't. A successful rerun proves that the Success Condition was met during the rerun; it doesn't erase an earlier Failed Outcome or explain why the browser previously stopped. This use case is about that gap: intermittent browser test failures, hard-to-reproduce browser bugs, and scheduled checks whose failure disappears before a developer opens the application. The use case: the overnight Run failed, but the morning rerun passed Imagine a scheduled checkout Test Case in a QA environment. It uses a dedicated QA Account and synthetic payment data. Its Success Condition is explicit: submitting the order must display a confirmation and add the order to the account history. At 02:00, the scheduled Run finishes as Failed. The payment form was submitted, but the browser remained on “Processing,” and the confirmation never appeared. At 09:15, a developer repeats the workflow. It passes.Evidence Overnight Run Morning rerunOutcome Failed PassedLast confirmed state Payment submitted Order confirmation displayedFinal observation Page remained on “Processing” Order appeared in account historyWhat it proves The earlier Success Condition wasn't met The workflow worked during the later evaluationThe second result is good news, but it leaves the investigation open. The earlier behavior could reflect application state, test data, authentication, a redirect, a delayed background job, or a dependency that recovered before morning. Browser Evidence shouldn't choose among those causes without support.Reality check: A pass after a failure isn't a root-cause analysis. It is one more observation.If this Test Case runs on a Daily or Weekly Schedule, see how recurring regression testing keeps critical flows under repeatable coverage. A passing rerun does not invalidate the original Outcome AlwaysQA separates Run State from Outcome:Field Values What it meansRun State Queued, Running, Finished Where the Run is in its lifecycleOutcome Passed, Failed, Needs Attention What a Finished Run establishedEach Run stands on its own Evidence:Passed: the Evidence confirms that the Success Condition was met in that Run. Failed: the Evidence confirms that the expected behavior didn't occur in that Run. Needs Attention: AlwaysQA couldn't responsibly confirm pass or failure from the available Evidence.A later pass doesn't rewrite a previous failure. The same is true in reverse: an earlier pass doesn't make a new Failed Outcome less important. Be careful with the word flaky. One failure followed by one pass is a mismatch, not a diagnosis. Before you blame the Test Case, compare what actually ran: the Test Case version, inputs, application environment, deployment context when available, and the point where browser behavior diverged. For the broader distinction between Failed, Needs Attention, and root-cause diagnosis, use the browser test failure diagnosis guide. Start with the original Run, not a manual reconstruction The original Run preserves the closest record of the failure as it happened. Review it in this order:Outcome and summary: what did the Run establish against the Success Condition? Observations: what meaningful browser states appeared before the Run finished? Action timeline: what did the browser do immediately before the unexpected state? Replay: does the visual sequence reveal timing, layout, redirect, or interaction context missing from the text?This order prevents replay from becoming another slow search. A final observation that names a visible validation error may be enough to route the issue. A loading overlay that briefly intercepts a click may require visual review. An illustrative original-Run record For the checkout scenario, the useful record might look like this:Test Case: Complete checkout with a QA Account Success Condition: Confirmation appears and the order is visible in account history Outcome: Failed Summary: Checkout was submitted, but the confirmation state wasn't observed. Last confirmed observation: The browser accepted the checkout submission and displayed “Processing.” Final observation: No confirmation appeared, and the order wasn't visible in account history. Last action: Waited for the checkout result after selecting “Place order.” Final URL: https://qa.example.test/checkout Replay: Available temporarilyThis record narrows the boundary to the transition after submission. It doesn't prove that the payment service failed, the frontend lost an update, or a background task exceeded a time limit. Those are hypotheses for the developer to test. Use the Evidence to choose the next investigation Don't ask every developer to watch every replay. Route the Run according to what the Evidence already shows.Original Run Evidence Best next action Do not assumeVisible validation or application error Check the relevant application logs and request records The Test Case is flakyPage remains in one state without progress Inspect the background operation, response timing, and state updates The browser simply needed more timeRedirect lands on an unexpected page Review routing, authentication, and recent configuration changes The original URL was wrongControl appears late, moves, or is obscured Review replay while available and inspect the UI transition The backend caused the failureNeeds Attention with a specific reason Follow the recommended action for the QA Account, Test File, Test Case, or Success Condition A product defect was confirmedLater Run passes under comparable conditions Compare the two Run records and look for a changing dependency or state The earlier failure no longer mattersThe goal is not to solve the incident inside the Run page. It is to enter engineering diagnosis with fewer unsupported branches. Deep dive: compare Runs only when their context is comparable “It passed for me” is weak evidence when the rerun didn't use the same conditions. Before comparing Outcomes, check the context attached to each Run.Comparison point Why it changes the interpretationTest Case and version An edited workflow or Success Condition may be testing something different.QA Account selection Account state, role, or authentication setup can change the path through the app.Test File metadata A replacement file is a new immutable input; its name, type, and size may differ.Deployment or environment metadata A rerun after a rollback, configuration change, or new release isn't the same condition.Time and application state Expiring sessions, queued work, rate limits, and data cleanup can make behavior time-dependent.AlwaysQA keeps each Run's historical record tied to the inputs and metadata available when it ran. Later changes don't rewrite that finished Outcome. Environment or deployment details can only help when they were captured, so don't invent missing context after the fact. Consider a passwordless SSO workflow. The original Run may show that SSO completed and the dashboard loaded, while the next protected page redirected back to sign-in. If that page depends on an internal API, browser Evidence can't distinguish an API error from rejected session state or a frontend routing decision. That is why application event records still matter; NIST's log-management guidance describes their value for operational analysis and reconstruction. A later pass with a fresh session doesn't prove why the earlier session failed. It tells the developer where to compare authentication events, application logs, and request records. This is where browser Evidence stops and application-side diagnosis starts. Without application-side records, the browser can identify the boundary but not explain an internal service decision. Review replay before it expires, but do not make it the system of record Browser replay is useful when sequence and appearance matter. It can show whether:a loading layer blocked the next action; the interface reset after submission; a redirect happened before the expected confirmation; a control appeared too late to be used; or the page stopped changing even though no explicit error appeared.This matches a broader browser-testing practice: Microsoft's Playwright Workspaces documentation uses reports and retained artifacts to investigate completed test Runs (Microsoft Learn: browser-test diagnostics). AlwaysQA's implementation is its own: structured Evidence remains with the Run, while the provider-hosted browser replay is temporary. Replay access is private and uses an authenticated, short-lived handoff. Review it promptly. If it has expired, the summary, observations, compact action timeline, and Run context remain available. There is operational friction here. Someone must review time-limited replay, preserve the relevant visual finding in the Issue Report, and keep the Test Case observable. Evidence reduces reproduction work; it doesn't remove maintenance, ownership, or application telemetry. Turn the disappearing failure into a useful Issue Report For a Failed Run, AlwaysQA can prepare an editable Markdown Issue Report. Its sections are predictable: What happened → Expected result → Steps to reproduce → Key observations → Evidence → Run context. The report includes up to five key observations, the first two and last three, plus the last recorded action. The authorized Run link remains the place to inspect the complete timeline and replay while it is available.Included in the handoff Kept out or deliberately limitedOutcome and evidence-backed summary Unsupported root-cause claimsExpected result and reproduction steps The full action timeline in copied MarkdownSelected observations and last action Test File contentsSanitized URL and relevant Run context URL query strings, fragments, and embedded user informationTest File name, type, and size when relevant Known QA Account credentialsCopying the report is explicit. AlwaysQA doesn't silently create an issue in an external tracker. The developer can add logs, a suspected component, impact, owner, and links to both the original Run and later rerun. Use Deployment QA History to keep the failed Run, later pass, Test Case version, and fix-and-rerun sequence connected over time. Treat visual replay as potentially sensitive AlwaysQA sanitizes exported Run information and excludes known credentials and Test File contents from the Issue Report. A replay, however, can display whatever the QA browser was allowed to see. Use dedicated QA Accounts, synthetic records, and non-sensitive Test Files. Don't put production credentials, payment-card details, confidential customer data, or regulated information into a browser workflow simply because the Run is automated. A practical workflow for a failure that disappearsDo not dismiss the original Run. Confirm it is Finished and read its Outcome.Preserve the fast-expiring context. If replay is available and visual behavior matters, review it now.Mark the failure boundary. Record the last expected observation, the first unexpected state, and the preceding action.Compare the rerun context. Check Test Case version, inputs, QA Account, environment, deployment metadata, and timing where available.Choose the right branch. Failed goes to product investigation; Needs Attention goes to the stated corrective action.Add application evidence. Inspect logs, traces, background work, authentication events, or configuration changes around the original Run time.File the edited Issue Report. Link the original failure and later pass, assign an owner, and keep the causal conclusion open until the evidence supports it.Verify the fix or monitor for recurrence. Run the same Test Case again and use later scheduled Runs to determine whether the behavior returns.See how AlwaysQA works for the complete Test Case, Run, Outcome, and Evidence flow. What changes for the teamWithout original-Run Evidence With AlwaysQAThe rerun passed, so the incident is closed. The pass and failure remain two separate, inspectable Outcomes.Someone tries to reproduce the entire workflow from memory. The failure boundary identifies where engineering should begin.“Flaky” becomes the default explanation. The team compares Run context before classifying the behavior.The replay expires and the investigation loses its record. Structured Evidence remains after visual replay is unavailable.The ticket says “sometimes checkout fails.” The Issue Report states expected behavior, observed behavior, selected Evidence, and Run context.The sales value is practical: fewer “cannot reproduce” loops, better developer handoffs, and less pressure to invent a cause just because the defect disappeared. Investigate the Run that failed, not only the rerun that passed Create a browser Test Case for a critical workflow and give it an observable Success Condition. When a scheduled Run fails, AlwaysQA preserves the original Evidence so your team can inspect what happened, compare later Runs, and start in the right part of the system. Start using AlwaysQA and keep the failed Run in front of the investigation. Frequently asked questions Is a Test Case flaky if it fails once and passes on rerun? Not necessarily. The two Outcomes show inconsistent observed behavior, but they don't identify the cause. Compare the Run context and failure boundary before deciding whether the Test Case, application, data, environment, or a dependency changed. What remains after browser replay expires? AlwaysQA retains the structured Evidence: summary, observations, compact action timeline, and Run context. Replay is temporary and should be reviewed promptly when visual context matters. Can AlwaysQA determine the root cause of an intermittent failure? AlwaysQA identifies what the browser observed and whether the Success Condition was met. A developer may still need application logs, traces, database records, or configuration history to establish the cause. What if the original Run finished as Needs Attention? Follow the stored Attention Reason and recommended action. Needs Attention means the Run couldn't support a pass-or-fail conclusion; it should not be filed as a confirmed product defect.