Turn a failed browser Run into a defensible next decision

Authenticated workflow testing must prove that the right user can sign in and finish the protected task after a deployment. AlwaysQA tests password and six-digit email-code sign-in through a fresh browser session, giving the team visibility into the complete workflow without giving the decrypted password, QA Inbox, Email Code, or raw authentication message to a Coding Agent or browser agent. Most teams make the mistake of treating login as a checkpoint. The expensive failures happen across the whole chain: form, credential submission, identity-provider redirect, session, permissions, and the product action the customer came to complete.Reality check: A green login page proves that the page loaded. It doesn't prove that an authenticated customer can use the product.The use case: verify the job behind the login Consider a SaaS team that has just shipped a small permissions change. Public pages respond normally. The sign-in form loads. Infrastructure monitoring is green. But standard users now return to the login page after entering a valid password. Administrators are unaffected, so the problem survives an internal spot check. Customers discover it first. A useful Test Case doesn't stop at “login succeeded.” It verifies a visible product result:Starting URL https://app.example.com/login Instructions Sign in as the configured standard user. Create a test project named with today's date. Confirm that it appears in the project list, then delete it. Success Condition The project appears after creation and no longer appears after deletion.That Run covers the sign-in form, credential acceptance, redirect, authenticated session, role permissions, record creation, persistence, and cleanup. It answers a business question, not a transport-level one. For broader coverage across signup, onboarding, permissions, billing, and other release risks, use the web app regression testing checklist. This page focuses specifically on authentication and the workflows gated by it. Why URL checks and login-only tests miss real failures Authentication is a dependency chain. A failure at any layer can block the customer while shallow monitoring still reports success.Layer What a shallow check sees What the customer experiencesLogin page The URL returned 200 OK The submit action does nothingPassword submission The form accepted input A valid credential is rejectedEmail delivery The email service is available The current six-digit code never arrivesCode verification The form accepts six characters A fresh code is rejected or expiresHosted authentication The identity provider responded The redirect returns to the wrong originSession Authentication completed The next page asks the customer to sign in againAuthorization The dashboard opened The account has the wrong role or permissionsProduct workflow Login passed The customer still can't complete the taskCustomers don't buy access to a login screen. They buy what the authenticated product lets them do. Use dedicated QA Accounts, not real customer credentials AlwaysQA uses a QA Account: a named, Project-owned authentication resource for a dedicated identity in your application. Multiple Test Cases in the same Project can reuse it. Create separate QA Accounts for the roles you need to verify, such as standard user, administrator, or billing manager. Give each one only the permissions and test data required for its workflows. That reduces operational risk and makes a failure easier to interpret. It also follows the least-privilege principle described in NIST SP 800-53 Revision 5. Each QA Account has one explicit, immutable Authentication Method:Authentication Method What you configure When to use itPassword A dedicated identifier and password The application signs in with username or email plus passwordEmail Code A dedicated identity using an AlwaysQA-controlled QA Inbox The application sends an exact six-digit code by emailAlwaysQA doesn't infer the method from the page. If the product supports both methods, create two QA Accounts and assign each Test Case to the appropriate one. The QA Account feature overview explains how this setup supports role-specific and permission-specific workflows. The trust boundary: the agent navigates, trusted code authenticates The browser agent needs to understand the interface. It doesn't need the authentication secret. During a Run, the agent navigates to the matching sign-in form and requests one QA Account Sign-in action. Trusted code completes the configured Authentication Method and returns only a sanitized result. After successful sign-in, the browser agent continues with the Test Case instructions. The boundary is easier to see as a handoff: Browser agent Trusted sign-in action Application | | | | navigate to login | | |-------------------------->| | | | submit protected material | | |-------------------------->| | | sanitized result | |<---------------------------| | | continue product workflow | |------------------------------------------------------->|What stays outside agent context:Authentication material Handled by Given to Coding Agent or browser agent? Retained as Run Evidence?Decrypted password Trusted sign-in action No NoQA Inbox address Authenticated user and trusted code No NoRaw authentication message Trusted code No NoSix-digit Email Code Trusted code No NoSanitized sign-in result Trusted code and Run Yes Only sanitized observationsIt's the product's core security distinction: the agent can evaluate the authenticated experience without receiving the material used to enter it. It's still a dedicated identity, and it still needs operational discipline. Use synthetic data, minimum permissions, a cleanup strategy, and an owner who can rotate or remove the identity when the application changes. How password authentication works For a Password QA Account, configure the identifier, password, Login Page URL, and approved Login Origin directly through AlwaysQA. Every Run starts with fresh browser state. The browser agent reaches the visible sign-in interface, then trusted code submits the configured authentication material. The agent receives a sanitized result and continues only after the sign-in action completes. The password is write-only from the perspective of product read interfaces. It isn't returned to your Coding Agent, placed in its prompt, or exposed to the browser agent. That's different from instructing an AI agent to retrieve a secret from an environment variable and type it into a page. The agent controls the workflow; AlwaysQA controls the credential boundary. How six-digit email-code authentication works Passwordless testing is harder because “use the latest email” isn't a reliable rule. An inbox may contain a code from an earlier attempt, delayed mail, or multiple messages that look valid. Each Email Code QA Account receives one stable, opaque QA Inbox. Its address contains no User, Project, or QA Account name. The authenticated user copies that address when provisioning the dedicated identity in the application; the Coding Agent and browser agent never receive it. For each unattended Run, AlwaysQA:opens a fresh browser session; navigates to the matching sign-in form; opens one short-lived authentication attempt; requests one authentication message; accepts only a fresh message delivered after that request; extracts one exact six-digit Email Code through trusted code; submits that code on the approved Login Origin; and continues with the authenticated Test Case.The Run doesn't reuse an old code, request an automatic resend, or try several candidates after rejection. Runs using the same Email Code QA Account execute one at a time. Serialization prevents overlapping messages and product activity from being attributed to the wrong Run. Other Test Cases remain independent. Each Email Code QA Account receives an AlwaysQA-controlled QA Inbox. Your company mailbox isn't part of the workflow. When trusted processing ends, the provider copy of the raw authentication message is deleted. The payload isn't stored in the AlwaysQA database, AI context, logs, or Run Evidence; only sanitized diagnostics such as delivery timing and failure classification remain. Deep dive: approve the exact destination before submitting a secret Keeping credentials out of an AI prompt solves only half the problem. Trusted code must also know where it is allowed to submit them. An application may start sign-in here: https://app.example.com/loginand collect the password or Email Code here: https://auth.example-provider.comThey're different origins. The Login Page URL tells AlwaysQA where to begin navigating; it isn't permission to submit authentication material. During setup, AlwaysQA follows the visible flow and detects the exact public HTTPS Login Origin that would receive the identifier or credential. The authenticated user must review and approve that origin before the QA Account becomes Ready. If a later Run reaches a different origin, AlwaysQA stops before submitting authentication material. The QA Account moves to Needs Attention, dependent Schedules pause, and the team must approve the new origin before those Runs can continue. This protects against an unexpected redirect receiving stored authentication material. It doesn't replace an application-security review. Teams should separately assess authenticator lifecycle, session management, recovery, and abuse controls against guidance such as NIST SP 800-63B. Where AlwaysQA fits alongside Playwright or Cypress Keep your existing lower-level automated coverage and scripted browser tests. They're the right choice for fast deterministic assertions in a development pipeline. AlwaysQA covers a different operating need: post-deploy QA that a Coding Agent can create, run, inspect, and use during fix-and-rerun work without receiving the QA Account's authentication material. The AlwaysQA workflow overview shows how Test Cases, Runs, Outcomes, and Evidence connect.Keep a scripted browser test when… Use AlwaysQA when…The team needs deterministic assertions on implementation details The team needs to verify an observable customer outcomeThe check belongs on every commit The check should run after a change, on demand, Daily, or WeeklyThe team owns and maintains the test code and selectors The workflow is defined through instructions and a Success ConditionExisting CI infrastructure already protects its secrets The Coding Agent should operate QA without retrieving the credentialA machine-readable failure is enough The team needs observations, a timeline, replay context, and an Issue ReportThe strongest setup isn't either/or. Scripted tests catch known implementation failures quickly. AlwaysQA verifies that the complete critical flow still works after the change reaches the application. Separate a broken product from an inconclusive Run Every finished Run receives one of three Outcomes:Passed: the Success Condition was observed. Failed: valid Evidence showed that the expected application behavior was broken. Needs Attention: AlwaysQA couldn't reach a trustworthy conclusion.The third Outcome matters in authenticated testing. A missing email, delayed delivery, expired Email Code, bot protection, or unclear result shouldn't be reported as proof that the product failed. Some problems affect only the current Run. If CAPTCHA appears, for example, the Run ends as Needs Attention without asking AI or trusted code to bypass it, while the QA Account remains Ready. Other problems require account-level action. An explicitly rejected password or a changed Login Origin can move the QA Account to Needs Attention, make it unavailable for new Runs, and pause dependent Schedules until the user resolves it. AlwaysQA preserves structured Evidence: a summary, browser observations, a compact action timeline, the final Outcome, guidance when action is required, and a temporary browser replay when available. A Failed Run can also produce an editable Markdown Issue Report without including QA Account secrets, Email Codes, or raw authentication messages. Use deployment QA history to connect that Evidence to the affected check version, fix the problem, and rerun the same Test Case. Current authentication coverage The safe product promise is a precise one.Supported now Not supported nowUsername or email plus password Email sign-in linksExact six-digit codes delivered by email SMS codesHosted authentication pages with an approved Login Origin TOTP authenticator codesFresh sign-in during every Run Reusing stored authenticated browser stateObservable post-login browser workflows SSO or OAuth popupsOn-demand, Daily, or Weekly Runs PasskeysHuman approval during a RunCAPTCHA bypassThese boundaries are useful qualification criteria. If a workflow needs a magic link, SMS, passkey, OAuth popup, or human approval, use a different testing approach for that path today. Put one authenticated workflow under post-deploy QA Start with the protected task whose unnoticed failure would cost the team most. Login by itself is usually too narrow. Before scheduling the Test Case:create a dedicated identity with minimum permissions; select Password or Email Code explicitly; approve the exact HTTPS Login Origin; use synthetic data that the Run can safely change; choose a workflow with a visible, business-relevant result; define cleanup for records the Run creates; make one person responsible for Failed and Needs Attention Outcomes; and run the Test Case once on demand to review its Evidence.Then add a Daily or Weekly Schedule when the workflow and test data are stable. For deployment-specific verification, run the Test Case again after the relevant change ships. Create your AlwaysQA account and put the authenticated workflow your customers depend on under post-deploy QA. Frequently asked questions Does AlwaysQA expose passwords or Email Codes to AI agents? No. Trusted code handles the configured password, raw authentication message, and six-digit Email Code. The Coding Agent and browser agent receive only the non-secret workflow context and sanitized sign-in result they need. Does AlwaysQA need access to an existing company mailbox? No. Each Email Code QA Account uses an AlwaysQA-controlled QA Inbox instead of connecting to an existing company mailbox. Can one QA Account test password and email-code authentication? No. Each QA Account has one immutable Authentication Method. Create separate Password and Email Code QA Accounts and assign each Test Case explicitly. What happens when the hosted authentication origin changes? AlwaysQA stops before submitting authentication material, moves the QA Account to Needs Attention, and pauses dependent Schedules. The user must review and approve the new Login Origin before Runs can continue.

Post-deployment smoke testing lets you monitor the high-risk customer flow affected by a release after the code reaches its target environment. AlwaysQA runs the saved browser Test Case on demand and returns Passed, Failed, or Needs Attention with Evidence your team can inspect. A green deployment pipeline proves that the release completed its configured steps. It doesn't prove that a user can sign in, complete checkout, submit a form, or reach the confirmation page.Reality check: Most teams don't have a deployment problem when regressions escape. They have a verification gap. The hard part is maintenance and release latency: too many smoke checks slow the decision, while too few leave the highest-risk behavior unverified.The use case at a glancePost-release smoke check with AlwaysQATrigger An important change reaches production, staging, or another target environmentTest selection The smallest set of Test Cases connected to the changed customer behaviorExecution An on-demand Run started in AlwaysQA or through a connected coding agent after confirmationResult Passed, Failed, or Needs AttentionEvidence Summary, browser observations, action timeline, stopped page, and temporary replay when availableCloseout Fix the regression and run the same Test Case againThe goal isn't to rerun every check after every deployment. It's to get a fast product-level answer about the behavior most likely to have changed. A practical example: the release is green but billing is unreachable Suppose a team changes an authentication callback used after passwordless login. The build passes. The deployment completes. The application starts, and its health endpoint responds. From the infrastructure side, the release looks healthy. The customer flow has more steps:Open the sign-in page. Request and submit the login code. Establish the authenticated session. Open account billing. See the current Plan and the option to manage it.The callback change preserves login but drops the redirect target. Users reach the dashboard, yet the billing route sends them back to sign-in. An AlwaysQA Test Case defines the complete observable behavior:Open the sign-in page, sign in with the configured QA Account, and navigate to billing. The billing page should display the current Plan and an option to manage it.After deployment, the team starts that Test Case on demand. A fresh browser agent attempts the workflow against the released application. If the agent reaches the billing page and observes the Success Condition, the Run passes. If valid Evidence shows the redirect loop or access failure, it fails. If an account or execution problem prevents a trustworthy conclusion, the Run receives Needs Attention instead of pretending that the product regression is confirmed. Here's the release loop: Deploy → run the affected behavior → inspect the Outcome → fix → run again. A successful deployment is not a behavior test Deployment automation answers delivery questions:Was the artifact built? Did the configured checks complete? Did the release reach the target environment? Did the service start and respond to its health probe?A browser smoke test answers a customer question: can someone still complete this important workflow? Those checks overlap, but they aren't interchangeable. A health endpoint can respond while a permission rule blocks the dashboard. A deployment can finish while form validation rejects every valid submission. A shared navigation component can load while sending users to the wrong page. Deployment monitoring, application monitoring, and customer-flow monitoring answer different questions. Deployment monitoring confirms rollout state; application monitoring exposes service health; customer-flow monitoring verifies what the user can complete. A smoke test belongs in the third layer, while still using the first two as release context. Microsoft's safe deployment guidance recommends health checks, issue detection, progressive exposure, and multiple testing methods rather than relying on one gate. Product-level smoke tests supply one missing layer: observable user behavior after the release is live. Match smoke tests to the release Here's the wrong smoke-test policy: “run everything.” That creates latency, spends Run allowance, and turns release verification into a slow regression cycle. Start from the changed behavior and its likely blast radius:Release area Immediate smoke-test candidates WhyAuthentication callback Passwordless login, dashboard access, account security Session and redirect behavior can affect every authenticated routeCheckout Product selection, checkout submission, confirmation A break blocks revenue and may leave ambiguous order stateOnboarding Registration, first setup step, initial dashboard New users can be acquired but fail before activationShared form component Lead form, support form, profile update One validation change may affect several workflowsPermissions Standard-user access, administrator action, restricted page The interface may load while the wrong role is allowed or blockedShared navigation Sign-in redirect, primary dashboard, billing route An unrelated-looking component can break routes across the productUse the regression testing checklist for web apps when the release touches a shared component and the impact isn't obvious. Ask three questions before choosing the Test CasesWhat customer outcome changed directly? Test that first. Which shared dependency could widen the impact? Add one or two adjacent flows when authentication, navigation, permissions, or shared validation changed. What failure would make the release unsafe to leave live? That's the flow for the immediate smoke set.That's what keeps the check set small without making it arbitrary. Save customer behavior as a reusable Test Case Define the workflow once as an AlwaysQA Test Case. Include:the starting URL; practical Instructions for the browser agent; an observable Success Condition; a dedicated QA Account or Test File when needed.Avoid implementation language in the Success Condition. “The callback handler returns the expected state” may be meaningful to a developer but invisible to a customer. Prefer the outcome:A signed-in customer can open billing and see the current Plan with an option to manage it.The Test Case becomes a reusable production smoke test. You don't have to rewrite the workflow after every release, and the saved definition gives later Runs a consistent behavior to evaluate. See how AlwaysQA Test Cases and Runs work for the full browser-evaluation sequence. Start the Run after the target environment is ready When the release reaches its target environment, start the relevant Test Cases with Run now. AlwaysQA creates a fresh Run from the saved Test Case definition. Its execution State moves through Queued, Running, and Finished. The final Outcome remains separate from that progress. If you're using a connected coding agent, you can request a confirmed Run there. This keeps the smoke test close to the repository and release conversation while AlwaysQA performs the actual browser evaluation. A practical release checklist looks like this:Deploy the change. Confirm the target environment is reachable. Start the smallest relevant Test Case set. Wait for every Run to finish. Review Failed and Needs Attention Outcomes. Fix or roll back when the customer impact requires it. Run the affected Test Case again after the repair.AlwaysQA doesn't decide which release is important enough to test. Your team owns that policy and the confirmation to start each Run. Treat Passed, Failed, and Needs Attention differently Every finished Run receives one Outcome. Passed AlwaysQA observed the Test Case's Success Condition during that Run. Passed is evidence that the saved workflow worked at that moment. It isn't proof that every application feature survived the release. Failed Valid Evidence showed that expected application behavior was broken. The browser may have reached an error message, returned to sign-in, stopped before confirmation, or displayed a state that contradicts the Success Condition. Needs Attention AlwaysQA couldn't reach a trustworthy conclusion. An unavailable QA Account, ambiguous Test Case, unusable Test File, or execution problem shouldn't be presented as a confirmed product regression. Needs Attention keeps uncertainty visible without converting it into a false failure. That's why this distinction matters during a release. Failed informs a product decision. Needs Attention tells the team that its verification is incomplete. Investigate the original Run before trying to reproduce it When a smoke test fails, the first useful artifact is the original post-release Run. AlwaysQA preserves structured Evidence that can include:a concise summary; meaningful browser observations; a compact action timeline; the page where the Run stopped; a temporary browser replay, when available.For a Failed Run, the team can prepare an editable Markdown Issue Report with the expected behavior, key Evidence, and a link back to the Run. You don't want the common waste pattern: a developer immediately reruns the workflow manually, gets a different result, and loses the exact state observed after deployment. Start with the Evidence. Reproduce only when the investigation needs more information. The AlwaysQA feature overview shows how browser Runs preserve Outcomes and Evidence. Fix, rerun, and keep both records After correcting the regression, start another Run of the same Test Case. The new Run evaluates the current saved definition. The earlier failure and its Evidence remain in history, so a passing rerun doesn't erase the record of what went wrong. That's important when expected behavior changes between releases. Test Case versioning keeps older results connected to the definition that produced them, while the active Test Case can evolve with the product. The deployment QA history turns the sequence into a durable record: deployment context, check version, failure Evidence, diagnosis, and fix-and-rerun status. The fix isn't verified because the code looks right. It's verified when a fresh Run observes the customer outcome again. Smoke tests, scheduled checks, and broader regression suites do different jobs One test strategy shouldn't be forced to answer every release question.Coverage type Trigger Best questionPost-deployment smoke test Immediately after an important release Did the changed critical flow survive this deployment?Daily or Weekly Schedule Recurring calendar cadence Does this workflow still work as the product and its dependencies change?Broader regression suite CI, release candidate, or planned QA cycle Did the change break a wider set of known behaviors?Progressive rollout During staged exposure Does the release remain healthy as more users receive it?Google Cloud's canary deployment guidance explains how progressive rollouts reduce exposure by testing a new version with part of the user base before a full release. An AlwaysQA smoke test complements that rollout strategy by checking a specific customer behavior in the target environment. Use on-demand Runs for release-specific verification and Daily or Weekly Schedules for ongoing protection. Neither removes the need for unit, integration, end-to-end, or manual testing. Build a release policy the team can follow A smoke-test process fails when “important release” means something different to every developer. Define the trigger before the next incident. Require an immediate smoke check when a release changes:authentication, sessions, or redirects; checkout, billing, or subscription management; onboarding or account creation; permissions or role-specific behavior; a shared form, navigation, or layout component; a third-party integration used inside a customer-critical flow.Then name the owner. Someone must choose the Test Cases, review every non-passing Outcome, and decide whether the release can stay live. Without that owner, “Run a smoke test after release” is advice, not an operating process. Give the next important release a customer-level check Start with one workflow that would make the release unsafe if it broke.Save that behavior as a Test Case. Run it once to confirm the definition and account are ready. Add the Test Case to the release checklist for relevant changes. Start an on-demand Run after deployment. Review the Outcome and original Evidence. If it failed, fix the application and run the same Test Case again.Do not stop at “deployment succeeded.” Check the behavior users depend on. Know whether the customer flow survived the release. Create your first release smoke test. Frequently asked questions What is post-deployment smoke testing? Post-deployment smoke testing runs a small set of high-value checks after a release reaches its target environment. The goal is to confirm that the customer behaviors most likely to be affected still work. How is a smoke test different from a full regression test? A smoke test is intentionally narrow and fast. It checks whether the release is usable at a basic product level. A broader regression suite covers more behaviors and usually takes longer to run and review. Which releases need an AlwaysQA smoke test? Prioritize releases affecting authentication, checkout, onboarding, permissions, shared UI components, forms, or integrations inside important customer flows. The right set depends on the change and its likely blast radius. Can I start a smoke test from my coding agent? Yes. A connected coding agent can request the relevant Test Case Run after explicit confirmation. AlwaysQA performs the browser evaluation and preserves the Run, Outcome, and Evidence. What should I do when a smoke test Needs Attention? Review why the Run was inconclusive. Resolve the account, Test Case, file, or execution problem, then start a fresh Run before treating the release as verified. Should smoke tests also run on a schedule? They can. Use an on-demand Run immediately after a relevant release, then keep the highest-impact workflows on a Daily or Weekly Schedule for ongoing regression coverage. Does a Passed smoke test prove the release is bug-free? No. It shows that the selected Success Condition was observed during that Run. Untested workflows and edge cases remain outside the result.

When intermittent browser test failures disappear on rerun, we recommend this process: investigate the original Run before calling the problem fixed or the test flaky. AlwaysQA preserves the Outcome and structured Evidence from that specific evaluation, with temporary browser replay when visual context matters. Most teams make the same mistake. They rerun the workflow immediately, see a pass, and treat the new result as an explanation for the old one. It isn't. A successful rerun proves that the Success Condition was met during the rerun; it doesn't erase an earlier Failed Outcome or explain why the browser previously stopped. This use case is about that gap: intermittent browser test failures, hard-to-reproduce browser bugs, and scheduled checks whose failure disappears before a developer opens the application. The use case: the overnight Run failed, but the morning rerun passed Imagine a scheduled checkout Test Case in a QA environment. It uses a dedicated QA Account and synthetic payment data. Its Success Condition is explicit: submitting the order must display a confirmation and add the order to the account history. At 02:00, the scheduled Run finishes as Failed. The payment form was submitted, but the browser remained on “Processing,” and the confirmation never appeared. At 09:15, a developer repeats the workflow. It passes.Evidence Overnight Run Morning rerunOutcome Failed PassedLast confirmed state Payment submitted Order confirmation displayedFinal observation Page remained on “Processing” Order appeared in account historyWhat it proves The earlier Success Condition wasn't met The workflow worked during the later evaluationThe second result is good news, but it leaves the investigation open. The earlier behavior could reflect application state, test data, authentication, a redirect, a delayed background job, or a dependency that recovered before morning. Browser Evidence shouldn't choose among those causes without support.Reality check: A pass after a failure isn't a root-cause analysis. It is one more observation.If this Test Case runs on a Daily or Weekly Schedule, see how recurring regression testing keeps critical flows under repeatable coverage. A passing rerun does not invalidate the original Outcome AlwaysQA separates Run State from Outcome:Field Values What it meansRun State Queued, Running, Finished Where the Run is in its lifecycleOutcome Passed, Failed, Needs Attention What a Finished Run establishedEach Run stands on its own Evidence:Passed: the Evidence confirms that the Success Condition was met in that Run. Failed: the Evidence confirms that the expected behavior didn't occur in that Run. Needs Attention: AlwaysQA couldn't responsibly confirm pass or failure from the available Evidence.A later pass doesn't rewrite a previous failure. The same is true in reverse: an earlier pass doesn't make a new Failed Outcome less important. Be careful with the word flaky. One failure followed by one pass is a mismatch, not a diagnosis. Before you blame the Test Case, compare what actually ran: the Test Case version, inputs, application environment, deployment context when available, and the point where browser behavior diverged. For the broader distinction between Failed, Needs Attention, and root-cause diagnosis, use the browser test failure diagnosis guide. Start with the original Run, not a manual reconstruction The original Run preserves the closest record of the failure as it happened. Review it in this order:Outcome and summary: what did the Run establish against the Success Condition? Observations: what meaningful browser states appeared before the Run finished? Action timeline: what did the browser do immediately before the unexpected state? Replay: does the visual sequence reveal timing, layout, redirect, or interaction context missing from the text?This order prevents replay from becoming another slow search. A final observation that names a visible validation error may be enough to route the issue. A loading overlay that briefly intercepts a click may require visual review. An illustrative original-Run record For the checkout scenario, the useful record might look like this:Test Case: Complete checkout with a QA Account Success Condition: Confirmation appears and the order is visible in account history Outcome: Failed Summary: Checkout was submitted, but the confirmation state wasn't observed. Last confirmed observation: The browser accepted the checkout submission and displayed “Processing.” Final observation: No confirmation appeared, and the order wasn't visible in account history. Last action: Waited for the checkout result after selecting “Place order.” Final URL: https://qa.example.test/checkout Replay: Available temporarilyThis record narrows the boundary to the transition after submission. It doesn't prove that the payment service failed, the frontend lost an update, or a background task exceeded a time limit. Those are hypotheses for the developer to test. Use the Evidence to choose the next investigation Don't ask every developer to watch every replay. Route the Run according to what the Evidence already shows.Original Run Evidence Best next action Do not assumeVisible validation or application error Check the relevant application logs and request records The Test Case is flakyPage remains in one state without progress Inspect the background operation, response timing, and state updates The browser simply needed more timeRedirect lands on an unexpected page Review routing, authentication, and recent configuration changes The original URL was wrongControl appears late, moves, or is obscured Review replay while available and inspect the UI transition The backend caused the failureNeeds Attention with a specific reason Follow the recommended action for the QA Account, Test File, Test Case, or Success Condition A product defect was confirmedLater Run passes under comparable conditions Compare the two Run records and look for a changing dependency or state The earlier failure no longer mattersThe goal is not to solve the incident inside the Run page. It is to enter engineering diagnosis with fewer unsupported branches. Deep dive: compare Runs only when their context is comparable “It passed for me” is weak evidence when the rerun didn't use the same conditions. Before comparing Outcomes, check the context attached to each Run.Comparison point Why it changes the interpretationTest Case and version An edited workflow or Success Condition may be testing something different.QA Account selection Account state, role, or authentication setup can change the path through the app.Test File metadata A replacement file is a new immutable input; its name, type, and size may differ.Deployment or environment metadata A rerun after a rollback, configuration change, or new release isn't the same condition.Time and application state Expiring sessions, queued work, rate limits, and data cleanup can make behavior time-dependent.AlwaysQA keeps each Run's historical record tied to the inputs and metadata available when it ran. Later changes don't rewrite that finished Outcome. Environment or deployment details can only help when they were captured, so don't invent missing context after the fact. Consider a passwordless SSO workflow. The original Run may show that SSO completed and the dashboard loaded, while the next protected page redirected back to sign-in. If that page depends on an internal API, browser Evidence can't distinguish an API error from rejected session state or a frontend routing decision. That is why application event records still matter; NIST's log-management guidance describes their value for operational analysis and reconstruction. A later pass with a fresh session doesn't prove why the earlier session failed. It tells the developer where to compare authentication events, application logs, and request records. This is where browser Evidence stops and application-side diagnosis starts. Without application-side records, the browser can identify the boundary but not explain an internal service decision. Review replay before it expires, but do not make it the system of record Browser replay is useful when sequence and appearance matter. It can show whether:a loading layer blocked the next action; the interface reset after submission; a redirect happened before the expected confirmation; a control appeared too late to be used; or the page stopped changing even though no explicit error appeared.This matches a broader browser-testing practice: Microsoft's Playwright Workspaces documentation uses reports and retained artifacts to investigate completed test Runs (Microsoft Learn: browser-test diagnostics). AlwaysQA's implementation is its own: structured Evidence remains with the Run, while the provider-hosted browser replay is temporary. Replay access is private and uses an authenticated, short-lived handoff. Review it promptly. If it has expired, the summary, observations, compact action timeline, and Run context remain available. There is operational friction here. Someone must review time-limited replay, preserve the relevant visual finding in the Issue Report, and keep the Test Case observable. Evidence reduces reproduction work; it doesn't remove maintenance, ownership, or application telemetry. Turn the disappearing failure into a useful Issue Report For a Failed Run, AlwaysQA can prepare an editable Markdown Issue Report. Its sections are predictable: What happened → Expected result → Steps to reproduce → Key observations → Evidence → Run context. The report includes up to five key observations, the first two and last three, plus the last recorded action. The authorized Run link remains the place to inspect the complete timeline and replay while it is available.Included in the handoff Kept out or deliberately limitedOutcome and evidence-backed summary Unsupported root-cause claimsExpected result and reproduction steps The full action timeline in copied MarkdownSelected observations and last action Test File contentsSanitized URL and relevant Run context URL query strings, fragments, and embedded user informationTest File name, type, and size when relevant Known QA Account credentialsCopying the report is explicit. AlwaysQA doesn't silently create an issue in an external tracker. The developer can add logs, a suspected component, impact, owner, and links to both the original Run and later rerun. Use Deployment QA History to keep the failed Run, later pass, Test Case version, and fix-and-rerun sequence connected over time. Treat visual replay as potentially sensitive AlwaysQA sanitizes exported Run information and excludes known credentials and Test File contents from the Issue Report. A replay, however, can display whatever the QA browser was allowed to see. Use dedicated QA Accounts, synthetic records, and non-sensitive Test Files. Don't put production credentials, payment-card details, confidential customer data, or regulated information into a browser workflow simply because the Run is automated. A practical workflow for a failure that disappearsDo not dismiss the original Run. Confirm it is Finished and read its Outcome.Preserve the fast-expiring context. If replay is available and visual behavior matters, review it now.Mark the failure boundary. Record the last expected observation, the first unexpected state, and the preceding action.Compare the rerun context. Check Test Case version, inputs, QA Account, environment, deployment metadata, and timing where available.Choose the right branch. Failed goes to product investigation; Needs Attention goes to the stated corrective action.Add application evidence. Inspect logs, traces, background work, authentication events, or configuration changes around the original Run time.File the edited Issue Report. Link the original failure and later pass, assign an owner, and keep the causal conclusion open until the evidence supports it.Verify the fix or monitor for recurrence. Run the same Test Case again and use later scheduled Runs to determine whether the behavior returns.See how AlwaysQA works for the complete Test Case, Run, Outcome, and Evidence flow. What changes for the teamWithout original-Run Evidence With AlwaysQAThe rerun passed, so the incident is closed. The pass and failure remain two separate, inspectable Outcomes.Someone tries to reproduce the entire workflow from memory. The failure boundary identifies where engineering should begin.“Flaky” becomes the default explanation. The team compares Run context before classifying the behavior.The replay expires and the investigation loses its record. Structured Evidence remains after visual replay is unavailable.The ticket says “sometimes checkout fails.” The Issue Report states expected behavior, observed behavior, selected Evidence, and Run context.The sales value is practical: fewer “cannot reproduce” loops, better developer handoffs, and less pressure to invent a cause just because the defect disappeared. Investigate the Run that failed, not only the rerun that passed Create a browser Test Case for a critical workflow and give it an observable Success Condition. When a scheduled Run fails, AlwaysQA preserves the original Evidence so your team can inspect what happened, compare later Runs, and start in the right part of the system. Start using AlwaysQA and keep the failed Run in front of the investigation. Frequently asked questions Is a Test Case flaky if it fails once and passes on rerun? Not necessarily. The two Outcomes show inconsistent observed behavior, but they don't identify the cause. Compare the Run context and failure boundary before deciding whether the Test Case, application, data, environment, or a dependency changed. What remains after browser replay expires? AlwaysQA retains the structured Evidence: summary, observations, compact action timeline, and Run context. Replay is temporary and should be reviewed promptly when visual context matters. Can AlwaysQA determine the root cause of an intermittent failure? AlwaysQA identifies what the browser observed and whether the Success Condition was met. A developer may still need application logs, traces, database records, or configuration history to establish the cause. What if the original Run finished as Needs Attention? Follow the stored Attention Reason and recommended action. Needs Attention means the Run couldn't support a pass-or-fail conclusion; it should not be filed as a confirmed product defect.