The rerun passed, but the original failure still matters

Authenticated workflow testing must prove that the right user can sign in and finish the protected task after a deployment. AlwaysQA tests password and six-digit email-code sign-in through a fresh browser session, giving the team visibility into the complete workflow without giving the decrypted password, QA Inbox, Email Code, or raw authentication message to a Coding Agent or browser agent. Most teams make the mistake of treating login as a checkpoint. The expensive failures happen across the whole chain: form, credential submission, identity-provider redirect, session, permissions, and the product action the customer came to complete.Reality check: A green login page proves that the page loaded. It doesn't prove that an authenticated customer can use the product.The use case: verify the job behind the login Consider a SaaS team that has just shipped a small permissions change. Public pages respond normally. The sign-in form loads. Infrastructure monitoring is green. But standard users now return to the login page after entering a valid password. Administrators are unaffected, so the problem survives an internal spot check. Customers discover it first. A useful Test Case doesn't stop at “login succeeded.” It verifies a visible product result:Starting URL https://app.example.com/login Instructions Sign in as the configured standard user. Create a test project named with today's date. Confirm that it appears in the project list, then delete it. Success Condition The project appears after creation and no longer appears after deletion.That Run covers the sign-in form, credential acceptance, redirect, authenticated session, role permissions, record creation, persistence, and cleanup. It answers a business question, not a transport-level one. For broader coverage across signup, onboarding, permissions, billing, and other release risks, use the web app regression testing checklist. This page focuses specifically on authentication and the workflows gated by it. Why URL checks and login-only tests miss real failures Authentication is a dependency chain. A failure at any layer can block the customer while shallow monitoring still reports success.Layer What a shallow check sees What the customer experiencesLogin page The URL returned 200 OK The submit action does nothingPassword submission The form accepted input A valid credential is rejectedEmail delivery The email service is available The current six-digit code never arrivesCode verification The form accepts six characters A fresh code is rejected or expiresHosted authentication The identity provider responded The redirect returns to the wrong originSession Authentication completed The next page asks the customer to sign in againAuthorization The dashboard opened The account has the wrong role or permissionsProduct workflow Login passed The customer still can't complete the taskCustomers don't buy access to a login screen. They buy what the authenticated product lets them do. Use dedicated QA Accounts, not real customer credentials AlwaysQA uses a QA Account: a named, Project-owned authentication resource for a dedicated identity in your application. Multiple Test Cases in the same Project can reuse it. Create separate QA Accounts for the roles you need to verify, such as standard user, administrator, or billing manager. Give each one only the permissions and test data required for its workflows. That reduces operational risk and makes a failure easier to interpret. It also follows the least-privilege principle described in NIST SP 800-53 Revision 5. Each QA Account has one explicit, immutable Authentication Method:Authentication Method What you configure When to use itPassword A dedicated identifier and password The application signs in with username or email plus passwordEmail Code A dedicated identity using an AlwaysQA-controlled QA Inbox The application sends an exact six-digit code by emailAlwaysQA doesn't infer the method from the page. If the product supports both methods, create two QA Accounts and assign each Test Case to the appropriate one. The QA Account feature overview explains how this setup supports role-specific and permission-specific workflows. The trust boundary: the agent navigates, trusted code authenticates The browser agent needs to understand the interface. It doesn't need the authentication secret. During a Run, the agent navigates to the matching sign-in form and requests one QA Account Sign-in action. Trusted code completes the configured Authentication Method and returns only a sanitized result. After successful sign-in, the browser agent continues with the Test Case instructions. The boundary is easier to see as a handoff: Browser agent Trusted sign-in action Application | | | | navigate to login | | |-------------------------->| | | | submit protected material | | |-------------------------->| | | sanitized result | |<---------------------------| | | continue product workflow | |------------------------------------------------------->|What stays outside agent context:Authentication material Handled by Given to Coding Agent or browser agent? Retained as Run Evidence?Decrypted password Trusted sign-in action No NoQA Inbox address Authenticated user and trusted code No NoRaw authentication message Trusted code No NoSix-digit Email Code Trusted code No NoSanitized sign-in result Trusted code and Run Yes Only sanitized observationsIt's the product's core security distinction: the agent can evaluate the authenticated experience without receiving the material used to enter it. It's still a dedicated identity, and it still needs operational discipline. Use synthetic data, minimum permissions, a cleanup strategy, and an owner who can rotate or remove the identity when the application changes. How password authentication works For a Password QA Account, configure the identifier, password, Login Page URL, and approved Login Origin directly through AlwaysQA. Every Run starts with fresh browser state. The browser agent reaches the visible sign-in interface, then trusted code submits the configured authentication material. The agent receives a sanitized result and continues only after the sign-in action completes. The password is write-only from the perspective of product read interfaces. It isn't returned to your Coding Agent, placed in its prompt, or exposed to the browser agent. That's different from instructing an AI agent to retrieve a secret from an environment variable and type it into a page. The agent controls the workflow; AlwaysQA controls the credential boundary. How six-digit email-code authentication works Passwordless testing is harder because “use the latest email” isn't a reliable rule. An inbox may contain a code from an earlier attempt, delayed mail, or multiple messages that look valid. Each Email Code QA Account receives one stable, opaque QA Inbox. Its address contains no User, Project, or QA Account name. The authenticated user copies that address when provisioning the dedicated identity in the application; the Coding Agent and browser agent never receive it. For each unattended Run, AlwaysQA:opens a fresh browser session; navigates to the matching sign-in form; opens one short-lived authentication attempt; requests one authentication message; accepts only a fresh message delivered after that request; extracts one exact six-digit Email Code through trusted code; submits that code on the approved Login Origin; and continues with the authenticated Test Case.The Run doesn't reuse an old code, request an automatic resend, or try several candidates after rejection. Runs using the same Email Code QA Account execute one at a time. Serialization prevents overlapping messages and product activity from being attributed to the wrong Run. Other Test Cases remain independent. Each Email Code QA Account receives an AlwaysQA-controlled QA Inbox. Your company mailbox isn't part of the workflow. When trusted processing ends, the provider copy of the raw authentication message is deleted. The payload isn't stored in the AlwaysQA database, AI context, logs, or Run Evidence; only sanitized diagnostics such as delivery timing and failure classification remain. Deep dive: approve the exact destination before submitting a secret Keeping credentials out of an AI prompt solves only half the problem. Trusted code must also know where it is allowed to submit them. An application may start sign-in here: https://app.example.com/loginand collect the password or Email Code here: https://auth.example-provider.comThey're different origins. The Login Page URL tells AlwaysQA where to begin navigating; it isn't permission to submit authentication material. During setup, AlwaysQA follows the visible flow and detects the exact public HTTPS Login Origin that would receive the identifier or credential. The authenticated user must review and approve that origin before the QA Account becomes Ready. If a later Run reaches a different origin, AlwaysQA stops before submitting authentication material. The QA Account moves to Needs Attention, dependent Schedules pause, and the team must approve the new origin before those Runs can continue. This protects against an unexpected redirect receiving stored authentication material. It doesn't replace an application-security review. Teams should separately assess authenticator lifecycle, session management, recovery, and abuse controls against guidance such as NIST SP 800-63B. Where AlwaysQA fits alongside Playwright or Cypress Keep your existing lower-level automated coverage and scripted browser tests. They're the right choice for fast deterministic assertions in a development pipeline. AlwaysQA covers a different operating need: post-deploy QA that a Coding Agent can create, run, inspect, and use during fix-and-rerun work without receiving the QA Account's authentication material. The AlwaysQA workflow overview shows how Test Cases, Runs, Outcomes, and Evidence connect.Keep a scripted browser test when… Use AlwaysQA when…The team needs deterministic assertions on implementation details The team needs to verify an observable customer outcomeThe check belongs on every commit The check should run after a change, on demand, Daily, or WeeklyThe team owns and maintains the test code and selectors The workflow is defined through instructions and a Success ConditionExisting CI infrastructure already protects its secrets The Coding Agent should operate QA without retrieving the credentialA machine-readable failure is enough The team needs observations, a timeline, replay context, and an Issue ReportThe strongest setup isn't either/or. Scripted tests catch known implementation failures quickly. AlwaysQA verifies that the complete critical flow still works after the change reaches the application. Separate a broken product from an inconclusive Run Every finished Run receives one of three Outcomes:Passed: the Success Condition was observed. Failed: valid Evidence showed that the expected application behavior was broken. Needs Attention: AlwaysQA couldn't reach a trustworthy conclusion.The third Outcome matters in authenticated testing. A missing email, delayed delivery, expired Email Code, bot protection, or unclear result shouldn't be reported as proof that the product failed. Some problems affect only the current Run. If CAPTCHA appears, for example, the Run ends as Needs Attention without asking AI or trusted code to bypass it, while the QA Account remains Ready. Other problems require account-level action. An explicitly rejected password or a changed Login Origin can move the QA Account to Needs Attention, make it unavailable for new Runs, and pause dependent Schedules until the user resolves it. AlwaysQA preserves structured Evidence: a summary, browser observations, a compact action timeline, the final Outcome, guidance when action is required, and a temporary browser replay when available. A Failed Run can also produce an editable Markdown Issue Report without including QA Account secrets, Email Codes, or raw authentication messages. Use deployment QA history to connect that Evidence to the affected check version, fix the problem, and rerun the same Test Case. Current authentication coverage The safe product promise is a precise one.Supported now Not supported nowUsername or email plus password Email sign-in linksExact six-digit codes delivered by email SMS codesHosted authentication pages with an approved Login Origin TOTP authenticator codesFresh sign-in during every Run Reusing stored authenticated browser stateObservable post-login browser workflows SSO or OAuth popupsOn-demand, Daily, or Weekly Runs PasskeysHuman approval during a RunCAPTCHA bypassThese boundaries are useful qualification criteria. If a workflow needs a magic link, SMS, passkey, OAuth popup, or human approval, use a different testing approach for that path today. Put one authenticated workflow under post-deploy QA Start with the protected task whose unnoticed failure would cost the team most. Login by itself is usually too narrow. Before scheduling the Test Case:create a dedicated identity with minimum permissions; select Password or Email Code explicitly; approve the exact HTTPS Login Origin; use synthetic data that the Run can safely change; choose a workflow with a visible, business-relevant result; define cleanup for records the Run creates; make one person responsible for Failed and Needs Attention Outcomes; and run the Test Case once on demand to review its Evidence.Then add a Daily or Weekly Schedule when the workflow and test data are stable. For deployment-specific verification, run the Test Case again after the relevant change ships. Create your AlwaysQA account and put the authenticated workflow your customers depend on under post-deploy QA. Frequently asked questions Does AlwaysQA expose passwords or Email Codes to AI agents? No. Trusted code handles the configured password, raw authentication message, and six-digit Email Code. The Coding Agent and browser agent receive only the non-secret workflow context and sanitized sign-in result they need. Does AlwaysQA need access to an existing company mailbox? No. Each Email Code QA Account uses an AlwaysQA-controlled QA Inbox instead of connecting to an existing company mailbox. Can one QA Account test password and email-code authentication? No. Each QA Account has one immutable Authentication Method. Create separate Password and Email Code QA Accounts and assign each Test Case explicitly. What happens when the hosted authentication origin changes? AlwaysQA stops before submitting authentication material, moves the QA Account to Needs Attention, and pauses dependent Schedules. The user must review and approve the new Login Origin before Runs can continue.

Post-deployment smoke testing lets you monitor the high-risk customer flow affected by a release after the code reaches its target environment. AlwaysQA runs the saved browser Test Case on demand and returns Passed, Failed, or Needs Attention with Evidence your team can inspect. A green deployment pipeline proves that the release completed its configured steps. It doesn't prove that a user can sign in, complete checkout, submit a form, or reach the confirmation page.Reality check: Most teams don't have a deployment problem when regressions escape. They have a verification gap. The hard part is maintenance and release latency: too many smoke checks slow the decision, while too few leave the highest-risk behavior unverified.The use case at a glancePost-release smoke check with AlwaysQATrigger An important change reaches production, staging, or another target environmentTest selection The smallest set of Test Cases connected to the changed customer behaviorExecution An on-demand Run started in AlwaysQA or through a connected coding agent after confirmationResult Passed, Failed, or Needs AttentionEvidence Summary, browser observations, action timeline, stopped page, and temporary replay when availableCloseout Fix the regression and run the same Test Case againThe goal isn't to rerun every check after every deployment. It's to get a fast product-level answer about the behavior most likely to have changed. A practical example: the release is green but billing is unreachable Suppose a team changes an authentication callback used after passwordless login. The build passes. The deployment completes. The application starts, and its health endpoint responds. From the infrastructure side, the release looks healthy. The customer flow has more steps:Open the sign-in page. Request and submit the login code. Establish the authenticated session. Open account billing. See the current Plan and the option to manage it.The callback change preserves login but drops the redirect target. Users reach the dashboard, yet the billing route sends them back to sign-in. An AlwaysQA Test Case defines the complete observable behavior:Open the sign-in page, sign in with the configured QA Account, and navigate to billing. The billing page should display the current Plan and an option to manage it.After deployment, the team starts that Test Case on demand. A fresh browser agent attempts the workflow against the released application. If the agent reaches the billing page and observes the Success Condition, the Run passes. If valid Evidence shows the redirect loop or access failure, it fails. If an account or execution problem prevents a trustworthy conclusion, the Run receives Needs Attention instead of pretending that the product regression is confirmed. Here's the release loop: Deploy → run the affected behavior → inspect the Outcome → fix → run again. A successful deployment is not a behavior test Deployment automation answers delivery questions:Was the artifact built? Did the configured checks complete? Did the release reach the target environment? Did the service start and respond to its health probe?A browser smoke test answers a customer question: can someone still complete this important workflow? Those checks overlap, but they aren't interchangeable. A health endpoint can respond while a permission rule blocks the dashboard. A deployment can finish while form validation rejects every valid submission. A shared navigation component can load while sending users to the wrong page. Deployment monitoring, application monitoring, and customer-flow monitoring answer different questions. Deployment monitoring confirms rollout state; application monitoring exposes service health; customer-flow monitoring verifies what the user can complete. A smoke test belongs in the third layer, while still using the first two as release context. Microsoft's safe deployment guidance recommends health checks, issue detection, progressive exposure, and multiple testing methods rather than relying on one gate. Product-level smoke tests supply one missing layer: observable user behavior after the release is live. Match smoke tests to the release Here's the wrong smoke-test policy: “run everything.” That creates latency, spends Run allowance, and turns release verification into a slow regression cycle. Start from the changed behavior and its likely blast radius:Release area Immediate smoke-test candidates WhyAuthentication callback Passwordless login, dashboard access, account security Session and redirect behavior can affect every authenticated routeCheckout Product selection, checkout submission, confirmation A break blocks revenue and may leave ambiguous order stateOnboarding Registration, first setup step, initial dashboard New users can be acquired but fail before activationShared form component Lead form, support form, profile update One validation change may affect several workflowsPermissions Standard-user access, administrator action, restricted page The interface may load while the wrong role is allowed or blockedShared navigation Sign-in redirect, primary dashboard, billing route An unrelated-looking component can break routes across the productUse the regression testing checklist for web apps when the release touches a shared component and the impact isn't obvious. Ask three questions before choosing the Test CasesWhat customer outcome changed directly? Test that first. Which shared dependency could widen the impact? Add one or two adjacent flows when authentication, navigation, permissions, or shared validation changed. What failure would make the release unsafe to leave live? That's the flow for the immediate smoke set.That's what keeps the check set small without making it arbitrary. Save customer behavior as a reusable Test Case Define the workflow once as an AlwaysQA Test Case. Include:the starting URL; practical Instructions for the browser agent; an observable Success Condition; a dedicated QA Account or Test File when needed.Avoid implementation language in the Success Condition. “The callback handler returns the expected state” may be meaningful to a developer but invisible to a customer. Prefer the outcome:A signed-in customer can open billing and see the current Plan with an option to manage it.The Test Case becomes a reusable production smoke test. You don't have to rewrite the workflow after every release, and the saved definition gives later Runs a consistent behavior to evaluate. See how AlwaysQA Test Cases and Runs work for the full browser-evaluation sequence. Start the Run after the target environment is ready When the release reaches its target environment, start the relevant Test Cases with Run now. AlwaysQA creates a fresh Run from the saved Test Case definition. Its execution State moves through Queued, Running, and Finished. The final Outcome remains separate from that progress. If you're using a connected coding agent, you can request a confirmed Run there. This keeps the smoke test close to the repository and release conversation while AlwaysQA performs the actual browser evaluation. A practical release checklist looks like this:Deploy the change. Confirm the target environment is reachable. Start the smallest relevant Test Case set. Wait for every Run to finish. Review Failed and Needs Attention Outcomes. Fix or roll back when the customer impact requires it. Run the affected Test Case again after the repair.AlwaysQA doesn't decide which release is important enough to test. Your team owns that policy and the confirmation to start each Run. Treat Passed, Failed, and Needs Attention differently Every finished Run receives one Outcome. Passed AlwaysQA observed the Test Case's Success Condition during that Run. Passed is evidence that the saved workflow worked at that moment. It isn't proof that every application feature survived the release. Failed Valid Evidence showed that expected application behavior was broken. The browser may have reached an error message, returned to sign-in, stopped before confirmation, or displayed a state that contradicts the Success Condition. Needs Attention AlwaysQA couldn't reach a trustworthy conclusion. An unavailable QA Account, ambiguous Test Case, unusable Test File, or execution problem shouldn't be presented as a confirmed product regression. Needs Attention keeps uncertainty visible without converting it into a false failure. That's why this distinction matters during a release. Failed informs a product decision. Needs Attention tells the team that its verification is incomplete. Investigate the original Run before trying to reproduce it When a smoke test fails, the first useful artifact is the original post-release Run. AlwaysQA preserves structured Evidence that can include:a concise summary; meaningful browser observations; a compact action timeline; the page where the Run stopped; a temporary browser replay, when available.For a Failed Run, the team can prepare an editable Markdown Issue Report with the expected behavior, key Evidence, and a link back to the Run. You don't want the common waste pattern: a developer immediately reruns the workflow manually, gets a different result, and loses the exact state observed after deployment. Start with the Evidence. Reproduce only when the investigation needs more information. The AlwaysQA feature overview shows how browser Runs preserve Outcomes and Evidence. Fix, rerun, and keep both records After correcting the regression, start another Run of the same Test Case. The new Run evaluates the current saved definition. The earlier failure and its Evidence remain in history, so a passing rerun doesn't erase the record of what went wrong. That's important when expected behavior changes between releases. Test Case versioning keeps older results connected to the definition that produced them, while the active Test Case can evolve with the product. The deployment QA history turns the sequence into a durable record: deployment context, check version, failure Evidence, diagnosis, and fix-and-rerun status. The fix isn't verified because the code looks right. It's verified when a fresh Run observes the customer outcome again. Smoke tests, scheduled checks, and broader regression suites do different jobs One test strategy shouldn't be forced to answer every release question.Coverage type Trigger Best questionPost-deployment smoke test Immediately after an important release Did the changed critical flow survive this deployment?Daily or Weekly Schedule Recurring calendar cadence Does this workflow still work as the product and its dependencies change?Broader regression suite CI, release candidate, or planned QA cycle Did the change break a wider set of known behaviors?Progressive rollout During staged exposure Does the release remain healthy as more users receive it?Google Cloud's canary deployment guidance explains how progressive rollouts reduce exposure by testing a new version with part of the user base before a full release. An AlwaysQA smoke test complements that rollout strategy by checking a specific customer behavior in the target environment. Use on-demand Runs for release-specific verification and Daily or Weekly Schedules for ongoing protection. Neither removes the need for unit, integration, end-to-end, or manual testing. Build a release policy the team can follow A smoke-test process fails when “important release” means something different to every developer. Define the trigger before the next incident. Require an immediate smoke check when a release changes:authentication, sessions, or redirects; checkout, billing, or subscription management; onboarding or account creation; permissions or role-specific behavior; a shared form, navigation, or layout component; a third-party integration used inside a customer-critical flow.Then name the owner. Someone must choose the Test Cases, review every non-passing Outcome, and decide whether the release can stay live. Without that owner, “Run a smoke test after release” is advice, not an operating process. Give the next important release a customer-level check Start with one workflow that would make the release unsafe if it broke.Save that behavior as a Test Case. Run it once to confirm the definition and account are ready. Add the Test Case to the release checklist for relevant changes. Start an on-demand Run after deployment. Review the Outcome and original Evidence. If it failed, fix the application and run the same Test Case again.Do not stop at “deployment succeeded.” Check the behavior users depend on. Know whether the customer flow survived the release. Create your first release smoke test. Frequently asked questions What is post-deployment smoke testing? Post-deployment smoke testing runs a small set of high-value checks after a release reaches its target environment. The goal is to confirm that the customer behaviors most likely to be affected still work. How is a smoke test different from a full regression test? A smoke test is intentionally narrow and fast. It checks whether the release is usable at a basic product level. A broader regression suite covers more behaviors and usually takes longer to run and review. Which releases need an AlwaysQA smoke test? Prioritize releases affecting authentication, checkout, onboarding, permissions, shared UI components, forms, or integrations inside important customer flows. The right set depends on the change and its likely blast radius. Can I start a smoke test from my coding agent? Yes. A connected coding agent can request the relevant Test Case Run after explicit confirmation. AlwaysQA performs the browser evaluation and preserves the Run, Outcome, and Evidence. What should I do when a smoke test Needs Attention? Review why the Run was inconclusive. Resolve the account, Test Case, file, or execution problem, then start a fresh Run before treating the release as verified. Should smoke tests also run on a schedule? They can. Use an on-demand Run immediately after a relevant release, then keep the highest-impact workflows on a Daily or Weekly Schedule for ongoing regression coverage. Does a Passed smoke test prove the release is bug-free? No. It shows that the selected Success Condition was observed during that Run. Untested workflows and edge cases remain outside the result.

To diagnose browser test failures, we recommend this process: start with what the Run can prove, not with a guess about root cause. AlwaysQA gives your Coding Agent a clear Outcome and structured Evidence, then provides private browser replay when the text alone can't explain the observed behavior. Most teams make the mistake of treating a red result or a replay as the diagnosis. Neither is enough. Failed tells you the Success Condition wasn't met; replay shows what the browser displayed. The useful question is narrower: what did the Run observe, what happened immediately before the workflow diverged, and what should the developer inspect next? That distinction matters after a deployment, when every blind reproduction attempt delays the fix. AlwaysQA turns one completed Run into a shared investigation record that the Coding Agent, developer, and product owner can use without retelling the failure from memory. The use case: investigate a failed browser workflow without starting from zero Imagine a critical Test Case that uploads a CSV file, maps its columns, starts an import, and waits for a completion message. Its Success Condition is explicit: the import must finish and the newly imported records must appear in the application. After a deployment, the Run finishes with a Failed Outcome. The final screen is still showing “Processing,” and the expected records aren't visible. A generic alert leaves the team with unanswered questions:Did the file upload succeed? Did the browser click the intended control, or did another element intercept the interaction? Did the page navigate or stall? Was there a visible validation error? Did the workflow demonstrably fail? Or did the Run lack enough evidence to confirm success?AlwaysQA preserves the investigation trail as structured Evidence. The developer can start with the summary, inspect the observations and compact action timeline, and open the temporary replay only if the text doesn't explain the behavior. The outcome isn't an automatic root-cause claim. It's a tighter, evidence-backed starting point that cuts out blind reproduction work. When this use case fitsSituation Why the Evidence mattersA critical flow fails after deployment The team needs to separate a confirmed regression from an unresolved Run.A failure is difficult to reproduce locally Observations preserve the last confirmed state and the action immediately before it.A visual interaction looks suspicious Temporary replay can reveal an overlay, redirect, reset, or timing problem.A Coding Agent must hand work to a developer The Issue Report carries expected behavior, observed behavior, reproduction steps, and Run context.The first decision: wait, investigate, or correct the Test Case? Reliable diagnosis starts with precise language. A Run State describes where execution is in its lifecycle. An Outcome describes what a finished Run established.Field Possible values What it answersRun State Queued, Running, Finished Is the Run still in progress?Outcome Passed, Failed, Needs Attention What did the finished Run establish?Only a Finished Run has an Outcome. Use it to choose the next move:What you see What it establishes Next moveQueued or Running The Run hasn't reached a conclusion. Wait for it to finish; don't diagnose partial Evidence.Finished + Passed The Evidence confirms the Success Condition. Record the pass and move on.Finished + Failed The Evidence confirms the expected behavior didn't occur. Inspect the failure boundary and prepare an Issue Report.Finished + Needs Attention AlwaysQA couldn't responsibly confirm pass or failure. Follow the Attention Reason instead of assigning a product defect.Needs Attention may point the team to a QA Account, unclear Test Case, unusable Test File, or Success Condition that can't be observed reliably. That's useful information. It prevents an uncertain automation result from becoming a misleading bug ticket. You can see how this fits into the broader verification workflow in How AlwaysQA works. Use an evidence ladder, not a replay-first habit AlwaysQA organizes a Run so the fastest evidence is available first. That creates a practical investigation ladder:Outcome and summary: read the conclusion first. Observations: identify the meaningful browser states, including the last one that matched expectations. Action timeline: connect the divergence to the preceding interaction. Browser replay: use the visual record only when timing, layout, or interaction details could change the diagnosis.This order keeps diagnosis efficient. A concise final observation may reveal a validation message immediately. There's no reason to scrub through an entire replay when the relevant evidence already says, for example, that the import remained in “Processing” and the expected completion state never appeared. The principle is simple: preserve the evidence needed for diagnosis before asking someone to form a theory. Microsoft's Playwright Workspaces guidance uses the same broad pattern for browser-test diagnostics: retain reports and test artifacts, then inspect them after the Run (Microsoft Learn: advanced browser-test diagnostics). AlwaysQA applies that principle through its own Outcome, Evidence, replay, and Issue Report model. What a useful Evidence record looks like Here is an illustrative record for the CSV import scenario. It shows the level of specificity a developer needs without pretending to identify an unsupported root cause.Test Case: Import customers from a valid CSV file Success Condition: Import completes and the imported records appear in the customer list Outcome: Failed Summary: The file was accepted and the import started, but the completion state wasn't observed. Final observation: The import page continued to display “Processing”; no completion message or imported records were visible. Last action: Waited for the import result after selecting “Start import.” Final URL: https://qa.example.test/imports Replay: Available temporarilyThis example proves that the expected end state wasn't reached during the Run. It doesn't prove whether the cause was a background job, an application response, a frontend state update, or something else. The developer still owns causal diagnosis, but begins with a far narrower search area.Reality check: Browser automation can show what happened in the browser. It can't, by itself, prove why an internal service behaved that way. Treat Evidence as the bridge to engineering diagnosis, not as a substitute for application logs, traces, or code inspection.Read each Evidence layer for a different answer Summary: what did the Run establish? The summary should let a developer understand the observed result without watching the whole session. It connects the Outcome to the Success Condition and stays within what the Run can support. Good summary:Checkout reached the payment confirmation step, but no order confirmation appeared and the order wasn't visible in order history.Weak summary:Checkout is broken because the payment service timed out.The second statement may sound more useful, but it invents a cause unless the browser Evidence directly supports it. AlwaysQA doesn't label the summary as an “AI diagnosis.” That restraint makes the record safer to use in triage. Observations: what meaningful states appeared? Observations preserve the significant states encountered during the Run: a successful sign-in, an accepted upload, an error message, a missing confirmation, or a page that stopped changing. Read them in sequence. The last successful observation identifies how far the workflow got; the first unexpected observation marks where the investigation should begin. If the final screen looks wrong but all earlier milestones are intact, the likely search area is much smaller than the complete user flow. Action timeline: what happened immediately beforehand? The compact action timeline connects an observation to browser activity: navigation, typing, selecting a file, clicking, or waiting for a visible result. It helps answer whether the failure followed a specific interaction or whether the page never reached a usable state. This is especially useful for intermittent failures. Comparing the final meaningful action and observation across Runs can expose a repeatable boundary, even when the eventual cause lives outside the browser. Replay: what did the interaction look like? Open the browser replay when the visual sequence matters. It can help answer questions such as:Did a loading overlay block the next control? Did a menu open, accept the click, and then close before the selection registered? Did the page redirect before the confirmation appeared? Was the expected element below the fold or obscured? Did a multi-step form reset? If so, at which transition?Replay is most useful for animation, timing, layout, and interaction ambiguity. It's less efficient when the summary already names a clear validation message. Browser replay is temporary; structured Evidence is the durable record AlwaysQA retains the structured Evidence associated with the Run. Browser replay is a provider-hosted recording and is available only for the provider's retention period. Because that availability can change, don't treat replay as the permanent system of record. The practical rule is simple:Review replay promptly when a failed Run needs visual investigation. Capture the relevant finding in the Issue Report or engineering notes. Rely on the structured summary, observations, timeline, and Run context for durable history.Replay access is private. AlwaysQA exposes it through an authenticated, short-lived handoff rather than placing a permanent public recording URL in the Run record. If the replay has expired, the structured Evidence remains available. For a longer-term view across deployments and Test Case versions, use Deployment QA History. That page owns the historical comparison workflow; this use case focuses on investigating one Run well. Technical deep dive: where browser Evidence stops Structured Evidence and browser replay serve different consumers. The Coding Agent needs compact text it can retrieve through MCP and reason over. A User needs a private visual session only when the interaction itself is unclear. AlwaysQA therefore returns Evidence with the Run and handles replay through a separate, authenticated browser handoff. That separation matters when replay playback has latency or the recording has expired. The Coding Agent can still inspect the Outcome, observations, and timeline without waiting for video. Now consider an SSO-backed import. Browser Evidence may show that the SSO redirect completed, the file was accepted, and the page remained on “Processing.” That bounds the failure. It still can't tell you whether an internal API returned an error, a background API call never completed, or the frontend ignored a valid response. That next step needs application-side records; NIST's log-management guidance explains why retained events matter for operational analysis and reconstruction. This is the handoff point. Take the last confirmed browser state and compare it with application logs, service telemetry, and recent code or configuration changes. There is still maintenance work. Someone must review temporary replay before it expires, add the relevant visual finding to the ticket, and keep Test Cases and Success Conditions specific enough to produce useful Evidence. AlwaysQA reduces reproduction work; it doesn't remove ownership or engineering judgment. Failed is evidence of a missed Success Condition, not evidence of a particular root cause. Needs Attention is uncertainty made explicit, not a softer word for failure. Turn a Failed Run into a developer-ready Issue Report When a Run fails, AlwaysQA can generate an editable Markdown Issue Report. It's designed to move the relevant Evidence into the team's existing engineering workflow without forcing someone to reconstruct the browser session from memory. The report uses a predictable sequence: What happened → Expected result → Steps to reproduce → Key observations → Evidence → Run context. A developer can scan the difference between expected and observed behavior before opening the full Run. The report is intentionally compact. It includes up to five key observations, the first two and last three, and the last action. The authorized Run link remains the place to inspect the complete timeline and available replay.Included Deliberately limited or excludedOutcome and evidence-backed summary Unsupported root-cause claimsExpected result and reproduction steps The full action timeline in the pasted reportSelected key observations and last action Test File contentsSanitized URL and relevant Run context URL query strings, fragments, and embedded user informationTest File name, type, and size when relevant Known QA Account credentialsThe Issue Report is editable before it leaves AlwaysQA. Copying it is an explicit User action; AlwaysQA doesn't silently create an issue in an external tracker. A developer can add service logs, an owner, severity, or a suspected code path before filing it. If your team wants to convert verified defects into permanent coverage, follow the workflow in How to turn bug reports into regression tests. Handle replay and test data as sensitive operational material AlwaysQA sanitizes exported Run information and excludes known credentials and Test File contents from the Issue Report. That reduces accidental exposure, but it doesn't make every visual browser session inherently non-sensitive. A replay can show whatever the test browser was allowed to display. Your QA environment might contain names, records, account details, or uploaded values visible on screen. Use dedicated QA Accounts, synthetic data, and purpose-built Test Files. Don't use production credentials or confidential customer data just because the test runs in an automated browser. This is the right security promise: the product limits what it stores and exports, access to replay is private and temporary, and your team still controls what data the QA workflow can see. You can review the broader product capabilities on the AlwaysQA features page. A six-step browser failure investigation workflow Use this sequence when a Run doesn't pass:Confirm the Run is Finished. Partial Evidence isn't a conclusion.Branch on the Outcome. Investigate Failed. For Needs Attention, correct the stated blocker before involving a developer.Find the failure boundary. Compare the last successful observation with the first unexpected state. Stop reading once the relevant transition is clear.Check the preceding action. What changed immediately before the browser diverged?Use replay selectively. Open it for timing, layout, redirects, or interaction ambiguity. Review it before retention expires.Complete the handoff. Edit the Issue Report, add application-side evidence, and assign an owner. After the fix, rerun the same Test Case against the same Success Condition.For developers working through MCP, the Coding Agent can retrieve the Run details and Evidence without a separate manual QA handoff. When visual review is necessary, the User opens the authenticated replay in the browser. Text stays agent-readable; recording access stays private. What changes for the teamWithout structured Evidence With AlwaysQA“The test is red.” The Outcome states Passed, Failed, or Needs Attention.Someone reruns the whole workflow from memory. The last confirmed state and preceding action define where to start.A developer watches every recording. The team opens replay only when visual context can change the decision.An ambiguous result becomes a low-quality bug. Needs Attention routes the team to the unresolved input or condition.The ticket lacks reproducible context. The editable Issue Report carries expected behavior, observations, and Run context.That is the commercial value: less time reconstructing the failure, fewer unsupported defect claims, and a cleaner handoff from verification to engineering. AlwaysQA doesn't guess the root cause. It removes avoidable uncertainty before the root-cause investigation begins. Diagnose the next failed Run with evidence, not guesswork Connect AlwaysQA to your Coding Agent, run a critical Test Case, and inspect the Evidence returned with the Outcome. If a workflow fails, you'll have a structured record for triage and a temporary replay when the browser sequence needs closer inspection. Start using AlwaysQA and give every failed Run a clearer path to a decision. Frequently asked questions Does AlwaysQA automatically identify the root cause of a failure? No. AlwaysQA reports what the Run observed and whether the Success Condition was met. Its Evidence can narrow the investigation substantially, but a developer may still need application logs, traces, database records, or code inspection to establish the underlying cause. What is the difference between Failed and Needs Attention? Failed means the Evidence supports the conclusion that the Success Condition wasn't met. Needs Attention means the Run couldn't responsibly confirm pass or failure and includes a reason plus a recommended next action. How long is browser replay available? Replay availability follows the browser provider's retention period, so it should be treated as temporary. Review it promptly. AlwaysQA's structured Evidence remains with the Run after the recording is no longer available. Does AlwaysQA create an issue in our tracker automatically? No. A Failed Run can produce an editable Markdown Issue Report, but copying and filing it is an explicit action. Your team decides what to add and where to send it.