Automation has a maintenance cost whether or not it provides useful release evidence. A suite can be large, run frequently, and still leave the team unsure whether a change is safe. The warning sign is not simply that tests sometimes fail. It is that engineers cannot tell what a failure means, or what a passing run actually protects.
Before replacing the framework, examine the economics of attention: which checks reveal meaningful product risk, which consume repeated investigation, and which duplicate evidence available elsewhere? The answer usually requires looking at failure history and product boundaries together.
Recognize the failure patterns
Look for behaviours in the delivery workflow rather than judging the suite by its test count. Engineers may rerun a job before reading the failure. Reviewers may merge while a familiar check is red. A small UI change may require updates across unrelated scenarios. A test might fail only when run after another test or alongside a parallel worker.
These symptoms have different causes. An intermittent failure can reveal a race in the application, shared test data, a dependency outage, or a synchronization error in the test itself. Calling every intermittent failure “flaky” before investigation risks dismissing a real product defect.
Start with a bounded sample of failed runs. Record the first failure, retry outcome, affected behaviour, cause when known, diagnosis effort, and owner. Keep unresolved cases separate. A classification system that forces every failure into “product” or “test” can create false certainty.
Ask what each expensive check proves
For the checks that demand the most attention, write one sentence describing the product risk they protect. Then ask:
- Would this check fail if the important behaviour regressed?
- Is the assertion about the business result or an incidental implementation detail?
- Could a lower-level check detect the same defect more directly?
- Does the scenario need a real external dependency, or is that boundary tested elsewhere?
- Is there an owner who understands why the test exists?
A browser test may be appropriate for a critical journey across the real application. It may be a poor place to enumerate dozens of validation combinations already covered at an API or component boundary. Conversely, moving every check below the UI can leave routing, rendering, browser behaviour, and integrated journeys unexamined.
The choice is not “UI tests are bad” or “more API tests are better.” Choose the boundary that gives useful evidence with manageable setup, execution, and diagnosis. Keep a small set of purposeful end-to-end journeys where the integration itself is the risk.
Example: a brittle account-settings scenario
Consider a hypothetical test that creates a customer through several screens, opens settings, changes a notification preference, and checks a success banner. It fails frequently because shared accounts are changed by other tests. It also passes when the banner appears even if the preference is not persisted.
There are at least three separate concerns: data isolation, UI feedback, and persistence. Repairing only the selector may leave the important weaknesses intact. A more useful approach could create an isolated account through a supported setup interface, change the setting through the UI, then verify the saved state through a reload or an appropriate independent observation.
Validation combinations could be covered closer to the API, while the browser scenario retains responsibility for the user journey. The decision depends on the application’s interfaces and testability; do not bypass the very behaviour the test is meant to examine.
Playwright’s best-practice guidance recommends isolated tests, user-facing locators, and web-first assertions. Those techniques can reduce avoidable coupling, but they do not decide which product risks deserve coverage. That remains an engineering decision.
Treat retries as evidence, not a repair
Retries can keep an isolated transient event from stopping all delivery, and they can provide diagnostic information. They also make an unreliable check appear healthy if reporting exposes only the final result.
Preserve first-run and retry outcomes separately. Playwright’s retry model distinguishes a first-run pass, a failure that passes on retry, and a failure that remains failed. Use those distinctions to identify recurring uncertainty. Do not claim a root cause from the retry outcome alone.
Avoid blanket increases to retry counts or timeouts. A longer timeout can conceal a performance regression; a retry can repeat a non-idempotent operation. Confirm that retries do not create duplicate transactions, corrupt shared state, or mask a recovery defect.
Where quarantine is necessary, make it a managed exception. Record the risk that loses automated protection, the person responsible, an investigation date, and the evidence used in the meantime. Keep the quarantined check visible in reports rather than making it disappear from the denominator.
Choose repair, relocation, or retirement deliberately
Repair a check when it protects a meaningful risk and the problem is identifiable: unstable data, weak synchronization, inadequate assertions, or missing diagnostics. Move it when another boundary can provide clearer evidence, while retaining any distinct integration coverage. Retire it when the behaviour no longer exists or the check duplicates evidence without adding useful protection.
For retirement, record what remains covered and where. Removing an expensive test is an engineering change, not a cosmetic way to improve pass rates. Review the decision with the product or service owner when it changes the release evidence for an important behaviour.
Avoid migrating the entire suite as the first intervention. Demonstrate the intended pattern on one representative path, run it in CI, and observe both successful and failed behaviour. A new tool can reproduce the same data coupling and unclear assertions if the design assumptions remain unchanged.
Evaluate whether confidence actually improved
Compare the same types of changes over a defined observation window. Examine diagnosis effort, repeated failure categories, time waiting for useful feedback, and evidence for release-critical paths. Include escaped defects and manual workarounds in the review. A cleaner pipeline obtained by disabling useful checks is not an improvement in release confidence.
Use the Release Readiness Scorecard to keep automation in the wider context of ownership, observability, and recovery. If the suite is costly but its value remains difficult to explain, a Quality Engineering Assessment can separate the framework problems from the product-risk and delivery-system problems.