AI Website Testing: How We Test Websites With AI (and Why the Machine Goes Second)

The most dangerous bug we found this month showed a thank-you message.

A tester on our team was running a routine fortnightly check on a software company’s website. She clicked “Schedule a Demo”, filled in the form and pressed submit. The confirmation page loaded exactly as designed. Almost any automated check pointed at that page would have passed it: the page loaded, there were no console errors, the heading and layout were correct.

Then she checked the inbox. Nothing had arrived. Her note in the task says it in one line: the thank-you page appeared, but the email never came.

That is the kind of failure this article is about. Not the broken layout everyone notices on day one, but the quiet failure that looks like success. A form that thanks a buyer and then loses them. A sync job that stops running while every page still loads. A mail connection that expires on a Tuesday and takes the order confirmations with it.

We spent most of 2026 rebuilding how we test, largely around AI. This is what we do now, written from our own task boards rather than a vendor’s brochure, including the parts that did not work.

What actually changed

For most of the web’s history, testing a site meant testing a sample of it. You checked ten pages out of 150 and hoped the other 140 behaved. Sampling was not a choice. It was the budget.

AI removed the budget. A model can read every page of a site in an afternoon, giving page 140 the same attention as page 1. That sounds like “AI finds more bugs”, but the real change is different: when you read everything, you stop finding bugs and start finding causes.

An example from a rebuild we are finishing now. We ran a site-wide check of Open Graph metadata, the tags that decide how a page appears when it is shared. The report surfaced something sampling would never have found. A real insight article was tagged as a “website”. Corporate pages such as About Us were tagged as an “article”. It was not one stray page. The logic was inverted across several of them, which looked like values carried over page by page from the old WordPress install.

The useful output was not the list. It was the question it forced us to ask our developer: is metadata defined in one shared function, or separately inside every page file? One answer means a single edit. The other means a correction across roughly 200 pages.

A person sampling ten pages finds a symptom. Reading all of them produces a better question.

Eight passes on every page

Before AI touches a page, several people already have. Every page task on our board moves through fixed phases:

1. Design (designer)
2. Client approval
3. Development (developer)
4. Screenshot test (the developer checks their own build against the design)
5. Basic test (our QA lead: does everything work?)
6. Design test (the designer checks what was built against what was drawn)
7. Final test (me, plus an AI content test where it applies)
8. Complete test (our QA lead, one last full pass after fixes)

Look at where AI sits: step seven, near the end. That placement is deliberate, and it is the most important decision in this article.

Rule one: humans first, machine second

When we planned the test week for our largest rebuild, the method fit in one sentence: everyone manually tests five pages, then we turn the faults we found into a prompt and run that prompt across the entire site.

Each person walks their five pages twice. Once as a customer trying to get something done, slowly, on the pages that matter most. Once as a tester trying to break things. On a phone as well as a desktop, because that is where most visitors will be.

The reasoning is simple. AI is excellent at finding the twentieth instance of a known fault and weak at noticing the first. It does not know that a client dislikes boxed rows in the mobile menu, or that French labels run 20 to 30 percent longer than English ones and will break a button. A human notices once. The machine then checks everywhere.

So the human pass is not a warm-up. It writes the test.

Rule two: the prompt is mostly things not to report

Our page-level prompt opens with a role and some context: an experienced tester, working for a four-person agency, on a staging site that is about to launch. Then it does something most AI testing advice skips. It spends most of its length saying what to ignore.

A shortened version of the real one:

We are about to launch and we are in the testing phase. Do not suggest changes that would rebuild the page or take a large amount of work.

Report any case where the old page had good UX or SEO that the new page lacks.

This is a staging site. We know the preview build can be indexed. We know about the stock photography.

Ignore heading hierarchy and file names; those are checked elsewhere. Skip minor grammar, but do report serious grammatical errors.

URLs from the old site must be preserved, so flag any mismatch between a URL and its page title.

List every text element on the site with a contrast problem.

Why so many exclusions? Because a report with 400 items in it reports nothing. Our internal note for this prompt states its purpose plainly: deliver the important things and do not make too much noise. Every false positive costs a person a minute to dismiss, and after enough of them people stop reading the report. Noise is not harmless. It trains your team to ignore the tool.

If we could keep only one instruction, it would be this one: report what the old page did well that the new one does not. Redesigns rarely fail by adding bad things. They fail by quietly removing good ones.

Rule three: test at three altitudes

A single-page prompt misses a whole class of bug, so we run three kinds.

Page. The prompt above, one URL at a time.

Family. We export the full page list and ask the model to group pages that belong together, usually because they share a template: industry pages, location pages, service pages. Then we ask a different question. These pages are built from the same pattern, and some parts should be specific to each one. Is there forgotten text, or an element that was never adapted to this page? Do any of them contradict each other?

This is the test that catches the industry page still carrying a sentence about bakeries, or two location pages quoting different timelines for the same process. No single-page review can find it, because each page looks fine on its own.

Site. Checks that only make sense across everything: every link and button that leads to a 404 or nowhere, sections that need a heading and lack one, missing social share images, SEO titles that run too long, inconsistent trailing slashes in links, publish and update dates, pages that still need structured data, and every quote on the site pulled into one list so a person can check each one against its source.

When the platform changes, AI writes the tests instead of running them

Reading pages works for content. It does not work for a platform migration, where the biggest risks are invisible on screen: redirects, headers, status codes, and what a crawler sees before any JavaScript runs.

For our current migration, from WordPress to Next.js hosted on Vercel, we used AI differently. Rather than asking it to test, we used it to help write a launch test plan in which every item states two things: what is being tested, and how a script will test it. The plan runs to ten sections, and each item is tagged P0 (blocks launch), P1 (fix before launch) or P2 (first week after). A few of the items show why this matters:

• Photograph the old site first. Before anything is compared, a full URL inventory is built from four sources: the sitemap, Search Console’s page report, every landing page from twelve months of analytics, and every URL that has backlinks. The pages that are missing from the sitemap but still earn traffic or links are exactly the ones migrations lose.
• The preview noindex trap. Vercel adds an X-Robots-Tag: noindex header to preview deployments by default. On staging that is correct. If any trace of it reaches production, the site drops out of search. The test expects the header on staging and its absence on the live domain.
• Trailing slashes. WordPress URLs end in a slash and Next.js does not by default. Get that setting wrong and every old URL picks up an extra redirect hop.
• 308, not 301. Next.js uses a 308 status for permanent redirects. Google treats 301 and 308 the same way, so this is not a problem in itself, but the test records which one is used and checks that it is used consistently.
• Raw HTML against the rendered page. For each template, a script captures the page twice: as plain HTML with no JavaScript, and again after a browser has run everything. Anything that appears only in the second version is content some crawlers will never see. Vercel’s own research found that none of the major AI crawlers, including GPTBot and ClaudeBot, execute JavaScript. On the site we are migrating, AI assistants are already a traffic source, and one that converts better than organic search.
• Forms, end to end. Every form is filled with test data by a script, submitted, then checked for the success state, the analytics event, the notification email and the CRM record. A thank-you page on its own proves nothing, as the opening of this article shows.

Every finding gets a severity from S0 to S3, and the whole run ends in one of three verdicts: BLOCKED, CONDITIONAL or CLEAR. The launch rule was written before any results existed: zero S0 findings, zero S1 findings unless each one has a written exception, and everything else goes to the post-launch backlog. Writing the rule first is what stops a launch date from quietly lowering the bar.

After launch: the boring tests are the ones that pay

Testing usually peaks before launch and then stops. For our retainer clients, launch is when the recurring tests begin, and they are deliberately dull.

Each retainer site has a written checklist with a cadence: weekly, every two weeks, monthly, quarterly. The weekly check for a wholesale tile manufacturer with a dealer portal, for example, includes:

• Canary searches with expected results. A search for “grey” should return more than 100 products. A search for “aura” should return fewer than ten, ideally three. If either number moves sharply, indexing or filtering has broken, even though the search page itself loads perfectly.
• Filters with expected ranges. One collection should show between 1 and 10 products; a usage-area filter should show more than 1,000.
• Stock sync health. Four named products that should always show stock somewhere, plus the count of products the inventory integration has synced.
• A real order through a test dealer account, walked all the way through checkout and then deleted.
• The mail log, not the inbox. Failed sends in the SMTP log, plus a look at a handful of recent emails for anything suspicious.
• Error logs and the 404 log, scanned for anything obvious.

Every quarter the checklist goes deeper: database queries for duplicate SKUs and duplicate categories, and confirmation that a deactivated dealer really cannot see prices, place orders, check stock or open the price book.

Almost none of these would be flagged by a crawler, a speed test or an accessibility scanner. They test the business, not the page. The expected numbers are the clever part. They turn “does search work?” into a yes or no that a new team member can answer in thirty seconds, and that an AI agent working in a browser can run without making judgment calls.

These checks catch real problems. A weekly check on a UK recruitment site surfaced a warning that the job board’s scheduled sync was failing: applications could be pushed across one at a time, but not in bulk. Our developer traced it the same day, in about ninety minutes, to a Cloudflare OWASP rule blocking the request. On another client site, a mail connection expired and the site stopped sending email entirely, order confirmations included. That is why our checklist opens the log instead of trusting the inbox.

Where AI is weak, and what we do about it

It tends to agree with you. Anthropic’s research on sycophancy found that leading AI assistants consistently lean toward what the user appears to believe. In testing, a leading question gets a confirming answer. So we ask important questions twice, once framed positively and once negatively, and watch whether the answer moves. If it moves, a person decides.

It describes the image it expects, not the one that is there. We use AI to draft alt text at scale, but a person opens the actual image before any of it goes live. A fluent description of the wrong photograph is worse than no description at all.

It does not know your client. Preferences like “no boxed rows in the mobile menu” live in people’s heads until someone writes them down. We now keep written rule files for design and for development, and every recurring fault becomes a new rule. The next project starts where the last one finished.

Automated accessibility checks are a floor, not a verdict. The WebAIM Million 2026 report found detectable WCAG failures on 95.9 percent of the top one million home pages, and low-contrast text on 83.9 percent of them. Tools catch the frequent problems well; a 2021 Deque analysis of more than 13,000 audited pages found automated testing covered 57 percent of issues by volume. The rest, including keyboard order, focus behaviour and whether an accordion makes sense to a screen reader, still needs a person.

Not every tool earns its place. We spent time evaluating a commercial visual regression platform. Our developer’s verdict: capable, but driven almost entirely from the command line, with very few documented use cases for a site like ours. We parked it. A screenshot test by the developer and a design review by the designer still do that job for us.

It never gets the last word. AI output is a draft with an unknown error rate. Anything that will be published or sent to a client is checked by a person, and every finding in a report has a named human who decided it was real.

What we are still working out

We are not finished. Our test standards document has been through several drafts, because the early versions were too long to use in the middle of a real test; the standing instruction now is to keep it short. We are turning each client’s checklist into something an agent can run end to end, with a person reading the result rather than performing every click. And deciding which check belongs at which cadence is a judgment we keep revisiting.

We also do not claim what we have not measured. We have not run controlled experiments on these testing methods, and we cannot give you a percentage of bugs caught. What we can show is the log: the tasks, the checklists, the findings and the dates.

Seven questions to ask whoever tests your website

If you are hiring an agency, or reviewing the one you have, these questions separate a testing process from a line on a proposal:

1. Do people test before the automated tools, or instead of them?
2. When a form is tested, does anyone confirm that the email arrived and the lead reached your CRM, or only that the thank-you page appeared?
3. What does your AI testing prompt tell the model to ignore?
4. Do you compare related pages against each other, or only check each page on its own?
5. For a migration, was a baseline of the old site captured before anything changed?
6. What is the written rule for when a launch is blocked?
7. After launch, what gets tested every week, and which number tells you it has broken?

Specific answers mean you are in good hands. General answers mean you should ask to see the checklist.

Sources

• WebAIM, The WebAIM Million: 2026 report
• Deque, Automated testing study identifies 57 percent of digital accessibility issues (2021)
• Vercel, Are Vercel preview deployments indexed by search engines?
• Vercel, The rise of the AI crawler (December 2024)
• Next.js, redirects documentation
• Google Search Central, Redirects and Google Search
• Sharma et al., Anthropic, Towards Understanding Sycophancy in Language Models (2023)