WTSers were chatting about Nozzle’s Poisoned Google SERPs research by Derek Perkins since it dropped.

Long-time supporters of WTSFest, when we reached out to Derek and the Nozzle team to share that our community members had questions - they jumped at the chance to answer them through this AMA.

Community members asked a whole range of questions from how to check their data and what to ask tracking providers, to how to explain unexpected changes to clients and leadership.

They also explored the implications for AI visibility and the future of rank tracking.

Understanding poisoned search results

Big picture, what’s your sense of Google’s motivations for making these changes?

100% this is anti-LLM.

Rank trackers have been around for decades, presenting at the same conferences Google sponsors. They could have made our lives much harder beforehand but never have.

While they may never admit to it publicly, in general I think rank trackers are a net-positive for Google.

We don’t make a dent in their costs while incentivising companies to do things in the way that Google wants.

Did your analysis reveal patterns in types of queries affected? (brand vs non, long-tail vs high-volume keywords, etc) Could those patterns help us focus on what needs the most scrutiny?

I initially did some research into search volume and it didn’t seem to correlate, but it’s worth revisiting.

Similar story for branded keywords. Let me take a look and report back.

There’s a big difference between industries from the research article, so it seems reasonable:

Source: Nozzle

Investigating changes in rankings, AI visibility & analytics

How can I check whether results are poisoned for my list of keywords?

The best way to verify is to do logged-in manual checks at this point. I haven’t seen any reports or evidence of poisoning logged-in results.

Data providers will need to keep track of expected results and flag anything deviating too far as poisoned.

The exact variance will take some tweaking over time.

We saw impressions dip by double digits on 17 September. Could some of those impressions have come from scrapers and rank trackers previously? Are there enough of them that, if Google blocked them, it could cause a drop that large?

I think GSC was unprepared for the changes and is having major issues itself.

There are a lot of scrapers out there, so it’s certainly possible that there was a real dip due to that. I just have a hard time trusting GSC or most rank-tracking sources right now.

Did the poisoning stop at the organic listings, or did it show up in AI Overviews and AI Mode too? Does our AI visibility reporting need the same second look?

At this point I have done a lot of checking into AIOs and haven’t found any credible evidence of poisoning there. It does seem generally restricted to organic.

For the specific type of poisoning where Google just ignores words from the query, that does seem like the poisoning happens early in the search process and impacts the inputs into their internal systems that generate videos, PAA, and related searches.I don’t believe those teams are independently poisoning, though.

Related resource shared: Nozzle’s Google Plays Dumb research.

Lots of keyword tools offer AIO reporting separately from traditional SERPs, but also flag it as a SERP feature in a traditional keyword report. What’s the value in tracking it both as a SERP feature and as an LLM for an AI prompt?

It is weird how it straddles both LLM and SERP tracking that way.

As an LLM, it answers questions just like any assistant, so knowing how it talks about you is important.

Since the AIO also changes the CTR on the SERP, understanding how it fits into the overall layout is necessary to understand how it is impacting your organic traffic.

The difference usually is more about the inputs. LLM tracking has longer, more conversational queries, versus traditional keyword tracking.

Derek shared this slide from Patrick Stox’s 2025 presentation when discussing the reach of AI Overviews.

Source: Patrick Stox, Ahrefs Evolve 2025

If Google is poisoning results to block LLM scrapers, could poisoning change which sources LLMs end up citing? If we’re planning off-site placements based on which domains AI tools cite today, should we expect that list to shift?

Yes, I think it’s going to cause a shift. I don’t know what that’ll do long term.

We know that OpenAI is building its own index, and I would be shocked if Anthropic isn’t doing similar. SerpApi has its own, and there is a new wave of other tools like Exa that are doing the same.

We’ll see where we end up.

Google is clearly trying to protect its competitive search advantage to make Gemini more effective than Claude/ChatGPT.

What are some practical steps we can take with clients to ID the sources and reduce instances of “not set” in our data? Landing pages are usually where I’m trying to troubleshoot it.

[DEREK]: I’ll defer to Dana DiTomaso here.

[DANA]: “(Not set)” is usually due to session timeouts if it’s in the source, medium, and landing page.

If someone times out with a tab open and then comes back to it again and an event fires, like a scroll, without a corresponding session_start—which happens when the page is refreshed, for example—you end up with “not set” values.

More info: GA4 unassigned traffic and confusing channel names.

[FOLLOW-UP]: Is that the case with Organic if the landing page is “not set”, too?

[DANA]: Yes, exactly.

Tabs that are left open and then revisited after the session has timed out are a big reason why you’d see “(not set)” in your landing pages or session sources and mediums.

This isn’t necessarily an issue unless you need to rely on tracking specific page views, such as page views of a thank-you page in order to record key events—then you can end up in the situation where you’re recording the same key event occurrence multiple times.

I’ve created an Exploration that can help me see what pages users are likely timing out on:

1. Create a blank free-form Exploration.
2. Add the dimensions of landing page, page path, and event name.
3. For the metrics, add total users.
4. Set your row to page path and your value to total users.
5. Add filters: event name does not exactly match session_start, and landing page exactly matches “(not set)”.

Scan through this list and see if you have pages where you’re recording a key event when it’s viewed. If you have a lot of users on those pages, then you know where you might possibly be overcounting key events as a result.

Are there ways we can track user agents in GA4? Also, can you talk about the pros and cons of default versus custom channel groupings for AI traffic?

[DEREK]: Again, Dana DiTomaso is much more qualified to answer this than me.

Derek shared Dana’s guide to AI traffic analytics and audience analysis.

[DANA]: You can definitely track user agents in GA4, but you’ll need to set it up custom.

GA4 limits dimension values to 100 characters, so you may want to split it up if you want the entire string.

I talk about this a little bit in my post on removing bot traffic in GA4.

Choosing data sources & understanding how tools are adapting

How can we rebuild trust in rank trackers and tools? What other verification should we do, and is it time to focus on GSC and webmaster tools data alone?

I don’t think GSC is exempt. Google is poisoning its own data.

I also don’t believe that the vast number of teams inside Google are coordinating enough to make sure that poisoned data isn’t being fed into GSC. Poisoned SERPs are what is actually being delivered, and thus it would take significant work to make sure that was filtered out.

Derek shared Joy Hawkins’s LinkedIn post about unusual Search Console activity.

Do you know of any studies comparing tracking tools—for example, which seems more reliable for position tracking?

I haven’t seen anything comparing tools.

For us at Nozzle, we’ve always stored the raw HTML and made it visible in-app so that it’s easy for anyone to spot-check the SERP. Other more enterprise-focused tools may have something similar.

In general, the larger the scale, the more difficult dealing with this change is going to be.

What is the most stable way of tracking performance right now?

Right now, your site analytics are the most stable reporting you have. If that isn’t changing, it’s likely that your SERPs haven’t been upended.

I would have a hard time trusting either rank trackers or GSC alone.

If you do see major shifts in analytics, I would still look at rank trackers and GSC with a healthy dose of spot checks.

Is it still possible to use SERP tracking for competitor benchmarking and general SEO visibility? Do poisoned SERPs influence everyone to the same degree?

I do think it’s possible, but I don’t feel confident that any tool has truly gotten to a place of more than 90% unpoisoned SERPs yet.

For most tools, it’s likely that they are less than 50% unpoisoned.

How are you currently preventing data pollution caused by the massive influx of bot traffic?

Nozzle’s checks on collected SERP data strategy is basically:

• Maintaining a rolling history of what we expect to see for any given keyword.
• Comparing the list of organic domains to the expected organic domains.
• If more than 30–50% have disappeared (still figuring out the right threshold) we treat it as poisoned.
• Re-crawl the same keyword up to five times, using different proxy networks each time.

Brand-new keywords we haven’t seen are tougher to verify since we don’t have a solid baseline.

How will Google serving URLs with ‘goto’ parameters impact tracking, and what is Nozzle doing to access reliable data? How should SEOs approach measurement when platforms make it more difficult?

Goto so far hasn’t been a significant issue in terms of tracking for most tools. At most it just requires some waiting to get the destination URLs.

If Google continues to fight back and only allows, say, 10% of links to be “clicked”, we’ll have to see what other workarounds are available. Having a URL index where we can match titles, paragraphs or chunks to a domain might come into play.

The biggest key going forward is to make sure you understand how your data platforms are ensuring data integrity. As Google drives that cost up, there are going to be lots of temptations to cut corners.

Derek shared this Resoneo post and the accompanying graphic.

Source: Resoneo.

How do we keep track of changes in LLM reporting? Every time I see a huge increase or dip, I need to research whether it was something we did or a change from the LLM. One example is the inability to track Copilot from 3 November.

It’s tough, for sure.

For Copilot specifically, I don’t have inside information on Semrush choosing to discontinue it, but I believe that it has more to do with customer interest than technical challenges.

Even if Microsoft moved parts behind a login, that’s no different from Claude. If people cared and were willing to pay, it would be solved.

[FOLLOW-UP]: Even Profound has discontinued collection for Copilot due to the changes, so it is definitely larger than Semrush. Profound’s collection notice.

Thanks for linking that. I still stand behind my original hypothesis that there isn’t enough customer interest to make it worthwhile to get around their changes.

We used to track Bing at Nozzle, but very few people were willing to pay for it and we discontinued it.

Decisions from data & explaining to stakeholders

How reliable is the data for making decisions around content planning, regular and LLM visibility, and SERP analysis?

Put big annotations around 13 September to today in any of your reports—rank tracking or GSC.

I don’t have any evidence that AIOs are poisoned, so LLM tracking should be generally unaffected.

There could very well be some knock-on effects, where other LLMs get poisoned results when fanning out and thus get worse results, but that’s also what real users are likely to see.

Plenty of “who’s winning in Google” studies have come out since mid-September. How worried should we be that some trends are poisoning artefacts? What should we ask before trusting a visibility study covering that period?

Every single one of those studies is based on poisoned data.

We source SERP HTML from every major proxy network and provider. I can promise that nobody is immune.

I think some studies, like ones showing growth in AIO coverage, are representing real numbers, but I wouldn’t trust anything reporting winners and losers in general.

What advice do you have for explaining the changes to leadership, particularly when giving context in reporting?

Definitely lean into annotations. Everything since 13 September is suspect.

• If you’re a large brand, any dips are very likely caused by poisoned data, as Google is more likely to remove major domains.
• If you’re a smaller brand, be similarly wary of big jumps, as that might just be your site jumping up to replace those larger brands.

How have reporting conversations with leadership changed? Given AIOs and SERP poisoning, are the primary metrics changing?

I don’t think primary metrics have changed yet, but I do think conversations are being had.

We have one major customer who told me at Tech SEO Connect that this break in reporting continuity got senior management off being overly focused on old ranking metrics and dashboards.

The future of tracking

I’m seeing ranking jumps that can’t be replicated on live SERPs. Is this likely to settle, or is it the new normal? Should we still do rank tracking, and what place does it have in a five-year SEO strategy?

I don’t think this is an existential threat to rank tracking, as much as it has felt like it.

While this is the most concentrated effort Google has put out to block us ever, there’s only so far that it can go without impacting real users.

There are always workarounds. The real question for me is what cost and scale look like.

At the most extreme end, we end up being forced to only use logged-in accounts. That’s annoying to manage and introduces some additional personalisation questions, though there is a query parameter to turn that off.

Google is still going to drive the lion’s share of traffic for the foreseeable future. SERP features and in-SERP AI changes are going to continue to change, so the value proposition of tracking is as high as it has ever been.

Today, we’re collecting up to five scrapes of a single keyword until we get an unpoisoned result, which is definitely improving things. As we continue the cat-and-mouse game with Google, trying to act more like regular humans when browsing, I’m hopeful that we land in a place that doesn’t require 10 times the cost to deliver the same data.

Assuming this is one escalation of the “LLM wars”, with more to come, what are your predictions for the future of SEO tracking and reporting?

I think it’s going to cause a lot of consolidation.

It used to be super easy to collect SERPs, so there were always tons of new rank trackers popping up.

Over the last two years, as Google has increasingly made things more difficult and expensive, tools have already shut down, and this is by far the biggest change that has happened.

I think in general that people need to be more aware about the provenance of their data.

It’s been too easy to just see a metric from a tool and trust it, so this is probably a good forcing mechanism for the industry.