⏱️
TL;DR
AI can easily turn your marketing data into a report, but it can only use what you give it. It won't know the full picture, but it'll sound confident anyway.

We've seen this lead to three costly mistakes recently. From deciding ads need fixes when they don't, to feeding incomplete sales data back into Google Ads so it optimises for fewer enquiries.

Keep using AI, but before you make important decisions, ask a specialist whether you’re missing something.

More and more of the businesses I work with are using AI to analyse their own marketing - that's a good thing. I use it every day, and so does my team.

However, something started happening recently that worries me a little bit. A pattern is emerging where AI is being used for analysis, but lacks crucial context and expertise to produce a useful output.

Worse, though, is when the output forms part of the decision-making process and even feeds the wrong data back into advertising systems to learn from. This can damage performance, instead of improving it.

A confused person holding three puzzle pieces next to a smiling robot holding a finished-looking puzzle with a growth chart on it.

It goes something like this: a marketer or founder exports data from their CRM and drops it into their AI tool of choice. A conversation follows, and they ask it how their ads are doing. Soon after, they get a neat-looking report with tables, benchmarks, and a clear verdict. It looks exactly like a specialist wrote it.

The trouble is, a specialist didn't write it.

It's missing:

  • The account's history
  • How the tracking setup actually works
  • An understanding of various attribution models used by the data provider
  • Conversions that never get tracked at all, thanks to cookie consent, tracking blockers, or other issues
  • The ones that get tracked incorrectly or dropped into other channels
  • It assumes that other channels don't contribute to a journey
  • And also, the context you only get from doing this for decades.
AI is confident. Too confident, in fact. And it makes the user confident. If you're not a specialist on the subject, it's very hard to know when it's wrong.

That isn't just my experience. Researchers at Aalto University found that people using ChatGPT on a logical reasoning test scored about three points higher out of 20 than people without it, but still overestimated their own score by about four points. People who said they understood AI better tended to be more confident, but less accurate about their own results.

This isn't an anti-AI post - I love AI. But it's only as good as what you put in, and who's checking the output it provides.

In this article, I'll start by explaining how this differs from a post I wrote last year. Then I'll walk through three patterns we've seen recently, what each one got wrong and why it matters commercially.

Finally, I'll share the checks I'd run before acting on any AI analysis of your marketing.

This isn't the "lazy marketers" problem

Last year I wrote that AI is making marketers lazy, and it's killing creativity. That article was about marketers outsourcing their own thinking. If they skip the struggle and skip the learning, they end up with generic work.

This is a different problem, and in some ways it could become a much more expensive or damaging one.

The people I'm giving advice to here aren't being lazy at all. They're extremely hard-working business owners and in-house marketing teams who are curious, motivated and trying to get more from their spend. They want to lean on AI to do their job better.

The problem is that they're using AI to do specialist work they don't have the background to check or understand fully. The AI's output is extremely convincing, and because it's convincing, people act on it.

That's where the damage happens: a budget cut, spend in the wrong channel, a change to what Google Ads optimises towards, or a batch of backlinks that puts rankings at risk.

Why AI sounds more expert than it is

AI works with what you give it. If you give it an export and a prompt, it doesn't know what's missing from that export.

It doesn't know how the tracking was set up, whether your HubSpot account is incorrectly pushing conversions into "direct", what changed in the account two years ago, what was tested and abandoned, or what the strategy is trying to achieve.

It also doesn't know that there is another big chunk of conversions that the export it received doesn't include. It might not know what impact your ads are having on driving genuinely direct traffic to your site. It certainly doesn't know what impact the ads are having on how many people are Googling your business.

It won't tell you any of that unless you include it in your conversation. It'll take the data at face value and give you an answer that sounds very certain.

The problem is, if you don't know it either, the output will be wrong.

A confident robot following a map and pointing ahead as it walks off the edge of a cliff.

The certainty of your AI's output is really persuasive. A University of California, Irvine study found that people consistently overestimated how accurate AI chatbot answers to quiz questions were. Longer explanations made them even more confident and made it harder for them to spot the wrong answers.

A Microsoft Research and Carnegie Mellon survey of 319 knowledge workers found that the more confident people were in generative AI, the less critical thinking they reported doing.

A specialist would first import a full data set and explain the nuances. But then they would look at the output and ask completely different questions about what's missing from the AI's output.

That process separates a useful analysis from a confidently wrong one. If you don't know how to ask, the AI usually won't check for you. And even if it does, it still doesn't understand the context of your actual situation.

Even experts get caught out by this. In a study of 758 BCG consultants by researchers from Harvard, Wharton, MIT, Warwick and BCG, one task gave the consultants a business problem with a spreadsheet that looked complete, but the real answer was hidden in the interview notes, and not the spreadsheet. Of those working without AI, 84.5% got it right, compared with 60% and 70.6% in the two groups using AI.

> Bar chart: on a business case task, 84.5% of consultants without AI got the right answer, compared with 60% and 70.6% in the two groups using AI.

The gap between those who used AI and those who didn't isn't even the scary part; the scary part is that the AI-assisted answers were still rated more convincing.

These were highly qualified consultants doing the work they do every day. So this goes beyond who's using the tool. Without the context behind the data, you can't check what it tells you.

What your AI doesn't see

Before I get into the patterns I'm seeing, it helps to know what's missing from the data most people hand to AI.

Two of the three patterns we've seen recently start in the same place. In one, a business exports its Analytics and CRM data, asks AI whether Google Ads is working, and gets told it needs improvement. In the other, a business builds its own dashboard to match ad enquiries to sales, then feeds that data back into Google Ads to optimise towards.

Both feel like the sensible thing to do. But in both cases, the data has gaps that AI can't see and won't point out unless an expert tells it. These are the ones that come up again and again.

Issue 1: One click, one conversion, one channel

The analysis assumes a customer arrives from one channel and converts there and then. Real customer journeys don't work like that. Someone might click an ad, leave, then come back through organic search a week later. Or they might become aware of you through organic search, then come back later from an Instagram post. If only one channel can get the credit, every other channel that played a part disappears.

Issue 2: Attribution gets ignored

Different tools credit different touches. In Pattern 1 below, the CRM report credited the last touch before someone enquired. But in our experience, paid search is often the first click - the one that introduced the business to the customer in the first place. Google Ads defaults to its own attribution model too (data-driven for most conversion actions). So the same customer can tell completely different stories depending on which data you use in your export, and an AI analysis that doesn't recognise that can undervalue whichever channel starts the journey.

Issue 3: The untracked bit

Not every journey can be tracked. People decline or ignore cookie banners. Some use ad blockers or browsers with stricter privacy settings. Links opened in apps, or some email clients, can lose their source along the way, especially without UTM tags. Those visits either land in "direct", get estimated by modelling, or never appear at all. An analysis that only counts what it can see will come up short - often by a lot.

Issue 4: Conversion lag

Related to issue 3, ad platforms only credit a conversion within a set window after the click. In Google Ads, that's 30 days by default (you can stretch it to 90). On Meta, it's 7 days. And even that assumes the cookie survives on the device; some browsers, like Safari, cap tracking cookies at 7 days, and people clear cookies or switch between devices all the time. If the time between the first ad click and the enquiry or conversion is longer than that - which it often is for bigger purchases or service enquiries - the tracking shows a broken journey.

Issue 5: Conversion rates

This is a big one. If your advertising is driving people directly to the website at a later stage in the journey, the conversion rate of that traffic is higher because they're closer to converting. This skews the analysis even more, because the AI compares Ads traffic against channels measured at a different point in the journey. The same applies to the opposite case - if you're driving people earlier in the funnel and another channel is getting the last click, conversion rates per channel will differ.

A customer journey of ad click, search, phone call and email, cut by scissors before it reaches the report a robot is reading.

Pattern 1: "Ads isn't working"

An agency who I consult with regularly recently ran a situation through me.

Their client pulled data from Analytics and their CRM, and pushed it through AI. They asked whether Google Ads is performing.

The analysis came back looking thorough, with a slick dashboard. It suggested that there's a lot to do to improve it.

The problem is, the export was wrong before AI even got involved.

Diagram of one customer journey, from an ad click to an organic search a week later to an enquiry. The CRM credits organic, Google Ads splits the credit between paid search and organic, and the AI report, seeing only the CRM data, credits organic.

It counted one channel per conversion, used the CRM's last-touch attribution, and couldn't see untracked journeys or those lost to conversion lag.

What to do instead

Put the gaps above together, and "Ads performance needs to improve" becomes an easy conclusion to reach. The problem is, it might not even be true.

Instead, treat any analysis like this as a simplified view of a single tool's tracked data, not the full picture, and ask:

  • Which attribution model sits behind every number
  • Where else you could find data to pull into the analysis
  • Look at what the channel itself (in this case, Google Ads) recorded alongside the CRM, and if they disagree, find out why before deciding which one is "right"
  • And if your own tracking has been changed, broken or rebuilt during the period you're looking at, factor that in before drawing any conclusion - you can't compare something that isn't comparable

Pattern 2: the dashboard that could make Google Ads worse

This one worries me the most, because it seems like the right thing to do, but it can damage ads performance.

A business builds its own attribution dashboard with the idea of matching enquiries from ads to the jobs or sales they turned into.

On the face of it, that's great, right? Actual revenue data from inside the business is exactly what can be fed to ad platforms to optimise towards the perfect visitor. When it's complete, it can be worth its weight in gold.

The problem comes when the data isn’t complete, and it rarely is.

Some enquiries can be matched to a sale, but most can't. There are many reasons why:

  • A customer might call from different numbers
  • Someone books through the office, and nobody records where they came from
  • They call from a withheld number
  • A customer converts weeks later through another route
  • The person clicking an ad passes it to someone in their team to get in touch
  • They have a broken journey that can't be tracked (as per the points above)
  • + many other reasons

Now imagine that partial data gets fed back into Google Ads as the main thing it optimises towards. This is what can happen:

  1. Ads is told that the matched enquiries are the valuable ones.
  2. Everything it can't see a sale for looks like a failure.
  3. Smart Bidding does exactly what it's designed to do and works hard to get more enquiries like the matched ones.
  4. It can start pulling back from the many other enquiries it generated that did convert, but were never matched in the client's own data.
Diagram of a feedback loop. Only the few enquiries matched to a sale are fed back into Google Ads Smart Bidding, which then chases more enquiries like those and learns to ignore the unmatched majority, even though many of them converted.

This results in fewer conversions overall, not more or better ones.

Nobody did anything reckless. The dashboard was built with good intentions, but without understanding how attribution works, the gaps in tracking, and how bidding algorithms learn from the data you give them, it's very easy to end up teaching Google Ads to chase fewer enquiries. We've seen this first-hand many times in the past year or two.

What to do instead

  • Fix how enquiries are recorded first, so the data going back is as complete as you can make it
  • Compare total enquiries and total sales before and after any change, not just the matched ones
  • Test value-based data on a small part of the account before rolling it out
  • Get whoever runs your ads to sign off any change to what the platform is optimising towards

The third pattern we've seen is in SEO.

A business owner, understandably impatient with how long SEO takes to have an impact, and how far ahead competitors might appear, starts using AI to build links themselves.

It feels productive and helpful. It only takes a few simple emails, some AI-created content and a quick reciprocal link swap - it feels too easy... because it is.

These links tend to have a few things in common:

  • They're reciprocal. "You link to me, I'll link to you"
  • The content around them is written by AI
  • The sites hosting them are often AI-generated at scale themselves
  • They're judged on a third-party score that can easily be manipulated, like Domain Authority, rather than on the many signals we know that search engines actually use. Google's John Mueller has said Google doesn't use DA 'at all'
Two cheerful robots shake hands while swapping chain links, beside a machine churning out near-identical web pages, each carrying a link.

None of that looks dangerous initially. The trouble is what the AI doesn't know. It has no idea what the current link strategy is, what the anchor text profile looks like, what the existing backlink profile is, what competitors' profiles look like, the way link equity is passed around internally, or the context and intent behind the pages that need to rank.

A handful of genuine reciprocal links between businesses that actually know each other is normal. At volume, it could become a problem. Google's spam policies specifically list "excessive link exchanges ('Link to me and I'll link to you') or partner pages exclusively for the sake of cross-linking" as spam.

The same policies cover scaled content abuse, which includes using generative AI to churn out lots of pages without adding value for users.

The best-case scenario here is that those links simply get ignored. Google's systems are designed to neutralise unnatural links, so any value they seemed to pass is lost and the time spent building them is wasted. The worst-case scenario is that a pattern like this can lead to a manual action. Either way, it can undo ranking gains that took months of careful work to build.

I've written more about how scaled AI content tends to play out over time in the long-term SEO risks of AI-generated content.

What to do instead

  • Genuine relationships and collaborations are great and can add value. Before arranging reciprocal links, though, I'd recommend running the idea past whoever looks after your SEO, because it could be that it won't do anything for you, or could even hurt you
  • Judge websites on what they're ranking for, who their audience is and whether they're relevant to the page you're linking to, instead of looking at a third-party domain strength score like DA
  • There are many other technical signals that could contribute to the value of a backlink, so your SEO partner should advise further
  • When budgets are tight, and the quick and easy option looks tempting, hold firm and stick to the plan - consistency is important, and that's exactly the moment when a second opinion is valuable

The common thread

In all three patterns, the business had some real data and a capable AI tool. They put in the effort and came away with a confident answer.

What none of them had was real context and expertise. They didn't know how the data was collected, what was missing, what happened in the account before, or what the strategy was meant to achieve. Because it's not their day job, they didn't know how to fill in the gaps and give as full a picture as possible, or how to challenge the AI's conclusions in a way that would produce a better answer.

That context is the bit a specialist brings. Without it, AI just works with what it's been given and sounds so sure about itself.

How to use AI on your marketing data safely

A thoughtful person examines a chart through a magnifying glass, spotting a question mark in the data, while a smiling robot beside the chart gives a confident thumbs up.

None of this means you should stop using AI to understand your own marketing efforts. Constantly striving for improvement is a good habit. But getting a good output from AI requires some thought. I'd recommend trying the following:

  1. Tell it what it doesn't know. Plan what you want to see as an output before starting the conversation, and ask it to list the assumptions it makes. Ask it would need to outline all of the information, and it would need to be accurate before feeding it any data.
  2. Ask it what information you could give it (and not give it) to make the answer incomplete. Tell it to challenge itself at every opportunity.
  3. Check the attribution of the data you provide. Ask which attribution model the tool you got the data from uses. If two tools disagree on the data, that's normal. In fact, it's rare that they'd match. Find out why they don't match before you pick a side and understand the gaps.
  4. Account for untracked enquiries and conversions. Cookies get declined often. They also get ignored. Ad blockers, phone calls, offline sales, and people who come back later on all impact the analysis output.
  5. One conversion or enquiry does not equal one channel. Investigate customer touchpoints. Look at Analytics to look at all of the possible journeys to understand that a conversion doesn't come from a single channel.
  6. Provide context and history. Give it all of your annotations. If you don't store them, your agency probably does. Include changes to tracking, new page launches, campaign changes, marketing activity, and seasonal information. Without a clear timeline of changes and activity, AI will read every decline or gap as a performance issue.
  7. Don't feed it straight back into an ad algorithm. Ad platforms can take your conversion data and use it to build out smart bidding systems to attract more of the right visitors. Get a specialist to check first, and treat all changes as small and reversible tests because if the data’s wrong or incomplete, you'll probably be optimising towards fewer conversions.
  8. Share it as questions, not evidence. Send the analysis to your agency or specialist with "what am I missing?" You'll get a better answer and a much better conversation.
  9. Bring in a specialist. If you don't have one, get one. They'll help you get the best from your analysis and performance. Speak to them before you cut or move your budget based on an AI report.

Conclusion

AI is changing the way business owners and marketing teams want to understand their own marketing. It's quick, it's easy, and it can help you ask much better questions of your agency and your data.

It can't tell you what it can't see, though. It doesn't know your account history, your tracking quirks or the strategy behind the work, and it'll give you a confident answer either way.

That matches independent research, too. An MIT analysis of 106 experiments found that people and AI usually did worse together than the better of the two working alone. The combination did best when the person already outperformed the machine.

In short, the businesses that get the most from AI over the next few years will be the ones that pair their own data with specialist context.

If you've run your marketing data through AI and you're not sure what to make of the answer, I'm always happy to take a look.