Why you need to test your tracking

TL;DR: In most companies, tracking is handled like documentation. It’s specified in a ticket or a wiki, implemented next to the actual feature, and mostly assumed to work. That’s why it breaks often and gets noticed late. Software engineering solved the same problems with rigorous testing, and tracking can work the same way. A tracking contract defines what correct looks like, a browser test checks the dataLayer against it in CI before deployment, and the same contract monitors the data in the warehouse.

In most companies I’ve worked with, tracking is handled like documentation. It gets specified in a Jira ticket or a wiki page. A developer implements it next to the actual feature, and after that it’s mostly assumed to work. I think this is why tracking breaks as often as it does, and why it usually takes so long until someone notices. I also think most of these problems go away when we treat tracking like everything else in software development.

Here is an example I have seen in one form or another at every company. Marketing runs a campaign. The clicks arrive on the landing page with the campaign parameters in the URL, and the visitor fills out a form two pages later. Somewhere on the way, a redirect or the routing of the single-page app has dropped the parameters, so the conversion arrives in Google Analytics without a campaign attached. The campaign did work, but the data says it didn’t. The marketing manager usually has no way to prove otherwise.

The second example is an A/B test. The tracking itself is fine, but the experiment runs in a different tool. Since the teams move fast, the tracking for the test variant didn’t go through the same rigorous QA process. The events of the variant fire on desktop but have missing tracking on mobile. Since nobody notices, the test gets evaluated on numbers where the mobile traffic is missing, and the wrong winner gets picked and shipped.

A very rigorous team might catch either of those two examples. But for a highly iterative product team and an agile marketing team that launches new campaigns frequently, manual testing does not scale. Especially for smaller tests and campaigns, tracking issues are usually only found after weeks of collecting data, or they never surface at all. It also happens that the initial implementation was never completely correct, so the baseline that people compare against is already wrong. An audit or a monitoring alert doesn’t change this, because both only tell you about the issue after the data is already missing. And that data is oftentimes gone for good. If you are lucky, it can be repaired in the transformation layer, which costs time. Otherwise it’s lost for the whole period.

Why do we treat tracking like documentation?

Tracking passes through many hands before it produces a single number. The requirements usually come from a product manager. A web analyst defines how the tracking should be generated, a developer implements it, and QA or the web analyst checks it before it gets deployed. After that it still depends on a tag management system, which sends the data on to the marketing tags, the analytics tool and the warehouse the data team builds on.

Usually someone owns the tracking, often the web analyst. But end-to-end ownership is rarely defined, and nobody controls the whole pipeline. The web analyst doesn’t control the software implementation and isn’t involved in every software or analytics change. Most companies also build far more software than they have web analysts to review it. For the developer, tracking is an outside requirement next to the feature they are actually building. It costs time and is often outside their expertise. Marketing managers, BI and data analysts depend on the data, but following every tracking change is not their job.

The spec is supposed to hold all of this together, but it doesn’t. It describes the tracking at the beginning. After that, the software and the tracking change through one change request after another, while the spec stays in a Jira ticket or a wiki page.

So the spec drifts in two directions: away from what the software actually sends, and away from what analytics and marketing expect. After a while nobody can say which version is the correct one. Even a dedicated web analyst has no tool to check every change against the spec, because a text document can’t be tested.

Software engineering has already solved this problem

Developers face the same problems in their software development lifecycle. Many people change the code, requirements move, and a spec in a ticket goes out of date. Software engineering solved this with rigorous testing. In test-driven development, you first write a test that describes what the code should do. Then you implement the code until the test passes. The test runs before every deployment, so a change that breaks the expected behaviour is caught before it reaches users.

If we treat tracking like code, it can go through the same loop. We define what the tracking should send, test the implementation against it, and only deploy when the test passes. Then we know the tracking works before it goes live. We just have to define what we test against.

Defining the tracking contract

To test the tracking, we need a definition of correct that a machine can read. This is the tracking contract, a data contract for the events the site sends. The developer or the web analyst doesn’t invent it alone. It comes from the business requirements and from the data the organization has already defined. Ideally it is part of the semantic layer, or a product of it, so it uses the same event names, business definitions and naming conventions that the data team and the dashboards downstream already rely on.

{
  "event": "purchase",
  "fires": { "after_click": "[data-testid='checkout-confirm']", "times": 1 },
  "schema": {
    "type": "object",
    "required": ["transaction_id", "value", "currency", "items"],
    "properties": {
      "transaction_id": { "type": "string", "pattern": "^ORD-\\d{4}-\\d{6}$" },
      "value": { "type": "number", "minimum": 0 },
      "currency": { "enum": ["EUR", "CHF"] },
      "items": { "type": "array", "minItems": 1 }
    }
  }
}

Testing the tracking contract

import { test, expect } from '@playwright/test';
import Ajv from 'ajv';
import contract from '../contracts/purchase.json' with { type: 'json' };

const validate = new Ajv().compile(contract.schema);
const purchases = (page) =>
  page.evaluate(() => window.dataLayer.filter((e) => e.event === 'purchase'));

test('purchase matches the tracking contract', async ({ page }) => {
  await page.goto('/checkout');
  await page.getByTestId('checkout-confirm').click();

  await expect.poll(async () => (await purchases(page)).length).toBe(1);
  const [purchase] = await purchases(page);
  expect(validate(purchase.ecommerce), JSON.stringify(validate.errors)).toBe(true);
});

The test runs in CI, for example in GitHub Actions, on every change before deployment. A developer who changes the checkout gets a failing build when the change breaks the tracking. The issue gets fixed before it is live, and nobody has to find it weeks later in the data.

Monitoring with the tracking contract

select
  event_date,
  ecommerce.transaction_id,
  count(*) as times_sent,
  logical_or(not regexp_contains(coalesce(ecommerce.transaction_id, ''), r'^ORD-\d{4}-\d{6}$')) as breaks_pattern
from `project.analytics_123456.events_*`
where event_name = 'purchase'
group by event_date, ecommerce.transaction_id
having times_sent > 1 or breaks_pattern
order by event_date desc

What testing and data contracts don’t fix

Testing improves the data quality, but it doesn’t fix the problems that are inherent to tracking. Even perfect tracking can’t measure a visitor who blocks it with an ad blocker or who denies consent. The same goes for a customer journey without user IDs, for example when a visitor starts on the phone and buys on their laptop.