How to run marketing experiments that produce decisions: hypotheses with mechanisms, stopping rules set in advance, and honest sample sizes.
Most marketing teams run tests. Far fewer run experiments that change what the business does. The difference is not statistical sophistication — it is whether the test was designed around a decision that was waiting to be made.
A test that produces a result nobody acts on has consumed traffic, time and attention to generate a fact. An experiment designed around a pending decision produces a commitment. The design work that separates them happens entirely before anything is launched.
Before designing anything, finish this sentence: depending on the outcome, we will do X or Y.
If it cannot be finished, the test should not run. This one constraint eliminates most of the waste in a typical testing backlog, because a great deal of what teams test is not connected to any pending choice. Testing two headline variants when the team has no intention of changing the messaging strategy either way produces a number and nothing else.
The decision framing also determines how much rigour is warranted, which is a question teams rarely ask. A reversible, low-cost choice — which of two subject lines to send — needs far less confidence than an irreversible one, such as rebuilding a checkout flow. Matching evidence to stakes prevents both failure modes: over-testing trivial decisions, and committing to expensive ones on a hunch.
A hypothesis worth testing states what will change, for whom, why, and what result would prove it false. "We think a shorter form will perform better" fails on every count. "Removing the company-size field will increase completed submissions among mobile visitors, because form analytics show most mobile abandonment occurring at that field; we will consider this disproved if completion does not improve" states a mechanism and a falsification condition.
The mechanism matters more than the prediction. When a test that had a stated mechanism wins, you have learned something transferable — mobile users abandon at fields requiring information they do not have to hand — which generates the next several hypotheses. When a test without a mechanism wins, you have learned that one specific variant beat another on one page, which transfers nowhere.
This is why organisations with long testing histories and no accumulated knowledge exist. They ran predictions rather than hypotheses.
Fix the sample size, the duration, and the primary metric in advance, and write them down.
The reason is that live experiments are unbearable to watch. Early results swing wildly. A variant leading by thirty per cent on day two will regress, and the temptation to stop while ahead is strong and specifically destructive. Stopping when the numbers look good means stopping at a moment of favourable randomness, which systematically over-reports wins.
Duration deserves the same discipline. Run full weeks. Behaviour differs across the week, and in the Saudi market the Sunday-to-Thursday working pattern makes partial-week tests particularly misleading — a test running Sunday to Wednesday captures a workweek audience and misses the weekend one entirely.
One primary metric, decided in advance. Secondary metrics can be observed, but the decision belongs to the primary. Teams that leave this open find that whichever metric happened to move becomes the metric that mattered, which is not analysis but justification.
There is a real trade-off here that is usually presented as a rule.
Changing one element isolates cause. If a single headline changes and conversion moves, you know why. The cost is that single-element changes usually produce small effects, and small effects need large samples — which many sites do not have.
Changing a whole page tests a concept. Effects are larger and easier to detect, and if it wins you know the new version is better without knowing which part did the work. For a site with modest traffic this is frequently the more honest choice: it is better to detect a real large effect than to fail to detect a real small one.
The decision should follow traffic. High volume supports isolation and the learning that comes with it. Lower volume favours bold concept tests, accepting less diagnostic precision in exchange for being able to conclude anything at all. Pretending otherwise — running granular tests on thin traffic — produces a stream of inconclusive results that slowly discredits the whole programme.
The most common failure in marketing experimentation is running a test that could never have detected the effect being sought.
The relationship is unforgiving: smaller expected effects need disproportionately larger samples. Detecting a two per cent relative improvement requires vastly more traffic than detecting a twenty per cent one. A page with a few hundred conversions a month cannot reliably detect small changes, and no amount of patience fixes it within a useful timeframe.
Calculating the required sample before launching is a five-minute exercise that prevents months of ambiguity. When the arithmetic says the test is not feasible, the options are honest: test a bolder change, test higher up the funnel where volume is greater, aggregate several pages into one test, or make the decision on judgement and stop pretending the data will settle it.
That last option is legitimate and under-used. Deciding by reasoning is respectable. Deciding by reasoning while claiming data support is not.
A healthy programme loses often. If most tests win, the hypotheses are too timid or the analysis is too generous.
Losses are also informative in a way wins are not. A losing test on a strong hypothesis has told you something specific about the audience, which is more than a marginal win on a weak one. The instinct to bury negative results costs the organisation exactly the knowledge it paid to acquire.
The habit to guard against is re-slicing. When a test loses overall, someone will find a segment where it won — new visitors on tablets, or one traffic source. With enough segments, something always looks positive. Unless the segment was specified in advance as part of the hypothesis, this is not a finding; it is the arithmetic of looking repeatedly. Segments should be declared before the test, not discovered after it.
Experimentation is usually confined to on-page conversion testing, which leaves its most valuable application unused: establishing whether marketing spend is causing outcomes at all.
Attribution models divide credit among touchpoints that occurred. They cannot observe the counterfactual. Only a deliberate absence can — a geographic holdout, a matched-market comparison, a scheduled pause on a channel that reports well. These are the tests that answer the question leadership actually asks, and they are run far less often than page tests because switching off something apparently working is uncomfortable.
They are also where the largest sums are decided. A page test might move conversion a few points. An incrementality test on a major channel can reveal that a substantial budget line is buying demand that already existed. Running the same discipline across media and on-site experiments, rather than treating them as separate crafts, is what turns testing from a conversion tactic into a budget instrument. It also depends on the reporting layers being clean, which is why it sits close to analytics and tracking work.
Record every experiment: the hypothesis, the mechanism, the design, the result, the decision taken, and the reasoning. Keep it somewhere the next person will find it.
This is the least glamorous element and the one that compounds. Without it, teams retest the same ideas every couple of years as people move on, and the accumulated understanding of what does not work in this particular market, for this particular audience, is lost with each departure. Institutions that test well are usually institutions that remember well.
The log also disciplines the programme. A column recording what decision each result produced makes it immediately visible when experiments are generating facts instead of choices — the entries where the decision column is empty are the tests that should not have been run.
Before launching anything, three questions. What decision is waiting on this. What outcome would prove us wrong. Can this test detect the effect we care about with the traffic we have.
Most proposed tests fail at least one, and abandoning them is a gain rather than a loss — the traffic goes to a test that can conclude something, and the team's confidence in the programme survives.
The useful question is not how many experiments you ran last quarter. It is how many decisions you made differently because of them.
Search is quietly changing shape. For twenty years the job was to earn a position in a list of links and wait for the click. Increasingly, the answer arrives before the list does — assembled by a language model, delivered in a paragraph, with a handful of sources credited underneath. Generative Engine Optimization is the discipline of making sure your brand is one of those sources.
The shift matters commercially, not just technically. If a potential client asks an AI assistant which firms handle performance marketing in Riyadh and receives a confident three-sentence answer naming three companies, the competition for that query was decided before any website was visited. Ranking fourth on a page nobody scrolls to is not a consolation prize. It is invisibility with extra steps.
Generative Engine Optimization, usually shortened to GEO, is the practice of making a brand's content retrievable, quotable, and attributable by AI answer engines — Google's AI Overviews, ChatGPT's search mode, Perplexity, and Microsoft Copilot among them. The objective is citation and inclusion rather than a numbered position.
In classic search, the unit of competition is the page. In generative search, the unit of competition is closer to the passage. The system is looking for a piece of text that cleanly answers the question it is trying to resolve. A page can be excellent overall and still be passed over because no individual passage inside it states an answer plainly enough to lift.
The honest case for acting early is not that GEO is a solved discipline. It is that it is an unsolved one, and the cost of entry is currently low. Generative search has no incumbency yet — the brands being cited today are frequently the ones whose content happens to be structured in a way the model can use, not the ones with the largest domain authority.
That window will close. As more organisations publish specifically for retrieval, the same accumulation dynamics that made classic SEO expensive will apply here too. The advantage available in the next year is a timing advantage, and timing advantages expire.