New Arbitron goof: rats ate the PPM data

Michael Hedges August 22, 2007

A new problem surfaced this week for US media measurement agency Arbitron in their roll-out of radio audience surveys collected with the Personal People Meter (PPM). A big chunk of data from the Houston, Texas survey disappeared. Arbitron calls it an “error.” Broadcasters call it a crisis.
followup(1);comment(0)

Arbitron supplies radio survey clients week-to-week sampling data, exact totals by age, gender and race of survey respondents. US broadcasters have been critical of sampling methods, compliance and results for many years and Arbitron responded by providing weekly figures. Difficulties meeting survey sample goals for diary-based surveys – apparent in the 1990’s - was one of several reasons Arbitron invested in passive electronic measurement systems.

Late Monday (August 20) data provided by Arbitron to broadcaster clients showed a one-third loss in carry-rate, the new PPM-speak term for compliance, for one week in the Houston survey. Philadelphia and Houston are the first US markets surveyed using the electronic device, officially from mid-July after two years of testing, squabbling and more testing. When questioned about the drop in “carry-rate,” one Arbitron official wrote to a curious broadcaster in an email Tuesday morning (obtained by ftm), “I’ll look into this and have someone get back to you today. It certainly seems possible that summer schedules and vacations could impact the number of people who qualified for 6 out of 7 days in August week 1, but we need to verify that's the cause.”

That would suggest one-third of the Houston population went on holiday during the first week of August. American holidays and vacations have disappeared in the Wal-Mart/Enron era. Another possibility is a quality control problem.

Yet another possibility for the “carry-rate” drop was mentioned a week earlier in an Arbitron conference call with key broadcasters. Compliance – the old term – had reared its head in both Philadelphia and Houston. Some folks in the survey panels just weren’t paying attention, not undocking the PPM and otherwise upsetting the sample. It seems survey panel participants grow less attentive to the PPM device with the passage of time. Of course, the primary purpose of passive measurement is elimination of that human touch. Arbitron started cutting the non-participants and recruiting new ones.

The importance of sampling in the PPM surveys cannot be understated as the results – that number media buyers demand and broadcasters pay for – are determined from two parallel samples: one to measure ‘cume’ (reach) and one to measure ‘average quarter hours’ (time spent listening). These numbers are then combined in a computer model to create a rating. Arbitron has also patented a confidential method of insuring compliance.

By early yesterday afternoon (Tuesday, August 21) one Arbitron official suggested a data processing error with one of the samples and, later, that the reported sample size was incorrect. In mid-afternoon, New York time, Arbitron officials turned the entire flap – fueled by the skeptical side of the US radio trade press – over to their public relations specialists. The official Arbitron response said, “The file used to process the weekly cume estimates…was incompletely loaded from the data warehouse to the production system.” In other words: rats ate the data file.

Said one broadcaster (email obtained by ftm): “Which is worse, a 30% drop in sample ... or an error of this scope that no one at Arbitron caught?”

All of this is more than vaguely, but materially similar to 2004 episode in the life of RadioControl – another electronic measurement system – in Switzerland. Changing sample provider in the midst of one survey period caused numerous changes in the results, including the ‘disappearance’ of several stations. Neither Radiocontrol nor PublicaData, its marketing service, informed clients of the change in advance and waited until one station owner – also a prominent Swiss political figure – threw what could only be described as a fit. The result, three years later, was a complete reorganization of the radio audience measurement system in Switzerland. Sampling, apparently, is even more volatile with electronic measurement systems.

Arbitron said yesterday – and last week, last month and last year – that sampling problems with PPM surveys would be corrected. Media measurement services operate like public utilities, mediating – like accounting – a relationship between broadcasters and media buyers. Confidence, on both sides, is built on transparency. Where transparency is weak, quality control weaker and the science reminiscent of Enron derivatives broadcasters and advertisers need to re-think whether or not they are getting what they’re paying for.

ftm Follow Up & Comments

Wake up and smell the coffee, Steve - November 15, 2007
This could be one of those days when Arbitron CEO Steve Morris wakes up asking himself why he ever left that good job running the Maxwell House coffee business for General Foods. Yesterday (Wednesday November 14) four of his biggest clients – Clear Channel Radio, Cox Radio, Cumulus Media and Radio One – fired off a letter, provided to ftm, that began, 'It is with the utmost urgency and objection that we, your customers, send you this letter.'...MORE