Crunch those ABS! (the ebb and flow of field)

So you're a Researcher and you get a request from your client to provide data on how the general public feels towards say, Child Obesity, a very important Social Research initiative.

You begin to discuss the requirements of the proposal.

In a nutshell the client has particular interest in learning what 30 - 65+ year olds in the Northern Territory (NT) think about this topic and how it currently affects, or is likely to affect their lives or lives of their loved ones in the future.

The client wishes to get a "spread" of this population. In theory this seems quite an easy task. Using RDD we can call into the NT and conduct the interviews over the telephone thus ending  up with a spread of ages, genders and locations.

But wait...After further discussion with the client we discover there is more to their requirements than we realised...

The client expects that we will capture a sufficient sample size to report on, ideally with ABS % in mind. This being the case (and thinking about an N=500 scenario) they would like:
(NT) 30-34 year olds: 80 (which is 8% of the Population, 16% of the quota group)
(NT) 35-39 year olds: 80 (which is 8% of the Population, 16% of the quota group)
(NT) 40-44 year olds: 70 (which is 7% of the Population, 14% of the quota group)
(NT) 45-49 year olds: 69 (which is 7% of the Population, 14% of the quota group)
(NT) 50-54 year olds: 61 (which is 6% of the Population, 12% of the quota group)
(NT) 55-59 year olds: 52 (which is 5% of the Population, 10% of the quota group)
(NT) 60-64 year olds: 37 (which is 4% of the Population, 7% of the quota group
(NT) 65+ year olds: 51 (which is 5% of the Population, 10% of the quota group)

Now we could say...hang on a sec...how about we just get a spread of the population (either via Last-Birthday, Rizzo or Kish Methods...each has its advantages) and then weight the data at the end of the project...Well with a target as low as N=500 to push quotas up or down with more than a 5% differential could cause a major skew in findings.

So we have a two pronged issue:
1) The Target of N=500 provides very little flexibility to adjust the quotas to suit the ebbs and flows of data collection;
2) The Population of the Northern Territory doesn't warrant a target much higher than N=500 considering the selection of the population we are targeting is only 117,000 (out of a TOTAL (NT) POPULATION of 226,000) and we will be spending a large amount of our hours budgeted potentially chasing tough to reach quotas.

But...the client then throws another spanner in the works..."We also need to analyse by Gender!" they say.

This is where we come up against the dreaded "Interlocking Quota" (shudders).

The interlocking quota is essentially a quota that is used to garner very specific information within a quota that already exists.

So we go from a "Mother Quota" of say (NT) 30-34 year olds to a couple of subsets of this as (NT) 30-34 year old Males and (NT) 30-34 year old Females. We then have a situation whereby the population incidence per quota has reduced even further....

(NT) 30-34 year old Males: 41 (which is 4% of the Population, 8% of the Quota Group)

(NT) 30-34 year old Females: 39 (which is 4% of the Population, 8% of the Quota Group)

(NT) 35-39 year old Males: 41(which is 4% of the Population, 8% of the Quota Group)

(NT) 35-39 year old Females: 39 (which is 4% of the Population, 8% of the Quota Group)

(NT) 40-44 year old Males: 36 (which is 4% of the Population, 7% of the Quota Group)

(NT) 40-44 year old Females: 34 (which is 4% of the Population, 7% of the Quota Group)

(NT) 45-49 year old Males: 36 (which is 4% of the Population, 7% of the Quota Group)

(NT) 45-49 year old Females: 33 (which is 3% of the Population, 7% of the Quota Group)

(NT) 50-54 year old Males: 31 (which is 3% of the Population, 6% of the Quota Group)

(NT) 50-54 year old Females: 30 (which is 3% of the Population, 6% of the Quota Group)

(NT) 55-59 year old Males: 28 (which is 3% of the Population, 6% of the Quota Group)

(NT) 55-59 year old Females: 24 (which is 3% of the Population, 5% of the Quota Group)

(NT) 60-64 year old Males: 21 (which is 2% of the Population, 4% of the Quota Group)

(NT) 60-64 year old Females: 16 (which is 2% of the Population, 3% of the Quota Group)

(NT) 65+ year old Males: 27 (which is 3% of the Population, 5% of the Quota Group)

(NT) 65+ year old Females: 24 (which is 2% of the Population, 5% of the Quota Group)

So let me tell you what the issues are with this approach:

1) The more specific the quota, the more costly it is to achieve;

2) Broader quotas (of say an age group without interlocking gender) are a lot quicker to achieve, thus reducing time in field, and making the project more affordable;

3) The more specific the quota, the more unknowns you introduce to the project (what if Males in the Northern Territory are less likely to have landlines to call? What if  Females 30-34 only work in very specific Metro locations within the NT so more time is necessary to track them down?)

Sooooo....The key message to take out of this is....

The more specific the quota, the lower the incidence, the higher the cost.

Why?
Your main cost driver are the casual CATI interviewers. They are on the phone trying to get in touch with these very hard to reach quota groups...and not only that, the CATI interviewer motivation for the project can wain if they're sifting through phone numbers chasing a quota with a 4% incidence.

Have you ever worked on the telephones trying to get in touch with a member of the population at less than 20% incidence rate? No? Well then have a chat to a CATI interviewer and they will tell you it's agonising and this frustration can come through in an interviewers voice. So when the interviewer eventually gets in contact with someone who qualifies they could lose the interview due to a refusal that has occurred because the interviewer is out of practice (as they are so used to having respondents disqualify).

Remember...at the end of the day you have "PEOPLE" calling "PEOPLE" so the process should be as user friendly as possible.

Lastly, if the client really MUST achieve these quotas we need to be as cost effective as possible. How do we do this?


Be honest with the client on likely scenarios good and bad (speak to your Field Team if you're unsure of the potential pitfalls). The client MUST be flexible.

In a recent project that had very similar quotas to those just discussed I discovered that although the Gender Split within the population as ABS states should be 52% Males and 48% Females, using the Last-Birthday method of random selection the quotas fell out 35% Males and 65% Females. Now this was very difficult to determine "WHY?". Are Males less likely to answer a telephone from an unknown number, are phones primarily issued to the Female in the household, or is there another factor causing this?

Well we may never know the true answer (or answers) but we can work with this information to mitigate risk of project failure. So HOW do we do this?

1. We can re-adjust our CATI quotas and target Males via a mixed-methodology approach;

2. We can use representative panel sample to boost quotas (but isolate data from this sample to determine any skews before merging it with the rest of the data collected);
3. We can snowball by contacting respondents who have already taken part in the study and attempt to get phone numbers of friends/family we can target direct.

A word of CAUTION...the alternative approaches just discussed are likely to produce at least a slight variation to the results we would produce if undertaking purely RDD.

So I'm going to leave you with a final point of note...think carefully about the questionnaire design to make it as iron clad as possible for there may be a need for you to be creative about how the data is collected while mid-field.

Thanks for reading.
U