True Randomness is Hard to Find

Soooooo here's the deal...

We want a Random sample of the population and we'd like it to fall by Age, Gender and Location...just as ABS % falls (purely by using CATI Last-Birthday method and no quota chasing).

Soooooo here's the fact....

IT WON'T!!

Just because we want our project to PERFECTLY reflect  ABS proportions it doesn't mean it will.

Just because I close my eyes and wish for an Aston Martin V8 Vantage to appear with two tickets to Fiji in the glove box it doesn't mean it will.

Just because I would like to have World Peace by the time you finish reading this blog it doesn't mean I will.

Now you may just think I'm being a bit silly but time and time again we spend too much time "hoping" for a result without actively getting involved to think about a solution that will work for our clients, and their budgets.

....lets continue....


Just hoping for the "right" fallout isn't enough to get by. We need to watch the proportions collected in field closely...if the incidence is not falling as expected then we may need to be creative in how the remainder of the project is executed.


...however, prior to going into field we'll need to ask ourselves these questions...

What does the client really need analysed? Do they really need absolute proportions of sample to reach these quotas?.....We need to be clear on the answers we receive from these questions as they will form the backbone of data collection.


Here is an example of a "quota table" that a client may provide...

  

	D
	Proportionate sample n=
ACT Male (18-34 yrs)	3
ACT Male (34-55 yrs)	3
ACT Male (55+ yrs)	2
ACT Female (18-34 yrs)	3
ACT Female (34-55 yrs)	3
ACT Female (55+ yrs)	2
NSW Male (18-34 yrs)	49
NSW Male (34-55 yrs)	61
NSW Male (55+ yrs)	51
NSW Female (18-34 yrs)	50
NSW Female (34-55 yrs)	63
NSW Female (55+ yrs)	57
NT Male (18-34 yrs)	-
NT Male (34-55 yrs)	2
NT Male (55+ yrs)	1
NT Female (18-34 yrs)	2
NT Female (34-55 yrs)	2
NT Female (55+ yrs)	1
QLD Male (18-34 yrs)	30
QLD  Male (34-55 yrs)	36
QLD Male (55+ yrs)	29
QLD Female (18-34 yrs)	30
QLD Female (34-55 yrs)	38
QLD Female (55+ yrs)	32
SA Male (18-34 yrs)	11
SA  Male (34-55 yrs)	14
SA Male (55+ yrs)	13
SA Female (18-34 yrs)	11
SA Female (34-55 yrs)	15
SA Female (55+ yrs)	15
TAS Male (18-34 yrs)	3
TAS Male (34-55 yrs)	4
TAS Male (55+ yrs)	4
TAS Female (18-34 yrs)	3
TAS Female (34-55 yrs)	5
TAS Female (55+ yrs)	5
VIC Male (18-34 yrs)	38
VIC Male (34-55 yrs)	46
VIC Male (55+ yrs)	37
VIC Female (18-34 yrs)	38
VIC Female (34-55 yrs)	48
VIC Female (55+ yrs)	43
WA Male (18-34 yrs)	15
WA Male (34-55 yrs)	19
WA Male (55+ yrs)	15
WA Female (18-34 yrs)	15
WA Female (34-55 yrs)	19
WA Female (55+ yrs)	17
Total	1004

          

......Is our project going to fall out perfectly like this table (above)...?

.....soooooo what do we need to check?

What is the client really looking for? Are WE (as THE EXPERTS) over complicating the brief?

More often than not the client just wants a reasonable spread of age, gender and location. It doesn't have to fully replicate ABS % and the most accurate way to undertake this (without blowing our budget) is with the.....

Batch and Exhaust Process

It’s important, in particular for Social Research projects, that we are using a Batch and Exhaust method of sample management.

The process is very easy to follow:

1. Source Nationally representative sample (by State);
2. Data Team Randomise entire sample;

3. Data Team split sample into number of Batches required (usually 3 labeled Batch A, Batch B
and Batch C but each project may differ depending on the ratio of sample records to interview
projected...usually 10:1 ratio will suffice unless there is specifc qualifying criteria likely to
TERMINATE a high proportion of candidates, in which case you may require more);

4. Data Team allocate sample locations;

5. All interviewers are placed into the first Batch and exhaust this Batch (as determined by
sample specifications. This can be anywhere from 3 dials to 15+ dials per active sample record);

6. Once interviewers begin running out of phone numbers in Batch A they are slowly placed in 
Batch B. We need to ensure that there are still interviewers moving in and out of the first Batch
to ensure the sample is exhausted and Appointments are honoured;

7. As the project is drawing to a close it should “tail off” so no fresh Batches are being accessed
in the last few days of fieldwork (unless it is absolutely necessary to reach quotas).

The aim of this process is to ensure that our random sample of the population allows a good spread. So all potential respondents have an opportunity to answer the survey (via Last-Birthday method) without an inherent bias on contacting respondents on a certain day of the week or time of the day.

The expectation is that EVERY Batch that has been accessed is exhausted by the end of the project. Although to achieve specific quotas a fresh Batch may need to be accessed on an Ad Hoc basis.

Lets go through the 2 PRIMARY questions clients have when concerned with this method:

1. Is it true that when we have one big "bucket" of sample we can't target individual locations?

Correct. Using Last-Birthday method, given sufficient timelines, we will get the spread our "ABS CATI Reality" provides. If for any reason we need to top up locations for analysis purposes then we can add further locations (specific by State, Metro, Rural etc. randomised and de-duped against the rest of the sample in the project). This will give us flexibility (but we should only use this as a LAST RESORT as this reduces the true Randomness of the process).

2. What if we just started with State/specific locations so we didn't have the issues with trying to chase these quotas in the end. Wouldn't this give the same result?

Incorrect. If we target individual states but we are trying to get a National spread we are likely to put a bias in our sample. Foreinstance: If our Call Centre is in the Eastern States we may only call WA after 7pm for ease of Call Centre management thus reducing our chance of candidates who are home before 7pm an opportunity to take part. We may also find that our older age brackets will fill up first when targeting QLD but our younger age brackets will fill up most when targeting NSW/VIC...

..then when we look at our data we will notice heavy skews in specific States. When we weight it we may end up with some very small counts by State and age.

So our aim is to take the "choice" away from the Call Centre and keep the process automated by our systems that are designed to do so.

At the end of the day using this method is a lot easier to manage (so we're not juggling multiple sample locations) and it gives us a much better spread (the closest to natural fallout we will find in the CATI environment).

Lastly, as you may or may not have read in my previous blogs...whatever doesn't fall out during this Batch and Exhaust process (whether it be age or gender) can be filled with 1 of 3 options.

Either:

1. Chase the quota/s at the incidence they consitute in the population (usually costing much more than a client is prepared to bear);

2. Use panel/purchased sample to top up quotas, analyse this data separately and if there is a major skew in findings with respondents of the same demographics then report separately. If there is little difference between the two sets of data then merge them and report as a whole; or

3. Accept the Fall Out as is!

Thanks for reading,

U
P.S. This process DOES NOT work without sufficient TIMELINES