2.Survey: Measurement and Scaling
2.5 Reliability and Validity
Multi-Item Scales: Measurement Accuracy
The True Score Model
where:
- ХO = the observed score of measurement
- ХT = the true score of characteristic
- ХS = systematic error
- ХR = random error
A measurement is not the true value of the characteristic of interest but rather an observation of it.
Reliability & Validity
Reliability:
- extent to which a scale produces consistent results in repeated measurements
- absence of random error
(ХR ⟶ 0 |⇒ ХO ⟶ ХT + ХS) - reliability of a multi-item scale is denoted as Cronbach’s alpha (0 ≥ α ≥ 1)
- values of α ≥ 0,7 are considered satisfactory
Validity:
- extent to which differences in observed scale scores reflect true differences among objects on the characteristic being measured
- no measurement error
(ХS ⟶ 0, ХR ⟶ 0 |⇒ ХO ⟶ ХT)
Relationship between Reliability & Validity
- validity implies reliability
(ХO = ХT |⇒ ХS = 0, ХR = 0) - unreliability implies invalidity
(ХR ≠ 0 |⇒ ХO = ХT + ХR ≠ ХT) - reliability does not imply validity
(ХR = 0, ХS ≠ 0 |⇒ ХO = ХT + ХS ≠ ХT) - reliability is a necessary, but not sufficient, condition of validity
“The purpose of a scale is to allow us to represent respondents with the highest accuracy and reliability. We can’t have one without the other and still believe in our data.”
Bart Gamble. Vice president client services, Burke, Inc. 2000-2003
Net Promoter Score
competitive growth rates?
How likely are you to recommend company/brand/product X to a friend/colleague/relative?
Is the scale reliable?
Is the scale valid?
NPS (-100% – +100%)
5-10% average companies
45% high potentials with open growth opportunity
50-80% market leaders with high growth potential
Net Promoter Score: Warning
“Though the “would recommend” question is far and away the best single-question predictor of customer behavior across a range of industries, it’s not the best for every industry…So, companies need to do their homework. They need to validate the empirical link between survey answers and subsequent customer behavior for their own business.”
Fred Reichheld, 2011
