don’t take SWS yolanda survey on aquino’s “satisfaction performance” seriously – defective design

i was very surprised when media carried this news on the latest SWS survey results that said super typhoon yolanda victims themselves gave president aquino a “very good” performance satisfaction rating.

SWS: PNoy gets ‘very good’ satisfaction ratings from Yolanda victims

Although he has received criticism from some parties for his administration’s response to super Typhoon Yolanda (Haiyan), President Benigno Aquino III got “very good” satisfaction marks from victims of the typhoon, according to a survey by Social Weather Stations.

SWS said this was the finding of its survey last Dec. 11 to 16, where Yolanda victims – which comprised 13 percent of Filipino families – gave Aquino a “very good” +54 net satisfaction rating.

“The December 2013 survey found higher satisfaction ratings for President Benigno ‘Noynoy’ Aquino III among victims of Yolanda compared to those who were not victimized by the super typhoon,” the survey said.

http://www.gmanetwork.com/news/story/345123/news/nation/sws-pnoy-gets-very-good-satisfaction-ratings-from-yolanda-victims\

essentially the same story was reported by other media outlets. reading the news stories alone, i immediately had a different reaction to the survey results which i tweeted (@wawam). will talk about this next. 

of course malacanang jumped on the “good news”.

Palace ecstatic over Aquino’s ‘very good’ satisfaction rating

MANILA, Philippines— Malacañang  gave itself a pat on the back on Thursday after the latest Social Weather Stations (SWS) survey showed that Super Typhoon “Yolanda” victims were satisfied with the performance of President Benigno Aquino III during and after its devastation.

“It is gratifying that those who suffered greatly appreciate what their President and their government have done to ease their pain and alleviate their plight despite the shortcomings and challenges still being hurdled,” Secretary Herminio Coloma Jr. said Thursday in a press briefing.

Read more: http://newsinfo.inquirer.net/568259/palace-ecstatic-over-aquino-very-good-satisfaction-rating#ixzz2rTD2g5nU
Follow us: @inquirerdotnet on Twitter | inquirerdotnet on Facebook

just based on the survey results published in media, i thought something was not right about the survey which i tweeted:

1. the respondent base for yolanda victims appeared to have been just extrapolated, not a stand alone respondent base. it appeared to me SWS re-looked at the survey results and counted the results of those who live in the yolanda areas and tabulated these results. i felt SWS did not actually conduct a separate survey to ask the respondents these questions but just looked at the survey results of an already finished survey
2.  coming from the above, i thought the question asked was a general “performance satisfaction” of president  aquino and not specific to aquino’s performance on the yolanda relief efforts. it is very possible that respondents will give very different answers to these questions.

last january 22, SWS posted the specifics of the survey results : i was both right and wrong on the point #1 and was right on point #2. conclusion remains – let us not take these results seriously, the results are what we call in research “soft results”.

i was wrong on point #1 because SWS did not just extrapolate the answers of those living in the yolanda areas what they did was they actually asked the respondents if they were affected by the yolanda.

there could have been 2 ways to extrapolate – 1 is by just applying the % of  respondents living in the yolanda areas to the total results of the total visayas results (a very wrong thing to do) or look at the respondent answer sheets and count the answers of those declaring their address to be in yolanda areas. none of these were done. what they did was they asked a direct question.

i was right in the sense that SWS did not conduct a separate and stand alone survey among those living in the yolanda areas. this should have been the cleanest way to do it. you get an actual sample size in the areas themselves.

since SWS just asked the question among respondents if they were affected by yolanda, this brings up two more questions – is the sample size of 13% enough to read the results and how accurate is it to say that these respondents actually “lived in yolanda areas” when these were just “claimed”.

on the point #2, i was proven right.

the question was a generic performance satisfaction question – “satisfied or dissatisfied in performance as president of the philippines”  and not specific to the performance of the president on post yolanda relief efforts.

the specific “performance on yolanda efforts” can get a very different answer from the generic “performance as president of the philippines”. the first zeroes in on yolanda while the second one includes everything about the presidency and the country in general.

for example, if we ask our spouse, we could be a “very good husband/wife”  as a whole but a “very poor sex partner” in a specific component of marriage.

we do not know why SWS released these survey results and how media plus malacanang take these results when the survey design is critically flawed.