Hello Mozilla Connect Community, I’m Chance York, a User Researcher on the Firefox User Research team. I’m reaching out because our team has created a survey to gather opinions on a handful of browser features, some of which were suggested previously on Mozilla Connect. Your feedback on this survey...
"
As a vegetarian, which of these three options you want the least and which you want the most? You MUST choose to continue:
Chicken sandwich.
A kick in the balls.
Pork sandwich.
"
You are saying that a reasonable person (vegetarian in this case, and disclaimer: I am not vegetarian) would say “well, I want a kick in the balls the least, so I’ll choose that. Now, fuck, I HATE chicken sandwiches and I HATE pork sandwiches. They both make me puke. But if I have to choose, I guess I’ll go for the chicken sandwich. Hey pollster, I want the chicken sandwich the most.” And the pollster writes “Chester wants the chicken sandwich the most.” Yeah, very clear.
They could’ve just said “Rank the features from one to three accordibg to how much you like it”. This seems unneccessarily more confusing. It isn’t all that cobfusing, but it is an odd way to formulate the question.
The person evaluating the poll will take away “person likes option 1 most” not “person absolutely wants none of these in their browser, ever”. That’s the issue. You should not phrase questions in a way that assumes parts of the answer, at least not if you want useful results.
A better way would have been to let us rate features 0 to 10 and just accept if people thought their feature ideas are all shit.
It’s not a better way to rate from 0 to 10. It takes way more effort from the user and leads to more people dropping out. And in the end, the result is the same in aggregate. If your opinion is popular, more people will vote like you because the sets of 3 are random. In a survey of thousands, individual opinions don’t matter. No one is going to evaluate the answers one by one.
It takes way more effort from the user and leads to more people dropping out.
Then make it 0 to 3 or 0 to 1 for all I care. You missed the point, which is: If I want or don’t want feature A doesn’t influence if I want or don’t want feature B, and linking the two distorts the results of the poll.
in the end, the result is the same in Aggregate.
Not if you include the human factor of the decision maker, who can twist “wanted less” into “still wanted a bit” as a justification if they want a certain feature for different reasons than user benefit (like, say, a “privacy friendly” but indeed not at all privacy friendly mechanism to give data to add networks). That doesn’t fly with “0 points”.
But the language is clear. From these 3 features, choose the one you want the most and the one you want the least 🤔
The language is not clear.
" As a vegetarian, which of these three options you want the least and which you want the most? You MUST choose to continue:
"
You are saying that a reasonable person (vegetarian in this case, and disclaimer: I am not vegetarian) would say “well, I want a kick in the balls the least, so I’ll choose that. Now, fuck, I HATE chicken sandwiches and I HATE pork sandwiches. They both make me puke. But if I have to choose, I guess I’ll go for the chicken sandwich. Hey pollster, I want the chicken sandwich the most.” And the pollster writes “Chester wants the chicken sandwich the most.” Yeah, very clear.
That’s the typical “if you were in a deserted island” scenario that vegetarians and vegans are very familiar with. Given those 3 options:
1 > 2 > 3
They could’ve just said “Rank the features from one to three accordibg to how much you like it”. This seems unneccessarily more confusing. It isn’t all that cobfusing, but it is an odd way to formulate the question.
That’s already suggestive. What if you want none of them, and strongly so?
If you hate or love them all equally, I guess a random score is fine.
The person evaluating the poll will take away “person likes option 1 most” not “person absolutely wants none of these in their browser, ever”. That’s the issue. You should not phrase questions in a way that assumes parts of the answer, at least not if you want useful results.
A better way would have been to let us rate features 0 to 10 and just accept if people thought their feature ideas are all shit.
It’s not a better way to rate from 0 to 10. It takes way more effort from the user and leads to more people dropping out. And in the end, the result is the same in aggregate. If your opinion is popular, more people will vote like you because the sets of 3 are random. In a survey of thousands, individual opinions don’t matter. No one is going to evaluate the answers one by one.
Then make it 0 to 3 or 0 to 1 for all I care. You missed the point, which is: If I want or don’t want feature A doesn’t influence if I want or don’t want feature B, and linking the two distorts the results of the poll.
Not if you include the human factor of the decision maker, who can twist “wanted less” into “still wanted a bit” as a justification if they want a certain feature for different reasons than user benefit (like, say, a “privacy friendly” but indeed not at all privacy friendly mechanism to give data to add networks). That doesn’t fly with “0 points”.