An ochre sieve holds a cluster of coloured circles while others pass beyond its edges, a visual reflection on the experiences a survey may miss.
A community survey becomes more useful when its organisers understand who could answer and what the results can reasonably tell them
By Arbab Naseebullah Kasi, Chief Executive Officer - CEO, Feel Worldwide Foundation Inc.
Imagine a community centre deciding when to offer its next digital skills course. A volunteer puts a survey link beside the reception desk and shares it during the centre's daytime activities. Eighty people answer the question about timing. Sixty-four prefer morning sessions.
The draft announcement almost writes itself: the community wants morning classes.
Before it goes out, someone asks where a resident working during the day would have encountered the survey. Another asks whether a person who needs help using a phone could have opened the link. The count is correct. The conclusion has travelled further than the evidence.
This is an invented example, including its numbers. It illustrates a practical problem for charities, small businesses and community groups: the people easiest to consult may have different circumstances from the people a decision will affect.
A survey can produce useful information without representing an entire neighbourhood. The responsibility is to understand its reach and describe its findings honestly. That work starts before the first question is sent.
Name the people the decision affects
The centre has several possible questions to investigate. When would current daytime visitors like another class? When could existing learners continue? What would enable adults across the neighbourhood to start learning digital skills?
Those are different questions about different groups. The first might reasonably begin with people already attending daytime activities. The last requires attention to residents the centre has not yet reached.
Write the intended group in an ordinary sentence. For this example, it might be adults living in the centre's service area who want help with everyday digital tasks. Then ask how each part of the recruitment plan connects to that group.
An email list identifies people whose contact details the centre holds. A social media post reaches whoever encounters it. A notice in reception is available to people who enter the building. None of those routes automatically includes every person in the intended group.
This exercise can reveal an operational problem before it becomes a statistical one. Perhaps the centre has never established a way for non-users to hear about its work. Improving that connection may matter as much as choosing the next course time.
Separate being missed from choosing not to answer
A resident who never encounters the invitation is in a different position from someone who sees it and declines. Someone who starts the questionnaire but cannot finish has had another experience again.
Statistics Canada distinguishes coverage problems from nonresponse and other sources of survey error. It also explains that systematic bias is not removed simply by increasing the sample size. These distinctions help identify which part of the process needs attention. Read Statistics Canada's explanation of non-sampling error.
In our imagined centre, sending the same link repeatedly to the same daytime visitors might increase the number of answers. It would still leave the organisers unsure about residents outside that circle.
Nor should silence be translated into satisfaction, disinterest or agreement. The team does not know why a particular person did not answer. They might have been busy, missed the invitation, disliked the question or preferred to keep their views private.
The practical response is to examine the process without inventing the missing opinions. Where did the invitation go? Was there a usable way to respond? Were people given enough time? What remains unknown after those questions have been considered?
A large total still needs a sound method
Watching a response counter rise can be encouraging. It represents people taking time to contribute. But a bigger total alone does not establish that the results describe the wider population.
Pew Research Center tested this issue in a study published in 2023, using six online surveys conducted in 2021 with nearly 30,000 U.S. adults overall. Across 28 benchmark measures, the three opt-in samples had an average absolute error of 5.8 percentage points, compared with 2.6 points for the three probability-based panels. Read Pew's comparison of online survey samples.
These figures describe those samples and benchmarks. They are not an error allowance that a community organisation can apply to its own survey. The study also does not establish that every difference came from people being missed. Its wider lesson for this discussion is that recruitment and data quality deserve attention alongside sample size.
An open questionnaire can still uncover a concern, suggest a useful idea or identify people interested in a pilot. Those are worthwhile purposes. Trouble begins when an organisation presents that information as a reliable estimate of what everyone thinks without a method that supports the claim.
If a decision genuinely depends on a population estimate, seek qualified survey advice early. A volunteer should not have to invent statistical assurances to make a funding proposal sound more convincing.
Change the invitation before blaming the audience
For the centre, a sensible next step would be to consider how a resident who never attends daytime sessions could participate. That might involve a paper response option, a telephone conversation at a suitable time or invitations through a local organisation serving a different group.
The American Association for Public Opinion Research explains that collection methods have different advantages, costs and limitations. Its best-practice guidance discusses combining methods when some people are more reachable in another way. A survey's recruitment method also matters: collecting answers online does not, by itself, tell us whether the sample was selected through probability-based or opt-in methods. Read AAPOR's survey research guidance.
For a small organisation, the useful question is which obstacle a proposed change would address. Printing a form may help someone who cannot use the link. It will not help if the paper is available only inside a building they never visit.
The centre could ask a trusted local partner to circulate an invitation without requesting that partner's private contact list. It could offer assistance while making clear that the answers belong to the participant. It should also respect a person's decision not to take part.
These steps can broaden a consultation. They do not automatically turn it into a representative survey, and the eventual report should preserve that distinction.
Make room for an answer you did not anticipate
Return to the course-time question. If the available choices are Monday morning and Wednesday morning, someone who can attend only after work has no answer that describes their situation.
The organisers might learn more by asking which of several realistic time windows a person could attend, with a way to indicate that none would work. A short follow-up could ask what makes attendance difficult. The aim is to understand practical possibilities without promising a timetable the centre cannot deliver.
Try the draft with a few people before launching it. Ask them to explain what they believe the question means and how they chose their answer. If someone interprets “regular attendance” as once a month while the organiser means every week, that difference deserves a revision.
Some uncertainties call for a conversation rather than another checkbox. The UK Government's service research guidance describes in-depth interviews as a way to understand people's circumstances and problems, using open questions and concrete examples. Read the guidance on in-depth interviews.
An account of someone's journey home after work might explain why a proposed evening start is still too early. That account can guide a better question or service trial. It should not be turned into a percentage describing all working residents.
Keep the limits beside the headline
Suppose the centre keeps its original findings. A careful summary would say that 64 of the 80 people who answered the timing question preferred morning sessions, and that recruitment took place through daytime centre activities and its reception notice.
That statement allows the reader to understand what happened. Calling the result “80 per cent of local residents” would claim something the exercise did not establish.
AAPOR's disclosure standards call for information such as the population studied, recruitment method, question wording, collection dates and sample sizes. They also place conditions on reporting measures of precision for non-probability samples. Read AAPOR's disclosure standards.
A short community report can adopt the same spirit of transparency without becoming a technical textbook. Include the question people actually answered and the number answering it. If some skipped a question, do not quietly count their silence as a response. Explain important gaps where readers will see them, rather than hiding them in a distant appendix.
The organisation should resist adding a conventional margin of sampling error to an open-link survey simply because a calculator offers one. Where specialist methods are used, their assumptions need a competent explanation.
Use incomplete evidence with proportionate care
The centre still needs to make a decision. Recognising limitations does not require waiting for perfect knowledge.
Its existing responses could justify keeping a morning option for people who clearly want it. They would provide a weaker basis for declaring that evening provision is unnecessary. The team might explore a limited evening trial, subject to staffing and cost, while gathering information from people who could not attend the current programme.
Before the trial, the organisers could decide what they need to learn: whether interested residents enrol, whether the time works in practice and what prevents attendance. They should avoid treating empty places as a complete explanation. A poorly advertised course and an unwanted course can leave the same number of chairs unused for different reasons.
The scale of the decision matters too. Choosing a date for one optional session allows room to adjust. Closing a service or withdrawing access from a group deserves stronger evidence and more careful consideration.
I would rather read a modest report that helps a team make its next decision than a confident headline that conceals how little is known about the people outside the sample.
Our imagined centre does not have to discard the 80 answers. It has to keep them in their proper place, listen further where the decision requires it and explain the basis of its choice.
The most useful question may arrive just before the chart is approved: whose circumstances could change what we do, and have we given those people a reasonable opportunity to be heard?
