When reviewing audit, survey or other project data and trying to interpret the results, you may have asked:
“Does my data achieve statistical significance?”
This is one of the biggest and arguably most important questions to answer before you start your project because it will inform how many samples you need to collect.
To help, CIF in partnership with the Waste Wiki team has created a Sample Size Calculator. The calculator is an excel document that will estimate the number of samples you need to collect from your community of interest to achieve a specified statistical confidence level.
Seeing an example is one of the best ways to learn: so, we teamed up with staff from Wellington County to assess the underlying statistics of their recent survey looking into resident behaviours at their solid waste depot collection sites. The household population breakdown and related statistics have been included in the Sample Size Calculator as an example to help in understanding how the calculator works.
Background
To make waste management more convenient for the more than 90,000 residents, Wellington County operates a depot solid waste collection program at six locations across the County. Annually, in coordination with residential waste audits, County staff survey residents at the depot sites to get an understanding of who is using their facilities and why.
This year, County staff were working on their long term waste management strategy and evaluating how best to optimize their depot operations using the audit and survey data to inform their decision making.
What did they collect and how does it all stack up?
Solid Waste Services staff collected 692 complete surveys from residents at the six waste sites over a two week period in August. Staff were pleased as this collection exceeded the requirement of 380 samples to achieve statistical confidence at a level above 95%. At this level, it is reasonable from a statistical standpoint to report the information from the survey represented the views and opinions of residents within the overall community.
The surveys collected were not, however, sufficient to compare the results between waste site locations. The table below identifies the number of sample (surveys) collected, households in each transfer site’s service area, and the number of samples that would need to be collected to statistically achieve a 95% confidence level.
We need information to help identify how and why residents use our waste drop-off facilities and we also need to ensure that the information we gather from our surveys and waste audits is representative of our municipality.
– Cathy Wiebe, Administration Supervisor, County of Wellington
| Depot site location | Aberfoyle | Belwood | Elora | Harriston | Riverstown | Rothsay |
|---|---|---|---|---|---|---|
| Samples collected | 80 | 203 | 191 | 94 | 80 | 44 |
| Households | 4947 | 9520 | 7654 | 3247 | 4682 | 3141 |
| Samples required @ 95% | 357 | 370 | 366 | 344 | 356 | 343 |
This means it would be inappropriate from a statistical standpoint to compare and contrast responses from residents across sample areas. In the absence of statistical significance, one should exhibit caution when making decision regarding the specific needs of individual waste sites.
Final Thoughts…
Completing studies of resident perceptions and the composition of the waste stream are implicit in the planning of effective and efficient programs. Knowing the weight of the information you gather is of the utmost importance and should be incorporated at the beginning of the data collection process.
“Achieving statistical representation through appropriate stratification of your community is appropriate to get the full picture of how residents are using your waste management program. Setting targets similar to academic or polling organization standards (i.e., 95%) is necessary to ensure the data or information you are using for decision making is valid and defensible”.
– Cal Lakhan, PHD York University
The example from the County of Wellington is a relatively straightforward representation of how to stratify the overall collection of data by geography and population. In other circumstances, the stratification of your dataset may be more complex incorporating other factors such as age, income levels, ethnicity, language of preference, and others which require more time up front to plan out your data collection. Knowing what you want to achieve with your survey will help determine the best way to collect the data and the appropriate sample size required to ensure the results are defensible.
For more information on project #1068 with the County of Wellington, or if you want to know more about the tools available to help set up your next project or analysis, please contact CIF staff.