Municipalities have been conducting waste composition studies in Ontario for decades. They provide valuable insights into program operations, aid in directing promotion & education (P&E) resources and developing long-term waste management strategies. However, trying to figure out the correct sample size, frequency of sampling and distribution is not as simple as it may seem.
How many samples should I take?
Waste composition studies aren’t cheap. So, it’s only reasonable that municipalities want to ensure they get the best possible results. At issue is, “how many samples are enough to give reliable results?” We’ve all seen survey results presented in terms of ‘confidence intervals’ (e.g., an election poll suggesting someone will win with a 95% confidence level ±5% margin of error). But what does this really mean and how does it apply to waste composition studies?
The confidence level and margin of error effectively represent a range, where if you repeat the same study, you can be ‘confident’ your results won’t change by more than the margin of error. It’s not to say that your results are, in the example above, 95% accurate. To better understand this distinction, imagine a person throwing darts. On average, they hit the bullseye five times for each 20 darts. Suppose we keep a running count of how many times the player hits the bullseye over many rounds of throwing 20 darts. With each round, we get a better idea of how accurately the dart player throws. Eventually, we will become 95% confident that in the next round of 20 darts, our player’s average number of bullseyes (i.e., five) won’t change by more than a small amount (the margin of error). This is different from stating that the dart player is 95% accurate (i.e., hits the bullseye 19 out of 20 times).
Factors Affecting Sample Size Determination
Most surveys typically consider a yes-or-no response, or a limited number of responses. Based on the number of surveys conducted, pollsters can determine the confidence level and margin of error of the data. However, waste composition studies are not just a yes-or-no issue. There is a broad range of materials that are sorted plus they vary in total amounts. So not only does the methodology need to consider whether the material is present (i.e., a ‘yes/no’ issue), but also how prevalent it is as a proportion of the total sample composition. Not surprisingly, materials present in smaller quantities require more samples to achieve the same confidence level and margin of error as those that are more prevalent.
Additionally, there is a long list of factors that affect material generation and composition. We’ve all seen the impact the COVID-19 lock down has had on waste generation but variables like household demographics, seasonality and program participation have a big impact on waste generation. In most cases, municipalities simply don’t have enough budget to develop a study that can consider all of the possible variables and achieve high confidence levels (i.e., > 90%) with low margins of error across the broad range of material typically present in the waste stream.
Trade-Offs
Recognizing that most municipalities have a limited budget, three key questions should be considered:
- How diverse is the population demographics?
- Are most residents provided with the same level of waste service?
- Are you looking for big picture trends or looking to target a specific material?
Available budget will ultimately dictate the number of samples that can be taken and the project team will have to decide how best to allocate them to examine the issues in question and address identified variables such as demographics. Obviously, the more consistent factors such as the waste service levels and population demographics are, the greater the data consistency will be and the higher the confidence level will be across a set number of samples.
Dealing with Demographics
For most municipalities, it will be more important to focus their efforts on getting the sample distribution across the community right, especially if the data is being used for program planning. Recognizing that many communities have distinct demographic groups, it’s typically easiest to divide a community based on income levels as a surrogate for demographic differences. This can be done by obtaining Stats Canada data on household income levels, and proportioning it out into Low, Medium and High Income. Alternatively, a more complex analysis can be done that considers multiple factors through an Analysis of Variance (ANOVA) test as outlined in CIF Project #1059: Residential Audit Sample Optimization Toolkit.
New tool for determining confidence levels and sample size
In order to help municipalities determine their confidence level for a set number of samples, the CIF has hired Martin Lysy, Associate Professor of Statistics and Director of the Statistical Consulting and Collaborative Unit at the University of Waterloo (PhD in Statistics, Harvard University, 2012) to develop a tool and guidance document to provide municipalities with an assessment of the trade-offs between statistical accuracy and budget.
The tool relies on ballpark estimates of waste composition data that the CIF has collected, or users can specify from their own historical waste audits. Based on these inputs and user-specified margins of error and confidence levels, the tool will estimate the number of samples required. Users can also test different sample sizes to see the resulting confidence levels and margin of error to ensure they can meet budget constraints. Work is still under way to finalize this new tool but if you want more information contact Neil Menezes at nmenezes@ecocompass.ca.