Members of the BRC lead the development of ecology’s first “risk of bias” tool


“Risk of bias” tools help researchers to establish whether biases in their data might threaten the validity of their conclusions. Such tools are common in disciplines such as medicine and psychology. Recently, members of the BRC took this idea and led the development of ecology’s first risk of bias tool. The tool, ROBITT, was designed specifically for analysts who intend to estimate trends in species’ abundances and distributions.

Three ‘heuristics’ indicating the potential for geographic biases in data on hummingbird occurrences collected in Ecuador and Colombia from 1950 to 2019.
Boyd et al, 2022


Information on temporal trends in species’ distributions and abundances is needed to monitor progress towards international biodiversity targets and the effectiveness of conservation actions. Ultimately, this information comes from data documenting the sighting of one or more individuals of some taxon at a given place and time. Depending on where, when and how these data were collected—e.g. by professional biologists according to a defined sampling protocol, or opportunistically by volunteer naturalists—they may or may not be suitable for trend estimation.

To understand why a dataset might be unsuitable for estimating trends in species’ abundances and distributions, one must first understand the concept of statistical inference (it’s actually quite simple). Statistical inference is the process of estimating some property of a statistical population from a sample, or subset, of that population. Note that the word “population” is used here in a statistical context, which is to say all possible units about which inferences are desired. For example, if a researcher wanted to say something about stars in the Milky Way, then the statistical population would be all stars in the Milky Way. If the sample is not representative of the statistical population, then there is a risk that the sample-based estimate will be biased; that is to say, it might systematically deviate from the truth.

To demonstrate the dangers of inference from unrepresentative samples, consider the following (made up) example. An analyst wants to estimate changes in bee distributions in the UK since the year 2000. The quantity of data collected has been relatively consistent over this period; however, the geographic distribution of sampling has shifted towards to the South East of the country. If the bees are faring poorly in the South, but better in the North, then the analyst might erroneously conclude that they are in decline. Similarly, if there has been an increase in amateur recorders over time, then there might be an apparent increase in the types of species they record. It is well known, for example, that amateurs tend to preferentially record “interesting” species whereas professionals more often record everything that they see. This could lead the analyst to erroneously conclude that interesting species faring better than the others. It is essential then, that analysts understand the potential for biases to undermine their inferences about how species’ distributions and abundances are changing over time.

In other disciplines, such as medicine and related areas, it is common for researchers to complete “risk of bias” assessments to expose and document potential mismatches between the sample and population of interest. Recently, members of the BRC, with a team of international experts, have taken this idea and developed ecology’s first risk of bias tool (Fig. 1). The tool, ROBITT, was designed specifically for use by analysts who intend to estimate temporal trends in species’ distributions and abundances. The development of ROBITT is an important step towards more robust trend estimation than is typical. However, there is still much work to do to convince the community of the tool’s value and establish it as a requirement.

Figure 1. Overview of the steps in a ROBITT assessment with brief details about what is required at each stage.  



Boyd, R. J., Powney, G. D., Burns, F., Danet, A., Duchenne, F., Grainger, M. J., Jarvis, S. G., Martin, G., Nilsen, E. B., Porcher, E., Stewart, G. B., Wilson, O. J., Pescott, O. L., & Boyd, R. J. (2022). ROBITT: A tool for assessing the risk-of-bias in studies of temporal trends in ecology. Methods in Ecology and Evolution, 2022(March), 1–11. ​