Expert Judgement: enhancing the human element in data-driven decision-making

Diarmuid Harvey | 4 July 2024

In today’s complex world, leaders frequently lean on information to make strategic decisions. However, while data is powerful, it can only ever inform a critical judgement, not make it. In such situations, expertise and intuition play a pivotal part in decision-making, but not without a cost. Even the most informed of experts are prone to errors of judgement, and this is often the result of an inescapable aspect of our cognitive processes – our inherent biases.

As our Head of Science, Diarmuid Harvey explains, ‘Expert Judgement’ methods can provide a sound scientific basis for addressing this issue. This concept blends data analysis with human insight to overcome cognitive biases and ensure that decisions are enriched by experience, not hindered by it.

The human factor

Information alone is often an insufficient basis upon which to make a reliable decision, even in cases where data is abundant. Leaders must also rely on their own or others’ intuition, which, in practice, often means decisions falling to the most confident, powerful, or outspoken individuals in a group.

While the inclusion of experts in the decision-making process can undoubtedly reduce uncertainty, it can also serve to reinforce rather than counteract this social bias. The appeal to authority is a well-known logical fallacy that is all the more inexcusable with the knowledge that bias is simply a natural and unavoidable by-product when human cognition is involved.

This is not to say that no expert should be believed, rather that even those most genuine are fallible. The key is how we acknowledge, understand and mitigate against this fallibility to ensure greater robustness in our decision-making processes.

Cutting through the noise

What’s more, individual biases aren’t the only issues that can introduce error and inconsistency into decision-making. Other factors, termed ‘noise’ by Nobel prize-winning Daniel Kahneman, also play a role.

While biases are types of errors that are consistent and to some extent predictable, noise is random and inconsistent. An example of bias is a judge that consistently tends to hand out harsher penalties than those given on average by his peers for similar crimes. The variation in penalties given by a judge for the same crimes, given the same factors, is an example of noise, and can be influenced by mood, among other things.

Noise can therefore arise among individuals with similar expertise and even within the same person over time, influenced by minor factors like whether they’ve eaten lunch. This source of randomness is particularly problematic because it adds unpredictability, and therefore increases uncertainty in the decision-making process.

Crowded house

An increasingly recognised solution to mitigating both noise and bias-induced errors is found in the ‘Wisdom of Crowds.’

This method aggregates information from groups to neutralise individual errors. Unlike crowd psychology, where people often influence each other, the Wisdom of Crowds depends on independent observations and the access each individual in a group has to accurate information.

The field of ‘Expert Judgement’ leverages this collective wisdom and has been successfully applied in financial risk assessment, climate science, and nuclear safety.

Noise cancelling

Standardisation of processes and procedures are critical to eliminating potential sources of noise. Structured means of gathering information like the Delphi method and Nominal Group Technique are therefore key to Expert Judgement methodologies.

However, these methods require significant time and effort, making them impractical for busy leaders and experts except for the most critical decisions. What’s more, even if they are successful in gaining a consensus, they can struggle to fully quantify the level of uncertainty in a decision resulting from residual noise or bias.

Fortunately, recent technological advances have reduced time and geographical barriers and allow for the application of more dynamic, advanced statistics. As a result, faster, more robust decision-making approaches are possible, enabling leaders to tackle complex business challenges with reduced time and effort while offering greater confidence and precision.

The NSCG approach

Expert Judgement practices are particularly critical in information gathering and hypothesis building situations where datasets are small and data sparse and/or imprecise.

This is frequently the case in the field of people analytics; consequently, Expert Judgement forms a central role in our methodology. We gather insights from industry veterans, thought leaders, consultants, and internal stakeholders, assessing their expertise, experience, and confidence levels. Using Cooke’s classical model for calibration, we employ seed questions to adjust for individual confidence errors. Put simply, we assess the extent to which the confidence expressed by experts in their estimations of known quantities reflects the accuracy of those estimates.

For instance, we might ask an expert to estimate a city’s population and then ask them their confidence in that estimate. Asking a range of similar questions, we measure and adjust errors relative to confidence. We then apply statistical methods like weighted averaging, regression analysis, and Bayesian inference to combine these judgements and quantify levels of uncertainty. This approach leads to more robust estimations and greater confidence in our inferences. For decision-makers, it enables them to maximise the available expertise with a substantial reduction in bias and noise-induced error.

These types of techniques are vital for stakeholder engagement and information elicitation, helping map appropriate people measures to organisational contexts, estimate the impact of individual profiles on team dynamics, and support decision-making in high-uncertainty situations. Beyond the most common use cases for us, these methods can be leveraged for any business-critical decision.

Balancing act

In today’s complex business landscape, data can often be sparse or imprecise. By blending data analysis with Expert Judgement, leaders can address the challenges of dynamic environments more effectively.

This approach ensures decisions are grounded in solid data and enriched by expert insights, leading to more robust and effective outcomes.

Want to find out more? Request a call now.

Share this: