Auditing Culture, and How Metrics Proliferate

Note: This post is in many ways an expansion or interpretation of the excellent fourth chapter in Limits Of The Numerical, written by Heather Steffen.

Previously, we’ve talked about high-stakes and low-stakes metrics, including their adverse effects. But if the effects can be this devastating, where do they come from and why do they proliferate?

Metrics used for accountability need to not only be captured, but also audited. In auditing, an auditor¹ goes to the organisation, and reviews how, why, and how well they are accomplishing their assigned goals. They do so by reviewing data the organisations are mandated to capture, either by law (for example, in finance or engineering), or by directive from above, and ideally, talking to the members of the audited organisation. Notably, the auditor need only have cursory awareness of the department they are auditing, they need to broadly know their goals, incentives and processes, but closer domain expertise is not required, or rather, not asked for.

It is in this audit that accountability metrics do their damage: Here, by someone not familiar with the closer circumstances of the audited organisation, judges the entire organisation with the help of metrics and measurements that the organisation has been, in one way or another, coerced to capture, meaning that the metrics audited on did not originate in the audited organisation for the purpose of diagnostics. It should be noted that this dimension even holds for your supervisor or boss reviewing your statistics to determine quality of performance.

The awareness that a regular, or even continuous, audit is going to happen, and knowing what metrics and measures are going to be paid attention to by the auditor, that (re)shapes how an organisation solves problems.² Knowing that the judgement of how well the organisation is doing in the way that matters most externally happens in a certain way warps the process of solving problems to be most advantageous to this audit. Problems are no longer solved in the way that would be most effcient for the organisation, the priorities have shifted: Now problems are solved in the way that is most visible in an audit, because that is how the organisation is judged as doing well.

The Proliferation Of Audit Culture

The story told here is that because the auditor is not directly involved in the organisation, and because the metrics captured are standardised, auditing is objective. No politicking has influenced the judgement of the auditor, no interpersonal squabbles with someone in the audited organisation are affecting them.

Yet, the interpretation and judgement of the captured metrics and data is still fundamentally filtered through the subjective view of the auditor. Because at the end of the day, it’s still humans dealing with humans, narrative can be, and is used, to influence this process. Any implementation of auditing that claims to be more objective and to remove narrative, is instead just creates new ones. Now, the narrative is based around the metrics: Why they are an exceptional case, why they should be particularly lauded this month because they were achieved against environmental resistance, what would be needed to better them. These sway the supposedly neutral auditor to varying degrees, sometimes the environment is such that the auditor is coerced to, or agrees to, rubberstamp the most disastrous trainwrecks of numbers.³

The idea of auditing being the transparency cure-all, fix-to-all-bad-actors pill itself propagates via narrative. A self-regulating, autonomous community of practice is deeply suspect to those convinced of the powers of auditing to regulate performance and outcomes. How are these communities swayed to embrace, or at least implement auditing culture?

Specifically, two narratives are told, one to the public about the community, and one to the individual members of the community, in order to convince them about the merits and benefits of auditing as means of acquiring truth.

The narrative to the public about the autonomous community primarily talks about the Bad Actors. The members of the community that maliciously do not perform as well as they should, or deliberately committing malpractice, are harbored and protected by the community. This is because nobody is taking a closer look at the community, therefore they are protected by the inaction of the community as they refuse to eliminate their bad actors.

The narrative told to invidual members of the community is instead about the alignment of values between the practitioner and the auditor: About convincing them that the auditor has the same interests in mind as the practitioner. It is, after all, about protecting the community as a whole, about upholding the standards of the community. To this end, auditing has to be adapted, to make sure. After all, we can’t trust what they’re saying, because they are deliberately and maliciously doing malpractice, or underperforming.

Consequences of External Auditors

Because ‘independence’ is a desired trait in auditors, the more important the audit in question, the more likely that the auditor is not, and can’t be, part of the organisation they are auditing. This means that them not being familiar with the politics of the organisation is considered good, as they are not influenced or part of these politics. The cost of this is that now they can’t account for the social dynamics in the audited organisation, making them likely to act to the benefit of those with structural and social power.

This means they have to fall back to auditing the industry “best practices”, and local adaptations are viewed, at best with suspicion, if not considered an outright flaw. It also means that the audited organisation, if they want to be able to pass an audit, need to adopt these best practices, regardless of them making sense for this organisation in particular. An external auditor coming to an organisation and finding well-made, locally-adapted processes is going to, at most, be able to tell that these are not the standard practices that are considered good.

This is exacerbated by the time limits the auditor faces. The neutrality of the auditor is limited, the longer they are embedded in the organisation, the more likely it is they will be accused of bias and having violated their impartiality. They have to hurry, they can’t fully understand the processes as it would be required. Therefore, both, the auditor and the audited default to “safe” processes and practices, no matter if these are any good for the organisation, or not.

This problem also plays out with executive hiring and senior leadership. Often, these are not brought up from inside the company, but hired from big, respected firms that have a reputation for Knowing Their Stuff. These are often hired specifically to fix situations, and so they are placed in exactly the same role as an external auditor: They have limited time, resources, and most crucially, understanding, but they are under the acute pressure to act and to be seen as deserving of their reputation as competent and decisive.

So, they standardise. They, now in a position of structural power, re-organise and restructure, until it resembles what they know and know how to manage. This is the second prong of how auditing culture propagates, and, so is my hypothesis, why most large companies have a very similar corporate culture.⁴

Auditing Purports To Be Its Own Solution: Transparency

All this time, the culture of auditing retains the siren song of metrics. If there is a problem that exists despite extensive metrics and auditing, then the answer is more metrics and auditing. If something has not been exposed and brought to light by auditing and metrics, then there is not enough auditing to successfully detect these hidden problems.

C. Thi Nguyen⁵ has written a fantastic paper⁶ that is, for us here, very aptly titled: Transparency Is Surveillance. In this, he quotes a lecture from Onara O’Neill that I shall quote here as well because it sums up my point far more concisely than I could:

Transparency can encourage people to be less honest so increasing deception and reducing reasons for trust: those who know that everything they say or write is to be made public may massage the truth. Public reports may underplay sensitive information; head teachers and employers may write blandly uninformative reports and references; evasive and uninformative statements may substitute for truth-telling. Demands for universal transparency are likely to encourage the evasions, hypocrisies and half-truths that we usually refer to as ‘political correctness’, but which might more forthrightly be called either ‘self- censorship’ or ‘deception.’

Then, when this deception is uncovered, the cries for transparency grow louder. After all, auditing has failed to uncover this deception, which means that people are still Getting Away With It. Trust has been eroded, and the situation is now worse. Trust has been eroded all along in this entire process, trust that people are doing what they are saying, and trust that people are doing it for the reasons they are proclaiming. Those audited feel now ever more justified in their deception, not least because of how little they are trusted to perform their job.

This makes the situation worse in more than just the obvious way: By giving more and more obligations to the audited for more auditing, more metrics, more data to capture, they now have less time to do what they were originally intended to do.

If the goal of auditing was to find underperformers and those with “too many” defects, they have now been found: Everyone audited, under perpetual pressure to provide so much data and transparency that the actual work has to be rushed. In this sense, it is a self-fulfilling prophecy acquired by what originally seemed like a good idea.

A Coda

I don’t have an answer for how to get out of this. Readers of Meadows will recognise this as a classic failure mode of a system with two actors and a shared goal: An arms race, or a ratchet. Increasing amounts of resources, time, and vitriol are spent on both sides in equal momentum. The result is embitterment and loss of all trust, but not victory or even any meaningful movement at all.

Meadows suggests the answer to this failure mode is unilateral disarmament and to reinvest the now-staggering bill of materials needed for this in other places. I suspect that might be the answer here as well, though I don’t see a way to convince those most involved in the perpetuation of auditng culture of this.

---