AI Risk Management
HOW 12 ANGRY MEN SHAPED AN AI RISK-MANAGEMENT STRATEGY.
I would like to take you back to the year 1957
Many events happened during that year which have left a lasting impression on the world. For example, the European Economic Community (EEC) was founded with six countries signing the Treaty of Rome (Belgium, France, Italy, Luxembourg, the Netherlands and West Germany). The Soviet Union launched their Sputnik programme and effectively triggered the Space Race with the United States. Dr Seuss published “The Cat in the Hat” which is a story that I used to read at night to my own children many years later. The Frisbee was invented…eh… Ok, maybe let’s forget about the frisbee!
There was one event in 1957 that I specifically want to draw your attention to. United Artists released a movie, directed by Sidney Lumet, which was based on a 1954 play by Reginald Rose. That movie was called 12 Angry Men.
It was produced by Henry Fonda who also played the main character “Juror 8”. The story was set in a New York courtroom where a young man was on trial for killing his abusive father. The jury consisted of 12 men, (well, it was 1957!) and the judge gave them the instruction that the verdict must be unanimous as the prosecutor was seeking the death penalty. If there was any “Reasonable Doubt” then they were directed to find the defendant “Not Guilty”.
The movie moves to the Jury’s deliberation room where they take an initial vote. Everyone votes “Guilty” except for Juror 8 who wants more discussion before deciding to send a young man to the electric chair.
Juror 8 makes rational arguments keeps the discussion going eventually turning the others’ opinions around one by one. In the end, Juror 3, played by Lee J. Cobb, makes a final attempt to push for a “Guilty” verdict but fails, realising that his judgement is clouded by his relationship with his own son, and he breaks down in distress and changes his vote. Juror 8 helps him up and puts on his jacket and they find the defendant “Not Guilty”.
Travelling forward to "a number" of years ago
It was a cold and wet Thursday evening in October a number of years ago. I found myself attending an executive team meeting in a hotel meeting room. There were 12 of us in the room and the topic for discussion was the idea of using machine learning technology to lower the cost of providing services without feeling the need to inform the customer.
As the presentations went on, I could see that I appeared to be the only one in the room against the idea. I literally was Juror 8.
I tried to make rational arguments to convince everyone that not telling the customer about this change was not right. It was in my opinion an unsustainable strategy that lacked any sense of integrity. The argument being put back to me was that “The customer does not care how we produce the work, so why tell them?”.
I argued that of course the customer cares and that this strategy would destroy their trust and the company would from this point on be built on unsustainable revenues and margins that one day would collapse.
Unlike Juror 8, I failed to change anybody’s mind, which I found a little strange, and then it dawned on me that the purpose of this meeting was not to discuss implementing this strategy.
It was to rubber-stamp the fact that it was already implemented almost everywhere in the business, with very few exceptions.
I now had a clear choice to make. Would I shut up and just go along, as it would be the easy thing to do or, like Juror 8, would I stand-up for the customers with integrity and accept whatever consequences that would bring for myself?
I excused myself and left the meeting. I was not going to go along with a strategy that included hiding things from the customer. This was against everything that I believed in and very so soon after, I subsequently stopped all collaboration with that company.
Another time jump to last week
Last week, I read about the widely reported Deloitte AI scandal: (https://www.theguardian.com/australia-news/2025/oct/06/deloitte-to-pay-money-back-to-albanese-government-after-using-ai-in-440000-report) where they used AI, without disclosing it to the customer.
The goal was to generate very expensive reports for the Australian Government. However, the LLM(s) hallucinated randomly and introduced AI Generated errors, including a fabricated quote from a federal court judgment and references to non-existent academic research papers. This resulted in Deloitte having to refund some of the $440,000 Australian Dollars that they had charged for this report and a loss of reputation for their brand.
It would seem that history does in fact repeat itself as this was not so different a scenario as my own previous experience, only this time there was no Juror 8, it seems.
Although, we are definitely in a long AI Technology wave right now, it is very clear that using LLMs to generate multilingual content is not a nailed-on guarantee of accuracy, quality or reliability. In my own company, we have been testing LLMs specifically to see if they can generate multilingual content with the same degree of accuracy and quality as when we take original source language content and translate it using our own highly trained and customised AI Translation models. The answer is almost always no.
The LLMs do perform well enough when we have very generic content and in high-resource languages, but even there they still tend to ignore context, and they often hallucinate. From what we have seen in our ongoing studies, they disregard some ideas from one language to the next, they sometimes guess at concepts differently in different languages, and they do (still) make basic errors.
Our findings are very similar to those already published is studies such as:
Managing risk in the AI world
What is very clear to me is that using AI it must be accompanied with a properly thought-out risk management strategy.
Risk-management must have full transparency, trust and most of all integrity.
If your suppliers/partners are hiding something in their process, pretending that their solution is better than it really is or they simply don’t really care about the risk that you are taking then they are the wrong partners to have on an AI journey with you.
It is better to work with a partner who wants to help you to achieve your goals and will help you mitigate the risks with integrity even if it means less revenue for themselves.
When facing risk there are four known strategies to manage that risk as follows:
-
Accept – In other words do it anyway.
-
Avoid – In other words just don’t do it.
-
Transfer – This is the laziest option because although it might absolve a company of direct liability, it doesn’t stop the risk from potentially happening.
-
Minimise – Reduce the risk factor to get it to an acceptable level.
We almost always recommend option 4 and let me give you a current example.
We are working with a number of clients who are experimenting with using LLMs to generate multi-lingual marketing content. Given what we already know about LLMs and their current limitations outside of English, (where 90%+ of their training data comes from), we almost always recommend an approach to minimise the risks.
Our approach is to use a LLM to generate the source content (usually English) and then we combine this with our custom-trained AI Translation models (plus optional human post-editing, if the customer wants this) to get to local market-ready content.
This process is fully transparent and brings the customer the best results in our experience.
My conclusion from “12 Angry Men”
Most companies have three thought-dimensions when it comes to multilingual AI.
-
Commercial perspective – Help by senior managers and C-Level execs. They want AI to save cost and reduce headcount, and they want it yesterday!
-
Engineering perspective – They love AI and want to keep trying new things to get that edge that the company demands.
-
Linguistic perspective – This is where the belief in humans being the “Gold Standard” for language comes in. They want to resist AI or at least slow it down by adding layers and layers of “QA checking” to make sure it is ok.
The linguistic perspective is clearly waning in many companies, and while that is entirely up to each company to determine how it wants these perspectives to balance out, it does open the company up to a higher risk profile.
The commercial and Engineering perspectives want AI to work so badly that they sometimes do not put in place an appropriate risk-management strategy.
At XTENSOS, we believe in integrity. It is one of our four core values, and we absolutely believe that a proper risk-management strategy must have full trust in the partner, open-kimono transparency and a huge amount of integrity.
We try to stay true to our values and bring that integrity to our customers by helping them to achieve their AI cost-reduction goals while, also, minimising and managing their risks.
We do this regardless of the consequences to ourselves in terms of cannibalising historical revenues because we know that it is the right thing to do for our customers.
And yes, you guessed it, we arrived at our approach under the heavy integrity-based influence of Juror 8 in 12 Angry Men.
