Assessing the Criticality of AI Failures
What is the criticality of a bad prediction from your algorithm? In my opinion this is one of the first questions to ask before considering a generative AI or AI project in general.
It's encouraging to see public knowledge about AI progressing. However, I think some aspects still require awareness-building, like choosing and framing use cases.
With the emergence of generative AI services (ChatGPT, MidJourney, etc.) and their democratization, business ideas are blossoming. I sometimes hear project ideas or startup ideas around me that make me wince because they don't take into account cases that could go wrong.
Machine learning solutions come from statistical methods, so a prediction will never be 100% reliable. What’s worse, in some cases they’re not even deterministic – they can vary from one run to another. We have to work with that and therefore choose use cases accordingly.
With that in mind, a question that naturally arises is: when the algorithm is wrong, what will the consequences be?
For a programming assistant solution like Github Copilot, criticality is almost zero. If the code suggestion is bad, it’s not serious, you’re there to correct it, there’s a human in the loop.
For an autonomous vehicle solution, a failure can be very critical – in the worst case, it can take a life.
In any case, whatever the project, I think it's always healthy to think about cases that can go wrong and their consequences. What happens if my solution discriminates against users? If it misinforms because of hallucinations? If it generates offensive content? Dangerous content? If it gives access to sensitive information? If it causes users to lose money?
All this to say that if the slightest prediction error is critical for your business, AI is probably not the right path.
In my opinion, the "good" use cases are those that tolerate a certain error rate and for which failures do not have serious consequences. In particular, use cases with a "human in the loop" are generally relatively safe.