From Black Box to Transparency: Why Validation Matters in AI Insights

Overview

Black box AI is artificial intelligence whose inner mechanisms are obscured. Much of the AI that you interact with on a day to day basis is ‘black box.’ In some circumstances, this doesn’t really matter. It’s not necessarily important to understand the exact mechanisms which lead a deep language model to recommend you a brunch spot or output all its answers like a Victorian child. However, when black box AI systems are tasked with meaningful and impactful decisions, transparency can be of utmost import.

Takeaways

• In a black box AI model, only user inputs and model outputs are interpretable.
• Explainable AI (XAI) wraps around existing systems and attempts to explain what a model has done.
• Transparent AI reveals the inner workings of the model, uncovering the links between neurons and pathways and the concepts they are associated with.

The Turtles

Sea turtles of the genus Cheloniidae Chelonia first enter the world in the dead of night. They emerge from their eggs under a ceiling of claustrophobic sand and spend their earliest moments desperately tunneling towards freedom. Once they’ve escaped the sand, they scramble with unwavering intent towards the dark, crashing waves of the ocean.

Why do they do this?

They are newly born, unacquainted with thought, yet they immediately know to tunnel skywards and crawl towards the violent water that will be their home. Call it instinct if you will, but we don’t really know what’s going on in the minds of those turtlings.

You can think of the actions of a black box AI system in much the same way.

When a black box AI model provides you with an insight, even it can’t tell you how it got to that answer. The turtles know to tunnel up and crawl towards water, but they don’t really know how they know. Similarly, a black box model can’t be sure how it got to an answer.

Neurons and Layers

A deep language model, like the LLMs of ChatGPT or Claude Opus, is made up of tens of millions of neurons. These neurons are little computational functions, taking in a variety of parameters, weighing them, summing them, and returning a value. There are dozens of layers of these neurons, each taking inputs from the last layer before finally synthesizing an output from your input by converting the final number output into a word. This process repeats itself (with the words outputted by the LLM thrown back into the model for added context) until the number the function outputs is an “end” command to finalize the answer.

As outside observers, we can track some of what occurs within these LLMs. For example, Anthropic has been able to observe some neuron clusters at work (i.e. what neurons activate inside a deep language model in response to certain concepts), but much of what occurs inside of a deep language model has yet to be observed.

Essentially, we know technically how these models work, but we don’t fully understand the connections that a deep language model makes between neurons and layers. We know that if you ask a deep language model what color the sky is, it will say blue. But we don’t know what part of the model — what neurons, what connectors — led it to that answer.

Why is this important?

When Benedict Cumberbatch’s Sherlock Holmes catches a criminal, he’ll walk the audience through every step in his decision process. Mud on the boots the color of the dirt from the Champagne region of France, the indentation of a cork on the hand, blood alcohol content slightly above normal levels — this must be the Champagne Killer!

Every step in Sherlock’s uncovering of the criminal is tracked and explained. We can trust that when Sherlock presents us with his findings, they are correct, because he lets us in on his decision process.

When I ask an AI model to say, draw a picture of a bus, I can’t really trace why they might draw a New York Transit bus instead of a London Double Decker. Is it because I’m US-based? Were they trained on single decker buses only? Had I said something previously that indicated I’d not want a double decker (even though they are more fun.)

While this double decker scenario is low stakes, imagine black box systems used in place of Benedict Cumberbatch’s Sherlock Homes, accusing suspects without human-discernible reason. Or, maybe closer to home, imagine allocating resources to a business decision steered by a black box AI whose reasonings and decision-making capability are unmeasured.

Explainable AI and Transparency

So, Black Box AI is a problem. What's the solution?

Explainable AI (or XAI) attempts to demystify black box systems. LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) are two of the most popular XAI models. These models sit around Black Box models and track how different changes in stimuli correspond to different outcomes.

If we gave a machine learning model hooked up with SHAP or LIME data surrounding our suspects for the Champagne Killer, the model would pick our most likely suspect, and the SHAP and LIME frameworks would be able to pinpoint what parts of our input data resulted in the accusation of our killer.

Importantly, it should be noted that this is different from the Anthropic neuron-mapping which was referenced above. XAI like SHAP or LIME carefully monitors what goes into and comes out of models, but doesn’t really pull back the curtain to take a look inside like Anthropic’s neuron cluster findings.

Anthropic’s research is a step towards implementing real transparency in LLMs, not just explanations regarding how input affects output, but real understandings of the neurons and connections that make up these advanced AI systems.

Soulmates.ai and Explainability

When it comes to Soulmates.ai, fidelity is the main way in which we approach explainability.

In XAI, fidelity is a popular metric that measures the efficacy of XAI systems.

In behavioral digital twin scoring, fidelity is a different, yet parallel concept. Fidelity scoring in behavioral twin validation is used as a metric for how based-in-truth a model’s responses are when compared to the person it’s mapping.

While fidelity doesn’t attempt to uncover the nuanced thought patterns of models like XAI or Transparent AI, it provides an easy metric with which to measure behavioral twin answers by benchmarking how accurately a person's data was mapped by their twins.

To put it this way, I’d trust a 93% fidelity Sherlock Holmes digital twin’s thoughts on the Champagne killer’s identity over a 50% fidelity twin’s.

Where does this leave us?

As the world’s dependence on AI systems increases, the need for transparency and explainability increases. Anthropic’s research highlights that real transparency in LLMs is a possibility, but the science just isn’t there yet.

It’s easy to place trust in deep language AI. It’s sycophantic; it sounds right.

But blind trust can lead to dire consequences.

Transparency in complex LLMs doesn’t exist, but when making truly important decisions (like those surrounding your business), it’s highly encouraged that you seek explainability.

And who knows—first we may peer inside the Black Box AI systems, then we may seek to understand the thoughts of the turtles of Cheloniidae Chelonia.

FAQ

What happens if I ask an LLM to explain its answer? Is that explainable AI?

No. When an LLM answers a “why did you say that” question, it doesn’t perform any introspection as to what occurred computationally inside itself. It’s an ‘after-the-fact,’ what sounds best answer.

If AI is a human invention, how do we not know what occurs within it?

This is where the underlying AI concept of machine learning pops in. Neural networks aren’t coded in a traditional sense, but rather are grown.

When will we reach full transparency in advanced AI systems?

It’s impossible to say. Research is promising, but we’re not nearly there yet.