Artificial Intelligence and the Expanding Black Box: Do We Need More AI Accountability, Risk, and Transparency?

Technology has always been built on an “if-then” framework. The logic was fully readable and the decision path was fully traceable, but then artificial intelligence emerged and “if-then” faded into history.

AI models don’t need a programmer’s “if-then” to move from input to output. They learn patterns and carry out processes that are based on probabilities. It’s next-level power, but it comes with a next-level problem: AI’s black box.

For AI users, the trade-off for more speed, automation, and analysis is less accountability. AI provides an answer, but the explanation for how it got there is concealed within an ever-growing black box of models, data, prompts, integrations, and automated decision-making processes.  The more complex AI technologies become, the harder it is to understand how a model reaches its answer, and the harder it is to establish an accountability framework.

“The black box problem is certainly getting worse as the systems get more layered,” says Jared Navarre, Founder of Keyni Consulting and CEO of ONNIX. “It’s not just one model anymore; it’s LLMs, agents, memory, APIs, permissions, and workflows all getting stitched together. That’s not a black box. That’s a black box with plumbing.”

Navarre is a creative strategist with multiple successes launching, scaling, and exiting ventures across entertainment, logistics, IT, and service industries. His work includes consulting with over 250 businesses on designing resilient technology infrastructures and developing multi-platform brand ecosystems that resonate with both niche and mainstream audiences. Navarre is also chairman of the humanitarian NGOs IN-Fire and Project AK-47 and the creator of ZILLION, an immersive music project that blends narrative, multimedia, and live performance into a cohesive storytelling experience.

AI accountability keeps bad decisions from becoming a corporate crisis

For developers, the emerging challenge is developing powerful AI that is also responsible. When the stakes are low, the lack of transparency is easy to ignore.

Someone watching Netflix probably won’t be concerned if AI can’t explain how it determined which movie they should watch next, but a bank customer applying for a loan will most likely want to know the reasoning behind an AI-powered denial. A doctor will want to know what it is about the MRI results that indicate a patient has cancer.

That’s the point where AI’s transparency problems become an accountability problem for the company using it.

“The risk is accountability,” Navarre says. “If AI makes a bad call in a high-stakes setting, that’s a big problem. But if nobody can explain why it happened, what data influenced it, or who was responsible, that’s a massive problem.”

The growing concern over accountability in AI has led to the introduction of explainable AI (XAI), an innovation aimed at shattering the black box and ensuring accountability. XAI addresses accountability issues either by simplifying the process or by reverse-engineering an explanation after an output has been provided.

The simplification method, which is sometimes referred to as working with a “white box,” gives human users a process that is easier to understand. By prioritizing interpretability over processing power, the white box approach can limit capabilities.

Reverse engineering AI decision-making doesn’t limit the model’s power, and it doesn’t necessarily provide the clear accountability needed in high-stakes applications. Complex models operate on complex processes. Finding a way to explain those processes to humans may increase comfort, but it doesn’t guarantee that accountability gaps have been addressed.

“Explainable AI helps, but with LLMs it’s often not a true audit trail,” Navarre explains. “Sometimes, it’s just the model giving you a very confident-sounding excuse.”

Functional transparency moves companies toward more effective AI governance

Companies awaiting an innovation that will provide them with complete access to AI that offers full transparency may have a long wait. Stanford University’s Human Centered Artificial Intelligence program recently reported that transparency has been on the decline as AI use has increased. And other experts say explainable AI is a “dangerous illusion.”

Still, there are measures companies can implement to ensure accountability. The key is implementing a governance framework that increases awareness at every step of the process.

“I don’t think full transparency is realistic, but functional transparency is — and should be a requirement,” Navarre says. “Companies need to know what went in, what came out, what tools were used, what actions were taken, and where human accountability sits.”

Companies want more automation and efficiency, but they can’t afford to adopt AI systems that derail their risk management practices. If AI is going to be used in ways that create more risk, accountability policies need to acknowledge, track, and address it in ways that demonstrate to stakeholders and regulators that the company isn’t acting recklessly.

“The goal isn’t to eliminate every black box,” Navarre explains. “It’s to make sure the black box doesn’t quietly become the boss.”