Smashing the black box: why glass box AI will rewrite the rules

We are now at the stage where the threat landscape is changing faster than ever before. A report from Accenture made one thing very clear: AI adoption is accelerating, and the speed, scale, and sophistication of attacks now are far beyond what most enterprise defences were originally built to handle.

Accenture found that 90% of organisations aren’t ready for AI driven attacks, and this statistic isn’t confined to critical infrastructure or nation-state targets. It’s every sector, every type of business model.

A great example of this is Anthropic’s recent announcement of the first large-scale AI-driven cyberattack, which is not a one-off, it’s an early warning. The direction of travel is clear, AI is becoming the engine behind many intrusion campaigns from threat actors. This is because it’s making attacks more autonomous, more adaptive, and harder to detect.

In this new era of AI-vs-AI conflict organisations need to remember that as useful as AI is in threat detection, blind trust in its usually opaque systems is a liability. Verifiable AI is the only solution to keep up with the ever evolving threat landscape.

Offense is scaling faster than the defense

The reason we’re now seeing more of this AI-vs-AI dynamic in cybersecurity is because traditional human-centric defences simply cannot keep pace with the speed and sophistication of modern threats. Attackers are using AI to automate code generation, adapt tactics in real time, and scale operations far beyond what manual processes can manage.

This shift into incorporating AI into cybersecurity isn’t about replacing humans entirely. Human judgement remains paramount, and although there is a lot of industry concern around AI supplanting security analysts, the reality is that AI makes mistakes. Mistakes which are especially hard to spot in traditional black box AI. AI should always augment, and not remove, the human element in cyber defences.  

So what is the best defense?

The most effective cyber defense combines transparent AI’s speed and scale with human judgment and context. Traditional AI can automate repetitive triage and surface actionable insights from vast data, but it cannot interpret nuance or ethical context the way a trained analyst can.

However, when analysts are provided with verifiable evidence rather than opaque suggestions, investigations that once took hours can often be resolved in minutes. This evidence-first approach in AI not only speeds up response times but also aids in the transfer of experience.

Junior analysts gain immediate context equivalent to that of seasoned professionals because the reasoning behind the AI’s findings is transparent and traceable.

Defenses that remain static or use opaque AI will inevitably fall behind. To avoid the AI hallucinations, it's important not to rely on simply “trust the machine” and instead equip them with clear, explainable evidence, or ‘verifiable AI’ that empowers informed, confident decision-making. This is the true foundation of resilient AI powered cyber defense.

Verifiable AI, the way forward

To understand Verifiable AI, it helps to think in terms of “black box” versus “glass box” systems.

Fundamentally, the difference is structural, “traditional” black-box AI produces probability-based outputs from large volumes of data, but it does not show how it reached its conclusions. This differs from, glass-box, or verifiable, AI which delivers deterministic outcomes grounded in clear and traceable evidence.

For example, say you had a verifiable AI system, it may flag that one of your business laptops is behaving oddly. In this scenario, instead of simply issuing a risk score, it would also present the exact SQL query executed, the telemetry examined, and the specific data rows that triggered the alert, effectively providing a reasoning receipt.

This creates a system where every conclusion is accompanied by a documented chain of evidence and logic. If an analyst, or regulator asks why an action was taken, the answer is not just “the model assessed this as risky,” but actually includes a clear, human-readable trail of machine reasoning that can be independently validated.

The reality is that, unfortunately, most security AI today still operates as a black box. It generates scores, classifications, and recommendations without revealing the logic behind them. That opacity was manageable when AI tools were merely assistive. But as AI systems take on more autonomous operational responsibility - and as regulation, liability, and accountability pressures increase - opaque decision-making becomes a significant risk. In a threat landscape where AI-driven attacks are scaling rapidly, defensive AI cannot afford to be unexplainable.

Where is all the Verifiable AI?

The limited adoption of Verifiable AI has not been a matter of unwillingness from vendors, but more an issue of technical architecture.

Most security platforms are built from layers of acquired tools, they could have one data model for cloud, another for endpoints, another for identity. These components are often connected through APIs rather than deeply integrated. When AI operates across such fragmented systems, it is effectively reasoning across multiple siloed “data sources” which leads to “educated guesses” from AI rather than a clear cut investigation.

Achieving verifiability requires the difficult, foundational work of building a unified ontology, a shared data language spanning AWS, Azure, GCP, on-prem systems, and endpoints alike. Only with that structural coherence can AI move beyond probabilistic guesswork and instead perform evidence-based analysis with outputs that are fully traceable.

As AI-enabled attacks grow in speed and autonomy, defense must also be AI-driven, but not blindly so. Organisations need verifiable AI that exposes the reasoning behind its conclusions, but human analysts still remain essential. Their judgment, experience, and contextual understanding cannot be replaced. But when paired with transparent, evidence-based AI, their effectiveness scales dramatically, creating a defense that is capable and ready to meet AI-driven threats.

‍