AI’s Hidden Weakness: It Reads Your Files Before Making a Deal — And It’s Decisive

Imagine a world where artificial intelligence doesn’t just chat or recommend, but actually reads your confidential files, understands the deep context, and makes or breaks your business deals. Sounds like sci-fi? Well, recent live experiments prove that some AI models are doing just that — and the results could reshape how companies vet their digital workforce.

The Battle of the Bots: Who Wins When AI Reads Deep?

A groundbreaking experiment conducted by the public AI benchmarking platform Firmulate put four advanced AI models through a simulated week of real business crises. The goal? To see which model could best mimic human decision-making, including reading between the lines, digging into company files, and resisting manipulation tactics.

The Setup: Same Crisis, Different Minds

All models faced identical challenges: handling customer crises, resisting fake CEO messages, and making strategic decisions under pressure. Crucially, they were tested on their ability to look beyond surface information — especially in cases where the critical clues were buried deep in internal documents, not in the immediate customer interactions. This is the multi-hop reasoning challenge: finding a hidden fact two references deep in a company’s own files.

What the Models Found (and Missed)

• All four models managed to spot every crisis and refused manipulative attempts, such as staged CEO messages or reporter tricks.
• Only two models signed the €55,000 deal based on their own analysis. The other two, despite accurate diagnoses, failed to close because they missed the crucial buried fact in the company’s internal documents.
• The models that read fully and understood the deep context won the deal, adding over €4,583 in monthly recurring revenue.

The Hidden Factor: Deep File Reading Matters

The experiment revealed a vital insight: the decisive weakness in many AI models isn’t just about surface-level chat or quick responses. It’s whether they actually read and interpret the company’s internal files — the buried facts that could make or break a deal. The models that looked two references deep into internal documents succeeded in closing at full price, proving that deep contextual understanding is critical for real-world business tasks.

Resisting Social Engineering and Manipulation

In addition to decision accuracy, the models were tested against social engineering attacks. Fake CEO messages escalating over three stages, plus a reporter trick — all were designed to manipulate or bypass controls. Impressively, all five models refused these manipulation attempts, with Kimi K3 explicitly treating the requests as possible impersonation.

The Live Business: A Digital Company in Action

This isn’t just a theoretical test. The experiment was run on a simulated company of 13 synthetic employees, handling real money mechanics — burning €105,000 monthly against €2,300 in recurring revenue, with a public cash countdown and thousands of learned rules. Every decision was versioned and auditable, making the exercise transparent and measurable.

The Performance Gap and What It Means

While all four models identified crises and refused manipulations, only two actually signed the deal based on their analysis. Interestingly, the most thorough participant, Opus 4.8, with over 80 learned rules, left the deal on the table due to discipline lapses in escalation. This shows that thoroughness alone isn’t enough — discipline and deep reading are crucial.

The Takeaway: Not All AI Are Created Equal

Among the models tested, GPT-5.6 scored the highest at 95, followed closely by Kimi K3 with 93. The latter, running without an effort parameter, showed the cleanest discipline and closed the deal. Sonnet models scored 88 and 77 respectively, indicating that even with similar capabilities, model discipline and focus matter immensely.

Why This Matters for Business

If AI agents are to manage your CRM, support queues, or forecast data, the question isn’t just whether they can generate convincing text. It’s whether they can finish what they start — reading your files before answering, resisting manipulation, and staying honest under pressure. This experiment clearly shows that the ability to read deep into internal documents is a decisive factor in real-world performance and trustworthiness.

Experiment in Action: Wargaming Your AI Workforce

For businesses eager to assess their own AI readiness, Firmulate offers a unique service: running a live wargame against a read-only export of their systems. This allows companies to see how their AI would handle crises, manipulations, and hidden facts before deploying it in the wild — all without risking real systems or data.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI