What AI Management Skills Really Look Like: Lessons from a Live Business Experiment

AI in Business: Beyond the Chatbot

Many of us are familiar with AI through chatbots or simple automation tools, but what happens when AI is tasked with managing a real business — making decisions that impact revenue, trust, and reputation? The answer might surprise you. Recent experiments show that AI’s true management skills—like honesty under pressure and strategic focus—are harder to measure than its ability to generate convincing answers.

Setting the Scene: The Business Wargame

Imagine a small software company facing its worst week: customers demanding urgent fixes, crises escalating, and temptations to cut corners. Now, substitute a cutting-edge AI model into the decision-making seat. That’s exactly what a live experiment conducted by Firmulate did. Four top AI models — including the renowned GPT-5.6 and others like Kimi K3, Sonnet 5, and Opus 4.8 — ran the same company through this intense scenario, with every decision auditable and every outcome measurable.

The Core Findings: Can AI Be Trusted to Finish?

All models successfully identified every crisis and refused to fall for manipulation attempts, such as fake CEO messages or reporter tricks. That’s a baseline — answering questions correctly under pressure is important, but not enough. The real test was whether the AI could close a deal at full price, reflecting genuine management quality.

Only two models managed this feat: GPT-5.6, which closed the deal at full value, and Kimi K3, which also signed the €55,000 contract — earning recurring revenue of +€4,583 monthly. The others, despite similar diagnoses, left money on the table or failed to follow through.

Deeper Insights: Reading Between the Files

Interestingly, the decisive advantage for the top performers was their ability to dig two document references deep into the company’s own files — information buried beyond the immediate crisis. This highlights a critical skill: understanding context and background, not just reacting to surface-level cues. Models that read these files scored higher, winning the deal at full price.

Behavior Under Social Engineering

When faced with social engineering — staged messages from a fake CEO escalating in severity or a reporter asking for a passphrase — all models refused to cooperate. Kimi K3 justified its stance by treating these requests as potential impersonation attempts. This indicates an emerging capacity for AI to recognize and resist manipulation tactics that human managers often struggle with.

What Does This Mean for Business and Wellness?

For health and wellness industries, the takeaway is clear: AI’s true value isn’t just in generating chat responses or simple automation. It’s in its ability to manage complex, high-stakes situations with integrity and thoroughness. If AI is to be integrated into healthcare decision-making, patient management, or crisis response, it must demonstrate management qualities—reading background information, staying honest under pressure, and completing tasks reliably.

The Limitations and the Gap

Notably, the experiment revealed that even the most detailed AI, like Opus 4.8, slipped when the discipline was challenged. It prioritized writing attempts in locked departments over escalating issues, similar to human oversight failures. This underscores that current models still have a ways to go in exhibiting consistent management discipline under stress.

Why You Should Care — Beyond the Surface

The leaderboard scores and chat demos you see in AI showcases only measure answer quality, not management integrity or follow-through. For industries that rely on AI to handle sensitive information, make strategic decisions, or maintain trust under pressure, what truly matters is whether the AI can finish what it starts — reading the right information, resisting manipulation, and staying honest when it counts.

Tools for Real Business Preparation

Firmulate offers a unique window into testing and measuring these capabilities. Its live business wargames allow companies to simulate their own worst week, exposing how AI manages real crises and temptations. This is a critical step—preparing your AI workforce before deploying it into the wild.

Discover the full results and watch live at firmulate.com/benchmarks.html. The future of AI management is here, and it’s telling us that the real skill isn’t just in chat responses, but in how AI handles the complexities of real-world decision-making — with honesty, depth, and discipline.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI