How Monzo runs data-driven AI experimentation

Engineering Enablement Podcast

0:00

-41:18

How Monzo runs data-driven AI experimentation

Monzo’s Fabien Deshayes shares how the bank is adopting AI responsibly—running structured trials, measuring impact, and balancing innovation with control.

Abi Noda

Oct 31, 2025

Transcript

Listen and watch now on YouTube, Apple, and Spotify.

In this episode of Engineering Enablement, I talk with Fabien Deshayes, who leads multiple platform engineering teams at Monzo Bank. Fabien shares how Monzo is adopting AI responsibly within a highly regulated industry, balancing innovation with control, structure, and strong data-driven decision-making.

We discuss how Monzo runs structured AI trials, measures adoption and satisfaction, and uses metrics to guide investment and training. Fabien explains why they moved from broad rollouts to small, focused cohorts, how they are addressing existing PR review bottlenecks that AI has intensified, and what they have learned from empowering product managers and designers to use AI tools directly.

He also shares insights into budgeting and experimentation, the results they are seeing from AI-assisted engineering, and his vision for what is next, from agent orchestration to more seamless collaboration across roles.

Some takeaways:

Evaluating AI in a regulated industry

Monzo treats AI enablement as a platform responsibility. The platform group centralizes tooling to ensure guardrails are in place for privacy, data access, and model usage across regulated domains.
Structure comes before scale. Before expanding access, Monzo limits tools to trained users and controlled contexts to reduce risk while maintaining speed of innovation.
Regulation became an advantage. The constraints of banking forced deliberate decision-making, leading to a tighter focus, better change management, and stronger adoption.

Experimentation strategy

AI adoption began with GitHub Copilot but lacked direction. Early usage was inconsistent, prompting Monzo to create structured trials with clear evaluation goals.
Each trial follows a defined process. The team selects one or two tools at a time, identifies specific use cases, and measures success against defined metrics such as adoption, satisfaction, and retention.
Trials are intentionally small and focused. Monzo learned that large-scale rollouts dilute feedback and waste time. Smaller cohorts of engineers with defined use cases yield higher-quality insights.

Evaluation criteria and metrics

Success is measured across ten to fifteen criteria. These include weekly and monthly active users, retention, percentage of AI-written code, suggestion acceptance rate, satisfaction, and cost.
Use-case analysis is part of every trial. Tools are compared across workflows like refactoring, documentation, migrations, and new feature creation to understand strengths and weaknesses.
Quantitative data is paired with survey feedback. Combining usage telemetry with engineer sentiment helps Monzo decide whether to scale a tool or end a trial.

Budgeting and cost control

Monzo sets clear spending limits before experimentation. After modeling expected usage, the team landed on a target of roughly $1,000 per engineer per year for AI tooling.
Budget discussions include leadership from the start. Transparent alignment with the CTO and finance ensures experimentation can continue without surprise costs.
Cost is balanced against value. Data from tool providers and internal telemetry validate ROI, while quarterly reviews adjust allocations based on usage trends.

Engineering outcomes

AI is increasing throughput and code volume. Pull requests per engineer are up 10–20%, with the average PR size growing by a similar amount. Around 20% of new code is now AI-generated.
Code review time is the new bottleneck. Larger, AI-assisted PRs take longer to review, creating a productivity drag that Monzo is actively addressing.
Preview environments are solving review delays. New “tenancies” allow engineers to test changes in isolated environments, improving speed and stability across teams.

Cross-functional adoption

AI use is spreading beyond engineering. Product managers and designers now prototype directly using AI tools integrated with Monzo’s design system.
Better context leads to better results. By exposing design components and brand guidelines to models, teams produce higher-quality prototypes that reduce rework later in development.

Leadership and enablement

Community drives adoption more than mandates. AI Engineering Champions across departments test tools, share learnings, and help others apply best practices in their collectives.
Upskilling is Monzo’s next investment. With only half of engineers feeling confident using AI, the focus is shifting to prompting education, internal workshops, and structured learning.
Success stories accelerate momentum. Highlighting early wins through Slack channels and company meetings keeps engineers engaged and inspired to experiment.