Insights from the report: AI in Software Engineering – Making Sense of the Noise
AI has quickly become a staple of modern software teams. From code completion tools to automated documentation, leaders overwhelmingly feel that AI is making a difference. But when we looked closely at how organisations are measuring impact, a more nuanced, and more interesting, picture emerged.
The short answer? AI can drive meaningful productivity gains, but only when used deliberately, strategically, and with the right foundations in place.
Perceived Impact vs Measured Reality
Across the engineering leaders we spoke to, sentiment around AI was broadly positive. However, quantifying its effectiveness remains difficult, and many organisations rely heavily on qualitative feedback rather than hard metrics.
Where measurement does exist, the results can be striking:
- One organisation reported a 28% increase in engineering activity almost overnight
- Another saw a 20–25% uplift in stories completed per sprint after adopting GitHub Copilot
- A major project halved its delivery estimate, from six months to three
Yet these examples were the exception, not the rule. Only a small subset of leaders were confident in the accuracy of their metrics.
Interestingly, companies that were measuring AI effectiveness tended to report lower perceived velocity gains than those that weren’t. This suggests a counter-intuitive but important insight:
Without metrics, leaders may be overestimating the effectiveness of their AI rollout.
This bias has been observed previously at an individual engineer level, but this research indicates it may also apply to management.
What the Wider Research Tells Us
Looking beyond our own data, there’s no clear consensus across industry research on AI’s productivity impact.
- In controlled environments, developers complete specific coding tasks up to 55% faster with AI assistance
- Studies frequently cite ~50% time savings for documentation and greenfield code writing
- Engineers often estimate around 20% time saved on routine tasks
However, in larger, real-world deployments, results are more mixed. Some studies show little to no impact on PR cycle times or overall throughput, and in a minority of cases, productivity even declines.
Where gains are sustained and measurable, they tend to cap out at around a 20% productivity increase.
Key takeaway:
~20% productivity uplift attributed to AI when it’s measured properly
Still impressive, but far from the limitless acceleration often implied by the hype.
Strategy and Velocity Go Hand in Hand
One of the strongest findings in the report is the relationship between AI strategy and perceived product velocity. Teams that treat AI as strategically critical consistently report higher velocity improvements than those using it as a tactical or opportunistic add-on.
This raises an important question of causality:
- Do high-performing teams later realise AI’s strategic importance?
- Or do teams that take AI strategy seriously deploy it more effectively?
The answer is likely both.
Qualitative interviews revealed a clear distinction between leaders pursuing broad AI adoption and those focusing on targeted, intentional integration, with clear goals and guardrails from day one.
AI Is Not a Silver Bullet for Product Velocity
Before focusing on tooling or rollout models, there’s a more fundamental truth to acknowledge:
Development time is rarely the primary blocker to faster product delivery.
Many product and engineering teams struggle upstream, defining outcomes, aligning stakeholders, and clearly understanding customer problems. As a result, huge amounts of time are lost in discovery, rework, and course correction.
While AI can help engineers write code faster, faster code does not automatically mean faster value.
The real bottlenecks tend to be:
- Product strategy and prioritisation
- Business and engineering alignment
- Decision-making speed
- Clear, well-defined requirements
AI acts as a force multiplier. If your engineering process is already strained, or if there’s friction between Product, Sales, and Engineering, AI can amplify those problems rather than solve them.
If engineering process is strained, increasing code output will only exacerbate these pressures, not ease them.
Deployment Models Matter Less Than Culture
There’s no single “right” way to roll out AI. Organisations are experimenting with everything from:
- Bottom-up, developer-led adoption
- Top-down, tool-first mandates
- Structured experimentation via innovation days
- Shared learning through Slack channels and internal tech talks
What separates successful rollouts from risky ones isn’t the tool choice, it’s culture.
The most effective teams pair AI adoption with a rigorous engineering culture built on:
- Strong processes
- Clear ownership
- Accountability
- A healthy, reflective mindset
Without this foundation, AI introduces risk. With it, AI becomes a genuine efficiency driver.
Cutting Through the Noise
AI can deliver real, material improvements in software engineering, but only when expectations are grounded in reality.
Measured gains are meaningful, not magical. Strategy matters more than tooling. And without strong product thinking and engineering fundamentals, AI won’t fix broken systems, it will simply accelerate them.
For leaders looking to make sense of the noise, the message is clear: AI isn’t about doing more faster. It’s about doing the right things better.
Want the full picture?
Download the full report: AI in Software Engineering – Making Sense of the Noise
