Microsoft's recent quarterly earnings report and AI productivity study have sparked conversations about the real-world impact of AI tools in the workplace.
While the tech giant claims impressive progress in AI services, their latest research on AI productivity raises questions about the effectiveness of current implementation strategies.
I broke down the earnings and research with Paul Rutzer, founder and CEO of the Marketing AI Institute, on episode 108 of The Artificial Intelligence Show.
The numbers game: Microsoft's AI Nemo
Microsoft's recent earnings presented a mixed picture.
On the one hand, the company beat expectations for overall revenue and earnings per share. Gross revenue grew by 21 percent year-on-year. And, the company noted that AI services added 8 percentage points to Azure and other cloud services revenue.
On the other hand, they fell short of cloud revenue expectations ($28.5 billion vs. an expected $28.7 billion).
That led to the stock falling slightly after earnings.
However, as Altimeter partner Jamin Ball pointed outthe AI piece of the business has some pretty compelling numbers.
The productivity puzzle: Microsoft's AI study
Now, at the same time as earnings, Microsoft also released a major study on AI productivity in the workplace.
The report is called Generative AI in Real-World Workplaces. It synthesizes findings from a number of recent Microsoft studies on the impact of creative AI in real-world work environments. And Microsoft calls it “the largest controlled study of productivity effects in real-world creative AI.”
Still, while you might expect some significant results from this type of study…
The results were a little, well, low. Microsoft found things like:
- Copilot users read 11% fewer emails.
- They spent 4% less time on emails.
- Users edited 10% more documents using the tool.
Roetzer expresses skepticism about the study's approach and findings.
“Is email really an interesting use case here?” Rutzer's Questions. “[The research] Clearly focusing on products and capabilities. [Copilot] enables. So they looked at appointments and emails and things like that. I feel like they're judging it against qualities that I don't find interesting.”
As Rutzer has personally seen in daily conversations, workshops, and interactions with customers and partners, there are many innovative use cases for creative AI within organizations today.
A wasted opportunity
Rutzer says part of the problem may be how Microsoft chose to structure its research.
- Limited scope: Research focused primarily on email and meeting metrics, ignoring more advanced use cases.
- Lack of training: There is no mention of user education or onboarding for Copilot.
- Uniform Applications: The study did not account for different roles or departments within organizations.
“If you're Microsoft and you want to show the value of Copilot, give it to 6,000 people in 60 organizations. [as one study did] And waiting to see if they sent fewer emails or if they spent less time in meetings, if that's your measure of whether or not they got value from Copilot, I think That you as Microsoft have a big problem on your hands,” argues Roetzer
The Big Picture: The True Potential of AI
While Microsoft's study may be short, it does Highlight a critical gap in current AI implementation strategies.
As companies invest heavily in AI tools, more creative and targeted approaches are needed to maximize their potential.
As AI tools increasingly permeate the workplace, companies must look beyond surface-level metrics and common use cases. The key to unlocking the true potential of AI lies in:
- Developing AI applications for specific roles and departments
- Investing in comprehensive training and education programs
- Establish clear standards and regularly measure impact
- Encouraging creative thinking about how AI can transform workflows.
“One has the opportunity to do the greatest work. [study] Who actually customized the use cases, trained people how to use the platform and benchmarked. [performance] Before and after,” Roetzer challenged.