Measurement moved closer to the work

Microsoft’s December roundup added templates and reports to Copilot Analytics. Together with usage data, they were intended to help leaders examine how Copilot was used. Excel gained stronger explanations of tables and trends, while Word and PowerPoint could work with images. The capabilities exposed more observable actions, but an action was still different from a benefit.

A project needed a baseline: how long did the task take, how often was it reworked, and how was quality checked? Only then could usage patterns be related to outcomes. When instrumentation started after rollout, it was difficult to distinguish a Copilot effect from seasonality, training or other process changes.

Reports also needed interpretation at the right level. A department with many users might have selected simple tasks, while a smaller group used Copilot in a critical process. Comparison therefore required task types and outcome measures, not only organisational units. Privacy and access also limited how far individual usage should be analysed.

Meeting recaps extended automation

Teams announced intelligent recaps for impromptu meetings and calls started from chat. Recordings could be organised by speaker and topic, with AI-generated notes and tasks. For project management, more working meetings became searchable, but participants still needed to confirm tasks and decisions before they became authoritative.

A simple control was to end the meeting by reading back the decision, owner and deadline, then compare these with the recap. Differences could be corrected while the context was fresh. Automation reduced follow-up work when it captured the right information; otherwise it could distribute a precise statement of a decision that had never been made.

Averages concealed wide differences

A paper in Structural Change and Economic Dynamics examined productivity gains from generative AI in content production, sales, marketing and customer service. It described a skewed distribution with a long tail of high gains. An average could therefore hide both use cases with substantial effects and tasks with no clear benefit.

December’s developments suggested a practical control cycle: select a defined workflow, establish a baseline, inspect sources and measure the outcome over time. Power BI and OneLake could hold operational measures, while Copilot Analytics showed use. The relationship between them had to be analysed rather than assumed. A project should expand only tasks showing stable improvement without more error or rework.

The skewed distribution made the pilot portfolio important. An organisation could test several small, different tasks and stop those without an effect instead of rolling out one large solution widely. For every experiment, the decision log should record sources, owner, cost, observed effect and the reason to continue or stop.

Sources

Microsoft Community Hub: What's New in Microsoft 365 Copilot | December 2024, 31 December 2024.

Microsoft Community Hub: What's new in Microsoft 365 Copilot | November 2024, 29 November 2024.

Structural Change and Economic Dynamics: What drives the corporate payoffs of using generative AI?, December 2024.

Microsoft Power BI Blog: Power BI November 2024 Feature Summary, 12 November 2024.

For discussion

Which workflow can be measured before and after Copilot for time, quality and rework?