Microsoft automates Copilot evaluation datasets, estimates $300,000 in savings

Microsoft says it has deployed a system for generating synthetic enterprise data as a standardized Copilot evaluation baseline in an internal shared catalog. It estimates savings of approximately $100,000 per evaluation dataset and $300,000 in total project savings, replacing a multi-day manual process with automated runs. The system creates simulated workplaces with hundreds of users, communication histories and industry-specific workflows across pharmaceutical, capital markets and IT/software environments. Public occupational data, job postings and SEC filings inform the simulations; no customer data enters the pipeline. Automated tools generate and validate questions against those environments. The reported 39% quality improvement measures synthetic data against Microsoft's initial generation approach, rather than Copilot's performance.