In a rapidly evolving AI landscape, the most successful organizations are those that pair technological adoption with disciplined experimentation. A recent CIO ThinkTank roundtable showed a growing agreement among senior IT leaders. They discussed how AI-powered agents are changing business operations and competition. The talk focused on important topics like AI testing in companies and how AI agents help with digital transformation.
The timing for this conversation couldn't be more critical. Enterprise AI adoption has accelerated dramatically, with McKinsey sharing that 65% of companies are using generative AI in early 2024—nearly double the rate from just ten months prior. This momentum continues to build, with analysts projecting AI will consume almost 20% of tech budgets in 2025, and the share of enterprises using generative AI applications expected to jump from 11% to 42% in just one year.
This article summarizes important insights from discussions about AI Agents and AI experimentation in businesses. It focuses on four key areas that will influence IT strategy in 2025 and beyond. One of these areas is how to decide between building or buying solutions.
For those interested in diving into the other articles in this CIO ThinkTank series, they have been listed here for your convenience:
Keep reading to learn how to balance experimentation with strategic governance.
The first major discussion point among CIOs, CTOs, and CISOs wasn't whether to build AI solutions in-house or buy them off the shelf—but rather how to develop decision frameworks that determine the right approach for each use case. AI agent use cases are central to these discussions, helping IT teams decide when to build and when to buy based on their business needs.
Many organizations are finding that hackathons and internal innovation programs serve as valuable proving grounds for in-house talent and ideas. Over 80% of Fortune 100 companies conduct hackathons to drive innovation, with nearly 50% of hackathons being recurring events and the amount of focus on AI in digital transformation scenarios in these hackathons continues to rapidly climb.
Over half of the leading innovative companies we work with have run internal AI-focused or specialized hackathons to spur experimentation. These events allow IT teams to prototype AI agent use cases in a low-risk environment, revealing whether the organization has the skills to build solutions internally.
This mix of top-down planning and bottom-up discovery helps IT leaders find a good balance. It pushes their limits while driving success in enterprise AI strategy. IT executives at a roundtable shared various methods. They talked about prompt-a-thons that develop into full automation-a-thons. These activities combine the energy of a Hackathon with expert guidance from the industry.
When evaluating build-vs-buy decisions, CIOs emphasized frameworks that prioritize:
Industry data supports this approach—enterprises selecting AI tools prioritize measurable ROI (30%) and industry-specific customization (26%) far above lowest price (only ~1%). This may be part of why there was a consensus with IT executives at the roundtable that build wasn’t just necessary but was often a leading preference during this period of AI in digital transformation.
While build might be a noted preference, it’s also clear that a hybrid strategy often delivers optimal results. For example, based on discussions at our CIO ThinkTank and roundtables:
One global bank built a custom AI model in-house for proprietary trading algorithms (a competitive differentiator). The same bank purchased established solutions for more common capabilities like document summarization.
This mix-and-match approach uses the best of both worlds. Build internally when you want a competitive edge. Buy a ready-made product when it can help you get there faster.
As technology platforms evolve, even the traditional boundaries between building and buying are beginning to dissolve.
Modern AI platforms are increasingly blurring the line between build and buy. Microsoft's Copilot Studio provides a low-code canvas for building custom AI agents with workflow automation. Similarly, Azure AI Foundry offers a unified environment to develop and deploy generative AI models, allowing organizations to plug in pre-built components while infusing their own data and logic.
These platforms enable IT teams to focus on tailoring AI to specific business needs rather than reinventing fundamental capabilities. The strategic blend of building and buying, guided by clear priorities, has emerged as best practice among the roundtable participants.
A recurring theme in the discussion was the difficulty of moving from promising AI pilots to production-scale implementations. This challenge is widespread—according to Boston Consulting Group, 74% of companies struggle to move AI projects from pilot to production, with only about 26% developing the necessary capabilities to generate tangible value.
The roundtable quickly identified data readiness as the critical success factor in scaling AI initiatives. Without interoperable systems and well-structured data, even the most advanced AI will stumble.
The numbers validate this experience:
Interestingly while AI drives legacy modernization and migration multiple IT executives shared at the event that they struggle with finding the capacity or expertise to accelerate some of these migrations while tackling other IT challenges and AI transformation priorities. Perhaps this is why many cited file shares or legacy content systems as a challenge and why it is always helpful to leverage industry best practices such as those outlined in this recent webinar on the secrets to SharePoint migration success and definitive guides to SharePoint migrations.
Several CIOs shared how they've implemented dedicated "innovation sandbox" environments for AI experimentation. These isolated, safe environments loaded with representative data (often a mix of anonymized real data and synthetic data) allow teams to quickly develop proofs-of-concept without risking disruption to production systems.
Key components of effective sandbox environments include:
While these sandboxes enable more advanced experimentation they don’t remove the need for advancing the right culture and business engagement.
Successful experimentation requires both cultural support and disciplined execution:
By addressing data readiness and integration planning from the start, IT executives significantly increase the odds that experiments will successfully transition from lab to enterprise-wide implementation. All the executives supported the need to safely fail and learn.
One top IT executive said, “We cannot know the limits of these technologies without using them ourselves. Failures guide us for future work. What seemed impossible months ago can become easy in less than a year.”
With a solid testing foundation, IT leaders can concentrate on an important aspect of artificial intelligence. This aspect is autonomous AI agents.
The most dynamic part of the roundtable focused on AI agents—autonomous or semi-autonomous AI-driven programs that can perform tasks, make recommendations, and take actions on behalf of users.
Unlike traditional RPA (Robotic Process Automation) which is limited to scripted, rule-based tasks, AI agents bring intelligence and adaptability. They understand context, learn from data, and collaborate with humans in a more interactive way.
There were many use cases discussed at the event but what follows are a few leading ones that we captured in our notes from discussions that leaders were leveraging or exploring themselves.
AI agents in IT operations can:
Beyond IT support, AI agents are expanding into core business functions:
In specialized fields like finance, AI agents are beginning to tackle activities such as M&A modeling:
The idea of using a cross-functional and strategic process like mergers and acquisitions came up on two different occasions as a great area to explore as it is front and center in executive visibility and can create momentum around other use cases and expansion of AI adoption in the organization.
These trends of complex domain applications are gaining momentum across industries—by late 2024, almost 70% of Fortune 500 companies had integrated Microsoft 365 Copilot into their daily operations.
Author’s Note: At the center of so much of this experimentation momentum lays business led agent development experiences like those accelerated by Copilot Studio. If you or your organization is trying to drive growth we strongly encourage having nominated team members join upcoming free Microsoft led workshops such like Copilot Sudio in a Day workshop’s that give participants hands-on experience with expert instructors.
The CIOs, CTOs, CISOs and IT executives emphasized pragmatic approaches to implementing AI agents:
The future vision shared in the roundtable sees AI agents as digital team members. They will handle routine and data-heavy tasks while boosting human skills. The plan is to have many agents for each employee.
These agents will be designed to solve business problems. This is not just about short-term gains from automation. It is also about better digitization. This will help improve, integrate, and advance projects across departments and roles over time.
As organizations deploy increasingly autonomous AI agents, however, questions of security, ethics, and governance become paramount—shifting the conversation from what AI can do to how it should be managed.
As organizations experiment with AI agents and models, security and responsible AI practices must evolve in parallel. Many roundtable attendees acknowledged that the excitement of AI adoption has sometimes outpaced the maturity of their governance frameworks. We had a separate dedicated set of roundtables for this topic (read all about their findings here), but naturally it came up throughout the discussion as we explore experimentation and agents.
Here were a few key insights that came up during our discussions on Agents and Experimentation:
Ensuring data trust emerged as the first pillar of Responsible AI:
The importance of these measures is underscored by industry research showing data issues (quality, availability, lineage) as a leading cause of AI project failures.
Many organizations at the roundtable had created or were creating AI steering committees or centers of excellence. These groups bring together IT, security, legal, and business leaders. These cross-functional groups typically create guidelines covering:
Responsible AI also encompasses clear lines of accountability for AI-driven decisions. Approaches shared by participants included:
From a security standpoint, AI agents introduce unique considerations because they may have access to wide swaths of data and execute actions automatically. Security best practices include:
Rather than allowing unrestricted AI deployment, some organizations are implementing structured adoption pipelines:
This approach balances innovation with necessary controls, ensuring new AI capabilities meet organizational standards.
With robust security and governance frameworks in place, organizations can accelerate their AI adoption with confidence—turning attention to the strategic priorities that will drive success in 2025 and beyond.
The CIO ThinkTank roundtable made one thing abundantly clear: AI agents and careful testing are essential for digital change. However, their successful implementation requires strategic vision, operational discipline, and robust governance.
Foster Strategic Experimentation
Prioritize Data Readiness
Deploy AI Agents for Maximum Impact
Balance Build vs. Buy Decisions
Embed Security and Ethics from Day One
As we're still in the early stages of the AI agent revolution, technologies and best practices will continue to evolve rapidly. Successful CIOs will position themselves as thought leaders—educating stakeholders, adapting policies, and continuously scanning the horizon for emerging opportunities and challenges.
The organizations that master this balance of bold experimentation and diligent governance will be the ones redefining their industries in the years ahead. The journey is just beginning—and IT Executives like CIOs, CTOs and CISOs are in the driver's seat.
Join Our Mailing List
