ctaio.dev
Ask AI Subscribe free

AI Team Design

AI Center of Excellence: Structure, Roles, and First 90 Days

Most AI CoEs die within 18 months. They become advisory committees producing frameworks nobody reads, or talent pools hoarding engineers nobody can access. The ones that survive share three traits: delivery mandate from day one, single accountable leader with budget, and a first project shipped inside 90 days.

By · Published May 25, 2026

Three Operating Models

Every AI CoE falls into one of three patterns. Pick one. Mixing them creates confusion about what the team delivers.

Advisory CoE

Does: Sets standards, evaluates vendors, publishes guidelines, reviews architectures. Doesn't build.

Works when: Large enterprises (2000+ employees) with BUs already doing AI independently. Value is coherence.

Fails when: Early-stage adoption where nobody's doing AI yet. Can't advise on something nobody's attempting. Also fails when BUs ignore standards because the CoE has no delivery credibility.

Delivery CoE

Does: Builds AI products for business units. Takes requests, prioritizes, delivers. Internal AI agency.

Works when: 3-8 BUs each need AI but can't justify their own ML team. CoE multiplexes scarce talent.

Fails when: Demand outstrips capacity. Every BU wants priority. Team gets pulled everywhere and ships nothing well. Becomes a bottleneck people route around.

Platform CoE

Does: Builds AI infrastructure, tooling, and self-serve platforms. BUs build on top.

Works when: Scale. 10+ models in production, 50+ people touching ML. Platform enables self-service without reinventing infra per project.

Fails when: Before you have enough users to justify the investment. Platform for 2 teams is over-engineering. Also fails if platform team loses touch with real product problems.

Start as Delivery CoE. Prove value by shipping. Evolve to Platform once you have 5+ active AI projects consuming shared infrastructure. Starting advisory is usually wrong because you can't earn credibility without demonstrating you can ship.

The Eight Roles

Not all from day one. Hire the first four in weeks 5-8. Add the rest as you scale past your second project.

Head of AI / CoE Lead — Owns budget, strategy, org politics. Must be technically credible AND organizationally savvy. This role determines whether the CoE lives or dies.
Senior ML Engineer — Architects and ships production models end-to-end. Not a researcher. A builder. Hire for production experience, not paper count.
Data Engineer — Builds pipelines that feed ML models. Without clean data infra, everything else is sandcastle engineering.
AI Product Manager — Translates business problems into ML-solvable specs. Different from regular PM: needs to understand what's technically feasible with current models.
MLOps Engineer — Model deployment, monitoring, lifecycle. Turns a Jupyter notebook into a production service that doesn't page anyone at 3am.
Prompt Engineering Lead — For LLM-heavy orgs. Owns prompt design patterns, evaluation, versioning. Didn't exist 3 years ago. Real now.
AI Safety / Ethics Lead — Red-teaming, bias testing, regulatory compliance (EU AI Act, NIST AI RMF). More critical in regulated industries.
Applied ML Engineers — Bulk of the team at scale. Each works on specific BU projects. Embedded in product context, reporting to CoE for technical standards.

90-Day Standup Plan

Weeks 1-4: Audit and Charter

  • Audit all existing AI usage across the org (you'll find more shadow AI than expected)
  • Interview each BU leader: what AI problem would save them the most time or money?
  • Write the charter: operating model, reporting line, budget, success metrics, scope boundaries
  • Get the charter signed by exec sponsor. Written and approved. No verbal agreements.
  • Pick first project. Criteria: clear ROI, deliverable in 60 days, visible to leadership, technically achievable with existing data

Weeks 5-8: Hire and Build

  • Hire core 3-4 people (ML engineer, data engineer, AI PM, lead if not already seated)
  • Set up infra: model registry, experiment tracking, deployment pipeline. Don't over-engineer. Managed services where possible.
  • Start building first project. Daily standups. Weekly stakeholder demos. Tight feedback with requesting BU.
  • Document architectural decisions as you go. This becomes your playbook.

Weeks 9-12: Ship and Document

  • Ship v1 to production. Doesn't need perfection. Needs measurable value.
  • Measure and report: what did this save/earn? Present to leadership.
  • Write the playbook: business problem to production in 60 days
  • Use success to secure budget for next two projects. Momentum over perfection.
  • Publish intake process: how BUs request, how you prioritize, what the SLA is

Anti-Patterns That Kill CoEs

The PowerPoint CoE

Advisory-only, no delivery. Produces strategy decks. Nobody follows them because the CoE has never shipped. Dies when budget gets questioned.

The Talent Hoarder

Attracts all ML engineers, puts them on internal research. BUs wait 6 months for capacity. Route around by hiring their own or buying vendor solutions. CoE becomes irrelevant.

The Orphan

No exec sponsor with real authority. Reports to a VP who reports to someone who might talk to the CEO quarterly. First overrun kills the team. Dies of starvation.

The Science Fair

Optimizes for technical impressiveness over business value. Ships amazing model on a problem nobody outside the team understands. "What's the dollar impact?" Can't answer. Budget cut.

Related Guides

AI Team Topology

Where LLM engineers sit. Hub-and-spoke vs embedded vs centralized.

AI Team Org Chart

Real org charts at 30, 200, and 1000+ person companies.

Conway's Law for AI

Your AI architecture mirrors your org chart. Design accordingly.

Frequently Asked Questions

What is an AI Center of Excellence?
An AI CoE is a cross-functional team responsible for AI strategy, standards, and delivery across the organization. It prevents every BU from reinventing AI infrastructure independently. Three operating models: advisory (sets standards, doesn't build), delivery (builds AI products for BUs), and platform (builds tooling that BUs self-serve on). Most orgs start as delivery and evolve to platform.
How many people do you need to start an AI CoE?
Four. One senior ML engineer who can architect and ship. One data engineer who can build pipelines. One AI product manager who translates business needs into specs. And a lead who can handle organizational politics and secure budget. You can stand this up in 90 days. Adding more people before shipping your first project is how CoEs become cost centers that die within 18 months.
Should the AI CoE report to the CTO or the CDO?
CTO if AI is core to your product. CDO if AI is primarily for internal analytics. CAIO if that role exists with real authority. The wrong answer is 'shared governance committee' because nobody owns delivery. Wherever the CoE sits, it needs a single accountable leader with budget authority and a direct line to build-vs-buy decisions.
Why do AI Centers of Excellence fail?
Three patterns. First: advisory-only mandate with no delivery responsibility. You write standards nobody follows. Second: hoarding talent so BUs can't get AI engineers on their roadmaps. You become a bottleneck. Third: no executive sponsor with real budget authority. When the first project overruns, nobody protects the team from being dissolved.
How long does it take to stand up an AI CoE?
90 days from charter to first delivery if you have executive buy-in. Weeks 1-4: audit current AI usage, write the charter, get budget approval. Weeks 5-8: hire core 3-4 people, pick your first project. Weeks 9-12: ship v1 and document the playbook. The playbook from your first project becomes the template for everything after.
·
Thomas Prommer
Thomas Prommer Technology Executive — CTO/CIO/CTAIO

These salary reports are built on firsthand hiring experience across 20+ years of engineering leadership (adidas, $9B platform, 500+ engineers) and a proprietary network of 200+ executive recruiters and headhunters who share placement data with us directly. As a top-1% expert on institutional investor networks, I've conducted 200+ technical due diligence consultations for PE/VC firms including Blackstone, Bain Capital, and Berenberg — work that requires current, accurate compensation benchmarks across every seniority level. Our team cross-references recruiter data with BLS statistics, job board salary disclosures, and executive compensation surveys to produce ranges you can actually negotiate with.

Profile LinkedIn Newsletter