AI Team Topology: Where LLM Engineers Actually Sit

One team owns all AI work for the entire company. Product teams submit requests, the AI team prioritizes and delivers. The team typically reports to a VP of Engineering or CTO, sits adjacent to data engineering, and operates its own backlog independent of product sprints.

When it works: Early maturity (0-2 models in production). The company is still figuring out what AI can do for them. Demand is low enough that one team can serve it without creating a months-long queue. The initial models are experimental, and centralizing expertise accelerates learning. Startups under 30 engineers and enterprises in the first year of an AI initiative fit here.

When it breaks: The moment demand outpaces bandwidth. Product teams start waiting weeks for the AI team's attention. Priority conflicts become political. The centralized team becomes a gatekeeper rather than an enabler. Two failure modes: (a) the team becomes so specialized in one domain that it can't context-switch to another, or (b) it spreads so thin across domains that quality drops everywhere.

Team Topologies mapping: The centralized AI team functions as an enabling team at best, a complicated-subsystem team at worst. Enabling when it's actively transferring knowledge to product teams and building toward its own dissolution. Complicated-subsystem when it's hoarding expertise and creating permanent dependencies.

Real example: A mid-market SaaS company (200 engineers, $50M ARR) stands up a 4-person ML team to ship their first recommendation engine. The team succeeds because they have one customer (the product team building recommendations) and one model to ship. Eighteen months later, five product teams want AI features, the ML team is a 6-month bottleneck, and the reorg begins.

AI engineers sit directly inside product teams. They report to the product team's engineering manager, attend the same stand-ups, and ship AI features alongside frontend and backend engineers. There is no centralized AI team; each product team owns its own AI capabilities end-to-end.

When it works: When AI is a feature, not the product. When the AI work is primarily integration (calling LLM APIs, building RAG pipelines, tuning prompts) rather than model development. When each product team's AI needs are distinct enough that shared infrastructure would be premature abstraction. AI-augmented products (vs. AI-native products) fit this model well.

When it breaks: Infrastructure duplication. Team A builds a vector database integration, Team B builds a different one. Team C builds a prompt management system, Team D rolls their own. Within a year you have four RAG architectures, three evaluation frameworks, and zero shared model registry. Inconsistency in AI behavior across the product surface becomes a UX problem. Embedded engineers get isolated from ML peers and stop developing their craft.

Team Topologies mapping: AI engineers are simply part of stream-aligned teams. This is the purest Team Topologies implementation, but it only works when the AI work is bounded enough to fit inside a single team's cognitive load budget without requiring deep ML platform knowledge.

Real example: A product-led growth company (80 engineers) where three product teams each have one AI engineer building LLM-powered features (smart search, content generation, customer support chatbot). Works fine for 18 months. Then the CEO asks "why do our three AI features behave so differently?" and "why are we paying three separate vector DB bills?" and the reorg conversation starts.

The AI team builds and operates shared ML infrastructure: model registries, feature stores, experiment tracking, model serving, evaluation pipelines, prompt management, vector databases, and guardrails. Product teams consume this platform to build their own AI features without needing deep ML ops expertise. The platform team ships tooling, not models.

When it works: At scale (10+ models in production, 5+ teams building AI features). When the infrastructure duplication from the embedded model has become too expensive. When you have enough internal demand to justify a dedicated platform investment. When the platform team has clear users and can define X-as-a-Service contracts with measurable SLOs.

When it breaks: When the platform gets too far from product reality. The classic failure: the platform team builds what they think product teams need (usually more abstraction, more configurability) while product teams need something simpler (a working example, good defaults, fast iteration cycles). The platform becomes overengineered for the sophistication of its users. Second failure mode: the platform team has no embedded context, so they can't help when things go wrong in production. Product teams still need "someone who knows ML" on the team, bringing you back to the hybrid model.

Team Topologies mapping: A clean platform team in the Team Topologies sense. Operates X-as-a-Service with stream-aligned teams as consumers. The interaction mode is clearly defined: product teams call the platform's APIs, the platform team maintains the infrastructure. This is the most natural Team Topologies fit of the four models.

Real example: Spotify's ML platform team. Uber's Michelangelo. Any company that has enough models in production that the build/serve/monitor loop needs to be a shared, maintained service rather than copy-pasted boilerplate in each team's repo. At this scale, the platform team is 8-15 engineers and product teams have 0-2 AI engineers each who focus on application logic, not infrastructure.

A central AI platform team provides shared infrastructure and best practices. Embedded AI specialists sit inside product teams and consume the platform while maintaining product context. An enabling function (sometimes called "AI guild" or "ML community of practice") connects the embedded specialists so they share learnings and prevent drift. Three layers: platform (the hub), embedded specialists (the spokes), and the connecting tissue (the guild).

When it works: For organizations with 50+ engineers and 3+ teams building AI features. This is the consensus model for most companies past the experimentation phase. It solves the three failure modes simultaneously: no bottleneck (product teams have their own AI people), no duplication (shared platform handles infrastructure), no isolation (guild connects embedded specialists). It maps directly to Team Topologies: a platform team, stream-aligned teams with embedded AI engineers, and a thin enabling team that facilitates knowledge sharing.

When it breaks: Coordination overhead. Three layers means three places where misalignment can hide. The platform team builds something the spokes don't use. The spokes go rogue and build custom solutions because the platform is "too slow." The guild becomes a talking shop without decision-making authority. The tax on this model is continuous alignment work: the Head of AI (or equivalent) spends 40% of their time on internal coordination rather than technical work.

Team Topologies mapping: This IS Team Topologies applied correctly. Platform team (AI infrastructure) + stream-aligned teams (with embedded AI specialists) + enabling team (the guild/community of practice). The interaction modes are: X-as-a-Service between platform and stream-aligned teams, facilitating between the enabling team and everyone else. Collaboration mode activates temporarily when a new capability is being built (platform collaborates with the first product team to use it, then shifts to X-as-a-Service for subsequent teams).

Real example: A Series D fintech (300 engineers, 40 in AI/ML across 8 product teams). Central ML platform team of 12 handles model serving, feature store, experiment tracking, and evaluation infrastructure. Each product team has 2-4 AI engineers who build models and RAG pipelines on the platform. Bi-weekly ML guild meetings share learnings. A Head of AI (reporting to CTO) owns the platform team directly and has dotted-line influence over embedded AI hires.