Enterprise AI Architecture and MLOps: Technical Decision-Maker Guide

The Production Gap: Why Notebooks Don't Scale

A data scientist builds a churn prediction model in a Jupyter notebook. The model achieves 84% AUC on the test set. Leadership approves production deployment. The data scientist exports a pickle file. The engineering team asks: how do we deploy this? The notebook imports 14 Python libraries at specific versions. It reads from a CSV the data scientist created manually from a SQL query. The feature engineering logic lives in 47 notebook cells with no tests, no documentation, and no error handling. The "deployment" plan is to run the notebook on a schedule.

This is the production gap — the architectural chasm between a model that works in a notebook and a model that serves predictions reliably at scale. The notebook proved the algorithm works. Production requires: a data pipeline that delivers features reliably, a serving infrastructure that responds at the latency the business requires, monitoring that detects when predictions degrade, and the automation that retrains the model when data patterns shift. These are architectural concerns, not data science concerns. And they're the concerns this guide addresses.

Reference Architecture: The 6-Layer AI Stack

Layer 1: Data Platform and Feature Store

The data platform provides the raw materials models consume — features for training and features for inference. The feature store is the architectural pattern that makes feature engineering reusable, consistent, and production-grade.

Feature Store Architecture

A feature store serves two purposes: offline store (historical features for model training — point-in-time correct, avoiding data leakage) and online store (low-latency feature serving for real-time inference — millisecond access to pre-computed features). Without a feature store, each model team engineers features independently — rebuilding the same calculations, risking inconsistency between training and serving, and creating maintenance burden as features multiply.

Feature store options: Feast (open-source, cloud-agnostic), Fabric feature store (integrated with OneLake and ML), Databricks Feature Store (integrated with MLflow and Unity Catalog), Tecton (managed, production-grade). The choice follows the data platform — a Fabric-based data estate uses Fabric's feature store; a Databricks estate uses Databricks Feature Store.

Feature Engineering as Data Engineering

Feature engineering for production AI is a data engineering discipline, not a notebook exercise. Features must be: computed by production-grade data pipelines (not notebook cells), version-controlled (which feature definition produced which model version), tested (unit tests for transformation logic, integration tests for pipeline reliability), and monitored (feature distribution tracking for drift detection). The data engineering team owns feature pipelines. The data science team defines what features to compute. This separation of concerns is critical for sustainable AI operations.

Layer 2: Training Infrastructure and Experiment Tracking

Training infrastructure provides the compute and tooling for model development. The key architectural decisions: managed vs. self-managed compute, experiment tracking platform, and resource governance.

Managed Compute

Azure ML Compute: Managed compute clusters that scale to GPU nodes for deep learning or CPU clusters for traditional ML. Auto-scaling based on job queue depth — no provisioning required. Integrated with Azure ML experiment tracking and model registry.

Databricks Clusters: Spark-based compute for distributed training on large datasets. Photon-optimized for SQL workloads, GPU-enabled for deep learning. Cluster policies control cost by limiting max nodes and auto-termination.

Self-managed (Kubernetes): Kubeflow on Kubernetes provides maximum flexibility — any framework (PyTorch, TensorFlow, XGBoost), any hardware (CPU, GPU, TPU), any scale. But operational overhead is high: cluster management, resource scheduling, and infrastructure maintenance are the team's responsibility.

Experiment Tracking

Experiment tracking records every model training run: hyperparameters, dataset version, feature configuration, performance metrics, and model artifacts. Without it, model development is unreproducible — "the model I trained last Tuesday worked better, but I can't remember the configuration."

MLflow is the de facto standard for experiment tracking — open-source, platform-agnostic, integrated with Databricks and Azure ML natively. Each experiment run logs parameters, metrics, and artifacts. The experiment history shows which configurations produced the best results and enables reproducibility.

Layer 3: Model Registry and Artifact Management

The model registry is the single source of truth for model versions — which model artifact is deployed to which environment, when it was trained, on what data, with what performance metrics. Without a registry, model deployment is manual and error-prone: "which pickle file is the current production model?"

Registry capabilities: versioning (each model version has a unique identifier), staging (models move through None → Staging → Production → Archived lifecycle), metadata (each version stores metrics, parameters, data lineage), and access control (who can promote models to production).

MLflow Model Registry, Azure ML Model Registry, and Databricks Unity Catalog all provide registry functionality. The choice follows the ML platform — consistency between experiment tracking and model registry reduces integration complexity.

Layer 4: Deployment and Serving

Deployment transforms a registered model artifact into a production prediction service. Three deployment patterns serve different use cases:

Real-Time Serving (Online Inference)

Model deployed behind an API endpoint. Applications send prediction requests and receive responses in milliseconds. Architecture: containerized model (Docker) behind a load-balanced endpoint with auto-scaling. Azure ML Managed Endpoints, SageMaker Endpoints, or Kubernetes-based serving (Seldon, KServe) provide the infrastructure.

Critical for: fraud detection (score each transaction before authorization), recommendation engines (personalize each page view), chatbots and AI agents (respond in real time), and pricing optimization (adjust prices per request).

Batch Scoring (Offline Inference)

Model scores the entire dataset periodically — nightly churn scores for all customers, weekly demand forecasts for all products, monthly credit risk scores for the portfolio. Architecture: scheduled pipeline that reads input data, runs model inference, and writes predictions to the analytical platform. Simpler infrastructure, lower cost, suitable for predictions consumed in dashboards or batch processes.

Streaming Inference

Model scores events as they arrive in a data stream. Architecture: model embedded in a stream processing job (Spark Streaming, Flink, or custom consumer) that reads from Kafka/Event Hubs and writes scored events downstream. Critical for: IoT anomaly detection, real-time quality inspection, and event-driven decisioning where batch is too slow and request/response isn't applicable.

Layer 5: Monitoring, Observability, and Drift Detection

A deployed model is a depreciating asset. Without monitoring, model performance degrades silently — predictions become less accurate as data patterns shift, but nobody notices until a business user reports that "the predictions don't seem right anymore." By then, trust is damaged and recovery takes months.

Four Monitoring Categories

Technical health: Endpoint latency, throughput, error rate, memory usage, GPU utilization. Standard infrastructure monitoring. Alert when latency exceeds SLA or error rate spikes.

Data quality: Input feature distributions compared to training distributions. Missing values, outliers, schema changes. Data quality issues are the most common cause of production model degradation — a source system change that alters feature values silently corrupts predictions.

Model performance: Prediction accuracy measured against ground truth (when available — churn predictions can be validated 90 days later when the customer churns or doesn't). Prediction distribution monitoring (are predictions shifting — more positives, fewer negatives, different confidence distribution?). Calibration monitoring (do predicted probabilities match observed frequencies?).

Data drift: Statistical comparison between the production feature distributions and the training feature distributions. Population Stability Index (PSI), Kolmogorov-Smirnov test, or Jensen-Shannon divergence — each measures how much the production data has diverged from training data. When drift exceeds thresholds, the model's assumptions about the data may no longer hold — triggering investigation and potential retraining.

Layer 6: MLOps Automation and CI/CD for ML

MLOps automates the ML lifecycle — from data ingestion through training, validation, deployment, monitoring, and retraining. MLOps maturity determines how many models the organization can operate simultaneously.

MLOps Maturity Levels

Most enterprises operate at Level 0-1. Production AI at scale requires Level 2+. The investment from Level 1 to Level 2 is primarily in pipeline automation, testing infrastructure, and deployment standardization — not in new platforms or tools.

Platform Comparison: Azure ML, Databricks, SageMaker, Vertex

Selection guidance: If your data platform is Azure/Fabric, choose Azure ML. If Databricks, choose Databricks ML. If AWS, choose SageMaker. If GCP, choose Vertex AI. Platform integration — not feature comparison — drives the right decision. An ML platform that integrates natively with the data platform reduces the data engineering effort by 40-60% compared to a platform that requires custom connectors.

MLOps Maturity: From Manual to Fully Automated

The path from Level 0 to Level 3 is incremental — each level builds on the previous one. Don't try to jump from Level 0 to Level 3 in a single project.

Go Deeper

Continue building your understanding with these related resources from our consulting practice.