Feature Engineering Playbook: Building the Data Foundation for ML Models

Why Feature Engineering Is the Highest-ROI ML Investment

Two data science teams compete on the same prediction problem — customer churn for a SaaS company. Team A uses advanced algorithms (gradient boosted trees, neural networks, ensemble methods) on basic features (tenure, plan type, last login date). Team B uses a simple logistic regression on thoughtfully engineered features (login frequency trend over 30/60/90 days, support ticket velocity, feature adoption breadth, contract renewal proximity, NPS trajectory). Team B wins. Not by a little — by 12 AUC points. The difference isn't the algorithm. It's the features.

This pattern repeats across every ML project we encounter in machine learning consulting. Sophisticated algorithms on poor features underperform simple algorithms on rich features. The reason is mathematical: an algorithm can only learn patterns that exist in the feature space. If the features don't capture the behavioral signal that predicts churn (declining engagement trend), no algorithm — regardless of sophistication — can learn it. The signal isn't in the data the algorithm sees.

This playbook covers the feature engineering patterns that capture predictive signal across the most common enterprise ML use cases — churn prediction, fraud detection, demand forecasting, risk scoring, and recommendation. Each pattern is production-ready: designed for implementation in data pipelines and feature stores, not just notebook exploration.

Feature Engineering Patterns by Data Type

Feature engineering patterns follow the data type of the raw input. Each data type has specific transformation patterns that extract predictive signal.

Temporal Features: The Time Dimension That Drives Most Models

Temporal features capture how behavior changes over time — and behavior change is the strongest predictor in most enterprise ML use cases. A customer who logged in 20 times last month and 3 times this month is churning, regardless of their total login count. The trend is the signal; the aggregate hides it.

Recency Features

Days since last event: Days since last purchase, last login, last support ticket, last payment. Recency captures immediacy — a customer who purchased yesterday is behaviorally different from one who purchased 6 months ago, even if their total purchase count is identical. Calculate for each meaningful event type independently.

Recency relative to cadence: A monthly subscriber who last logged in 45 days ago is overdue. A quarterly buyer who last purchased 45 days ago is on schedule. Recency must be interpreted relative to the entity's typical cadence — not absolute days.

Frequency Features

Rolling window counts: Count of events in the last 7, 30, 60, 90 days. Multiple windows capture short-term behavior (7-day) and medium-term patterns (90-day). The ratio between windows reveals acceleration or deceleration — a 7-day count that's 50% of the 30-day count indicates acceleration; 10% indicates deceleration.

Frequency trend: Is the frequency increasing, stable, or decreasing? Calculate the slope of the frequency over time (linear regression on weekly counts over the last 12 weeks). A negative slope is the single strongest churn predictor in most subscription businesses — stronger than any static attribute.

Periodicity Features

Day-of-week and hour-of-day patterns: A user who always logs in on weekday mornings and suddenly starts logging in at midnight on weekends exhibits behavioral change. Extract the dominant period (most common day/hour) and the deviation from it.

Seasonality alignment: Is this entity's behavior following expected seasonal patterns? A retail customer whose December spending is 20% below their typical December represents an anomaly — not captured by comparing to other months.

Categorical Features: Encoding for ML Consumption

ML algorithms consume numbers. Categorical features (product category, industry, country, plan type) must be encoded as numbers — and the encoding method affects what the model can learn.

One-hot encoding: Creates a binary column for each category value. "Industry = Healthcare" becomes a column that's 1 for healthcare, 0 otherwise. Works well for low-cardinality features (10-50 categories). Fails for high cardinality (10,000 product IDs) — creates 10,000 sparse columns that slow training and overfit.

Target encoding: Replaces each category with the mean target value for that category. "Industry = Healthcare" becomes 0.15 (the average churn rate for healthcare customers). Captures the predictive signal of the category directly. Requires careful implementation to avoid data leakage — use leave-one-out or cross-validated encoding to prevent the target from leaking into the feature.

Frequency encoding: Replaces each category with its frequency in the dataset. Rare categories get small values; common categories get large values. Useful when category frequency itself is predictive — a customer on a rare plan type may behave differently from one on the most popular plan.

Embeddings (for high cardinality): Learned dense vector representations that capture similarity between categories. Product IDs, user IDs, and ZIP codes with thousands of unique values are better represented as 8-64 dimensional embeddings than 10,000 one-hot columns. Train embeddings as part of a neural network or use pre-trained embeddings from collaborative filtering.

Text Features: From Unstructured Language to Numerical Signal

Enterprise data is rich with unstructured text — support tickets, customer reviews, email communications, survey responses, internal documents. Text features extract the predictive signal hidden in language.

Sentiment features: Positive/negative/neutral sentiment score for support tickets, reviews, or survey comments. A customer whose sentiment trajectory is declining (positive → neutral → negative over 3 months) is a churn risk. Sentiment is a leading indicator — it changes before behavior changes.

Topic features: What topics does the customer engage with? A customer who starts asking about billing and cancellation policies (detected via NLP topic classification) exhibits pre-churn behavior that no numerical feature captures. Topic distribution as a feature vector lets the model learn which topic patterns predict which outcomes.

Embedding features: Dense vector representations of text from transformer models (BERT, GPT, sentence-transformers). These capture semantic meaning that keyword-based approaches miss — "I'm frustrated with the product" and "this software doesn't meet our needs" have similar meaning but no keyword overlap. Embeddings capture the similarity. Use for: support ticket classification, email intent prediction, and document similarity.

Feature Interactions and Derived Features

Individual features capture individual signals. Feature interactions capture the combined signal that neither feature captures alone.

Ratio features: Revenue per employee (captures efficiency), support tickets per login (captures friction), purchase value per visit (captures monetization). Ratios normalize for scale — a company with 100 employees and $1M revenue is different from a company with 10 employees and $1M revenue, even though the revenue is identical.

Difference features: This month's value minus last month's value (change), this quarter's value minus same quarter last year (year-over-year change). Differences capture momentum — the direction and magnitude of change matter more than the absolute level for most prediction tasks.

Interaction features: Multiplying two features creates an interaction that captures the combined effect. "High tenure × declining engagement" is a different signal from "high tenure" alone or "declining engagement" alone — it captures the specific pattern of long-term customers losing interest, which is the highest-risk churn segment.

Feature Selection: Removing Noise Without Losing Signal

More features isn't always better. Beyond a point, additional features add noise that degrades model performance — the curse of dimensionality. Feature selection identifies the features that carry signal and removes the ones that don't.

Filter methods: Rank features by statistical relationship with the target (correlation, mutual information, chi-squared). Remove features below a threshold. Fast, simple, but doesn't account for feature interactions.

Wrapper methods: Iteratively add/remove features and measure model performance. Forward selection (add the best feature at each step), backward elimination (remove the worst feature at each step), or recursive feature elimination (RFE). More accurate than filters but computationally expensive for large feature sets.

SHAP importance: Train the model with all features, then use SHAP values to measure each feature's contribution to predictions. Features with near-zero SHAP importance can be removed without affecting performance. SHAP-based selection accounts for interactions and provides the most reliable importance ranking for tree-based models.

Production Feature Engineering: Feature Store Architecture

Production feature engineering is a data engineering discipline, not a notebook exercise. Features computed in notebooks can't be reused, aren't tested, and create the training-serving skew that silently corrupts production predictions.

Feature Store: The Production Pattern

A feature store provides: feature registry (catalog of all features with definitions, owners, and lineage), offline store (historical features for model training — point-in-time correct to prevent data leakage), online store (low-latency current features for real-time inference), and feature computation pipeline (automated, tested, monitored pipelines that compute features from raw data).

Feature store options for enterprise:

Data Leakage: The Silent Accuracy Killer

Data leakage is the most dangerous bug in ML — it makes models appear accurate during development and fail in production. Leakage occurs when information from the future leaks into features used for training.

Common Leakage Patterns

Target leakage: A feature that is a consequence of the target, not a predictor. Predicting churn using "cancellation_reason" as a feature — the reason exists because the customer churned. The model learns to predict churn perfectly by detecting the presence of a cancellation reason, which is only available after the fact.

Temporal leakage: Using future data to predict the past. Computing "next 30 days login count" using data from after the prediction date. The model learns from information it won't have at prediction time, producing artificially high accuracy that collapses in production.

Train-test leakage: Information from the test set leaking into training — through global statistics (computing mean using all data including test set), through entity overlap (the same customer appearing in both train and test), or through temporal overlap (training on March data and testing on February data).

Go Deeper

Continue building your understanding with these related resources from our consulting practice.