Why Accuracy Isn't Enough: Comprehensive ML Model Evaluation for Production Systems

Ali Mahmoudi

Why Accuracy Isn’t Enough: Comprehensive ML Model Evaluation for Production Systems

Building ML models that work in production requires fundamentally different evaluation approaches than academic exercises. Over the past three years architecting real-time sports prediction systems and customer intelligence platforms at a leading Australian sports technology company, I’ve learned that a model showing 95% accuracy in testing can still fail catastrophically in production.

The difference between research and production ML isn’t just scale—it’s understanding that evaluation is where theory meets business reality.

The Accuracy Trap: Lessons from Sports Betting

Accuracy seems intuitive: how often does our model make correct predictions? But in high-stakes production environments, this metric can be dangerously misleading.

Real example: Our early sports prediction models achieved 85% accuracy predicting match winners. Sounds impressive, right? Wrong. The business lost money because the model was biased toward favorites—it was right about obvious outcomes but missed the profitable edge cases where underdogs had value.

In sports analytics, profitability matters more than accuracy. A model that’s 60% accurate but identifies profitable opportunities outperforms an 85% accurate model that only predicts obvious outcomes.

This is why production ML requires a sophisticated evaluation toolkit, with each metric revealing different aspects of business performance.

Classification Evaluation: The Complete Framework

Understanding the Confusion Matrix

The confusion matrix forms the foundation of classification evaluation. From this 2×2 table (for binary classification), we derive all other metrics:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.model_selection import cross_val_score, StratifiedKFold

def create_detailed_confusion_matrix(y_true, y_pred, labels=None):
    """Create an enhanced confusion matrix with detailed analysis"""
    
    cm = confusion_matrix(y_true, y_pred)
    
    # Calculate percentages
    cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
    
    # Visualization
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
    
    # Raw counts
    sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', ax=ax1)
    ax1.set_title('Confusion Matrix (Counts)')
    ax1.set_ylabel('True Label')
    ax1.set_xlabel('Predicted Label')
    
    # Normalized
    sns.heatmap(cm_normalized, annot=True, fmt='.2%', cmap='Blues', ax=ax2)
    ax2.set_title('Confusion Matrix (Percentages)')
    ax2.set_ylabel('True Label')
    ax2.set_xlabel('Predicted Label')
    
    plt.tight_layout()
    return cm

# Example usage
y_true = np.array([1, 0, 1, 1, 0, 1, 0, 0, 1, 0])
y_pred = np.array([1, 0, 1, 0, 0, 1, 1, 0, 1, 0])
cm = create_detailed_confusion_matrix(y_true, y_pred)

Precision, Recall, and F1-Score: The Balancing Act

These metrics address the accuracy limitation by focusing on specific aspects of performance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

from sklearn.metrics import precision_score, recall_score, f1_score, fbeta_score

def comprehensive_classification_metrics(y_true, y_pred, beta=1.0):
    """Calculate comprehensive classification metrics"""
    
    precision = precision_score(y_true, y_pred, average='weighted')
    recall = recall_score(y_true, y_pred, average='weighted')
    f1 = f1_score(y_true, y_pred, average='weighted')
    f_beta = fbeta_score(y_true, y_pred, beta=beta, average='weighted')
    
    metrics = {
        'Precision': precision,
        'Recall': recall,
        'F1-Score': f1,
        f'F{beta}-Score': f_beta
    }
    
    # Interpretation guide
    interpretations = {
        'Precision': f"Of all positive predictions, {precision:.1%} were correct",
        'Recall': f"Of all actual positives, {recall:.1%} were identified", 
        'F1-Score': f"Harmonic mean balancing precision and recall",
        f'F{beta}-Score': f"Weighted harmonic mean (β={beta})"
    }
    
    return metrics, interpretations

# Apply to real scenario
metrics, interpretations = comprehensive_classification_metrics(y_true, y_pred)

for metric, value in metrics.items():
    print(f"{metric}: {value:.3f}")
    print(f"  → {interpretations[metric]}")
    print()

ROC Analysis: Understanding Trade-offs

ROC curves reveal how well a model discriminates across different decision thresholds:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

from sklearn.metrics import roc_curve, auc, precision_recall_curve
import matplotlib.pyplot as plt

def comprehensive_curve_analysis(y_true, y_scores):
    """Generate ROC and Precision-Recall curves with detailed analysis"""
    
    # ROC Curve
    fpr, tpr, roc_thresholds = roc_curve(y_true, y_scores)
    roc_auc = auc(fpr, tpr)
    
    # Precision-Recall Curve  
    precision, recall, pr_thresholds = precision_recall_curve(y_true, y_scores)
    pr_auc = auc(recall, precision)
    
    # Find optimal threshold using Youden's J statistic
    j_scores = tpr - fpr
    optimal_idx = np.argmax(j_scores)
    optimal_threshold = roc_thresholds[optimal_idx]
    
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))
    
    # ROC Curve
    ax1.plot(fpr, tpr, color='darkorange', lw=2, 
             label=f'ROC Curve (AUC = {roc_auc:.3f})')
    ax1.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--', 
             label='Random Classifier')
    ax1.scatter(fpr[optimal_idx], tpr[optimal_idx], color='red', s=100, 
                label=f'Optimal Threshold = {optimal_threshold:.3f}')
    ax1.set_xlim([0.0, 1.0])
    ax1.set_ylim([0.0, 1.05])
    ax1.set_xlabel('False Positive Rate')
    ax1.set_ylabel('True Positive Rate')
    ax1.set_title('ROC Curve Analysis')
    ax1.legend(loc="lower right")
    ax1.grid(True, alpha=0.3)
    
    # Precision-Recall Curve
    ax2.plot(recall, precision, color='blue', lw=2,
             label=f'PR Curve (AUC = {pr_auc:.3f})')
    baseline_precision = sum(y_true) / len(y_true)
    ax2.axhline(y=baseline_precision, color='red', linestyle='--', 
                label=f'Baseline = {baseline_precision:.3f}')
    ax2.set_xlim([0.0, 1.0])
    ax2.set_ylim([0.0, 1.05])
    ax2.set_xlabel('Recall')
    ax2.set_ylabel('Precision')
    ax2.set_title('Precision-Recall Curve')
    ax2.legend()
    ax2.grid(True, alpha=0.3)
    
    plt.tight_layout()
    
    return {
        'roc_auc': roc_auc,
        'pr_auc': pr_auc, 
        'optimal_threshold': optimal_threshold,
        'optimal_tpr': tpr[optimal_idx],
        'optimal_fpr': fpr[optimal_idx]
    }

# Example with synthetic probability scores
y_scores = np.random.beta(2, 1, size=len(y_true))  # Biased toward higher scores
results = comprehensive_curve_analysis(y_true, y_scores)

Regression Evaluation: Beyond R-Squared

Regression evaluation requires different thinking. We’re not just asking “is this prediction right?” but “how wrong is this prediction, and does it matter?”

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
from sklearn.metrics import mean_absolute_percentage_error
import scipy.stats as stats

def comprehensive_regression_evaluation(y_true, y_pred, sample_weight=None):
    """Comprehensive regression model evaluation"""
    
    # Core metrics
    mse = mean_squared_error(y_true, y_pred, sample_weight=sample_weight)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y_true, y_pred, sample_weight=sample_weight)
    r2 = r2_score(y_true, y_pred, sample_weight=sample_weight)
    
    # Additional insights
    residuals = y_true - y_pred
    
    # Mean Absolute Percentage Error (be careful with zeros)
    non_zero_mask = y_true != 0
    if np.any(non_zero_mask):
        mape = np.mean(np.abs(residuals[non_zero_mask] / y_true[non_zero_mask])) * 100
    else:
        mape = float('inf')
    
    # Symmetric MAPE (handles zeros better)
    smape = np.mean(2 * np.abs(residuals) / (np.abs(y_true) + np.abs(y_pred))) * 100
    
    # Statistical tests
    # Normality of residuals (Shapiro-Wilk)
    if len(residuals) <= 5000:  # Test works well for smaller samples
        normality_stat, normality_p = stats.shapiro(residuals)
    else:
        normality_stat, normality_p = stats.normaltest(residuals)
    
    # Heteroscedasticity test (simple version)
    # Split residuals in half and compare variances
    mid_point = len(residuals) // 2
    first_half_var = np.var(residuals[:mid_point])
    second_half_var = np.var(residuals[mid_point:])
    heteroscedasticity_ratio = max(first_half_var, second_half_var) / min(first_half_var, second_half_var)
    
    metrics = {
        'MSE': mse,
        'RMSE': rmse,
        'MAE': mae,
        'R²': r2,
        'MAPE (%)': mape,
        'SMAPE (%)': smape,
        'Residuals Normal': normality_p > 0.05,
        'Heteroscedasticity Ratio': heteroscedasticity_ratio
    }
    
    return metrics, residuals

# Example usage
np.random.seed(42)
y_true_reg = np.random.normal(100, 20, 1000)
y_pred_reg = y_true_reg + np.random.normal(0, 5, 1000)  # Add noise

reg_metrics, residuals = comprehensive_regression_evaluation(y_true_reg, y_pred_reg)

print("Regression Evaluation Results:")
print("=" * 40)
for metric, value in reg_metrics.items():
    if isinstance(value, bool):
        print(f"{metric}: {'✓' if value else '✗'}")
    elif isinstance(value, float):
        print(f"{metric}: {value:.4f}")

Cross-Validation: Robust Performance Estimation

Single train-test splits can be misleading. Cross-validation provides more reliable performance estimates:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

from sklearn.model_selection import cross_validate, StratifiedKFold, TimeSeriesSplit
from sklearn.ensemble import RandomForestClassifier
import pandas as pd

def robust_model_evaluation(model, X, y, cv_strategy='stratified', scoring=None):
    """Perform robust model evaluation using cross-validation"""
    
    if scoring is None:
        scoring = ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']
    
    # Choose appropriate CV strategy
    if cv_strategy == 'stratified':
        cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    elif cv_strategy == 'time_series':
        cv = TimeSeriesSplit(n_splits=5)
    else:
        cv = 5  # Default k-fold
    
    # Perform cross-validation
    cv_results = cross_validate(model, X, y, cv=cv, scoring=scoring, 
                               return_train_score=True, n_jobs=-1)
    
    # Organize results
    results_summary = {}
    
    for metric in scoring:
        test_scores = cv_results[f'test_{metric}']
        train_scores = cv_results[f'train_{metric}']
        
        results_summary[metric] = {
            'test_mean': np.mean(test_scores),
            'test_std': np.std(test_scores),
            'train_mean': np.mean(train_scores),
            'train_std': np.std(train_scores),
            'overfitting_gap': np.mean(train_scores) - np.mean(test_scores),
            'test_scores': test_scores
        }
    
    return results_summary

# Example with synthetic data
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=20, n_informative=10,
                          n_redundant=5, random_state=42)

model = RandomForestClassifier(n_estimators=100, random_state=42)
cv_results = robust_model_evaluation(model, X, y)

print("Cross-Validation Results:")
print("=" * 50)
for metric, stats in cv_results.items():
    print(f"\n{metric.upper()}:")
    print(f"  Test Score: {stats['test_mean']:.3f} (±{stats['test_std']:.3f})")
    print(f"  Train Score: {stats['train_mean']:.3f} (±{stats['train_std']:.3f})")
    print(f"  Overfitting Gap: {stats['overfitting_gap']:.3f}")

Learning Curves: Diagnosing Model Problems

Learning curves reveal whether your model suffers from bias or variance issues:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

from sklearn.model_selection import learning_curve
import matplotlib.pyplot as plt

def plot_learning_curves(model, X, y, cv=5):
    """Generate and plot learning curves for bias-variance analysis"""
    
    train_sizes = np.linspace(0.1, 1.0, 10)
    
    train_sizes_abs, train_scores, val_scores = learning_curve(
        model, X, y, cv=cv, n_jobs=-1, train_sizes=train_sizes,
        scoring='accuracy', random_state=42
    )
    
    # Calculate means and standard deviations
    train_mean = np.mean(train_scores, axis=1)
    train_std = np.std(train_scores, axis=1)
    val_mean = np.mean(val_scores, axis=1)
    val_std = np.std(val_scores, axis=1)
    
    plt.figure(figsize=(12, 8))
    
    # Plot learning curves
    plt.subplot(2, 2, 1)
    plt.plot(train_sizes_abs, train_mean, 'o-', color='blue', label='Training Score')
    plt.fill_between(train_sizes_abs, train_mean - train_std, train_mean + train_std, 
                     alpha=0.2, color='blue')
    
    plt.plot(train_sizes_abs, val_mean, 'o-', color='red', label='Validation Score')
    plt.fill_between(train_sizes_abs, val_mean - val_std, val_mean + val_std,
                     alpha=0.2, color='red')
    
    plt.xlabel('Training Set Size')
    plt.ylabel('Accuracy Score')
    plt.title('Learning Curves')
    plt.legend()
    plt.grid(True, alpha=0.3)
    
    # Gap analysis
    gap = train_mean - val_mean
    plt.subplot(2, 2, 2)
    plt.plot(train_sizes_abs, gap, 'o-', color='green')
    plt.xlabel('Training Set Size')
    plt.ylabel('Training - Validation Gap')
    plt.title('Overfitting Analysis')
    plt.grid(True, alpha=0.3)
    
    # Final performance analysis
    final_train_score = train_mean[-1]
    final_val_score = val_mean[-1]
    final_gap = gap[-1]
    
    plt.subplot(2, 1, 2)
    plt.text(0.1, 0.8, f"Final Training Score: {final_train_score:.3f}", fontsize=12)
    plt.text(0.1, 0.6, f"Final Validation Score: {final_val_score:.3f}", fontsize=12)
    plt.text(0.1, 0.4, f"Overfitting Gap: {final_gap:.3f}", fontsize=12)
    
    # Diagnosis
    if final_gap > 0.05:
        diagnosis = "High Variance (Overfitting) - Try regularization or more data"
    elif final_val_score < 0.7:
        diagnosis = "High Bias (Underfitting) - Try more complex model"
    else:
        diagnosis = "Good Balance - Model appears well-tuned"
    
    plt.text(0.1, 0.2, f"Diagnosis: {diagnosis}", fontsize=12, weight='bold')
    plt.axis('off')
    
    plt.tight_layout()
    return train_sizes_abs, train_scores, val_scores

# Example usage
learning_data = plot_learning_curves(model, X, y)

Business-Oriented Evaluation

Technical metrics are important, but business impact is what matters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

def business_impact_analysis(y_true, y_pred, y_proba=None, cost_matrix=None):
    """Evaluate model from business perspective"""
    
    if cost_matrix is None:
        # Default cost matrix (can be customized based on business case)
        cost_matrix = {
            'true_positive_value': 100,    # Revenue from correctly identifying positive
            'false_positive_cost': -25,    # Cost of false alarm
            'false_negative_cost': -150,   # Cost of missing positive
            'true_negative_value': 5       # Small value for correct rejection
        }
    
    # Calculate confusion matrix elements
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    
    # Business metrics
    total_value = (tp * cost_matrix['true_positive_value'] + 
                  tn * cost_matrix['true_negative_value'] + 
                  fp * cost_matrix['false_positive_cost'] + 
                  fn * cost_matrix['false_negative_cost'])
    
    # ROI calculation
    investment_cost = len(y_true) * 10  # Assume $10 per prediction
    roi = (total_value - investment_cost) / investment_cost * 100
    
    # Threshold optimization (if probabilities available)
    if y_proba is not None:
        thresholds = np.linspace(0.1, 0.9, 9)
        threshold_values = []
        
        for threshold in thresholds:
            pred_thresh = (y_proba >= threshold).astype(int)
            tn_t, fp_t, fn_t, tp_t = confusion_matrix(y_true, pred_thresh).ravel()
            
            value_t = (tp_t * cost_matrix['true_positive_value'] + 
                      tn_t * cost_matrix['true_negative_value'] + 
                      fp_t * cost_matrix['false_positive_cost'] + 
                      fn_t * cost_matrix['false_negative_cost'])
            
            threshold_values.append(value_t)
        
        optimal_threshold = thresholds[np.argmax(threshold_values)]
        max_value = max(threshold_values)
    else:
        optimal_threshold = None
        max_value = total_value
    
    business_metrics = {
        'Total Business Value': total_value,
        'ROI (%)': roi,
        'Value per Prediction': total_value / len(y_true),
        'Optimal Threshold': optimal_threshold,
        'Maximum Achievable Value': max_value
    }
    
    return business_metrics

# Example usage with business context
business_results = business_impact_analysis(y_true, y_pred, 
                                          cost_matrix={
                                              'true_positive_value': 500,  # High-value customer identified
                                              'false_positive_cost': -50,  # Marketing cost wasted
                                              'false_negative_cost': -300, # Lost high-value customer
                                              'true_negative_value': 0     # No cost/benefit
                                          })

print("Business Impact Analysis:")
print("=" * 30)
for metric, value in business_results.items():
    if metric.endswith('(%)'):
        print(f"{metric}: {value:.1f}%")
    elif isinstance(value, (int, float)) and value is not None:
        print(f"{metric}: ${value:,.2f}")
    else:
        print(f"{metric}: {value}")

Model Comparison Framework

When comparing multiple models, systematic evaluation prevents biased decisions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
import pandas as pd

def compare_models(models_dict, X, y, cv=5, scoring_metrics=None):
    """Comprehensive model comparison framework"""
    
    if scoring_metrics is None:
        scoring_metrics = ['accuracy', 'precision', 'recall', 'f1', 'roc_auc']
    
    results = []
    
    for model_name, model in models_dict.items():
        print(f"Evaluating {model_name}...")
        
        # Cross-validation
        cv_results = cross_validate(model, X, y, cv=cv, scoring=scoring_metrics,
                                   return_train_score=True, n_jobs=-1)
        
        # Compile results
        model_results = {'Model': model_name}
        
        for metric in scoring_metrics:
            test_scores = cv_results[f'test_{metric}']
            train_scores = cv_results[f'train_{metric}']
            
            model_results[f'{metric}_mean'] = np.mean(test_scores)
            model_results[f'{metric}_std'] = np.std(test_scores)
            model_results[f'{metric}_overfitting'] = np.mean(train_scores) - np.mean(test_scores)
        
        # Fit time
        model_results['fit_time_mean'] = np.mean(cv_results['fit_time'])
        model_results['score_time_mean'] = np.mean(cv_results['score_time'])
        
        results.append(model_results)
    
    # Create comparison DataFrame
    comparison_df = pd.DataFrame(results)
    
    # Ranking
    rankings = {}
    for metric in scoring_metrics:
        rankings[f'{metric}_rank'] = comparison_df[f'{metric}_mean'].rank(ascending=False)
    
    rankings_df = pd.DataFrame(rankings)
    rankings_df['Model'] = comparison_df['Model']
    rankings_df['Average_Rank'] = rankings_df[[col for col in rankings_df.columns if col.endswith('_rank')]].mean(axis=1)
    
    return comparison_df, rankings_df

# Example model comparison
models = {
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
    'Gradient Boosting': GradientBoostingClassifier(random_state=42),
    'Logistic Regression': LogisticRegression(random_state=42, max_iter=1000),
    'SVM': SVC(probability=True, random_state=42)
}

comparison_results, rankings = compare_models(models, X, y)

print("\nModel Performance Comparison:")
print("=" * 50)
print(comparison_results[['Model', 'accuracy_mean', 'accuracy_std', 'f1_mean', 'f1_std', 'roc_auc_mean']].round(4))

print("\nModel Rankings:")
print("=" * 30)
print(rankings[['Model', 'Average_Rank']].sort_values('Average_Rank'))

Key Principles for Model Evaluation

Through years of developing production ML systems for sports analytics and customer intelligence, these principles have proven essential:

  1. Match metrics to business objectives - Sports prediction models need profitability metrics, customer intelligence needs retention metrics, not just accuracy

  2. Always use multiple metrics - No single metric captures the full picture. We evaluate sports models on accuracy, calibration, profitability, and edge case performance

  3. Validate on truly unseen data - Your final evaluation should be on data that never influenced any modeling decisions. For sports, this means holdout seasons, not just random samples

  4. Consider operational constraints - Real-time sports predictions must complete in under 50ms during peak traffic. Beautiful models are useless if they can’t meet production SLAs

  5. Think about failure modes - In customer intelligence, false positives waste marketing budget; false negatives lose high-value customers. These costs aren’t equal

  6. Monitor performance over time - Sports strategies evolve, customer behavior shifts. Models degrade faster than you think—build monitoring from day one

Conclusion

Building ML systems that work in production requires evaluation approaches that go far beyond academic metrics. The frameworks presented here have been battle-tested in high-stakes environments where model failures have immediate business consequences.

The key insight: Evaluation isn’t just about measuring performance—it’s about understanding failure modes, business impact, and operational constraints before they become production problems.

Whether you’re building sports prediction models, customer intelligence platforms, or any enterprise ML system, invest heavily in comprehensive evaluation. It’s the difference between models that work in notebooks and models that create business value.

Ali Mahmoudi is Research Lead at a leading Australian sports technology company, where he architects enterprise ML systems for sports analytics and customer intelligence. He holds a PhD in Statistics from the University of Melbourne and has published research in computational biology.

Questions about production ML evaluation? Connect on LinkedIn or email me—I’m always happy to discuss building systems that actually work in production.