Why Percentile Errors Matter

Oct 13, 2025 by wbrum.org 12 views

Introduction

In machine learning, validation metrics are crucial for evaluating the performance of a model. However, traditional metrics such as mean absolute error (MAE) or mean squared error (MSE) may not provide a complete picture of a model's performance. Large errors can significantly impact the overall performance of a model, and ignoring them can lead to overconfidence in the model's predictions. In this article, we will discuss how to add percentile errors to validation metrics for W&B (Weights & Biases), a popular platform for tracking and visualizing machine learning experiments.

Why Percentile Errors Matter

Percentile errors provide a more comprehensive understanding of a model's performance by highlighting the distribution of errors. By tracking the 95th and 5th percentile of normalized errors, for example, you can identify the largest and smallest errors, respectively. This information can be particularly useful in identifying outliers and improving the robustness of the model.

Current Validation Metrics

The current validation metrics in the base_model.py file include:

def validate(self, data):
    # ...
    metrics = {
        'mae': self.mae,
        'mse': self.mse,
        'rmse': self.rmse,
        'mape': self.mape,
        'r2': self.r2,
    }
    # ...

Adding Percentile Errors to Validation Metrics

To add percentile errors to the validation metrics, we can modify the validate function to include the following code:

import numpy as np

def validate(self, data):
    # ...
    metrics = {
        'mae': self.mae,
        'mse': self.mse,
        'rmse': self.rmse,
        'mape': self.mape,
        'r2': self.r2,
    }

    # Calculate percentile errors
    errors = np.array([self.mae, self.mse, self.rmse, self.mape, self.r2])
    percentiles = [0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.98]

    for percentile in percentiles:
        metric_name = f'p{int(percentile*100)}_mae'
        metrics[metric_name] = np.percentile(errors, percentile*100)

    # Save metrics to W&B
    self.log_metrics(metrics)
    # ...

In this modified code, we first calculate the errors for each metric and store them in a numpy array. We then define a list of percentiles to calculate, including the 5th, 10th, 25th, 50th, 75th, 90th, 95th, and 98th percentiles. For each percentile, we calculate the corresponding metric by using the np.percentile function and store it in the metrics dictionary. Finally, we save the updated metrics to W&B using the log_metrics function.

Example Use Case

To demonstrate the effectiveness of adding percentile errors to validation metrics, let's consider an example use case. Suppose we have a regression model that predicts house prices based on several features. We want to evaluate the model's performance using the mean absolute error (MAE) and the 95th percentile of normalized errors.

# Define the model and data
model = MyRegressionModel()
data = pd.DataFrame({'feature1': [1, 2, 3], 'feature2': [4, 5, 6], 'target': [10, 20, 30]})

# Train and validate the model
model.train(data)
metrics = model.validate(data)

# Print the metrics
print(metrics)

In this example, the output will include the MAE and the 95th percentile of normalized errors, providing a more comprehensive understanding of the model's performance.

Conclusion

Q: What are percentile errors, and why are they important?

A: Percentile errors provide a more comprehensive understanding of a model's performance by highlighting the distribution of errors. By tracking the 95th and 5th percentile of normalized errors, for example, you can identify the largest and smallest errors, respectively. This information can be particularly useful in identifying outliers and improving the robustness of the model.

Q: How do I calculate percentile errors?

A: To calculate percentile errors, you can use the np.percentile function in Python. This function takes an array of values and a percentile value as input and returns the value at that percentile. For example, to calculate the 95th percentile of an array of errors, you can use the following code:

import numpy as np

errors = np.array([1, 2, 3, 4, 5])
percentile = 0.95
p95_error = np.percentile(errors, percentile*100)
print(p95_error)

Q: How do I add percentile errors to my validation metrics?

A: To add percentile errors to your validation metrics, you can modify the validate function in your model to include the following code:

import numpy as np

def validate(self, data):
    # ...
    metrics = {
        'mae': self.mae,
        'mse': self.mse,
        'rmse': self.rmse,
        'mape': self.mape,
        'r2': self.r2,
    }

    # Calculate percentile errors
    errors = np.array([self.mae, self.mse, self.rmse, self.mape, self.r2])
    percentiles = [0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.98]

    for percentile in percentiles:
        metric_name = f'p{int(percentile*100)}_mae'
        metrics[metric_name] = np.percentile(errors, percentile*100)

    # Save metrics to W&B
    self.log_metrics(metrics)
    # ...

Q: What are some common percentile errors to track?

A: Some common percentile errors to track include:

5th percentile: The smallest error
10th percentile: The smallest 10% of errors
25th percentile: The smallest 25% of errors
50th percentile: The median error
75th percentile: The largest 25% of errors
90th percentile: The largest 10% of errors
95th percentile: The largest 5% of errors
98th percentile: The largest 2% of errors

Q: How do I interpret percentile errors?

A: To interpret percentile errors, you can use the following guidelines:

If the 95th percentile of errors is high, it may indicate that the model is overfitting to the training data.
If the 5th percentile of errors is high, it may indicate that the model is underfitting to the training data.
If the 50th percentile of errors is high, it may indicate that the model is not performing well on average.

Q: Can I use percentile errors with other metrics?

A: Yes, you can use percentile errors with other metrics, such as accuracy, precision, and recall. However, you should be careful when combining multiple metrics, as it can lead to overfitting or underfitting.

Q: How do I visualize percentile errors?

A: To visualize percentile errors, you can use a histogram or a box plot. A histogram can show the distribution of errors, while a box plot can show the median and interquartile range (IQR) of errors.