Implications for computational statistics and machine learning

Oct 13, 2025 by wbrum.org 9 views

Introduction

The Wasserstein distance, also known as the Earth Mover's Distance (EMD), is a measure of the difference between two probability distributions. It is a popular metric in computational statistics and machine learning, particularly in the field of generative models and data analysis. In theory, the Wasserstein distance between two samples drawn from the same distribution should be zero, as there is no difference between the two distributions. However, in practice, the Wasserstein distance is often far away from zero, even when the samples are drawn from the same distribution. In this article, we will explore the reasons behind this phenomenon and discuss the implications for computational statistics and machine learning.

What is the Wasserstein distance?

The Wasserstein distance is a measure of the difference between two probability distributions. It is defined as the minimum cost of transforming one distribution into another. The cost of transformation is measured by the amount of "work" required to move the mass of one distribution to match the mass of the other distribution. The Wasserstein distance is a powerful tool for comparing distributions, as it takes into account the shape and location of the distributions, as well as their scale and orientation.

Why should the Wasserstein distance be zero when samples are drawn from the same distribution?

In theory, the Wasserstein distance between two samples drawn from the same distribution should be zero, as there is no difference between the two distributions. This is because the two samples are drawn from the same underlying distribution, and therefore, they should have the same shape, location, scale, and orientation. The Wasserstein distance is a measure of the difference between two distributions, and if the two distributions are the same, then the difference should be zero.

Why is the Wasserstein distance far away from zero when samples are drawn from the same distribution?

Despite the theoretical expectation that the Wasserstein distance should be zero when samples are drawn from the same distribution, in practice, the Wasserstein distance is often far away from zero. There are several reasons for this phenomenon:

1. Sampling variability

When we draw samples from a distribution, we are subject to sampling variability. This means that the samples we draw may not be representative of the underlying distribution, and therefore, the Wasserstein distance between the samples may not be zero. Sampling variability is a fundamental aspect of statistical inference, and it is a major contributor to the non-zero Wasserstein distance.

2. Numerical instability

The Wasserstein distance is a complex metric that requires numerical computation. In practice, the computation of the Wasserstein distance can be unstable, particularly when the samples are drawn from a distribution with a complex shape or when the sample size is small. Numerical instability can lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

3. Computational approximations

The Wasserstein distance is often computed using approximations, such as the Kantorovich-Rubinstein duality or the Sinkhorn algorithm. These approximations can introduce errors and lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

4. Distributional differences

Even when the samples are drawn from the same distribution, there may be subtle differences between the two distributions. For example, the two distributions may have different modes or different tail behaviors. These differences can lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

5. Statistical bias

The Wasserstein distance is a statistical metric that is subject to bias. This means that the Wasserstein distance may not be an unbiased estimator of the true difference between the two distributions. Statistical bias can lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

Implications for computational statistics and machine learning

The non-zero Wasserstein distance between samples drawn from the same distribution has important implications for computational statistics and machine learning. In particular:

1. Model evaluation

The Wasserstein distance is often used as a metric for evaluating the performance of generative models. However, the non-zero Wasserstein distance between samples drawn from the same distribution can lead to biased model evaluation. This means that the model may be evaluated as performing worse than it actually is.

2. Data analysis

The Wasserstein distance is often used in data analysis to compare the distribution of different variables. However, the non-zero Wasserstein distance between samples drawn from the same distribution can lead to incorrect conclusions about the distribution of the variables.

3. Machine learning

The Wasserstein distance is often used in machine learning to compare the distribution of different classes or clusters. However, the non-zero Wasserstein distance between samples drawn from the same distribution can lead to biased classification or clustering results.

Conclusion

In conclusion, the Wasserstein distance is a powerful metric for comparing distributions, but it is not always zero when samples are drawn from the same distribution. The non-zero Wasserstein distance is due to several factors, including sampling variability, numerical instability, computational approximations, distributional differences, and statistical bias. These factors have important implications for computational statistics and machine learning, and they must be taken into account when using the Wasserstein distance as a metric.

Simulation Code

To illustrate the non-zero Wasserstein distance between samples drawn from the same distribution, we can use the following Python code:

import numpy as np
from scipy.stats import norm
from scipy.optimize import linear_sum_assignment
from scipy.spatial.distance import cdist
def distribution(x):
return norm.pdf(x, loc=0, scale=1)
np.random.seed(0)
n_samples = 1000
x1 = np.random.normal(loc=0, scale=1, size=n_samples)
x2 = np.random.normal(loc=0, scale=1, size=n_samples)
def wasserstein_distance(x1, x2):
n1, n2 = len(x1), len(x2)
cost_matrix = cdist(x1[:, None], x2[None, :])
row_ind, col_ind = linear_sum_assignment(cost_matrix)
return cost_matrix[row_ind, col_ind].sum() / n1
wasserstein_distance_value = wasserstein_distance(x1, x2)
print("Wasserstein distance:", wasserstein_distance_value)

This code draws two samples from a normal distribution and computes the Wasserstein distance between the two samples. The result is a non-zero Wasserstein distance, even though the two samples are drawn from the same distribution.

References

[1] M. Cuturi, "Sinkhorn distances: Lightspeed computation of optimal transport," in Advances in Neural Information Processing Systems, 2013.
[2] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein generative adversarial networks," in Proceedings of the International Conference on Machine Learning, 2017.
[3] P. M. Clarkson, "Lower bounds for the size of sample spaces," in Proceedings of the 30th Annual Symposium on Foundations of Computer Science, 1989.
Q&A: Understanding the Wasserstein Distance =============================================

Q: What is the Wasserstein distance, and why is it important in machine learning?

A: The Wasserstein distance, also known as the Earth Mover's Distance (EMD), is a measure of the difference between two probability distributions. It is a powerful tool in machine learning, particularly in generative models, data analysis, and clustering. The Wasserstein distance is important because it takes into account the shape and location of the distributions, as well as their scale and orientation.

Q: Why is the Wasserstein distance not always zero when samples are drawn from the same distribution?

A: The Wasserstein distance is not always zero when samples are drawn from the same distribution because of several factors, including sampling variability, numerical instability, computational approximations, distributional differences, and statistical bias. These factors can lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

Q: What is sampling variability, and how does it affect the Wasserstein distance?

A: Sampling variability refers to the fact that the samples we draw from a distribution may not be representative of the underlying distribution. This means that the Wasserstein distance between the samples may not be zero, even if the samples are drawn from the same distribution. Sampling variability is a fundamental aspect of statistical inference and is a major contributor to the non-zero Wasserstein distance.

Q: What is numerical instability, and how does it affect the Wasserstein distance?

A: Numerical instability refers to the fact that the computation of the Wasserstein distance can be unstable, particularly when the samples are drawn from a distribution with a complex shape or when the sample size is small. This can lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

Q: What are computational approximations, and how do they affect the Wasserstein distance?

A: Computational approximations refer to the fact that the Wasserstein distance is often computed using approximations, such as the Kantorovich-Rubinstein duality or the Sinkhorn algorithm. These approximations can introduce errors and lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

Q: What are distributional differences, and how do they affect the Wasserstein distance?

A: Distributional differences refer to the fact that even when the samples are drawn from the same distribution, there may be subtle differences between the two distributions. For example, the two distributions may have different modes or different tail behaviors. These differences can lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

Q: What is statistical bias, and how does it affect the Wasserstein distance?

A: Statistical bias refers to the fact that the Wasserstein distance is a statistical metric that is subject to bias. This means that the Wasserstein distance may not be an unbiased estimator of the true difference between the two distributions. Statistical bias can lead to a non-zero Wasserstein distance, even when the samples are drawn from the same distribution.

Q: How can I use the Wasserstein distance in machine learning?

A: The Wasserstein distance can be used in machine learning in a variety of ways, including:

Generative models: The Wasserstein distance can be used to evaluate the performance of generative models, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs).
Data analysis: The Wasserstein distance can be used to compare the distribution of different variables, such as the distribution of ages or incomes.
Clustering: The Wasserstein distance can be used to cluster data points based on their distribution.

Q: What are some common applications of the Wasserstein distance?

A: The Wasserstein distance has a wide range of applications, including:

Image processing: The Wasserstein distance can be used to compare the distribution of pixel values in images.
Speech recognition: The Wasserstein distance can be used to compare the distribution of acoustic features in speech signals.
Recommendation systems: The Wasserstein distance can be used to compare the distribution of user preferences in recommendation systems.

Q: What are some common challenges when using the Wasserstein distance?

A: Some common challenges when using the Wasserstein distance include:

Computational complexity: The computation of the Wasserstein distance can be computationally expensive, particularly for large datasets.
Numerical instability: The computation of the Wasserstein distance can be unstable, particularly when the samples are drawn from a distribution with a complex shape or when the sample size is small.
Statistical bias: The Wasserstein distance is a statistical metric that is subject to bias, which can lead to incorrect conclusions about the distribution of the variables.