Is there an efficient way to add two normal distributions?

Let's say I have two multivariate normal distributions with means m1 and m2, and covariance matrices C1 and C2, and that the number of elements in each distribution is n1 and n2.

The mean of the sum of the distributions would then be

(m1 * n1 + m2 * n2) / (n1 + n2)

But is there an efficient way to calulate the new covariance matrix, other than iterating over all of the points of the two distributions? I have a feeling there should be, but I can't see it now.

Let's say I have two multivariate normal distributions with means m1 and m2, and covariance matrices C1 and C2, and that the number of elements in each distribution is n1 and n2.

The mean of the sum of the distributions would then be

(m1 * n1 + m2 * n2) / (n1 + n2)

But is there an efficient way to calulate the new covariance matrix, other than iterating over all of the points of the two distributions? I have a feeling there should be, but I can't see it now.

x1 = n1 x 1 vector of variables in first distribution

x2 = n2 x 1 vector

C1 = n1 x n1 covariance matrix

C2 = n2 x n2 covariance matrix

You need to figure out the cross-covariance values between elements of x1 and elements of x2. If you can assume that the values are independent, then XC12 is all zeros.

XC12 = n1 x n2 cross-covariance matrix between x1 and x2

x12 = transpose(x1 x2)

C12 is the (n1+n2)x(n1+n2) covariance matrix for x12

C12 = [ C1 XC12 ]

[ transpose(XC12) C2 ]

Both distributions have elements that are three-dimensional vectors (actually, they represent groups of points in 3D-space). So both C1, C2, and C12 are 3x3 matrices, right? What I'm after is to compute the covariance for the collection of all points in the two distributions. So the dimensionality is still the same, it's only the number of points that is bigger.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.

I think it's doable.

variance(x) = b * ( sum(x^2) - n*sum(x)^2 )

There are two ways to calculate variance and covariance, the population method using b=1/n, and the unbiased method using b=1/(n-1). See http://en.wikipedia.org/wiki/Variance, section "Population variance and sample variance". I don't know which method you used, but you need to know it to dig the sum(x^2) value back out from your variance calculations.

sum(x) = n*mean(x)

sum(x^2) = variance(x)/b + n*sum(x)^2 = variance(x)/b + n^2*mean(x)^2

If x12 is the combined set of x1 and x2, then

sum(x12) = sum(x1) + sum(x2)

sum(x12^2) = sum(x1^2) + sum(x2^2)

variance(x12) = b*(sum(x12^2) - n*sum(x12)^2)

In code:

Given n1 (# of points in first collection), v1 (variance of a variable in the first collection), m1 (mean of that same variable), and similar n2,v2,m2 from the second collection:

sumsq1 = v1/b + n1*(n1*m1)^2 = v1/b+n1^3*m1^2

sumsq2 = v2/b + n2^3*m2^2

n12 = n1 + n2

sumsq12 = sumsq1 + sumsq2

m12 = (n1 * m1 + n2 * m2) / n12

v12 = b*(sumsq12 - n12^2 * m12^2)

That's how you do variance for each dimension. Now for covariance:

cov(x,y) = E((x-mx)*(y-my))

= E(xy - y*mx - x*my + mx*my)

= E(xy) - mx*E(y) - E(x)*my + mx*my

= E(xy) - mx*my - mx*my + mx*my

= E(xy) - mx*my

cov(x,y) = sum(x*y)/n - sum(x)*sum(y)/n^2

sum(x*y) = n*cov(x,y) + sum(x)*sum(y)/n

Like before

x12 is x1 and y1 combined, likewise for y12.

sum(x12*y12) = sum(x1*y1) + sum(x2*y2)

In code:

Given n1 (# of points in first collection), cxy1 (covariance(x,y) from first collection), mx1 (mean of x from first collection), my1 (mean of y from first collection), and similar n2,cxy2,mx2,my2 for second collection:

sumxy1 = n1 * cxy1 + mx1 * my1 * n1

sumxy2 = n2 * cxy2 + mx2 * my2 * n2

sumxy12 = sumxy1 + sumxy2

cxy12 = sumxy12 / n12 - mx12*my12

That's the pattern.