Efficient moving window average implementation

Hi *,

I have the attached code, which I use to compute a moving window average. However, it is rather slow. Is there a more efficient way to implement it? Is there a built in function for this kind of thing.

outputTabs and outputTrel are waht I want moving averages of.

I am using Octave, however as it is so similar to Matlab any suggestions for Matlab would also be useful, and I may be able to implement it in Octave as well.

Many thanks,
James
``````inc         = 1000;
windowWidth = 1000;
beginning   = 1;%finish-windowWidth-inc;
k = 1;
for start=beginning:inc:finish-windowWidth
% Find window of absolute spikes
window1    = find(outputTabs(numberIndices{n})<=start+windowWidth);
window2    = find(outputTabs(numberIndices{n}(window1))>=start);
window 	   = outputTrel(numberIndices{n}(window1(window2)));
if numel(window)>0
avg{n}(k)      = mean(window);
error{n}(k)    = std(window);
numInPat{n}(k) = (100/(numel(window)))*(numel(find(window>0)));
else
avg{n}(k)      = 0;
error{n}(k)    = 0;
numInPat{n}(k) = 0;
end
k = k + 1;
end
``````
LVL 4
Who is Participating?

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Commented:
Are the avg{n}(k) arrays built dynamically or you you specify the size beforehand? I know MATLAB lets you just grow arrays but it slows it down a lot. If you allocate the arrays for their max size before you start the loop, it will run much faster.

MATLAB has a function that gets the mean and standard deviation at the same time. If you have a function like that it could speed it up a bit.
\Commented:
For an average of n values, define window:
``````     avg_i   = (x_i   + x_i+1 + ... + x_n+i-1)/n
``````
Then, the next window moving one element forward is:
``````     avg_i+1 = (        x_i+1 + ... + x_n+i-1 + x_n+i)/n
``````
Notice that:
``````     avg_i+1 = avg_i + (x_n+i - x_i)/n
``````
Commented:
For standard deviation remember that stddev = sqrt(mean(x^2) - mean(x)^2) so if you use phoffric's excellent suggestion to speed up the means, also track the mean of the squares so you can get the stddev just as quickly.
In case the above formula is diffucult to read:
standard deviation = square root of (the average of the X^2 values minus the square of the mean)
Commented:
One way to calculate moving average without for loop is using build in MATLAB FILTER function:
http://www.mathworks.com/help/techdoc/ref/filter.html
See the Example section.

However, this works if the signal is spaced regularly. It looks like this is not your case since you have to find outputTabs values in a current window. I don't think you can avoid loops.

One possible way to speed this up is to avoid running find operation across whole outputTabs vector all the times.

First, store temporary vectors into a variables. Indexing and other operations do take time. For example, why you have to index outputTabs(numberIndices{n}) at every loop?

Second, make sure the vector is sorted. Since your segments for moving average are not overlapping, you can take advantage removing those values that has been used already.

Something like this (I didn't test it, sorry for possible mistakes):
``````tmpTabs = outputTabs(numberIndices{n});
tmpTrel = outputTrel(numberIndices{n});
lastfound = 0;
for start=beginning:inc:finish-windowWidth
winidx = find(tmpTabs(lastfound=1:end)>=start && tmpTabs(lastfound=1:end)<=start+windowWidth);
if numel(winidx)>0
window = tmpTrel(winidx+lastfound);
lastfound = winidx(end);
...
else
...
end
end
``````

If you want more help please make your code runnable defining all variables and providing small data to test.
Commented:
I forgot, that you are running Octave, not MATLAB. As for FILTER function, I think Octave has it. Hope other thoughts will be also compatible.
Commented:
as it is a convolution why not using the conv function?

conv() and kernel with (ones(windowsize)/windowsize

I've tested it out and its quite fast:

``````len = [10000,300000,2000000];
windowsize = 2000;

% convolution
for turn=1:numel(len)
randomsignal = rand(len(turn),1);
tic
kernel = ones(windowsize,1)/windowsize;
movingavg_c = conv(randomsignal,kernel);
toc
end
``````
``````Elapsed time is 0.012252 seconds.
Elapsed time is 0.330076 seconds.
Elapsed time is 2.189201 seconds.
``````
Author Commented:
Many thanks for all your suggestions.

phorric, your suggestions is very elegant however for simplicity I think Kendor's suggestion lends itself better to Octave/Matlab. If I was doing this in C (and I may be) this this would be perfect (and I shall remember it!).

yuk99: as you point out the code I already have could be made more streamline, however it still seems to be going quite slow do to the loops.

Kendor: is there a way to do the std like this as well? I can't think how to form the kernel.

Many thanks,
James
\Commented:
James,

My implementation in matlab (years ago) was extremely easy. A convolution has many more operations. You should try to implement the moving average suggestion in octave, and you should see a significant performance improvement.

Paul
Commented:
James,

I would then go with the function filter... something as proposed by Scott:
``````% the mean using filter
mean = filter(ones(1,n)/n,1,signal);

% moving SD using filter
signal2 = filter(ones(1,n),1,signal.^2);
variance=(signal2-n*mean.^2)/(n-1);
stddev=sqrt(variance);

% remember: the last n-1 values ar invalid...
mean(1:(n-1))=[];
stddev(1:(n-1))=[];
``````

or with convolutions this would somehow look like:
``````    kern = ones(N,1);
meanT = conv(signal,kern);

squared = signal.*signal;
stddev=sqrt((conv(squared,kern)-(meanT.^2)/N)/(N-1));
mean = meanT/N;
% remember here: you have done a convolution this adds padding - so the first N-1 values as well as the last N-1 values are not valid...
stddev(1:(N-1))=[]
% etc..
``````

Experts Exchange Solution brought to you by

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Commented:
remains to say that the implementation you choose depends on how good/fast matlab/octave has implemented either conv or filter - I believe they must be super fast as they are often used. Otherwise you better go with phoffrics style of implementation.

you find some pseudo codeand nice explanation here:
http://www.dspguide.com/ch2/2.htm
Author Commented:
Many thanks to all for suggestions.

Cheers,
James
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Programming Languages-Other

From novice to tech pro — start learning today.