Efficient moving window average implementation

Hi *,

I have the attached code, which I use to compute a moving window average. However, it is rather slow. Is there a more efficient way to implement it? Is there a built in function for this kind of thing.

outputTabs and outputTrel are waht I want moving averages of.

I am using Octave, however as it is so similar to Matlab any suggestions for Matlab would also be useful, and I may be able to implement it in Octave as well.

Many thanks,
James
inc         = 1000;
			windowWidth = 1000;
			beginning   = 1;%finish-windowWidth-inc;
			k = 1;
			for start=beginning:inc:finish-windowWidth
				% Find window of absolute spikes
				window1    = find(outputTabs(numberIndices{n})<=start+windowWidth);
				window2    = find(outputTabs(numberIndices{n}(window1))>=start);
				window 	   = outputTrel(numberIndices{n}(window1(window2)));
				if numel(window)>0
					avg{n}(k)      = mean(window);
					error{n}(k)    = std(window);
					numInPat{n}(k) = (100/(numel(window)))*(numel(find(window>0)));
				else
					avg{n}(k)      = 0;
					error{n}(k)    = 0;
					numInPat{n}(k) = 0;
				end
				k = k + 1;
			end

Open in new window

LVL 4
James_h1023Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

TommySzalapskiCommented:
Are the avg{n}(k) arrays built dynamically or you you specify the size beforehand? I know MATLAB lets you just grow arrays but it slows it down a lot. If you allocate the arrays for their max size before you start the loop, it will run much faster.

MATLAB has a function that gets the mean and standard deviation at the same time. If you have a function like that it could speed it up a bit.
0
phoffricCommented:
For an average of n values, define window:
     avg_i   = (x_i   + x_i+1 + ... + x_n+i-1)/n

Open in new window

Then, the next window moving one element forward is:
     avg_i+1 = (        x_i+1 + ... + x_n+i-1 + x_n+i)/n

Open in new window

Notice that:
     avg_i+1 = avg_i + (x_n+i - x_i)/n

Open in new window

0
TommySzalapskiCommented:
For standard deviation remember that stddev = sqrt(mean(x^2) - mean(x)^2) so if you use phoffric's excellent suggestion to speed up the means, also track the mean of the squares so you can get the stddev just as quickly.
In case the above formula is diffucult to read:
standard deviation = square root of (the average of the X^2 values minus the square of the mean)
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

yuk99Commented:
One way to calculate moving average without for loop is using build in MATLAB FILTER function:
http://www.mathworks.com/help/techdoc/ref/filter.html
See the Example section.

However, this works if the signal is spaced regularly. It looks like this is not your case since you have to find outputTabs values in a current window. I don't think you can avoid loops.

One possible way to speed this up is to avoid running find operation across whole outputTabs vector all the times.

First, store temporary vectors into a variables. Indexing and other operations do take time. For example, why you have to index outputTabs(numberIndices{n}) at every loop?

Second, make sure the vector is sorted. Since your segments for moving average are not overlapping, you can take advantage removing those values that has been used already.

Something like this (I didn't test it, sorry for possible mistakes):
tmpTabs = outputTabs(numberIndices{n});
tmpTrel = outputTrel(numberIndices{n});
lastfound = 0;
for start=beginning:inc:finish-windowWidth
    winidx = find(tmpTabs(lastfound=1:end)>=start && tmpTabs(lastfound=1:end)<=start+windowWidth);
    if numel(winidx)>0
        window = tmpTrel(winidx+lastfound);
        lastfound = winidx(end);
        ...
    else
        ...
    end
end

Open in new window


If you want more help please make your code runnable defining all variables and providing small data to test.
0
yuk99Commented:
I forgot, that you are running Octave, not MATLAB. As for FILTER function, I think Octave has it. Hope other thoughts will be also compatible.
0
KendorCommented:
as it is a convolution why not using the conv function?

conv() and kernel with (ones(windowsize)/windowsize

I've tested it out and its quite fast:

len = [10000,300000,2000000];
windowsize = 2000; 



% convolution
for turn=1:numel(len)
    randomsignal = rand(len(turn),1);
    tic
    kernel = ones(windowsize,1)/windowsize;
    movingavg_c = conv(randomsignal,kernel);
    toc
end

Open in new window

Elapsed time is 0.012252 seconds.
Elapsed time is 0.330076 seconds.
Elapsed time is 2.189201 seconds.

Open in new window

0
James_h1023Author Commented:
Many thanks for all your suggestions.

phorric, your suggestions is very elegant however for simplicity I think Kendor's suggestion lends itself better to Octave/Matlab. If I was doing this in C (and I may be) this this would be perfect (and I shall remember it!).

yuk99: as you point out the code I already have could be made more streamline, however it still seems to be going quite slow do to the loops.

Kendor: is there a way to do the std like this as well? I can't think how to form the kernel.

Many thanks,
James
0
phoffricCommented:
James,

My implementation in matlab (years ago) was extremely easy. A convolution has many more operations. You should try to implement the moving average suggestion in octave, and you should see a significant performance improvement.

   Paul
0
KendorCommented:
James,

I would then go with the function filter... something as proposed by Scott:
% the mean using filter
mean = filter(ones(1,n)/n,1,signal);

% moving SD using filter
signal2 = filter(ones(1,n),1,signal.^2);
variance=(signal2-n*mean.^2)/(n-1);
stddev=sqrt(variance);

% remember: the last n-1 values ar invalid...
mean(1:(n-1))=[];
stddev(1:(n-1))=[];

Open in new window



or with convolutions this would somehow look like:
    kern = ones(N,1);
    meanT = conv(signal,kern);

    squared = signal.*signal;    
    stddev=sqrt((conv(squared,kern)-(meanT.^2)/N)/(N-1));
    mean = meanT/N;
    % remember here: you have done a convolution this adds padding - so the first N-1 values as well as the last N-1 values are not valid...
    stddev(1:(N-1))=[] 
    % etc..

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
KendorCommented:
remains to say that the implementation you choose depends on how good/fast matlab/octave has implemented either conv or filter - I believe they must be super fast as they are often used. Otherwise you better go with phoffrics style of implementation.

you find some pseudo codeand nice explanation here:
http://www.dspguide.com/ch2/2.htm
0
James_h1023Author Commented:
Many thanks to all for suggestions.

Cheers,
James
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Programming Languages-Other

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.