Speed up Calcs using INDIRECT function.

I can't help you with your existing formula, but I assume from column E's heading you are calculating STDEV.S. Are you aware that there's a built-in function for that? Here's an article from Microsoft that shows how to use it.

Martin, I was using an array which that article alludes and works fine on a single series of variables but when using an indexed range coupled with 100,000+ rows it is not practical. Thanks anyway.

Ian

I've not had a chance to look at your workbook properly but I was wondering why you think using INDIRECT will speed things up.

Are you using formulas to determine the ranges to apply other formulas to?

I use that sample formula for another purpose and the INDIRECT function speeds it up dramatically

Here's a couple of links that suggest using INDIRECT might not actually help in speeding up calculation.

Excel Performance

OEvil Functions

Norie I can assure you it does for the purposes I use it for.
I just need someone to add the INDIRECT function to the formula in E column.

Ian

Which part of the formula do you want to change to use INDIRECT?

not changing, just adding.

...when using an indexed range coupled with 100,000+ rows it is not practical.

If that's because you would have to manually add formulas to multiple sections of the column then perhaps I could write code for you that would automate the process of adding the STDEV.S formulas.

What are your thoughts on the sample formula in column 'M' of the worksheet ?

I assume you are not asking that question of me.

Can you explain why "...it is not practical"?

It was not practical because the methodology was meant for non indexed series and certainly not 6 figure rows of data

I was asking you .... are you able to add the INDIRECT as shown in the sample I provided ??
If not no probs.

Yes, probably. Please show me what the formula for E2 and E3 should be.

I don't understand your question.
I'm asking YOU how the formula in column E should be ?
Please look at the formula I am using in column E and also the SAMPLE ONLY formula in column M
and see if you could use the formulas in columns G:H to help  speed up the calculation process.

As an example the below formula which is the sample formula before speeding modification was added
=IF(WF5="","",SUMPRODUCT((A$5:$A$1000=A5)*(B$5:B$1000>B5))+1)
Then after added it look like this.
IF(ISBLANK(A2),"",IF(AND(A2=A1,K2=K1),L1,SUMPRODUCT((INDIRECT("$A"&G2&":$A"&H2)=A2)*(K2<INDIRECT("$K"&G2&":$K"&H2)))+1))))
I changed the columns to G and H to show how those two columns can be used to assist.

An expert on the forum modified the formula for me a long time ago and speeded up the process immensely.
Please don't ask me questions about the formula as I won't be able to answer them.
If you cannot combine them no problem.

I have code that generates the following formula for E2.

=IF(A2="","",IF(J2=0,0,IF(ISBLANK(A2),"",IF(AND(A2=A1,K2=K1),L1,SUMPRODUCT((INDIRECT("$A"&G2&":$A"&H2)=A2)*(K2<INDIRECT("$K"&G2&":$K"&H2)))+1))))

I used the formula in M2 as a template but the result of the new formula in E2 is a #REF! error which may be due to my assumptions which are that in "translating" the formula in M2, column 'J' in column M should be column 'B' in E2, and similarly that 'K' s/b 'C', and 'L' s/b 'D'. Hopefully you or Norie can help me correct the formula.

You have used the whole of the sample formula. Remember the sample formula is just that, a sample. The formula that needs modifyiing is column E which means you keep all of the components like SQRT, SUMIF and COUNTIF and only add INDIRECT and ISBLANK if needed
Ignore column range  J:M
Only use columns  A  E  and D  as well as the helper columns  G and H

The problem is that the sumif and countif formulas in column E use all 120,000 rows.
Including rows above, where there will never be a match, since the calculation is only done on the first occurrence of the Index number.
With indirect you still need to find the end row with a countif.
That would calculate faster if it only looked from actual row and perhaps 100 rows down, and not all 120,000.
The number of rows must be greater than the maximum number of repeated values in Index.
But even using 1,000 is much less than 120,000.

But then the formula in column E could do the same without indirect, only calculate 100 rows down.
Then the formula would be
=IF(D2=0,0,IF(A2=A1,E1,SQRT(SUMIF(A2:A101,A2,D2:D101)/(COUNTIF(A2:A101,A2)-1))))

I have a total of 20 worksheets each one with number of rows varying between 70,000 and 170,000 and forever expanding.
Each worksheet has 87 column of variables making a total of say 100,000 rows * 87 * 20 = 174 million cells and to be updated every three months.
Seems an impossible task in Excel, I will have to look at other options. One I have in mind is with Stata for which I have a licensed copy.

You might also consider Access which is much better than Excel at handling large amounts of data.

True, Access can handle large amounts of data however I doubt very much that it can aid in converting values to z scores.

I found this article and SELECT STDEV(Bonus) FROM SalesPerson; among others on the web.

Based on that article are you able to offer a solution on how it can be done ?

No but I'd bet the information is out there, or ask another question here that includes the Access topic. In my administrative capacity I've added Access to this question so perhaps someone will respond that can answer you last question, because I can't.

Ejgil,  If you refer to the sample data column 'M' you will see that it uses the 2 helper columns G and H Start and End. That is the idea behind the speeding up process. It only uses max number of rows between each series. I want to add those columns G:H to the formula in column 'E' and the way to do it if I'm not mistaken is by adding the INDIRECT function. But if you have an alternative way to add those columns then I'm all ears.

Thanks for all your help Martin it's keeping the thread alive if nothing else.

Ejgil, perfect, thanks a million. It took 2 seconds for 116,000 rows on my blistering fast computer with an i9 processor and 64 gb ram.
That INDIRECT function is really powerful. You have nailed it. I should use it for my other formulas that I use.
Thanks again
Ian
Thanks also to Norie and Martin for your contributions.

Ejgil, I have another formula I would like the INDIRECT condition added to help speed up. It is an array. should I open a new question ?

The functions could be more simple.
Since the range is set with the columns G and H, there is no need for if in the formulas.
H2

=IF(A2="","",IF(A2=A3,H3,ROW()))

Select allOpen in new window

E2

=IF(D2=0,0,IF(A2=A1,E1,SQRT(SUM(INDIRECT("$D$"&$G2&":$D$"&$H2))/(H2-G2))))

Select allOpen in new window

D2

=POWER(B2-C2, 2)

Select allOpen in new window

C2

=IF(A2=A1,C1,SUM(INDIRECT("$B$"&$G2&":$B$"&$H2))/(H2-G2+1))

Select allOpen in new window

Yes, open a new question.
You could make a reference here.

thanks