Solved

COldfusion: Memory allocation failed during query processing: error on CF8 but not on CF7

Posted on 2008-10-31
6
972 Views
Last Modified: 2013-12-24
Hello,
I am having problems performing a task on a CF8 server that works on a CF7 server. The CF7 & CF8 servers are both windows. What I am trying to do is use cfdirectory to get a list of picture files (about 15,000) then insert a record for each file into a table. When I run the cfm template on the CF7 server it eventually finishes, slow but it finishes. When I run the same template on the cf8 server it eventually comes back with this error

[Macromedia][Informix JDBC Driver][Informix]Memory allocation failed during query processing

I looked in the cf8 monitor and the jvm memory peaks to about 120M out of 512. Here are some cf8 settings which, btw, are the same as on the cf7 server

Maximum number of running JRun threads : 50
Maximum number of queued JRun Threads : 1000
Java Virtual Machine Path : D:/ColdFusion8/runtime/jre
Minimum JVM Heap Size (MB): 512        Maximum JVM Heap Size (MB) : 512
ColdFusion Class Path:  "c:\Program Files\IBM\Informix JDBC Driver 2.21.JC6\lib","D:\cfsftp\cfsftp.jar"
JVM Arguments: -server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=128m -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib

Any help or ideas is appreciated!
Regards,
Joe

Here is the code

<CFQUERY datasource="foo" name="delcfd">
      delete from ws_pics
</CFQUERY>

<cfif DirectoryExists("//server/A/B/C/skupics/")>
<cfdirectory directory="//server/A/B/C/skupics/" action="list" name="myD" recurse="no">
<CFLOOP query="myD"> <!--- A --->
<cfif myD.type EQ "Dir" AND  DirectoryExists("//server/A/B/C/skupics/#trim(myD.name)#/")>
<cfdirectory directory="//server/A/B/C/skupics/#trim(myD.name)#/" action="list" name="myD2" filter="a*.jpg|b*.jpg|c*.jpg|d*.jpg|e*.jpg|f*.jpg|g*.jpg|h*.jpg|i*.jpg|j*.jpg|k*.jpg|l*.jpg|m*.jpg|n*.jpg|o*.jpg|p*.jpg|q*.jpg|r*.jpg|s*.jpg|t*.jpg|u*.jpg|v*.jpg|w*.jpg|x*.jpg|y*.jpg|z*.jpg" recurse="no" type="File"> ***** anyone know of a better way to do this ? ****
<CFLOOP query = "myD2" > <!--- B --->
<CFSET variables.nlen = #len(myd2.name)#>
<CFIF nlen GTE 6>
:#myD.name#/#myd2.name#:
<CFQUERY datasource="famousviews" name="instcfd">
insert into ws_pics (num, picfile) values (0, '#trim(myd2.name)#')
</CFQUERY>
</CFIF>
<cfflush> ***print to browser ***
</CFLOOP> <!--- B --->
</cfif>
</CFLOOP> <!--- A --->
...
0
Comment
Question by:ckercher
  • 3
  • 2
6 Comments
 
LVL 16

Expert Comment

by:duncancumming
ID: 22856074
I think this
filter="a*.jpg|b*.jpg|c*.jpg|d*.jpg|e*.jpg|f*.jpg|g*.jpg|h*.jpg|i*.jpg|j*.jpg|k*.jpg|l*.jpg|m*.jpg|n*.jpg|o*.jpg|p*.jpg|q*.jpg|r*.jpg|s*.jpg|t*.jpg|u*.jpg|v*.jpg|w*.jpg|x*.jpg|y*.jpg|z*.jpg"
can just be rewritten as
filter="*.jpg"
unless your intention is to exclude files name 1.jpg etc?  

take out your cfflush to speed things up a bit (although I'm guessing you've kept it in as a tracer while trying to debug this?)

avoid setting an unnecessary variable
<CFSET variables.nlen = #len(myd2.name)#>
<CFIF nlen GTE 6>
becomes
<CFIF len(myd2.name) GTE 6>

also note you don't need the # # around your function call inside a cfset.

in your inner cfdirectory, try setting listinfo="name", as you're only wanting the filenames, although read this article:
http://www.bennadel.com/blog/886-Learning-ColdFusion-8-CFDirectory-Improvements.htm
you could probably just use LIstLast(name, "/") to then extract just the filename part.

in your outer cfdirectory, try setting type="dir", then remove your cfif myD.type EQ "Dir" AND  DirectoryExists if clause.  

Could you redo the whole thing as one cfdirectory with recurse=true, applying your filter to just get the jpgs?  saves you having to do nested cfdirectory.
0
 

Author Comment

by:ckercher
ID: 22866994
Hi duncan...
Thanks for the reply. There are folders and files under /a/b/c that I do not want to process, hence the long filter and multiple cfdirectory loops. Its too bad cfdirectory can not process regex for the filter :(  I'll have to play with the new cfdirectory fields in cf8 to make it more efficient. The odd thing is the cf8 monitor did showed the jvm memory did not fill up which is what I would have expected if that was the issue. Any other ideas aer appreciated.
Joe
0
 
LVL 61

Expert Comment

by:gheist
ID: 22867631
What is java version - i.e java -version?

-XX: parameter may not perform as expected when another JVM is used.

MaxPermSize and PermSize should be same due to bugs in old JVMs (if you use SUN's JVM)

Does it work when you get rid of parameter?
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 

Author Comment

by:ckercher
ID: 22868117
Something I just found to be curious is it always bombs on the insertion of the same file. db spaces and file systems are not filling up.

The JVM is Sun's 1.6.0_04
I have not tried getting rid of the MaxPermSize but that will be my next step.

Thanks!
Joe
0
 
LVL 61

Expert Comment

by:gheist
ID: 22868279
Is there any Informix session identifiable? onstat -g ses

You should be running later JDK http://secunia.com/advisories/product/12878/?task=advisories_2008

I do not see valid reason for -XX tuning unless system does not cope with JVM's memory use well.

Setting -Xms and -Xmx to same value offloads memory management to JVM - which usually is more optimal that one provided by OS.
0
 

Accepted Solution

by:
ckercher earned 0 total points
ID: 22880783
It turns out that the cfquery was using the native cf8 Informix DB drivers. When I switched it to use the downloaded IBM JDBC drivers it works to completion. Unfortunately I don't have time to dig into why the issue with the native drivers. Thanks for your suggestions.
Joe
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Recently while working on a project I got a very annoying cfdocument has no body error message. I had never seen this error before. So I checked the code. The code was pretty simple; it was Just showing me the cfdocumnt tag and inside that tag a …
I annotated my article on ransomware somewhat extensively, but I keep adding new references and wanted to put a link to the reference library.  Despite all the reference tools I have on hand, it was not easy to find a way to do this easily. I finall…
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now