Solved

COldfusion: Memory allocation failed during query processing: error on CF8 but not on CF7

Posted on 2008-10-31
6
1,009 Views
Last Modified: 2013-12-24
Hello,
I am having problems performing a task on a CF8 server that works on a CF7 server. The CF7 & CF8 servers are both windows. What I am trying to do is use cfdirectory to get a list of picture files (about 15,000) then insert a record for each file into a table. When I run the cfm template on the CF7 server it eventually finishes, slow but it finishes. When I run the same template on the cf8 server it eventually comes back with this error

[Macromedia][Informix JDBC Driver][Informix]Memory allocation failed during query processing

I looked in the cf8 monitor and the jvm memory peaks to about 120M out of 512. Here are some cf8 settings which, btw, are the same as on the cf7 server

Maximum number of running JRun threads : 50
Maximum number of queued JRun Threads : 1000
Java Virtual Machine Path : D:/ColdFusion8/runtime/jre
Minimum JVM Heap Size (MB): 512        Maximum JVM Heap Size (MB) : 512
ColdFusion Class Path:  "c:\Program Files\IBM\Informix JDBC Driver 2.21.JC6\lib","D:\cfsftp\cfsftp.jar"
JVM Arguments: -server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=128m -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib

Any help or ideas is appreciated!
Regards,
Joe

Here is the code

<CFQUERY datasource="foo" name="delcfd">
      delete from ws_pics
</CFQUERY>

<cfif DirectoryExists("//server/A/B/C/skupics/")>
<cfdirectory directory="//server/A/B/C/skupics/" action="list" name="myD" recurse="no">
<CFLOOP query="myD"> <!--- A --->
<cfif myD.type EQ "Dir" AND  DirectoryExists("//server/A/B/C/skupics/#trim(myD.name)#/")>
<cfdirectory directory="//server/A/B/C/skupics/#trim(myD.name)#/" action="list" name="myD2" filter="a*.jpg|b*.jpg|c*.jpg|d*.jpg|e*.jpg|f*.jpg|g*.jpg|h*.jpg|i*.jpg|j*.jpg|k*.jpg|l*.jpg|m*.jpg|n*.jpg|o*.jpg|p*.jpg|q*.jpg|r*.jpg|s*.jpg|t*.jpg|u*.jpg|v*.jpg|w*.jpg|x*.jpg|y*.jpg|z*.jpg" recurse="no" type="File"> ***** anyone know of a better way to do this ? ****
<CFLOOP query = "myD2" > <!--- B --->
<CFSET variables.nlen = #len(myd2.name)#>
<CFIF nlen GTE 6>
:#myD.name#/#myd2.name#:
<CFQUERY datasource="famousviews" name="instcfd">
insert into ws_pics (num, picfile) values (0, '#trim(myd2.name)#')
</CFQUERY>
</CFIF>
<cfflush> ***print to browser ***
</CFLOOP> <!--- B --->
</cfif>
</CFLOOP> <!--- A --->
...
0
Comment
Question by:ckercher
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
6 Comments
 
LVL 16

Expert Comment

by:duncancumming
ID: 22856074
I think this
filter="a*.jpg|b*.jpg|c*.jpg|d*.jpg|e*.jpg|f*.jpg|g*.jpg|h*.jpg|i*.jpg|j*.jpg|k*.jpg|l*.jpg|m*.jpg|n*.jpg|o*.jpg|p*.jpg|q*.jpg|r*.jpg|s*.jpg|t*.jpg|u*.jpg|v*.jpg|w*.jpg|x*.jpg|y*.jpg|z*.jpg"
can just be rewritten as
filter="*.jpg"
unless your intention is to exclude files name 1.jpg etc?  

take out your cfflush to speed things up a bit (although I'm guessing you've kept it in as a tracer while trying to debug this?)

avoid setting an unnecessary variable
<CFSET variables.nlen = #len(myd2.name)#>
<CFIF nlen GTE 6>
becomes
<CFIF len(myd2.name) GTE 6>

also note you don't need the # # around your function call inside a cfset.

in your inner cfdirectory, try setting listinfo="name", as you're only wanting the filenames, although read this article:
http://www.bennadel.com/blog/886-Learning-ColdFusion-8-CFDirectory-Improvements.htm
you could probably just use LIstLast(name, "/") to then extract just the filename part.

in your outer cfdirectory, try setting type="dir", then remove your cfif myD.type EQ "Dir" AND  DirectoryExists if clause.  

Could you redo the whole thing as one cfdirectory with recurse=true, applying your filter to just get the jpgs?  saves you having to do nested cfdirectory.
0
 

Author Comment

by:ckercher
ID: 22866994
Hi duncan...
Thanks for the reply. There are folders and files under /a/b/c that I do not want to process, hence the long filter and multiple cfdirectory loops. Its too bad cfdirectory can not process regex for the filter :(  I'll have to play with the new cfdirectory fields in cf8 to make it more efficient. The odd thing is the cf8 monitor did showed the jvm memory did not fill up which is what I would have expected if that was the issue. Any other ideas aer appreciated.
Joe
0
 
LVL 62

Expert Comment

by:gheist
ID: 22867631
What is java version - i.e java -version?

-XX: parameter may not perform as expected when another JVM is used.

MaxPermSize and PermSize should be same due to bugs in old JVMs (if you use SUN's JVM)

Does it work when you get rid of parameter?
0
Free NetCrunch network monitor licenses!

Only on Experts-Exchange: Sign-up for a free-trial and we'll send you your permanent license!

Here is what you get: 30 Nodes | Unlimited Sensors | No Time Restrictions | Absolutely FREE!

Act now. This offer ends July 14, 2017.

 

Author Comment

by:ckercher
ID: 22868117
Something I just found to be curious is it always bombs on the insertion of the same file. db spaces and file systems are not filling up.

The JVM is Sun's 1.6.0_04
I have not tried getting rid of the MaxPermSize but that will be my next step.

Thanks!
Joe
0
 
LVL 62

Expert Comment

by:gheist
ID: 22868279
Is there any Informix session identifiable? onstat -g ses

You should be running later JDK http://secunia.com/advisories/product/12878/?task=advisories_2008

I do not see valid reason for -XX tuning unless system does not cope with JVM's memory use well.

Setting -Xms and -Xmx to same value offloads memory management to JVM - which usually is more optimal that one provided by OS.
0
 

Accepted Solution

by:
ckercher earned 0 total points
ID: 22880783
It turns out that the cfquery was using the native cf8 Informix DB drivers. When I switched it to use the downloaded IBM JDBC drivers it works to completion. Unfortunately I don't have time to dig into why the issue with the native drivers. Thanks for your suggestions.
Joe
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Never store passwords in plain text or just their hash: it seems a no-brainier, but there are still plenty of people doing that. I present the why and how on this subject, offering my own real life solution that you can implement right away, bringin…
In this series, we will discuss common questions received as a database Solutions Engineer at Percona. In this role, we speak with a wide array of MySQL and MongoDB users responsible for both extremely large and complex environments to smaller singl…
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question