Request body over 1400KB cannot be read

I have recently come across a problem where my CGI page will fail to read stdin when data being sent in is over 1400KB.

An overview of what I am currently doing is as follows:
1. HTML form contains information.
2. Onsubmit, Javascript will parse form and create XML string with all information.
3. AJAX will pass XML string to CGI page to be processed using POST.

The problem comes in after #3, where stdin is being read.  I read the content length is 1375, and the entire XML can be read immediately.  I read the content length of 1407, and the CGI hangs, then there is a page refresh.  But the XML file is not completely read.

Originally I was using Appweb 3.2.2-1.  After reading post on forum about possible fix, I upgraded to Appweb 3.3.0-2, but problem still exists.

There is not output in the appweberror.log file.

Attached is my appweb.conf file. appweb.conf.txt
sctccommAsked:
Who is Participating?
 
Dave BaldwinConnect With a Mentor Fixer of ProblemsCommented:
Something outside of your code is limiting what you're doing.  Look for the prototype of "read()".  In Unix C, there is a value defined as 'SSIZE_MAX' that sets a limit on the size of the "read()" data returned.  Something not obvious is interfering.
0
 
Dave BaldwinFixer of ProblemsCommented:
I assume you mean 1,400,000 bytes.  At the bottom of your config file are values that set the upload limits.  You can try increasing both of them.

LimitRequestBody 2000000
LimitUploadSize 1000000

What computer are you running this on?  Is this a 'PC'?
0
 
Dave BaldwinFixer of ProblemsCommented:
Here is the info in the FAQ: http://appwebserver.org/support/faq.html
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
sctccommAuthor Commented:
When I read back the content length, I see that it is 14XX for which I thought that was in Bytes.  In the configuration, the numbers are also in Bytes.  I also tried setting the LimitRequestBody to 0 (supposedly disables the limit), but it still did not work.
0
 
sctccommAuthor Commented:
Btw, I'm running appweb on an embedded system using ARM9.
0
 
sctccommAuthor Commented:
I have tried to increase the Limits to 10000000, but it makes no difference.
0
 
Dave BaldwinFixer of ProblemsCommented:
If the content length is 1407 bytes, not Kilobytes, then you have something else in your code limiting the length.  See if your buffer is too small.
0
 
sctccommAuthor Commented:
My buffer is dynamically allocated.

1. Get content length. (Read to be 1407)
2. buf = (SINT8 *)malloc((length + 1) * sizeof(SINT8));
3. size = read(0, buf, length);
4. If size != length then return FAIL (size = 536).

My function always fails at the read.
0
 
Dave BaldwinFixer of ProblemsCommented:
Try using a static buffer of twice the expected size.  Then just use the size that is read without comparing it to the supposed length to see what you get.
0
 
sctccommAuthor Commented:
I removed the return FAIL if size read is not equal to supposed length for both a static buffer and a dynamic buffer, and the behavior is consistent for both.

I read back only the 537 characters that were read, which is less than half of the data that is supposed to be sent.
0
 
Dave BaldwinFixer of ProblemsCommented:
Do you have the XML files on your computer so you can check the size directly?  Something about your program or the interaction with the host is misleading.  Can you run the HTML page and javascript on another host to see what you get?
0
 
sctccommAuthor Commented:
Well the XML file is generated by Javascript and then sent to the server.  So I compare the size of the XML string created by Javascript from within the script and I set the content length of the XML string to be read by the page.

As for running the page on another host, it will be difficult because my CGI files are cross-compiled for an embedded CPU and I would need to recompile them for x86, then copy them to a PC before testing it there.  If possible, I would like to avoid this action.
0
 
sctccommAuthor Commented:
I also forgot to mention that it seems to work fine for any XML string less than 1400 bytes, which I stated in my original question.
0
 
Dave BaldwinFixer of ProblemsCommented:
I read your original question.  Just beating on the CGI code won't produce an answer.  I would try to find an alternate way of testing that could reveal more information.  HTML/javascript should be able to post the XML files to another page written in something PHP or ASP that won't have the limits you're running into.  The test page doesn't have to do what your CGI code does, it just has to receive the files and tell you what's in them and how large they are so you can verify that you're getting what you are supposed to.

Can you post the HTML page that generates the XML?
0
 
sctccommAuthor Commented:
My Javascript scans all forms and the inputs and selects within those forms and generates an XML.

I have posted the XML string generated.
<?xml version="1.0" encoding="UTF-8" ?>
<INFO><BUTTON><ID>2</ID><NAME>Apply</NAME><TYPE>dropbox</TYPE><VALUE>Apply</VALUE></BUTTON><ENTRY><NAME>IPv4 Address_ip1</NAME><TYPE>text</TYPE><VALUE>192</VALUE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>IPv4 Address_ip2</NAME><TYPE>text</TYPE><VALUE>168</VALUE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>IPv4 Address_ip3</NAME><TYPE>text</TYPE><VALUE>1</VALUE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>IPv4 Address_ip4</NAME><TYPE>text</TYPE><VALUE>87</VALUE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>Subnet Mask_ip1</NAME><TYPE>text</TYPE><VALUE>255</VALUE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>Subnet Mask_ip2</NAME><TYPE>text</TYPE><VALUE>255</VALUE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>Subnet Mask_ip3</NAME><TYPE>text</TYPE><VALUE>255</VALUE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>Subnet Mask_ip4</NAME><TYPE>text</TYPE><VALUE>0</VALUE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>IPv6 Address</NAME><TYPE>text</TYPE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>IPv6 Address_prefix</NAME><TYPE>text</TYPE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY><ENTRY><NAME>IPv6 Default Gateway</NAME><TYPE>text</TYPE><CHECKED>false</CHECKED><CLICKED>false</CLICKED></ENTRY></INFO>

Open in new window

0
 
Dave BaldwinFixer of ProblemsCommented:
I get 1476 characters for that string.
0
 
sctccommAuthor Commented:
That is what I see from my Javascript and the size of my static buffer was 5000.
0
 
Dave BaldwinFixer of ProblemsCommented:
And the routine still only read 537 bytes?
0
 
sctccommAuthor Commented:
Yes, that is correct.  For both dynamically and statically allocated buffers.
0
 
sctccommAuthor Commented:
Okay, I understand what you are saying.

I have changed the code to perform a while loop reading 500 characters at a time, and it seems to be more stable now.

I am going to clean up the code and check to make sure it is working as expected.  Once I can confirm everything is working I will close this question.

Thanks.
0
 
sctccommAuthor Commented:
On thing that I have noticed that is strange is that when reading a body that is of size 1362 (less than 1400), the processing is almost immediate.  But when reading a body of size 1475 (over 1400), the processing takes almost a minute.

Is there a known reason for this difference in processing time?
0
 
Dave BaldwinFixer of ProblemsCommented:
I can't find anything about it.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.