Solved

CFHTTP not returning all content from certain websites

Posted on 2003-11-03
5
321 Views
Last Modified: 2013-12-24
I'm trying to get the source for the search results on dice.com using CFHTTP using the following code:

---------------------------------------
<cfhttp method = "GET" url = "http://seeker.dice.com/jobsearch/index.jsp" resolveurl="yes">
</cfhttp>

<CFOUTPUT>
content = #cfhttp.filecontent# <BR>
mimetype = #cfhttp.mimeType# <BR>
header = #cfhttp.header# <BR>
</cfoutput>
---------------------------------------

but it's only returning partial content. I think this is because the server is using "transfer encoding: chunked" so only 1 chunk is being sent to me.

Is there a way to get all of the content?

Your prompt response is greatly appreciated.

Johnny G.
0
Comment
Question by:javanigus
  • 3
  • 2
5 Comments
 
LVL 4

Expert Comment

by:procept
ID: 9669659
Hi,

I just tested your code and it returned a complete page, up to the closing </html> tag. SO maybe you just had a bad connection when you tried it. ;-))

You can add a timeout value to the <cfhttp> tag:

<cfhttp ...... timeout="timeout_in_seconds">

HTH,

Chris
0
 

Author Comment

by:javanigus
ID: 9669821
Oops, I gave you the wrong code snippet. The code that doesn't give the complete page is

--------------------------------------
<cfhttp method = "POST" url = "http://seeker.dice.com/jobsearch/servlet/JobSearch" resolveurl="yes">
      <cfhttpparam name = "PREV_DOC" type = "FormField" value = "0">
      <cfhttpparam name = "NEXT_DOC" type = "FormField" value = "31">
      <cfhttpparam name = "op" type = "FormField" value = "1001">
      <cfhttpparam name = "FILTER_TEXT" type = "FormField" value = "">
</cfhttp>

<CFOUTPUT>
content = #cfhttp.filecontent# <BR>
mimetype = #cfhttp.mimeType# <BR>
header = #cfhttp.header# <BR>
</cfoutput>

-------------------------------------------

If you go to

http://askabboudi.com/misc/httppost_dice.cfm

you'll see the code in action not showing the whole page.

Any ideas?

Thanks,

Johnny G.
0
 
LVL 4

Accepted Solution

by:
procept earned 250 total points
ID: 9669903
Hi Johnny,

I took your code, played around with it, looked at the dice-site, played some more, and found... you need to change the value of the paramter "op" to "1000" and it will work. Don't know why, but, it works. :-))

HTH,

Chris
0
 

Author Comment

by:javanigus
ID: 9669913
Dude, thanks alot.

Johnny G.
0
 
LVL 4

Expert Comment

by:procept
ID: 9669971
You're welcome. :-))

Chris
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: kevp75
Hey folks, 'bout time for me to come around with a little tip. Thanks to IIS 7.5 Extensions and Microsoft (well... really Windows 8, and IIS 8 I guess...), we can now prime our Application Pools, when IIS starts. Now, though it would be nice t…
Periodically we have to update or add SSL certificates for customers. Depending upon your hosting plan you may be responsible for the installation and/or key generation. In the wake of Heartbleed many sites were forced to re-key. We will concen…
This tutorial gives a high-level tour of the interface of Marketo (a marketing automation tool to help businesses track and engage prospective customers and drive them to purchase). You will see the main areas including Marketing Activities, Design …
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

810 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question