Solved

CFHTTP not returning all content from certain websites

Posted on 2003-11-03
5
324 Views
Last Modified: 2013-12-24
I'm trying to get the source for the search results on dice.com using CFHTTP using the following code:

---------------------------------------
<cfhttp method = "GET" url = "http://seeker.dice.com/jobsearch/index.jsp" resolveurl="yes">
</cfhttp>

<CFOUTPUT>
content = #cfhttp.filecontent# <BR>
mimetype = #cfhttp.mimeType# <BR>
header = #cfhttp.header# <BR>
</cfoutput>
---------------------------------------

but it's only returning partial content. I think this is because the server is using "transfer encoding: chunked" so only 1 chunk is being sent to me.

Is there a way to get all of the content?

Your prompt response is greatly appreciated.

Johnny G.
0
Comment
Question by:javanigus
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
5 Comments
 
LVL 4

Expert Comment

by:procept
ID: 9669659
Hi,

I just tested your code and it returned a complete page, up to the closing </html> tag. SO maybe you just had a bad connection when you tried it. ;-))

You can add a timeout value to the <cfhttp> tag:

<cfhttp ...... timeout="timeout_in_seconds">

HTH,

Chris
0
 

Author Comment

by:javanigus
ID: 9669821
Oops, I gave you the wrong code snippet. The code that doesn't give the complete page is

--------------------------------------
<cfhttp method = "POST" url = "http://seeker.dice.com/jobsearch/servlet/JobSearch" resolveurl="yes">
      <cfhttpparam name = "PREV_DOC" type = "FormField" value = "0">
      <cfhttpparam name = "NEXT_DOC" type = "FormField" value = "31">
      <cfhttpparam name = "op" type = "FormField" value = "1001">
      <cfhttpparam name = "FILTER_TEXT" type = "FormField" value = "">
</cfhttp>

<CFOUTPUT>
content = #cfhttp.filecontent# <BR>
mimetype = #cfhttp.mimeType# <BR>
header = #cfhttp.header# <BR>
</cfoutput>

-------------------------------------------

If you go to

http://askabboudi.com/misc/httppost_dice.cfm

you'll see the code in action not showing the whole page.

Any ideas?

Thanks,

Johnny G.
0
 
LVL 4

Accepted Solution

by:
procept earned 250 total points
ID: 9669903
Hi Johnny,

I took your code, played around with it, looked at the dice-site, played some more, and found... you need to change the value of the paramter "op" to "1000" and it will work. Don't know why, but, it works. :-))

HTH,

Chris
0
 

Author Comment

by:javanigus
ID: 9669913
Dude, thanks alot.

Johnny G.
0
 
LVL 4

Expert Comment

by:procept
ID: 9669971
You're welcome. :-))

Chris
0

Featured Post

Save the day with this special offer from ATEN!

Save 30% on the CV211 using promo code EXPERTS30 now through April 30th. The ATEN CV211 connects a laptop directly to any server allowing you instant access to perform data maintenance and local operations, for quick troubleshooting, updating, service and repair.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: kevp75
Hey folks, 'bout time for me to come around with a little tip. Thanks to IIS 7.5 Extensions and Microsoft (well... really Windows 8, and IIS 8 I guess...), we can now prime our Application Pools, when IIS starts. Now, though it would be nice t…
Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
In this video, viewers will be given step by step instructions on adjusting mouse, pointer and cursor visibility in Microsoft Windows 10. The video seeks to educate those who are struggling with the new Windows 10 Graphical User Interface. Change Cu…
Monitoring a network: why having a policy is the best policy? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the enormous benefits of having a policy-based approach when monitoring medium and large networks. Software utilized in this v…

695 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question