CFHTTP not returning all content from certain websites

I'm trying to get the source for the search results on dice.com using CFHTTP using the following code:

---------------------------------------
<cfhttp method = "GET" url = "http://seeker.dice.com/jobsearch/index.jsp" resolveurl="yes">
</cfhttp>

<CFOUTPUT>
content = #cfhttp.filecontent# <BR>
mimetype = #cfhttp.mimeType# <BR>
header = #cfhttp.header# <BR>
</cfoutput>
---------------------------------------

but it's only returning partial content. I think this is because the server is using "transfer encoding: chunked" so only 1 chunk is being sent to me.

Is there a way to get all of the content?

Your prompt response is greatly appreciated.

Johnny G.
javanigusAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

proceptCommented:
Hi,

I just tested your code and it returned a complete page, up to the closing </html> tag. SO maybe you just had a bad connection when you tried it. ;-))

You can add a timeout value to the <cfhttp> tag:

<cfhttp ...... timeout="timeout_in_seconds">

HTH,

Chris
0
javanigusAuthor Commented:
Oops, I gave you the wrong code snippet. The code that doesn't give the complete page is

--------------------------------------
<cfhttp method = "POST" url = "http://seeker.dice.com/jobsearch/servlet/JobSearch" resolveurl="yes">
      <cfhttpparam name = "PREV_DOC" type = "FormField" value = "0">
      <cfhttpparam name = "NEXT_DOC" type = "FormField" value = "31">
      <cfhttpparam name = "op" type = "FormField" value = "1001">
      <cfhttpparam name = "FILTER_TEXT" type = "FormField" value = "">
</cfhttp>

<CFOUTPUT>
content = #cfhttp.filecontent# <BR>
mimetype = #cfhttp.mimeType# <BR>
header = #cfhttp.header# <BR>
</cfoutput>

-------------------------------------------

If you go to

http://askabboudi.com/misc/httppost_dice.cfm

you'll see the code in action not showing the whole page.

Any ideas?

Thanks,

Johnny G.
0
proceptCommented:
Hi Johnny,

I took your code, played around with it, looked at the dice-site, played some more, and found... you need to change the value of the paramter "op" to "1000" and it will work. Don't know why, but, it works. :-))

HTH,

Chris
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
javanigusAuthor Commented:
Dude, thanks alot.

Johnny G.
0
proceptCommented:
You're welcome. :-))

Chris
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Servers

From novice to tech pro — start learning today.