Solved

Fetching Data From WebSites

Posted on 2001-08-04
36
336 Views
Last Modified: 2008-02-01
Dear Experts,

I would like to get the HTML file from the URL.. I know
CInternetSession can be used and I am having a sample program on that.. Is there anyother method or Code sample or some URL?  I need your assistance experts. Give me some URLs and Guidelines.

If the Code or URL is really very good I can give more points.

Thanks in advance..

With Regards
0
Comment
Question by:ksenthilraj
  • 17
  • 13
  • 2
  • +4
36 Comments
 
LVL 32

Expert Comment

by:jhance
ID: 6351557
What is it about CInternetSession that is unsatisfactory?
0
 
LVL 10

Expert Comment

by:makerp
ID: 6352812
i found this knocking about from early days, it works ok but aint perfect.

/*
     written by     : Paul Maker
     Description : command line browser
*/

#include <afxinet.h>
#include <stdio.h>
#include <malloc.h>
#include <iostream.h>
#include <io.h>

int main(void)
{
     CInternetSession session("My Session");
     CHttpFile *file = NULL;

     unsigned long code = 0, con_len = 0;
     char buff[1000] = "";
     long i = 0;
     char *file_text = NULL;

     printf("getting file\n\n");
     file = (CHttpFile*)session.OpenURL("http://stan/default.asp",1, INTERNET_FLAG_TRANSFER_ASCII, NULL,0);

     if(file)
     {
          file->QueryInfoStatusCode(code);

          printf("Got It\n\n**status code is : %i**\n\n",code);
          i = file->Read(buff,999);
          buff[i + 1] = '\0';
          file_text = (char *)malloc(i + 2);
          strcpy(file_text,buff);
          if(i > 0)
          {
               do
               {
                    i = file->Read(buff,999);
                    buff[i + 1] = '\0';
                    file_text = (char *)realloc(file_text, _msize(file_text) + i + 1);
                    strcat(file_text,buff);
               }while(i == 999);
               file_text[strlen(file_text) - 1] = '\0';
               printf("%s\n\n",file_text);
          }
     }



     return 0;
}
0
 
LVL 10

Expert Comment

by:makerp
ID: 6352814
no one please comment on the code, its 2 years old, :), i was not that good then
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6354612
Dear Experts,
CInternetSession is OK. But I need good examples in that. I am not very much strict with that alone that's is my point if it is other than that also ok for me.

I am expecting some examples or code or URL from you experts..

With Cheers.
0
 
LVL 10

Expert Comment

by:makerp
ID: 6354926
have you tried the code i posted
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6355012
Hai makerp,

It gives some errors can you refine and give me.

With Cheers..
0
 
LVL 10

Expert Comment

by:makerp
ID: 6355021
it gives me no error, you must chose to use MFC in project settings, maybe this is your problem, i will refine it anyway
0
 
LVL 10

Accepted Solution

by:
makerp earned 100 total points
ID: 6355036
#include <afxinet.h>
#include <stdio.h>
#include <malloc.h>

int main(void)
{
    CInternetSession session("My Session");
    CHttpFile *file = NULL;

    unsigned long code = 0, con_len = 0;
    char buff[1000] = "";
    long i = 0, so_far = 0;
    char *file_text = NULL;

    printf("getting file\n\n");
    file = (CHttpFile*)session.OpenURL("http://stan/",1, INTERNET_FLAG_TRANSFER_ASCII, NULL,0);

    if(file)
    {
         file->QueryInfoStatusCode(code);

         printf("Got It **status code is : %i**\n\n",code);

           while(true)
           {
               i = file->Read(buff,999);
               file_text = (char*)realloc(file_text,(so_far += i) + 1);
               memcpy(file_text + so_far - i,buff,i);
               if(i < 999) break;
           }
           file_text[so_far] = '\0';
           printf("CONTENT\n%s\n",file_text);
     }        
    return 0;
}
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6355831
makerp,

The code runs fine.  But i need to put this in my database can you intergrate this thing with the database..
I will check this code today fully for my compatiability and let you know tommorrow.

Very good response makerp,

Thanks,

With Cheers.
0
 
LVL 10

Expert Comment

by:makerp
ID: 6358759
yes database is easy, see your other question on databases, i posted some info there to
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6358789
makerp,

don't mistake me.. I will accept your both comments..If you don't mine can you give me integrated code in which the internet connection and database feeding happedned using a exe for three times a week.. Since i am novice in VC++ i face so many problem in every aspects

With Cheers.
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6358790
sorry 3 times a day..

With Cheers.
0
 
LVL 10

Expert Comment

by:makerp
ID: 6358912
right what exactly do you want it to do. get a web page and put it in the database ?
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6362679
makerp,

Nice response..Sorry for the delay..
What i actually want is:

I have a web site like www.google.com/

Using CInternetSession I fetch the html page from the website by passing the apprpriote additional string like
  1)computer
I mean,  www.google.com/computer and get the information as a text file
Like this i have 10 various variables stored in the array something like that..
etc.,  string[0]=computer
       string[1]=textile
       string[2]=information

Like this i have to change the final string in the www.google.com like site the above arrays as..(this is always constant one)

www.google.com/computers
www.google.com/textile
www.google.com/information
By passing like this 10 infomation which can be hard coded (predefined) and get the 10 different html files from the net.. After getting the files I will parse that file where the information avl..( I will design this after getting the html only.. ) So atpresent i have only little design only in my mind.
You can simulate a Html file and get the information and store the necessary (Take as you like from 10 files )and store it in a database in the different columns.

I think this will clear your doubt..

Thanks in advance.

With Cheers.

0
 
LVL 10

Expert Comment

by:makerp
ID: 6363215
i can not write the code for you, it will take to long but i will advise. you have most of the code already, the fetching code here and the database examples, you just have to intergrate it. i would start with a simple console app that gets one page and saves it to a access dtabase in a memo field.

Paul
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6363315
Dear paul,

Ok that's fine..
>>but i will advise

This is equavalent to your answer..

I am expecting you simple console app..

With Cheers.
0
 
LVL 1

Expert Comment

by:RideOn
ID: 6364562
hi makerp

i'm interested in your sample, too.
i'm using c++builder and i don't have the afxinet.h header-file. can u post it for me?
thx

:RideOn
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6367182
Hai makerp,

Waiting eagerly

With Cheers
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 
LVL 2

Expert Comment

by:ssr
ID: 6371062
Try looking at the TEAR MFC sample.

SSR
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6376741
Hai makerp,

Waiting eagerly

With Cheers
0
 
LVL 10

Expert Comment

by:makerp
ID: 6376891
i am going on holiday, therefore i will not get a chance. all you need to do is tie together the code i gave you here with the code i gave you in the database question. its easy.

i think that it would be best to close the two questions as you have answers for your original questions, then write the code and if you get problems ask a new question. most experts on here will not write whole programs for people, rather they will look at existing codes and help

Paul
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6377016
makerp,

But i am giving here 100 points and there 50 points .. think of it.. 100 points for this simple question that's why i asked to integrate and give.. and i am novice  too.

Or else give some URL's where i get more help in this regard,

Mainly for parsing the content which i got from the website

storing in the array and executing the different url


Hope you understand my difficulties.
With Cheers.
0
 
LVL 10

Expert Comment

by:makerp
ID: 6379104
parsing the content of a HTML page is quite complex. you can look at the SGML HTML parser

http://validator.w3.org/

you see getting the site and storing it is easy, you must be able to see how to do that from what i have given you. it is the parsing in between getting and storing that will be hard
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6408455
makerp,

This is a reminder for you

With Cheers
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6624379
makerp,

any comments
0
 
LVL 10

Expert Comment

by:makerp
ID: 6624388
hi there, sorry ive been ofline for a while. remind me of what you want
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6624476
dear makerp,

tomorrow i will tell my requirement,.

Bye for now

Cheers
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6803457
Makerp,

Very Long Gap.. Any Way I am continue now.. That problem over.

Now we will move to Proxy Settings..

I am behind proxy settings.

For Example;;
my proxy script;
http://proxy:8080/config.pac

I am aloted for some UserName, Password for the Proxy:

I know the Class InternetSetOption.. Can you give me some suggestion in that.. to get your points makerp..

or Else Give suggestions or Clues to Get Your Points.

I would like to put my exe into system tray when I minimise the window..

Shell_NotifyIcon
NOTIFYICONDATA

Give some links or working example .

Cheers..
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6812389
Are U there??
 
Can I delete the Question?

Regards..
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6812762
No response from Experts Quiet Long.

Regards,
0
 
LVL 10

Expert Comment

by:makerp
ID: 6815500
what help do you want now?
0
 
LVL 10

Expert Comment

by:makerp
ID: 6815502
what help do you want now?
0
 
LVL 1

Author Comment

by:ksenthilraj
ID: 6818341
Dear makerp,

Question now changed...

Herewith I gave my code FYR:

I would like to copy a whole directory to my ftp site as it is.. and delete it after copying ..

I am trying this with recursive function..

For Example..

d:\localdirectory\filecopy

filecopy directory contains another directory called dir1,dir2...

Inside dir1,dir2 still more files are there..

I would like to copy my whole filecopy directory to my ftp root directory and delete local hard disk directory completely.


My ftp site for example. ftp://210.4.7.8

Copy filecopy like this

ftp://210.4.7.8/filecopy..

Hope I clear the request.

     CInternetSession ftpSession("my session");
     ptrFtpConnection =  ftpSession.GetFtpConnection(strFtpServerIp,strFtpUserName,strFtpPassword,INTERNET_INVALID_PORT_NUMBER,FALSE);

Regards
0
 
LVL 11

Expert Comment

by:griessh
ID: 6845255
I think you forgot this question. I will ask Community Support to close it unless you finalize it within 7 days. Unless there is objection or further activity,  I will suggest to accept "makerp" comment(s) as an answer.

ksenthilraj
You changed the scope of your question the second time now. I think for all the help makerp has provided you should award him the points. If you want to ask new questions, please do what everybody in EE does, ask a new question and offer new points.

If you think your question was not answered at all, you can post a request in Community support (please include this link) to refund your points.
The link to the Community Support area is: http://www.experts-exchange.com/jsp/qList.jsp?ta=commspt

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!
======
Werner
0
 
LVL 11

Expert Comment

by:griessh
ID: 6845262
BTW: This question was answered long time ago. There is no way to just delete it ...

======
Werner
0
 
LVL 1

Expert Comment

by:Computer101
ID: 6848832
ksenithilraj,
It seems you asked a question and it was answered.  Now you have another question.  I would say to award an expert who assisted you in the original question and make another question.

Thank you
Computer101
Community Support Moderator
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

Errors will happen. It is a fact of life for the programmer. How and when errors are detected have a great impact on quality and cost of a product. It is better to detect errors at compile time, when possible and practical. Errors that make their wa…
Container Orchestration platforms empower organizations to scale their apps at an exceptional rate. This is the reason numerous innovation-driven companies are moving apps to an appropriated datacenter wide platform that empowers them to scale at a …
The goal of the video will be to teach the user the difference and consequence of passing data by value vs passing data by reference in C++. An example of passing data by value as well as an example of passing data by reference will be be given. Bot…
The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now