Fetching Data From WebSites

Dear Experts,

I would like to get the HTML file from the URL.. I know
CInternetSession can be used and I am having a sample program on that.. Is there anyother method or Code sample or some URL?  I need your assistance experts. Give me some URLs and Guidelines.

If the Code or URL is really very good I can give more points.

Thanks in advance..

With Regards
LVL 1
ksenthilrajAsked:
Who is Participating?
 
makerpConnect With a Mentor Commented:
#include <afxinet.h>
#include <stdio.h>
#include <malloc.h>

int main(void)
{
    CInternetSession session("My Session");
    CHttpFile *file = NULL;

    unsigned long code = 0, con_len = 0;
    char buff[1000] = "";
    long i = 0, so_far = 0;
    char *file_text = NULL;

    printf("getting file\n\n");
    file = (CHttpFile*)session.OpenURL("http://stan/",1, INTERNET_FLAG_TRANSFER_ASCII, NULL,0);

    if(file)
    {
         file->QueryInfoStatusCode(code);

         printf("Got It **status code is : %i**\n\n",code);

           while(true)
           {
               i = file->Read(buff,999);
               file_text = (char*)realloc(file_text,(so_far += i) + 1);
               memcpy(file_text + so_far - i,buff,i);
               if(i < 999) break;
           }
           file_text[so_far] = '\0';
           printf("CONTENT\n%s\n",file_text);
     }        
    return 0;
}
0
 
jhanceCommented:
What is it about CInternetSession that is unsatisfactory?
0
 
makerpCommented:
i found this knocking about from early days, it works ok but aint perfect.

/*
     written by     : Paul Maker
     Description : command line browser
*/

#include <afxinet.h>
#include <stdio.h>
#include <malloc.h>
#include <iostream.h>
#include <io.h>

int main(void)
{
     CInternetSession session("My Session");
     CHttpFile *file = NULL;

     unsigned long code = 0, con_len = 0;
     char buff[1000] = "";
     long i = 0;
     char *file_text = NULL;

     printf("getting file\n\n");
     file = (CHttpFile*)session.OpenURL("http://stan/default.asp",1, INTERNET_FLAG_TRANSFER_ASCII, NULL,0);

     if(file)
     {
          file->QueryInfoStatusCode(code);

          printf("Got It\n\n**status code is : %i**\n\n",code);
          i = file->Read(buff,999);
          buff[i + 1] = '\0';
          file_text = (char *)malloc(i + 2);
          strcpy(file_text,buff);
          if(i > 0)
          {
               do
               {
                    i = file->Read(buff,999);
                    buff[i + 1] = '\0';
                    file_text = (char *)realloc(file_text, _msize(file_text) + i + 1);
                    strcat(file_text,buff);
               }while(i == 999);
               file_text[strlen(file_text) - 1] = '\0';
               printf("%s\n\n",file_text);
          }
     }



     return 0;
}
0
Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

 
makerpCommented:
no one please comment on the code, its 2 years old, :), i was not that good then
0
 
ksenthilrajAuthor Commented:
Dear Experts,
CInternetSession is OK. But I need good examples in that. I am not very much strict with that alone that's is my point if it is other than that also ok for me.

I am expecting some examples or code or URL from you experts..

With Cheers.
0
 
makerpCommented:
have you tried the code i posted
0
 
ksenthilrajAuthor Commented:
Hai makerp,

It gives some errors can you refine and give me.

With Cheers..
0
 
makerpCommented:
it gives me no error, you must chose to use MFC in project settings, maybe this is your problem, i will refine it anyway
0
 
ksenthilrajAuthor Commented:
makerp,

The code runs fine.  But i need to put this in my database can you intergrate this thing with the database..
I will check this code today fully for my compatiability and let you know tommorrow.

Very good response makerp,

Thanks,

With Cheers.
0
 
makerpCommented:
yes database is easy, see your other question on databases, i posted some info there to
0
 
ksenthilrajAuthor Commented:
makerp,

don't mistake me.. I will accept your both comments..If you don't mine can you give me integrated code in which the internet connection and database feeding happedned using a exe for three times a week.. Since i am novice in VC++ i face so many problem in every aspects

With Cheers.
0
 
ksenthilrajAuthor Commented:
sorry 3 times a day..

With Cheers.
0
 
makerpCommented:
right what exactly do you want it to do. get a web page and put it in the database ?
0
 
ksenthilrajAuthor Commented:
makerp,

Nice response..Sorry for the delay..
What i actually want is:

I have a web site like www.google.com/

Using CInternetSession I fetch the html page from the website by passing the apprpriote additional string like
  1)computer
I mean,  www.google.com/computer and get the information as a text file
Like this i have 10 various variables stored in the array something like that..
etc.,  string[0]=computer
       string[1]=textile
       string[2]=information

Like this i have to change the final string in the www.google.com like site the above arrays as..(this is always constant one)

www.google.com/computers
www.google.com/textile
www.google.com/information
By passing like this 10 infomation which can be hard coded (predefined) and get the 10 different html files from the net.. After getting the files I will parse that file where the information avl..( I will design this after getting the html only.. ) So atpresent i have only little design only in my mind.
You can simulate a Html file and get the information and store the necessary (Take as you like from 10 files )and store it in a database in the different columns.

I think this will clear your doubt..

Thanks in advance.

With Cheers.

0
 
makerpCommented:
i can not write the code for you, it will take to long but i will advise. you have most of the code already, the fetching code here and the database examples, you just have to intergrate it. i would start with a simple console app that gets one page and saves it to a access dtabase in a memo field.

Paul
0
 
ksenthilrajAuthor Commented:
Dear paul,

Ok that's fine..
>>but i will advise

This is equavalent to your answer..

I am expecting you simple console app..

With Cheers.
0
 
RideOnCommented:
hi makerp

i'm interested in your sample, too.
i'm using c++builder and i don't have the afxinet.h header-file. can u post it for me?
thx

:RideOn
0
 
ksenthilrajAuthor Commented:
Hai makerp,

Waiting eagerly

With Cheers
0
 
ssrCommented:
Try looking at the TEAR MFC sample.

SSR
0
 
ksenthilrajAuthor Commented:
Hai makerp,

Waiting eagerly

With Cheers
0
 
makerpCommented:
i am going on holiday, therefore i will not get a chance. all you need to do is tie together the code i gave you here with the code i gave you in the database question. its easy.

i think that it would be best to close the two questions as you have answers for your original questions, then write the code and if you get problems ask a new question. most experts on here will not write whole programs for people, rather they will look at existing codes and help

Paul
0
 
ksenthilrajAuthor Commented:
makerp,

But i am giving here 100 points and there 50 points .. think of it.. 100 points for this simple question that's why i asked to integrate and give.. and i am novice  too.

Or else give some URL's where i get more help in this regard,

Mainly for parsing the content which i got from the website

storing in the array and executing the different url


Hope you understand my difficulties.
With Cheers.
0
 
makerpCommented:
parsing the content of a HTML page is quite complex. you can look at the SGML HTML parser

http://validator.w3.org/

you see getting the site and storing it is easy, you must be able to see how to do that from what i have given you. it is the parsing in between getting and storing that will be hard
0
 
ksenthilrajAuthor Commented:
makerp,

This is a reminder for you

With Cheers
0
 
ksenthilrajAuthor Commented:
makerp,

any comments
0
 
makerpCommented:
hi there, sorry ive been ofline for a while. remind me of what you want
0
 
ksenthilrajAuthor Commented:
dear makerp,

tomorrow i will tell my requirement,.

Bye for now

Cheers
0
 
ksenthilrajAuthor Commented:
Makerp,

Very Long Gap.. Any Way I am continue now.. That problem over.

Now we will move to Proxy Settings..

I am behind proxy settings.

For Example;;
my proxy script;
http://proxy:8080/config.pac

I am aloted for some UserName, Password for the Proxy:

I know the Class InternetSetOption.. Can you give me some suggestion in that.. to get your points makerp..

or Else Give suggestions or Clues to Get Your Points.

I would like to put my exe into system tray when I minimise the window..

Shell_NotifyIcon
NOTIFYICONDATA

Give some links or working example .

Cheers..
0
 
ksenthilrajAuthor Commented:
Are U there??
 
Can I delete the Question?

Regards..
0
 
ksenthilrajAuthor Commented:
No response from Experts Quiet Long.

Regards,
0
 
makerpCommented:
what help do you want now?
0
 
makerpCommented:
what help do you want now?
0
 
ksenthilrajAuthor Commented:
Dear makerp,

Question now changed...

Herewith I gave my code FYR:

I would like to copy a whole directory to my ftp site as it is.. and delete it after copying ..

I am trying this with recursive function..

For Example..

d:\localdirectory\filecopy

filecopy directory contains another directory called dir1,dir2...

Inside dir1,dir2 still more files are there..

I would like to copy my whole filecopy directory to my ftp root directory and delete local hard disk directory completely.


My ftp site for example. ftp://210.4.7.8

Copy filecopy like this

ftp://210.4.7.8/filecopy..

Hope I clear the request.

     CInternetSession ftpSession("my session");
     ptrFtpConnection =  ftpSession.GetFtpConnection(strFtpServerIp,strFtpUserName,strFtpPassword,INTERNET_INVALID_PORT_NUMBER,FALSE);

Regards
0
 
griesshCommented:
I think you forgot this question. I will ask Community Support to close it unless you finalize it within 7 days. Unless there is objection or further activity,  I will suggest to accept "makerp" comment(s) as an answer.

ksenthilraj
You changed the scope of your question the second time now. I think for all the help makerp has provided you should award him the points. If you want to ask new questions, please do what everybody in EE does, ask a new question and offer new points.

If you think your question was not answered at all, you can post a request in Community support (please include this link) to refund your points.
The link to the Community Support area is: http://www.experts-exchange.com/jsp/qList.jsp?ta=commspt

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!
======
Werner
0
 
griesshCommented:
BTW: This question was answered long time ago. There is no way to just delete it ...

======
Werner
0
 
Computer101Commented:
ksenithilraj,
It seems you asked a question and it was answered.  Now you have another question.  I would say to award an expert who assisted you in the original question and make another question.

Thank you
Computer101
Community Support Moderator
0
All Courses

From novice to tech pro — start learning today.