Link to home
Create AccountLog in
Avatar of CipherIS
CipherISFlag for United States of America

asked on

Get Data from WebService

Creating a WebService that will obtain approx 4,000,000 records.  

Are there any issues that I need to be aware of?  
How do I develop?  

Also, it is a possibility to call a web service and the data is dumped to a csv file and then sent to the client requesting the data?

Thx
Avatar of Misbah
Misbah
Flag of United States of America image

will the web service receive or return this 4 million  records ?
Avatar of CipherIS

ASKER

it will return.  The user will make a request and receive the 4,000,000 records
i'd run it as an async process or the client will have to wait until all the records are received. (at the client end)
ASKER CERTIFIED SOLUTION
Avatar of developmentguru
developmentguru
Flag of United States of America image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
You can't bring all the 4,000,000  records at a time. By default 4 MB of data can be sent back. This can be increased to in the web.config file. You should basically see how my bytes of data ur 4,000,000 rows going to get it back.  If you sent it back in JSON format from the web methods then u can send more data than XML format.

Best way is to have a page size and bring the subsequent pages on demand to the client.
can anyone provide an example of how to return data using "GetNextN" like developmentguru suggested?
SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
4 million records at 1 byte each is 4Megabytes of data to which we are going to add
<xmldata> <data> </data></xmldata> to each record let say we optimize and so its 8 x 4 = 48 Megabytes plus a little overhead to send 4 million records of one character
Would you be open to using PHP as the server side coding?  I could easily provide some PHP code that reads the database and returns the records in XML format.  As ve3ofa said, XML is quite verbose.  Many developers prefer JSON due to the relatively concise nature.  I used XML because I was able to format the XML in a way that let me assign it directly to a TClientDataset and use the in memory dataset in my Delphi application.  Delphi can handle JSON as well.  You could also modify the code to handle comma delimited if you desire although that can cause problems of it's own if your data contains text fields.

If you are interested in the PHP approach, I should mention that I am using Ubuntu 11.10 with a LAMP stack (in case you want to set up a free server to test with).