We help IT Professionals succeed at work.

Get Data from WebService

Medium Priority
315 Views
Last Modified: 2013-12-16
Creating a WebService that will obtain approx 4,000,000 records.  

Are there any issues that I need to be aware of?  
How do I develop?  

Also, it is a possibility to call a web service and the data is dumped to a csv file and then sent to the client requesting the data?

Thx
Comment
Watch Question

MisbahSr. Software Engineer
CERTIFIED EXPERT

Commented:
will the web service receive or return this 4 million  records ?

Author

Commented:
it will return.  The user will make a request and receive the 4,000,000 records
Dushan De SilvaTechnology Architect

Commented:
You may implement to call web service as a batch task.
http://forum.springsource.org/showthread.php?69907-Calling-webservice-as-batch-task
David Johnson, CDSimple Geek from the '70s
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
i'd run it as an async process or the client will have to wait until all the records are received. (at the client end)
President
Commented:
I have, in the past, made web services that could return this many records.  I made it return XML because it fit better with the rest of my design.  You can easily make it return CDL instead.  

ve3ofa has a good point.  I made my web service call to use a GetNextN style record call so the client could determine how many records it would retrieve in one shot.  This left the client more configurable and gave me the capability of showing status while it retrieved the large data set.  Getting 1,000 records per shot seemed to be a good number to keep things moving smoothly but this would depend on MANY criteria.  You should experiment and see what works for you.

Another concern is to make sure that the data you are sending is encrypted.  You can make your web service work over HTTPS, but I tend to encrypt it before sending it through HTTPS.

I made sure to place the work of getting the records into a thread to keep the UI responsive too.
Aaron JabamaniTechnical Architect

Commented:
You can't bring all the 4,000,000  records at a time. By default 4 MB of data can be sent back. This can be increased to in the web.config file. You should basically see how my bytes of data ur 4,000,000 rows going to get it back.  If you sent it back in JSON format from the web methods then u can send more data than XML format.

Best way is to have a page size and bring the subsequent pages on demand to the client.

Author

Commented:
can anyone provide an example of how to return data using "GetNextN" like developmentguru suggested?
David Johnson, CDSimple Geek from the '70s
CERTIFIED EXPERT
Distinguished Expert 2019
Commented:
it is normally called paging http://www.15seconds.com/issue/010308.htm
David Johnson, CDSimple Geek from the '70s
CERTIFIED EXPERT
Distinguished Expert 2019

Commented:
4 million records at 1 byte each is 4Megabytes of data to which we are going to add
<xmldata> <data> </data></xmldata> to each record let say we optimize and so its 8 x 4 = 48 Megabytes plus a little overhead to send 4 million records of one character
developmentguruPresident

Commented:
Would you be open to using PHP as the server side coding?  I could easily provide some PHP code that reads the database and returns the records in XML format.  As ve3ofa said, XML is quite verbose.  Many developers prefer JSON due to the relatively concise nature.  I used XML because I was able to format the XML in a way that let me assign it directly to a TClientDataset and use the in memory dataset in my Delphi application.  Delphi can handle JSON as well.  You could also modify the code to handle comma delimited if you desire although that can cause problems of it's own if your data contains text fields.

If you are interested in the PHP approach, I should mention that I am using Ubuntu 11.10 with a LAMP stack (in case you want to set up a free server to test with).

Explore More ContentExplore courses, solutions, and other research materials related to this topic.