Smanyx
asked on
Effective way to deal with long running server-side web requests
Hi,
I'm writing a web application using python flask micro-framework.
A request is sent to the server and must return a list of documents by categories, with relevant information for each doc such as author(s), url, publication date, summary etc...
Each request can return up to 10, 000 or even 20, 000+ documents for popular topics.
The search and clustering being done on the server, a template is used to render the output on the browser.
The problem I'm facing is that the rendering is taking way too long. Sometimes even around the 10 minute mark, which is really just unacceptable.
I am looking at ways to optimize this.
But I'm not quite sure which approach I should take.
One line of thought was to limit the number of docs returned from the initial request query (typed in the input form) and have that rendered on the browser and then, subsequently send multiple ajax/jquery requests adding each time the number of documents until I reach the maximum number of docs that could be retrieved for that request (need a way to check this), updating the browser each time ...
I'm still not sure though how or what would trigger those ajax/jquery requests (there won't be a button click or anything ...)
Would having a div id that might contain a value to be checked in order to trigger the jquery request work?
What's the best/practicable way to deal with such scenario?
Thanks.
I'm writing a web application using python flask micro-framework.
A request is sent to the server and must return a list of documents by categories, with relevant information for each doc such as author(s), url, publication date, summary etc...
Each request can return up to 10, 000 or even 20, 000+ documents for popular topics.
The search and clustering being done on the server, a template is used to render the output on the browser.
The problem I'm facing is that the rendering is taking way too long. Sometimes even around the 10 minute mark, which is really just unacceptable.
I am looking at ways to optimize this.
But I'm not quite sure which approach I should take.
One line of thought was to limit the number of docs returned from the initial request query (typed in the input form) and have that rendered on the browser and then, subsequently send multiple ajax/jquery requests adding each time the number of documents until I reach the maximum number of docs that could be retrieved for that request (need a way to check this), updating the browser each time ...
I'm still not sure though how or what would trigger those ajax/jquery requests (there won't be a button click or anything ...)
Would having a div id that might contain a value to be checked in order to trigger the jquery request work?
What's the best/practicable way to deal with such scenario?
Thanks.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I'm really interested in the practicality aspect of it.
Any code snippets, links, tutorials, books .. where I can see an actual implementation?
Cheers.