• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 665
  • Last Modified:

Upload and parse EXTREMELY large CSV file COLDFUSION/MySQL

Hello experts

I have an application where I need to allow users to be able to upload CSV files via the site. The site is built in ColdFusion 9 with a MySQL back-end. These files are LARGE. Sometimes millions of lines. I was thinking of writing something like this:
User selects file using CFFILE to upload to a directory on the site
Use a MySQL stored proc to process
Email or alert the user the parsing is done.

Is there a way to submit something like this to the server and "release" the browser so the user can do other things while the file is processed?
Maybe a status bar showing the progress while the ColdFusion server does'nt time-out?

Any advice would be appreciated

1 Solution
Just make the browser action the user's upload... so the user uploads the file, place it in the "processing area" .. .maybe add a corresponding record to the database and let the user know he will be emailed when it's ready.

Then have a scheduled task look for files every few minutes by looking for the database record with status = 'New'... start processing the file by changing the status to "In Progress" ...    the processing is finished alert the user

To process the file, look for MySQL import functions that can go directly against a physical file... that will like be the fastest way
MuzickProAuthor Commented:

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now