Upload and parse EXTREMELY large CSV file COLDFUSION/MySQL

Posted on 2011-10-09
Last Modified: 2012-05-12
Hello experts

I have an application where I need to allow users to be able to upload CSV files via the site. The site is built in ColdFusion 9 with a MySQL back-end. These files are LARGE. Sometimes millions of lines. I was thinking of writing something like this:
User selects file using CFFILE to upload to a directory on the site
Use a MySQL stored proc to process
Email or alert the user the parsing is done.

Is there a way to submit something like this to the server and "release" the browser so the user can do other things while the file is processed?
Maybe a status bar showing the progress while the ColdFusion server does'nt time-out?

Any advice would be appreciated

Question by:MuzickPro
    LVL 39

    Accepted Solution

    Just make the browser action the user's upload... so the user uploads the file, place it in the "processing area" .. .maybe add a corresponding record to the database and let the user know he will be emailed when it's ready.

    Then have a scheduled task look for files every few minutes by looking for the database record with status = 'New'... start processing the file by changing the status to "In Progress" ...    the processing is finished alert the user

    To process the file, look for MySQL import functions that can go directly against a physical file... that will like be the fastest way

    Author Closing Comment


    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How to improve team productivity

    Quip adds documents, spreadsheets, and tasklists to your Slack experience
    - Elevate ideas to Quip docs
    - Share Quip docs in Slack
    - Get notified of changes to your docs
    - Available on iOS/Android/Desktop/Web
    - Online/Offline

    Hi. There are several upload tutorials using jquery and coldfusion. I found a very interesting one here Upload Your Files using Jquery & ColdFusion and Preview them ( . I did keep the main js functions but made sever…
    Sometimes databases have MILLIONS of records and we need a way to quickly query that table to return the results me need. Sure you could use CFQUERY but it takes too long when there are millions of records. That is why SOLR was invented. Please …
    It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.
    Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

    737 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    22 Experts available now in Live!

    Get 1:1 Help Now