Starting point in a foreach()

Posted on 2007-08-07
Last Modified: 2013-12-12

I have a tab delimited text file which contains 9,000 rows and 23 columns.  I am importing this into a MySQL database.  However, the script which imports it is timing out around row 4860.

What I want to do is make it so the script gets to row 1000, then stops and refreshes and continues from that point to break the process down a bit.  I can store the row number in a SQL database ready to be read and continued from by the script on refresh.

However, the method I am using to break the file down is as follows.

if (file_exists($file_name)) { // Check the file can be opened
$file_to_process = file($file_name); // put file into an array

Then once I have the file in an array I do the following to process each row

foreach($file_to_process as $row) {
$data = explode("\t",$row);
// From here I do my validation and insert into the database

However, if I am restarting the script, my current script is going to always start from row zero and will never get anywhere.  How can I do the foreach starting at row 1001 (or whatever value I call from the database to continue from) so that only the data which has not been processed is then processed.

Also, each time the script runs I need to call row 0 to get the column names as this is part of my verification system.

Note:  The 9,000 rows is an example.  The file could contains 23,000 rows, or just 10 rows.  I depends on what my customers upload to be processed.

I did wonder if it was possible to do the following

foreach($file_to_process[i] as $row)

and then i++ after processing although I have a strong feeling that would not work as it would only retrieve one row.

I am sure I am missing something simple but for the life of me cannot figure it out.

Question by:ParadyneDesigns
    LVL 10

    Expert Comment

    instead, why not count the rows in a file

    next set a for loop delimiter and work with a for loop instead?

    LVL 10

    Expert Comment

    $starting_offset = "0";
    $stopping_offset = "999";
    $current_count = "0";
    for($i = $starting_offset; $stopping_offset; $i ++){
           $current_count ++;

    use the current count to check if all was succesfull and to know where the seccond run shoudl start...
    LVL 10

    Expert Comment

    You might also want to read this function (do be carefull though)

    LVL 10

    Expert Comment

    oeps, that for will walk forever :P

    for($i = $starting_offset; $i <= $stopping_offset; $i ++)

    Author Comment

    Thanks for the ideas but the loop suggested still would not split the array.  Each iteration would still commence from the begining of the array.  What I needed was something to move the internal pointer to the new starting point in the array when the script restarts.

    What I have done in the time being is add an array slice to see if that will work.  I am just uploading the file to my FTP to run the tasks and seeing it that solved it.

    The time set limit would not really have worked either as a 16,000 row file would take my server around 1 hour to process (the validation in my cron script is extremely intensive (I forgot to add that this was a cron)).  There is still the time set limit of the server which in my case overrules the time set limit added in php.
    LVL 11

    Accepted Solution

    if ($_SESSION["position"]<1){
    $_SESSION["position"] = 1;

    if (file_exists($file_name))
    $file_to_process = file($file_name);

          echo $file_to_process[$i];
                $_SESSION["position"] = $i + 1;

    At the point where it says       echo $file_to_process[$i];
    you can put whatever you need for that 1000 worth of records.

    Whats happening here is that you position is held in a session variable.  That way that position can be pulled up when you come back to the page so as to "resume" from that point.  Once another set is done, it gets incremented for another "resume" once you reload the page (or implement some other way to restart it).

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How to run any project with ease

    Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
    - Combine task lists, docs, spreadsheets, and chat in one
    - View and edit from mobile/offline
    - Cut down on emails

    Introduction HTML checkboxes provide the perfect way for a web developer to receive client input when the client's options might be none, one or many.  But the PHP code for processing the checkboxes can be confusing at first.  What if a checkbox is…
    I imagine that there are some, like me, who require a way of getting currency exchange rates for implementation in web project from time to time, so I thought I would share a solution that I have developed for this purpose. It turns out that Yaho…
    Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
    The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

    737 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    23 Experts available now in Live!

    Get 1:1 Help Now