• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 323
  • Last Modified:

Starting point in a foreach()

Hello

I have a tab delimited text file which contains 9,000 rows and 23 columns.  I am importing this into a MySQL database.  However, the script which imports it is timing out around row 4860.

What I want to do is make it so the script gets to row 1000, then stops and refreshes and continues from that point to break the process down a bit.  I can store the row number in a SQL database ready to be read and continued from by the script on refresh.

However, the method I am using to break the file down is as follows.

if (file_exists($file_name)) { // Check the file can be opened
$file_to_process = file($file_name); // put file into an array

Then once I have the file in an array I do the following to process each row

foreach($file_to_process as $row) {
$data = explode("\t",$row);
// From here I do my validation and insert into the database

However, if I am restarting the script, my current script is going to always start from row zero and will never get anywhere.  How can I do the foreach starting at row 1001 (or whatever value I call from the database to continue from) so that only the data which has not been processed is then processed.

Also, each time the script runs I need to call row 0 to get the column names as this is part of my verification system.

Note:  The 9,000 rows is an example.  The file could contains 23,000 rows, or just 10 rows.  I depends on what my customers upload to be processed.

I did wonder if it was possible to do the following

foreach($file_to_process[i] as $row)

and then i++ after processing although I have a strong feeling that would not work as it would only retrieve one row.

I am sure I am missing something simple but for the life of me cannot figure it out.

Cheers
0
ParadyneDesigns
Asked:
ParadyneDesigns
  • 4
1 Solution
 
Chris GralikeSpecialistCommented:
instead, why not count the rows in a file

next set a for loop delimiter and work with a for loop instead?

regards,
0
 
Chris GralikeSpecialistCommented:
$starting_offset = "0";
$stopping_offset = "999";
$current_count = "0";
for($i = $starting_offset; $stopping_offset; $i ++){
       blabla
       $current_count ++;
}

use the current count to check if all was succesfull and to know where the seccond run shoudl start...
0
 
Chris GralikeSpecialistCommented:
You might also want to read this function (do be carefull though)
http://nl3.php.net/manual/en/function.set-time-limit.php

regards,
0
Cloud Class® Course: CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

 
Chris GralikeSpecialistCommented:
oeps, that for will walk forever :P

for($i = $starting_offset; $i <= $stopping_offset; $i ++)
0
 
ParadyneDesignsAuthor Commented:
Thanks for the ideas but the loop suggested still would not split the array.  Each iteration would still commence from the begining of the array.  What I needed was something to move the internal pointer to the new starting point in the array when the script restarts.

What I have done in the time being is add an array slice to see if that will work.  I am just uploading the file to my FTP to run the tasks and seeing it that solved it.

The time set limit would not really have worked either as a 16,000 row file would take my server around 1 hour to process (the validation in my cron script is extremely intensive (I forgot to add that this was a cron)).  There is still the time set limit of the server which in my case overrules the time set limit added in php.
0
 
AlexanderREnterprise Web DeveloperCommented:
session_start();
if ($_SESSION["position"]<1){
$_SESSION["position"] = 1;
}

if (file_exists($file_name))
$file_to_process = file($file_name);


for($i=$_SESSION["position"];$i<=($_SESSION["position"]+1000);$i++){
      echo $file_to_process[$i];
      if(($i+1)==($_SESSION["position"]+1000)){
            $_SESSION["position"] = $i + 1;
            break;
      }
}

At the point where it says       echo $file_to_process[$i];
you can put whatever you need for that 1000 worth of records.

Whats happening here is that you position is held in a session variable.  That way that position can be pulled up when you come back to the page so as to "resume" from that point.  Once another set is done, it gets incremented for another "resume" once you reload the page (or implement some other way to restart it).
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now