How would I decompress and parse 365 JSON files, each over a gig?

I've got the script I need to decompress and parse the files, but what I need now is something that can "look" into the directory and automatically grab each file, process it and then move on to the next.

I can't imagine how I could do that, unless I had the name of each file loaded in a database somewhere.

Perhaps some clever spin on fopen?

How could I do it so I could initiate the process as I'm headed out the door and the process continue automatically through the nite and be done in the morning?
brucegustPHP DeveloperAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Brian TaoSenior Business Solutions ConsultantCommented:
This skeleton may be what you need:
if ($dh = opendir("$dir_name")){
  while (($file = readdir($dh)) !== false){
    // code for processing each individual file
    // e.g. print the file name
    echo "$file <br>\n";
  }
  closedir($dh);
}

Open in new window

0
Ray PaseurCommented:
Here is what I would do.

Get a list of the files.  Scandir() will handle that part.  Then with each file name, start a process to do whatever you want with the file.  You can use fsockopen() or cURL to start the process.  Give the process script the name of the file and let it run.  You will want to start the process with a POST-method request, so you can disconnect and let the process run asynchronously.

You probably want to sleep() a few moments between starting the processes.  You probably want to keep a log of the file names and a timestamp when the process was started.  You probably want to keep a log of the times when each process ended, so you know what succeeded and what failed.
0
brucegustPHP DeveloperAuthor Commented:
Gentlemen!

Thanks so much for your willingness to share your expertise!

Question: In both your examples, the output includes two rows of "blank" values. By that I mean, in my current directory, I have one file. Rather than that file being listed by itself, with taoyipai' s suggestion I get:

.
 ..
 00_8ptcd6jgjn201311060000_day.json

Ray, with your scenario I get:

Array ( [0] => . [1] => .. [2] => 00_8ptcd6jgjn201311060000_day.json ) Array ( [0] => 00_8ptcd6jgjn201311060000_day.json [1] => .. [2] => . )

Again, you're getting those "dots" and I'm wondering, first of all, what they represent and, secondly, how can I remove them from the list of files that I want to preform some code on? In other words, how do I ensure that the list of files in the directory do not include "." and ".."?
0
Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

Chris GralikeSpecialistCommented:
This will remove the dots from the array.

$d = scandir($path);

foreach($d as $k => $v){
        if( !(( $v === '.') || ($v === '..')) ){
                $files[]= $v";
        }
}

print_r($files);

Open in new window

0
Ray PaseurCommented:
getting those "dots" and I'm wondering, first of all, what they represent
They are directory indicators and irrelevant to your application.  You would only be looking for files that end in ".json" right?  Skip the others as you process the array.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
brucegustPHP DeveloperAuthor Commented:
Got it!

Thank you!

Also, feel free to head out to http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/Q_28526319.html for a question that pertains to the next piece of scaffolding for this project...
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
PHP

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.