asked on

How would I decompress and parse 365 JSON files, each over a gig?

I've got the script I need to decompress and parse the files, but what I need now is something that can "look" into the directory and automatically grab each file, process it and then move on to the next.

I can't imagine how I could do that, unless I had the name of each file loaded in a database somewhere.

Perhaps some clever spin on fopen?

How could I do it so I could initiate the process as I'm headed out the door and the process continue automatically through the nite and be done in the morning?

SOLUTION

Brian Tao

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

SOLUTION

Ray Paseur

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Bruce Gust

ASKER

Gentlemen!

Thanks so much for your willingness to share your expertise!

Question: In both your examples, the output includes two rows of "blank" values. By that I mean, in my current directory, I have one file. Rather than that file being listed by itself, with taoyipai' s suggestion I get:

.
..
00_8ptcd6jgjn201311060000_day.json

Ray, with your scenario I get:

Array ( [0] => . [1] => .. [2] => 00_8ptcd6jgjn201311060000_day.json ) Array ( [0] => 00_8ptcd6jgjn201311060000_day.json [1] => .. [2] => . )

Again, you're getting those "dots" and I'm wondering, first of all, what they represent and, secondly, how can I remove them from the list of files that I want to preform some code on? In other words, how do I ensure that the list of files in the directory do not include "." and ".."?

SOLUTION

Member_2_3684445

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER CERTIFIED SOLUTION

Ray Paseur

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Bruce Gust

ASKER

Got it!

Thank you!

Also, feel free to head out to https://www.experts-exchange.com/questions/28526319/What-would-be-the-best-approach.html for a question that pertains to the next piece of scaffolding for this project...