Solved

Search many Zip Files

Posted on 2011-03-09
7
281 Views
Last Modified: 2013-12-26
Our email system does a backup each day of all email sent and received, and stores it in a directory with the date stamp, so:

mail/2010-10-10/2010-10-10.zip
mail/2010-10-11/2010-10-11.zip

etc

I now have to provide all emails from jsmith@whatver.com from 2010-10-01 to 2010-12-31.  Of course, this is impossible to do until I unzip everything!

There has got to be a way to automate this!  I have tried Recoll, but I must be doing something wrong, every search returns nothing (even searching items I KNOW are there).

Appreciate any help!
0
Comment
Question by:dougp23
  • 5
  • 2
7 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 35086111
It can sure be scripted very easily.  But I would need more information.

First step is to unzip those files to a directory.
Then command to upzip all would look like this, may be not exactly


 
for m in {10..12}
do
    for d in $(seq -w 01 31)
   do
        date="2010-$m-$d"
        unzip -j mail/$date/$date.zip
    done
done

Open in new window


Then get one email and then the question would become, how to extract the emails you want from those files.
0
 
LVL 1

Author Comment

by:dougp23
ID: 35086209
Your bash script is a little beyond me (as I fiddle with the variables,lol).  All the directories that I need to step through are in one top level dir (/mail).  

So could we say

for all directories in /mail
unzip the zip in each directory.

For the 2nd part of your question, each email is extracted with a unique time stamp for that day, so once unzipped, each dir will include every email sent and received that day.  I can then grep or egrep for a person's email address that I need.

Thanks!
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35086307
First part.
I would like to keep your zip files intact.  So, please put those to a new location, say test folder.  This way you would not regret.

Second part, I would need to see the file structure to construct a regular expression for extraction because, I am expecting emails to be in the following format

HEADERS
body

NEXT EMAIL HEADERS

I think you want to get the emails along with the body, so I need to setup a rule like
start from a line that contains this specific field and end with a line where you see this.
0
Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

 
LVL 1

Author Comment

by:dougp23
ID: 35086358
If I unzip one of the zips, I get files like this:

09:39:27-0400.12630.0
09:40:09-0400.12630.0
09:40:09-0400.12630.1

So as you can see, each email is self-contained.  If I do a cat or a less of the file, I can see all the headers as well as the body.  So I just need to walk through each directory and unzip.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35087497
Ok, I would give you the code if you answer three questions.

In an email, is there a field in the header called from: jsmith@whatver.com something like this.  I want the exact syntax.

Second, should I follow exactly this directory structure?

Third, what should be the file or folder where I save it?
0
 
LVL 31

Accepted Solution

by:
farzanj earned 500 total points
ID: 35090445
Since I did not receive any response from you, I would make some assumptions

My script would extract some files in a tmp directory that you would need to set and copy the files that included the email address 'jsmith@whatver.com' (may be send or to field in the header) and copy it over to the target folder.  You would need to set the values of TMP and TARGET.  Hope it helps but would eagerly wait for your response
#!/bin/bash

#####
#PLEASE MAKE DIRS AND SET THESE VALUES
TMP=<temp dir, should be writable and empty>
TARGET=<this would contain your desired emails>

for m in {10..12}
do
    for d in $(seq -w 01 31)
   do
        date="2010-$m-$d"
        unzip mail/$date/$date.zip > $TMP
    done
done

files=$(grep -i "jsmith@whatver.com" $TMP/* | cut -d: -f1 | sort | uniq)

mv $files $TARGET

Open in new window

0
 
LVL 31

Expert Comment

by:farzanj
ID: 35090488
Sorry.

Change line 13 above to:
unzip mail/$date/$date.zip -d $TMP
0

Featured Post

Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Uninstall via powershell 1 74
How can I make a command only be ran by sudo permission 8 92
UNiX Script filesystem space usage 19 74
AWS CLI issues with Tags 3 75
This Windows batch file is useful for organizing image files from a digital camera or other source, but can have many other uses.  It simply renames the file(s) to match their create date.  For example, if you took a picture today at 1:40pm and the …
Background Still having to process all these year-end "csv" files received from all these sources (including Government entities), sometimes we have the need to examine the contents due to data error, etc... As a "Unix" shop, our only readily …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…

786 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question