Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

scripting help

Posted on 2013-06-17
21
Medium Priority
?
173 Views
Last Modified: 2013-08-30
I have a folder with about 20,000 files in it, I need to find a way to delete the first four lines out of each file and append an @example.com to the end of each to and from address.  I'm trying to find a way to script this and I have no programming background.  Is there an easy way to do this, or can someone point me in the right direction?
0
Comment
Question by:PSGITech
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 8
  • 3
21 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 39254602
Make a backup of your files.

To delete four lines of  files you can do something like

cd /to/the/folder
for filename in *
do
perl -i -ne 'print if $. >4' $filename
done

Open in new window


For appending @example.com, you need to provide a sample
0
 
LVL 42

Expert Comment

by:sedgwick
ID: 39255174
Where the address is located in the file?
0
 
LVL 2

Author Comment

by:PSGITech
ID: 39255863
I will try this, this morning I have never used perl before.
0
TCP/IP Network Protocol Cheat Sheet

TCP/IP is a set of network protocols which is best known for connecting the machines that make up the Internet. The truth is that TCP/IP is one of the oldest network protocols and its survival is mainly based on its simplicity and universality.

 
LVL 2

Author Comment

by:PSGITech
ID: 39256012
Here is the contents of the files, I'm pasting in what it looks like now and what I need it to look live afterwards

THE FILE AS IT APPEARS NOW

X-BAAuthOn: <null>
x-BA-Received: from johnsmith@logprod.contoso.com (EHLO logprod.contoso.com 10.10.10.10 [10.10.10.10] )
      by 10.10.10.15 with ESMTP id LMVE5CWFPKZLEXZJL3CVARULDM51LHXNB
      for <>; Fri, 6 Jan 2012 12:00:18 -0500  (EST)
Received: from mail pickup service by logprod.contoso.com with Microsoft SMTPSVC;
       Fri, 6 Jan 2012 12:00:19 -0500
Message-ID: <16245629-FAD6-45BB-B07D-D8BC45DD299520726083742758233>
Date: Fri, 06 Jan 2012 13:43:06 UTC
From: johnsmith
To: billjones; johnsmith


HERE IS WHAT I NEED IT TO LOOK LIKE AFTERWARDS, I NEED THE FIRST 6 LINES REMOVED AND REPLACED WITH X-Receiver: bs@bm3.contoso, I THEN NEED TO APPEND @EXAMPLE.COM AT THE END OF THE ALIAS IN THE FROM AND TO FIELDS

X-Receiver: bs@bm3.contoso
Message-ID: <16245629-FAD6-45BB-B07D-D8BC45DD299520726083742758233>
Date: Fri, 06 Jan 2012 13:43:06 UTC
From: johnsmith@example.com
To: billjones@example.com; johnsmith@example.com

Any help would be greatly appreciated, to have to do this manually would take a year.
0
 
LVL 42

Expert Comment

by:sedgwick
ID: 39256071
powershell:
$rootfolder = 'c:\temp\test'
gci $rootfolder | %{
	$lines = gc $_.fullname | select -Skip 6
	$lines = ,"X-Receiver: bs@bm3.contoso" + $lines
	$lastline = $lines | select -Last 1
	$lastline = "{0}{1}" -f $lastline.Replace(";", "@example.com;"), "@example.com"
	$lines[$lines.Count-1] = $lastline
	$lines | Set-Content $_.fullname
}

Open in new window

0
 
LVL 2

Author Comment

by:PSGITech
ID: 39256102
thank you going to try this right now
0
 
LVL 2

Author Comment

by:PSGITech
ID: 39256137
I ran the script and it seems to have pulled off a little too much, here is the result

X-Receiver: bs@bm3.contoso
X-OriginalArrivalTime: 06 Jan 2012 17:00:19.0212 (UTC) FILETIME=[AB8AACC0:01CCCC94]

I do see it did append @example.com at the end of the file but I didn't paste the whole file in the example for confidential purposes, it looks like in all the files @example.com needs to be appended in lines 9 and 10
0
 
LVL 42

Expert Comment

by:sedgwick
ID: 39256153
can u post the file that wasn't processed as it should?
i'll test it again.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39256190
Try this one.

It is a Perl one liner.  The advantage is that you can test it on any file and if you have happy with the output, you can simply use option -i to change the file

perl -ne 'BEGIN{print "X-Receiver: bs@bm3.contoso\n";}if($.>6){if(/From:/){s/$/\@example.com/;print;}elsif(/^To:/){s/To: //;s/(\w+)/$1\@example.com/g;s/^/To: /; print;} else{ print;}}' filename

Open in new window


Once you are happy, you can do:
perl -i -ne 'BEGIN{print "X-Receiver: bs@bm3.contoso\n";}if($.>6){if(/From:/){s/$/\@example.com/;print;}elsif(/^To:/){s/To: //;s/(\w+)/$1\@example.com/g;s/^/To: /; print;} else{ print;}}' tt2

Open in new window

0
 
LVL 2

Author Comment

by:PSGITech
ID: 39256197
Here is a before and after with what the full file would look like with some dummy data, thanks for you help

WHAT IT LOOKS LIKE NOW

X-BAAuthOn: <null>
x-BA-Received: from johnsmith@logprod.contoso.com (EHLO logprod.contoso.com 10.10.10.10 [10.10.10.10] )
      by 10.10.10.15 with ESMTP id LMVE5CWFPKZLEXZJL3CVARULDM51LHXNB
      for <>; Fri, 6 Jan 2012 12:00:18 -0500  (EST)
Received: from mail pickup service by logprod.contoso.com with Microsoft SMTPSVC;
       Fri, 6 Jan 2012 12:00:19 -0500
Message-ID: <16245629-FAD6-45BB-B07D-D8BC45DD299520726083742758233>
Date: Fri, 06 Jan 2012 13:43:06 UTC
From: johnsmith
To: billjones; johnsmith
Subject: Giggins, Dave: buyer 25k now
X-OriginalArrivalTime: 06 Jan 2012 17:00:17.0962 (UTC) FILETIME=[AACBF0A0:01CCCC94]

Friday, January 06, 2012 11:11:35 AM EST
        Giggins, Dave started conversation.
Friday, January 06, 2012 11:11:35 AM EST
        philspencer has entered the conversation.
Friday, January 06, 2012 11:11:35 AM EST
        Giggins, Dave:     buyer 25k now
Friday, January 06, 2012 11:14:59 AM EST



WHAT I'M LOOKING TO ACHIEVE


X-Receiver: bs@bm3.contoso
Message-ID: <16245629-FAD6-45BB-B07D-D8BC45DD299520726083742758233>
Date: Fri, 06 Jan 2012 13:43:06 UTC
From: johnsmith@example.com
To: billjones@example.com; johnsmith@example.com
Subject: Giggins, Dave: buyer 25k now

Friday, January 06, 2012 11:11:35 AM EST
        Giggins, Dave started conversation.
Friday, January 06, 2012 11:11:35 AM EST
        philspencer has entered the conversation.
Friday, January 06, 2012 11:11:35 AM EST
        Giggins, Dave:     buyer 25k now
Friday, January 06, 2012 11:14:59 AM EST
0
 
LVL 42

Expert Comment

by:sedgwick
ID: 39256265
that should work:
$rootfolder = 'c:\temp\test'
gci $rootfolder | %{
	$lines = gc $_.fullname | select -Skip 6
	$lines = ,"X-Receiver: bs@bm3.contoso" + $lines
	$toline = $lines | where {$_ -match "to:"}
	$tolineindex = [array]::IndexOf($lines, $toline)
	$toline = "{0}{1}" -f $toline.Replace(";", "@example.com;"), "@example.com"
	$fromline = $lines | where {$_ -match "from:"}
	$fromlineindex = [array]::IndexOf($lines, $fromline)
	$fromline = "{0}{1}" -f $fromline.Replace(";", "@example.com;"), "@example.com"
	$lines[$tolineindex] = $toline
	$lines[$fromlineindex] = $fromline
}

Open in new window

0
 
LVL 31

Expert Comment

by:farzanj
ID: 39256278
I had tested it on my system and gave it only when it worked correctly
0
 
LVL 2

Author Comment

by:PSGITech
ID: 39256317
Weird I changed the first line to match the folder path where I have my test file, and I pasted it into my powershell session and it didn't error out or anything but the file is unchanged, could i be doing something wrong here?
0
 
LVL 42

Expert Comment

by:sedgwick
ID: 39256340
sorry i forgot to update the file after i changed it. here:
$rootfolder = 'c:\temp\test'
gci $rootfolder | %{
	$lines = gc $_.fullname | select -Skip 6
	$lines = ,"X-Receiver: bs@bm3.contoso" + $lines
	$toline = $lines | where {$_ -match "to:"}
	$tolineindex = [array]::IndexOf($lines, $toline)
	$toline = "{0}{1}" -f $toline.Replace(";", "@example.com;"), "@example.com"
	$fromline = $lines | where {$_ -match "from:"}
	$fromlineindex = [array]::IndexOf($lines, $fromline)
	$fromline = "{0}{1}" -f $fromline.Replace(";", "@example.com;"), "@example.com"
	$lines[$tolineindex] = $toline
	$lines[$fromlineindex] = $fromline
$lines | set-content $_.fullname

}

Open in new window

0
 
LVL 2

Author Comment

by:PSGITech
ID: 39256366
Great!!! thank you for you help, I now need to fun this against a folder that now has about 200,000 messages, this should make my life much easier, all the files have a .txt extension is there also a way to save them as .doc after the update is completed also?
0
 
LVL 42

Expert Comment

by:sedgwick
ID: 39256380
yes, and also i can change the script to run parallel jobs so it would take much less time to process the 200k files.

in the meantime here's the update script which saves as doc:
$rootfolder = 'c:\temp\test'
gci $rootfolder | %{
	$lines = gc $_.fullname | select -Skip 6
	$lines = ,"X-Receiver: bs@bm3.contoso" + $lines
	$toline = $lines | where {$_ -match "to:"}
	$tolineindex = [array]::IndexOf($lines, $toline)
	$toline = "{0}{1}" -f $toline.Replace(";", "@example.com;"), "@example.com"
	$fromline = $lines | where {$_ -match "from:"}
	$fromlineindex = [array]::IndexOf($lines, $fromline)
	$fromline = "{0}{1}" -f $fromline.Replace(";", "@example.com;"), "@example.com"
	$lines[$tolineindex] = $toline
	$lines[$fromlineindex] = $fromline
	$lines | Set-Content $_.fullname.Replace('.txt','.doc')
}

Open in new window

0
 
LVL 2

Author Comment

by:PSGITech
ID: 39256446
Thank you for all your help this is a hugh help after I have all the file updated and renamed I have to move them to a different folder so they will be picked up by my smtp server, i figured i'd do this with an xcopy job, unless it would be easy to add this to the script also?  When I run this on a folder with alot of files is there a way to monitor its progress, will it run through the files as its doing it or will it just show a blinking cursor?
0
 
LVL 42

Expert Comment

by:sedgwick
ID: 39256479
here the script with some nice UI progress bar and the copy stuff, change $target_dir to whatever u need.

$rootfolder = 'c:\temp'
$target_dir = '\\ws-meirr\shared\files'
$files = gci $rootfolder
$files | %{
	$i++
	Write-Progress -activity "Processing files" -status "Percent complete: " -PercentComplete (($i / $files.length)  * 100)
	$lines = gc $_.fullname | select -Skip 6
	$lines = ,"X-Receiver: bs@bm3.contoso" + $lines
	$toline = $lines | where {$_ -match "to:"}
	$tolineindex = [array]::IndexOf($lines, $toline)
	$toline = "{0}{1}" -f $toline.Replace(";", "@example.com;"), "@example.com"
	$fromline = $lines | where {$_ -match "from:"}
	$fromlineindex = [array]::IndexOf($lines, $fromline)
	$fromline = "{0}{1}" -f $fromline.Replace(";", "@example.com;"), "@example.com"
	$lines[$tolineindex] = $toline
	$lines[$fromlineindex] = $fromline
$target_file = $_.fullname.Replace('.txt','.doc')
	$lines | sc $target_file
copy-item $target_file $target_dir

}

Open in new window

0
 
LVL 2

Author Comment

by:PSGITech
ID: 39256591
I can't thank you enough for the help here, you've saved me a boat load of time here, when running this it creating a new copy of the file and saving it as a .doc and copying it to the other folder, if i change the command in line 19 to move-item will that work.  This way I don't have a .txt and .doc in the source directory.  I did try that and it seems to work but errors pertaining to percentcomplete Write-Progress : Cannot validate argument on parameter 'PercentComplete'. The 120 argument is greater than the maximum
allowed range of 100. Supply an argument that is less than 100 and then try the command again.
At line:3 char:94
0
 
LVL 42

Accepted Solution

by:
sedgwick earned 2000 total points
ID: 39256660
in regards to move-item, you better use this approach instead:
$rootfolder = 'c:\temp\test'
$targetfolder = 'c:\temp\test2'
$files = gci $rootfolder
$files | %{
	$i++
	Write-Progress -activity "Processing files" -status "Percent complete: " -PercentComplete (($i / $files.length)  * 100)
	$lines = gc $_.fullname | select -Skip 6
	$lines = ,"X-Receiver: bs@bm3.contoso" + $lines
	$toline = $lines | where {$_ -match "to:"}
	$tolineindex = [array]::IndexOf($lines, $toline)
	$toline = "{0}{1}" -f $toline.Replace(";", "@example.com;"), "@example.com"
	$fromline = $lines | where {$_ -match "from:"}
	$fromlineindex = [array]::IndexOf($lines, $fromline)
	$fromline = "{0}{1}" -f $fromline.Replace(";", "@example.com;"), "@example.com"
	$lines[$tolineindex] = $toline
	$lines[$fromlineindex] = $fromline
	$target_file = Join-Path $targetfolder $_.name.Replace('.txt','.doc')
	$lines | sc $target_file
}

Open in new window

0
 
LVL 2

Author Comment

by:PSGITech
ID: 39256762
As it looks now, this is working great, I will test it in a full batch after hours, thanks for all your help on this issue.
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Ready to improve network connectivity? Watch this webinar to learn how SD-WANs and a one-click instant connect tool can boost provisions, deployment, and management of your cloud connection.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Making a simple AJAX shopping cart Couple years ago I made my first shopping cart, I used iframe and JavaScript, it was very good at that time, there were no sessions or AJAX, I used cookies on clients machine. Today we have more advanced techno…
In this tutorial I will show you how to make a simple HTML bar chart with the usage of WhizBase, If you want more information about WhizBase please read my previous articles at http://www.experts-exchange.com/ARTH_5123186.html (http://www.experts-ex…
Learn the basics of while and for loops in Python.  while loops are used for testing while, or until, a condition is met: The structure of a while loop is as follows:     while <condition>:         do something         repeate: The break statement m…
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question