Solved

# Need a way to loop a input file

Posted on 2009-02-16
331 Views
Hello all,  I am in need of a way to create a loop effect to enter information from this file 2 lines at a time. I need a way to strip this file and have it run. The way it stands now, the CLI ( command line interface does not understand multiple lines. So I need to break it down by these two variables

Domain=emcbi.com beer.com is.beer.com fi.beer.com emc.beer.com

The CLI input string is used like to

qip-setsubnet - u username -p password -f <input file name > -o "Organization name" -t <to append these to existing subnets already in  the system>

so

qip-setsubnet - u username -p password -f e:\ 4-5-fix-subnets.txt -o rich -t

I have perl  on my system if this is the way to go, but I dont have the skills yet to do the above. I can except any ideas that will make this work. I need to be able to take this solution and use it on either windows or Sun O/S's

I have attached the file that needs to be stripped and ran with two variables at a time.

4-5-fix-subnets.txt
0
Question by:richsark

LVL 40

Expert Comment

see the script below which is an example:

cat 4-5-fix-subnets.txt | while read line1
do
echo "$line1$line2"
done

you may replace the echo line by any line of your's
0

LVL 1

Author Comment

Hi, so how do I call it to run on the next set all the way down the file?
Thanks
0

LVL 40

Expert Comment

it does that in the while loop, it reads one line with the while read and the next line by the 2nd read. this will loop till the input file is read till end of file
0

LVL 1

Author Comment

Ok. so where do I put my cli string to get them in?
qip-setsubnet - u username -p password -f e:\ 4-5-fix-subnets.txt -o rich -t
also, when I ran a test of the cat  it shows this
SubnetAddress=172.30.80.0 Domain=beer.com is.beer.com fi.beer.com emc.beer.com encorecredit.com
SubnetAddress=172.30.82.0 Domain=beer.com is.beer.com fi.beer.com emc.beer.com encorecredit.com
SubnetAddress=207.162.226.0 Domain=bear.com is.beer.com fi.beer.com emc.beer.com clearco.com
I need it to be on two lines like so:
Domain=bear.com is.beer.com fi.bear.com emc.beer.com encorecredit.com

Thanks
0

LVL 40

Expert Comment

cat 4-5-fix-subnets.txt | while read line1
do
echo $line1 echo$line2
done

I am not sure what you want to do with

qip-setsubnet - u username -p password -f e:\ 4-5-fix-subnets.txt -o rich -t
0

LVL 1

Author Comment

Hello Sir,
in order to get this two lines in the system, I have to call a CLI (see above) and reference the txt file.
The issue I have is that the CLI does not understand this massive file and I need to call it by two lines at a time. The two lines are mandatory and must have the header
Domain=
So I need a way to automate this so its takes my input file, strip it down into two headers have them go into the system use the
CLI qip-setsubnet string and run all the way down the files.
0

LVL 40

Expert Comment

can you give example of how the cli should be called with sample data?
0

LVL 1

Author Comment

Hi,
I need to take the file I have attached "4-5-fix-subnets.txt "
cut it down into two headers

Domain=
and run using the cli to call the file.
So I guess need to perhaps pipe each of the two headers out into a scratch file that makes an input file that is called by the CLI and the whole process repeats all the way down the master file called 4-5-fix-subnets.txt

Does that help?
0

LVL 28

Expert Comment

What is the syntax of the qip-setsubnet command when passing the 2 parameters instead of the file?
0

LVL 40

Expert Comment

try this:

cat 4-5-fix-subnets.txt | while read line1
do
echo "$line1$line2" | qip-setsubnet - u username -p password -o rich -t
done
0

LVL 1

Author Comment

Hi,
We need to create a scratch file called perhaps step1.txt which has the two headers from the master file "4-5-fix-subnets.txt"
Then take the file step1.txt and have it standby so the CLI can call it using
qip-setsubnet - u username -p password -f e:\ step1.txt -o rich -t
Then it will need to repeat to the next set ofheaders create another file called perhaps step2.txt have it standby while the CLI is called agian to reference the step2.txt
qip-setsubnet - u username -p password -f e:\ step2.txt -o rich -t
and so on.... all the way down the master file "4-5-fix-subnets.txt"
Does that help?
0

LVL 40

Expert Comment

try

cat 4-5-fix-subnets.txt | while read line1
do
echo "$line1$line2" > step.txt
done

0

LVL 1

Author Comment

Hello, so your last commet will run all the way down the file calling the same file step.txt until its done? so it step.txt will be overwritten with the next step right?
Is there a stop or a way to find out if it failed at some stage?
0

LVL 40

Expert Comment

yes it will overwrite the temp file.

you could save output or errors into file:

cat 4-5-fix-subnets.txt | while read line1
do
echo "$line1$line2" > step.txt
qip-setsubnet - u username -p password -f step.txt -o rich -t >> log.txt 2>&1
done
0

LVL 1

Author Comment

Ok, will try it now and let you know in a few minutes. I need to install CYGWIN on my windows to test this as I dont have access today to a sun box
Thanks
0

LVL 1

Author Comment

HI, I am having a rough time getting this to work on windows, I installed cygwin but I am getting errors

C:\cygwin\home\false>cat  E:\FalseRoot-New\2-9-09\4-5-fix-subnets.txt   | while read line1
'while' is not recognized as an internal or external command,
operable program or batch file.

Any ideas?
0

LVL 28

Expert Comment

You need to be in a shell environment.

First, execute bash.exe (which is in the cygwin bin directory), then run the command as omarfarid showed.
0

LVL 28

Expert Comment

Instead of executing bash.exe directly, you could run cygwin.bat and it will cd you into C:\cygwin\bin and then execute 'bash --login -i'

When you installed cygwin, it should have created a link in "Start -> Programs -> Cygwin"
0

LVL 1

Author Comment

Ok, will do that now, also is there a way to make the output of the above script to left justify?

It seems that the Domain= is tabbed over more the SubnetAddress=

See below:

Domain=disclosure.com private.disclosure.com
Domain=beers.companystore.com

Thanks
0

LVL 1

Author Comment

HI, back to Fishmoger, yes I have a shortcut on my desk top. After I launched cygwin, I ran

my file is called setsub.pl  its located in /home/Administrator/

The contents of setsub.pl is

cat 4-5-fix-subnets.txt | while read line1
do
echo "$line1$line2" > step.txt
qip-setsubnet - u name-p pass -f /home/Administrator/4-5-fix-subnets.txt -o rich -t >> log.txt 2>&1
done
~

I executed ./setsub.pl

I got an error:

./setsub.pl: line 8: syntax error: unexpected end of file

Also, any ideas how to make the output line up to the left as my previous note above. I guess first things first :)

Thanks
0

LVL 28

Expert Comment

The .pl extension is normally for Perl scripts.  The code that omarfarid provided would be a shell script.  Which did you want?

If you want, I could show you the Perl script version.
0

LVL 28

Expert Comment

#!/usr/bin/perl

use strict;

use warnings;

open my $FH1, '<', '4-5-fix-subnets.txt' or die "Can't open '4-5-fix-subnets.txt'$!";

while ( my $line1 = <$FH1> ) {

my $line2 = <$FH1>;

open my $FH2, '>', 'step.txt' or die "Can't open 'step.txt'$!";

print $FH2$line1,$line2; system("qip-setsubnet -u name -p pass -f /home/Administrator/4-5-fix-subnets.txt -o rich -t >> log.txt 2>&1"); }  0 LVL 1 Author Comment Hi FishMonger, please show me the perl version as another option. I like Omarfarid's version but I am against the clock and I need to test this ASAP. Could I also ask for a way to make the output left justify, I cant have any tabs where Domain= SubnetAddress=10.3.2.0 Domain=disclosure.com private.disclosure.com SubnetAddress=10.3.4.0 Domain=beers.companystore.com Need it to look like: SubnetAddress=10.3.2.0 Domain=disclosure.com private.disclosure.com SubnetAddress=10.3.4.0 Domain=beers.companystore.com Thanks all ! 0 LVL 28 Expert Comment I forgot 1 line. Add this just after the print statement close$FH2;
0

LVL 28

Expert Comment

If there is leading whitespace on the Domain= line, you can strip it out with a regex (regular expression).
#!/usr/bin/perl

use strict;

use warnings;

open my $FH1, '<', '4-5-fix-subnets.txt' or die "Can't open '4-5-fix-subnets.txt'$!";

while ( my $line1 = <$FH1> ) {

my $line2 = <$FH1>;

$line2 =~ s/^\s*//; open my$FH2, '>', 'step.txt' or die "Can't open 'step.txt' $!"; print$FH2 $line1,$line2;

close $FH2; system("qip-setsubnet -u name -p pass -f /home/Administrator/4-5-fix-subnets.txt -o rich -t >> log.txt 2>&1"); }  0 LVL 28 Expert Comment One more thing, it might be better to use single quotes instead of double quotes in the system call. 0 LVL 28 Expert Comment Hmm, the woos of not paying attention when doing copy/paste. system("qip-setsubnet -u name -p pass -f /home/Administrator/4-5-fix-subnets.txt -o rich -t >> log.txt 2>&1"); should be: system('qip-setsubnet -u name -p pass -f /home/Administrator/step.txt -o rich -t >> log.txt 2>&1'); 0 LVL 1 Author Comment qip-setsubnet.exe :/home/Administrator/4-5-fix-subnets.txt: Invalid File Name?!qip-setsubnet.exe :/home/Administrator/4-5-fix-subnets.txt: Invalid File Name?! I have the files located in /home/Administrator Thanks 0 LVL 1 Author Comment Hi FishMonger, I modified your above with: system('qip-setsubnet -u name -p pass -f /home/Administrator/step.txt -o rich -t >> log.txt 2>&1'); When I run t he script and look at the log.txt I see tons of messages: qip-setsubnet.exe :/home/Administrator/step.txt: Invalid File Name?!qip-setsubnet.exe :/home/Administrator/step.txt: Invalid File Name?! Thanks 0 LVL 1 Author Comment FishMonger, I ran the file manually, I get the same message, but the dir and path are good, see below: Administrator@mobile6 ~$ /cygdrive/e/qip-62/cli/qip-setsubnet -u xx -p xx -f /home/Administrator/step.txt -o rich -t

$more step.txt SubnetAddress=63.247.112.0 Domain=econfirm.com theice.com Administrator@mobile6 ~$ pwd

The file is there, I am not understanding why its invalid??
0

LVL 28

Expert Comment

I suspect that qip-setsubnet.exe expects a Windows path.  Exit out of the bash shell and run the perl script from a normal Windows command prompt.  You'll also need to change the path in the system call to use the full Windows path to the file.  I don't know if the cygwin install setup the proper file ext associations you may need to specify the full path to perl.exe and pass your script as a parameter.

Like this (but use the proper paths on your system):

c:\perl\bin\perl c:\cygwin\home\Administrator\setsub.pl

0

LVL 1

Author Comment

Cool, that worked in Windows after I made the path changes!! Awesome !

I guess when I goto to the office, I will need to modify the script from windows to unix. I Assume it will work the same :)

Thanks guys.

0

LVL 28

Expert Comment

Yes, it will work the same once the path is adjusted.  With a few extra lines we could make it so that it will work on either system without having to adjust paths.
0

LVL 1

Author Comment

Hi FishMonger, do you want to do that? If not I can modify the file. Its up to you !

Thanks
0

LVL 1

Author Comment

Hi FishMonger,  in reference to the log.txt file. Does that get over-written on each new step.txt call?

I like for the logs to not over-write so I know what failed ( if any )

Could you confirm and or modify the perl script please?

Thanks!

0

LVL 28

Expert Comment

There are several ways to check which OS the script is running on and the one I'm using is probably the most common.

I changed the name of the output file from step.txt to subnet.txt because that seemed more accurate.  I also corrected the output to the log file so that it will append instead of overwrite.

This script has the bare minimum amount of error handling.  Normally I'd add more, but this should get you started.
#!/usr/bin/perl

use strict;

use warnings;

my $subnet; if ($^O eq 'MSWin32' ) {

$subnet = 'c:\path\to\subnet.txt'; } else {$subnet = '/path/to/subnet.txt';

}

open my $FH1, '<', '4-5-fix-subnets.txt' or die "Can't open '4-5-fix-subnets.txt'$!";

while ( my $line1 = <$FH1> ) {

my $line2 = <$FH1>;

open my $FH2, '>>', 'subnet.txt' or die "Can't open 'subnet.txt'$!";

print $FH2$line1,$line2; close$FH2;

system("qip-setsubnet -u name -p pass -f $subnet -o rich -t >> log.txt 2>&1"); }  0 LVL 1 Author Comment Hi FishMonger, It appears that this new script will run on Win and Sun without any modifications besides where the files are? I appericate your help and everyone else involved. I will try this new script on a Sun box when I get to the office and will update you during the day. Cheers ! 0 LVL 1 Author Comment HI FishMOnger, I ran the perl scripts on my Sun box today in the office, But I got these errors: Can't locate warnings.pm in @INC (@INC contains: /usr/perl5/5.00503/sun4-solaris /usr/perl5/5.00503 /usr/perl5/site_perl/5.005/sun4-solaris /usr/perl5/site_perl/5.005 .) at ./setsubnet-Unix.pl line 24. BEGIN failed--compilation aborted at ./setsubnet-Unix.pl line 24. I used your old veriosn of the code: use strict; use warnings; open my$FH1, '<', '4-5-fix-subnets.txt' or die "Can't open '4-5-fix-subnets.txt' $!"; while ( my$line1 = <$FH1> ) { my$line2 = <$FH1>; open my$FH2, '>', 'step.log' or die "Can't open 'step.log' $!"; print$FH2 $line1,$line2;
system('/opt/qip/usr/bin/qip-setsubnet -u Xxx -p Xxx-f /opt/qip/rich/scripts/step.log -o rich -t >> setsubnet.log 2>&1'
);
}

Any clues? or how can we modify it to work with this version of perl?
0

LVL 1

Author Comment

HI, I also tried to take out the two warnings from the file, but then I got different errors:
The code looks like:
open my $FH1, '<', '4-5-fix-subnets.txt' or die "Can't open '4-5-fix-subnets.txt'$!";
while ( my $line1 = <$FH1> ) {
my $line2 = <$FH1>;
open my $FH2, '>', 'step.log' or die "Can't open 'step.log'$!";
print $FH2$line1,$line2; system("/opt/qip/usr/bin/qip-setsubnet -u XxX -p XxX-f /opt/qip/rich/scripts/step.log -o rich-t >> setsubnet.log 2>&1"); The error is : <', '4-5-fix-subnets.txt' or die "Can't open '4-5-fix-subnets.txt'$!";
while ( my $line1 = <$FH1> ) {
my $line2 = <$FH1>;
open my $FH2, '>', 'step.log' or die "Can't open 'step.log'$!";
print $FH2$line1,$line2; system("/opt/qip/usr/bin/qip-setsubnet -u XxX -p XxX -f /opt/qip/rich/scripts/step.log -o rich -t >> setsubnet.log 2>&1"); Please Help ! 0 LVL 28 Expert Comment That's a very old/outdated version (circa 1998). I would highly recommend upgrading perl to at least 5.8.6 or preferably 5.10 That old version doesn't support the warnings or strict pragmas i.e., use strict; use warnings; It also doesn't support the use of the my keyword when declaring the vars as well as a number of other features. Those features were introduced in 2000 in either v5.5 or v5.6 See if this version works better. #!/usr/bin/perl -w if ($^O eq 'MSWin32' ) {

$subnet = 'c:\path\to\subnet.txt'; } else {$subnet = '/path/to/subnet.txt';

}

open FH1, '4-5-fix-subnets.txt' or die "Can't open '4-5-fix-subnets.txt' $!"; while ($line1 = <FH1> ) {

$line2 = <FH1>; open FH2, '>>subnet.txt' or die "Can't open 'subnet.txt'$!";

print FH2 $line1,$line2;

close FH2;

system("qip-setsubnet -u name -p pass -f $subnet -o rich -t >> log.txt 2>&1"); } close FH1;  0 LVL 28 Expert Comment I just noticed a goof on my part. I'm opening subnet.txt in apend mode, instead it should overwite the file. change open FH2, '>>subnet.txt' or die "Can't open 'subnet.txt'$!";

to
open FH2, '>subnet.txt' or die "Can't open 'subnet.txt' \$!";
0

LVL 1

Author Comment

Hi, Yea it was a surprise for me as well :)

I will try it in the morning.  I am not able to upgrade perl in the limited time I have. So I hope your script will have the work around for the version I have.

Do you have any doubts?

Thanks

0

LVL 1

Author Comment

Hey FishMonger,

Great work, the new code worked for the perl I have.  I will run this in production this weekend and let you know.

Thanks a bunch !
0

LVL 1

Author Comment

Hi FishMonger, can I ask you for a slight change in your perl to work with another request that is simular with what we done above?
Here is what I need to do
take a file called 2.1-rr-for-dns.csv using the same string with a different word:
qip-setzoneprof -u name -p pass -f 2.1-rr-for-dns.csv  -o rich
and have it run one section at a time: a section will be:
see file attached ( for format purposes)
you will notice a section is from rows 1 through 8 (and of course retain the colums)
I need to strip out a master file like we did above and run it 1 section at a time. A new section starts at Row 9 just so we are on the same page
I will throw in more points for ya !
Can you help me today :)
Thanks

2.1-rr-for-dns.csv.xls
0

LVL 28

Expert Comment

It would be more appropriate to close this question and start a new one with the new requirements.

The file you've attached is not a plain text csv file, it's a binary excel spreadsheet.  The parsing of that file will be different and a little more complex.  If there aren't too many records in the file, you could add a blank line between each record which will help to simplify the parsing.  Saving/exporting the file to a plain text csv file would also help to simply the parsing.
0

LVL 1

Author Comment

Hi, Sorry about that, I ment to save it as a text file. I will re-upload.

Since it was my mistake on the file extension, could you take a look at it?

If not, I would be more then happy to open a new thread. But I was hoping that I could retain your services since you know what I need.

2.1-rr-for-dns.txt
0

LVL 1

Author Comment

HI, Sorry, I must be dumb, I attached the wrong format. It wont not allow me to upload as csv, I changed it to .txt

FishMonger. I would accept any ideas to get this to work. I am desperate !

Thanks

2.1-rr-for-dns.txt
0

LVL 28

Accepted Solution

I'm happy to help you on this next step, but it still would be more appropriate to close this question and start a new one.

I have some other chores that I need to work on right now and I might not be able to do anything on this until tomorrow.  So, it would be to your benefit to open a new question to give others a chance to help in the mean time.
0

LVL 1

Author Comment

Ok, cool, I was hoping I would have something by mid-afternoon Thurs. Anyway Please do what you can do, and I will open a new thread.

Thanks!
0

LVL 1

Author Comment

Thanks FishMonger !!
0

## Featured Post

I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
The following is a collection of cases for strange behaviour when using advanced techniques in DOS batch files. You should have some basic experience in batch "programming", as I'm assuming some knowledge and not further explain the basics. For some…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…