Korn Shell - Search Directory Structure with grep

I have this command now:

find . -type d | grep -i -v "agg" | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {print "export database " $2 "." $3 " level0 data to data_file '\''F:\\EssbaseBackup\\Daily\\Export\\" $2 $3 ".txt'\'';"}' >> $ess_scripts_bkup/DailyExport.bat


My dir structure looks like:

lev 1 Hyperion
lev 2      Essbase
lev 3           App
lev 4                fcl_agg
lev 5                     fcl_agg
lev 5                     default
lev 5                     log
lev 5                     metadata
lev 5                     temp
lev 4                fcl
lev 5                     fcl

I need the DailyExport.bat file in my command to change the way it's created.  Instead of excluding if it contains 'agg'.  I need it to look at lev 5 and if a dir named 'metadata' exists, then it excludes the lev 4 dir, which in this case would be 'fcl_agg'.   This way if the lev 4 dir name does not contain 'agg' but it does have a subdirectory named 'metadata', it still excludes it from the DailyExport.bat file.

elwayisgodAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
amit_gConnect With a Mentor Commented:
"test -d " $0 "/../metadata && echo Exists"

tests if the current directory being tested should be used later for export or not. It does that by checking if the parent directory contains a directory named as metadata. This is done using test. If you see a typical command is

test -d ./allocate/allocate/../metadata && echo Exists

This echoes Exists only if the current directory's parent directory contains a directory named as metadata. This word is stored in a variable named as output and is later tested.
0
 
amit_gCommented:
Try this way ...

find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {"file " $0 "/metadata" | getline output; if (output !~ /metadata: directory/) print "export database " $2 "." $3 " level0 data to data_file'\''F:\\EssbaseBackup\\Daily\\Export\\" $2 $3 ".txt'\'';"}'

It this doesn't work, execute the following command (from Hyperion directory) and post the output

file Essbase
file Essbase_DoesNotExists

0
 
elwayisgodAuthor Commented:
when I run the above modified: find command, i get the following which is missing all the 'export' lines.


/* ****************************************************************** */
/*   File is dynamically built by the DailyExport.ksh script!!!!!!!!  */
/* ****************************************************************** */
login 'cntrlm_hal'       'prod123$' on sdntheb2;


spool on to 'F:/EssbaseBackupLogs/Step_5.txt';


exit;
0
Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
elwayisgodAuthor Commented:
[F:/Hyperion] file essbase
essbase:      directory
[F:/Hyperion] file Essbase_DoesNotExists
Essbase_DoesNotExists: cannot open: The system cannot find the file specified.
0
 
elwayisgodAuthor Commented:
How do I get YYYY, MM and DD to display 2007, 03, 26?  Is this not correct?

#!/bin/ksh


Day=`date +%A`
YYYY='date +%y'
MM='date +%m'
DD='date +%d'
0
 
elwayisgodAuthor Commented:
got the date thing.  Wrong ' should have been ` :)
0
 
amit_gCommented:
In the command change

/metadata: directory/

to

/metadata:      directory/

make sure to have exactly as you get it from command

file Essbase

Is there a tab between : and directory?
0
 
elwayisgodAuthor Commented:
I'm struggling today.  I forgot how to copy a directory and all it's subdirectories and files.  I need

From:

F:\Program Files\cognos\cer3\upfront

To:

F:\cognos\backups\Cognos7_Daily_Backups\Monday\UpfrontThemes

This is what I have:  Sorry, I'm just struggling today.

mv F:/Program Files/cognos/cer3/templates/upfront/*.* F:\cognos\backups\Cognos7_Daily_Backups\Monday\UpfrontThemes\*.*
0
 
amit_gCommented:
cp -r  F:/Program Files/cognos/cer3/templates/upfront/* F:/cognos/backups/Cognos7_Daily_Backups/Monday/UpfrontThemes/
0
 
amit_gCommented:
This might work better ...

find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {"test -d " $0 "/metadata && echo Exists" | getline output; if (output !~ /Exists/) print "export database " $2 "." $3 " level0 data to data_file'\''F:\\EssbaseBackup\\Daily\\Export\\" $2 $3 ".txt'\'';"}'
0
 
elwayisgodAuthor Commented:
Why does this not work, I'm trying to get a directory created.

cd F:/cognos/backups/Cognos7_Daily_Backups/Monday

md templates
0
 
elwayisgodAuthor Commented:
md works when I type it into the shell, but mkdir only works in my .ksh file.  I'm going crazy here :)  Now I have a file named logfile.txt. I need to search this file for the word 'error'.  Then if 'error' is found I need to 'exit 1'.  How do I do that?  My current logic here is wrong because the $? variable is just whether the grep command ran successful.  What I need is to have it know if it found 'error' in the file or not.

Wrong:

grep -i error Logfile.txt >> $cog7_daily_bkup_ksh/$Day/Logfile.txt
 echo "$?" >> $cog7_daily_bkup_ksh/$Day/Logfile.txt
if [ "$?" -eq "1" ]; then
 echo "\nStep 18 - FAILURE!!!  Backup Process completed with ERRORS!!!  Please review the logfile.txt for more details!!!" >> $cog7_daily_bkup_ksh/$Day/Logfile.txt
 exit 1
else
 echo "\nStep 18 - Success!!   The BACKUP process had no ERRORS!!" >> $cog7_daily_bkup_ksh/$Day/Logfile.txt
fi

0
 
amit_gCommented:
$? is last command's return status. So you are checking status of

echo "$?" >> $cog7_daily_bkup_ksh/$Day/Logfile.txt

as that is the command before the if and echo would always be successful. do it this way...

grep -i error Logfile.txt >> $cog7_daily_bkup_ksh/$Day/Logfile.txt
SavedReturnValue="$?"
echo "$SavedReturnValue" >> $cog7_daily_bkup_ksh/$Day/Logfile.txt
if [ "$SavedReturnValue" -ne "0" ]; then

If you notice, I changed the if condition also to -ne 0 as 0 is the success and anything other than 0 is failure. The command could return 1 for one failure, 2 for another and so on.
0
 
amit_gCommented:
Oh one more thing, you since the return status is numeric, you need not use "".

grep -i error Logfile.txt >> $cog7_daily_bkup_ksh/$Day/Logfile.txt
SavedReturnValue=$?
echo $SavedReturnValue >> $cog7_daily_bkup_ksh/$Day/Logfile.txt
if [ $SavedReturnValue -ne 0 ]; then
0
 
elwayisgodAuthor Commented:
Here's Logfile.txt

Step 1 Process Completed Successfully!
Step 2 Process Completed Successfully!
Step 3 Process Completed Successfully!
Step 4 Process Completed Successfully!
Step 5 Process Completed Successfully!
Step 6 Process Completed Successfully!
Step 7 Process Completed Successfully!
Step 8 Process Completed Successfully!
Step 9 Process Completed Successfully!
Step 10 - Success!! Removal of 'templates' UpFrontThemes folder was successfull!
Step 10 - Success!! Removal of 'webcontent' UpFrontThemes folder was successfull!
Step 10 - Success!! Korn Shell Execution Completed Successfully!
Step 11 Process Completed Successfully!
Step 12 Process Completed Successfully!
Step 13 Process Completed Successfully!
Step 14 Process Completed Successfully!
Step 15 Process Completed Successfully!
Step 16 Process Completed Successfully!
Step 17 Process Completed Successfully!
Step 18 - Success!! Removal of current days backup folder was successfull!
Step 18 - Success!! Creation of current days backup folder was successfull!
Step 18 - Success!! Move of working backup folder to current days backup folder  was successfull!
Step 18 - Success!! Removal of 'working' Folder was successfull!

0
 
elwayisgodAuthor Commented:
Here's the results of the code.  There is no 'error' in the file, thus why would it return a '1'?

1

Step 18 - FAILURE!!!  Backup Process Failed!!!  Please review the logfile.txt for more details!!!
0
 
amit_gCommented:
grep returns 0 (success) when a match is found. So your test should be

if [ $SavedReturnValue -eq 0 ]; then
0
 
elwayisgodAuthor Commented:
Got that to work.  Now the original issue, have you had any thoughts on it?  Is it possible?  I will increase pts since you solved many issues for me today.
0
 
amit_gCommented:
Try my last comment at 03.26.2007 at 02:49PM PDT
0
 
elwayisgodAuthor Commented:
Still Missing the 'export' lines in the DailyExport.bat file :(

/* ****************************************************************** */
/*   File is dynamically built by the DailyExport.ksh script!!!!!!!!  */
/* ****************************************************************** */
login 'cntrlm_hal'       'prod123$' on sdntheb2;


spool on to 'F:/EssbaseBackupLogs/Step_5.txt';


exit;
0
 
amit_gCommented:
Next to
find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {"test -d " $0 "/metadata && echo Exists" | getline output; if (output !~ /Exists/) print "export database " $2 "." $3 " level0 data to data_file'\''F:\\EssbaseBackup\\Daily\\Export\\" $2 $3 ".txt'\'';"}'

add these line to debug

find . -type d | grep -i -v "DM_APP"
find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {print}'
find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {"test -d " $0 "/metadata && echo Exists" | getline output; print "output : " output;}'

and post the output.
0
 
elwayisgodAuthor Commented:
What do you mean by next to? Just put them below or on the same lines?
0
 
amit_gCommented:
Below.
0
 
elwayisgodAuthor Commented:
.
./allocate
./allocate/allocate
./allocate/allocate/dmf
./allocate/allocate/trig
./fcl
./fcl/fcl
./fcl/fcl/dmf
./fcl/fcl/trig
./fcl_agg
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/fcl_agg/trig
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam
./sam/test
./sam/test/dmf
./sam/test/trig
./sam/test2
./sam/test2/trig
./allocate/allocate
./fcl/fcl
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam/test
./sam/test2
output :
output :
output :
output :
output :
output :
output :
output :
output :
0
 
amit_gCommented:
Change those lines to

find . -type d | grep -i -v "DM_APP"
echo
find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {print}'
echo
find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {"test -d " $0 "../metadata && echo Exists" | getline output; print "output : " output;}'
0
 
elwayisgodAuthor Commented:
Here is output separated by ***** for each command

.
./allocate
./allocate/allocate
./allocate/allocate/dmf
./allocate/allocate/trig
./fcl
./fcl/fcl
./fcl/fcl/dmf
./fcl/fcl/trig
./fcl_agg
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/fcl_agg/trig
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam
./sam/test
./sam/test/dmf
./sam/test/trig
./sam/test2
./sam/test2/trig
/* ***************************************************************** */
./allocate/allocate
./fcl/fcl
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam/test
./sam/test2
/* ***************************************************************** */
output :
output :
output :
output :
output :
output :
output :
output :
output :
0
 
elwayisgodAuthor Commented:
read your mind :)
0
 
amit_gCommented:
Change

/metadata

to

../metadata

in 2 lines you have in your code.
0
 
elwayisgodAuthor Commented:
on this one ?  03.26.2007 at 07:18PM MST
0
 
amit_gCommented:
Yes

find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {"test -d " $0 "../metadata && echo Exists" | getline output; if (output !~ /Exists/) print "export database " $2 "." $3 " level0 data to data_file'\''F:\\EssbaseBackup\\Daily\\Export\\" $2 $3 ".txt'\'';"}'

and the debug line also

find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {"test -d " $0 "../metadata && echo Exists" | getline output; print "output : " output;}'
0
 
elwayisgodAuthor Commented:
/* ****************************************************************** */
/*   File is dynamically built by the DailyExport.ksh script!!!!!!!!  */
/* ****************************************************************** */
login 'cntrlm_hal'       'prod123$' on sdntheb2;


spool on to 'F:/EssbaseBackupLogs/Step_5.txt';


exit;
0
 
amit_gCommented:
What was the output of debug line with this change?
0
 
elwayisgodAuthor Commented:
I put 'YES' or 'NO' next to the lines I would want to appear in the 'export' line, maybe this will help?


/allocate
./allocate/allocate
./allocate/allocate/dmf
./allocate/allocate/trig
./fcl
./fcl/fcl
./fcl/fcl/dmf
./fcl/fcl/trig
./fcl_agg
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/fcl_agg/trig
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam
./sam/test
./sam/test/dmf
./sam/test/trig
./sam/test2
./sam/test2/trig
/* ***************************************************************** */
./allocate/allocate = YES
./fcl/fcl =YES
./fcl_agg/default =NO
./fcl_agg/fcl_agg =NO
./fcl_agg/log =NO
./fcl_agg/metadata =NO
./fcl_agg/temp =NO
./sam/test =YES
./sam/test2 =YES
/* ***************************************************************** */
output :
output :
output :
output :
output :
output :
output :
output :
output :
0
 
elwayisgodAuthor Commented:
whoops, forgot to direct the output to my file, here it is:

/* ****************************************************************** */
/*   File is dynamically built by the DailyExport.ksh script!!!!!!!!  */
/* ****************************************************************** */
login 'cntrlm_hal'       'prod123$' on sdntheb2;


spool on to 'F:/EssbaseBackupLogs/Step_5.txt';


.
./allocate
./allocate/allocate
./allocate/allocate/dmf
./allocate/allocate/trig
./fcl
./fcl/fcl
./fcl/fcl/dmf
./fcl/fcl/trig
./fcl_agg
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/fcl_agg/trig
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam
./sam/test
./sam/test/dmf
./sam/test/trig
./sam/test2
./sam/test2/trig

./allocate/allocate
./fcl/fcl
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam/test
./sam/test2

output :
output :
output :
output :
output :
output :
output :
output :
output :
exit;
0
 
amit_gCommented:
Hmm. what is the output of command

test -d ExistingDir
echo $?
test -d NonExistingDir
echo $?

Replace ExistingDir with the value of some existing dir and vice versa for NonExistingDir
0
 
amit_gCommented:
How did you get YES and NO in the output?
0
 
elwayisgodAuthor Commented:
How about something like this.  Do something like:

If 'metadata' exists as the level 2 dir, then change the name of level 1 to 'agg'.  And then ignore 'agg'.  Don't actually change the name of the directory, but only do it temporariy while this runs.  So if you had a directory:

test/test
test/metadata

It would somehow temporarily change it to:
agg/test
agg/metadata

Then ignore 'agg' in the print of the 'export' line?

Just trying to think of alternate ways that may make it easier.
0
 
elwayisgodAuthor Commented:
Manually typed it in.  
0
 
amit_gCommented:
>>Manually typed it in.  

That is what I was wondering as it seemed like perfect output but I had no idea what code would have produced YES and NO :)
0
 
elwayisgodAuthor Commented:
I see what your thinking.  Is there a way to get Yes and No in the lines and only print the YES ones?
0
 
elwayisgodAuthor Commented:
Now I'm trying to brainstorm, do you have a IM account on MSN or Yahoo?  Might be easier to exchange ideas?
0
 
amit_gCommented:
That is what output var should have. Execute this and post the result...

echo '$$$$'
test -d ExistingDir
echo $?
test -d NonExistingDir
echo $?
test -d ExistingDir && echo Exists
test -d NonExistingDir && echo Exists
echo '$$$$'
0
 
elwayisgodAuthor Commented:
Post it with the original and the debug lines or by itself?  
0
 
amit_gCommented:
No this is from command line. I need to see if test command does what I expect it to.
0
 
elwayisgodAuthor Commented:
If just paste it below the others and run I get the same:

/* ****************************************************************** */
/*   File is dynamically built by the DailyExport.ksh script!!!!!!!!  */
/* ****************************************************************** */
login 'cntrlm_hal'       'prod123$' on sdntheb2;


spool on to 'F:/EssbaseBackupLogs/Step_5.txt';


.
./allocate
./allocate/allocate
./allocate/allocate/dmf
./allocate/allocate/trig
./fcl
./fcl/fcl
./fcl/fcl/dmf
./fcl/fcl/trig
./fcl_agg
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/fcl_agg/trig
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam
./sam/test
./sam/test/dmf
./sam/test/trig
./sam/test2
./sam/test2/trig

./allocate/allocate
./fcl/fcl
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./sam/test
./sam/test2

output :
output :
output :
output :
output :
output :
output :
output :
output :
exit;
0
 
elwayisgodAuthor Commented:
Disregard last post. I forgot to redirect output, I'm retarted.  Here it is:

$$$$
1
1
$$$$
0
 
amit_gCommented:
Use this command in place of what you have now ...

find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {output=""; "test -d " $0 "/../metadata && echo Exists" | getline output; if (output !~ /Exists/) print "export database " $2 "." $3 " level0 data to data_file'\''F:\\EssbaseBackup\\Daily\\Export\\" $2 $3 ".txt'\'';"}'
0
 
elwayisgodAuthor Commented:
Still same result - no export lines in the .bat.  I was thinking.  If you have a dir  fcl_agg/metadata.  If 'fcl_agg' is $2 and 'metadata' is $3, couldnlt you do something like:

If $3 = 'metadata' then make the corresponding $2 equal to $skip, which in this case would be 'fcl_agg'

Then plug in $skip into original find statement???

find . -type d | grep -i -v $skip | grep -i -v "DM_APP" | awk -F/ 'NF ==
3 {print "export database " $2 "." $3 " level0 data to data_file
'\''F:\\EssbaseBackup\\Daily\\Export\\" $2 $3 ".txt'\'';"}' >>
$ess_scripts_bkup/DailyExport.bat
0
 
amit_gCommented:
What directory this is being executed from? Next to the last command (latest version), add the debug lies and post he result. You can remove the old debug lines.

find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {output=""; "test -d " $0 "/../metadata && echo Exists" | getline output; if (output !~ /Exists/) print "export database " $2 "." $3 " level0 data to data_file'\''F:\\EssbaseBackup\\Daily\\Export\\" $2 $3 ".txt'\'';"}'
echo "****"
find . -type d | grep -i -v "DM_APP"
echo "****"
find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {print}'
echo "****"
find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {print "test -d " $0 "/../metadata && echo Exists"}'
echo "****"
find . -type d | grep -i -v "DM_APP" | awk -F/ 'NF == 3 {output=""; "test -d " $0 "/../metadata && echo Exists" | getline output; print "output: " output}'

$3 is metadata only when we get to ./fcl_agg/metadata. For other (like ./fcl_agg/log) it is log but we still want to ignore.
0
 
elwayisgodAuthor Commented:
vpn went down, will have to post tomorrow.  Is that ok?
0
 
amit_gCommented:
Sure.
0
 
elwayisgodAuthor Commented:
I believe it works.  Now can you explain the code so I know what it's doing?  I appreciate your help.

/* ****************************************************************** */
/*   File is dynamically built by the DailyExport.ksh script!!!!!!!!  */
/* ****************************************************************** */
login 'cntrlm_hal'       'prod123$' on sdntheb2;


spool on to 'F:/EssbaseBackupLogs/Step_5.txt';


export database allocate.allocate level0 data to data_file'F:\EssbaseBackup\Daily\Export\allocateallocate.txt';
export database fcl.fcl level0 data to data_file'F:\EssbaseBackup\Daily\Export\fclfcl.txt';
export database sam.test level0 data to data_file'F:\EssbaseBackup\Daily\Export\samtest.txt';
export database sam.test2 level0 data to data_file'F:\EssbaseBackup\Daily\Export\samtest2.txt';
****
.
./allocate
./allocate/allocate
./allocate/allocate/dmf
./allocate/allocate/trig
./aso_test
./aso_test/default
./aso_test/log
./aso_test/metadata
./aso_test/sammy
./aso_test/sammy/trig
./aso_test/temp
./fcl
./fcl/fcl
./fcl/fcl/dmf
./fcl/fcl/trig
./fcl_agg
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/fcl_agg/trig
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./loan
./loan/default
./loan/loancube
./loan/loancube/trig
./loan/log
./loan/metadata
./loan/temp
./sam
./sam/test
./sam/test/dmf
./sam/test/trig
./sam/test2
./sam/test2/trig
****
./allocate/allocate
./aso_test/default
./aso_test/log
./aso_test/metadata
./aso_test/sammy
./aso_test/temp
./fcl/fcl
./fcl_agg/default
./fcl_agg/fcl_agg
./fcl_agg/log
./fcl_agg/metadata
./fcl_agg/temp
./loan/default
./loan/loancube
./loan/log
./loan/metadata
./loan/temp
./sam/test
./sam/test2
****
test -d ./allocate/allocate/../metadata && echo Exists
test -d ./aso_test/default/../metadata && echo Exists
test -d ./aso_test/log/../metadata && echo Exists
test -d ./aso_test/metadata/../metadata && echo Exists
test -d ./aso_test/sammy/../metadata && echo Exists
test -d ./aso_test/temp/../metadata && echo Exists
test -d ./fcl/fcl/../metadata && echo Exists
test -d ./fcl_agg/default/../metadata && echo Exists
test -d ./fcl_agg/fcl_agg/../metadata && echo Exists
test -d ./fcl_agg/log/../metadata && echo Exists
test -d ./fcl_agg/metadata/../metadata && echo Exists
test -d ./fcl_agg/temp/../metadata && echo Exists
test -d ./loan/default/../metadata && echo Exists
test -d ./loan/loancube/../metadata && echo Exists
test -d ./loan/log/../metadata && echo Exists
test -d ./loan/metadata/../metadata && echo Exists
test -d ./loan/temp/../metadata && echo Exists
test -d ./sam/test/../metadata && echo Exists
test -d ./sam/test2/../metadata && echo Exists
****
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
output:
exit;
0
 
elwayisgodAuthor Commented:
Raised as helped with multiple issues.  Awesome!!!!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.