AIX/Linux - Permissions ksh script help

I'm working very large migration between different platforms. Part of my process, I need to capture the permissions and user/group ownership on the source area.  Than I recursively change permissions and ownership on the source target to an App ID that we need to make specific changes. After that, I migrate the data over to the new platform to the target area. Than I need to take the captured permissions from the source area and restore it to the target area. Also, I will need a log file to output any errors in the restore of permissions, such as "could not find directory or file". Another problem is that there are tons of imbedded spaces with file and directory names. Please help
AIX25Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

woolmilkporcCommented:
Hi,

I fear I didn't copmpletely understand your requrements.

But if your "source" platform is AIX I can at least help you with recording and recreating the permissions.

Here is a script which creates "chmod" statements according to the current permissions of all files/directories under the current directory.

#!/bin/ksh
find . | while read file
 do
  P=$(istat "$file" |grep -i prot | \
      awk '{P="u=" substr($2,1,3) ",g=" substr($2,4,3) ",o=" substr($2,7,3);
            gsub("-","",P); print P}')
  echo "chmod" $P "\"$file\""
 done

"cd" to the "source", then run the above script, redirecting its output to a file.
This output file will then contain the appropriate "chmod" statements.

Please note that broken lnks are reported on stderr during the script's execution.

Once you migrated the data to the new platform (including the directory structure, of course), you can "cd" to the target root there and simply run the script you just created.

Redirect stderr to a logfile ("scriptname 2>/path/to/logfile") to get a record of errors.

wmp
0
Ernie GronblomCloud Systems EngineerCommented:
Are there any files with any embedded double quotes (")?  Are the absolute paths of the files going to change?

Ernie
0
AIX25Author Commented:
@woolmilkporc:
Will your script factor/catch files with embedded spaces?

Also, the top level directories maybe different on the source and target areas, but all the sub-directores will be the same. For example:

source: /www/mobile

target:/mobile

Will this be an issue with you script and how do I handle that when i restore the permissions on the target area?
0
CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

woolmilkporcCommented:
The script works on filenames with embedded spaces, but not on filenames with embedded quotes (").

The second point is not an issue. Just run

cd /www/mobile
/path/to/script > /tmp/chmodscript 2>/tmp/script.errors

Now transfer "/tmp/chmodscript" to the target system and run there:

cd /mobile
sh /tmp/chmodscript 2>/tmp/chmod.errors
0
AIX25Author Commented:
Ok, I need to make sure I do check before this script run. Do you know of a way to check on ", \, leading spaces, trailing space, etc" filenames?

Also, is there anyway you can explain your script? I would like to understand it better.
0
woolmilkporcCommented:
You could run my script to create the "chmod" script and then check thoroughly what this new script contains. Let me know if you notice any blurs/inconsistencies etc.!
The script I posted just creates a new one if redirected, it won't do any harm to anything!

Are there indeed embedded backslashes? If so it's going to be a bit harder!

My script runs "find" to print out all files/dirs from the current directory and below.
The it runs the AIX tool "istat" against each of the files/dirs found (embedded spaces allowed) and uses grep/awk (awk could have done it alone, but anyway) to parse the line containing "Protection".

Such a line basically looks e.g like this:

Protection: rw-r--r--

We split the second field into three triplets and prefix each triplet with "u=" (user), "g=" (group) and "o=" (others) respectively, and separate them with commas.
Since dashes are not allowed with chmod we eliminate them and finally we print the result, which goes to a variable "P" due to command substitution.

Finally we echo out the "chmod" statement itself, the permission string we just created and the filename, the latter surrounded by protected quotes, so they will find their way to the output file, to keep chmod from complaining should the filename contain spaces.

A resulting line basically looks like this:

chmod u=rw,g=r,o=r "filename"
0
AIX25Author Commented:
One more requirement I need in my capture...I need to collect the timestamp on the files and directory too. Is there anyway you can integrate that with script?
0
woolmilkporcCommented:
Yes,

but I assume that you want to keep the functionality to create a chmod script, or am I wrong?

#!/bin/ksh
rm /tmp/chmodscript /tmp/timestampfile
find . | while read file
 do
  P=$(istat "$file" |grep -i prot | \
      awk '{P="u=" substr($2,1,3) ",g=" substr($2,4,3) ",o=" substr($2,7,3);
            gsub("-","",P); print P}')
  echo "chmod" $P "\"$file\"" >> /tmp/chmodscript
  echo "\"$file\"" $(istat "$file"  | grep "updated") >> /tmp/timestampfile
 done

If you prefer collecting the timestamp of the last modification or the one of the last access then grep for "modified" or "accessed" instead of "updated" in the new "echo" line.
0
AIX25Author Commented:
but I assume that you want to keep the functionality to create a chmod script, or am I wrong?

Sorry, I did not do a good job explaining this. What I need with the script you have in ID: 39518928 posting...is to add to it to capture timestamps on the source area, and that will allow me to restore that on the target area along with permission too of course.
0
woolmilkporcCommented:
OK,

here is a new version which will create one script containing the chmod statements like the previous version plus "touch" statements with the current "update" timestamps of the respective files as the "-t" parameter.

If you want two distinct scripts please add the appropriate redirections to the "echo" lines, or let me know if you want me to assist you.

#!/bin/ksh
find . | while read -r file
 do
  P=$(istat "$file" |grep -i prot | \
      awk '{P="u=" substr($2,1,3) ",g=" substr($2,4,3) ",o=" substr($2,7,3);
            gsub("-","",P); print P}')
  T=$(istat "$file" | grep -i updated | \
      awk '{M="0"(match("JanFebMarAprMayJunJulAugSepOctNovDec", $4)+2)/3;
            print $NF substr(M,length(M)-1,2) $5 substr($6,1,2) substr($6,4,2)}')
  echo "chmod" $P "\"$file\""
  echo "touch -t" $T  "\"$file\""
 done
0
AIX25Author Commented:
Ok just to confirm.

I take this script above and output it to a file and than execute when ready to restore the target area with the permissions and timestamp?

Also, will this capture the current timestamp on the source area and when I run the output file it will restore the "captured" timestamp on the target area?
0
woolmilkporcCommented:
Yes to all.

Please consider using "grep -i modified" instead of "grep -i updated" if you want to restore the file's modification date instead of the date where the inode was last updated. "ls" displays the modification date by default.

In fact, that's also the timestamp which "touch" sets, so using the modification date would make things more consistent.

#!/bin/ksh
find . | while read -r file
 do
  P=$(istat "$file" |grep -i prot | \
      awk '{P="u=" substr($2,1,3) ",g=" substr($2,4,3) ",o=" substr($2,7,3);
            gsub("-","",P); print P}')
  T=$(istat "$file" | grep -i modified | \
      awk '{M="0"(match("JanFebMarAprMayJunJulAugSepOctNovDec", $4)+2)/3;
            print $NF substr(M,length(M)-1,2) $5 substr($6,1,2) substr($6,4,2)}')
  echo "chmod" $P "\"$file\""
  echo "touch -t" $T  "\"$file\""
 done
0
AIX25Author Commented:
The script did not work properly. I received for every file and and directory, istat not found error. And the chmods and touches did not look correct. In this case...I had to run it from Red Ht...does that have anything to do with this error? Majority of the time I will be running this script on AIX and than taking my output file and executing it on Linux

My errors are below.

Output from when I ran the script
/script/path/perms_timestamp.ksh: line 4: istat: not found
/script/path/perms_timestamp.ksh: line 7: istat: not found

Snippet from my output file:
chmod "./filename"
touch -t "./filename"
chmod "./filename"
touch -t "./filename"
0
woolmilkporcCommented:
In my very first post I wrote

>> if your "source" platform is AIX <<

Now we can't do anything else than start over and make a completely new script for Linux.

I'll see if I can find the time.
0
woolmilkporcCommented:
Here is the script made portable between Linux and AIX.
You can run it on either platform without modification.
If it has to support additional OSes please let me know.

#!/bin/sh
find . | while read -r file
 do
   if [[ $(uname) = "AIX" ]] ; then
    OPT=""
    P=$(istat "$file" |grep -i prot | \
        awk '{P="u=" substr($2,1,3) ",g=" substr($2,4,3) ",o=" substr($2,7,3);
              gsub("-","",P); print P}')
    T=$(istat "$file" | grep -i modified | \
        awk '{M="0"(match("JanFebMarAprMayJunJulAugSepOctNovDec", $4)+2)/3;
              print $NF substr(M,length(M)-1,2) $5 substr($6,1,2) substr($6,4,2)}')
    elif [[ $(uname) = "Linux" ]] ; then
     OPT="-m"
     P=$(stat -c "%A" "$file" | \
         awk '{P="u=" substr($1,1,3) ",g=" substr($1,4,3) ",o=" substr($1,7,3);
               gsub("-","",P); print P}')
     T=$(stat -c "%y" "$file" | \
         awk -F"-| |:" '{print $1 $2 $3 $4 $5}')
     else echo "$(uname) Unsupported OS, exiting" ; exit
   fi
  echo "chmod" $P "\"$file\""
  echo "touch" $OPT "-t" $T  "\"$file\""
 done
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AIX25Author Commented:
I'm getting the error below using the script above. I have validated the Directory is there. This script has worked on hundreds of other embedded space directories, but for some odd reason the one below is not working. Please let me know what I can do.

stat: cannot stat `./apps/Admin/Prod/Data Accees Regulatory Policy': No such file or directory
0
woolmilkporcCommented:
Is this perhaps a dead link?

Issue

ls -l "./apps/Admin/Prod/Data Accees Regulatory Policy"

Do you see a link ("l" in the very first column, "-> ....." at the end)?

If it's a link - does the file/directory following "-> " exist? Check with "ls -l ....".

Or is there an embedded backslash in the name? My script does not (and will never) work on names containing backslashes!
0
AIX25Author Commented:
There is no link or dead link. Also, no embedded backslashes. Any other reason why only one directory of the whole area would fail??
0
woolmilkporcCommented:
Does this

ls -l "./apps/Admin/Prod/Data Accees Regulatory Policy"

indeed yield a result? If so, what exactly do you see?

If not please run

ls -bl ./apps/Admin/Prod/Data*

(using the "-b" flag of "ls", putting an asterisk after "Data" and omitting the quotes).

Is the directory in question among the displayed results? How exactly does it look?

And, by the way, is "Accees" a typo of yours, or should we dig deeper here?
0
AIX25Author Commented:
[root@servername development]# ls -l "./apps/Admin/Prod/Data Accees Regulatory Policy"

ls: cannot access ./apps/Admin/Prod/Data Accees Regulatory Policy: No such file or directory

 But....when I:
cd /apps/Admin/Prod
ls -ld Data*
drwxrwxr-x 2 user group    4096 Feb 12  2013 Data Accees Regulatory Policy

So...its there and I'm able to cd into Data Accees Regulatory Policy and see all the files under it. This is odd???

No, its not a typo...I have copy/pasted it exactly from my terminal. But, it is a typo spelling as far as the naming of the directory goes. No digging here needed on the "Accees".

I'm not sure what to do here...script worked 99%, except for the mentioned directory in the questions and another one. The other has underscores in the path name...so there is no embedded spaces there.

Any ideas?
0
AIX25Author Commented:
We have another problem with this script.

The capturing portion does not work. Here is a snippet of the output file of the capture part when running your script.

chmod u=drw,g=xrw,o=sr "."
touch -m -t 201307261407 "."
chmod u=drw,g=xrw,o=xr "./app"
touch -m -t 201307161419 "./app"

Where is the capturing of the user and group ownership? I only see the timestamp and permissions, but do not see the group/user ownership captured.

Also, here is a snippet of the error log when running the script.

chmod: invalid mode: `u=drw,g=xrw,o=sr'
Try `chmod --help' for more information.
chmod: invalid mode: `u=drw,g=xrw,o=xr'
Try `chmod --help' for more information.
chmod: invalid mode: `u=drw,g=xrw,o=xr'
Try `chmod --help' for more information.

The script does not restore anything in the output file. It fails on every line. The capture and restore do not work.

Please advise
0
woolmilkporcCommented:
1) I think the directory in question has a space/spaces at the end. See the below new version which takes this into account.

2) It wasn't quite clear to me that changing owner/group was among the tasks of the script. See the new version. The change is made based on numeric IDs , so missing entries in /etc/passwd or /etc/group will not lead to an error.

3) The "invalid mode" thing comes from different output formats of the various "stat" implementations. I tried to incorporate this.

New version:

#!/bin/sh
find . | while IFS="" read -r file
 do
   if [[ $(uname) = "AIX" ]] ; then
    OPT=""
    P=$(istat "$file" |grep -i prot | \
        awk '{P="u=" substr($2,1,3) ",g=" substr($2,4,3) ",o=" substr($2,7,3);
              gsub("-","",P); print P}')
    T=$(istat "$file" | grep -i modified | \
        awk '{M="0"(match("JanFebMarAprMayJunJulAugSepOctNovDec", $4)+2)/3;
              print $NF substr(M,length(M)-1,2) $5 substr($6,1,2) substr($6,4,2)}')
    O=$(istat "$file" | awk -F"[ (]" '/Owner/ {print $2 ":" $4}')
    elif [[ $(uname) = "Linux" ]] ; then
     OPT="-m"
     P=$(stat -c "%A" "$file" | \
         awk '{P="u=" substr($1,2,3) ",g=" substr($1,5,3) ",o=" substr($1,8,3);
               gsub("-","",P); print P}')
     T=$(stat -c "%y" "$file" | \
         awk -F"-| |:" '{print $1 $2 $3 $4 $5}')
     O=$(stat -c "%u:%g" "$file")
     else echo "$(uname) Unsupported OS, exiting" ; exit
   fi
  echo "chmod" $P "\"$file\""
  echo "chown" $O "\"$file\""
  echo "touch" $OPT "-t" $T  "\"$file\""
 done
0
AIX25Author Commented:
1) Thanks for figuring that one out...regarding the extra space at the end.

2) The user's UID and the group's GID will differ from the source and target areas. I need the script to capture the username and groupname from the source area, than when I run the restore on the target system; restore the username and groupname, and if the user or group is not located in /etc/passwd and /etc/group...set it to the UID or GID. Is that how this portion of the script will work?

3) Thank you for the new version.
0
woolmilkporcCommented:
2) It will work only if username/userid combos are consistent on both systems, so if "AIX25" has ID 209 on the old system then "AIX25" must have ID 209 on the new system.

If this is inconsistent - how should the script be aware what's been going on in your mind when creating new users?

We can revert to using names, but then you will get errors with users/groups not existing on the new system.
0
AIX25Author Commented:
Please see my (2) response above. I need it to capture the user and group name based on the name and not the UID/GID. The UID and GID will differ on the source and target systems. Please factor that in.
0
woolmilkporcCommented:
Aye-aye, SIR!

Did you notice what I wrote about errors with non-existing names?

#!/bin/sh
find . | while IFS="" read -r file
 do
   if [[ $(uname) = "AIX" ]] ; then
    OPT=""
    P=$(istat "$file" |grep -i prot | \
        awk '{P="u=" substr($2,1,3) ",g=" substr($2,4,3) ",o=" substr($2,7,3);
              gsub("-","",P); print P}')
    T=$(istat "$file" | grep -i modified | \
        awk '{M="0"(match("JanFebMarAprMayJunJulAugSepOctNovDec", $4)+2)/3;
              print $NF substr(M,length(M)-1,2) $5 substr($6,1,2) substr($6,4,2)}')
    O=$(istat "$file" | awk -F"[()]" '/Owner/ {print $2 ":" $4}')
    elif [[ $(uname) = "Linux" ]] ; then
     OPT="-m"
     P=$(stat -c "%A" "$file" | \
         awk '{P="u=" substr($1,2,3) ",g=" substr($1,5,3) ",o=" substr($1,8,3);
               gsub("-","",P); print P}')
     T=$(stat -c "%y" "$file" | \
         awk -F"-| |:" '{print $1 $2 $3 $4 $5}')
     O=$(stat -c "%U:%G" "$file")
     else echo "$(uname) Unsupported OS, exiting" ; exit
   fi
  echo "chmod" $P "\"$file\""
  echo "chown" $O "\"$file\""
  echo "touch" $OPT "-t" $T  "\"$file\""
 done
0
AIX25Author Commented:
The O= line?

One more thing...since I already ran this and have been running the chowns and chmods manually because this is time sensitive. Anyway you can provide me a script that will only capture the permissions and datestamp?? Similar to the one above...but without the user and group ownership part. For my next run...I will use your script from start to finish, but since your initial scrip that I ran failed on the chmods. I took like 4 hours to run the whole thing...and I now only need the capture of the permissions and timestamp because of my current situation. Please help with that.
0
woolmilkporcCommented:
>> Anyway you can provide me a script that will only capture the permissions and datestamp?? Similar to the one above...but without the user and group ownership part. <<

The script without ownership processing is what I presented all the time until you came up with this additional thing.
0
AIX25Author Commented:
My apologies...its early morning here 5:30AM

I'm not trying to test your patience. I'm just confused.

My true apologies for having you repeat yourself.

So we are on the same page...the script needs:

1) I do need the user and group ownership captured based off of NAME. 99% of the time the UID and GID on the source and target systems will be different.  But, the user and group names will be the same on the source and target. And if the user name and group name are not located on the target system...I'm ok with it creating the ownership with an orphaned ownership. Please incorporate this into it. I really appreciate your help on this

2) you covered it, with permission capture.

3) you covered it, with timestamp.

Sorry to keep bugging you on this, but my fault for not explaining it correctly to you.
0
woolmilkporcCommented:
The script without ownership processing is already there.

Take the version inside the accepted answer and change

find . | while read -r file

to

find . | while IFS="" read -r file


The last complete version I posted does exactly what you're requesting in (1).

You will get errors for non-existing users/groups. Ownership will stay as it is for those files.
0
AIX25Author Commented:
I used your script "Posted on 2013-09-26 at 03:05:40ID: 39524120". That is the one that that gave me the "chmod invalid errors" and did not factor in the spacing issues I was having. Will changing the line to "find . | while IFS="" read -r file" help with this?

I just want to make sure I use the correct script. Thank you for your assistance on this.
0
woolmilkporcCommented:
Yes, I forgot.

Change

 awk '{P="u=" substr($1,1,3) ",g=" substr($1,4,3) ",o=" substr($1,7,3);

to

 awk '{P="u=" substr($1,2,3) ",g=" substr($1,5,3) ",o=" substr($1,8,3);

in the "Linux" part.

Don't forget the change I mentioned in my previous post:

find . | while read -r file

to

find . | while IFS="" read -r file
0
AIX25Author Commented:
I tested this script below. The timestamp and permissions pieces of the script work great. Its still missing a piece I need for user and group ownership. I need to be able to capture the user and group ownership on the source area, so that I restore it on the target area. The capture needs to be based on user name and group name...not by UID and GID. Please help

#!/bin/sh
find . | while IFS="" read -r file
 do
   if [[ $(uname) = "AIX" ]] ; then
    OPT=""
    P=$(istat "$file" |grep -i prot | \
        awk '{P="u=" substr($2,1,3) ",g=" substr($2,4,3) ",o=" substr($2,7,3);
              gsub("-","",P); print P}')
    T=$(istat "$file" | grep -i modified | \
        awk '{M="0"(match("JanFebMarAprMayJunJulAugSepOctNovDec", $4)+2)/3;
              print $NF substr(M,length(M)-1,2) $5 substr($6,1,2) substr($6,4,2)}')
    elif [[ $(uname) = "Linux" ]] ; then
     OPT="-m"
     P=$(stat -c "%A" "$file" | \
         awk '{P="u=" substr($1,2,3) ",g=" substr($1,5,3) ",o=" substr($1,8,3);
               gsub("-","",P); print P}')
     T=$(stat -c "%y" "$file" | \
         awk -F"-| |:" '{print $1 $2 $3 $4 $5}')
     else echo "$(uname) Unsupported OS, exiting" ; exit
   fi
  echo "chmod" $P "\"$file\""
  echo "touch" $OPT "-t" $T  "\"$file\""
 done
0
woolmilkporcCommented:
#39532640
0
AIX25Author Commented:
I'm running script in #39532640 and capturing 6.5TB of permissions, ownership, and timestamp. It's been over 5 hours and about 75% done. Is there any kind of performance tuning we can do to make the script run faster? I still haven't executed the output to how long restoring part will??
0
AIX25Author Commented:
The capture script only took 2 hours on AIX for 6.TB. But, it took over 7 hours on Linux. Any idea why one would take longer than the other? Any way we can tune the performance of the script?
0
woolmilkporcCommented:
Maybe we could tune the collector script a bit, but we can't tune the restore step. You requested three properties to be changed which requires running three commands against each file, not more and not less.

Difference AIX/Linux?

Could it be that the AIX box has faster processors or I/O? Or maybe AIX's istat is faster than Linux's stat.

There's nothing we can influence, after all.
0
AIX25Author Commented:
I ran the capture script on AIX. Took the output file and executed the restore output file on Linux. But, it I did not work and did not restore any permissions See snippet of errors below.

chown: invalid group: `adm4:mktgrp'
chown: invalid group: `u4872:mktgrp'
chown: invalid group: `u2347:mktgrp'
touch: invalid date format `20120361441'
chown: invalid group: `u2189:mktgrp'
touch: invalid date format `20120361441'

How do you fix the "invalid date format"? Also, how do you the "invalid group" error? Options could be...if the group is not there...set a random GID. Or maybe we can alter the script to just capture user ownership and not group ownership. I can handle the group ownership manually. Any thoughts?
0
AIX25Author Commented:
I ran a test from Linux to Linux. Timestamp worked properly. But it would not restore my output file because of group being unknown. Is there a way to allow the script to skip the group if its not there, but still assign the user ownership? Because of the group being unknown...it won't do either. If not, I'm ok with just capturing the user ownership and running the chgrp manually.  Please see snippet below.

chown: invalid group: `u9283:UNKNOWN'
chown: invalid group: `u2493:UNKNOWN'
chown: invalid group: `u0946:UNKNOWN'
chown: invalid group: `us1273:UNKNOWN'
chown: invalid group: `u0936:UNKNOWN'
0
woolmilkporcCommented:
I already told you that we will run into problems when using user/group names. Numeric IDs would noz have led to those errors.

Anyway, you can change the result file coming from Linux to use some arbitrary numeric group ID, like "9999" instead of "UNKNOWN", or to use some existing groupname like "staff":

sed 's/UNKNOWN/9999/' resultfile > resultfile.new

or

sed 's/UNKNOWN/staff/' resultfile > resultfile.new

By the way, this "UNKNOWN" string comes from the source system. Perhaps you should do some cleanup there on occasion.

For the AIX errors:

I could change the result script by splitting the "chown" line into two lines "chown" and "chgrp" and adding some logic to change user or group to numeric "9999" if the original ones do not exist, and by adding the missing "0" to the touch command where needed.
This would require quite a bit of "awk" coding.

On the other hand, I could change the capture script to generate the correct output.

Which option do you prefer?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Unix OS

From novice to tech pro — start learning today.