Need help modifying my script

I need to write a script that loops throught all the directories that contain $var in the path and get all the files with .txt extension and calculate the total size


for example if $var="sample" and directories below contain

/u01/test/samples

test1.txt 1GB
test2.txt 1GB

/u04/data/sample

sample2.txt 2Gb

then the script should add up file sizes in those directories and assing them to a variable

So far I came up with something like this:

$dir=/u01/test/samples/*.txt
du -g $dir | total_size=`awk '{c+=$1};END { print c }'`

which works great but only returns total file size for one specified directory. Can someone help fix my code so that it returns total size fo all directories containing a specific variable in the path?
LVL 35
YZlatAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ozoCommented:
wc -c `find . -path '*/sample/*.txt'`
0
YZlatAuthor Commented:
ozo, but it will not loop through all fielsystems, will it?

The thing is I have a number of different filesystems: /u01, /u02, /u03, /u04 and many more. I need to loop through them because almost every one of them have word "sample" in it
0
YZlatAuthor Commented:
i get an error

find: 0652-017 -path is not a valid option.

Also wc command counts the characters not file size, I know it is basically the same thing but I feel like it is not the right way to go.

The bottom line this code does not work either way
0
Build an E-Commerce Site with Angular 5

Learn how to build an E-Commerce site with Angular 5, a JavaScript framework used by developers to build web, desktop, and mobile applications.

simon3270Commented:
A command to generate the result could be:
find /u0? -type f | grep "/${var}/.*\.txt$" | xargs ls -s | awk '{c+=$1};END { print c }'

Open in new window

To get this into a variable, use
totsize=$(find /u0? -type f | grep "/${var}/.*\.txt$" | xargs ls -s | awk '{c+=$1};END { print c }')

Open in new window


(editted to add the "-type f", in case a directory name ended with ".txt")
0
skullnobrainsCommented:
du -ch `find / -type d -name sample`

you should try and toy with the find command to make it faster. maybe something like this

du -ch `find /u* -maxdepth 4 -type d -name sample`

i guess you can figure that part out easily if you know where to expect the files

if locate is available and reasonably up to date, you may want to replace find with "locate sample"
0
YZlatAuthor Commented:
simon3270, your code produces an error:

find: 0652-019 The status on /u0? is not valid.
0
simon3270Commented:
That's /u zero questionmark (or whatever matches the base directories you want to search)
0
YZlatAuthor Commented:
yes, I tried /u zero question mark and that's what gave me the error. I also tried /u? but still same error
0
simon3270Commented:
I think you get that error either if there is a problem with a disk (are they NFS mounts with a dead server?  Or actual disk failures?) or simply if the path specified doesn't exist - you don't want to try text until it works - use a pattern which finds the actual disks (assuming that /u01 or /uo1 for example is the mount point for a mounted disk).  I can't see your system, so unless you copy and paste the output, for example, or "ls -p /" or "mount", I can't see what your directories are called.

If find can't get past the lowest level directory, it won't be able to check for "sample" directories - you need to fix that problem first.

I only used /u0? because you implied that you needed to check all such directories (and I didn't expect them to be /uo?).  if you only need to check some of them, first check that the ones you want work with "find" (test them one at a time with, for example, "find /u01 | head"), then just list them on the command line, like:

    find /u01 /disk2 /cc0703 /mdisk04 -type f |

(and then the rest of the line is the same as it was).
0
skullnobrainsCommented:
here is an example on my machine

$ du -ch `find /v* -maxdepth 2 -type d -name \*sa\*` | tail
4,0K	/var/log/boot-sav/log/2013-06-21__14h13boot-repair20/sdb1
4,0K	/var/log/boot-sav/log/2013-06-21__14h13boot-repair20/sda5
1,1M	/var/log/boot-sav/log/2013-06-21__14h13boot-repair20/sda
4,0K	/var/log/boot-sav/log/2013-06-21__14h13boot-repair20/sda1
4,0K	/var/log/boot-sav/log/2013-06-21__14h13boot-repair20/sda2
1,2M	/var/log/boot-sav/log/2013-06-21__14h13boot-repair20
7,1M	/var/log/boot-sav/log
7,1M	/var/log/boot-sav
4,0K	/var/log/samba
7,2M	total

Open in new window


try something like this

du -ch `find /u[0-9] -maxdepth 4 -type d -name sample` | cut -d \  -f 1

Open in new window

0
YZlatAuthor Commented:
simon, directories are called

/u01
/u02
/u10
/u40
/u80
0
YZlatAuthor Commented:
skullnobrains, what does this part do?

cut -d \  -f 1
0
simon3270Commented:
The command becomes:
find /u01 /u02 /u10 /u40 /u80 -type f | grep "/${var}/.*\.txt$" | xargs ls -s | awk '{c+=$1};END { print c }'

Open in new window

If that returns the same "find" error, check that each disk can be read by running:
    ls /u01 /u02 /u10 /u40 /u80 >/dev/null
and letting us know if there are any errors reported
0
skullnobrainsCommented:
skullnobrains, what does this part do?

cut -d \  -f 1

cut the string using ' ' (space) as the delimiter and print field number 1 (try "man du")

this parses du's output so you only get the size and not the whole line. i printed a greater part of the output for clarity.

- start here : http://www.experts-exchange.com/Programming/Languages/Scripting/Shell/Q_28230945.html#a39478341
- you can add "| tail -n 1" to retrieve the last line
- and add "|cut -d ' ' -f1' to only get the first field which contains the total size

-->
du -ch `find /u[0-9] -maxdepth 4 -type d -name sample` | tail -n1 | cut -d \  -f 1

Open in new window


it would be MUCH easier to help you out if you tried the provided code and posted some feedback. does it provide the information you wanted ? if not what did you get and how different is that from your expectations ?
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
simon3270Commented:
I thought you only wanted to add up the size of the *.txt files, not of the entire contents of the directories called "sample".
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.