unix shell commands  ssh

Posted on 2009-04-24
Last Modified: 2012-05-06
Hi everyone,

I have a number of things I want to do that seem like they should be possible and I was hoping for simple commands like these two which I have found very useful for deleting every file of a certain type from a directory tree or copying a file to every directory in a directory tree.

find . -type d -print0 | xargs -0 -iD cp myfile 'D'
find -type f -name "*.txt" -exec rm {} \;

1)  How can I count how many files of a certain type are in a directory or its subdirectories?
    I keep getting in trouble for using WinSCP to access my filestore on the server.  It hangs when I go in a directory that contains thousands of files to see how many are in there or not.  Using commands like ls are not practical as well unless someone knows how to process the ls output somehow.

Also, on our server queue I can see the queue of jobs using qstat and can see my jobs using

  qstat | grep username

or count how many jobs I have in the queue using

  qstat | grep username | wc -l

because qstat just gives a list of output where each line represents a job and the username is one of the elements in the line.  Another element tells you how long the job is running and another tells you which machine and the job ID and so on.

3)  How can I tell how many jobs are in the queue above me. i.e. which line of output has my username on first?  The number of lines before the first instance of a line with my username on should tell me how many are above me.

4)  How can I see how many of my jobs are running.  This will be lines in the output, with my username on, where one of the values (say the 6th) is something like 0:04:32 rather than just 0.

5) Is there an easy way to kill all my jobs at once if I need to? To kill jobs on my server you need to know the job ID and use the qdel command.  something like

  qdel 1256346.qm01

Now the job ID is an element on a line produced from qstat.  There seems there should be a way to take qstat output, grep it for my username, then apply qdel to all of the job IDs (which are first element on a line of qstat output).

I wil be happy to give clarifiication if I can on any of my questions and say they are worth 100 pts for each answer.

Thanks for your help

Question by:WillFletcher
    LVL 10

    Assisted Solution

    1) the following command shows how many files called *.txt are to be found in /export/petur/tmp and it's subdirectories

     find /export/petur/tmp -type f -name "*.txt"|wc -l
    LVL 58

    Accepted Solution

    find . -type f -name "*.txt" | wc -l

    qstat | tr -s " " | awk 'BEGIN {count = 0;} $3 == "username" {Found = 1;} NR > 2 && Found != 1 {count++} END {print count;}'

    qstat | tr -s " " | awk 'BEGIN {count = 0;} $3 == "username" && $4 == "0" {count++} END {print count;}'

    qstat | grep "username" | cut -d" " -f1 | xargs qdel

    Author Comment

    Hi there,

    Thank you both for your answers!  I have tried your suggestions out and found the following:

    1) Say I have a directory x and that contains 1033 text files and it also contains 567 directories all that contain one text file.  So there are 1600 text files all together.

    If I am inside x then PeturlingEgilsson's answer doesn't work.  But if I am in the directory that contains x then without the forward slash before the directory name it does

       find x -type f -name "*.txt"|wc -l          

    gives 1004.  Also, if I am inside x then the answer from amit_q works too.

       find . -type f -name "*.txt" | wc -l

    also gives 1004. I'll consider it answered by both of you.

    Both of these answers are useful to me but if I could also run a command that would give 567 too (count the text files in all subdirectories of a directory) or 1600 (the count of both) then I could know all the info. I need.  Any ideas?

    2) I realise I didn;t put this in!

    3)  I don;t know if the answer

       qstat | tr -s " " | awk 'BEGIN {count = 0;} $3 == "username" {Found = 1;} NR > 2 && Found != 1 {count++} END {print count;}'

      works properly as I don't *actually* know how many jobs are in front of me right now but after trying it out, and also counting total jobs with "qstat | wc -l" and total jobs of mine "qstat | grep username | wc -l" the answer seems consistent so thankyou!

     In this case total jobs was 57924, my total jobs was 45299 and this answer gave jobs infront of me as 154 which certainly doesn't fall outside the possible range.  I'll consider this one answered by amit_q.  PS.  I have permission to run so many jobs.  The cluster is in a testing phase and I am helping! ;-)     I will confirm tomorrow if lots of my jobs ran overnight (i.e. they were in the queue but literally just about to start after hours of waiting)

    4) I see what this answer is trying to do but I think it is making a mistake somewhere. Maybe I explained it wrong so ask me if I can tell more.

      qstat | tr -s " " | awk 'BEGIN {count = 0;} $3 == "username" && $4 == "0" {count++} END {print count;}'

    gives the same answer as

       qstat | grep username | wc -l

    which can't be right in my situation. i.e. the number of jobs of mine in the queue = the number running.   maybe there is a typo?

    5)  I can't test this one right now as I have jobs that have been queueing for hours that will run soon but I will let you know!

    Any advances on number 1 or 4 for now?

    And thanks again for all the help!


    Author Comment

    Actually it just occurred to me that I think I may have mistook answer 4 to be wrong.  I see it tells me how many ARE NOT running of course because jobs not running all have "0"  and jobs running will all have "x" where x will be a different time string in each case making the count impossible to directly calculate.  I am half asleep as it i late here!

    I will double check tomorrow.

    advances on 1 are still welcome though!
    LVL 58

    Expert Comment

    You don't have to be inside the dir

    find /path/to/WhateverDir -type f -name "*.txt" | wc -l

    for the directories

    find /path/to/WhateverDir -type d | wc -l

    the above would include the /path/to/WhateverDir in the directory count so you may want to subtract one.

    Author Comment

    Thanks for the comment amit_q, but I don't want to count directories but the numbers of files of a certain type in all subdirectories of a directory.

    The reason is that when jobs of mine finish running they output files in thousands of different subdirectories.  I need to count how many e.g. .txt files are in all subdirectories in total.

    So say I have a directory x and that contains 1033 text files and it also contains 567 directories all that contain 1,2,3,... text files.  I don't know how many are in each directory, or even if there are any at all.  But I need either the total count (1033 + however many text files are in the 567 sub directories) or just the count of the how many are in the 567 subdirectories.

    Thanks again!

    LVL 61

    Expert Comment

    Do you want to count files of particular content type?

    find /dir -type f -print0 | xargs -0 file | grep -c "file type name from /etc/magic"

    LVL 58

    Expert Comment

    find /path/to/WhateverDir -type f -name "*.txt" | wc -l

    should count the number of .txt files within /path/to/WhateverDir and any sub-directories within it. What are yoi getting from this? Take out the last wc -l and redirect the output to a file and check that file.

    Author Comment

    Hi Amit_q.

    My mistake.  It does work.  I can only think I somehow tested it in the wrong place before.

    Thank you all for your comments, all questions now answered and I allocated the points.


    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Threat Intelligence Starter Resources

    Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

    I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
    Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
    This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
    Video by: Phil
    This video goes over how to configure and start a jail in FreeBSD.  This video is meant to supplement the article included with this course.

    761 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now