WillFletcher
asked on
unix shell commands ssh
Hi everyone,
I have a number of things I want to do that seem like they should be possible and I was hoping for simple commands like these two which I have found very useful for deleting every file of a certain type from a directory tree or copying a file to every directory in a directory tree.
find . -type d -print0 | xargs -0 -iD cp myfile 'D'
find -type f -name "*.txt" -exec rm {} \;
1) How can I count how many files of a certain type are in a directory or its subdirectories?
I keep getting in trouble for using WinSCP to access my filestore on the server. It hangs when I go in a directory that contains thousands of files to see how many are in there or not. Using commands like ls are not practical as well unless someone knows how to process the ls output somehow.
Also, on our server queue I can see the queue of jobs using qstat and can see my jobs using
qstat | grep username
or count how many jobs I have in the queue using
qstat | grep username | wc -l
because qstat just gives a list of output where each line represents a job and the username is one of the elements in the line. Another element tells you how long the job is running and another tells you which machine and the job ID and so on.
3) How can I tell how many jobs are in the queue above me. i.e. which line of output has my username on first? The number of lines before the first instance of a line with my username on should tell me how many are above me.
4) How can I see how many of my jobs are running. This will be lines in the output, with my username on, where one of the values (say the 6th) is something like 0:04:32 rather than just 0.
5) Is there an easy way to kill all my jobs at once if I need to? To kill jobs on my server you need to know the job ID and use the qdel command. something like
qdel 1256346.qm01
Now the job ID is an element on a line produced from qstat. There seems there should be a way to take qstat output, grep it for my username, then apply qdel to all of the job IDs (which are first element on a line of qstat output).
...
I wil be happy to give clarifiication if I can on any of my questions and say they are worth 100 pts for each answer.
Thanks for your help
Will
I have a number of things I want to do that seem like they should be possible and I was hoping for simple commands like these two which I have found very useful for deleting every file of a certain type from a directory tree or copying a file to every directory in a directory tree.
find . -type d -print0 | xargs -0 -iD cp myfile 'D'
find -type f -name "*.txt" -exec rm {} \;
1) How can I count how many files of a certain type are in a directory or its subdirectories?
I keep getting in trouble for using WinSCP to access my filestore on the server. It hangs when I go in a directory that contains thousands of files to see how many are in there or not. Using commands like ls are not practical as well unless someone knows how to process the ls output somehow.
Also, on our server queue I can see the queue of jobs using qstat and can see my jobs using
qstat | grep username
or count how many jobs I have in the queue using
qstat | grep username | wc -l
because qstat just gives a list of output where each line represents a job and the username is one of the elements in the line. Another element tells you how long the job is running and another tells you which machine and the job ID and so on.
3) How can I tell how many jobs are in the queue above me. i.e. which line of output has my username on first? The number of lines before the first instance of a line with my username on should tell me how many are above me.
4) How can I see how many of my jobs are running. This will be lines in the output, with my username on, where one of the values (say the 6th) is something like 0:04:32 rather than just 0.
5) Is there an easy way to kill all my jobs at once if I need to? To kill jobs on my server you need to know the job ID and use the qdel command. something like
qdel 1256346.qm01
Now the job ID is an element on a line produced from qstat. There seems there should be a way to take qstat output, grep it for my username, then apply qdel to all of the job IDs (which are first element on a line of qstat output).
...
I wil be happy to give clarifiication if I can on any of my questions and say they are worth 100 pts for each answer.
Thanks for your help
Will
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Actually it just occurred to me that I think I may have mistook answer 4 to be wrong. I see it tells me how many ARE NOT running of course because jobs not running all have "0" and jobs running will all have "x" where x will be a different time string in each case making the count impossible to directly calculate. I am half asleep as it i late here!
I will double check tomorrow.
advances on 1 are still welcome though!
I will double check tomorrow.
advances on 1 are still welcome though!
You don't have to be inside the dir
find /path/to/WhateverDir -type f -name "*.txt" | wc -l
for the directories
find /path/to/WhateverDir -type d | wc -l
the above would include the /path/to/WhateverDir in the directory count so you may want to subtract one.
find /path/to/WhateverDir -type f -name "*.txt" | wc -l
for the directories
find /path/to/WhateverDir -type d | wc -l
the above would include the /path/to/WhateverDir in the directory count so you may want to subtract one.
ASKER
Thanks for the comment amit_q, but I don't want to count directories but the numbers of files of a certain type in all subdirectories of a directory.
The reason is that when jobs of mine finish running they output files in thousands of different subdirectories. I need to count how many e.g. .txt files are in all subdirectories in total.
So say I have a directory x and that contains 1033 text files and it also contains 567 directories all that contain 1,2,3,... text files. I don't know how many are in each directory, or even if there are any at all. But I need either the total count (1033 + however many text files are in the 567 sub directories) or just the count of the how many are in the 567 subdirectories.
Thanks again!
The reason is that when jobs of mine finish running they output files in thousands of different subdirectories. I need to count how many e.g. .txt files are in all subdirectories in total.
So say I have a directory x and that contains 1033 text files and it also contains 567 directories all that contain 1,2,3,... text files. I don't know how many are in each directory, or even if there are any at all. But I need either the total count (1033 + however many text files are in the 567 sub directories) or just the count of the how many are in the 567 subdirectories.
Thanks again!
Do you want to count files of particular content type?
find /dir -type f -print0 | xargs -0 file | grep -c "file type name from /etc/magic"
find /dir -type f -print0 | xargs -0 file | grep -c "file type name from /etc/magic"
find /path/to/WhateverDir -type f -name "*.txt" | wc -l
should count the number of .txt files within /path/to/WhateverDir and any sub-directories within it. What are yoi getting from this? Take out the last wc -l and redirect the output to a file and check that file.
should count the number of .txt files within /path/to/WhateverDir and any sub-directories within it. What are yoi getting from this? Take out the last wc -l and redirect the output to a file and check that file.
ASKER
Hi Amit_q.
My mistake. It does work. I can only think I somehow tested it in the wrong place before.
Thank you all for your comments, all questions now answered and I allocated the points.
W
My mistake. It does work. I can only think I somehow tested it in the wrong place before.
Thank you all for your comments, all questions now answered and I allocated the points.
W
ASKER
Thank you both for your answers! I have tried your suggestions out and found the following:
1) Say I have a directory x and that contains 1033 text files and it also contains 567 directories all that contain one text file. So there are 1600 text files all together.
If I am inside x then PeturlingEgilsson's answer doesn't work. But if I am in the directory that contains x then without the forward slash before the directory name it does
find x -type f -name "*.txt"|wc -l
gives 1004. Also, if I am inside x then the answer from amit_q works too.
find . -type f -name "*.txt" | wc -l
also gives 1004. I'll consider it answered by both of you.
Both of these answers are useful to me but if I could also run a command that would give 567 too (count the text files in all subdirectories of a directory) or 1600 (the count of both) then I could know all the info. I need. Any ideas?
2) I realise I didn;t put this in!
3) I don;t know if the answer
qstat | tr -s " " | awk 'BEGIN {count = 0;} $3 == "username" {Found = 1;} NR > 2 && Found != 1 {count++} END {print count;}'
works properly as I don't *actually* know how many jobs are in front of me right now but after trying it out, and also counting total jobs with "qstat | wc -l" and total jobs of mine "qstat | grep username | wc -l" the answer seems consistent so thankyou!
In this case total jobs was 57924, my total jobs was 45299 and this answer gave jobs infront of me as 154 which certainly doesn't fall outside the possible range. I'll consider this one answered by amit_q. PS. I have permission to run so many jobs. The cluster is in a testing phase and I am helping! ;-) I will confirm tomorrow if lots of my jobs ran overnight (i.e. they were in the queue but literally just about to start after hours of waiting)
4) I see what this answer is trying to do but I think it is making a mistake somewhere. Maybe I explained it wrong so ask me if I can tell more.
qstat | tr -s " " | awk 'BEGIN {count = 0;} $3 == "username" && $4 == "0" {count++} END {print count;}'
gives the same answer as
qstat | grep username | wc -l
which can't be right in my situation. i.e. the number of jobs of mine in the queue = the number running. maybe there is a typo?
5) I can't test this one right now as I have jobs that have been queueing for hours that will run soon but I will let you know!
Any advances on number 1 or 4 for now?
And thanks again for all the help!
Will