shell script urgent help

I am trying to find checksum values of files and insert them in oracle database ...  following script is having issue reading from files.txt file... I am getting error "no file or directory" - However, files do exist..... I think it has to do with "$LINE" variable... Does anybody has any suggestion?

'files.txt'
"/opt/app/file1.txt"
"/opt/app/file2.txt"

Open in new window


FILES='files.txt'
cat $FILES |
while read LINE
   do
echo $LINE
     checksum=`sha256sum "$LINE"|awk '{print $1}'`
echo $checksum
     path=`sha256sum "$LINE"|awk '{print $2}'`

         sqlplus -s <DB details> <<EOF
         insert into checksum (path,checksum_value) values ('$path', '$checksum');
         commit;
         EXIT;
EOF
done

Open in new window

LVL 5
CalmSoulAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ozoCommented:
If the " are not part of the actual file name, and you can't remove them from files.txt, it may be better to use  ${LINE//\"} in place of $LINE
CalmSoulAuthor Commented:
Its not working basically files.txt has "path" but the command I want to execute is

sha256sum "path"

Open in new window

arnoldCommented:
Instead of running the command twice, sha256sum, run it once
Data=$(sha256sum $LINE)
set $DATA
$1 will have the path information
$2 will have the cksum.


I do not see where you set the oracle related also another issue I see you seem to be processing one at a time, instead of possibly.

Can you run the script and output the inser line to screen to make sure the insert command is correct.

You do not test the exit code from sqlplus to see whether there was an issue.
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

CalmSoulAuthor Commented:
here is the output

- don't worry about ORA stuff its working ...
- test1.txt and test2.txt files exist

/usr/bin/sha256sum: "/opt/app/xxx/test/test1.txt": No such file or directory
/usr/bin/sha256sum: "/opt/app/xxx/test/test1.txt": No such file or directory

1 row created.


Commit complete.

/usr/bin/sha256sum: "/opt/app/xxx/test/text2.txt": No such file or directory
/usr/bin/sha256sum: "/opt/app/xxx/test/text2.txt": No such file or directory

1 row created.


Commit complete.

Open in new window

ozoCommented:
Did you change $LINE to ${LINE//\"} ?
Which shell are you running?
CalmSoulAuthor Commented:
is there a way to check which shell i have ?
arnoldCommented:
If your content is in quotes, you can skip the quoting within the sha256 line.
You are double quoting.
i.e. the data in the text file is quoted, and then your command assignment puts quotes around the $LINE

You should always test the output status of commands you run to make sure they are successful.

What shell are you using for the script, sh, bash, csh,ksh.
you can check the $? for sh, bash
csh I think it is $status
if the value is 0 the command executed without an issue, if it is anything else, there was an error.
The above suggestion will eliminate the inserts in the above case.

Try the following script, ,incorporating ozo's suggestion to avoid issue when the line in the text does not include quotes but does include spaces.

#!/bin/sh
FILES='files.txt'
cat $FILES |
while read LINE
   do
echo $LINE
LINE=${LINE//\"}
if [ -e "$LINE"]; then   # test if the file exists
     DATA=`sha256sum "$LINE"`
if [ $? -eq 0 ]; then   #test if the prior command run was successful
set ${DATA} #do not place this variable in quotes or you will always get a single value the only issue this will run into is if there is a space in the path.
checksum=$1
file_path=$LINE
echo $checksum
 #    path=`sha256sum "$LINE"|awk '{print $2}'`

         sqlplus -s <DB details> <<EOF
         insert into checksum (path,checksum_value) values ('$file_path', '$checksum');
         commit;
         EXIT;
EOF
fi
fi
done

Open in new window

arnoldCommented:
echo $0 will return the shell.
as the example, the first line of a script
#!/bin/bash specifies the shell.
CalmSoulAuthor Commented:
@arnold:

I am getting following error

./script.sh: line 9: syntax error near unexpected token `then'
./script.sh: line 9: `     DATA=`sha256sum "$LINE"`if [ $? -gt 0 ]; then '
ozoCommented:
if [ $? -gt 0 ]; then
should be on a different line from
DATA=`sha256sum "$LINE"`
arnoldCommented:
Sorry, the two commands should be on separate lines. Corrected it in the post, but guess not fast enough.
note the change in the check from greater than to equal.
DATA=`sha256sum "$LINE"`
if [ $? -eq 0 ]; then  #test command exit status status
CalmSoulAuthor Commented:
adding this line

DATA=`sha256sum "$LINE"`
if [ $? -eq 0 ];

Open in new window


Now getting out put:

"/opt/app/xxx/test/test1.txt"
./script.sh: line 8: [: missing `]'
"/opt/app/xxx/test/text2.txt"
./script.sh: line 8: [: missing `]'

Open in new window

arnoldCommented:
Seems to be missing small things:
Changing shell to bash
added the semi-colon to after the running of the set command.

#!/bin/bash
FILES='files.txt'
cat $FILES |
while read LINE
   do
echo $LINE
LINE=${LINE//\"}
if [ -e "$LINE" ]; then   # test if the file exists
     DATA=`sha256sum "$LINE"`
if [ $? -eq 0 ]; then   #test if the prior command run was successful
set ${DATA} ; #do not place this variable in quotes or you will always get a single value the only issue this will run into is if there is a space in the path.
checksum=$1
file_path=$LINE
echo $checksum
 #    path=`sha256sum "$LINE"|awk '{print $2}'`

         sqlplus -s <DB details> <<EOF
         insert into checksum (path,checksum_value) values ('$file_path', '$checksum');
         commit;
         EXIT;
EOF
else 
     echo "The command sha256sum encountered an error with $LINE"
fi
else
     echo "The file $LINE does not exist"
fi
done

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ozoCommented:
if [ -e "$LINE"]; then
should be
if [ -e "$LINE" ]; then
CalmSoulAuthor Commented:
ozo:

what do you mean ?
if [ -e "$LINE"]; then 
should be
if [ -e "$LINE" ]; then

Open in new window

arnoldCommented:
The test of the existence of the file was missing a space between the "$LINE" and the closure of the test.
This is why the line 8 reference was output .  adding the space matches the opening and closing of the test condition.

I believe the prior posted complete code and should be without errors
ozoCommented:
if [ -e "$LINE"]; then  from http:#a40751935 is missing a space, which may have caused the error in http:#a40751958 
it should be
if [ -e "$LINE" ]; then  as in http:#a40751965
CalmSoulAuthor Commented:
still not getting the checksum values, is this because I am going this on mount NAS Point?
arnoldCommented:
What is the output that you are getting when running the script?
ozoCommented:
Can you show us a command you can give directly, not in a shell script, which does get the checksum values, and show us exactly what that command returns?
CalmSoulAuthor Commented:
script output:
"/xxx/004853226__0001__082113c.bus"
 does not existrd-xxx/004853226__0001__082113c.bus

Open in new window



as single command line:

>sha256sum "/xxx/004853226__0001__082113c.bus"
78e48c8239557b21be4c7b9b247cd311dffb8449a0b6228929d568a3a84024eb  /xxx/004853226__0001__082113c.bus

Open in new window

arnoldCommented:
The error tells that the test to check whether the file exists is coming back as not existing.

realize what I ask, but could you post the script as you have it?
CalmSoulAuthor Commented:
here is the script:

>vi script.sh
#!/bin/bash
FILES='files.txt'
cat $FILES |
while read LINE
   do
echo $LINE
LINE=${LINE//\"}
if [ -e "$LINE" ]; then   # test if the file exists
     DATA=`sha256sum "$LINE"`
if [ $? -eq 0 ]; then   #test if the prior command run was successful
set ${DATA} ; #do not place this variable in quotes or you will always get a single value the only issue this will run into is if there is a space in the path.
checksum=$1
file_path=$LINE
echo $checksum
 #    path=`sha256sum "$LINE"|awk '{print $2}'`

         sqlplus -s 'xxx' <<EOF
         insert into CHECKSUM1 (path,checksum_value) values ('$file_path', '$checksum');
         commit;
         EXIT;
EOF
else
     echo "The command sha256sum encountered an error with $LINE"
fi
else
     echo "The file $LINE does not exist"
fi
done

Open in new window

ozoCommented:
the "rd-" in the output seems strange to me.
Is it possible that there is a \r at the end of the lines in files.txt?
CalmSoulAuthor Commented:
script is chopping 7 chars from the path
arnoldCommented:
I think it has some control characters that lead to cursor movement
cat -v files.txt | more
Are there ^M and similar funny characters ^@ ^[ etc.?
CalmSoulAuthor Commented:
yes filename has funny charaters ...
CalmSoulAuthor Commented:
^M
is in the end of each line
ozoCommented:
^M characters can be removed from $LINE the same way that " characters can, although the command can be tricky to cut and paste into a [code] section
arnoldCommented:
Those are being interpreted by the shell and that leads to the test for the existence of the file to fail.

try the following

sha256sum `head -1 files.txt`
what do you get? do you get an error from sha256sum or do you get the checksum??

This presumes that all entries have funny characters.

What generates this list of files?
The funny characters need to be stripped from the file for the script to output and do what you expect.
CalmSoulAuthor Commented:
can you paste command here ? http://pastebin.com/
CalmSoulAuthor Commented:
is there a way to remove ^M? I am generating files from toad for oracle
arnoldCommented:
What are you using to edit the script, vi nano?

^M is \r
to actually get the ^M in the code, you would have to use ctrl+v [enter].
in the below replace <> with the key combo reflected within
LINE=${LINE//\<Control+V [enter]>}

Open in new window

The result should be
LINE=${LINE//\^M}

Open in new window

Where ^M is a single character within the editor

There are dos2unix files.txt what will strip out the ^M characters from the file.
sed -i.bak -e 's/\r//g' files.txt
the above will do a replace with a files.txt.bak as the backup.
arnoldCommented:
Add the sed -i.bak -e 's/\r//g' files.txt in the script on line 2 pushing the FILES= one line down.
ozoCommented:
Might as well do sed -i.bak -e 's/["\r]//g' while you're at it, then you don't need the ${LINE//\"}
Or use ${LINE//[\"^M]} (edited as arnold described), and then you don't need the sed.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.