Solved

Linux Script to validate the file exists ?? and to filter the CR from the file.

Posted on 2015-01-21
12
514 Views
Last Modified: 2015-01-21
I get .txt file in my Linux. File is exported to oracle database through some oracle scripts.

I have to validate whether the file exists or not and if the file has any CR or CR LF,
i need to filter from the input .txt file.

I am very new to Linux Shell scripting and having a tough time to getting into it.
0
Comment
Question by:chokka
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
12 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562658
Use the "-e" test to check if the file exists.
Use "cat -v filename" to display the CR characters as "^M" (if any) and use "grep -F" to find them.
Finally, use "tr -d" to delete the unwanted CR charactes ("\r"). I think you want to leave the linefeed characters (LF, "\n") in place, right?

FILE="/path/to/file.txt"
if [[ -e "$FILE" ]]; then
   if cat -v "$FILE" | grep -q -F "^M"; then
       tr -d '\r' < "$FILE" > "${FILE}.new"
   fi
fi

If the file exists and contains CR the the above will create a file "/path/to/file.txt.new".
If you're satisfied with the content of this new file rename it to its previous name, thus overwriting the old file:

FILE="/path/to/file.txt"
mv "${FILE}.new" "$FILE"

Note: I used quotation marks around the filenames to provide for the case that the names might contain spaces.
0
 

Author Comment

by:chokka
ID: 40562679
Thank you @woolmilkporc,

I am so thank ful. I am not pushing you hard.

However, i need to get this syntax in shape to run and make the file to work.

I have attached the Sample Data - TestData.txt ( This is how we get our Input Source File ).

On opening with Notepad++, I am able to see the CR in each line.

I dont need CR or LR. Yes , CR goes for \n

I can remove manually by replacing \n with empty .

I have attached the screenshot of Notepad++
D--LINUX-File-Validation-TESTDATA.TXT
C--Users-ZHQ8HPV-Desktop-TESTDATA-CR.jpg
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562713
CR ("carriage return") is "\r", not "\n"! The latter is LF ("linefeed").

Your file contains Windows-style CR characters at some line ends. The code I gave you will remove these.

You said you're on Linux, so you can use this (shorter) code:

FILE="TESTDATA.txt"
if [[ -e "$FILE" ]]; then
   if grep -q $'\r' "$FILE"; then
      sed -i.bak 's/\r$//' "$FILE"
   fi
fi

The above will create an unchanged backup file TESTDATA.txt.bak and change TESTDATA.txt in place.
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 

Author Comment

by:chokka
ID: 40562804
I ran the shell script but file is not changed.
Still i can see the CR.

Do i made any mistake in my script

I tried both by giving only file  /home/chokka/file.txt and filename as file.txt.
D--LINUX-File-Validation-TestFileValidat
D--LINUX-File-Validation-TestFilewithCR.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562851
Did you run the script from the directory where the file "TestFilewithCR.TXT" is in?
You must provide the full absolute path (with "/" on the left) if the file is not in your currrent directory.

Are you aware that case matters in Linux? For example ".TXT" is not the same as ".txt"!
0
 

Author Comment

by:chokka
ID: 40562897
Yes, i changed the file path as

/home/chokka/TestFileValidation.sh

script creates another file as

TestFilewithCR.TXT.bak


When i checked both .bak file and original file, i still have the CR

How should i take off the CR ?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562918
Could you please post the output of

od -tcx TestFilewithCR.TXT
0
 

Author Comment

by:chokka
ID: 40563027
Thank you for your help, I have attached the output from running the script -

od -tcx TestFilewithCR.TXT


Please find the attachment in a notepad. It has \r
C--Users-ZHQ8HPV-Desktop-Output.txt
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 40563045
OK, we definitely have CR (\r) in that file.

Now we could try to find out why "grep" or "sed" or both obviously don't support
$'\r' or '\r$', but you could also try my first version.

FILE="/path/to/file.txt"
if [[ -e "$FILE" ]]; then
   if cat -v "$FILE" | grep -q -F "^M"; then
       tr -d '\r' < "$FILE" > "${FILE}.new"
   fi
fi

Please don't forget that the changed file gets a ".new" suffix with this code!

What is your OS? (uname -a; cat /proc/version), what is your shell (echo $0)?
0
 
LVL 48

Expert Comment

by:Tintin
ID: 40563197
Most Linux systems have the dos2unix command installed, so you could simply do

[ -e $FILE ] && dos2unix $FILE
0
 

Author Comment

by:chokka
ID: 40563210
Woolmilkporc,

Comment ID: 40563045 - Works

I am closing this posting. I have to do few more testing !!

I am so thank ful for your efforts !!
0
 

Author Closing Comment

by:chokka
ID: 40563570
Great !! Thanks !!
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses

628 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question