Solved

Linux Script to validate the file exists ?? and to filter the CR from the file.

Posted on 2015-01-21
12
506 Views
Last Modified: 2015-01-21
I get .txt file in my Linux. File is exported to oracle database through some oracle scripts.

I have to validate whether the file exists or not and if the file has any CR or CR LF,
i need to filter from the input .txt file.

I am very new to Linux Shell scripting and having a tough time to getting into it.
0
Comment
Question by:chokka
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
12 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562658
Use the "-e" test to check if the file exists.
Use "cat -v filename" to display the CR characters as "^M" (if any) and use "grep -F" to find them.
Finally, use "tr -d" to delete the unwanted CR charactes ("\r"). I think you want to leave the linefeed characters (LF, "\n") in place, right?

FILE="/path/to/file.txt"
if [[ -e "$FILE" ]]; then
   if cat -v "$FILE" | grep -q -F "^M"; then
       tr -d '\r' < "$FILE" > "${FILE}.new"
   fi
fi

If the file exists and contains CR the the above will create a file "/path/to/file.txt.new".
If you're satisfied with the content of this new file rename it to its previous name, thus overwriting the old file:

FILE="/path/to/file.txt"
mv "${FILE}.new" "$FILE"

Note: I used quotation marks around the filenames to provide for the case that the names might contain spaces.
0
 

Author Comment

by:chokka
ID: 40562679
Thank you @woolmilkporc,

I am so thank ful. I am not pushing you hard.

However, i need to get this syntax in shape to run and make the file to work.

I have attached the Sample Data - TestData.txt ( This is how we get our Input Source File ).

On opening with Notepad++, I am able to see the CR in each line.

I dont need CR or LR. Yes , CR goes for \n

I can remove manually by replacing \n with empty .

I have attached the screenshot of Notepad++
D--LINUX-File-Validation-TESTDATA.TXT
C--Users-ZHQ8HPV-Desktop-TESTDATA-CR.jpg
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562713
CR ("carriage return") is "\r", not "\n"! The latter is LF ("linefeed").

Your file contains Windows-style CR characters at some line ends. The code I gave you will remove these.

You said you're on Linux, so you can use this (shorter) code:

FILE="TESTDATA.txt"
if [[ -e "$FILE" ]]; then
   if grep -q $'\r' "$FILE"; then
      sed -i.bak 's/\r$//' "$FILE"
   fi
fi

The above will create an unchanged backup file TESTDATA.txt.bak and change TESTDATA.txt in place.
0
The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

 

Author Comment

by:chokka
ID: 40562804
I ran the shell script but file is not changed.
Still i can see the CR.

Do i made any mistake in my script

I tried both by giving only file  /home/chokka/file.txt and filename as file.txt.
D--LINUX-File-Validation-TestFileValidat
D--LINUX-File-Validation-TestFilewithCR.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562851
Did you run the script from the directory where the file "TestFilewithCR.TXT" is in?
You must provide the full absolute path (with "/" on the left) if the file is not in your currrent directory.

Are you aware that case matters in Linux? For example ".TXT" is not the same as ".txt"!
0
 

Author Comment

by:chokka
ID: 40562897
Yes, i changed the file path as

/home/chokka/TestFileValidation.sh

script creates another file as

TestFilewithCR.TXT.bak


When i checked both .bak file and original file, i still have the CR

How should i take off the CR ?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562918
Could you please post the output of

od -tcx TestFilewithCR.TXT
0
 

Author Comment

by:chokka
ID: 40563027
Thank you for your help, I have attached the output from running the script -

od -tcx TestFilewithCR.TXT


Please find the attachment in a notepad. It has \r
C--Users-ZHQ8HPV-Desktop-Output.txt
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 40563045
OK, we definitely have CR (\r) in that file.

Now we could try to find out why "grep" or "sed" or both obviously don't support
$'\r' or '\r$', but you could also try my first version.

FILE="/path/to/file.txt"
if [[ -e "$FILE" ]]; then
   if cat -v "$FILE" | grep -q -F "^M"; then
       tr -d '\r' < "$FILE" > "${FILE}.new"
   fi
fi

Please don't forget that the changed file gets a ".new" suffix with this code!

What is your OS? (uname -a; cat /proc/version), what is your shell (echo $0)?
0
 
LVL 48

Expert Comment

by:Tintin
ID: 40563197
Most Linux systems have the dos2unix command installed, so you could simply do

[ -e $FILE ] && dos2unix $FILE
0
 

Author Comment

by:chokka
ID: 40563210
Woolmilkporc,

Comment ID: 40563045 - Works

I am closing this posting. I have to do few more testing !!

I am so thank ful for your efforts !!
0
 

Author Closing Comment

by:chokka
ID: 40563570
Great !! Thanks !!
0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
rDNS on single IP and multiple domains 11 77
Linux MD5 Hash 7 97
umask commands 5 15
bash: ./Facebooker.pl :Permission Denied 9 52
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Fine Tune your automatic Updates for Ubuntu / Debian
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question