Solved

Linux Script to validate the file exists ?? and to filter the CR from the file.

Posted on 2015-01-21
12
486 Views
Last Modified: 2015-01-21
I get .txt file in my Linux. File is exported to oracle database through some oracle scripts.

I have to validate whether the file exists or not and if the file has any CR or CR LF,
i need to filter from the input .txt file.

I am very new to Linux Shell scripting and having a tough time to getting into it.
0
Comment
Question by:chokka
  • 6
  • 5
12 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
Comment Utility
Use the "-e" test to check if the file exists.
Use "cat -v filename" to display the CR characters as "^M" (if any) and use "grep -F" to find them.
Finally, use "tr -d" to delete the unwanted CR charactes ("\r"). I think you want to leave the linefeed characters (LF, "\n") in place, right?

FILE="/path/to/file.txt"
if [[ -e "$FILE" ]]; then
   if cat -v "$FILE" | grep -q -F "^M"; then
       tr -d '\r' < "$FILE" > "${FILE}.new"
   fi
fi

If the file exists and contains CR the the above will create a file "/path/to/file.txt.new".
If you're satisfied with the content of this new file rename it to its previous name, thus overwriting the old file:

FILE="/path/to/file.txt"
mv "${FILE}.new" "$FILE"

Note: I used quotation marks around the filenames to provide for the case that the names might contain spaces.
0
 

Author Comment

by:chokka
Comment Utility
Thank you @woolmilkporc,

I am so thank ful. I am not pushing you hard.

However, i need to get this syntax in shape to run and make the file to work.

I have attached the Sample Data - TestData.txt ( This is how we get our Input Source File ).

On opening with Notepad++, I am able to see the CR in each line.

I dont need CR or LR. Yes , CR goes for \n

I can remove manually by replacing \n with empty .

I have attached the screenshot of Notepad++
D--LINUX-File-Validation-TESTDATA.TXT
C--Users-ZHQ8HPV-Desktop-TESTDATA-CR.jpg
0
 
LVL 68

Expert Comment

by:woolmilkporc
Comment Utility
CR ("carriage return") is "\r", not "\n"! The latter is LF ("linefeed").

Your file contains Windows-style CR characters at some line ends. The code I gave you will remove these.

You said you're on Linux, so you can use this (shorter) code:

FILE="TESTDATA.txt"
if [[ -e "$FILE" ]]; then
   if grep -q $'\r' "$FILE"; then
      sed -i.bak 's/\r$//' "$FILE"
   fi
fi

The above will create an unchanged backup file TESTDATA.txt.bak and change TESTDATA.txt in place.
0
 

Author Comment

by:chokka
Comment Utility
I ran the shell script but file is not changed.
Still i can see the CR.

Do i made any mistake in my script

I tried both by giving only file  /home/chokka/file.txt and filename as file.txt.
D--LINUX-File-Validation-TestFileValidat
D--LINUX-File-Validation-TestFilewithCR.
0
 
LVL 68

Expert Comment

by:woolmilkporc
Comment Utility
Did you run the script from the directory where the file "TestFilewithCR.TXT" is in?
You must provide the full absolute path (with "/" on the left) if the file is not in your currrent directory.

Are you aware that case matters in Linux? For example ".TXT" is not the same as ".txt"!
0
 

Author Comment

by:chokka
Comment Utility
Yes, i changed the file path as

/home/chokka/TestFileValidation.sh

script creates another file as

TestFilewithCR.TXT.bak


When i checked both .bak file and original file, i still have the CR

How should i take off the CR ?
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 68

Expert Comment

by:woolmilkporc
Comment Utility
Could you please post the output of

od -tcx TestFilewithCR.TXT
0
 

Author Comment

by:chokka
Comment Utility
Thank you for your help, I have attached the output from running the script -

od -tcx TestFilewithCR.TXT


Please find the attachment in a notepad. It has \r
C--Users-ZHQ8HPV-Desktop-Output.txt
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
Comment Utility
OK, we definitely have CR (\r) in that file.

Now we could try to find out why "grep" or "sed" or both obviously don't support
$'\r' or '\r$', but you could also try my first version.

FILE="/path/to/file.txt"
if [[ -e "$FILE" ]]; then
   if cat -v "$FILE" | grep -q -F "^M"; then
       tr -d '\r' < "$FILE" > "${FILE}.new"
   fi
fi

Please don't forget that the changed file gets a ".new" suffix with this code!

What is your OS? (uname -a; cat /proc/version), what is your shell (echo $0)?
0
 
LVL 48

Expert Comment

by:Tintin
Comment Utility
Most Linux systems have the dos2unix command installed, so you could simply do

[ -e $FILE ] && dos2unix $FILE
0
 

Author Comment

by:chokka
Comment Utility
Woolmilkporc,

Comment ID: 40563045 - Works

I am closing this posting. I have to do few more testing !!

I am so thank ful for your efforts !!
0
 

Author Closing Comment

by:chokka
Comment Utility
Great !! Thanks !!
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

Suggested Solutions

This article will explain how to establish a SSH connection to Ubuntu through the firewall and using a different port other then 22. I have set up a Ubuntu virtual machine in Virtualbox and I am running a Windows 7 workstation. From the Ubuntu vi…
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now