Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Linux Script to validate the file exists ?? and to filter the CR from the file.

Posted on 2015-01-21
12
Medium Priority
?
520 Views
Last Modified: 2015-01-21
I get .txt file in my Linux. File is exported to oracle database through some oracle scripts.

I have to validate whether the file exists or not and if the file has any CR or CR LF,
i need to filter from the input .txt file.

I am very new to Linux Shell scripting and having a tough time to getting into it.
0
Comment
Question by:chokka
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
12 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562658
Use the "-e" test to check if the file exists.
Use "cat -v filename" to display the CR characters as "^M" (if any) and use "grep -F" to find them.
Finally, use "tr -d" to delete the unwanted CR charactes ("\r"). I think you want to leave the linefeed characters (LF, "\n") in place, right?

FILE="/path/to/file.txt"
if [[ -e "$FILE" ]]; then
   if cat -v "$FILE" | grep -q -F "^M"; then
       tr -d '\r' < "$FILE" > "${FILE}.new"
   fi
fi

If the file exists and contains CR the the above will create a file "/path/to/file.txt.new".
If you're satisfied with the content of this new file rename it to its previous name, thus overwriting the old file:

FILE="/path/to/file.txt"
mv "${FILE}.new" "$FILE"

Note: I used quotation marks around the filenames to provide for the case that the names might contain spaces.
0
 

Author Comment

by:chokka
ID: 40562679
Thank you @woolmilkporc,

I am so thank ful. I am not pushing you hard.

However, i need to get this syntax in shape to run and make the file to work.

I have attached the Sample Data - TestData.txt ( This is how we get our Input Source File ).

On opening with Notepad++, I am able to see the CR in each line.

I dont need CR or LR. Yes , CR goes for \n

I can remove manually by replacing \n with empty .

I have attached the screenshot of Notepad++
D--LINUX-File-Validation-TESTDATA.TXT
C--Users-ZHQ8HPV-Desktop-TESTDATA-CR.jpg
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562713
CR ("carriage return") is "\r", not "\n"! The latter is LF ("linefeed").

Your file contains Windows-style CR characters at some line ends. The code I gave you will remove these.

You said you're on Linux, so you can use this (shorter) code:

FILE="TESTDATA.txt"
if [[ -e "$FILE" ]]; then
   if grep -q $'\r' "$FILE"; then
      sed -i.bak 's/\r$//' "$FILE"
   fi
fi

The above will create an unchanged backup file TESTDATA.txt.bak and change TESTDATA.txt in place.
0
Will your db performance match your db growth?

In Percona’s white paper “Performance at Scale: Keeping Your Database on Its Toes,” we take a high-level approach to what you need to think about when planning for database scalability.

 

Author Comment

by:chokka
ID: 40562804
I ran the shell script but file is not changed.
Still i can see the CR.

Do i made any mistake in my script

I tried both by giving only file  /home/chokka/file.txt and filename as file.txt.
D--LINUX-File-Validation-TestFileValidat
D--LINUX-File-Validation-TestFilewithCR.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562851
Did you run the script from the directory where the file "TestFilewithCR.TXT" is in?
You must provide the full absolute path (with "/" on the left) if the file is not in your currrent directory.

Are you aware that case matters in Linux? For example ".TXT" is not the same as ".txt"!
0
 

Author Comment

by:chokka
ID: 40562897
Yes, i changed the file path as

/home/chokka/TestFileValidation.sh

script creates another file as

TestFilewithCR.TXT.bak


When i checked both .bak file and original file, i still have the CR

How should i take off the CR ?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40562918
Could you please post the output of

od -tcx TestFilewithCR.TXT
0
 

Author Comment

by:chokka
ID: 40563027
Thank you for your help, I have attached the output from running the script -

od -tcx TestFilewithCR.TXT


Please find the attachment in a notepad. It has \r
C--Users-ZHQ8HPV-Desktop-Output.txt
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 2000 total points
ID: 40563045
OK, we definitely have CR (\r) in that file.

Now we could try to find out why "grep" or "sed" or both obviously don't support
$'\r' or '\r$', but you could also try my first version.

FILE="/path/to/file.txt"
if [[ -e "$FILE" ]]; then
   if cat -v "$FILE" | grep -q -F "^M"; then
       tr -d '\r' < "$FILE" > "${FILE}.new"
   fi
fi

Please don't forget that the changed file gets a ".new" suffix with this code!

What is your OS? (uname -a; cat /proc/version), what is your shell (echo $0)?
0
 
LVL 48

Expert Comment

by:Tintin
ID: 40563197
Most Linux systems have the dos2unix command installed, so you could simply do

[ -e $FILE ] && dos2unix $FILE
0
 

Author Comment

by:chokka
ID: 40563210
Woolmilkporc,

Comment ID: 40563045 - Works

I am closing this posting. I have to do few more testing !!

I am so thank ful for your efforts !!
0
 

Author Closing Comment

by:chokka
ID: 40563570
Great !! Thanks !!
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses

704 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question