Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17


Currupt text file

Posted on 1999-07-28
Medium Priority
Last Modified: 2010-04-21
I frequently have students coming to me with currupt text data files which I assume that have backed-up incorrectly.  On our unix system, most people just ftp their datasets (which consist of binary and text data) to their pc/mac and use whatever backup-resources they have there.  When students try to reload their data to the workstation, the software says that it is currupt.  Looking at the data, all the variables are on one or 2 lines instead of each variable being on its own line.  Loading the text file into "joe" indicates that there is a <cr> in between each variable.  I'm assuming this is a carriage return since a search and replace for <cr> doesn't work.
  So, is there any way I could use sed or some other nifty tool to add a linefeed after each carriage return to make this data usable again?  I know how to use sed, however I don't know how to replace/add special characters.
Question by:BoB
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
  • +4

Expert Comment

ID: 2011710
They need to transfer the files in binary mode (typing bi before doing the transfer)

Author Comment

ID: 2011711
I know that, and they know that (theoretically), but they don't do it and I need to figure out a way to fix their mistakes. :)

Expert Comment

ID: 2011712
Ah :) well to replace text with sed you could do this:

sed -e "s/<cr>//g" input_file > output_file

Good Luck

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!


Expert Comment

ID: 2011713
I do not have access to Unix machine right now to test your thing, but I think "tr" command is the best option for working with special characters.
LVL 19

Expert Comment

ID: 2011714
I think you'll need something like.....

sed -e 's/(CTRL)V (CTRL)M//g' > outputfile

The <cr> probably looks something like ^M in the unix file, CTRL V allow you to insert control chars so you'll have to try that first.


Expert Comment

ID: 2011715
Yes, they should use binary mode to transfer them.
Anyway, on your UNIX system you shoudl find two utilities:
to correct carriage return and linefeed between DOS and UNIX
LVL 19

Expert Comment

ID: 2011716
Note that dos2unix and unix2dos may not exist if nfs is not installed...

Author Comment

ID: 2011717
I don't have unix2dos or dos2unix on my computer.  The only solution I could get to work was from ginde and I did it like this:

tr "\r" "\n" < > file.out

ginde, please answer the question so you can get the credits.
LVL 51

Accepted Solution

ahoffmann earned 200 total points
ID: 2011718
the problem is described in previous comments: use bin mode in ftp, if the text files are still in wrong format use one of folloing methods:
   1. depending on your UNIX you have
          dos2unix - unix2dos, dos2ux - ux2dos, to_dos - to_unix, duxonv, dosread - doswrite
   2. use tr or sed or perl or whatever to remove ^M (be prepared for some shell dragons, see prev. comments)
   3. use vi
    a) :%s/^M//g
    b) :g/^M/s///g
       (where ^M is  CTRL-V followed by Return
   4. use vim (a vi clone)
       :unset textmode
      you may get vim (or any other vi clone from
   5. use any other editor which can handle it

Author Comment

ID: 2011719
The person who answered didn't actually answer my question.  Ginde should actually get credit for doing so.  Unless there is no argument, however, I'll just close the question as it stands.

Expert Comment

ID: 6821554
This question was awarded, but never cleared due to the JSP-500 errors of that time.  It was "stuck" against userID -1 versus the intended expert whom you awarded.  This corrects that and the expert will now receive these points, all verified.

Please click on your Member Profile, select "View Question History" to navigate through any open or locked questions you may have to update and finalize them.
Moderator @ Experts Exchange

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question