Solved

Trying to find non-printable characters in a file

Posted on 2013-11-25
17
2,036 Views
Last Modified: 2013-11-27
I would like to find all non-printable characters in a file. Questions:

1) Can somebody give me an example script?
2) If I want to replace all of them with a blank space or just nothing, how would I do it?

Thanks.
0
Comment
Question by:soccerplayer
  • 9
  • 8
17 Comments
 
LVL 13

Expert Comment

by:duncanb7
ID: 39675159
Please try this tr command,
tr -cd '\11\12\15\40-\176'  yourfile.txt(with non-printable char)  outputfile.txt(take out all non-printable char)


Please take a look at this article that might help
http://alvinalexander.com/blog/post/linux-unix/how-remove-non-printable-ascii-characters-file-unix
http://www.asciitable.com/


Hope understand your question completely, if not please point it out


Duncan
0
 

Author Comment

by:soccerplayer
ID: 39675661
duncanb7, your link allows me to remove characters. What if I want to replace them? For example, I came across characters such as ^M and <92> or <93>. Questions:

1) What are the actual characters for <92> and <93> in Unix?
2) How do I replace them with a space or some printable character?
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39675689
To remove ^M, you can do this as follows , what your mean <90> ?  you mean 90 in DEC
at http://www.asciitable.com/

Duncan

How we can solve it then?
To Solve ^M at last in each line we can do following things:
1-Linux Command shell
     dos2unix file-name new-file-name
     (dos2unix is a Linux utility tool which can convert windows oriented file to Linux compatible, so it will remove ^M        automatically from the end of each line
 
2-By setting vim configuration file i.e. .vimrc
        Open .vimrc which will be at your home location i.e. /home/aliencoders/.vimrc
        And save this line
        set ffs=unix,dos
 
3-By editing the file manually . Open test.txt in vim editor and use the following command to substitute in command mode i.e after pressing esc button.
       :1,$s/^M//g or :%s/^M//g     (Actually ctrl+v and ctrl+ M will show you ^M)
 
4-By using strings command in Linux. This command can omit all non-printable characters from the file.
       strings file-name > new-file-name
      Now, this new-file-name will not contain those non-printable characters.

Taken from www.aliencoders.com/content/how-remove-m-and-other-non-printable-characters-file-linux?page=1
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 

Author Comment

by:soccerplayer
ID: 39675708
When I did a vi file_name, I found some characters such as <92> and <93>. I am trying to find out how I can search for them and replace them if needed.
0
 
LVL 13

Accepted Solution

by:
duncanb7 earned 500 total points
ID: 39675730
To relace <90> string into "newstr" on linux

sed -i  's/<90>/newstr/g' yourfile.txt

To find and replace non-printable char with other char like  'x' or anything you wnat
try it on linux .

tr '\033' 'x' < yourfile.txt >output.txt

or

 sed   's/\o33/x/g' yourfile.txt >output.txt;

or in hex

 sed   's/\x1b/x/g' yourfile.txt >output.txt;

or with output to the same file

 sed -i  's/\o33/x/g' yourfile.txt

Hope it help

Duncan
0
 

Author Comment

by:soccerplayer
ID: 39676137
<92> is not a string. It is a non-printable character in my file just like ^M. My biggest problem is, I have them in my file but how do I search for them?
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39676173
you can try 92 as 33 in my example  if it is non-printable.

And please review those post I wrote in this thread.


Duncan
0
 

Author Comment

by:soccerplayer
ID: 39679871
Two questions:

1) How do I search after getting into vi? I tried doing:
/<92>
It didn't work.

2) How do I find out what the values of these characters are?

Thanks.
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39679878
Could you send your issue file to us so that we can  solve it ?

After sending your file, please download the file you send to us and
test it at your side again to cehck whether it is issue file you're talking about
that action is helping we will receive the correct issue file you sent for cross-check
If not repeat your issue at our side , it might be related to unix and dos format file issue


Duncan
0
 

Author Comment

by:soccerplayer
ID: 39681372
Can you tell me if I have a non-printable character such as <92>, <93> or ^M., how do I search for it using vi? That's all I need to know.
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39681662
Dear soccerplayer,

On linux shell , you can dos2unix command covert the fle from dos to unix format
On vi editor, you can convert the file from dos to unix format by command

:set fileformat=unix

If those don't help at all,

-try it in VI to detele ^M

:s/^M/x /   ###that will convert ^M to x or other char you like. you can type ctrl-v to
get "^"

-for<92>,  <93>, I believe that is extended ascill code in Hex
http://www.ascii-code.com/, and that are  ’  and  “ respectively

-try it in VI to replace or delete

:s/\x92/x/   ###-replace hex92 to x
:s/\x93/x/   ###-repace hex93 to x

If you send us the file, that will be easier for us to help

Duncan
0
 

Author Comment

by:soccerplayer
ID: 39681785
I tried, after vi-ing the file:
:s/\x92/ /

It tells me:
Pattern not found: \x92

whereas I clearly see the <92> in the file.
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39681800
if you actually see the <92> that is printable char

so why not just

:s/<92>/x/  to replace to x  or delete it

DUncan
0
 

Author Comment

by:soccerplayer
ID: 39681835
I see ^M character as well and it certainly is not printable. <92> is a non-printable character for apostrophe. How do I search for it is my issue.
0
 

Author Comment

by:soccerplayer
ID: 39681854
I found the solution through browsing. Thank for you very much.
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39681863
Just try those command I posted

such as  tr, sed, vi    

or send us your issue file if it is not too big

Otherwise, I have no more idea or room for your issue
and I will quit this thread

have a nice day

Duncan
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39681866
Thanks for your pt

Have a nice working day

Duncan
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

My previous tech tip, Installing the Solaris OS From the Flash Archive On a Tape (http://www.experts-exchange.com/articles/OS/Unix/Solaris/Installing-the-Solaris-OS-From-the-Flash-Archive-on-a-Tape.html), discussed installing the Solaris Operating S…
Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question