Solved

Trying to find non-printable characters in a file

Posted on 2013-11-25
17
1,967 Views
Last Modified: 2013-11-27
I would like to find all non-printable characters in a file. Questions:

1) Can somebody give me an example script?
2) If I want to replace all of them with a blank space or just nothing, how would I do it?

Thanks.
0
Comment
Question by:soccerplayer
  • 9
  • 8
17 Comments
 
LVL 13

Expert Comment

by:duncanb7
ID: 39675159
Please try this tr command,
tr -cd '\11\12\15\40-\176'  yourfile.txt(with non-printable char)  outputfile.txt(take out all non-printable char)


Please take a look at this article that might help
http://alvinalexander.com/blog/post/linux-unix/how-remove-non-printable-ascii-characters-file-unix
http://www.asciitable.com/


Hope understand your question completely, if not please point it out


Duncan
0
 

Author Comment

by:soccerplayer
ID: 39675661
duncanb7, your link allows me to remove characters. What if I want to replace them? For example, I came across characters such as ^M and <92> or <93>. Questions:

1) What are the actual characters for <92> and <93> in Unix?
2) How do I replace them with a space or some printable character?
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39675689
To remove ^M, you can do this as follows , what your mean <90> ?  you mean 90 in DEC
at http://www.asciitable.com/

Duncan

How we can solve it then?
To Solve ^M at last in each line we can do following things:
1-Linux Command shell
     dos2unix file-name new-file-name
     (dos2unix is a Linux utility tool which can convert windows oriented file to Linux compatible, so it will remove ^M        automatically from the end of each line
 
2-By setting vim configuration file i.e. .vimrc
        Open .vimrc which will be at your home location i.e. /home/aliencoders/.vimrc
        And save this line
        set ffs=unix,dos
 
3-By editing the file manually . Open test.txt in vim editor and use the following command to substitute in command mode i.e after pressing esc button.
       :1,$s/^M//g or :%s/^M//g     (Actually ctrl+v and ctrl+ M will show you ^M)
 
4-By using strings command in Linux. This command can omit all non-printable characters from the file.
       strings file-name > new-file-name
      Now, this new-file-name will not contain those non-printable characters.

Taken from www.aliencoders.com/content/how-remove-m-and-other-non-printable-characters-file-linux?page=1
0
 

Author Comment

by:soccerplayer
ID: 39675708
When I did a vi file_name, I found some characters such as <92> and <93>. I am trying to find out how I can search for them and replace them if needed.
0
 
LVL 13

Accepted Solution

by:
duncanb7 earned 500 total points
ID: 39675730
To relace <90> string into "newstr" on linux

sed -i  's/<90>/newstr/g' yourfile.txt

To find and replace non-printable char with other char like  'x' or anything you wnat
try it on linux .

tr '\033' 'x' < yourfile.txt >output.txt

or

 sed   's/\o33/x/g' yourfile.txt >output.txt;

or in hex

 sed   's/\x1b/x/g' yourfile.txt >output.txt;

or with output to the same file

 sed -i  's/\o33/x/g' yourfile.txt

Hope it help

Duncan
0
 

Author Comment

by:soccerplayer
ID: 39676137
<92> is not a string. It is a non-printable character in my file just like ^M. My biggest problem is, I have them in my file but how do I search for them?
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39676173
you can try 92 as 33 in my example  if it is non-printable.

And please review those post I wrote in this thread.


Duncan
0
 

Author Comment

by:soccerplayer
ID: 39679871
Two questions:

1) How do I search after getting into vi? I tried doing:
/<92>
It didn't work.

2) How do I find out what the values of these characters are?

Thanks.
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 13

Expert Comment

by:duncanb7
ID: 39679878
Could you send your issue file to us so that we can  solve it ?

After sending your file, please download the file you send to us and
test it at your side again to cehck whether it is issue file you're talking about
that action is helping we will receive the correct issue file you sent for cross-check
If not repeat your issue at our side , it might be related to unix and dos format file issue


Duncan
0
 

Author Comment

by:soccerplayer
ID: 39681372
Can you tell me if I have a non-printable character such as <92>, <93> or ^M., how do I search for it using vi? That's all I need to know.
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39681662
Dear soccerplayer,

On linux shell , you can dos2unix command covert the fle from dos to unix format
On vi editor, you can convert the file from dos to unix format by command

:set fileformat=unix

If those don't help at all,

-try it in VI to detele ^M

:s/^M/x /   ###that will convert ^M to x or other char you like. you can type ctrl-v to
get "^"

-for<92>,  <93>, I believe that is extended ascill code in Hex
http://www.ascii-code.com/, and that are  ’  and  “ respectively

-try it in VI to replace or delete

:s/\x92/x/   ###-replace hex92 to x
:s/\x93/x/   ###-repace hex93 to x

If you send us the file, that will be easier for us to help

Duncan
0
 

Author Comment

by:soccerplayer
ID: 39681785
I tried, after vi-ing the file:
:s/\x92/ /

It tells me:
Pattern not found: \x92

whereas I clearly see the <92> in the file.
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39681800
if you actually see the <92> that is printable char

so why not just

:s/<92>/x/  to replace to x  or delete it

DUncan
0
 

Author Comment

by:soccerplayer
ID: 39681835
I see ^M character as well and it certainly is not printable. <92> is a non-printable character for apostrophe. How do I search for it is my issue.
0
 

Author Comment

by:soccerplayer
ID: 39681854
I found the solution through browsing. Thank for you very much.
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39681863
Just try those command I posted

such as  tr, sed, vi    

or send us your issue file if it is not too big

Otherwise, I have no more idea or room for your issue
and I will quit this thread

have a nice day

Duncan
0
 
LVL 13

Expert Comment

by:duncanb7
ID: 39681866
Thanks for your pt

Have a nice working day

Duncan
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Suggested Solutions

Hello fellow BSD lovers, I've created a patch process for patching openjdk6 for BSD (FreeBSD specifically), although I tried to keep all BSD versions in mind when creating my patch. Welcome to OpenJDK6 on BSD First let me start with a little …
Have you ever been frustrated by having to click seven times in order to retrieve a small bit of information from the web, always the same seven clicks, scrolling down and down until you reach your target? When you know the benefits of the command l…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now