Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2474
  • Last Modified:

Trying to find non-printable characters in a file

I would like to find all non-printable characters in a file. Questions:

1) Can somebody give me an example script?
2) If I want to replace all of them with a blank space or just nothing, how would I do it?

Thanks.
0
soccerplayer
Asked:
soccerplayer
  • 9
  • 8
1 Solution
 
duncanb7Commented:
Please try this tr command,
tr -cd '\11\12\15\40-\176'  yourfile.txt(with non-printable char)  outputfile.txt(take out all non-printable char)


Please take a look at this article that might help
http://alvinalexander.com/blog/post/linux-unix/how-remove-non-printable-ascii-characters-file-unix
http://www.asciitable.com/


Hope understand your question completely, if not please point it out


Duncan
0
 
soccerplayerAuthor Commented:
duncanb7, your link allows me to remove characters. What if I want to replace them? For example, I came across characters such as ^M and <92> or <93>. Questions:

1) What are the actual characters for <92> and <93> in Unix?
2) How do I replace them with a space or some printable character?
0
 
duncanb7Commented:
To remove ^M, you can do this as follows , what your mean <90> ?  you mean 90 in DEC
at http://www.asciitable.com/

Duncan

How we can solve it then?
To Solve ^M at last in each line we can do following things:
1-Linux Command shell
     dos2unix file-name new-file-name
     (dos2unix is a Linux utility tool which can convert windows oriented file to Linux compatible, so it will remove ^M        automatically from the end of each line
 
2-By setting vim configuration file i.e. .vimrc
        Open .vimrc which will be at your home location i.e. /home/aliencoders/.vimrc
        And save this line
        set ffs=unix,dos
 
3-By editing the file manually . Open test.txt in vim editor and use the following command to substitute in command mode i.e after pressing esc button.
       :1,$s/^M//g or :%s/^M//g     (Actually ctrl+v and ctrl+ M will show you ^M)
 
4-By using strings command in Linux. This command can omit all non-printable characters from the file.
       strings file-name > new-file-name
      Now, this new-file-name will not contain those non-printable characters.

Taken from www.aliencoders.com/content/how-remove-m-and-other-non-printable-characters-file-linux?page=1
0
NFR key for Veeam Backup for Microsoft Office 365

Veeam is happy to provide a free NFR license (for 1 year, up to 10 users). This license allows for the non‑production use of Veeam Backup for Microsoft Office 365 in your home lab without any feature limitations.

 
soccerplayerAuthor Commented:
When I did a vi file_name, I found some characters such as <92> and <93>. I am trying to find out how I can search for them and replace them if needed.
0
 
duncanb7Commented:
To relace <90> string into "newstr" on linux

sed -i  's/<90>/newstr/g' yourfile.txt

To find and replace non-printable char with other char like  'x' or anything you wnat
try it on linux .

tr '\033' 'x' < yourfile.txt >output.txt

or

 sed   's/\o33/x/g' yourfile.txt >output.txt;

or in hex

 sed   's/\x1b/x/g' yourfile.txt >output.txt;

or with output to the same file

 sed -i  's/\o33/x/g' yourfile.txt

Hope it help

Duncan
0
 
soccerplayerAuthor Commented:
<92> is not a string. It is a non-printable character in my file just like ^M. My biggest problem is, I have them in my file but how do I search for them?
0
 
duncanb7Commented:
you can try 92 as 33 in my example  if it is non-printable.

And please review those post I wrote in this thread.


Duncan
0
 
soccerplayerAuthor Commented:
Two questions:

1) How do I search after getting into vi? I tried doing:
/<92>
It didn't work.

2) How do I find out what the values of these characters are?

Thanks.
0
 
duncanb7Commented:
Could you send your issue file to us so that we can  solve it ?

After sending your file, please download the file you send to us and
test it at your side again to cehck whether it is issue file you're talking about
that action is helping we will receive the correct issue file you sent for cross-check
If not repeat your issue at our side , it might be related to unix and dos format file issue


Duncan
0
 
soccerplayerAuthor Commented:
Can you tell me if I have a non-printable character such as <92>, <93> or ^M., how do I search for it using vi? That's all I need to know.
0
 
duncanb7Commented:
Dear soccerplayer,

On linux shell , you can dos2unix command covert the fle from dos to unix format
On vi editor, you can convert the file from dos to unix format by command

:set fileformat=unix

If those don't help at all,

-try it in VI to detele ^M

:s/^M/x /   ###that will convert ^M to x or other char you like. you can type ctrl-v to
get "^"

-for<92>,  <93>, I believe that is extended ascill code in Hex
http://www.ascii-code.com/, and that are  ’  and  “ respectively

-try it in VI to replace or delete

:s/\x92/x/   ###-replace hex92 to x
:s/\x93/x/   ###-repace hex93 to x

If you send us the file, that will be easier for us to help

Duncan
0
 
soccerplayerAuthor Commented:
I tried, after vi-ing the file:
:s/\x92/ /

It tells me:
Pattern not found: \x92

whereas I clearly see the <92> in the file.
0
 
duncanb7Commented:
if you actually see the <92> that is printable char

so why not just

:s/<92>/x/  to replace to x  or delete it

DUncan
0
 
soccerplayerAuthor Commented:
I see ^M character as well and it certainly is not printable. <92> is a non-printable character for apostrophe. How do I search for it is my issue.
0
 
soccerplayerAuthor Commented:
I found the solution through browsing. Thank for you very much.
0
 
duncanb7Commented:
Just try those command I posted

such as  tr, sed, vi    

or send us your issue file if it is not too big

Otherwise, I have no more idea or room for your issue
and I will quit this thread

have a nice day

Duncan
0
 
duncanb7Commented:
Thanks for your pt

Have a nice working day

Duncan
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

  • 9
  • 8
Tackle projects and never again get stuck behind a technical roadblock.
Join Now