Solved

Substitutions involving non-ASCII characters

Posted on 2013-11-21
2
275 Views
Last Modified: 2013-11-21
I have a file with some binary characters that are unrecognizable in any of my text editors.

For example, I would see something like this:

\xC2Freien - with \xC2 being some binary character. I attempted to make this substitution in my processing script to clean these out:

s/[^[:ascii:]]Freien/Freien/g;

However, it failed to do so. I know I can do something like this:

s/[^[:ascii:]]//g;

While this will get rid of those binary characters, it will also strip out non-ASCII characters I do what to keep.

I know I can also do hexadecimal substitution like this:

s/\xC2Freien/Freien/g;

However, in future cases I won't always know the hexadecimal value.

I can't cut & paste binary characters like this because \xC2 will transform into this: ? in the cut & paste process
0
Comment
Question by:hadrons
2 Comments
 
LVL 27

Accepted Solution

by:
ddrudik earned 500 total points
ID: 39667524
Determine your range of disallowed characters (refer to ASCII chart for your desired range) and specify as such:
s/[\xC2-\xCF]//g;

Open in new window


You could also specify the negation of your allowed range if it's quicker to specify that (replace with your desired range):
s/[^\x00-\x7F]//g;

Open in new window

0
 

Author Closing Comment

by:hadrons
ID: 39667612
s/[^\x00-\x7F]Freien/Freien/g; is the winner; I didn't have any results with the other expression ... and I know of the [^\x00-\x7F], but I didn't think to use it. Thanks
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

713 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question