Solved

Substitutions involving non-ASCII characters

Posted on 2013-11-21
2
271 Views
Last Modified: 2013-11-21
I have a file with some binary characters that are unrecognizable in any of my text editors.

For example, I would see something like this:

\xC2Freien - with \xC2 being some binary character. I attempted to make this substitution in my processing script to clean these out:

s/[^[:ascii:]]Freien/Freien/g;

However, it failed to do so. I know I can do something like this:

s/[^[:ascii:]]//g;

While this will get rid of those binary characters, it will also strip out non-ASCII characters I do what to keep.

I know I can also do hexadecimal substitution like this:

s/\xC2Freien/Freien/g;

However, in future cases I won't always know the hexadecimal value.

I can't cut & paste binary characters like this because \xC2 will transform into this: ? in the cut & paste process
0
Comment
Question by:hadrons
2 Comments
 
LVL 27

Accepted Solution

by:
ddrudik earned 500 total points
ID: 39667524
Determine your range of disallowed characters (refer to ASCII chart for your desired range) and specify as such:
s/[\xC2-\xCF]//g;

Open in new window


You could also specify the negation of your allowed range if it's quicker to specify that (replace with your desired range):
s/[^\x00-\x7F]//g;

Open in new window

0
 

Author Closing Comment

by:hadrons
ID: 39667612
s/[^\x00-\x7F]Freien/Freien/g; is the winner; I didn't have any results with the other expression ... and I know of the [^\x00-\x7F], but I didn't think to use it. Thanks
0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Regular expression patterns 2 74
.net string parse 18 55
perl: Cleaning meta tags using RegEX 12 81
Regex code:separate email:ip from email:pass ? 6 55
As most anyone who uses or has come across them can attest to, regular expressions (regex) are a complicated bit of magic. Packed so succinctly within their cryptic syntax lies a great deal of power. It's not the "take over the world" kind of power,…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question