Solved

Substitutions involving non-ASCII characters

Posted on 2013-11-21
2
270 Views
Last Modified: 2013-11-21
I have a file with some binary characters that are unrecognizable in any of my text editors.

For example, I would see something like this:

\xC2Freien - with \xC2 being some binary character. I attempted to make this substitution in my processing script to clean these out:

s/[^[:ascii:]]Freien/Freien/g;

However, it failed to do so. I know I can do something like this:

s/[^[:ascii:]]//g;

While this will get rid of those binary characters, it will also strip out non-ASCII characters I do what to keep.

I know I can also do hexadecimal substitution like this:

s/\xC2Freien/Freien/g;

However, in future cases I won't always know the hexadecimal value.

I can't cut & paste binary characters like this because \xC2 will transform into this: ? in the cut & paste process
0
Comment
Question by:hadrons
2 Comments
 
LVL 27

Accepted Solution

by:
ddrudik earned 500 total points
ID: 39667524
Determine your range of disallowed characters (refer to ASCII chart for your desired range) and specify as such:
s/[\xC2-\xCF]//g;

Open in new window


You could also specify the negation of your allowed range if it's quicker to specify that (replace with your desired range):
s/[^\x00-\x7F]//g;

Open in new window

0
 

Author Closing Comment

by:hadrons
ID: 39667612
s/[^\x00-\x7F]Freien/Freien/g; is the winner; I didn't have any results with the other expression ... and I know of the [^\x00-\x7F], but I didn't think to use it. Thanks
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now