Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Substitutions involving non-ASCII characters

Posted on 2013-11-21
2
Medium Priority
?
294 Views
Last Modified: 2013-11-21
I have a file with some binary characters that are unrecognizable in any of my text editors.

For example, I would see something like this:

\xC2Freien - with \xC2 being some binary character. I attempted to make this substitution in my processing script to clean these out:

s/[^[:ascii:]]Freien/Freien/g;

However, it failed to do so. I know I can do something like this:

s/[^[:ascii:]]//g;

While this will get rid of those binary characters, it will also strip out non-ASCII characters I do what to keep.

I know I can also do hexadecimal substitution like this:

s/\xC2Freien/Freien/g;

However, in future cases I won't always know the hexadecimal value.

I can't cut & paste binary characters like this because \xC2 will transform into this: ? in the cut & paste process
0
Comment
Question by:hadrons
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 27

Accepted Solution

by:
ddrudik earned 2000 total points
ID: 39667524
Determine your range of disallowed characters (refer to ASCII chart for your desired range) and specify as such:
s/[\xC2-\xCF]//g;

Open in new window


You could also specify the negation of your allowed range if it's quicker to specify that (replace with your desired range):
s/[^\x00-\x7F]//g;

Open in new window

0
 

Author Closing Comment

by:hadrons
ID: 39667612
s/[^\x00-\x7F]Freien/Freien/g; is the winner; I didn't have any results with the other expression ... and I know of the [^\x00-\x7F], but I didn't think to use it. Thanks
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Whatever be the reason, if you are working on web development side,  you will need day-today validation codes like email validation, date validation , IP address validation, phone validation on any of the edit page or say at the time of registration…
As most anyone who uses or has come across them can attest to, regular expressions (regex) are a complicated bit of magic. Packed so succinctly within their cryptic syntax lies a great deal of power. It's not the "take over the world" kind of power,…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Six Sigma Control Plans

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question