Converting unicode punctuation to ASCII

Posted on 2008-06-19
Last Modified: 2012-06-22

I am reading in some text from a file. The text was saved (as tab-delimited text) from Excel, but may have originally been copied into Excel from Word. As a result of this (I assume) the text has characters such as the 'left single quotation mark' 8216 rather than the ASCII equivalent single quote. Another example is unicode 8230, which is the ellipsis (...) character which Exel seems to insert for some reason. I would like to convert any such characters into the ASCII equivalent. I understand not all incode characters have an ASCII equivalents, and those I will simple filter out, but does anyone know of a definitive list / table of which characters I can convert ? Java code would be even better of course. Thanks in advance.

Question by:emsttam
  • 4
  • 3
LVL 86

Expert Comment

ID: 21821452
>>Java code would be even better of course

.. and yet it's been posted in the C# TA? I *can* give you a Java answer if that's really what you want

Author Comment

ID: 21821488

The Java bit was optional. The question asked for a list / table.
LVL 86

Expert Comment

ID: 21821543
OK. I'm not sure if there's a list available - replacements would be subjective and differ among locales. Given access to the original Unicode string, you could do something like the following for the characters you mention:

s = s.replaceAll("\u2018", "'");

s = s.replaceAll("\u2019", "'");

s = s.replaceAll("\u2026", "");

Open in new window


Author Comment

ID: 21821759

Sorry, perhaps I wasn't clear. It's not the coding that's the difficulty, it's getting a definitive list of which unicode characters to look out for which can be converted into an ASCII equivalent (such as the examples in the question).
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

LVL 13

Accepted Solution

MicheleMarcon earned 500 total points
ID: 21822035
LVL 86

Expert Comment

ID: 21822117
>>it's getting a definitive list of which unicode characters to look out for which can be converted into an ASCII equivalent

As i mentioned, such things are subjective: do you want to delete ellipses or replace them with three dots? Do you want to replace the left quote with a backtick or a normal single quote? Only you know

Author Closing Comment

ID: 31468735
Well that's certainly complete :)
I'll extract the most likely candidates from that page, the punctuation in particular. Thanks.
LVL 86

Expert Comment

ID: 21822213
I'm confused emsttam. It would appear from your chosen answer that the question was really 'can you show me a table of Unicode character codes?' If so, the definitive ones are here:

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Suggested Solutions

A short article about problems I had with the new location API and permissions in Marshmallow
It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
Viewers learn how to read error messages and identify possible mistakes that could cause hours of frustration. Coding is as much about debugging your code as it is about writing it. Define Error Message: Line Numbers: Type of Error: Break Down…
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now