emsttam
asked on
Converting unicode punctuation to ASCII
Hi,
I am reading in some text from a file. The text was saved (as tab-delimited text) from Excel, but may have originally been copied into Excel from Word. As a result of this (I assume) the text has characters such as the 'left single quotation mark' 8216 rather than the ASCII equivalent single quote. Another example is unicode 8230, which is the ellipsis (...) character which Exel seems to insert for some reason. I would like to convert any such characters into the ASCII equivalent. I understand not all incode characters have an ASCII equivalents, and those I will simple filter out, but does anyone know of a definitive list / table of which characters I can convert ? Java code would be even better of course. Thanks in advance.
I am reading in some text from a file. The text was saved (as tab-delimited text) from Excel, but may have originally been copied into Excel from Word. As a result of this (I assume) the text has characters such as the 'left single quotation mark' 8216 rather than the ASCII equivalent single quote. Another example is unicode 8230, which is the ellipsis (...) character which Exel seems to insert for some reason. I would like to convert any such characters into the ASCII equivalent. I understand not all incode characters have an ASCII equivalents, and those I will simple filter out, but does anyone know of a definitive list / table of which characters I can convert ? Java code would be even better of course. Thanks in advance.
ASKER
CEHJ,
The Java bit was optional. The question asked for a list / table.
The Java bit was optional. The question asked for a list / table.
OK. I'm not sure if there's a list available - replacements would be subjective and differ among locales. Given access to the original Unicode string, you could do something like the following for the characters you mention:
s = s.replaceAll("\u2018", "'");
s = s.replaceAll("\u2019", "'");
s = s.replaceAll("\u2026", "");
ASKER
CEHJ:
Sorry, perhaps I wasn't clear. It's not the coding that's the difficulty, it's getting a definitive list of which unicode characters to look out for which can be converted into an ASCII equivalent (such as the examples in the question).
Sorry, perhaps I wasn't clear. It's not the coding that's the difficulty, it's getting a definitive list of which unicode characters to look out for which can be converted into an ASCII equivalent (such as the examples in the question).
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
>>it's getting a definitive list of which unicode characters to look out for which can be converted into an ASCII equivalent
As i mentioned, such things are subjective: do you want to delete ellipses or replace them with three dots? Do you want to replace the left quote with a backtick or a normal single quote? Only you know
As i mentioned, such things are subjective: do you want to delete ellipses or replace them with three dots? Do you want to replace the left quote with a backtick or a normal single quote? Only you know
ASKER
Well that's certainly complete :)
I'll extract the most likely candidates from that page, the punctuation in particular. Thanks.
I'll extract the most likely candidates from that page, the punctuation in particular. Thanks.
I'm confused emsttam. It would appear from your chosen answer that the question was really 'can you show me a table of Unicode character codes?' If so, the definitive ones are here:
http://www.unicode.org/charts/
http://www.unicode.org/charts/
.. and yet it's been posted in the C# TA? I *can* give you a Java answer if that's really what you want