July 24, 2008 02:38am pdt

1. jkr 716,400
2. Infinity08 471,651
3. evilrix 468,580
4. itsmeand... 171,101
5. jaime_ol... 96,523
6. josgood 76,798
7. ikework 62,815
8. Gurudenis 55,037
9. SeanDurkin 49,861
10. DanRollins 48,566
11. rstaveley 46,482
12. alb66 42,870
13. Zoppo 40,400
14. Kdo 35,444
15. JohnGaby 25,090

1. jkr 6,976,677
2. itsmeand... 1,842,868
3. Axter 1,462,534
4. Infinity08 1,287,848
5. nietod 859,677
6. AlexFM 732,762
7. jaime_ol... 658,178
8. DanRollins 614,769
9. evilrix 585,010
10. rstaveley 511,924
11. grg99 390,024
12. efn 315,320
13. chensu 313,112
14. AlexViro... 263,393
15. Salte 247,544

01.10.2004 at 12:18AM PST, ID: 20846903

	[x] Attachment Details

[x]

The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

The Grade of the Solution
The Zone Rank of the Expert Providing the Solution
The Number of Author and Expert Comments
The Number of Experts Contributing
The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!

7.4

Convert ASCII string to UTF8 string

Zone: C++ Programming Language

Tags: convert, utf8, ascii, string

Title explains it all ;)

I want to convert a string like "ûüâäç" etc to UTF8 encoding...

How can I do this with an easy C(++) function?

Start your free trial to view this solution

Question Stats

Zone: Programming

Question Asked By: G00fy

Solution Provided By: Axter

Participating Experts: 2

Solution Grade: A

Translate:

Loading Advertisement...

Microsoft

Internet Protocols
Applications

Development
OS

Hardware
Windows Security

Apple

Operating Systems
Hardware

Programming
Networking

Software

Internet

Search Engines
File Sharing
WebTrends / Stats
Spy / Ad Blockers

Web Browsers
New Net Users
Web Development
Chat / IM

Anti Spam
Web Servers
Anti-Virus
Email Clients

Gamers

Tips
Online / MMORPG
Puzzle
Emulators

Action / Adventure
Role Playing
Consoles
Game Programming

Strategy
Sports
Misc
Computer Games

Digital Living

Hardware
Automotive
New Net Users
New Users

Software
Digital Music
Gaming World
Home Security

Apple
Networking Hardware

Virus & Spyware

Vulnerabilities
IDS
Encryption
Anti-Virus

Operating Systems Security
Software Firewalls
WebApplications
Cell Phones

Operating Systems
Internet
Hardware Firewalls

Hardware

Displays / Monitors
Handhelds / PDAs
Components
Peripherals
Laptops/Notebooks

Servers
Misc
Apple
Embedded Hardware
Networking Hardware

Storage
Desktops
New Users

Software

System Utilities
Industry Specific
Network Management
Photos / Graphics
Page Layout
VMware
Misc
Web Development

OS
CYGWIN
Voice Recognition
Virtualization
Message Queue
Quality Assurance
Security
Firewalls

MultiMedia Applications
Development
Database
Office / Productivity
Business Management
OS/2 Apps
Server Software
Internet / Email

ITPro

OS
Storage
Encryption
Operating Systems Security
Apple Hardware
Laptops & Notebooks
Servers
Networking Hardware
Peripherals
Devices

Displays / Monitors
WebTrends / Stats
Search Engines
Firewalls
Web Computing
WebApplications
IDS
Vulnerabilities
Email Clients
File Sharing

Spy / Ad Blockers
Web Browsers
Web Servers
Networking
Anti-Virus
Consulting
Chat / IM
Anti Spam

Developer

Web Servers
Web Browsers
Game Programming
Dev Tools
Industry Specific
Office / Productivity

Database
CYGWIN
Web Development
Search Engines
File Sharing
WebTrends / Stats

Programming
Content Management
Application Servers
Protocols

Storage

Removable Backup Media
Storage Technology
Servers

Grid
Remote Access
Backup / Restore

Misc
Hard Drives

Miscellaneous
Security
Development
Linux
VMware

MainFrame OS
Unix
Apple
OS / 2
AS / 400

BeOS
Microsoft
VMS / OpenVMS

Database

Oracle
Miscellaneous
MySQL
Software
Sybase
Contact Management
PostgreSQL
Data Manipulation
Clarion
InterSystems Cache

Siebel
MUMPS
OLAP
SQLBase
SAS
GIS & GPS
4GL
Berkeley DB
DB2
Informix

Interbase / Firebird
FoxPro
Reporting
LDAP
Filemaker Pro
MS SQL Server
dBase
MS Access

Security

Misc
Web Browsers
Software Firewalls
Operating Systems Security
File Sharing

Spy / Ad Blockers
Vulnerabilities
WebApplications
IDS
Anti-Virus

Encryption
Anti Spam
Email Clients
VPN
Chat / IM

Programming

Editors IDEs
Installation
Handhelds / PDAs
Multimedia Programming
System / Kernel
Automation

Algorithms
Game
Signal Processing
Project Management
Open Source
Database

Misc
Languages
Processor Platforms
Theory

Web Development

Scripting
Blogs
Web Servers
Software
Search Engines
Web Graphics
Web Services

Images
Internet Marketing
Images and Photos
Components
Document Imaging
Web Languages/Standards
Illustration

WebApplications
Fonts
WebTrends / Stats
Authoring
Digital Camera Software
Miscellaneous

Networking

Protocols
Apple Networking
Network Management
Message Queue
Application Servers
Content Management
File Servers
Email Servers
Misc
Java Editors & IDEs

Wireless
Networking Hardware
Backup / Restore
System Utilities
ISPs & Hosting
Web Servers
Storage Technology
Removable Backup Media
Servers
Web Computing

Broadband
Grid
OS / 2
Novell Netware
Unix Networking
Windows Networking
Security
Telecommunications
Operating Systems
Linux Networking

Other

Lounge
Business Travel
Community Support
New Net Users

Philosophy / Religion
Math / Science
Miscellaneous
URLs

Expert Lounge
Politics
Puzzles / Riddles
Automotive

Community Support

Suggestions
New to EE
New Topics
CleanUp

Announcements
General
Feedback

Input
EE Bugs

01.10.2004 at 12:21AM PST, ID: 10086062

Rank: Genius

Axter:

Your question does not match the title of your question.

Do you want to convert ASCII to UTF8?
Or do you want to convert UNICODE to UTF8?

01.10.2004 at 12:22AM PST, ID: 10086064

Rank: Genius

Axter:

A windows project can use the [b]MultiByteToWideChar[/b] API function to convert an ANSI string to a UNICODE string.
Example:
[code]
void Function(void)
{
char dataBuff[] = "abcdefghijklmnopq";

DWORD Pos = 10;

CString tmpStr = "";
wchar_t* pwsz = tmpStr.GetBufferSetLength ((Pos+1)*sizeof(wchar_t));
MultiByteToWideChar(CP_ACP, 0, dataBuff, strlen(dataBuff), pwsz, (Pos+1)*sizeof(wchar_t));
tmpStr.ReleaseBuffer();
}
[/code]

Accepted Solution

01.10.2004 at 12:23AM PST, ID: 10086065

Rank: Genius

Axter:

The C/C++ mbstowcs function can be used to convert an ANSI string to UNICODE.

mbstowcs is more portable then MultiByteToWideChar, and should work on any C/C++ compliant compiler

01.10.2004 at 12:24AM PST, ID: 10086073

Rank: Genius

Axter:

To convert UNICODE string to ANSI string, check out the following link:

http://www.axter.com/faq/topic.asp?TOPIC_ID=63&FORUM_ID=4&CAT_ID=9

01.10.2004 at 12:34AM PST, ID: 10086092

Rank: Genius

Axter:

For VC++, if you want to convert an ASCII to a UTF8 you could use MultiByteToWideChar and then use WideCharToMultiByte.

Use the MultiByteToWideChar to convert ASCII to UNICODE, and then use WideCharToMultiByte to conver from UNICODE to UTF8.

01.11.2004 at 02:22PM PST, ID: 10092559

G00fy:

No, what I meant is to convert an ansii string to utf8 encoding...

so it means convert the 'ü' character (char -4) to utf8 (-62 -81 if I remember correctly)?

[btw, is it logical I did see emails coming in with replies from you, but that I didn't see the posts itself?]

01.11.2004 at 02:24PM PST, ID: 10092566

G00fy:

Isn't there an easier way then WC2MB & MB2WC ?

That works ... But :S It's so slow (I mean there SHOULD be something like a 3 lines function or so)

01.11.2004 at 02:27PM PST, ID: 10092573

Rank: Genius

Axter:

>>[btw, is it logical I did see emails coming in with replies from you, but that I didn't see the posts itself?]

You have to click on the link to Experts-Exchange, to see the reply.

>>so it means convert the 'ü' character (char -4) to utf8 (-62 -81 if I remember correctly)?

Did you try the functions I posted?

FYI:
'ü' is not an ASCII character.

Where are you getting this character from? How is it introduced into your code?

01.11.2004 at 02:32PM PST, ID: 10092588

Rank: Genius

Axter:

>>That works ... But :S It's so slow (I mean there SHOULD be something like a 3 lines function or so)

What do you mean it's slow?
How do you know it's slow?
Did you do a bench mark test?

Can you post your code?

01.11.2004 at 02:34PM PST, ID: 10092592

G00fy:

I tried it, it works, but when importing like 20k lines from an ascii file, this is getting too slow for me...

the characters come to me via an ascii file...
I read line per line, parse it & then I convert for example the names of the people in it to UTF8-encoding... (actually all the non-numeric fields are being converted).

And then I need it to submit it to SQLite, which is compiled in UTF8-mode

01.11.2004 at 02:40PM PST, ID: 10092618

G00fy:

char* lijn; // here is something inside I need to convert
wchar_t * lijn2 = new wchar_t[strlen(lijn)+1]
MultiByteToWideChar(CP_ACP, 0, lijn, strlen(lijn), lijn2, strlen(lijn));
delete [] lijn;
lijn = new char[wcslen(lijn2)*3+1] // ugly yes :p
WideCharToMultiByte(CP_UTF8, 0, lijn2, wcslen(lijn2), lijn, wcslen(lijn2)*3, 0, NULL);

--> was something like that ... already ditched it
(currently going via wxWindows methods)
wxString test( lijn, wxConvLibc );
test.mb_str( wxConvUTF8 );

works OK for me ... But this also is ways too slow :(

01.11.2004 at 02:43PM PST, ID: 10092630

Rank: Genius

Axter:

>>I tried it, it works, but when importing like 20k lines from an ascii file, this is getting too slow for me...

Again, how do you know it's slow?
Did you run any type of valid test to see if it is slow?

If so, please explain.

This method should not impact your code, since the real bottle neck will be in reading the file.

Do a test with the function calls, and compare it to running your code without the function calls. I would be very surprise if you could measure a significant difference.

01.11.2004 at 02:44PM PST, ID: 10092635

Rank: Genius

Axter:

>>works OK for me ... But this also is ways too slow :(

Please post your method for testing speed.

01.11.2004 at 02:49PM PST, ID: 10092650

Rank: Genius

Axter:

Why are you using UTF8 instead of wide string (UNICODE)?

01.11.2004 at 02:51PM PST, ID: 10092656

half_life_fool:

I used to have the same problem before:

*lijn2++ = (char)(192 + (((unsigned char)lijn[current_number]) / 64));
*lijn2++ = (char)(128 + (((unsigned char)lijn[current_number]) % 64));

this converts lijn to lijn2 where lijn = ansi, lijn2 = utf8
only usuable if used for ansi-strings!

Assisted Solution

01.11.2004 at 02:55PM PST, ID: 10092673

G00fy:

wxStopWatch sw;
wxMessageBox( wxString::Format( "Time elapsed: %ldms", sw.Time() ) );

This stopwatch starts before the file being read in, and stops after the file is read in...

It takes +- 5.6s to read in the file via wxString, via the other calls it takes 7.2s ...

Not a huge difference, but I think the real bottleneck is when assigning the memory for the second string ...

01.11.2004 at 03:04PM PST, ID: 10092714

G00fy:

Checked them all out:

wchar_t * lijn2 = new wchar_t[MAX_BUFFER_LENGTH];
MultiByteToWideChar(CP_ACP, 0, abuffer, strlen(abuffer)+1, lijn2, MAX_BUFFER_LENGTH);
WideCharToMultiByte(CP_UTF8, 0, lijn2, wcslen(lijn2)+1, abuffer, MAX_BUFFER_LENGTH, 0, NULL);

==> 1200ms <-> 1300ms

*lijn2++ = (char)(192 + (((unsigned char)lijn[current_number]) / 64));
*lijn2++ = (char)(128 + (((unsigned char)lijn[current_number]) % 64));
==> 1046ms <-> 1000ms

wxString test( abuffer, wxConvLibc );
strcpy(abuffer, test.mb_str( wxConvUTF8 ) );
==> 1360ms <-> 2703ms

01.11.2004 at 03:04PM PST, ID: 10092718

G00fy:

PS: I took the writing to the database out of it, so it would be faster

20080236-EE-VQP-29