Solved

Handling UTF8 Unix flat file, while maintaining international text characters

Posted on 2004-09-17
8
1,755 Views
Last Modified: 2008-03-10
I am trying to write a series of VBA macros to parse and validate flat files, ready for import into a database.  The database, file format and technology are not changeable.

The flat file is sent to us, containing data that should be added, as a Unix text file, with UTF-8 encoding and no Byte Object Marker.  However, it can contain any and all number of foreign characters (Chinese, Taiwanese, Japanese, Sri Lankan, Arabic, etc).  

In its current state, my Excel application can read data, validate and recompile the desired output to another sheet, ready for export, without affecting the characters.  

The primary problem occurs when I try to export the file.  The exported file format has to match that of the imported file.  When I view the original file in SC Unipad, I can see the file is a Unix UTF-8 file, with no Byte Order Marker.  The output file, however, is a Unix UTF-16 Little Endian file, with a Byte Object Marker.

I am using the File System Object; creating the file with
    Set FileObj = FSO.CreateTextFile(OutputFile, Overwrite, True)

and writing with
    FileObj.Write (Sheets(OutSheet).Cells(RowCount, 1).Value)
    FileObj.Write EndLine                                                        ' Endline = Chr(10)

to write each individual row.

I have tried using the WideCharToMultiByte() external function (with Codepage = 65001) in a UTF8Encode() function, as suggested in previous questions, so the write line is
    FileObj.Write (UTF8Encode (Sheets(OutSheet).Cells(RowCount, 1).Value))

but this still creates a UTF-16 file and the foreign characters are lost in favour of ANSI(?) counterparts.

I figure the UTF-16 output might be due to the Unicode flag in FSO.CreateTextFile() being set to True, but I cannot see any other way of maintaining the characters.

I would prefer to avoid opening SC Unipad in a shell and using it to open the UTF-16 file, remove the BOM and convert to UTF-8, as it would rely too heavily on third party software.

Any help anyone can give would be greatly appreciated...
0
Comment
Question by:jimbobmcgee
  • 5
  • 3
8 Comments
 
LVL 16

Author Comment

by:jimbobmcgee
ID: 12142251
Anybody?

...

:)
0
 
LVL 16

Author Comment

by:jimbobmcgee
ID: 12190911
Please?

...

:)
0
 
LVL 3

Accepted Solution

by:
Arundel_Castle earned 500 total points
ID: 12200047
Try this link Jim Bob. :)

http://www.cyberactivex.com/UnicodeTutorialVb.htm

By the way my name is also Magee. Magee and McGee are the same name.
0
 
LVL 3

Expert Comment

by:Arundel_Castle
ID: 12200056
Thanks for the points Jim, by the way my first name is also James:) Though I have no bob.
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 3

Expert Comment

by:Arundel_Castle
ID: 12200075
Actually thats a pretty interesting question in working with multinational encoding from a unix based system.
0
 
LVL 3

Expert Comment

by:Arundel_Castle
ID: 12200385
You don't have to give me the extra points for this question Jim. But it would be nice if you did. I would reach my experts total. I haven't used this site in a couple of years but I was an expert before.
0
 
LVL 16

Author Comment

by:jimbobmcgee
ID: 12200599
>> Actually thats a pretty interesting question in working with multinational encoding from a unix based system.

It was a pain!!  I think my original problem was due to opening the text stream in unicode (utf16).  The second problem I had was due to my UTF8Encode function using WideCharToMultiByte and then returning a Left(...) on the result.  I think this converted the string back to UTF16.

Either way, all cleared up by the link you posted...
0
 
LVL 3

Expert Comment

by:Arundel_Castle
ID: 12200765
Thanks for the points Jim, by the way it does sound like a nightmare to write for. Thanks again.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Introduction While answering a recent question (http://www.experts-exchange.com/Q_27402310.html) in the VB classic zone, I wrote some VB code in the (Office) VBA environment, rather than fire up my older PC.  I didn't post completely correct code o…
Enums (shorthand for ‘enumerations’) are not often used by programmers but they can be quite valuable when they are.  What are they? An Enum is just a type of variable like a string or an Integer, but in this case one that you create that contains…
As developers, we are not limited to the functions provided by the VBA language. In addition, we can call the functions that are part of the Windows operating system. These functions are part of the Windows API (Application Programming Interface). U…
Get people started with the utilization of class modules. Class modules can be a powerful tool in Microsoft Access. They allow you to create self-contained objects that encapsulate functionality. They can easily hide the complexity of a process from…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now