Convert Unicode file to "normal" file.

ooh, I forgot to give you this link which explains endianess...

http://en.wikipedia.org/wiki/Endianness

...basically, it just denotes which way around the bytes are ordered in data types that are compound bytes.

So, imagine we have UTF16, the 16 bit word is made up of 2 bytes, let's call them H for high order and L for low order. This is how they would be ordered depending upon their endianess...

Address         0 1  2 3  4 5  6 7
---------------------------------------
Big Endian:    [H|L][H|L][H|L][H|L]
Little Endian: [L|H][L|H][L|H][L|H]

                                              

1:
2:
3:
4:

>> Does anyone know of a command line utility (or freeware program) that will convert my file to normal ASCII?
It's a bit quite and dirtly, but try compiling the code below. It takes standard in as a wide stream and write to standard out as a narrow stream.

Assume you call it conv.exe, you use it like this...
'type utf16.txt | conv.exe  > utf8.txt'

type is a DOS command that will stream a file through the standard input of a program and > is an operator that will take the standard output of a program and redirect it to a file. In this case the input file is called utf16.txt and the output file is called utf8.txt. I can't promise this'll work with any fancy languages like Chinese but it seemed to work fine with English saved as UTF16 and then converted back. Also, there is no error handling, I leave that to you to implement as you see fit.

I hope this helps.

-Rx.

#include <iostream>
 
int main()
{
	wchar_t wc = 0;;
	while(!std::wcin.get(wc).eof())
	{
		char c = 0;
		wcstombs(&c, &wc, 1);
		std::cout.put(c);
	}
}
                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:

IN all fairness, I haven't yet studied what you wrote.
Is there a dog-simple way to do this.
For example a command-line runnablle program that I could use
    Convert old-file.txt new-file.txt
Thanks

I just now so your code-snippet.  That's possible too.  But the command-line would be nice.  Forgive my delay in studying what you wrote yet.

The iconv utility is what Linux/UN*X people use to convert between character formats.

  iconv --from utf-16  --to utf-8 < source-file.txt > destination-file.txt

You can Google for a Win32 port or get up and running with Cygwin.

If you have a couple of weird looking binary characters at the beginning of your UTF-16 file (0xffef as it is little-endian), and want to use evilrix's utility in http:#21644289, you will need to discard the first wchar_t. This is the BOM (see http://en.wikipedia.org/wiki/Byte_Order_Mark).

>> Is there a dog-simple way to do this.
I thought what I posted was simple... 7 lines of code to convert wide to narrow... even hello world is about 3 :)

>> I just now so your code-snippet.  That's possible too.  But the command-line would be nice
That's why I gave you the code... all you have to do is build it and you have a simple command line tool


BTW: It is a quick and dirty bit of code, it has been provided by way of showing you the basics so you can build on it yourself. It won't as rstaveley points out, handle BOM markers correctly. Handling that, though, is simple (as rstaveley explains) and I've left it up to you to do as a learning exercise.

try this
cmd /A /C type "Norton-Ghost.log.txt" >newNG.txt

Canali: I tried it.  It didn't work.  Did you try it?  A copy of the actual file is attached to the top of this thread.  You'll find it at the bottom of the initial question, in the Attachments section.  I've attached it here again.  Thanks.

Did you try the code I posted above? I see no reason why this wouldn't work.

I wrote the following program called "utf16_to_utf8.exe":  (See attached code snippet.)
Then I ran the following command from the CMD prompt:
    type "norton Ghost 9.0.log.txt" | utf16_to_utf8.exe >utf8.txt
The program just sat there - it didn't end.  I mean I waited what seemed to be a long time, an dit didn't end.
The part of the program that is unclear is how I am handling the BOM (first) byte.  Maybe I'm not doing that correctly.  (See "if bFirst == FALSE...)
The only other difference is me having added "#include "stdafx.h".
Where did I go wrong?  Thanks

// utf16_to_utf8.cpp : Defines the entry point for the console application.
//
 
#include "stdafx.h"
#include <iostream>
 
int main()
{
	bool bFirst = false;
	wchar_t wc = 0;
	while (!std::wcin.get(wc).eof())
	{
		if (bFirst == false)
		{
			bFirst = true;
			continue;
		}
		char c = 0;
		wcstombs(&c, &wc, 1);
		std::cout.put(c);
	}
	return 0;
}

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:

Well, I'm doing great!
Because I had problems with evilrix's solution, I tried rstaveley's solution.
I downloaded iconv.exe from here: http://sourceforge.net/project/showfiles.php?group_id=25167&package_id=51458
It put an iconv.exe in a Bin folder.  I copied that iconv.exe to the folder with my UTF16 file and entered
       iconv --from utf-16 --to utf-8 < "Norton Ghost 9.0.log.txt" > fixed.txt
It finished VERY fast and created an empty output file
So ... I'm batting 0 for 2 today!

>> Because I had problems with evilrix's solution
What problem was that exactly? You can ask for more help you know, I'm happy to give it.

Oh I forgot about this problem.... You'll find that wcin.get() extracts only one byte from stdin in certain locales (same as wfstream), because the external format is narrow. You'll need to open the file in binary mode to get it to read 16-bits at a time.

Here's some rumination about this the last time I messed up on this: http:Q_20797258.html#9753379

> external format

"External representation" is what they call it.

Try writing "Hello, world" into an ofstream in most locales and you'll get a file of 12 bytes (not 24/48 as you might expect in Windows/Linux for wchar_t). That's because your external representation is typically optimised as narrow (char) and internal representation optimised as wide (wchar_t).

I had to read the file as Binary and then convert it to UTF16 Little Endian to read the file

I wrote a script with AutoIT ( I was feeling bored) which will work from Command Prompt i.e
Convert OldText.txt NewText.txt.

I've compiled to an exe and can be found here http://matrixnz.321.co.nz/Convert.exe

Or if you wish to compile yourself

Download and Install AutoIT and also AutoIT Script Editor found here - http://www.autoitscript.com/autoit3/downloads.shtml
AutoIt is a freeware Windows automation language.

Once installed, copy and paste the code snippet into Scite (AutoIT Script Editor), save the file as Convert.au3 and then click Tools -> Build this will create the Convert.exe in the same folder as the Script.

From a command prompt you can use:
Convert /? for help
examples
Convert Norton-Ghost.log.txt NewText.txt
or
Convert "Norton Ghost.log.txt" "New Text.txt"

Cheers

#NoTrayIcon
#AutoIt3Wrapper_Change2CUI=y
 
If	$CmdLine[0] < 3 Then
	If $CmdLine[1] = "/?" Then
		ConsoleWrite("Usage:" & @CRLF)
		ConsoleWrite(@TAB & "Convert [/? | <inputfile.txt> | <outputfile.txt> |" & @CRLF & @CRLF)
		ConsoleWrite(@TAB & "Options:" & @CRLF)
		ConsoleWrite(@TAB & "   /?" & @TAB & @TAB & @TAB & "Display this help message" & @CRLF)
		ConsoleWrite(@TAB & "   <inputfile.txt>" & @TAB & "UDF Text File to Convert" & @CRLF)
		ConsoleWrite(@TAB & "   <outputfile.txt>" & @TAB & "Converted Text File" & @CRLF & @CRLF)
		ConsoleWrite(@TAB & "Example:" & @CRLF)
		ConsoleWrite(@TAB & "   Convert ""UDF File.txt"" ""Output File.txt""" & @CRLF & @CRLF)
		Exit
	Else
		$Input = StringRight($CmdLine[1], 4)
		If $Input = ".txt" Then
			$INP_FILE = FileOpen($CmdLine[1], 16)
			If @error = -1 Then
				ConsoleWrite("Input File is invalid")
			EndIf
			$READ_FILE = FileRead($INP_FILE)
			FileClose($INP_FILE)
		Else
			ConsoleWrite("Error: No Input File specified.  Please see help [ /? ] for options" & @CRLF)
			Exit
		EndIf
	EndIf
 
	If $CmdLine[0] = 2 Then
		$Output = StringRight($CmdLine[2], 4)
		If $Output = ".txt" Then
			$OUT_FILE = FileOpen($CmdLine[2], 34)
			If @error = -1 Then
				ConsoleWrite("Output File is invalid")
			EndIf
			FileWrite($OUT_FILE, $READ_FILE)
			FileClose($OUT_FILE)
			Exit
		Else
			ConsoleWrite("Error: No Output File specified. Please see help [ /? ] for options")
			Exit
		EndIf
	Else
		ConsoleWrite("Error: No Output File specified. Please see help [ /? ] for options")
		Exit
	EndIf
Else
		ConsoleWrite("Error: Please see help [ /? ] for options")
		Exit
EndIf
                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:

Typo

> Try writing "Hello, world" into an ofstream in most locales

Try writing "Hello, world" into an owfstream in most locales

> I had to read the file as Binary

Yes, that's the right thing to do.

At the risk of flogging a dead horse, the following snippet shows you how to have a wide "external representation".

There is no BOM handling, but I notice DFlaschen's .txt file doesn't appear to have a BOM anyhow and I couldn't think of an elegant way to handle the BOM in IOStreams, when I last played with this in 2003/2004.

This writes four characters into a "wide" file and reads them from a "wide" file, doing the right thing with IOStreams, by imbuing a "no conversion" locale to see that the external representation is wide. You can of course hack this to read the .txt file and then write a .txt without imbuing the "no conversion" locale.

Very silly, isn't it?  I hope it's educational though.

#include <iostream>
#include <fstream>
#include <string>
#include <locale>
 
namespace noconv {
 
class codecvt : public std::codecvt<wchar_t,char,mbstate_t> {
public:
	explicit codecvt(size_t refs = 0) : std::codecvt<wchar_t,char,mbstate_t>(refs) {}
protected:
	virtual	result do_in(
		state_type& state
		,const extern_type *from_begin
		,const extern_type *from_end
		,const extern_type *&from_next
		,intern_type *to_begin
		,intern_type *to_end
		,intern_type *&to_next
		) const
	{
	        return noconv;
	}
 
/* Here's where we convert from the internal representation to the external
   representation written to disk */
 
	virtual	result do_out(
		state_type& state
		,const intern_type *from_begin
		,const intern_type *from_end
		,const intern_type *&from_next
		,extern_type *to_begin
		,extern_type *to_end
		,extern_type *&to_next
		) const
	{
		const intern_type *src = from_begin;
		const intern_type *src_end = from_end;
		intern_type *dst = reinterpret_cast<intern_type*>(to_begin);
		intern_type *dst_end = reinterpret_cast<intern_type*>(to_end);
 
		while (dst+1 <= dst_end && src < src_end)
			*dst++ = *src++;
 
		from_next = src;
		to_next = reinterpret_cast<extern_type*>(dst);
 
	        return ok;
	}
	virtual	result do_unshift(
		state_type& state
		,extern_type *to_begin
		,extern_type *to_end
		,extern_type *&to_next
		) const
	{
	        return noconv;
	}
 
	virtual	int do_length(
		state_type& state
		, const extern_type *from_begin
		,const extern_type *from_end
		,size_t max_internal_chars
		) const throw()
	{
	        return std::min(max_internal_chars,size_t(from_end-from_begin));
	}
 
/* Never converts? Not true for us. */
 
	virtual	bool do_always_noconv() const throw()
	{
	        return false;
	}
 
/* Max extern_type for one intern_type */
 
	virtual	int do_max_length() const throw()
	{
	        return sizeof(intern_type);
	}
 
/* do_encoding returns one of the following:
	-1, if the external representation of a character uses
	a stateful encoding
 
	a constant number representing the maximum width in externT
	elements used to represent a character in a fixed-width encoding
 
	0, if the external representation of the characters in the
	character set uses a variable size encoding
 */
 
	virtual	int do_encoding() const throw()
	{
	        return sizeof(intern_type);
	}
};
} // noconv namespace
 
int main()
{
	try {
		std::locale loc(std::locale::classic(),new noconv::codecvt);
		std::wofstream file;
		file.imbue(loc);
		file.open("four_wchars.txt");
		if (!file) {
			std::cerr << "Error: Unable to create file\n";
			return 1;
		}
 
		//std::wstring wstr(L"abc");
		//file << wstr /*<< std::endl*/;
 
		file <<	L"1234";
 
		file.close();
	}
	catch (std::exception e) {
		std::cerr << "Exception: " << e.what() << std::endl;
	}
 
	try {
		std::locale loc(std::locale::classic(),new noconv::codecvt);
		std::wifstream file;
		file.imbue(loc);
		file.open("four_wchars.txt");
		if (!file) {
			std::cerr << "Error: Unable to open file\n";
			return 2;
		}
 
		std::wstring wstr;
 
		file >> wstr;
 
		file.close();
 
		std::wcout << "String is \"" << wstr << "\"\n";
	}
	catch (std::exception e) {
		std::cerr << "Exception: " << e.what() << std::endl;
	}
 
}

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:
108:
109:
110:
111:
112:
113:
114:
115:
116:
117:
118:
119:
120:
121:
122:
123:
124:
125:
126:
127:
128:
129:
130:
131:
132:
133:
134:
135:
136:
137:
138:
139:
140:
141:
142:
143:
144:
145:
146:
147:
148:

MatrixNZ:
Thank you for your program.  I ran it and it worked!
Just one odd thing:  During the run (twice) a message box containing only ".txt" appeared - with an OK button.  I had to press OK twice.  Then the program ended with correct output.  The actual comand line I issued was:
    convert "Norton Ghost 9.0.log.txt" newNG.txt
Thanks again.