Convert files in any unicode encoding type to UTF-8 in Java

thanks objects..

this is what im doing:

//-----------------------------------------------------------------------------------------------------
BufferedReader read = new BufferedReader(new InputStreamReader(myFile.getInputStream(), getEncodingTypeCode("utf16")));

ByteArrayOutputStream temp = new ByteArrayOutputStream();
BufferedWriter write = new BufferedWriter(new OutputStreamWriter(temp, "UTF8"));
//-----------------------------------------------------------------------------------------------------




is there a way to convert "any" unicode files to UTF8 without having  to know before hand what type of encoding type the file is?

currently i just check the extension names, something like:

//-----------------------------------------------------------------------------------------------------
private static String getEncodingTypeCode (String extension_name){
      String encoding_type = "";

      if(extension_name.equalsIgnoreCase("utf8"))
      {                                          
            return "UTF8";
      }
      if(extension_name.equalsIgnoreCase("utf-8"))
      {                                          
            return "UTF8";
      }
      if(extension_name.equalsIgnoreCase("utf16"))
      {                                          
            return "UTF-16";
      }
      if(extension_name.equalsIgnoreCase("utf-16"))
      {                                          
            return "UTF-16";
      }
      if(extension_name.equalsIgnoreCase("utf-16le"))
      {                                          
            return "UnicodeLittle";
      }
      if(extension_name.equalsIgnoreCase("utf-16be"))
      {                                          
            return "UnicodeBig";
      }
}
//-----------------------------------------------------------------------------------------------------


but that "if" condition can go on and on...i still need to add ucs, ucs2, ucs-2l2, Cp037, Cp273 etc (basically everything here: http://java.sun.com/j2se/1.3/docs/guide/intl/encoding.doc.html)....

and aside from those "if" conditions, if it's a .txt file i also need to check the byte order marks to see if it's just a plain .txt file or if its a unicode...

so i have something like:

//-----------------------------------------------------------------------------------------------------
      if(first_byte.equalsIgnoreCase("FE") && sec_byte.equalsIgnoreCase("FF")){
            return sEncodingType = "UnicodeBig";//java.sun.com/j2se/1.3/docs/guide/intl/encoding.doc.html
      }
      else if (first_byte.equalsIgnoreCase("FF") && sec_byte.equalsIgnoreCase("FE")){
            return sEncodingType = "UnicodeLittle";
      }
      else if (first_byte.equalsIgnoreCase("EF") && sec_byte.equalsIgnoreCase("BB") && third_byte.equalsIgnoreCase("BF")){
            return sEncodingType = "UTF8";
      }
//-----------------------------------------------------------------------------------------------------
      

that would work fine but a problem would arise when the .txt file has unicode characters but its unmarked (no byte order marks)...
and ofcourse i also still need to check the BOM (byte order marks) for all the other extension names so i would know what encoding type to pass in my BufferedReader.

i'm curious if there's a shorter way to convert any files to UTF8 so i dont have to go through all this checking anymore and  just convert whatever file, unmarked or not to UTF8 w/o having to check first what type of encoding the file has..

not really, you need to know the encoding to be able to tell the Reader what to use

i see.... oh well, i guess i dont have a choice but to put each one of those encoding types...

anyway, i tried your solution but i noticed the BOM (byte order marks) are still there after i convert a file to UTF8. is there a way to remove the BOM before converting it to UTF8?

thanks again!

you need to remove them manually

yeah, i figured there's no shortcut to that too....

for the sake of those reading this here's how i removed the BOM...

//encoding_type = can be UTF8, UnicodeBig, UnicodeLil, etc
BufferedReader read = new BufferedReader(new InputStreamReader(file.getInputStream(), encoding_type));
ByteArrayOutputStream temp = new ByteArrayOutputStream();
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(temp, "UTF8"));

line = read.readLine();

//offset = 1 for BE/LE and 2 for UTF8
writer.write(line, offset, line.length()-offset);

solution is excellent but kinda vague that i had to do additional research and had to do some code experiments  to fill up some missing parts of the solution...