Solved

Problem with Base64 encoders stripping 0x00 characters?

Posted on 2004-10-14
7
672 Views
Last Modified: 2010-05-18
Hi,

I am using a Base64 encoder to encode a binary message for sending across SOAP. The message being encoded is sent from a mobile device.

The message looks like this:

82000085860088898A

and is made up of Hex values.

When it comes into my Base64 encoder the resulting Base64 String looks like:

ggAAhYYAiImK

and when decoded back out to binary (hex values) it looks like:

8285 8688 898A

As you can see the 0x00 characters are being stripped out of my input String?

I have tried various different Base64 classes and they all do this? This http://iharder.sourceforge.net/base64/ is one of the Base64 implementations I have tried.

Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?

Thanks
Scott

0
Comment
Question by:scurtis_1
  • 3
  • 2
  • 2
7 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 12307337
>>
Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?
>>

That should not happen


    byte[] bytes = { 'a', (byte)0, 'c' };
    System.out.println("Before encode:");
    System.out.println(new sun.misc.HexDumpEncoder().encode(bytes));
    String s = new sun.misc.BASE64Encoder().encode(bytes);
    byte[] decoded = new sun.misc.BASE64Decoder().decodeBuffer(s);
    System.out.println("After round-trip Base64 encode/decode");
    System.out.println(new sun.misc.HexDumpEncoder().encode(decoded));
0
 
LVL 13

Expert Comment

by:Webstorm
ID: 12307382
Hi scurtis_1,

the length of the encoded bytes seems ok.

>> I have tried various different Base64 classes and they all do this?
Then, it may come from elsewhere, and not from base64 decoding.

Have you tried all the ones found with google ?
http://www.google.com/search?sourceid=navclient&hl=en&ie=UTF-8&q=Java+fast+Base64
0
 
LVL 13

Expert Comment

by:Webstorm
ID: 12307455
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:scurtis_1
ID: 12307610
I am starting to think it may have something to do with how we get the byte[] to encode? In order to pass 8bit characters round our code as Strings (we had to do this because some characters above ascii 128 were being negated and coming out incorrectly) we used a class called ByteArrayHexStringConverter. This takes the bytes as they come in from the mobile and immediately converts them into a String representation of the Hex values. This hexstring is then converted back to a byte[] just before being encoded to Base64 further down in the system. I think it might be this conversion from the HexString back into the actual hex values byte[] that the problem arises?

I tried that CEHJ and it still did the same thing using the above scenario.

Following is the code used:

public class BytePrinter
{

    public static void printBytes( byte[] bytes )
    {
        System.out.println( hexstring( bytes ) );
    }

    public static String hexstring( byte[] bytes)
    {
          return hexstring(bytes, bytes.length);
    }

    public static String hexstring( byte[] bytes, int length )
    {

        // int max = bytes.length;
        StringBuffer sb = new StringBuffer();

        for ( int i = 0; i < length; i++ )
        {
            int val = ( int ) bytes[i];
            val = val < 0 ? 256 + val : val;

            String tmp = "0" + Integer.toString( val, 16 );
            String tmp2 = tmp.substring( tmp.length() - 2, tmp.length() )
                    .toUpperCase();
            sb.append( tmp2 );

        }
        return sb.toString();
    }

}

public class ByteArrayHexStringConverter extends BytePrinter {
    public static byte[] toByteArray(String hexString) throws OddNumberOfBytesException {
        int stringLength = hexString.length();
        if ((stringLength & 0x1) != 0) {
            throw new OddNumberOfBytesException("toByteArray requires an even number of hex characters");
        }
        byte[] b = new byte[stringLength / 2];
        for (int i = 0, j = 0; i < stringLength; i += 2, j++) {
            int high = charToNibble(hexString.charAt(i));
            int low = charToNibble(hexString.charAt(i + 1));
            b[j] = (byte) ((high << 4) | low);
        }
        return b;

    }

    private static int charToNibble(char c)
    {
        if ('0' <= c && c <= '9')
        {
            return c - '0';
        } else if ('a' <= c && c <= 'f')
        {
            return c - 'a' + 0xa;
        } else if ('A' <= c && c <= 'F')
        {
            return c - 'A' + 0xa;
        } else
        {
            throw new IllegalArgumentException("Invalid hex character: " + c);
        }
    }

    // explicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes, int length) {
        return hexstring(bytes,length);
    }

    // implicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes) {
        return hexstring(bytes);
    }

    public static String hexStringToString(String hexString) throws OddNumberOfBytesException {
        return new String(toByteArray(hexString));
    }
}

This is the main method I use to test when reading bytes in from a file:

public static void main(String[] args) {

        try {

            String hexString = Base64.loadMessage();
            byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);
            String base64 = Base64.encodeBytes(hexBytes);
           
            System.out.println("Base64 string: " + base64);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static String loadMessage() throws FileNotFoundException, IOException, NamedPropertiesNotFoundException {

        System.out.println("loadMessage()");

        DataInputStream in = new DataInputStream(new FileInputStream("c:/binarymessage.txt"));

        StringBuffer sbData = new StringBuffer();
        boolean body = true;
        int val;
        byte[] dataArray = new byte[100];
        while (body) {
            val = in.read(dataArray);
            sbData.append(ByteArrayHexStringConverter.toHexString(dataArray, val));
            if (val < 100 || val == -1)
                body = false;
        }

        return sbData.toString();
    }

The message contents are fine when I display the loaded hexString variable but after calling ByteArrayHexStringConverter.toByteArray(hexString); to get a byte[] of the hex values and encode them the 0x00 characters are removed?

Sorry to post so much code here. Any help would be very welcome?

Thanks
Scott
0
 

Author Comment

by:scurtis_1
ID: 12307766
I have actually just confirmed that it is my call to ByteArrayHexStringConverter.toByteArray(hexString); that is the problem using the following code (thanks to CEHJ for pointing out this HexDumpEncoder):

String hexString = Base64.loadMessage();
byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);

System.out.println(new sun.misc.HexDumpEncoder().encode(hexBytes));

So I suppose the question is actually:

Can anyone spot why my ByteArrayHexStringConverter might be stripping out 0x00 characters?

scott
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 500 total points
ID: 12307814
No time now - gotta fly. This works:

    public static byte[] hexStringToByteArray(String s) {
      String charIndex = "0123456789abcdef";
      int stringIndex = 0;
      byte[] bytes = new byte[s.length() / 2];
      s = s.toLowerCase();
      for (int i = 0; i < bytes.length; i++) {
        stringIndex = i * 2;
        byte n = (byte) charIndex.indexOf(s.charAt(stringIndex));
        n <<= 4;
        bytes[i] |= n;
        n = (byte) charIndex.indexOf(s.charAt(stringIndex + 1));
        bytes[i] |= n;
      }
      return bytes;
    }

0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12310762
8-)
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Weekend adv creator 3 64
expectj telnet failing 5 49
How to configure empty element in XML Document parser? 15 44
batch vs regular insert in spring DAO 2 20
Java Flight Recorder and Java Mission Control together create a complete tool chain to continuously collect low level and detailed runtime information enabling after-the-fact incident analysis. Java Flight Recorder is a profiling and event collectio…
Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Viewers will learn about if statements in Java and their use The if statement: The condition required to create an if statement: Variations of if statements: An example using if statements:
The viewer will learn how to implement Singleton Design Pattern in Java.

685 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question