Solved

Problem with Base64 encoders stripping 0x00 characters?

Posted on 2004-10-14
7
647 Views
Last Modified: 2010-05-18
Hi,

I am using a Base64 encoder to encode a binary message for sending across SOAP. The message being encoded is sent from a mobile device.

The message looks like this:

82000085860088898A

and is made up of Hex values.

When it comes into my Base64 encoder the resulting Base64 String looks like:

ggAAhYYAiImK

and when decoded back out to binary (hex values) it looks like:

8285 8688 898A

As you can see the 0x00 characters are being stripped out of my input String?

I have tried various different Base64 classes and they all do this? This http://iharder.sourceforge.net/base64/ is one of the Base64 implementations I have tried.

Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?

Thanks
Scott

0
Comment
Question by:scurtis_1
  • 3
  • 2
  • 2
7 Comments
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
>>
Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?
>>

That should not happen


    byte[] bytes = { 'a', (byte)0, 'c' };
    System.out.println("Before encode:");
    System.out.println(new sun.misc.HexDumpEncoder().encode(bytes));
    String s = new sun.misc.BASE64Encoder().encode(bytes);
    byte[] decoded = new sun.misc.BASE64Decoder().decodeBuffer(s);
    System.out.println("After round-trip Base64 encode/decode");
    System.out.println(new sun.misc.HexDumpEncoder().encode(decoded));
0
 
LVL 13

Expert Comment

by:Webstorm
Comment Utility
Hi scurtis_1,

the length of the encoded bytes seems ok.

>> I have tried various different Base64 classes and they all do this?
Then, it may come from elsewhere, and not from base64 decoding.

Have you tried all the ones found with google ?
http://www.google.com/search?sourceid=navclient&hl=en&ie=UTF-8&q=Java+fast+Base64
0
 
LVL 13

Expert Comment

by:Webstorm
Comment Utility
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 

Author Comment

by:scurtis_1
Comment Utility
I am starting to think it may have something to do with how we get the byte[] to encode? In order to pass 8bit characters round our code as Strings (we had to do this because some characters above ascii 128 were being negated and coming out incorrectly) we used a class called ByteArrayHexStringConverter. This takes the bytes as they come in from the mobile and immediately converts them into a String representation of the Hex values. This hexstring is then converted back to a byte[] just before being encoded to Base64 further down in the system. I think it might be this conversion from the HexString back into the actual hex values byte[] that the problem arises?

I tried that CEHJ and it still did the same thing using the above scenario.

Following is the code used:

public class BytePrinter
{

    public static void printBytes( byte[] bytes )
    {
        System.out.println( hexstring( bytes ) );
    }

    public static String hexstring( byte[] bytes)
    {
          return hexstring(bytes, bytes.length);
    }

    public static String hexstring( byte[] bytes, int length )
    {

        // int max = bytes.length;
        StringBuffer sb = new StringBuffer();

        for ( int i = 0; i < length; i++ )
        {
            int val = ( int ) bytes[i];
            val = val < 0 ? 256 + val : val;

            String tmp = "0" + Integer.toString( val, 16 );
            String tmp2 = tmp.substring( tmp.length() - 2, tmp.length() )
                    .toUpperCase();
            sb.append( tmp2 );

        }
        return sb.toString();
    }

}

public class ByteArrayHexStringConverter extends BytePrinter {
    public static byte[] toByteArray(String hexString) throws OddNumberOfBytesException {
        int stringLength = hexString.length();
        if ((stringLength & 0x1) != 0) {
            throw new OddNumberOfBytesException("toByteArray requires an even number of hex characters");
        }
        byte[] b = new byte[stringLength / 2];
        for (int i = 0, j = 0; i < stringLength; i += 2, j++) {
            int high = charToNibble(hexString.charAt(i));
            int low = charToNibble(hexString.charAt(i + 1));
            b[j] = (byte) ((high << 4) | low);
        }
        return b;

    }

    private static int charToNibble(char c)
    {
        if ('0' <= c && c <= '9')
        {
            return c - '0';
        } else if ('a' <= c && c <= 'f')
        {
            return c - 'a' + 0xa;
        } else if ('A' <= c && c <= 'F')
        {
            return c - 'A' + 0xa;
        } else
        {
            throw new IllegalArgumentException("Invalid hex character: " + c);
        }
    }

    // explicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes, int length) {
        return hexstring(bytes,length);
    }

    // implicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes) {
        return hexstring(bytes);
    }

    public static String hexStringToString(String hexString) throws OddNumberOfBytesException {
        return new String(toByteArray(hexString));
    }
}

This is the main method I use to test when reading bytes in from a file:

public static void main(String[] args) {

        try {

            String hexString = Base64.loadMessage();
            byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);
            String base64 = Base64.encodeBytes(hexBytes);
           
            System.out.println("Base64 string: " + base64);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static String loadMessage() throws FileNotFoundException, IOException, NamedPropertiesNotFoundException {

        System.out.println("loadMessage()");

        DataInputStream in = new DataInputStream(new FileInputStream("c:/binarymessage.txt"));

        StringBuffer sbData = new StringBuffer();
        boolean body = true;
        int val;
        byte[] dataArray = new byte[100];
        while (body) {
            val = in.read(dataArray);
            sbData.append(ByteArrayHexStringConverter.toHexString(dataArray, val));
            if (val < 100 || val == -1)
                body = false;
        }

        return sbData.toString();
    }

The message contents are fine when I display the loaded hexString variable but after calling ByteArrayHexStringConverter.toByteArray(hexString); to get a byte[] of the hex values and encode them the 0x00 characters are removed?

Sorry to post so much code here. Any help would be very welcome?

Thanks
Scott
0
 

Author Comment

by:scurtis_1
Comment Utility
I have actually just confirmed that it is my call to ByteArrayHexStringConverter.toByteArray(hexString); that is the problem using the following code (thanks to CEHJ for pointing out this HexDumpEncoder):

String hexString = Base64.loadMessage();
byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);

System.out.println(new sun.misc.HexDumpEncoder().encode(hexBytes));

So I suppose the question is actually:

Can anyone spot why my ByteArrayHexStringConverter might be stripping out 0x00 characters?

scott
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 500 total points
Comment Utility
No time now - gotta fly. This works:

    public static byte[] hexStringToByteArray(String s) {
      String charIndex = "0123456789abcdef";
      int stringIndex = 0;
      byte[] bytes = new byte[s.length() / 2];
      s = s.toLowerCase();
      for (int i = 0; i < bytes.length; i++) {
        stringIndex = i * 2;
        byte n = (byte) charIndex.indexOf(s.charAt(stringIndex));
        n <<= 4;
        bytes[i] |= n;
        n = (byte) charIndex.indexOf(s.charAt(stringIndex + 1));
        bytes[i] |= n;
      }
      return bytes;
    }

0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
8-)
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Suggested Solutions

Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Viewers learn about the third conditional statement “else if” and use it in an example program. Then additional information about conditional statements is provided, covering the topic thoroughly. Viewers learn about the third conditional statement …
This tutorial will introduce the viewer to VisualVM for the Java platform application. This video explains an example program and covers the Overview, Monitor, and Heap Dump tabs.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now