Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Problem with Base64 encoders stripping 0x00 characters?

Posted on 2004-10-14
7
Medium Priority
?
702 Views
Last Modified: 2010-05-18
Hi,

I am using a Base64 encoder to encode a binary message for sending across SOAP. The message being encoded is sent from a mobile device.

The message looks like this:

82000085860088898A

and is made up of Hex values.

When it comes into my Base64 encoder the resulting Base64 String looks like:

ggAAhYYAiImK

and when decoded back out to binary (hex values) it looks like:

8285 8688 898A

As you can see the 0x00 characters are being stripped out of my input String?

I have tried various different Base64 classes and they all do this? This http://iharder.sourceforge.net/base64/ is one of the Base64 implementations I have tried.

Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?

Thanks
Scott

0
Comment
Question by:scurtis_1
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
7 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 12307337
>>
Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?
>>

That should not happen


    byte[] bytes = { 'a', (byte)0, 'c' };
    System.out.println("Before encode:");
    System.out.println(new sun.misc.HexDumpEncoder().encode(bytes));
    String s = new sun.misc.BASE64Encoder().encode(bytes);
    byte[] decoded = new sun.misc.BASE64Decoder().decodeBuffer(s);
    System.out.println("After round-trip Base64 encode/decode");
    System.out.println(new sun.misc.HexDumpEncoder().encode(decoded));
0
 
LVL 13

Expert Comment

by:Webstorm
ID: 12307382
Hi scurtis_1,

the length of the encoded bytes seems ok.

>> I have tried various different Base64 classes and they all do this?
Then, it may come from elsewhere, and not from base64 decoding.

Have you tried all the ones found with google ?
http://www.google.com/search?sourceid=navclient&hl=en&ie=UTF-8&q=Java+fast+Base64
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:scurtis_1
ID: 12307610
I am starting to think it may have something to do with how we get the byte[] to encode? In order to pass 8bit characters round our code as Strings (we had to do this because some characters above ascii 128 were being negated and coming out incorrectly) we used a class called ByteArrayHexStringConverter. This takes the bytes as they come in from the mobile and immediately converts them into a String representation of the Hex values. This hexstring is then converted back to a byte[] just before being encoded to Base64 further down in the system. I think it might be this conversion from the HexString back into the actual hex values byte[] that the problem arises?

I tried that CEHJ and it still did the same thing using the above scenario.

Following is the code used:

public class BytePrinter
{

    public static void printBytes( byte[] bytes )
    {
        System.out.println( hexstring( bytes ) );
    }

    public static String hexstring( byte[] bytes)
    {
          return hexstring(bytes, bytes.length);
    }

    public static String hexstring( byte[] bytes, int length )
    {

        // int max = bytes.length;
        StringBuffer sb = new StringBuffer();

        for ( int i = 0; i < length; i++ )
        {
            int val = ( int ) bytes[i];
            val = val < 0 ? 256 + val : val;

            String tmp = "0" + Integer.toString( val, 16 );
            String tmp2 = tmp.substring( tmp.length() - 2, tmp.length() )
                    .toUpperCase();
            sb.append( tmp2 );

        }
        return sb.toString();
    }

}

public class ByteArrayHexStringConverter extends BytePrinter {
    public static byte[] toByteArray(String hexString) throws OddNumberOfBytesException {
        int stringLength = hexString.length();
        if ((stringLength & 0x1) != 0) {
            throw new OddNumberOfBytesException("toByteArray requires an even number of hex characters");
        }
        byte[] b = new byte[stringLength / 2];
        for (int i = 0, j = 0; i < stringLength; i += 2, j++) {
            int high = charToNibble(hexString.charAt(i));
            int low = charToNibble(hexString.charAt(i + 1));
            b[j] = (byte) ((high << 4) | low);
        }
        return b;

    }

    private static int charToNibble(char c)
    {
        if ('0' <= c && c <= '9')
        {
            return c - '0';
        } else if ('a' <= c && c <= 'f')
        {
            return c - 'a' + 0xa;
        } else if ('A' <= c && c <= 'F')
        {
            return c - 'A' + 0xa;
        } else
        {
            throw new IllegalArgumentException("Invalid hex character: " + c);
        }
    }

    // explicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes, int length) {
        return hexstring(bytes,length);
    }

    // implicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes) {
        return hexstring(bytes);
    }

    public static String hexStringToString(String hexString) throws OddNumberOfBytesException {
        return new String(toByteArray(hexString));
    }
}

This is the main method I use to test when reading bytes in from a file:

public static void main(String[] args) {

        try {

            String hexString = Base64.loadMessage();
            byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);
            String base64 = Base64.encodeBytes(hexBytes);
           
            System.out.println("Base64 string: " + base64);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static String loadMessage() throws FileNotFoundException, IOException, NamedPropertiesNotFoundException {

        System.out.println("loadMessage()");

        DataInputStream in = new DataInputStream(new FileInputStream("c:/binarymessage.txt"));

        StringBuffer sbData = new StringBuffer();
        boolean body = true;
        int val;
        byte[] dataArray = new byte[100];
        while (body) {
            val = in.read(dataArray);
            sbData.append(ByteArrayHexStringConverter.toHexString(dataArray, val));
            if (val < 100 || val == -1)
                body = false;
        }

        return sbData.toString();
    }

The message contents are fine when I display the loaded hexString variable but after calling ByteArrayHexStringConverter.toByteArray(hexString); to get a byte[] of the hex values and encode them the 0x00 characters are removed?

Sorry to post so much code here. Any help would be very welcome?

Thanks
Scott
0
 

Author Comment

by:scurtis_1
ID: 12307766
I have actually just confirmed that it is my call to ByteArrayHexStringConverter.toByteArray(hexString); that is the problem using the following code (thanks to CEHJ for pointing out this HexDumpEncoder):

String hexString = Base64.loadMessage();
byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);

System.out.println(new sun.misc.HexDumpEncoder().encode(hexBytes));

So I suppose the question is actually:

Can anyone spot why my ByteArrayHexStringConverter might be stripping out 0x00 characters?

scott
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 2000 total points
ID: 12307814
No time now - gotta fly. This works:

    public static byte[] hexStringToByteArray(String s) {
      String charIndex = "0123456789abcdef";
      int stringIndex = 0;
      byte[] bytes = new byte[s.length() / 2];
      s = s.toLowerCase();
      for (int i = 0; i < bytes.length; i++) {
        stringIndex = i * 2;
        byte n = (byte) charIndex.indexOf(s.charAt(stringIndex));
        n <<= 4;
        bytes[i] |= n;
        n = (byte) charIndex.indexOf(s.charAt(stringIndex + 1));
        bytes[i] |= n;
      }
      return bytes;
    }

0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12310762
8-)
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Java contains several comparison operators (e.g., <, <=, >, >=, ==, !=) that allow you to compare primitive values. However, these operators cannot be used to compare the contents of objects. Interface Comparable is used to allow objects of a cl…
Java functions are among the best things for programmers to work with as Java sites can be very easy to read and prepare. Java especially simplifies many processes in the coding industry as it helps integrate many forms of technology and different d…
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
Suggested Courses

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question