Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 730
  • Last Modified:

Problem with Base64 encoders stripping 0x00 characters?

Hi,

I am using a Base64 encoder to encode a binary message for sending across SOAP. The message being encoded is sent from a mobile device.

The message looks like this:

82000085860088898A

and is made up of Hex values.

When it comes into my Base64 encoder the resulting Base64 String looks like:

ggAAhYYAiImK

and when decoded back out to binary (hex values) it looks like:

8285 8688 898A

As you can see the 0x00 characters are being stripped out of my input String?

I have tried various different Base64 classes and they all do this? This http://iharder.sourceforge.net/base64/ is one of the Base64 implementations I have tried.

Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?

Thanks
Scott

0
scurtis_1
Asked:
scurtis_1
  • 3
  • 2
  • 2
1 Solution
 
CEHJCommented:
>>
Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?
>>

That should not happen


    byte[] bytes = { 'a', (byte)0, 'c' };
    System.out.println("Before encode:");
    System.out.println(new sun.misc.HexDumpEncoder().encode(bytes));
    String s = new sun.misc.BASE64Encoder().encode(bytes);
    byte[] decoded = new sun.misc.BASE64Decoder().decodeBuffer(s);
    System.out.println("After round-trip Base64 encode/decode");
    System.out.println(new sun.misc.HexDumpEncoder().encode(decoded));
0
 
WebstormCommented:
Hi scurtis_1,

the length of the encoded bytes seems ok.

>> I have tried various different Base64 classes and they all do this?
Then, it may come from elsewhere, and not from base64 decoding.

Have you tried all the ones found with google ?
http://www.google.com/search?sourceid=navclient&hl=en&ie=UTF-8&q=Java+fast+Base64
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
scurtis_1Author Commented:
I am starting to think it may have something to do with how we get the byte[] to encode? In order to pass 8bit characters round our code as Strings (we had to do this because some characters above ascii 128 were being negated and coming out incorrectly) we used a class called ByteArrayHexStringConverter. This takes the bytes as they come in from the mobile and immediately converts them into a String representation of the Hex values. This hexstring is then converted back to a byte[] just before being encoded to Base64 further down in the system. I think it might be this conversion from the HexString back into the actual hex values byte[] that the problem arises?

I tried that CEHJ and it still did the same thing using the above scenario.

Following is the code used:

public class BytePrinter
{

    public static void printBytes( byte[] bytes )
    {
        System.out.println( hexstring( bytes ) );
    }

    public static String hexstring( byte[] bytes)
    {
          return hexstring(bytes, bytes.length);
    }

    public static String hexstring( byte[] bytes, int length )
    {

        // int max = bytes.length;
        StringBuffer sb = new StringBuffer();

        for ( int i = 0; i < length; i++ )
        {
            int val = ( int ) bytes[i];
            val = val < 0 ? 256 + val : val;

            String tmp = "0" + Integer.toString( val, 16 );
            String tmp2 = tmp.substring( tmp.length() - 2, tmp.length() )
                    .toUpperCase();
            sb.append( tmp2 );

        }
        return sb.toString();
    }

}

public class ByteArrayHexStringConverter extends BytePrinter {
    public static byte[] toByteArray(String hexString) throws OddNumberOfBytesException {
        int stringLength = hexString.length();
        if ((stringLength & 0x1) != 0) {
            throw new OddNumberOfBytesException("toByteArray requires an even number of hex characters");
        }
        byte[] b = new byte[stringLength / 2];
        for (int i = 0, j = 0; i < stringLength; i += 2, j++) {
            int high = charToNibble(hexString.charAt(i));
            int low = charToNibble(hexString.charAt(i + 1));
            b[j] = (byte) ((high << 4) | low);
        }
        return b;

    }

    private static int charToNibble(char c)
    {
        if ('0' <= c && c <= '9')
        {
            return c - '0';
        } else if ('a' <= c && c <= 'f')
        {
            return c - 'a' + 0xa;
        } else if ('A' <= c && c <= 'F')
        {
            return c - 'A' + 0xa;
        } else
        {
            throw new IllegalArgumentException("Invalid hex character: " + c);
        }
    }

    // explicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes, int length) {
        return hexstring(bytes,length);
    }

    // implicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes) {
        return hexstring(bytes);
    }

    public static String hexStringToString(String hexString) throws OddNumberOfBytesException {
        return new String(toByteArray(hexString));
    }
}

This is the main method I use to test when reading bytes in from a file:

public static void main(String[] args) {

        try {

            String hexString = Base64.loadMessage();
            byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);
            String base64 = Base64.encodeBytes(hexBytes);
           
            System.out.println("Base64 string: " + base64);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static String loadMessage() throws FileNotFoundException, IOException, NamedPropertiesNotFoundException {

        System.out.println("loadMessage()");

        DataInputStream in = new DataInputStream(new FileInputStream("c:/binarymessage.txt"));

        StringBuffer sbData = new StringBuffer();
        boolean body = true;
        int val;
        byte[] dataArray = new byte[100];
        while (body) {
            val = in.read(dataArray);
            sbData.append(ByteArrayHexStringConverter.toHexString(dataArray, val));
            if (val < 100 || val == -1)
                body = false;
        }

        return sbData.toString();
    }

The message contents are fine when I display the loaded hexString variable but after calling ByteArrayHexStringConverter.toByteArray(hexString); to get a byte[] of the hex values and encode them the 0x00 characters are removed?

Sorry to post so much code here. Any help would be very welcome?

Thanks
Scott
0
 
scurtis_1Author Commented:
I have actually just confirmed that it is my call to ByteArrayHexStringConverter.toByteArray(hexString); that is the problem using the following code (thanks to CEHJ for pointing out this HexDumpEncoder):

String hexString = Base64.loadMessage();
byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);

System.out.println(new sun.misc.HexDumpEncoder().encode(hexBytes));

So I suppose the question is actually:

Can anyone spot why my ByteArrayHexStringConverter might be stripping out 0x00 characters?

scott
0
 
CEHJCommented:
No time now - gotta fly. This works:

    public static byte[] hexStringToByteArray(String s) {
      String charIndex = "0123456789abcdef";
      int stringIndex = 0;
      byte[] bytes = new byte[s.length() / 2];
      s = s.toLowerCase();
      for (int i = 0; i < bytes.length; i++) {
        stringIndex = i * 2;
        byte n = (byte) charIndex.indexOf(s.charAt(stringIndex));
        n <<= 4;
        bytes[i] |= n;
        n = (byte) charIndex.indexOf(s.charAt(stringIndex + 1));
        bytes[i] |= n;
      }
      return bytes;
    }

0
 
CEHJCommented:
8-)
0

Featured Post

Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

  • 3
  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now