Solved

Problem with Base64 encoders stripping 0x00 characters?

Posted on 2004-10-14
7
651 Views
Last Modified: 2010-05-18
Hi,

I am using a Base64 encoder to encode a binary message for sending across SOAP. The message being encoded is sent from a mobile device.

The message looks like this:

82000085860088898A

and is made up of Hex values.

When it comes into my Base64 encoder the resulting Base64 String looks like:

ggAAhYYAiImK

and when decoded back out to binary (hex values) it looks like:

8285 8688 898A

As you can see the 0x00 characters are being stripped out of my input String?

I have tried various different Base64 classes and they all do this? This http://iharder.sourceforge.net/base64/ is one of the Base64 implementations I have tried.

Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?

Thanks
Scott

0
Comment
Question by:scurtis_1
  • 3
  • 2
  • 2
7 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 12307337
>>
Can anyone spot why it and seemingly all other Base64 encoders might be removing 0x00 hex values from the string?
>>

That should not happen


    byte[] bytes = { 'a', (byte)0, 'c' };
    System.out.println("Before encode:");
    System.out.println(new sun.misc.HexDumpEncoder().encode(bytes));
    String s = new sun.misc.BASE64Encoder().encode(bytes);
    byte[] decoded = new sun.misc.BASE64Decoder().decodeBuffer(s);
    System.out.println("After round-trip Base64 encode/decode");
    System.out.println(new sun.misc.HexDumpEncoder().encode(decoded));
0
 
LVL 13

Expert Comment

by:Webstorm
ID: 12307382
Hi scurtis_1,

the length of the encoded bytes seems ok.

>> I have tried various different Base64 classes and they all do this?
Then, it may come from elsewhere, and not from base64 decoding.

Have you tried all the ones found with google ?
http://www.google.com/search?sourceid=navclient&hl=en&ie=UTF-8&q=Java+fast+Base64
0
 
LVL 13

Expert Comment

by:Webstorm
ID: 12307455
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:scurtis_1
ID: 12307610
I am starting to think it may have something to do with how we get the byte[] to encode? In order to pass 8bit characters round our code as Strings (we had to do this because some characters above ascii 128 were being negated and coming out incorrectly) we used a class called ByteArrayHexStringConverter. This takes the bytes as they come in from the mobile and immediately converts them into a String representation of the Hex values. This hexstring is then converted back to a byte[] just before being encoded to Base64 further down in the system. I think it might be this conversion from the HexString back into the actual hex values byte[] that the problem arises?

I tried that CEHJ and it still did the same thing using the above scenario.

Following is the code used:

public class BytePrinter
{

    public static void printBytes( byte[] bytes )
    {
        System.out.println( hexstring( bytes ) );
    }

    public static String hexstring( byte[] bytes)
    {
          return hexstring(bytes, bytes.length);
    }

    public static String hexstring( byte[] bytes, int length )
    {

        // int max = bytes.length;
        StringBuffer sb = new StringBuffer();

        for ( int i = 0; i < length; i++ )
        {
            int val = ( int ) bytes[i];
            val = val < 0 ? 256 + val : val;

            String tmp = "0" + Integer.toString( val, 16 );
            String tmp2 = tmp.substring( tmp.length() - 2, tmp.length() )
                    .toUpperCase();
            sb.append( tmp2 );

        }
        return sb.toString();
    }

}

public class ByteArrayHexStringConverter extends BytePrinter {
    public static byte[] toByteArray(String hexString) throws OddNumberOfBytesException {
        int stringLength = hexString.length();
        if ((stringLength & 0x1) != 0) {
            throw new OddNumberOfBytesException("toByteArray requires an even number of hex characters");
        }
        byte[] b = new byte[stringLength / 2];
        for (int i = 0, j = 0; i < stringLength; i += 2, j++) {
            int high = charToNibble(hexString.charAt(i));
            int low = charToNibble(hexString.charAt(i + 1));
            b[j] = (byte) ((high << 4) | low);
        }
        return b;

    }

    private static int charToNibble(char c)
    {
        if ('0' <= c && c <= '9')
        {
            return c - '0';
        } else if ('a' <= c && c <= 'f')
        {
            return c - 'a' + 0xa;
        } else if ('A' <= c && c <= 'F')
        {
            return c - 'A' + 0xa;
        } else
        {
            throw new IllegalArgumentException("Invalid hex character: " + c);
        }
    }

    // explicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes, int length) {
        return hexstring(bytes,length);
    }

    // implicitly tell the BytePrinter how many characters to convert
    public static String toHexString(byte[] bytes) {
        return hexstring(bytes);
    }

    public static String hexStringToString(String hexString) throws OddNumberOfBytesException {
        return new String(toByteArray(hexString));
    }
}

This is the main method I use to test when reading bytes in from a file:

public static void main(String[] args) {

        try {

            String hexString = Base64.loadMessage();
            byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);
            String base64 = Base64.encodeBytes(hexBytes);
           
            System.out.println("Base64 string: " + base64);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static String loadMessage() throws FileNotFoundException, IOException, NamedPropertiesNotFoundException {

        System.out.println("loadMessage()");

        DataInputStream in = new DataInputStream(new FileInputStream("c:/binarymessage.txt"));

        StringBuffer sbData = new StringBuffer();
        boolean body = true;
        int val;
        byte[] dataArray = new byte[100];
        while (body) {
            val = in.read(dataArray);
            sbData.append(ByteArrayHexStringConverter.toHexString(dataArray, val));
            if (val < 100 || val == -1)
                body = false;
        }

        return sbData.toString();
    }

The message contents are fine when I display the loaded hexString variable but after calling ByteArrayHexStringConverter.toByteArray(hexString); to get a byte[] of the hex values and encode them the 0x00 characters are removed?

Sorry to post so much code here. Any help would be very welcome?

Thanks
Scott
0
 

Author Comment

by:scurtis_1
ID: 12307766
I have actually just confirmed that it is my call to ByteArrayHexStringConverter.toByteArray(hexString); that is the problem using the following code (thanks to CEHJ for pointing out this HexDumpEncoder):

String hexString = Base64.loadMessage();
byte[] hexBytes = ByteArrayHexStringConverter.toByteArray(hexString);

System.out.println(new sun.misc.HexDumpEncoder().encode(hexBytes));

So I suppose the question is actually:

Can anyone spot why my ByteArrayHexStringConverter might be stripping out 0x00 characters?

scott
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 500 total points
ID: 12307814
No time now - gotta fly. This works:

    public static byte[] hexStringToByteArray(String s) {
      String charIndex = "0123456789abcdef";
      int stringIndex = 0;
      byte[] bytes = new byte[s.length() / 2];
      s = s.toLowerCase();
      for (int i = 0; i < bytes.length; i++) {
        stringIndex = i * 2;
        byte n = (byte) charIndex.indexOf(s.charAt(stringIndex));
        n <<= 4;
        bytes[i] |= n;
        n = (byte) charIndex.indexOf(s.charAt(stringIndex + 1));
        bytes[i] |= n;
      }
      return bytes;
    }

0
 
LVL 86

Expert Comment

by:CEHJ
ID: 12310762
8-)
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Cipher Configuration on Apache HTTPD 4 53
Error trying to install RTMT Win7 5 41
windows explorer path to command prompt 5 33
servlet example 17 10
Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
The viewer will learn how to implement Singleton Design Pattern in Java.
This video teaches viewers about errors in exception handling.

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

24 Experts available now in Live!

Get 1:1 Help Now