Handling Binary Data in Coldfusion

Hello,

I am using an API from a site on the web and I am having trouble figuring out how to deal with the data once it has been retrieved.  In other words, it comes back in (what I assume to be) binary data and I want to convert it into something usable.  Any help would be tremendously appreciated!!!!!!!!!!!!!!!!!!!!!!!!!!!

This is what I am doing:

1.  I am sending an <cfhttp> request to the web service with the proper parameters using <cfhttpparam>
2. The data that is returned (specifically the .FileContent), is in some funky binary or character encoding.  Here's an example:

AAPL  C9aHC9¿C9L¿C9Y¿E¿f $ J¿¿C9 ¿C9w¿C9 ¿C9\)Db¿ $ K¿ C8¿{C9 {C8¿¿C9 \Dm¿f             .......... etc.

I don't have a clue what that is........ The API's documentation says this: [the]  response is in binary format.

3. I've used binarydecode(), but that doesn't return anything.  I've also used CharsetDecode() with the encoding set to "us-ascii" and that returns a long string of numbers.

What I do after this is a mystery to me.  How do I convert that pile of junk you see above into a usable format???

Thank you!!!!!!!!!!!!!!!!!!!!
stracqanAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

_agx_Commented:
It's hard to say without seeing a sample, or more of the api..  But if it's really binary, you're definitely looking in the right area as far as functions go. What kind of binary data is the api returning, an image, text...? If it's text, what does the api say about encoding ?

0
stracqanAuthor Commented:
Thanks for your fast responce!!!!!!!

The data being returned is supposed to be text.  In the documentation, it looks like its HEX?  Also, the only mention of encoding is in the URL, and that more for like if you have a parameter that you want to pass that has a space in it.

This is what they say should be the responce.....

00 00 00 01 00 04 41 4D 54 44 00 00 00 00 02 41
97 33 33 41 99 5C 29 41 90 3D 71 41 91 D7 0A 47
0F C6 14 00 00 01 16 6A E0 68 80 41 93 B4 05 41
97 1E B8 41 90 7A E1 41 96 8F 57 46 E6 2E 80 00
00 01 16 7A 53 7C 80 FF FF

and this should somehow mean this: (The data coppied a little funky, so there are basically three columns: Data, Type and Description.  That is how each block is formatted... sorry about that)

DATA
 TYPE
 DESCRIPTION
 
00 00 00 01
 4 bytes
 Symbol Count =1
 
00 04
 2 bytes
 Symbol Length = 4
 
41 4D 54 44
 4 bytes
 Symbol = AAPL
 
00
 1 byte
 Error code = 0 (OK)
 
00 00 00 02
 4 bytes
 Bar Count =  2
 
41 97 33 33
 4 bytes
 Close = 18.90
 
41 99 5C 29
 4 bytes
 High = 19.17
 
41 90 3D 71
 4 bytes
 Low = 18.03
 
41 91 D7 0A
 4 bytes
 Open = 18.23
 
47 0F C6 14
 4 bytes
 Volume = 3,680,608
 
00 00 01 16 6A E0 68 80
 8 bytes
 Timestamp = November 23,2007

41 93 B4 05
 4 bytes
 Close = 18.4629
 
41 97 1E B8
 4 bytes
 High = 18.89
 
41 90 7A E1
 4 bytes
 Low = 18.06
 
41 96 8F 57
 4 bytes
 Open = 18.82
 
46 E6 2E 80
 4 bytes
 Volume = 2,946,325
 
00 00 01 16 7A 53 7C 80
 8 bytes
 Timestamp = November 26,2007

FF FF
 2 bytes
 

----------------------------------
--------------------------------
-------------------------------
 
How they get from that paragraph to usuable data is where I dont't have a clue.  However, even getting to that first paragraph is a challenege because binarydecode() isn't showing me anything.........



0
stracqanAuthor Commented:
and characterdecode() returns just a long string of ASCII numbers...
0
Cloud Class® Course: Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

stracqanAuthor Commented:
I meant charsetdecode() ... my bad
0
_agx_Commented:
> The data being returned is supposed to be text.  In the documentation, it looks like its HEX?

That does look like hex. So when you say you tried binarydecode(...), you were using "hex" for the second parameter?
0
_agx_Commented:
> you tried binarydecode(...),

BTW: Did you mean to say BinaryEncode() ?
0
stracqanAuthor Commented:
I've been using decode, which is probably why it's been coming back blank...

I tried binaryencode() with "HEX" as the second object, but I got an error:

Parameter 1 of the BinaryEncode function, which is now  AAPL  C9aHC9¿ C9L¿C9Y¿E ¿f  ......................., must be a valid binary object.

So does that mean they are not actually sending the data in binary format?
0
_agx_Commented:
Not necessarily... But you really need to find out a little more about exactly what formats they're using and how developer's are expected to decode the information.  Otherwise, you'll probably waste a lot of time guessing. (That's is all I can do right now without access to the api)  Once you know the format(s) they're using, it will be a lot easier to find the correct CF functions to use.
0
stracqanAuthor Commented:
Sounds good! I'll contact them and see what they say .... Thank you!!!!
0
_agx_Commented:
Sounds like a plan.  I'll be heading out soon. But will check back tomorrow.  
0
_agx_Commented:
> <cfhttp> request

Total shot in the dark ...  but if you're not already, you might try using getasbinary="true". See if it has an effect either way..
0
stracqanAuthor Commented:
THAT WORKED!!!!!!!!!!!! binaryencode() worked perfectly then!!!!!!!

Now all I have to do is convert the HEX values into something workable....

wow........ very impressed...

Thank you!!!!!! Enjoy your evening!!!
0
_agx_Commented:
Great! I guess that old saying is true - always go with your instincts :)
0
stracqanAuthor Commented:
ok....... sooooo.... the API team said that they don't know coldfusion so they can not provide guidance... All they said is that it comes in binary format and from there I need to convert it to something useful for Coldfusion to understand..... Any ideas on how to convert binary and/or hex into something useful?

Thanks again!!!!
0
_agx_Commented:
It's certainly no problem breaking up the binary into bytes (if needed), or hex to plain string.  cflib.org has lots of functions for hex, converting hex in both directions...

http://www.cflib.org/index.cfm?event=page.udfbyid&udfid=1424
http://www.cflib.org/index.cfm?event=page.udfbyid&udfid=1016

The real question is what rules should be used to parse the results? (Generally, not in CF terms. ) For example, are you expected parse the binary data positionally?

ie    Take bytes 0-4, convert it to hex then string, to get value X
      Take bytes 5-8, convert it to hex then string, to get value Y
      etc...

In the documentation you posted earlier, it looks like hex, but some of the values don't make sense.  AAPL comes out as 41 41 50 4C
in most translators. Any chance you could post a dummy sample of the raw binary data?

41 4D 54 44
 4 bytes
 Symbol = AAPL


0
_agx_Commented:
> The API team said that they don't know coldfusion so they can not provide guidance

   Do they have any java examples? That might help.  Also, is their API public / online?
0
stracqanAuthor Commented:
Their Java exmaples don't cover what I'm trying to do....

However, I have attached the source for the documentation and attached it here as a .txt file.  Just pop that into a browser and you'll be able to read these two sections of the documentation. Maybe you can make sense of it?  

P.S. I tried the HEX to string converter (thanks again for that), but it's not giving me anything useful after the ticker symbol (AAPL).  

Thanks again!!!!
pricehistoryresponse-2-.txt
pricehistorysamples-1-.txt
0
stracqanAuthor Commented:
> Also, is their API public / online?

Unfortunately it is not public....
0
_agx_Commented:
Are you using CF8+?
0
stracqanAuthor Commented:
yuppp .. CF8

Here's a PHP example if you know PHP..
class historicalDataTDAMTD
{        
    # Converts hexadecimal to float
    function hex2float($hex)
    {
        $bin = str_pad(base_convert($hex, 16, 2), 32, "0", STR_PAD_LEFT);
        $sign = $bin[0];
        $exp = bindec(substr($bin, 1, 8)) - 127;
        $man = (2 << 22) + bindec(substr($bin, 9, 23));
        $dec = $man * pow(2, $exp - 23) * ($sign ? -1 : 1);
        return $dec;
    }
    # Checks if request was valid (doesn't have errors)
    function errorResponse($response)
    {
        //possible errors
        $errors = array("error","fail","failed","not valid");
        foreach($errors as $error)
        {
            if(strpos(strtolower($response),$error))
            {
                return TRUE; //Add code to send email about the error
            }
        }
    }
    # Moves through array data from an element to a specified position
    function walk($arr,$start,$pos)
    {
        $arr = array_slice($arr,$start);
            $y="";
            for($i=0;$i<$pos;$i++)
            {
                $y=$y." ".$arr[$i];
            }
            return $y;
    }
    # Handles AMTD response data
    function dataHandling($arr,$ticker)
    {
        $totalElements = count($arr); //total array elements
        if($totalElements>4)
        {
            // Get Total Symbols
            $start = 0;
            $pos = 4;
            $totalSymbols = hexdec($this->walk($arr,$start,$pos)); 
            if($totalSymbols>0)
            {
                //Get Symbol Length
                $start += $pos;
                $pos = 2;
                $symbolLength = hexdec($this->walk($arr,$start,$pos));
                //Get Symbol Name
                $start += $pos;
                $symbol="";
                foreach(explode(" ",$this->walk($arr,$start,$symbolLength)) as $symbolLetter)
                {
                    $symbol = $symbol.chr(hexdec($symbolLetter));
                }
                //Get Error Code
                $start += $symbolLength;
                $pos = 1;
                $errorCode = hexdec($this->walk($arr,$start,$pos));
                if($errorCode==0)
                {
                    //Get Bar Count
                    $start += $pos;
                    $pos = 4;
                    $barCount = hexdec($this->walk($arr,$start,$pos));
                    //Check data consistency
                    //Symbol Count + Symbol Length + Symbol + Error Code + Bar Count + Bar Count*28 + Terminator = Total Array Elements
                    if((4+2+$symbolLength+1+4+$barCount*28+2)==$totalElements)
                    {
                        //Get OHLC data
                        $sliceStart = $start + $pos;
                        $ohlcData = array_chunk(array_slice($arr,$sliceStart,-2),28);
                        foreach($ohlcData as $dataArr)
                        {
                            
                            $data = array_chunk($dataArr,4);
                            
                            # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                            
                            $date = date('Y-m-d',strtotime("+1 day",(hexdec(implode($data[5]).implode($data[6])))/1000));
                            $open = number_format($this->hex2float(implode($data[3])),2);
                            $high = number_format($this->hex2float(implode($data[1])),2);
                            $low = number_format($this->hex2float(implode($data[2])),2);
                            $close = number_format($this->hex2float(implode($data[0])),2);
                            $volume = number_format($this->hex2float(implode($data[4]))*100,0);
                            
                            # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                            
                            //do whatever with data
                            //print symbol, date, open, high, low, close, volume
                            echo $symbol." *** ".$date." *** ".$open." *** ".$high." *** ".$low." *** ".$close." *** ".$volume."\n";
                            
                        }
                        //Get Terminator
                        //I'm not sure if this is necessary for this class but here it is..
                        $start += $barCount*28 + $pos;
                        $pos = 2;
                        $terminator = $this->walk($arr,$start,$pos);
                    }
                    else
                    {
                        //Add code to send email about the error
                        echo $ticker." elements count error: ".(4+2+$symbolLength+1+4+$barCount*28+2)." vs ".$totalElements." total";
                    }
                }
            }
        }
    }
}

Open in new window

0
_agx_Commented:
> Here's a PHP example if you know PHP..

   Not really, but enough to understand out what's going on from the code.  

   Just before you posted that I just figured out how you need to process the information ;-)  It's not hard,
   but it's going to take some coding.  


    Basically the response is just a big string of HEX.  The two files you posted give you a map of what's
    in the response (ie format) and how many bytes (or hex characters) are in each value.  You just have
    to walk the HEX string, and grab X bytes or characters and decode the substring to get the value.  Then
    move to the next value and repeat.

    For example the first 8 characters (or 4 bytes) in the string represent the SymbolCount in HEX.
   
              00 00 00 01      4      Symbol Count =1


     SymbolCount is an INTEGER. So you'll need the HexToDec function at cflib.org to decode the value
     http://www.cflib.org/index.cfm?event=page.udfbyid&udfid=347

     Just use CF string functions to grab the first 8 characters. Then use HexToDec() function to convert it.
     The result will equal 1, just as in their example


     <cfset theValue = hexToDec(mid(theHexString, 1, 8))>


     You'll need different functions depending on the value data type (SymbolCount = integer,
     Symbol = string, etc...). But that's the basic idea.  Though the response can contain repeated
     data (ie results for 1 to 4 symbols), so you'll have to factor that into the algorithm.




0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
stracqanAuthor Commented:
That makes so much sence now!!!!!!!!!!!!!!!!!!!!!!!!!!!  I basically have to build a custom algo to walk down the string and pick out what I need and convert it from HEX based on their documentation.... completely get it now... I thought there was going to be one CF function for it all, but I guess that would be too easy ;-)

Thank you so so so much for this .... I wish I could give you more then 500 points for this because you really did a great job in helping me figure out what I need to do.  Very very impressed.... thank you very much and you sure have earned that "genious" rank.... wow.....
0
stracqanAuthor Commented:
Thank you so so so much for this .... I wish I could give you more then 500 points for this because you really did a great job in helping me figure out what I need to do.  Very very impressed.... thank you very much and you sure have earned that "genious" rank.... wow.....
0
_agx_Commented:
> I thought there was going to be one CF function for it all, but I guess that would be too easy ;-)

Yes, that's what I was thinking at first. But once I saw their docs I realized it was formatted totally different than I was thinking.   Just keep in mind the "repeats". You'll have to use things like SymbolCount to determine the number of times to loop. But it's pretty straight-forward other than that. Plus you can always use the php base.  

Anyway, glad I could help and let me know if you get stuck anywhere :)
0
stracqanAuthor Commented:
I'm running into issues regarding the conversion from Hex to "FLOAT".... I'm going to open another question so you could earn more points.. thanks again
0
_agx_Commented:
> I'm running into issues regarding the conversion from Hex to "FLOAT"...

   I'll keep an eye out for the question.  I was curious and had just started looking at that conversion
   in php... I haven't cracked it yet. But am close.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Web Servers

From novice to tech pro — start learning today.