Handling Binary Data in Coldfusion

It's hard to say without seeing a sample, or more of the api..  But if it's really binary, you're definitely looking in the right area as far as functions go. What kind of binary data is the api returning, an image, text...? If it's text, what does the api say about encoding ?

Thanks for your fast responce!!!!!!!

The data being returned is supposed to be text.  In the documentation, it looks like its HEX?  Also, the only mention of encoding is in the URL, and that more for like if you have a parameter that you want to pass that has a space in it.

This is what they say should be the responce.....

00 00 00 01 00 04 41 4D 54 44 00 00 00 00 02 41
97 33 33 41 99 5C 29 41 90 3D 71 41 91 D7 0A 47
0F C6 14 00 00 01 16 6A E0 68 80 41 93 B4 05 41
97 1E B8 41 90 7A E1 41 96 8F 57 46 E6 2E 80 00
00 01 16 7A 53 7C 80 FF FF

and this should somehow mean this: (The data coppied a little funky, so there are basically three columns: Data, Type and Description.  That is how each block is formatted... sorry about that)

DATA
 TYPE
 DESCRIPTION
 
00 00 00 01
 4 bytes
 Symbol Count =1
 
00 04
 2 bytes
 Symbol Length = 4
 
41 4D 54 44
 4 bytes
 Symbol = AAPL
 
00
 1 byte
 Error code = 0 (OK)
 
00 00 00 02
 4 bytes
 Bar Count =  2
 
41 97 33 33
 4 bytes
 Close = 18.90
 
41 99 5C 29
 4 bytes
 High = 19.17
 
41 90 3D 71
 4 bytes
 Low = 18.03
 
41 91 D7 0A
 4 bytes
 Open = 18.23
 
47 0F C6 14
 4 bytes
 Volume = 3,680,608
 
00 00 01 16 6A E0 68 80
 8 bytes
 Timestamp = November 23,2007

41 93 B4 05
 4 bytes
 Close = 18.4629
 
41 97 1E B8
 4 bytes
 High = 18.89
 
41 90 7A E1
 4 bytes
 Low = 18.06
 
41 96 8F 57
 4 bytes
 Open = 18.82
 
46 E6 2E 80
 4 bytes
 Volume = 2,946,325
 
00 00 01 16 7A 53 7C 80
 8 bytes
 Timestamp = November 26,2007

FF FF
 2 bytes
 

----------------------------------
--------------------------------
-------------------------------
 
How they get from that paragraph to usuable data is where I dont't have a clue.  However, even getting to that first paragraph is a challenege because binarydecode() isn't showing me anything.........

and characterdecode() returns just a long string of ASCII numbers...

I meant charsetdecode() ... my bad

> The data being returned is supposed to be text.  In the documentation, it looks like its HEX?

That does look like hex. So when you say you tried binarydecode(...), you were using "hex" for the second parameter?

> you tried binarydecode(...),

BTW: Did you mean to say BinaryEncode() ?

I've been using decode, which is probably why it's been coming back blank...

I tried binaryencode() with "HEX" as the second object, but I got an error:

Parameter 1 of the BinaryEncode function, which is now
AAPL  C9aHC9¿ C9L¿C9Y¿E ¿f  ......................., must be a valid binary object.

So does that mean they are not actually sending the data in binary format?

Not necessarily... But you really need to find out a little more about exactly what formats they're using and how developer's are expected to decode the information.  Otherwise, you'll probably waste a lot of time guessing. (That's is all I can do right now without access to the api)  Once you know the format(s) they're using, it will be a lot easier to find the correct CF functions to use.

Sounds good! I'll contact them and see what they say .... Thank you!!!!

Sounds like a plan.  I'll be heading out soon. But will check back tomorrow.  

> <cfhttp> request

Total shot in the dark ...  but if you're not already, you might try using getasbinary="true". See if it has an effect either way..

THAT WORKED!!!!!!!!!!!! binaryencode() worked perfectly then!!!!!!!

Now all I have to do is convert the HEX values into something workable....

wow........ very impressed...

Thank you!!!!!! Enjoy your evening!!!

Great! I guess that old saying is true - always go with your instincts :)

ok....... sooooo.... the API team said that they don't know coldfusion so they can not provide guidance... All they said is that it comes in binary format and from there I need to convert it to something useful for Coldfusion to understand..... Any ideas on how to convert binary and/or hex into something useful?

Thanks again!!!!

It's certainly no problem breaking up the binary into bytes (if needed), or hex to plain string.  cflib.org has lots of functions for hex, converting hex in both directions...

http://www.cflib.org/index.cfm?event=page.udfbyid&udfid=1424
http://www.cflib.org/index.cfm?event=page.udfbyid&udfid=1016

The real question is what rules should be used to parse the results? (Generally, not in CF terms. ) For example, are you expected parse the binary data positionally?

ie    Take bytes 0-4, convert it to hex then string, to get value X
      Take bytes 5-8, convert it to hex then string, to get value Y
      etc...

In the documentation you posted earlier, it looks like hex, but some of the values don't make sense.  AAPL comes out as 41 41 50 4C
in most translators. Any chance you could post a dummy sample of the raw binary data?

> The API team said that they don't know coldfusion so they can not provide guidance

   Do they have any java examples? That might help.  Also, is their API public / online?

Their Java exmaples don't cover what I'm trying to do....

However, I have attached the source for the documentation and attached it here as a .txt file.  Just pop that into a browser and you'll be able to read these two sections of the documentation. Maybe you can make sense of it?  

P.S. I tried the HEX to string converter (thanks again for that), but it's not giving me anything useful after the ticker symbol (AAPL).  

Thanks again!!!!
pricehistoryresponse-2-.txt
pricehistorysamples-1-.txt

> Also, is their API public / online?

Unfortunately it is not public....

Are you using CF8+?

yuppp .. CF8

Here's a PHP example if you know PHP..

class historicalDataTDAMTD
{        
    # Converts hexadecimal to float
    function hex2float($hex)
    {
        $bin = str_pad(base_convert($hex, 16, 2), 32, "0", STR_PAD_LEFT);
        $sign = $bin[0];
        $exp = bindec(substr($bin, 1, 8)) - 127;
        $man = (2 << 22) + bindec(substr($bin, 9, 23));
        $dec = $man * pow(2, $exp - 23) * ($sign ? -1 : 1);
        return $dec;
    }
    # Checks if request was valid (doesn't have errors)
    function errorResponse($response)
    {
        //possible errors
        $errors = array("error","fail","failed","not valid");
        foreach($errors as $error)
        {
            if(strpos(strtolower($response),$error))
            {
                return TRUE; //Add code to send email about the error
            }
        }
    }
    # Moves through array data from an element to a specified position
    function walk($arr,$start,$pos)
    {
        $arr = array_slice($arr,$start);
            $y="";
            for($i=0;$i<$pos;$i++)
            {
                $y=$y." ".$arr[$i];
            }
            return $y;
    }
    # Handles AMTD response data
    function dataHandling($arr,$ticker)
    {
        $totalElements = count($arr); //total array elements
        if($totalElements>4)
        {
            // Get Total Symbols
            $start = 0;
            $pos = 4;
            $totalSymbols = hexdec($this->walk($arr,$start,$pos)); 
            if($totalSymbols>0)
            {
                //Get Symbol Length
                $start += $pos;
                $pos = 2;
                $symbolLength = hexdec($this->walk($arr,$start,$pos));
                //Get Symbol Name
                $start += $pos;
                $symbol="";
                foreach(explode(" ",$this->walk($arr,$start,$symbolLength)) as $symbolLetter)
                {
                    $symbol = $symbol.chr(hexdec($symbolLetter));
                }
                //Get Error Code
                $start += $symbolLength;
                $pos = 1;
                $errorCode = hexdec($this->walk($arr,$start,$pos));
                if($errorCode==0)
                {
                    //Get Bar Count
                    $start += $pos;
                    $pos = 4;
                    $barCount = hexdec($this->walk($arr,$start,$pos));
                    //Check data consistency
                    //Symbol Count + Symbol Length + Symbol + Error Code + Bar Count + Bar Count*28 + Terminator = Total Array Elements
                    if((4+2+$symbolLength+1+4+$barCount*28+2)==$totalElements)
                    {
                        //Get OHLC data
                        $sliceStart = $start + $pos;
                        $ohlcData = array_chunk(array_slice($arr,$sliceStart,-2),28);
                        foreach($ohlcData as $dataArr)
                        {
                            
                            $data = array_chunk($dataArr,4);
                            
                            # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                            
                            $date = date('Y-m-d',strtotime("+1 day",(hexdec(implode($data[5]).implode($data[6])))/1000));
                            $open = number_format($this->hex2float(implode($data[3])),2);
                            $high = number_format($this->hex2float(implode($data[1])),2);
                            $low = number_format($this->hex2float(implode($data[2])),2);
                            $close = number_format($this->hex2float(implode($data[0])),2);
                            $volume = number_format($this->hex2float(implode($data[4]))*100,0);
                            
                            # +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                            
                            //do whatever with data
                            //print symbol, date, open, high, low, close, volume
                            echo $symbol." *** ".$date." *** ".$open." *** ".$high." *** ".$low." *** ".$close." *** ".$volume."\n";
                            
                        }
                        //Get Terminator
                        //I'm not sure if this is necessary for this class but here it is..
                        $start += $barCount*28 + $pos;
                        $pos = 2;
                        $terminator = $this->walk($arr,$start,$pos);
                    }
                    else
                    {
                        //Add code to send email about the error
                        echo $ticker." elements count error: ".(4+2+$symbolLength+1+4+$barCount*28+2)." vs ".$totalElements." total";
                    }
                }
            }
        }
    }
}

Select allOpen in new window

That makes so much sence now!!!!!!!!!!!!!!!!!!!!!!!!!!!  I basically have to build a custom algo to walk down the string and pick out what I need and convert it from HEX based on their documentation.... completely get it now... I thought there was going to be one CF function for it all, but I guess that would be too easy ;-)

Thank you so so so much for this .... I wish I could give you more then 500 points for this because you really did a great job in helping me figure out what I need to do.  Very very impressed.... thank you very much and you sure have earned that "genious" rank.... wow.....

Thank you so so so much for this .... I wish I could give you more then 500 points for this because you really did a great job in helping me figure out what I need to do.  Very very impressed.... thank you very much and you sure have earned that "genious" rank.... wow.....

> I thought there was going to be one CF function for it all, but I guess that would be too easy ;-)

Yes, that's what I was thinking at first. But once I saw their docs I realized it was formatted totally different than I was thinking.   Just keep in mind the "repeats". You'll have to use things like SymbolCount to determine the number of times to loop. But it's pretty straight-forward other than that. Plus you can always use the php base.  

Anyway, glad I could help and let me know if you get stuck anywhere :)

I'm running into issues regarding the conversion from Hex to "FLOAT".... I'm going to open another question so you could earn more points.. thanks again

> I'm running into issues regarding the conversion from Hex to "FLOAT"...

   I'll keep an eye out for the question.  I was curious and had just started looking at that conversion
   in php... I haven't cracked it yet. But am close.