Link to home
Start Free TrialLog in
Avatar of scope_creep
scope_creep

asked on

Using utl_raw package wrapped in Connor McDonald package to do bitand operations on numbers, but returning the wrong answer.

Folks,

I'm using the following code to perform a bitand of two numbers.


FUNCTION raw_ascii(p_dec number) RETURN raw
IS

  v_result varchar2(1999);
  v_tmp1   number := p_dec;

BEGIN

  LOOP
    v_result := CHR(MOD(v_tmp1,256)) || v_result ;
    v_tmp1 := TRUNC(v_tmp1/256);
    exit when v_tmp1 = 0;
  END LOOP;
   
  RETURN utl_raw.cast_to_raw(v_result);

END;


FUNCTION ascii_raw(p_raw varchar2) RETURN number
IS
  v_result number := 0;

BEGIN
 
  FOR i in 1 .. LENGTH(p_raw)
  LOOP
    v_result := v_result * 256 + ASCII(SUBSTR(p_raw,i,1));
  END LOOP;
  RETURN v_result;

END;

FUNCTION bitand(p_dec1 number, p_dec2 number) RETURN varchar2
IS

BEGIN
   
   dbms_output.put_line('pdec1 ' || p_dec1);
   dbms_output.put_line('pdec2 ' || p_dec2);

   RETURN
   ascii_raw(
     utl_raw.cast_to_varchar2(
       utl_raw.bit_and(
         raw_ascii(p_dec1),
         raw_ascii(p_dec2)
       )
     )
   );
END;



FUNCTION Determine_TLQ(det IN NUMBER) RETURN BOOLEAN
IS

Day_Mean      CONSTANT NUMBER:=72;

tlq_mask_bin    CONSTANT VARCHAR2(32):='11000010000000000000000000000001';
tlq_mask_dec      CONSTANT NUMBER:=3254779905;
tlq_mask_hex      CONSTANT VARCHAR2(8):='C2000001';

cnt            NUMBER;
bad_flag_count      NUMBER;
temp_num      NUMBER;
temp_field      varchar2(16);
bit_field      varchar2(16);            
temp_bin      varchar2(256);

BEGIN
      bad_flag_count:=0;
            
      FOR cnt IN 1..Det
      LOOP
            temp_field:=v_tlq_list(cnt);
                  dbms_output.put_line('before ' || temp_field);      
            
            temp_num:=to_dec(temp_field,16);
                  dbms_output.put_line('Dec ' || temp_num);
                        
            bit_field:=bitand(temp_num, tlq_mask_dec);                        
            
            dbms_output.put_line('bit output ' || bit_field);
      
              dbms_output.put_line('bin temp num   ' || to_bin(temp_num));
            dbms_output.put_line('mask           ' || tlq_mask_bin);
            dbms_output.put_line('bin bit output ' || to_bin(bit_field));
                  
 
             IF (TO_NUMBER(bit_field)) >0
            THEN                  
                  bad_flag_count:=bad_flag_count+1;
                        dbms_output.put_line('bad ' || bad_flag_count);
            END IF;

            IF (Bad_Flag_Count>Day_Mean)
            THEN
                  RETURN TRUE;
            END IF;
      
      END LOOP;            
      RETURN FALSE;            

END Determine_TLQ;

This is the output.

before 00000001
Dec 1
pdec1 1
pdec2 3254779905
bit output 1
bin temp num   1
mask           11000010000000000000000000000001
bin bit output 1
bad 1
before C2000001
Dec 3254779905
pdec1 3254779905
pdec2 3254779905
bit output 3254779905
bin temp num   11000010000000000000000000000001
mask           11000010000000000000000000000001
bin bit output 11000010000000000000000000000001
bad 2
before 00008000
Dec 32768
pdec1 32768
pdec2 3254779905
bit output 2147483649
bin temp num   1000000000000000
mask           11000010000000000000000000000001
bin bit output 10000000000000000000000000000001
bad 3
before C0000001
Dec 3221225473
pdec1 3221225473
pdec2 3254779905
bit output 3221225473
bin temp num   11000000000000000000000000000001
mask           11000010000000000000000000000001
bin bit output 11000000000000000000000000000001
bad 4
before 00000000
Dec 0
pdec1 0
pdec2 3254779905
bit output 1
bin temp num   0
mask           11000010000000000000000000000001
bin bit output 1
bad 5
before 82000001
Dec 2181038081
pdec1 2181038081
pdec2 3254779905
bit output 2181038081
bin temp num   10000010000000000000000000000001
mask           11000010000000000000000000000001
bin bit output 10000010000000000000000000000001
bad 6
before 42000001
Dec 1107296257
pdec1 1107296257
pdec2 3254779905
bit output 1107296257
bin temp num   1000010000000000000000000000001
mask           11000010000000000000000000000001
bin bit output 1000010000000000000000000000001
bad 7


The problem I have is that the oracle docs states that when bit ands are done, the unprocessed part of the longer raw string, is appended to the output, which results in a wrong value. So when I enter two numbers of different lengths, and do the bitand, I get the wrong result.

I could do the bitand  by converting the numbers to binary strings, and using a related bitand function, but this would be slow.

Does anybody have a good way of performing a bitand on numbers of uneven length and getting the correct result back. For example, is it possible to 'lpad' a number to make it the same size as the other parameter to bitand, so the resultant value is correct.


regards
Bob.



Avatar of pennnn
pennnn

If "the unprocessed part of the longer raw string, is appended to the output", then why not just trim the appended unprocessed part before returning the result? Just get the length of the shorter string and after the bit_and operation just trim the result to the desired length.
Hope that helps!
ASKER CERTIFIED SOLUTION
Avatar of stemu2000
stemu2000
Flag of Japan image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of scope_creep

ASKER


Stemu,

I've tested your solution, and it works well.
Have some points

regards
Bob.