scope_creep
asked on
Using utl_raw package wrapped in Connor McDonald package to do bitand operations on numbers, but returning the wrong answer.
Folks,
I'm using the following code to perform a bitand of two numbers.
FUNCTION raw_ascii(p_dec number) RETURN raw
IS
v_result varchar2(1999);
v_tmp1 number := p_dec;
BEGIN
LOOP
v_result := CHR(MOD(v_tmp1,256)) || v_result ;
v_tmp1 := TRUNC(v_tmp1/256);
exit when v_tmp1 = 0;
END LOOP;
RETURN utl_raw.cast_to_raw(v_resu lt);
END;
FUNCTION ascii_raw(p_raw varchar2) RETURN number
IS
v_result number := 0;
BEGIN
FOR i in 1 .. LENGTH(p_raw)
LOOP
v_result := v_result * 256 + ASCII(SUBSTR(p_raw,i,1));
END LOOP;
RETURN v_result;
END;
FUNCTION bitand(p_dec1 number, p_dec2 number) RETURN varchar2
IS
BEGIN
dbms_output.put_line('pdec 1 ' || p_dec1);
dbms_output.put_line('pdec 2 ' || p_dec2);
RETURN
ascii_raw(
utl_raw.cast_to_varchar2(
utl_raw.bit_and(
raw_ascii(p_dec1),
raw_ascii(p_dec2)
)
)
);
END;
FUNCTION Determine_TLQ(det IN NUMBER) RETURN BOOLEAN
IS
Day_Mean CONSTANT NUMBER:=72;
tlq_mask_bin CONSTANT VARCHAR2(32):='11000010000 0000000000 0000000000 1';
tlq_mask_dec CONSTANT NUMBER:=3254779905;
tlq_mask_hex CONSTANT VARCHAR2(8):='C2000001';
cnt NUMBER;
bad_flag_count NUMBER;
temp_num NUMBER;
temp_field varchar2(16);
bit_field varchar2(16);
temp_bin varchar2(256);
BEGIN
bad_flag_count:=0;
FOR cnt IN 1..Det
LOOP
temp_field:=v_tlq_list(cnt );
dbms_output.put_line('befo re ' || temp_field);
temp_num:=to_dec(temp_fiel d,16);
dbms_output.put_line('Dec ' || temp_num);
bit_field:=bitand(temp_num , tlq_mask_dec);
dbms_output.put_line('bit output ' || bit_field);
dbms_output.put_line('bin temp num ' || to_bin(temp_num));
dbms_output.put_line('mask ' || tlq_mask_bin);
dbms_output.put_line('bin bit output ' || to_bin(bit_field));
IF (TO_NUMBER(bit_field)) >0
THEN
bad_flag_count:=bad_flag_c ount+1;
dbms_output.put_line('bad ' || bad_flag_count);
END IF;
IF (Bad_Flag_Count>Day_Mean)
THEN
RETURN TRUE;
END IF;
END LOOP;
RETURN FALSE;
END Determine_TLQ;
This is the output.
before 00000001
Dec 1
pdec1 1
pdec2 3254779905
bit output 1
bin temp num 1
mask 11000010000000000000000000 000001
bin bit output 1
bad 1
before C2000001
Dec 3254779905
pdec1 3254779905
pdec2 3254779905
bit output 3254779905
bin temp num 11000010000000000000000000 000001
mask 11000010000000000000000000 000001
bin bit output 11000010000000000000000000 000001
bad 2
before 00008000
Dec 32768
pdec1 32768
pdec2 3254779905
bit output 2147483649
bin temp num 1000000000000000
mask 11000010000000000000000000 000001
bin bit output 10000000000000000000000000 000001
bad 3
before C0000001
Dec 3221225473
pdec1 3221225473
pdec2 3254779905
bit output 3221225473
bin temp num 11000000000000000000000000 000001
mask 11000010000000000000000000 000001
bin bit output 11000000000000000000000000 000001
bad 4
before 00000000
Dec 0
pdec1 0
pdec2 3254779905
bit output 1
bin temp num 0
mask 11000010000000000000000000 000001
bin bit output 1
bad 5
before 82000001
Dec 2181038081
pdec1 2181038081
pdec2 3254779905
bit output 2181038081
bin temp num 10000010000000000000000000 000001
mask 11000010000000000000000000 000001
bin bit output 10000010000000000000000000 000001
bad 6
before 42000001
Dec 1107296257
pdec1 1107296257
pdec2 3254779905
bit output 1107296257
bin temp num 10000100000000000000000000 00001
mask 11000010000000000000000000 000001
bin bit output 10000100000000000000000000 00001
bad 7
The problem I have is that the oracle docs states that when bit ands are done, the unprocessed part of the longer raw string, is appended to the output, which results in a wrong value. So when I enter two numbers of different lengths, and do the bitand, I get the wrong result.
I could do the bitand by converting the numbers to binary strings, and using a related bitand function, but this would be slow.
Does anybody have a good way of performing a bitand on numbers of uneven length and getting the correct result back. For example, is it possible to 'lpad' a number to make it the same size as the other parameter to bitand, so the resultant value is correct.
regards
Bob.
I'm using the following code to perform a bitand of two numbers.
FUNCTION raw_ascii(p_dec number) RETURN raw
IS
v_result varchar2(1999);
v_tmp1 number := p_dec;
BEGIN
LOOP
v_result := CHR(MOD(v_tmp1,256)) || v_result ;
v_tmp1 := TRUNC(v_tmp1/256);
exit when v_tmp1 = 0;
END LOOP;
RETURN utl_raw.cast_to_raw(v_resu
END;
FUNCTION ascii_raw(p_raw varchar2) RETURN number
IS
v_result number := 0;
BEGIN
FOR i in 1 .. LENGTH(p_raw)
LOOP
v_result := v_result * 256 + ASCII(SUBSTR(p_raw,i,1));
END LOOP;
RETURN v_result;
END;
FUNCTION bitand(p_dec1 number, p_dec2 number) RETURN varchar2
IS
BEGIN
dbms_output.put_line('pdec
dbms_output.put_line('pdec
RETURN
ascii_raw(
utl_raw.cast_to_varchar2(
utl_raw.bit_and(
raw_ascii(p_dec1),
raw_ascii(p_dec2)
)
)
);
END;
FUNCTION Determine_TLQ(det IN NUMBER) RETURN BOOLEAN
IS
Day_Mean CONSTANT NUMBER:=72;
tlq_mask_bin CONSTANT VARCHAR2(32):='11000010000
tlq_mask_dec CONSTANT NUMBER:=3254779905;
tlq_mask_hex CONSTANT VARCHAR2(8):='C2000001';
cnt NUMBER;
bad_flag_count NUMBER;
temp_num NUMBER;
temp_field varchar2(16);
bit_field varchar2(16);
temp_bin varchar2(256);
BEGIN
bad_flag_count:=0;
FOR cnt IN 1..Det
LOOP
temp_field:=v_tlq_list(cnt
dbms_output.put_line('befo
temp_num:=to_dec(temp_fiel
dbms_output.put_line('Dec ' || temp_num);
bit_field:=bitand(temp_num
dbms_output.put_line('bit output ' || bit_field);
dbms_output.put_line('bin temp num ' || to_bin(temp_num));
dbms_output.put_line('mask
dbms_output.put_line('bin bit output ' || to_bin(bit_field));
IF (TO_NUMBER(bit_field)) >0
THEN
bad_flag_count:=bad_flag_c
dbms_output.put_line('bad ' || bad_flag_count);
END IF;
IF (Bad_Flag_Count>Day_Mean)
THEN
RETURN TRUE;
END IF;
END LOOP;
RETURN FALSE;
END Determine_TLQ;
This is the output.
before 00000001
Dec 1
pdec1 1
pdec2 3254779905
bit output 1
bin temp num 1
mask 11000010000000000000000000
bin bit output 1
bad 1
before C2000001
Dec 3254779905
pdec1 3254779905
pdec2 3254779905
bit output 3254779905
bin temp num 11000010000000000000000000
mask 11000010000000000000000000
bin bit output 11000010000000000000000000
bad 2
before 00008000
Dec 32768
pdec1 32768
pdec2 3254779905
bit output 2147483649
bin temp num 1000000000000000
mask 11000010000000000000000000
bin bit output 10000000000000000000000000
bad 3
before C0000001
Dec 3221225473
pdec1 3221225473
pdec2 3254779905
bit output 3221225473
bin temp num 11000000000000000000000000
mask 11000010000000000000000000
bin bit output 11000000000000000000000000
bad 4
before 00000000
Dec 0
pdec1 0
pdec2 3254779905
bit output 1
bin temp num 0
mask 11000010000000000000000000
bin bit output 1
bad 5
before 82000001
Dec 2181038081
pdec1 2181038081
pdec2 3254779905
bit output 2181038081
bin temp num 10000010000000000000000000
mask 11000010000000000000000000
bin bit output 10000010000000000000000000
bad 6
before 42000001
Dec 1107296257
pdec1 1107296257
pdec2 3254779905
bit output 1107296257
bin temp num 10000100000000000000000000
mask 11000010000000000000000000
bin bit output 10000100000000000000000000
bad 7
The problem I have is that the oracle docs states that when bit ands are done, the unprocessed part of the longer raw string, is appended to the output, which results in a wrong value. So when I enter two numbers of different lengths, and do the bitand, I get the wrong result.
I could do the bitand by converting the numbers to binary strings, and using a related bitand function, but this would be slow.
Does anybody have a good way of performing a bitand on numbers of uneven length and getting the correct result back. For example, is it possible to 'lpad' a number to make it the same size as the other parameter to bitand, so the resultant value is correct.
regards
Bob.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Stemu,
I've tested your solution, and it works well.
Have some points
regards
Bob.
Hope that helps!