I'm working on a binary ecryption library for perl. The library written in C but my problem is rather perl API related so I'm posting here.
The library receives a few parameters, does the encryption and returns the encrypted data. Before the encryption takes place the required buffer size for the encrypted data is calculated. When this value (nOutDataLen) is calculated sv_setpvn is used to allocate the buffer for the result. The whole context is
/* set up our output buffer */
sv_setpvn(svOutData, "", nOutDataLen);
/* get the pointer to the buffer */
pOutData = (unsigned char*)SvPV(svOutData, nOutDataLen);
/* now try and create the blob */
bRet = EncryptData(pRemoteKey,
This code worked fine so far but it turned out recently that if the calculated result length is too long the code is failing. I did some sanity check and found that if nOutDataLen >= 7361 then the library crashes but working fine if nOutDatalength <= 7360.
According to the debug session sv_setpvn is failing.
Interestingly the code is working fine when the compiler optimalisations (used /O1) are turned off.
I have used Win32 for the investigation but the library has similar problems on Linux.