[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More


How to process returned pointer structure data from Windows DLL in Ruby

Published on
10,437 Points
Last Modified:
In Ruby, Call or invoke a API DLL library is easily via Win32API class, win32-api gem or other gems. For general DLL API call, there are quite a few references, some good tips list below:

But all of these articles, tips don't mention how to process the returned pointer of a struct or memory block. As we known, standard DLL routine should return a code and return pointer/struct/memory block by function arguments, memory should free by claimer, so we can allocate memory in our codes and pass this pointer into DLL, the DLL routine save returned data into that pointer, after processed we need to free the allocated pointer ourselves. Anyway, the DLL author might not follow the General Programming Rules, or have some special reason, the DLL maybe return a pointer structure data directly instead of function arguments.

For example, if a definition like below:
typedef  struct Data {
    int len;
    int flag;
    char values[1000];
  } *Data;
  PData = *Data;

Open in new window

and the exports function of the DLL like below:
Data* GetData(int index, int flag); cdcel;

Open in new window

Win32API only provider a few data types like "0", "V", "I", "L", "P", other data types of other languages will mapping to these types, then we using "P" to receive the pointer data. unfortunately, we can't get the correct data by Win32API in Ruby via normally way:
require "Win32API"
getData = Win32API.new('filename.dll', 'GetData', ['I', 'I'], 'P');
pdata = getData.call(100, 10)

Open in new window

We might not get the correct data by pdata, because the pdata might be truncated. The "P" declaration only worked for import sections but does not work as well for export section.

In fact, Ruby doesn't know the size of the return pointer's object, thus Ruby just process pointer's data like "null-terminated" string. so the returned data might be truncated if the returned pointer's object is a complex struct. A similar question appears here demonstrating this problem: http://www.iteye.com/topic/47936

We hope Ruby will improve this later, but how do we process the PData pointer after the call now?

If we want to get returned data correctly, first, we need to define another API first:
@CopyMemory = Win32API.new('Kernel32.dll', 'RtlMoveMemory', ['P', 'I', 'I'], '0')

Open in new window

then, we need to change the Win32API export of the API as 'L':
getData = Win32API.new('filename.dll', 'GetData', ['I', 'I'], 'L')

Open in new window

now, we can call the API like before:
pd = getData.call(100, 10)

Open in new window

Last, we get data by the pointer via CopyMemory API:
buf = Array.new(buf_size, 0).pack('L*')
@CopyMemory.call(buf, pd, buf_size)

Open in new window

The buf saved the returned data now. you can process them, for this tips, we can process the data like below:
len, flag = buf.unpack('I2')
values = buf[9, len]

Open in new window

Of course, you should give the right buf_size, otherwise you might get AV error or crash.

Featured Post

OWASP Proactive Controls

Learn the most important control and control categories that every architect and developer should include in their projects.

Join & Write a Comment

Overview of OneDrive and collaboration.
This is Part-2 of Learning to use the Power of Mailwasher Pro so if you haven't watched Part-1 yet, I urge you to do so before watching this video. Click this link to watch Part-1 (https://www.experts-exchange.com/videos/56638/Learn-to-use-the-POWER…

Keep in touch with Experts Exchange

Tech news and trends delivered to your inbox every month