How to process returned pointer structure data from Windows DLL in Ruby

Published on
10,549 Points
Last Modified:
In Ruby, Call or invoke a API DLL library is easily via Win32API class, win32-api gem or other gems. For general DLL API call, there are quite a few references, some good tips list below:

But all of these articles, tips don't mention how to process the returned pointer of a struct or memory block. As we known, standard DLL routine should return a code and return pointer/struct/memory block by function arguments, memory should free by claimer, so we can allocate memory in our codes and pass this pointer into DLL, the DLL routine save returned data into that pointer, after processed we need to free the allocated pointer ourselves. Anyway, the DLL author might not follow the General Programming Rules, or have some special reason, the DLL maybe return a pointer structure data directly instead of function arguments.

For example, if a definition like below:
typedef  struct Data {
    int len;
    int flag;
    char values[1000];
  } *Data;
  PData = *Data;

Open in new window

and the exports function of the DLL like below:
Data* GetData(int index, int flag); cdcel;

Open in new window

Win32API only provider a few data types like "0", "V", "I", "L", "P", other data types of other languages will mapping to these types, then we using "P" to receive the pointer data. unfortunately, we can't get the correct data by Win32API in Ruby via normally way:
require "Win32API"
getData = Win32API.new('filename.dll', 'GetData', ['I', 'I'], 'P');
pdata = getData.call(100, 10)

Open in new window

We might not get the correct data by pdata, because the pdata might be truncated. The "P" declaration only worked for import sections but does not work as well for export section.

In fact, Ruby doesn't know the size of the return pointer's object, thus Ruby just process pointer's data like "null-terminated" string. so the returned data might be truncated if the returned pointer's object is a complex struct. A similar question appears here demonstrating this problem: http://www.iteye.com/topic/47936

We hope Ruby will improve this later, but how do we process the PData pointer after the call now?

If we want to get returned data correctly, first, we need to define another API first:
@CopyMemory = Win32API.new('Kernel32.dll', 'RtlMoveMemory', ['P', 'I', 'I'], '0')

Open in new window

then, we need to change the Win32API export of the API as 'L':
getData = Win32API.new('filename.dll', 'GetData', ['I', 'I'], 'L')

Open in new window

now, we can call the API like before:
pd = getData.call(100, 10)

Open in new window

Last, we get data by the pointer via CopyMemory API:
buf = Array.new(buf_size, 0).pack('L*')
@CopyMemory.call(buf, pd, buf_size)

Open in new window

The buf saved the returned data now. you can process them, for this tips, we can process the data like below:
len, flag = buf.unpack('I2')
values = buf[9, len]

Open in new window

Of course, you should give the right buf_size, otherwise you might get AV error or crash.
Enjoy this complimentary article view.

Get unlimited access to our entire library of technical procedures, guides, and tutorials written by certified industry professionals.

Get 7 days free
Click here to view the full article

Using this article for work? Experts Exchange can benefit your whole team.

Learn More
Experts Exchange is a tech solutions provider where users receive personalized tech help from vetted certified professionals. These industry professionals also write and publish relevant articles on our site.
Ask questions about what you read
If you have a question about something within an article, you can receive help directly from the article author. Experts Exchange article authors are available to answer questions and further the discussion.
Learn from the best.