Link to home
Start Free TrialLog in
Avatar of binkzz
binkzz

asked on

system interaction

Hi!

Say I was wanting to create a small personal systems macro editor. I'd need to be able to read screen content, and be able to use the mouse freely outside my own application. How would I do this in Delphi?

Mr B
ASKER CERTIFIED SOLUTION
Avatar of Member_2_248744
Member_2_248744
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of binkzz
binkzz

ASKER

>I'm not to clear on what you want to do. You say "read >screen content, and be able to use the mouse
>freely outside my own application"

Well, what I meant was.. To give my macro application (or macapp) the ability to click outside it's own program, as well as be able to type into other programs.

The WH_JOURNAL looks very promising, I'll look into that properly when I've had some sleep.

Also, the other thing, how could I get a screen shot in delphi (or even better, a screenshot of the selected app only)?

Mr B
Here's a little bit of code to capture the window underneath your mouse.  It will need to be refined a bit, but the basics are here:

This is the main unit:

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs,
  ExtCtrls;

type
  TForm1 = class(TForm)
    Timer1: TTimer;
    Image1: TImage;
    procedure Timer1Timer(Sender: TObject);
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.DFM}


procedure TForm1.Timer1Timer(Sender: TObject);
var
  Hnd: Integer;
  R: TRect;
  Window : TCanvas;
  BMP: TBitmap;
begin
  Timer1.Enabled := false;
  Hnd := WindowFromPoint(Mouse.CursorPos);
  if Hnd > 0 then
    begin
      GetWindowRect(Hnd, R);
      BMP := TBitmap.Create;
      BMP.Height := R.Bottom;
      BMP.Width := R.Right;
      Window := TCanvas.Create;
      Window.Handle := GetWindowDC (Hnd) ;
      With BMP.Canvas do
        CopyRect (Rect (0, 0, R.Right, R.Bottom),
          Window, Rect (0, 0, R.Right, R.Bottom));
       Image1.Picture.Bitmap.Assign(BMP);
       BMP.Free;
       Window.Free;
    end;
  Timer1.Enabled := true;
end;

end.


and the DFM:

object Form1: TForm1
  Left = 263
  Top = 145
  Width = 696
  Height = 480
  Caption = 'Form1'
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'MS Sans Serif'
  Font.Style = []
  OldCreateOrder = False
  PixelsPerInch = 96
  TextHeight = 13
  object Image1: TImage
    Left = 30
    Top = 24
    Width = 421
    Height = 375
  end
  object Timer1: TTimer
    Interval = 100
    OnTimer = Timer1Timer
    Left = 514
    Top = 76
  end
end


If you need any more help, let me know.

Stu
gets full screen bitmap

procedure TForm1.FormShow(Sender: TObject);
var
DeskDC: HDC;
ScreenBmp: TBitmap;
begin
{full screen bitmap}
ScreenBmp := TBitmap.Create;
ScreenBmp.Width := Screen.Width;
ScreenBmp.Height := Screen.Height;
DeskDC := GetDC(0);
BitBlt(ScreenBmp.Canvas.Handle, 0, 0, Screen.Width, Screen.Height, DeskDC, 0, 0, SRCCOPY);
StretchBlt(DeskDC, 0, Screen.Height,
    Screen.Width, -Screen.Height, ScreenBmp.Canvas.Handle,
    0, 0, Screen.Width, Screen.Height, SRCCOPY);
ReleaseDC(GetDesktopWindow,DeskDC);
FreeAndNil(ScreenBmp);
end;

if you use windows hDC then creating a Canvas isn't nessary
Had some time this weekend, so I took the code for a Journaling hook together for ya. You do Not need to run this code in a .DLL, like other Hooks, you can just put in in your app. I had a very difficult time trying to use the JournalPlaybackProc function, The documentation and examples I had for it were crap (including the borland page above), not really saying what was going on with it and sometimes giving wierd behavior, look at my comments in the code. Here is some code from an app that records keyboard and mouse events and then is able to play them back at another time. It can save the events as a file, to be loaded and played later. I set it so it does not record the 1000's of mouse move messages that are useless in a macro play back. Because the mouse clicks are based on position, I had to check the user's desktop Work Area, to see if the screen Resolution or TaskBar Position had changed.



private
    { Private declarations }
    EventMsgArray: Array of TEVENTMSG;
    TempBmp: TBitmap;
    TimerStr: String;
    procedure DrawDesk(DisText: String; Red: Boolean);


var
  Form1: TForm1;
  Track, SysModal, DoDelay, GotMove: Boolean;
  FirstGo: Byte;
  StartTime: Cardinal;
  TempEvMsg: TEventMsg;
  SRect: TRect;
  NumMsgs, PlayNum: Integer;
  JHook: THandle;


procedure TForm1.FormCreate(Sender: TObject);
begin
NumMsgs := 0;
JHook := 0;
PlayNum := 0;
Track := False;
SysModal := False;
SetLength(EventMsgArray,1);
end;


procedure TForm1.FormClose(Sender: TObject; var Action: TCloseAction);
begin
SetLength(EventMsgArray,0);
UnhookWindowsHookEx( JHook );
end;


procedure TForm1.ApplicationEvents1Message(var Msg: tagMSG;
  var Handled: Boolean);
begin
{this is the Application OnMessage event.
You need to check for the WM_CANCELJOURNAL message
which will happen if the user presses Ctrl+Esc, Ctrl+Alt+Del and maybe Ctrl+Break}
Handled := False;
if Msg.message = WM_CANCELJOURNAL then
  begin
  Timer1.Enabled := False;
  InvalidateRect(0,@SRect, True);
  SetForegroundWindow(Handle);
  Perform(WM_SYSCOMMAND,SC_RESTORE,0);
  ShowMessage('Journaling has been canceled');
  end;
end;


procedure TForm1.DrawDesk(DisText: String; Red: Boolean);
var
DeskDC, hBrush: Integer;
begin
{I had to remind the user that Journaling was happing,
so I draw a Rect in the lower right corner of the desktop}
if not SystemParametersInfo(SPI_GETWORKAREA,0,@SRect,0) then
SRect := Rect(0,0,Screen.Width,Screen.Height - 32);
SRect.TopLeft := Point(SRect.Right-158,SRect.Bottom-60);
DeskDC := GetDC(0);
hBrush := 0;
if Red then
  begin
  hBrush := CreateSolidBrush($000000FF);
  SelectObject(DeskDC,hBrush);
  SetBkColor(DeskDC,$000000FF);
  end;
Rectangle(DeskDC,SRect.Left,SRect.Top,SRect.Right,SRect.Bottom);
InflateRect(SRect,-3,-2);
DrawText(DeskDC,PChar(DisText),Length(DisText),SRect, DT_LEFT or DT_WORDBREAK);
DeleteObject(hBrush);
ReleaseDC(GetDesktopWindow,DeskDC);
TimerStr := DisText;
Timer1.Tag := 0;
Timer1.Interval := 700;
Timer1.Enabled := True;
end;


function Hook2Proc(Code:integer; wParam: Longint; var EventStrut: TEVENTMSG): Longint; stdcall;
begin
{this is the JournalRecordProc}
Result := CallNextHookEx( JHook, Code, wParam, Longint(@EventStrut) );
if Code < 0 then Exit;

{you should cancel operation if you get HC_SYSMODALON}
if Code = HC_SYSMODALON then SysModal := True
else if Code = HC_SYSMODALOFF then SysModal := False;
if Code = HC_ACTION then
  begin
  if SysModal then Exit;
  if EventStrut.message = WM_KEYDOWN then
  begin
  if (EventStrut.paramL = 17923) {17923 is for Ctrl+Break}
  or (Chr(LOBYTE(LOWORD(EventStrut.paramL)))='Q') and (GetKeyState(VK_CONTROL) < 0) then
    Form1.but_EndMTClick(Form1)  {Ctrl+Q and Ctrl+Break will End the Journal record}
  end  else if (Chr(LOBYTE(LOWORD(EventStrut.paramL)))='S') and (GetKeyState(VK_CONTROL) < 0) then
    begin
    {Ctrl+S will start the journal record}
    if Track = False then
      begin
      StartTime := GetTickCount;
      {It is nessarry to subtract the TickCount to get the amount of ticks between events
      I do not use the Event Time in my playback Proc, but you might want to}
      FirstGo := 0;
      {First go is used to remove the Ctrl and S key up events}
      Form1.DrawDesk('JOURNAL NOW RECORDING'#10'Press Ctrl+q to Stop', False);
      end;
    Track := True;
    end;
If Track then
    begin
    if NumMsgs < 1024 then
    {I have limited the event log to 1024 for NO reason
    except for those who forget or don't know how to stop
    the the journaling}
      begin
      if EventStrut.message = WM_MOUSEMOVE then
        begin
        {I found the 1000's of mouse move messages to be
        totally useless, so I skip all the Mouse Move messages
        except the Last one before another type of event}
        TempEvMsg := EventStrut;
        GotMove := True;
        end else
        begin
        if GotMove then
          begin
          GotMove := False;
          Form1.EventMsgArray[NumMsgs] := TempEvMsg;
          Dec(Form1.EventMsgArray[NumMsgs].time,StartTime);
          Inc(NumMsgs);
          SetLength(Form1.EventMsgArray, NumMsgs+1);
          end;
        if FirstGo < 2 then
        if (EventStrut.message = WM_KEYUP) then
          begin
          {this prevents the Ctrl and s key up from being recorded}
          if EventStrut.paramL = 7441 then Inc(FirstGo);
          {7441 is Ctrl key}
          if Chr(LOBYTE(LOWORD(EventStrut.paramL))) = 'S' then Inc(FirstGo);
          Exit
          end else FirstGo := 2;
        Form1.EventMsgArray[NumMsgs] := EventStrut;
        Dec(Form1.EventMsgArray[NumMsgs].time,StartTime);
        Inc(NumMsgs);
        SetLength(Form1.EventMsgArray, NumMsgs+1);
        end;
      end;
    end;
  end;
end;


procedure TForm1.but_StartMTClick(Sender: TObject);
var
Rect1: TRect;
begin
{this button click starts the Journal Record hook}
if Track then
  begin
  Messagebox(0,'Mouse tracking has already been started', 'No can Restart', MB_OK or MB_ICONQUESTION);
  Exit;
  end;

GotMove := False;
FirstGo := 0;
JHook := SetWindowsHookEx(WH_JOURNALRECORD , @Hook2Proc, 0{hInstance}, 0);

if JHook > 0 then
  begin
  NumMsgs := 1;
{set NumMsgs to 1 because the first EventMsgArray is used for
file verification and NOT for events}
  Track := False;
  SysModal := False;
  SetLength(EventMsgArray,2);

  {If the user has changed screen Resolution OR moved the task bar
  then the Position of mouse clicks will NOT be correct, So get the
  Work Area and record it, so you can compare it later for playback}
  if not SystemParametersInfo(SPI_GETWORKAREA,0,@Rect1,0) then
  Rect1 := Rect(0,0,Screen.Width,Screen.Height - 32);
  EventMsgArray[0].message := Rect1.Left;
  EventMsgArray[0].paramL := Rect1.Top;
  EventMsgArray[0].paramH := Rect1.Right;
  EventMsgArray[0].time := Rect1.Bottom;
  EventMsgArray[0].hwnd := 65432;
  Perform(WM_SYSCOMMAND,SC_MINIMIZE or SC_ICON,0);
{I minimaze the main Form so the user can click other things}
  DrawDesk('JOURNAL RECORD READY'#10'Press Ctrl+s to Start', False);
  end else
  ShowMessage('Journal Hook could not be set');
end;


procedure TForm1.but_EndMTClick(Sender: TObject);
var
i: Integer;
begin
{This is an Emergency Stop Journaling button click}
Track := False;
if JHook < 1 then
NumMsgs := 0 else
UnhookWindowsHookEx(JHook);
Timer1.Enabled := False;
SetForegroundWindow(Handle);
Perform(WM_SYSCOMMAND,SC_RESTORE,0);
Application.ProcessMessages;
InvalidateRect(0,@SRect, True);
Sleep(200);
JHook := 0;
if NumMsgs >1 then
  begin
  Memo1.Clear;
    {start the for loop with 1 because the 0 if for Work Area verification}
  for i := 1 to NumMsgs-1 do
    begin
    case EventMsgArray[i].message of
      WM_LBUTTONDOWN: Memo1.Lines.Add('L Down '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
      WM_LBUTTONUP: Memo1.Lines.Add('L UP '+IntToStr(LOWORD(EventMsgArray[i].paramL))+'  '+IntToStr(HIWORD(EventMsgArray[i].paramL))+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
      WM_MOUSEMOVE: Memo1.Lines.Add('Move '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
      WM_RBUTTONDOWN: Memo1.Lines.Add('R Down '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
      WM_RBUTTONUP: Memo1.Lines.Add('R UP '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
      WM_KEYDOWN: Memo1.Lines.Add('Key Down '+IntToStr(EventMsgArray[i].paramL){Chr(LOBYTE(LOWORD(EventMsgArray[i].paramL)))}+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
      WM_KEYUP: Memo1.Lines.Add('Key UP '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
      else Memo1.Lines.Add('Other '+IntToStr(EventMsgArray[i].message)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
      end;
    end;

      while EventMsgArray[NumMsgs-1].paramL = 7441 do
      begin
      {This removes the Ctrl key mouse down repeat messages}
      Dec(NumMsgs);
      if NumMsgs < 3 then Break;
      end;

  end else
  ShowMessage('No Journal Messages are recorded');
end;


function PlaybackProc(Code:integer; wParam: Integer; var EventStrut: TEVENTMSG): Integer; stdcall;
begin
{this journal playback function is very difficult to understand.
Each event in the EventMsgArray is called here 2 or 3 times or more,
first with a HC_GETNEXT code to get a delay Result, if there is a Delay
Result larger than 0,then the event is NOT Fired, and a HC_GETNEXT is
called again after the delay period. If the Delay Result is 0 then the event
is fired and the HC_SKIP code is called, which advances the PlayNum to the
next EventMsgArray and then it does the HC_GETNEXT to start the loop again.}
Result := 0;

if Code < 0 then
  begin
  {if code is less than 0 pass it}
  Result := CallNextHookEx( JHook, Code, wParam, Integer(@EventStrut) );
  Exit;
  end;

if Code = HC_SKIP then
  begin
  if SysModal then Exit;
  Inc(PlayNum);
  DoDelay := True;
  {set this to true to get a Delay in the HC_GETNEXT}
  if PlayNum >= NumMsgs-1 then
    begin
    {this ends the playback}
    UnhookWindowsHookEx(JHook);
    Form1.Timer1.Enabled := False;
    Sleep(700);
    InvalidateRect(0,@SRect, True);
    SetForegroundWindow(Form1.Handle);
    Form1.Perform(WM_SYSCOMMAND,SC_RESTORE,0);
    end;
  Exit;
  end;

if Code = HC_GETNEXT then
  begin
  if SysModal then Exit;
  Form1.Label3.Caption := 'HC_GETNEXT';
  Move(Form1.EventMsgArray[PlayNum], EventStrut, SizeOf(TEVENTMSG));
  if DoDelay then
    begin
    Result := 380;
    {I tried lots of different combinations of setting the
    delay Result, but for playback using a constant between 200 and 1000
    seemed best}

    //Result := Form1.EventMsgArray[PlayNum].time;
    {Setting the Result to time of EventStrut will simulate the
    event timing, more or less}
    DoDelay := False;
    end; {else
  EventStrut.time := 300{EventStrut.time+ StartTime};
  {I tried setting this time to different values, but it seems to
  have little or No effect}

  {unlike many other windows system Proc, this uses the memory value
  in the Proc parameter EventStrut (lParam), when this function returns.
  Notice that it is declared as a var. You do NOT need to to use
  CallNextHookEx, because this is not used like other Hooks, to get messages,
  this fires mouse and keyboard events outside of a "hook Chain"}
  Exit;
  end else
if Code = HC_SYSMODALON then  SysModal := True
else if Code = HC_SYSMODALOFF then
  begin
  SysModal := False;
  end;

CallNextHookEx( JHook, Code, wParam, Integer(@EventStrut));
{I'm not sure you need this CallNextHookEx here}
end;


procedure TForm1.but_PlayBackClick(Sender: TObject);
var
i: Integer;
begin
ShowMessage('NumMsgs '+IntToStr(NumMsgs));
if NumMsgs > 0 then
  begin
  PlayNum := 1;
  DoDelay := False;
  ReleaseCapture;
  SysModal := False;
  Form1.Perform(WM_SYSCOMMAND,SC_MINIMIZE or SC_ICON,0);
  DrawDesk('JOURNAL PLAYBACK'#10'Please Wait', False);
  StartTime := GetTickCount;
  JHook := SetWindowsHookEx(WH_JOURNALPLAYBACK , @PlaybackProc, 0{hInstance}, 0);
  end;
end;


procedure TForm1.sbut_SaveTFileClick(Sender: TObject);
var
  SaveFile: File of TEventMsg;
  i: Integer;
  FileName: String;
begin
if NumMsgs > 1 then
  begin
  FileName := 'C:\Stuff\Play Journal1.pbj';
  AssignFile(SaveFile, FileName);
  Rewrite(SaveFile);
  for i := 0 to NumMsgs-1 do
  Write(SaveFile,EventMsgArray[i]);
  CloseFile(SaveFile);
  end;
end;

procedure TForm1.sbut_LoadTFileClick(Sender: TObject);
var
  FileName: String;
  LoadFile: File of TEventMsg;
  i: Integer;
  Rect1: TRect;
begin
FileName := 'C:\Stuff\Play Journal1.pbj';
if FileExists(FileName) then
  begin
  AssignFile(LoadFile, FileName);
  Reset(LoadFile);
  NumMsgs := FileSize(LoadFile);
  if NumMsgs > 1 then
    begin
    SetLength(EventMsgArray, NumMsgs);
    for i := 0 to NumMsgs-1 do
    Read(LoadFile,EventMsgArray[i]);
    CloseFile(LoadFile);

    {this is to check the file to see if it is a valid .ppj file for this
    version AND to see if the DeskTop work Area matches the file's work area}
    if not SystemParametersInfo(SPI_GETWORKAREA,0,@Rect1,0) then
    Rect1 := Rect(0,0,Screen.Width,Screen.Height - 32);
    if EventMsgArray[0].hwnd <> 65432 then
      begin
      {the 65432 is a number just to verify that this is a valid .pbj file}
      SetLength(EventMsgArray, 0);
      ShowMessage('the file "'+FileName+'" is NOT an Event File compatible with this version. No Playback is availible');
      end else
    {check to see if the Desktop work area matches}
      if (EventMsgArray[0].message <> Rect1.Left) or
      (EventMsgArray[0].paramL <> Rect1.Top) or
      (EventMsgArray[0].paramH <> Rect1.Right) or
      (EventMsgArray[0].time <> Rect1.Bottom) then
        begin
        SetLength(EventMsgArray, 0);
        ShowMessage('the DeskTop Dimentions are NOT compatible with this Playback File. No Playback is availible');
        end else
        begin
        Memo1.Clear;
        for i := 1 to NumMsgs-1 do
          begin
          {this adds the messages to a memo, just to check in a practice version}
            case EventMsgArray[i].message of
              WM_LBUTTONDOWN: Memo1.Lines.Add('L Down '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
              WM_LBUTTONUP: Memo1.Lines.Add('L UP '+IntToStr(LOWORD(EventMsgArray[i].paramL))+'  '+IntToStr(HIWORD(EventMsgArray[i].paramL))+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
              WM_MOUSEMOVE: Memo1.Lines.Add('Move '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
              WM_RBUTTONDOWN: Memo1.Lines.Add('R Down '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
              WM_RBUTTONUP: Memo1.Lines.Add('R UP '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
              WM_KEYDOWN: Memo1.Lines.Add('Key Down '+IntToStr(EventMsgArray[i].paramL){Chr(LOBYTE(LOWORD(EventMsgArray[i].paramL)))}+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
              WM_KEYUP: Memo1.Lines.Add('Key UP '+IntToStr(EventMsgArray[i].paramL)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
              else Memo1.Lines.Add('Other '+IntToStr(EventMsgArray[i].message)+'  '+IntToStr(EventMsgArray[i].paramH)+'  hwnd '+IntToStr(EventMsgArray[i].hwnd));
            end;
          end;
        end;
    end;
  end;
end;


procedure TForm1.Timer1Timer(Sender: TObject);
begin
If Timer1.Tag = 0 then
  begin
  DrawDesk(TimerStr, True);
  Timer1.Tag := 1;
  Timer1.Interval := 300;
  end else
  begin
  DrawDesk(TimerStr, False);
  Timer1.Tag := 0;
  Timer1.Interval := 800;
  end;
end;

procedure TForm1.FormActivate(Sender: TObject);
begin
{you may not need this, it's kind of insurance to end the timer}
Timer1.Enabled := False;
UnhookWindowsHookEx(JHook);
end;

- - - - - - - - - - - - - - - - - - - - - - - - -

let me know if this helps, ask questions if it is unclear.
Avatar of binkzz

ASKER

Excellent, just what I needed!

Thanks a lot, much appreciated.

There's an extra 200 for the effort in the question list, and 100 for mr Stuart as his bit of code was quite helpful.

Mr B

Stuart Johnson,

The points for Q for you is found at:
https://www.experts-exchange.com/questions/20291778/Points-for-Stuart-Johnson.html

modulo

Community Support Moderator
Experts Exchange