troubleshooting Question

CreateProcess/Pipe Problem

Avatar of BrianGEFF719
BrianGEFF719Flag for United States of America asked on
Visual Basic Classic
7 Comments1 Solution1991 ViewsLast Modified:
Here is what I am trying to do. I am trying to create a program that will spawn a DOS Program and then redirect the output to a byte array and then I also want to be able to send input to the DOS Program via a byte buffer. The only way I could think of doing this to keep the process open and allow the user to send the input was to create a Class Module, below is what I have so far.

Private Declare Function CreatePipe Lib "kernel32" (phReadPipe As Long, phWritePipe As Long, lpPipeAttributes As SECURITY_ATTRIBUTES, ByVal nSize As Long) As Long
Private Declare Sub GetStartupInfo Lib "kernel32" Alias "GetStartupInfoA" (lpStartupInfo As STARTUPINFO)
Private Declare Function CreateProcess Lib "kernel32" Alias "CreateProcessA" (ByVal lpApplicationName As String, ByVal lpCommandLine As String, lpProcessAttributes As Any, lpThreadAttributes As Any, ByVal bInheritHandles As Long, ByVal dwCreationFlags As Long, lpEnvironment As Any, ByVal lpCurrentDriectory As String, lpStartupInfo As STARTUPINFO, lpProcessInformation As PROCESS_INFORMATION) As Long
Private Declare Function SetWindowText Lib "user32" Alias "SetWindowTextA" (ByVal hwnd As Long, ByVal lpString As String) As Long
Private Declare Function ReadFile Lib "kernel32" (ByVal hFile As Long, lpBuffer As Any, ByVal nNumberOfBytesToRead As Long, lpNumberOfBytesRead As Long, lpOverlapped As Any) As Long
Private Declare Function SendMessage Lib "user32" Alias "SendMessageA" (ByVal hwnd As Long, ByVal wMsg As Long, ByVal wParam As Long, lParam As Any) As Long
Private Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
 nLength As Long
 lpSecurityDescriptor As Long
 bInheritHandle As Long
End Type
 hProcess As Long
 hThread As Long
 dwProcessId As Long
 dwThreadId As Long
End Type
 cb As Long
 lpReserved As Long
 lpDesktop As Long
 lpTitle As Long
 dwX As Long
 dwY As Long
 dwXSize As Long
 dwYSize As Long
 dwXCountChars As Long
 dwYCountChars As Long
 dwFillAttribute As Long
 dwFlags As Long
 wShowWindow As Integer
 cbReserved2 As Integer
 lpReserved2 As Byte
 hStdInput As Long
 hStdOutput As Long
 hStdError As Long
End Type
   ternal As Long
   ternalHigh As Long
   offset As Long
   OffsetHigh As Long
   hEvent As Long
End Type
Private Const SW_HIDE = 0
Private Const EM_SETSEL = &HB1
Private Const EM_REPLACESEL = &HC2
Private hStdOut As Long, hStdIn As Long 'pipes for use by the Shell
Private hRead As Long 'Pipes used by us ;/
Private hWrite As Long
Private ProcessAttrib As SECURITY_ATTRIBUTES
Public Function CreateCMD()
Dim startInfo As STARTUPINFO
Dim lngPipe1ret As Long, lngPipe2ret As Long
Dim lngProcessRet As Long

'Create Two Pipes
pipeAttrib.nLength = Len(pipeAttrib)
pipeAttrib.lpSecurityDescriptor = 0
pipeAttrib.bInheritHandle = True
lngPipe1ret = CreatePipe(hRead, hStdOut, pipeAttrib, 0)     'hStdOut -> hRead
lngPipe2ret = CreatePipe(hStdIn, hWrite, pipeAttrib, 0)       'hWrite -> hStdIn

'make sure pipes were created
If (lngPipe1ret = 0) Or (lngPipe2ret = 0) Then
    CreateCMD = -2  'return -2 and exit
    Exit Function
End If

'Set Startup Info
startInfo.cb = Len(startInfo)
GetStartupInfo startInfo

'Set Pipes
startInfo.hStdOutput = hStdOut
startInfo.hStdError = hStdOut
startInfo.hStdInput = hStdIn

'Set Flags
startInfo.wShowWindow = SW_HIDE

'Create the process
ProcessAttrib.nLength = Len(ProcessAttrib)
ThreadAttrib.nLength = Len(ThreadAttrib)
lngProcessRet = CreateProcess(vbNullString, "program.exe", ProcessAttrib, ThreadAttrib, True, 0, Null, vbNullString, startInfo, processInfo)
If lngProcessRet > 0 Then
    CreateCMD = ProcessInfo.dwProcessId
    CreateCMD = -1
End If
End Function

Public Function ReadCMD() As Byte()
Dim lngRet As Long
Dim lngBytesRead As Long
Dim byteBuff(1024) As Byte
lngRet = ReadFile(hRead, byteBuff(0), 1023, lngBytesRead, 0&)
End Function

As of now the CreateCMD works, however, when I do a readCMD it reads the data, but right after that my program freezes.

Here is the code I am using in my form:

Private cmdShell As New cmd
Private hasShell As Boolean

Private Sub Form_Load()
Dim retCode As Long
retCode = cmdShell.CreateCMD
If retCode > 0 Then hasShell = True
End Sub

Private Sub Timer1_Timer()
Dim byteBuff(1024) As Byte
If hasShell = True Then
    byteBuff(0) = cmdShell.ReadCMD
   ' Text1.Text = Text1.Text & byteBuff()  <--- dont pay attention to this, just for testing
End If
End Sub

if I disable the timer after the first call of ReadCmd it wont freeze on me, however, I need to be able to let the program to hang out until the user tells it what to do.

Join the community to see this answer!
Join our exclusive community to see this answer & millions of others.
Unlock 1 Answer and 7 Comments.
Join the Community
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 7 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros