asked on

Need help with System.Diagnostics.Process.StandardOutput buffer size.

Hi I'm using vs2005, .net 2 for windows application. The application I started using System.Diagnostics.Process is having a "listFiles.StandardOutput" buffer size problem. I was wondering where and how can I adjust the buffer size problem. I don't want to be limited to a set buffer size, is that possible? thank you.

public static int GetNisFile(System.Diagnostics.ProcessStartInfo psi,
ref DataTable dtAccounts,
ref ToolStripStatusLabel slMain, ref StatusStrip ssMain, string userOrGroup)
{

string accountFile="", output="";
int numOfAccounts = 0;
System.IO.StreamReader fileOutput = null, myOutput = null;
System.Diagnostics.Process listFiles;
try
{
listFiles = System.Diagnostics.Process.Start(psi);
if (listFiles != null)
{
myOutput = listFiles.StandardError;
fileOutput = listFiles.StandardOutput;
listFiles.WaitForExit(30000);
if (listFiles.HasExited)
{
// Read and display lines from the file until the end of
// the file is reached.
accountFile = fileOutput.ReadToEnd();
output = myOutput.ReadToEnd();
if (accountFile != null && accountFile != "")
numOfAccounts = CPAUnix.ParseAndInsertAccounts(accountFile,
ref dtAccounts, ref slMain, ref ssMain, userOrGroup);
}
else
MessageBox.Show("Time out retrieving NIS" + userOrGroup + "accounts.",
"PowerADvantage");
}
else
MessageBox.Show("Error starting process to read file.", "PowerADvantage");

return numOfAccounts;

}
catch (Win32Exception wex)
{
MessageBox.Show(wex.Message, "PowerADvantage");
return numOfAccounts;
}
}

Livanta

From what i remember when implementing the RedirectStandardOutput or RedirectStandardInput in ProcessStartInfo, you must implement all 3 of them. You have to set

System.Diagnostics.ProcessStartInfo proc = new System.Diagnostics.ProcessStartInfo;
proc.RedirectStandardError = true;
proc.RedirectStandardInput = true;
proc.RedirectStandardOutput = true;

lapucca

ASKER

I tried it but problem still exist. I'm pretty sure it's the buffer size problem in reading the StandardInput. Thanks.

illusio

This is very possible - the STDOUT has only a limited buffer after all.
You fill the buffer (possibly drown it) and when everything is over you read everything at once.
Have you already tried to read with BeginRead and EndRead (thus asynchronously) while processing is busy? This will empty the buffer at the speed it's filling...

Kind regards,
Peter

lapucca

ASKER

Hi Peter,
I tried the readline and then discardbuffer until endofstream but not working. Can you show me the code for BeginRead and EndRead? Thank you.

ASKER CERTIFIED SOLUTION

illusio

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

lapucca

ASKER

I tried using the proc.BeginOutputReadLine();
but I'm still only getting like the first 5 lines of the data. There are still about 15 lines missing. I think even with this the buffer size is still limited.

illusio

hmm, strange, I retested the program and threw a 5 Mb long file through it (using the code above) and it worked as a charm. I changed one block and one line for my test:
if (proc.WaitForExit(10000))
{
Console.WriteLine("process is ended");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\resfile.txt", false, System.Text.Encoding.Default);
sw.Write(sb.ToString());
sw.Close();
}
else
Console.WriteLine("process not properly ended");

And changed:
sb.Append(e.Data); -->>> sb.AppendLine(e.Data);

The writing is to be able to verify more quickly and the exit timeout, I increased because otherwise the program quitted before all the lines where received. The appendline is because the linebreaks are not included in the OutputDataReceived event.
Are you processing your input while receiving the data or are you waiting for everything to finish before you start processing? Best is buffer your data and process it afterwards. And if you have to do it "on-the-fly", queue the data and use a seperate thread to process. Otherwise the data might be comming in faster then you can process.
Are the lines you are receiving (amount of text till you reach a CR/LF or LF) not too long?

I will continue trying to help!

Kind regards,
Peter

lapucca

ASKER

Here is my code using BeginOutputReadLine. This code doesn't have the time
out issue but it only returns 5 lines of data. What am I doing different here? I tried running the ypcat.exe (the c code application) that gives the output line by line and it does get all the data. Thank you.

static System.Text.StringBuilder sb = new System.Text.StringBuilder();
static void proc_OutputDataReceived(object sender, DataReceivedEventArgs e)
{
sb.Append(e.Data);
}

public static int GetNisFile(System.Diagnostics.ProcessStartInfo psi,
ref DataTable dtAccounts,
ref ToolStripStatusLabel slMain, ref StatusStrip ssMain, string
userOrGroup)
{

string accountFile = "", output = "", line = "", theResult="";
int numOfAccounts = 0;
System.IO.StreamReader fileOutput = null, myOutput = null;
System.Diagnostics.Process listFiles = new Process();
listFiles.StartInfo = psi;
try
{
String inputText;
int numInputLines = 0;
listFiles.OutputDataReceived += new
DataReceivedEventHandler(proc_OutputDataReceived);
//listFiles = System.Diagnostics.Process.Start(psi);
// Use a stream writer to synchronously write the sort input.
listFiles.Start();
//StreamReader nisReader = listFiles.StandardOutput;
listFiles.BeginOutputReadLine();
listFiles.WaitForExit();
theResult = sb.ToString();
listFiles.CancelOutputRead();
listFiles.Kill();
return numOfAccounts;

}
catch (Win32Exception wex)
{
MessageBox.Show(wex.Message, "PowerADvantage");
return numOfAccounts;
}
}

illusio

The code looks ok.
The lines listFile.CancelOutputRead() and listFiles.Kill() are not necessary for a far as i can see.
better use: listFile.Close();

For a better understanding: those 5 lines are those the first 5 or the last 5? Or some lines that go in between. Further, can you also couple the ErrorStream and see if nothing if coming through there?
Are there special characters in the stream (EOF's or other control characters)?

Meanwhile I will create a sample that uses asynchronous reading and doesn't rely on the event (although I doubt that is the problem).

Kind regards,
Peter

lapucca

ASKER

Also, I forgot to mention that I'm running on Windows 2000 server. Would this be the problem?

lapucca

ASKER

I just ran the C application to verify, and the 5 lines read are the FIRST 5 lines of data. Thank you.

lapucca

ASKER

Hi Pete,
thank you for all your help. It's now working. The eventhandler was stripping the \n (in String.Append) and through my code it was retuning 0 records when it didn't find \n. But the process redirect is wokring properly. Many thanks again.