[Webinar] Streamline your web hosting managementRegister Today

x
?
Solved

Need help with System.Diagnostics.Process.StandardOutput buffer size.

Posted on 2007-07-25
12
Medium Priority
?
1,764 Views
Last Modified: 2008-10-16
Hi I'm using vs2005, .net 2 for windows application.  The application I started using System.Diagnostics.Process is having a "listFiles.StandardOutput" buffer size problem.  I was wondering where and how can I adjust the buffer size problem.  I don't want to be limited to a set buffer size, is that possible?  thank you.


public static int GetNisFile(System.Diagnostics.ProcessStartInfo psi,
    ref DataTable dtAccounts,
    ref ToolStripStatusLabel slMain, ref StatusStrip ssMain, string userOrGroup)
{

    string accountFile="", output="";
    int numOfAccounts = 0;
    System.IO.StreamReader fileOutput = null, myOutput = null;
    System.Diagnostics.Process listFiles;
    try
    {
        listFiles = System.Diagnostics.Process.Start(psi);
        if (listFiles != null)
        {
            myOutput = listFiles.StandardError;
            fileOutput = listFiles.StandardOutput;
            listFiles.WaitForExit(30000);
            if (listFiles.HasExited)
            {
            // Read and display lines from the file until the end of
            // the file is reached.
                accountFile = fileOutput.ReadToEnd();
                output = myOutput.ReadToEnd();
                if (accountFile != null && accountFile != "")
                    numOfAccounts = CPAUnix.ParseAndInsertAccounts(accountFile,
                        ref dtAccounts, ref slMain, ref ssMain, userOrGroup);
            }
            else
                MessageBox.Show("Time out retrieving NIS" + userOrGroup + "accounts.",
                    "PowerADvantage");
        }
        else
            MessageBox.Show("Error starting process to read file.", "PowerADvantage");

        return numOfAccounts;

    }
    catch (Win32Exception wex)
    {
        MessageBox.Show(wex.Message, "PowerADvantage");
        return numOfAccounts;
    }
}
0
Comment
Question by:lapucca
  • 7
  • 4
12 Comments
 
LVL 2

Expert Comment

by:Livanta
ID: 19570010
From what i remember when implementing the RedirectStandardOutput or RedirectStandardInput in ProcessStartInfo, you must implement all 3 of them. You have to set

System.Diagnostics.ProcessStartInfo proc = new System.Diagnostics.ProcessStartInfo;
proc.RedirectStandardError = true;
proc.RedirectStandardInput = true;
proc.RedirectStandardOutput = true;
0
 

Author Comment

by:lapucca
ID: 19570178
I tried it but problem still exist.  I'm pretty sure it's the buffer size problem in reading the StandardInput.  Thanks.
0
 
LVL 7

Expert Comment

by:illusio
ID: 19572675
This is very possible - the STDOUT has only a limited buffer after all.
You fill the buffer (possibly drown it) and when everything is over you read everything at once.
Have you already tried to read with BeginRead and EndRead (thus asynchronously) while processing is busy? This will empty the buffer at the speed it's filling...

Kind regards,
Peter
0
The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

 

Author Comment

by:lapucca
ID: 19576679
Hi Peter,
I tried the readline and then discardbuffer until endofstream but not working.  Can you show me the code for BeginRead and EndRead?  Thank you.
0
 
LVL 7

Accepted Solution

by:
illusio earned 2000 total points
ID: 19582102
For the beginread function you use the basestream on the streamreader but  i searched a little and came up with following solution which is far simpler.
        static void Main()
        {
            Process proc = new Process();
            proc.OutputDataReceived += new DataReceivedEventHandler(proc_OutputDataReceived);
            proc.StartInfo = new ProcessStartInfo("cmd", @"/C type C:\testfile.txt");
            proc.StartInfo.UseShellExecute = false;
            proc.StartInfo.RedirectStandardOutput = true;
            if (proc.Start())
            {
                proc.BeginOutputReadLine();
                if (proc.WaitForExit(3000))
                    string theResult = sb.ToString()
            }
        }

        static System.Text.StringBuilder sb = new System.Text.StringBuilder();
        static void proc_OutputDataReceived(object sender, DataReceivedEventArgs e)
        {
            sb.Append(e.Data);
        }

Can you let met know if this helps you out?

Peter
0
 

Author Comment

by:lapucca
ID: 19621971
I tried using the  proc.BeginOutputReadLine();
 but I'm still only getting like the first 5 lines of the data.  There are still about 15 lines missing.  I think even with this the buffer size is still limited.
0
 
LVL 7

Expert Comment

by:illusio
ID: 19623257
hmm, strange, I retested the program and threw a 5 Mb long file through it (using the code above) and it worked as a charm. I changed one block and one line for my test:
  if (proc.WaitForExit(10000))
  {
    Console.WriteLine("process is ended");
    System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\resfile.txt", false, System.Text.Encoding.Default);
    sw.Write(sb.ToString());
    sw.Close();
  }
  else
    Console.WriteLine("process not properly ended");

And changed:
  sb.Append(e.Data);  -->>> sb.AppendLine(e.Data);

The writing is to be able to verify more quickly and the exit timeout, I increased because otherwise the program quitted before all the lines where received. The appendline is because the linebreaks are not included in the OutputDataReceived event.
Are you processing your input while receiving the data or are you waiting for everything to finish before you start processing? Best is buffer your data and process it afterwards. And if you have to do it "on-the-fly", queue the data and use a seperate thread to process. Otherwise the data might be comming in faster then you can process.
Are the lines you are receiving (amount of text till you reach a CR/LF or LF) not too long?

I will continue trying to help!

Kind regards,
Peter


0
 

Author Comment

by:lapucca
ID: 19628715
Here is my code using BeginOutputReadLine.  This code doesn't have the time
out issue but it only returns 5 lines of data.  What am I doing different here?  I tried running the ypcat.exe (the c code application) that gives the output line by line and it does get all the data.    Thank you.

static System.Text.StringBuilder sb = new System.Text.StringBuilder();
static void proc_OutputDataReceived(object sender, DataReceivedEventArgs e)
{
    sb.Append(e.Data);
}

public static int GetNisFile(System.Diagnostics.ProcessStartInfo psi,
    ref DataTable dtAccounts,
    ref ToolStripStatusLabel slMain, ref StatusStrip ssMain, string
userOrGroup)
{

    string accountFile = "", output = "", line = "", theResult="";
    int numOfAccounts = 0;
    System.IO.StreamReader fileOutput = null, myOutput = null;
    System.Diagnostics.Process listFiles = new Process();
    listFiles.StartInfo = psi;
    try
    {
        String inputText;
        int numInputLines = 0;
        listFiles.OutputDataReceived += new
DataReceivedEventHandler(proc_OutputDataReceived);
        //listFiles = System.Diagnostics.Process.Start(psi);
        // Use a stream writer to synchronously write the sort input.
        listFiles.Start();
        //StreamReader nisReader = listFiles.StandardOutput;
        listFiles.BeginOutputReadLine();
        listFiles.WaitForExit();
        theResult = sb.ToString();
        listFiles.CancelOutputRead();
        listFiles.Kill();
        return numOfAccounts;

    }
    catch (Win32Exception wex)
    {
        MessageBox.Show(wex.Message, "PowerADvantage");
        return numOfAccounts;
    }
}
0
 
LVL 7

Expert Comment

by:illusio
ID: 19637197
The code looks ok.
The lines listFile.CancelOutputRead()  and listFiles.Kill() are not necessary for a far as i can see.
better use: listFile.Close();

For a better understanding: those 5 lines are those the first 5 or the last 5? Or some lines that go in between. Further, can you also couple the ErrorStream and see if nothing if coming through there?
Are there special characters in the stream (EOF's or other control characters)?

Meanwhile I will create a sample that uses asynchronous reading and doesn't rely on the event (although I doubt that is the problem).

Kind regards,
Peter

0
 

Author Comment

by:lapucca
ID: 19639940
Also, I forgot to mention that I'm running on Windows 2000 server.  Would this be the problem?
0
 

Author Comment

by:lapucca
ID: 19640246
I just ran the C application to verify, and the 5 lines read are the FIRST 5 lines of data.  Thank you.
0
 

Author Comment

by:lapucca
ID: 19642011
Hi Pete,
thank you for all your help.  It's now working.  The eventhandler was stripping the \n (in String.Append) and through my code it was retuning 0 records when it didn't find \n.  But the process redirect is wokring properly.    Many thanks again.

0

Featured Post

Take Control of Web Hosting For Your Clients

As a web developer or IT admin, successfully managing multiple client accounts can be challenging. In this webinar we will look at the tools provided by Media Temple and Plesk to make managing your clients’ hosting easier.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article introduced a TextBox that supports transparent background.   Introduction TextBox is the most widely used control component in GUI design. Most GUI controls do not support transparent background and more or less do not have the…
High user turnover can cause old/redundant user data to consume valuable space. UserResourceCleanup was developed to address this by automatically deleting user folders when the user account is deleted.
With just a little bit of  SQL and VBA, many doors open to cool things like synchronize a list box to display data relevant to other information on a form.  If you have never written code or looked at an SQL statement before, no problem! ...  give i…
Enter Foreign and Special Characters Enter characters you can't find on a keyboard using its ASCII code ... and learn how to make a handy reference for yourself using Excel ~ Use these codes in any Windows application! ... whether it is a Micr…
Suggested Courses
Course of the Month9 days, 13 hours left to enroll

591 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question