asked on
C# SFTP program hangs under task scheduler
I have a C# program that grabs a file each day off an SFTP server. It was running fine for months. It is run as a scheduled task under Task Scheduler. Over the past couple of weeks the task is left hanging when normally it runs in under a minute. I am beginning to wonder if perhaps when this code tries to connect to the SFTP it fails and gets stuck. Does this seem plausible? What else would cause the scheduled task to just hang? If I terminate the task and run it manually the transfer goes right through.
try
{
// Connect
sftpClient = new Renci.SshNet.SftpClient("bolagna.sandwich.io", "my_sftp", "password");
sftpClient.Connect();
// List the contents of a given folder and loop through them
var dirContents = sftpClient.ListDirectory("./upload").ToArray();
foreach (var entry in dirContents)
{
// Check to see if the base filename of the remote file contains "hm_orders_"
if (entry.Name.Contains("hm_orders_"))
{
Do Work
}
}
}
I have always only used WinSCP .NET without any troubles.
And for your actual code: There are different levels, where I would use separate methods, thus separate error handling. E.g., something like this:
using Renci.SshNet;
private bool Connect(string hostNameOrIP, string userName, string password, out SftpClient sftpClient)
{
try
{
sftpClient = new SftpClient(hostNameOrIP, userName, password);
sftpClient.Connect();
return true;
}
catch (Exception exception)
{
// TODO: Log error.
return false;
}
}
private bool ListDirectory(SftpClient sftpClient, string directory, out useCorrectDataType directoryContent)
{
try
{
directoryContent = sftpClient.ListDirectory(directory).ToArray();
return true;
}
catch (Exception exception)
{
// TODO: Log error.
return false;
}
}
private bool ProcessFile(useCorrectDataType directoryEntry)
{
try
{
// Process entry.
return true;
}
catch (Exception exception)
{
// TODO: Log error.
return false;
}
}
SftpClient sftpClient = null;
useCorrectDataType directoryContent = null;
if Connect("bolagna.sandwich.io", "my_sftp", "password", out sftpClient)
{
if ListDirectory(sftClient, "./upload", out directoryContent)
{
foreach (useCorrectDataType directoryEntry in directoryContent)
{
if (directoryEntry.Name.Contains("hm_orders_"))
{
ProcessFile(directoryEntry)
}
}
}
}
Depending on your used SSH library, the directory parameter for the list method does already take a pattern. Then you don't need the if.
ASKER
Good points. I was going to add the logging next. I am just perplexed as this had been running fine since the fall of 2021 and within the past two weeks this pops up but not all the time. I need to find where the hanging begins. I will rearrange some of the code.
ASKER
ste5an : What is
useCorrectDataType
ASKER
I will defer to whatever you feel is best. The code needs to go to the SFTP server and look for any files with the specified prefix on the day the program is run. The filename returned is the most recent file. So given that it needs to be an array of file names, what is the correct type? Here is what I have so far. I am open to any suggestions.
ASKER
I tried this to find out the type but it comes back as "No such host is known"
How do I find out what type this is?
var dirContents = sftpClient.ListDirectory("./upload").ToArray();
MessageBox.Show("The type of dircontents is : " + dirContents.GetType());
The return type for ListDirectory on Renci SSH / SFTPClient is an IENumerable of "SftpFile"s. You're converting it to an array, so the variable type of dirContents is SftpFile[].
ASKER
I found that this was the final solution. I searched the internet for the return type of sftpClient.ListDirectory(directory).ToArray() and was not able to find it. Where did you find it?
private bool ListDirectory(SftpClient sftpClient, string directory, out Renci.SshNet.Sftp.SftpFile[] directoryContent)
{
try
{
directoryContent = sftpClient.ListDirectory(directory).ToArray();
return true;
}
catch (Exception ex)
{
// TODO: Log error.
string eMsg = "Error - 002: Cannot ListDirectory";
MessageBox.Show(eMsg + "\n" + ex);
directoryContent = null;
return false;
}
}
I have that library on my PC with an old project that uses it so I just glanced at the old code. I'm sure it's in the documentation somewhere, too.
ASKER
Ste5an:
Changing my code to your example I am getting errors on ListDirectory, foreach and ProcesFile.
ListDirectory - Object reference is required for the non-static field
foreach - Cannot convert type Renci.SshNet.Sftp.SftpFile to Renci.SshNet.Sftp.SftpFile[]
ProcessFile - Object reference is required for the non-static field
public static void DownloadSalesOrderFile(string dataDrive)
{
SftpClient sftpClient = null;
Renci.SshNet.Sftp.SftpFile[] directoryContent = null;
if (Connect(Globals.SFTP_HostName, Globals.SFTP_UserName, Globals.SFTP_Password, out sftpClient)
{
if (ListDirectory(sftpClient, "./upload", out directoryContent))
{
foreach (Renci.SshNet.Sftp.SftpFile[] directoryEntry in directoryContent)
{
if (directoryEntry.Name.Contains("hm_orders_"))
{
ProcessFile(directoryEntry);
}
}
}
// Disconnect at the end if needed
if ((sftpClient != null) && (sftpClient.IsConnected))
{
sftpClient.Disconnect();
}
}
}
ASKER
OK, I resolved the object reference messages but this one still remains:
foreach (Renci.SshNet.Sftp.SftpFile[] directoryEntry in directoryContent)
'Cannot convert type Renci.SshNet.Sftp.SftpFile to Renci.SshNet.Sftp.SftpFile[]'
Why is one array(directoryContent) and the other is not(directoryEntry)?
ASKER
Here is the current code. I am going to revert to an earlier working version for now until I can resolve this 'convert type' issue.
using Renci.SshNet;
public class SftpHelper
{
private IEnumerable<SftpFile> remoteFiles;
private SftpClient sftpClient;
public static bool DownloadFiles(string hostName, string userName, string password, string remoteDirectory, string localDirectory, Func<SftpFile, bool> acceptFile)
{
bool result = false;
SftpHelper sftpHelper = new SftpHelper()
if (sftpHelper.Connect(hostName, userName, password))
{
if sftpHelper.ListDirectory(remoteDirectory)
result = sftpHelper.DownloadFiles(localDirectory, acceptFile);
sftpHelper.Disconnect();
}
return result;
}
private bool Connect(string hostName, string userName, string password)
{
try
{
this.sftpClient = new SftpClient(hostName, userName, password);
this.sftpClient.Connect();
return true;
}
catch (Exception exception)
{
MessageBox.Show($"Cannot connect to server '{hostName}':\n" + exception);
this.sftpClient = null;
return false;
}
}
private bool Disconnect()
{
try
{
this.sftpClient.Disconnect();
this.sftpClient = null;
this.remoteFiles = null;
return true;
}
catch (Exception exception)
{
MessageBox.Show("Cannot disconnect from server:\n" + exception);
return false;
}
}
private bool ListDirectory(string remoteDirectory)
{
try
{
this.remoteFiles = this.sftpClient.ListDirectory(remoteDirectory);
return true;
}
catch (Exception exception)
{
MessageBox.Show($"Cannot list remote directory '{remoteDirectory}':\n" + exception);
return false;
}
}
private bool DownloadFiles(string localDirectory, Func<SftpFile, bool> acceptFile)
{
foreach (SftpFile remoteFile in this.remoteFiles)
{
if (acceptFile(remoteFile))
{
if (!this.DownloadFile(remoteFile, localDirectory))
return false;
}
}
return true;
}
private DownloadFile(SftpFile remoteFile, string localDirectory)
{
try
{
// The actual download code for the Renci client.
return true;
}
catch (Exception exception)
{
MessageBox.Show($"Cannot download remote file '{remoteFile.Name}' to '{localDirectory}' :\n" + exception);
return false;
}
}
}
public class YourForm: Form
{
// Your other form code..
public static void DownloadSalesOrderFile(string dataDrive)
{
// Having dataDrive as parameter and hardcoding the partial path to the final local directory is suspicious.
string localDirectory = dataDrive + @"\GPShares\ImportFiles\SOPImport\";
if (SftpHelper.DownloadFiles(
Globals.SFTP_HostName,
Globals.SFTP_UserName,
Globals.SFTP_Password,
"./upload",
localDirectory,
f => f.Name.Contains("hm_orders_")
))
MessageBox.Show("Success.");
}
}
p.s. skip the array cast. Use the IEnumerable directly.
p.p.s. as you run it in Task Scheduler, this program should be a console application. And instead of using MessageBox.Show() you need to log the results to a log file.
p.p.p.s. when to solely purpose of this application is to download the files, then I would use a PowerShell script or CommandShell batch using WinScp. This would be easier to maintain and especially debug as you always could check with the WinScp UI what happens.
Just responding to this:
foreach (Renci.SshNet.Sftp.SftpFile[] directoryEntry in directoryContent)
'Cannot convert type Renci.SshNet.Sftp.SftpFile to Renci.SshNet.Sftp.SftpFile[]'
Why is one array(directoryContent) and the other is not(directoryEntry)?
A foreach will loop through a collection (e.g. array) one item at a time.
The [] indicates an array. For example, this creates an array of strings:
var arrayOfStrings = new string[] { "apple", "orange" };
If I did a foreach loop through this array of strings, I would get one string at a time.
foreach(string oneString in arrayOfStrings)
{
}
Similarly, your previous code was getting the directory listing as an array of SftpFile objects, so the output of ListDirectory(blah blah) was a special type of collection called an IEnumerable. It's essentially the most basic type of collection there is in .NET.
You then converted that IEnumerable to an array with the ToArray() code, so that meant you have an array of SftpFiles, or SftpFile[].
If you were looping through that array, you'd get one SftpFile at a time.
foreach(SftpFile entry in directoryContent)
{
}
Lots of us just use the var keyword shortcut which lets .NET figure out the type when it goes to compile things. It's shorter and easier in most cases.
foreach(var entry in directoryContent)
{
}
Lots of us just use the var keyword shortcut which lets .NET figure out the type when it goes to compile things. It's shorter and easier in most cases.But it makes it a lot harder to refactor it with paper and pencil, when you don't know the used third-party library ;)
True, although I would say that we should write code with the goal of easy maintenance by people who have access to the code. Writing code for the sake of a 3rd party who doesn't have access to your code is an edge case.
If someone has access to the code, they PROBABLY have Visual Studio and can just hover over the variable to see the type, if there's any confusion:
On a side note - as I went to go grab a screenshot of this code, I realized that the code snippet that rwheeler23 is using is from me. I gave them a code snippet on using this SftpClient a while ago (back in September 2021, it seems), so I was the one who added the ToArray(). I'm not sure why I did that. You're correct that IEnumerable is fine as-is. Oops!
ASKER
We use PowerShell for all of our exports. The code you are seeing here is part of a much larger application. It is called via Task Scheduler using arguments indicating what type of files need to be imported. That is why you see 'hm_orders'. We are sent various files some of which there is one per day and others six per day. The messagebox calls were a typo. I will review what was recently sent and incorporate it into this program. A big part of this code is to grab the most recent version of the file on any given day.
ASKER
For what it is worth here is the code I now have for for the Connect method.
I am outputting to a text file. This because we need to see if there is a failure where does the failure occur.
ASKER
You folks are full of great ideas. Speaking to Ste5an's point, datadrive is in the config file. I should also put the path to each type of file in there as well in the event the locations change.
ASKER
OK, one last question on this thread. This code is now in place.
public static void DownloadSalesOrderFile()
{
string localDirectory = Globals.DataDrive.Trim() + Globals.SOPImportPath.Trim(); ;
if (SftpHelper.DownloadFiles(
Globals.SFTP_HostName,
Globals.SFTP_UserName,
Globals.SFTP_Password,
"./upload",
localDirectory,
f => f.Name.Contains("hm_orders_")
))
Console.WriteLine("Success - File Imported.");
}
==================================================================================
Which leads to this where I will insert my previously existing code
private bool DownloadFile(SftpFile remoteFile, string localDirectory)
{
try
{
// The actual download code for the Renci client.
return true;
}
catch (Exception exception)
{
Console.WriteLine($"Cannot download remote file '{remoteFile.Name}' to '{localDirectory}' :\n" + exception);
return false;
}
}
====================================================================================
What my code needs to do is loop through all the files found in this directory and grab only the most recent file.
I was doing this for the loop:
foreach (var entry in dirContents)
{
do stuff by going through list and determine file with most recent date
}
What do I loop through now? Or do I get the most recent file someplace else?
ASKER
Please let me now how to wrap this up. I really appreciate everyone's help on this and I would think we are almost there. How do I get the most recent file?
using System;
using System.Collections.Generic;
using System.Linq;
using Renci.SshNet;
public class SftpHelper
{
private IEnumerable<SftpFile> remoteFiles;
private SftpClient sftpClient;
public static bool DownloadFiles(string hostName, string userName, string password, string remoteDirectory, string localDirectory, Func<IEnumerable<SftpFile>, IEnumerable<SftpFile>> acceptFiles)
{
bool result = false;
SftpHelper sftpHelper = new SftpHelper()
if (sftpHelper.Connect(hostName, userName, password))
{
if sftpHelper.ListDirectory(remoteDirectory)
{
this.remoteFiles = acceptFiles(this.remoteFiles);
result = sftpHelper.DownloadFiles(localDirectory);
}
sftpHelper.Disconnect();
}
return result;
}
private bool Connect(string hostName, string userName, string password)
{
try
{
this.sftpClient = new SftpClient(hostName, userName, password);
this.sftpClient.Connect();
return true;
}
catch (Exception exception)
{
Console.WriteLine($"Cannot connect to server '{hostName}':\n" + exception);
this.sftpClient = null;
return false;
}
}
private bool Disconnect()
{
try
{
this.sftpClient.Disconnect();
this.sftpClient = null;
this.remoteFiles = null;
return true;
}
catch (Exception exception)
{
Console.WriteLine("Cannot disconnect from server:\n" + exception);
return false;
}
}
private bool ListDirectory(string remoteDirectory)
{
try
{
this.remoteFiles = this.sftpClient.ListDirectory(remoteDirectory);
return true;
}
catch (Exception exception)
{
Console.WriteLine($"Cannot list remote directory '{remoteDirectory}':\n" + exception);
return false;
}
}
private bool DownloadFiles(string localDirectory)
{
foreach (SftpFile remoteFile in this.remoteFiles)
{
if (!this.DownloadFile(remoteFile, localDirectory))
return false;
}
return true;
}
private DownloadFile(SftpFile remoteFile, string localDirectory)
{
try
{
// The actual download code for the Renci client.
return true;
}
catch (Exception exception)
{
Console.WriteLine($"Cannot download remote file '{remoteFile.Name}' to '{localDirectory}' :\n" + exception);
return false;
}
}
}
public class Programm
{
// Your other console code..
public static void DownloadSalesOrderFile(string dataDrive)
{
string localDirectory = dataDrive + @"\GPShares\ImportFiles\SOPImport\";
if (SftpHelper.DownloadFiles(
Globals.SFTP_HostName,
Globals.SFTP_UserName,
Globals.SFTP_Password,
"./upload",
localDirectory,
remoteFiles =>
remoteFiles
.Where(remoteFile => remoteFile.Name.Contains("hm_orders_"))
.OrderBy(remoteFile => remoteFile.LastModified)
.Take(1)
))
Console.WriteLine("Success.");
}
}
ASKER
This is failing on this:
remoteFiles =>
remoteFiles
.Where(remoteFile => remoteFile.Name.Contains("hm_orders_"))
.OrderBy(remoteFile => remoteFile.LastModified)
.First()
The message is Sftpfile does not contain a definition for Where.
Add "using System.Linq;" to the top of your code.
ASKER
using System.Linq; was already there.
ASKER
SftpHelper.DownloadFiles is called with 6 arguments in Main but in SftHelper it has seven arguments.
I am getting a 'Cannot be null' error message.
===================================================================================
public static bool DownloadFiles(string hostName, string userName, string password, string remoteDirectory, string localDirectory, Func<IEnumerable<SftpFile>, IEnumerable<SftpFile>> acceptFiles)
{
bool result = false;
SftpHelper sftpHelper = new SftpHelper();
if (sftpHelper.Connect(hostName, userName, password))
{
if (sftpHelper.ListDirectory(remoteDirectory))
{
sftpHelper.remoteFiles = acceptFiles(sftpHelper.remoteFiles);
result = sftpHelper.DownloadFiles(localDirectory);
}
sftpHelper.Disconnect();
}
return result;
}
=====================================================================================
class Program
{
static void Main(string[] args)
{
string localDirectory = @"\GPShares\ImportFiles\SOPImport\";
if (SftpHelper.DownloadFiles(
"Globals.SFTP_HostName",
"Globals.SFTP_UserName",
"Globals.SFTP_Password",
"./upload",
localDirectory,
remoteFiles =>
remoteFiles
.Where(remoteFile => remoteFile.Name.Contains("hm_orders_"))
.OrderBy(remoteFile => remoteFile.LastWriteTime)
.Take(1)
))
Console.WriteLine("Success.");
Console.WriteLine("Press any key to quit.");
Console.ReadLine();
}
}
ASKER
Ste5an: I decided to completely rewrite the code based on your code. That cleared up several issues. Now I am stuck here:
public static bool DownloadFiles(string hostName, string userName, string password, string remoteDirectory, string localDirectory, Func<IEnumerable<SftpFile>, IEnumerable<SftpFile>> acceptFiles)
{
bool result = false;
SftpHelper sftpHelper = new SftpHelper(); <<<<--- It crashes here. The message is 'Value Cannot be null'
if (sftpHelper.Connect(hostName, userName, password))
{
sftpHelper.remoteFiles = acceptFiles(sftpHelper.remoteFiles);
result = sftpHelper.DownloadFiles(localDirectory);
sftpHelper.Disconnect();
}
return result;
}
=================================================================================
I am going to wrap this inside a try/catch to get further information.
ASKER
After taking a few hours off I came back to discovered a typo. This code now runs. The next step I need to accomplish is inserting my old code to actually transfer the file. That will go into the ProcessFile method.
ASKER
Running the code is returning the oldest file. I need it to return the most recent file.
The old code would actually limit the date to today's date and then find the latest version.
public static void DownloadSalesOrderFile()
{
string localDirectory = Globals.SOPImportPath.Trim();
if (SftpHelper.DownloadFiles(
Globals.SFTP_HostName.Trim(),
Globals.SFTP_UserName.Trim(),
Globals.SFTP_Password.Trim(),
"./upload",
localDirectory,
remoteFiles =>
remoteFiles
.Where(remoteFile => remoteFile.Name.StartsWith("hm_orders_"))
.OrderBy(remoteFile => remoteFile.LastWriteTime)
.Take(1)
))
return;
Console.WriteLine("Success - File Imported.");
}
ASKER
I was able to limit the file(s) to today's date but how do I get only the most recent file of the day?
public static void DownloadSalesOrderFile()
{
string localDirectory = Globals.SOPImportPath.Trim();
if (SftpHelper.DownloadFiles(
Globals.SFTP_HostName.Trim(),
Globals.SFTP_UserName.Trim(),
Globals.SFTP_Password.Trim(),
"./upload",
localDirectory,
remoteFiles =>
remoteFiles
.Where(remoteFile => remoteFile.Name.StartsWith("hm_orders_") && remoteFile.Attributes.LastWriteTime.Date == DateTime.Now.Date)
.OrderBy(remoteFile => remoteFile.LastWriteTime)
.Take(1)
))
return;
Console.WriteLine("Success - File Imported.");
}
ASKER
I think this will do it.
public static void DownloadSalesOrderFile()
{
string localDirectory = Globals.SOPImportPath.Trim();
if (SftpHelper.DownloadFiles(
Globals.SFTP_HostName.Trim(),
Globals.SFTP_UserName.Trim(),
Globals.SFTP_Password.Trim(),
"./upload",
localDirectory,
remoteFiles =>
remoteFiles
.Where(remoteFile => remoteFile.Name.StartsWith("hm_orders_") && remoteFile.Attributes.LastWriteTime.Date == DateTime.Now.Date)
.OrderByDescending(remoteFile => remoteFile.LastWriteTime.TimeOfDay)
.Take(1)
))
return;
Console.WriteLine("Success - File Imported.");
}
ASKER
ste5an and gr8gonzo, thank you both for your help providing a solution. I appreciate all the time you spent providing assistance. I will award as many points as possible plus be willing to provide any recommendations to Experts Exchange so they know the high level of assistance and competence you both displayed.
By the way, here the final small piece of code I had to add to get the file to transfer.
private bool DownloadFile(SftpFile remoteFile, string localDirectory)
{
try
{
Console.WriteLine(remoteFile.Name);
Console.WriteLine(localDirectory.Trim());
using (FileStream localfile = new FileStream(localDirectory.Trim()+ remoteFile.Name.Trim(), FileMode.Create))
{
sftpClient.DownloadFile("./upload/" + remoteFile.Name.Trim(), localfile);
}
// The actual download code for the Renci client.
return true;
}
catch (Exception exception)
{
Console.WriteLine($"Cannot download remote file '{remoteFile.Name}' to '{localDirectory}' :\n" + exception);
return false;
}
}
private bool DownloadFile(SftpFile remoteFile, string localDirectory)
{
try
{
string localFileName = localDirectory.Trim()+ remoteFile.Name.Trim();
using (FileStream localfile = new FileStream(localFileName, FileMode.Create))
{
sftpClient.DownloadFile(remoteFile.FullName, localfile);
return true;
}
}
catch (Exception exception)
{
Console.WriteLine($"Cannot download remote file '{remoteFile.FullName}' to '{localDirectory}' :\n" + exception);
return false;
}
}
It could be there's some kind of prompt and the task doesn't know how to handle it. It's hard to tell, but I would suggest doing two things:
First, wrap everything all in a try/catch block so you can log any exceptions that happen:
BEFORE:
Open in new window
AFTER:
Open in new window
Second, in case it's not an exception (which I would expect would crash the job rather than hang it), try writing the code's progress to a log.
BEFORE:
Open in new window
AFTER:
Open in new window
Then run the app via the task scheduler and check out that log file and see where the code manages to reach before it hangs. If it doesn't generate a log at all, it could be some issue with the startup/execution of the app, but again, I would expect that to fail rather than hang.