Issue with SFTP using JSch library
Hi,
I am making an SFTP connection using JSch libaray in java.
Strangely enough the first time i call the api which internally sftp to the server it succeeds. But the second time it fails.
what could be the reason for that ?
Each time a file gets uploaded to the server the following piece of code gets called :
One thing i noticed is that i am not disconnecting the session or closing the connection explicitly here after all the files are uploaded to the sftp server.
Although each time i upload a file i open a sftp channel and then close it.
So the next time when i make the api call ideally the same connection must be available if only it doesnt get disconnected due to inactivity.
Even in that case establishConnection should make a connection :
But i still dont see why this could be the reason for the failure.
In the first api call everything goes fine..
In the second i see the following JSCH logs :
Disconnection from host ...
Caught an exception, leaving main loop due to Connection reset
So in the next call i straight away see the disconnection from host in jsch log.
And the detailed exception is :
java.util.concurrent.Compl
what could be the reason for this. How can i debug this further...
I also see a connection already established message once on the second api call..
Here is the first exception that i see the second time :
This seems to be the main error.
I am making an SFTP connection using JSch libaray in java.
Strangely enough the first time i call the api which internally sftp to the server it succeeds. But the second time it fails.
what could be the reason for that ?
Each time a file gets uploaded to the server the following piece of code gets called :
private CompletionStage<ChannelSftp> getChannel() {
return sessionManagerInit
.thenCompose(__ -> sessionManager.establishConnection(timeoutMillis))
.thenCompose(__ -> sessionManager.openSftpChannel())
.exceptionally(throwable -> {
logger.error("Error in getChannel", throwable).attr("message", throwable.getMessage()).log();
throw new ChannelException(throwable);
});
}One thing i noticed is that i am not disconnecting the session or closing the connection explicitly here after all the files are uploaded to the sftp server.
Although each time i upload a file i open a sftp channel and then close it.
So the next time when i make the api call ideally the same connection must be available if only it doesnt get disconnected due to inactivity.
Even in that case establishConnection should make a connection :
But i still dont see why this could be the reason for the failure.
In the first api call everything goes fine..
In the second i see the following JSCH logs :
Disconnection from host ...
Caught an exception, leaving main loop due to Connection reset
So in the next call i straight away see the disconnection from host in jsch log.
And the detailed exception is :
java.util.concurrent.Compl
what could be the reason for this. How can i debug this further...
I also see a connection already established message once on the second api call..
Here is the first exception that i see the second time :
This seems to be the main error.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
If you'd like your code review, attach a text file. Images are very tough to review.
From your comment, likely trying to reuse a session means a teardown/disconnect is never done correctly.
Just change your code to create/destroy a session each connection. See if that works. If it does, then you know the teardown/disconnection is tied to your object/session destruction + is required to correctly manage your SFTP server connections correctly.
From your comment, likely trying to reuse a session means a teardown/disconnect is never done correctly.
Just change your code to create/destroy a session each connection. See if that works. If it does, then you know the teardown/disconnection is tied to your object/session destruction + is required to correctly manage your SFTP server connections correctly.
ASKER
Hi,
I found that performing session.disconnect() after each file uploads solves the issue.
But that is one approach.
I am uploading approx 1000 times at a time. So dont want to create a new session every time as it would lead to overhead (correct me if i am wrong).
So the alternative is to do something like -
try {
if (session.isConnected() == false)
session.connect(connectTim
ChannelSftp channelSftp = (ChannelSftp) session.openChannel("sftp"
channelSftp.connect();
//perform operation..
channelSftp.disconnect();
} catch (Exception e) {
session.disconnect();
session = jSch.getSession(user, host, port);
session.connect(connectTim
}
So that if there is some issue with the connection then i renew it.
This aproach should work but i had to test it yet also i have noticed that normally a session stays fine for approx 15 mins..
(I dont know how this exact value can be determined ?) But it always errors out after this much time...
This approach seems better because we only renew most likely once in 15 mins... and so overhead is considerably reduced.
Other approach i can think of it to send keep alives...But i am not sure how to do it properly.
https://epaul.github.io/jsch-documentation/javadoc/com/jcraft/jsch/Session.html#setServerAliveInterval-int-
there are two functions i see of relevance -
public void setServerAliveInterval(int
sendKeepAliveMsg()
this should be the best approach i can think of as of now.. Because there wont be any need ever of renewing the session.
Although after uploading of 1000 files i will indeed renew the session. But at least during the one processing i wont have to renew it.
But then the following question appears
1) when to call the sendKeepAlive
2) what value of interval should be set in setServerAliveInterval
Please provide your view on the above
Also even though the original code i tested was opening sftp channel and then closing the sftp channel. I guess this should be treated as an activity but still the session got down after approx 20 mins..
I am not sure but that should mean that even after keepalive it will go down and the only way will be to renew the session..
what do you suggest whats the ideal and best appraoch
Thanks
I found that performing session.disconnect() after each file uploads solves the issue.
But that is one approach.
I am uploading approx 1000 times at a time. So dont want to create a new session every time as it would lead to overhead (correct me if i am wrong).
So the alternative is to do something like -
try {
if (session.isConnected() == false)
session.connect(connectTim
ChannelSftp channelSftp = (ChannelSftp) session.openChannel("sftp"
channelSftp.connect();
//perform operation..
channelSftp.disconnect();
} catch (Exception e) {
session.disconnect();
session = jSch.getSession(user, host, port);
session.connect(connectTim
}
So that if there is some issue with the connection then i renew it.
This aproach should work but i had to test it yet also i have noticed that normally a session stays fine for approx 15 mins..
(I dont know how this exact value can be determined ?) But it always errors out after this much time...
This approach seems better because we only renew most likely once in 15 mins... and so overhead is considerably reduced.
Other approach i can think of it to send keep alives...But i am not sure how to do it properly.
https://epaul.github.io/jsch-documentation/javadoc/com/jcraft/jsch/Session.html#setServerAliveInterval-int-
there are two functions i see of relevance -
public void setServerAliveInterval(int
sendKeepAliveMsg()
this should be the best approach i can think of as of now.. Because there wont be any need ever of renewing the session.
Although after uploading of 1000 files i will indeed renew the session. But at least during the one processing i wont have to renew it.
But then the following question appears
1) when to call the sendKeepAlive
2) what value of interval should be set in setServerAliveInterval
Please provide your view on the above
Also even though the original code i tested was opening sftp channel and then closing the sftp channel. I guess this should be treated as an activity but still the session got down after approx 20 mins..
I am not sure but that should mean that even after keepalive it will go down and the only way will be to renew the session..
what do you suggest whats the ideal and best appraoch
Thanks
ASKER
As per my understanding Here i am using the same session sessionManagerInit always. Once this object is constructed i am using the same session over and over which causes error.
Also here the same connection is used unless the connection is disconnected. Which i think is fine.
What should be the ideal approach in such a case.
One approach that i can think of but need suggestions is :
1) Establish a new session at each call
2) Now there are two possibilities
i) establish a new connection whenever i need to upload a file
* This will be an additional overhead.
ii) use the same connection each time i upload a file
* This approach although seems logical. and we can always reestablish the connection whenever the connection breaks
So if i take this approach of establishing a new session with each api call . An api call needs to upload say a max of 3000 files. And all those will happen over the same connection and i will reestablish the connection in case it is disconnected.
But what if the session also gets disconnected while say after uploading the 1000 th file.
So i need to also check each time if the session is still there or not...