Solved

automate sftp without key authentication and expect

Posted on 2010-08-18
22
1,846 Views
Last Modified: 2013-11-17
Here is the requirement and want to know whether i would be able to do it or not:

I want to automate the sftp to get a file from a remote server without key authentication and expect. can anybody provide any script for this to achieve?
0
Comment
Question by:kalyan_arjun
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 7
  • 3
  • +2
22 Comments
 
LVL 34

Expert Comment

by:Shreedhar Ette
ID: 33471928
Hi,

Refer this:
http://cpanforum.com/threads/3329

It might help,
Shree
0
 

Author Comment

by:kalyan_arjun
ID: 33471945
The link you had sent us linked to another one which i am not able to access. And moreover, it seems like he is using expect with the perl. Like I said, i am looking for a script without key authentication and expect.
Anyway, Thanks for the quick response.
0
 
LVL 7

Expert Comment

by:lewisg
ID: 33472047
This is what I have used before http://troy.jdmz.net/rsync/index.html

Works GREAT!
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 

Author Comment

by:kalyan_arjun
ID: 33472169
Hi lewis, The link you had sent me is really a nice but it mentioned that it needs to have the keys on the remote host(check Configuring remotehost section). This will not work for me.

Let me be more specific on my requirement.
I dont have any permissions/authority to create a file on the remote host and remote directory. A file would be placed daily and im looking for a script to get that file from the remote host.
I am able to do sftp@remotehost and get that file after entering the password. I want to acheive this in the script without paswordkey and expect.

Lewis, I appreciate for the link that you had sent and is really a nice article regarding the secuirty tip where you can mention the ipadreeses that would allow the sftp. Thanks!
0
 
LVL 7

Expert Comment

by:lewisg
ID: 33472233
If your system has autoexpect here is a slick scripting solution:

http://dbaspot.com/forums/shell/196804-novice-needs-help-using-expect-automate-sftp.html

0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 33472789
No way.
There is no possibility in sftp to provide a password on the commandline or to have it read from a file or stdin or the like.
You will have to go with key authentication or with something like expect.
ssh-agent will not work either, as it relies on keys and not system passwords.
wmp
 
0
 
LVL 26

Expert Comment

by:arober11
ID: 33478537
Do you have php installed?  As the SFTP sub-system permits password authentication, see: http://www.phpbuilder.com/manual/function.ssh2-sftp.php
0
 
LVL 26

Expert Comment

by:arober11
ID: 33526860
Along similiar lines, if Java is your language of choice, the mindterm java library and SFTPCopyFile.java example provide the functionality you require:

Usage: SFTPCopyFile <server:port> <username> <password> to|from <src_file> <dst_file>

See: http://www.appgate.com/index/products/mindterm/
0
 

Author Comment

by:kalyan_arjun
ID: 33551987
i was able to do sftp with the expect as there were no options until I use another language. Thanks everybody for your suggestions.

And one more thing is, i need to get the latest file from the other server. Eveyday one file is copied to the remote server and i need to get the latest file from the remote server:

sftp> latest_file=`ls -ltr | tail -1 | awk '{print $9}'
Invalid command.
It seems like only the sftp server is installed on this server

but ls -ltr works and it will give a list of files. Not sure what the other vendor has it. Is there anyway i could do it?
0
 

Author Comment

by:kalyan_arjun
ID: 33551991

#!/usr/local/bin/expect

#DATE1=`date "+%m%d%y%H%M%S"'
#LOGFILE= /ul/batch/logs/log_$DATE1.log
spawn sftp abc@hostname.com
expect "password:"
send "pass\n";

expect sftp>
send "get abc.txt\r"
expect sftp>
send "exit\r"

expect eof
0
 

Author Comment

by:kalyan_arjun
ID: 33555636
Any help with this or do i need to open another question for this?
0
 
LVL 7

Expert Comment

by:lewisg
ID: 33559643
It looks like the question was answered with information that appears on the dbaspot.com.

As for getting the newest file that sounds like a new question.

SFTP has a fairly limited command set and since you can't run anything on the remote machine you might try capturing a list of file names to your machine and then figuring out what the latest file is. What do the file names look like?
0
 

Author Comment

by:kalyan_arjun
ID: 33564448

#!/usr/local/bin/expect

#DATE1=`date "+%m%d%y%H%M%S"'
#LOGFILE= /ul/batch/logs/log_$DATE1.log
spawn sftp abc@hostname.com
expect "password:"
send "pass\n";

expect sftp>
send "ls /ul/test\n"
expect "No Such File" {send "exit\n"}\
expect sftp>
send "get abc.txt\r"
expect sftp>
send "exit\r"
expect eof

I tried using exit and quit and neither of them are working. Even though the file is not there, it is executing the next steps.

I am running with an agent where it looks for an exit code 0 for successful and 1 for failure. I am looking for a script for this..
0
 
LVL 26

Expert Comment

by:arober11
ID: 33577535
Hi, you now appear to be using "expect" but  your question: "automate sftp without key authentication and expect"
states you require a solutions that does not use "expect", have you changed your mind?
0
 

Author Comment

by:kalyan_arjun
ID: 33578227
sorry I should have changed it. Now,i am using expect for this sftp.Like, i mentioned, i want to quit or exit if the file is not there.
0
 
LVL 26

Expert Comment

by:arober11
ID: 33579238
Replace your:

expect "No Such File" {send "exit\n"}\


with:

expect "No such" { send "exit\r"
                           exit 2}
0
 

Author Comment

by:kalyan_arjun
ID: 33580160
still not working..
0
 
LVL 26

Expert Comment

by:arober11
ID: 33598740
Sorry forgot it's FORMAT sensitive, you need a blank line after the "{", try:

expect "No such" {
                           send "exit\r";
                           exit 2
                         }

If still not working add a "-d" debug flag to first line e.g.

#!/usr/local/bin/expect -d

Should show you were you going wrong, ELSE try the following, which works:


#!/bin/sh
# Name:    sftp_get_using_pwd.sh
# Purpose: SFTP test and pull a remote file, using password authentication.
#
EXPECT_DIR="/usr/local/bin/"
LOG_DIR="/tmp"
LOG_FILE="no_key-sftp"
DEST_DIR="."
SFTP_USER="abcdef"
SFTP_PASS="xxxxxx"
SFTP_HOST="some.host"
SFTPEXIT=0
tstFile="/ul/test"

# ----- Procedures -----
sftp_batch() {

$EXPECT_DIR/expect 2>&1 > "${LOG_DIR}/${LOG_FILE}" <<EOF
set timeout 100
spawn  /usr/bin/sftp $SFTP_USER@$SFTP_HOST
log_file ${LOG_DIR}/${LOG_FILE}-sftp
expect "Connecting to $SFTP_HOST..." {
  expect {
           -re ".*assword:" {
                              send -- "$SFTP_PASS\r";
                              expect {
                                       "Password:" {
                                                       send_user "ERROR SFTP password invalid\n";
                                                       exit 1
                                                      }
                                       "sftp> " {
                                                       send "ls $tstFile\r";
                                                       expect {
                                                                "No such file or directory" {
                                                                              send_user "ERROR No such File: $tstFile\n";
                                                                              send "quit\r";
                                                                              exit 8
                                                                             }
                                                                "sftp> " {
                                                                        send "get $tstFile $DEST_DIR\r";
                                                                        expect "sftp>" {
                                                                                        send "quit\r";
                                                                                        exit
                                                                                      }
                                                                        send_user "ERROR SFTP get failed OR timed out\n";
                                                                        send "quit\r";
                                                                        exit 7
                                                                        }
                                                            }
                                                        send_user "ERROR SFTP listing failed\n";
                                                        send "quit\r";
                                                        exit 6
                                                   }
                                     }
                           }
       }
       send_user "ERROR SFTP connection failed\n"
       exit 3
}
send_user "ERROR SFTP initialization failed\n"
exit 4
EOF
SFTPEXIT=$?
0
 
LVL 26

Expert Comment

by:arober11
ID: 33598774
Woops, cut'n'paste failure, the whole script is below:


#!/bin/sh
# Name:    sftp_get_using_pwd.sh
# Purpose: SFTP test and pull a remote file, using password authentication.
#
EXPECT_DIR="/usr/local/bin/"
LOG_DIR="/tmp"
LOG_FILE="no_key-sftp"
DEST_DIR="."
SFTP_USER="abcdef"
SFTP_PASS="xxxxxx"
SFTP_HOST="some.host"
SFTPEXIT=0
tstFile="/ul/test"

# ----- Procedures -----
sftp_batch() {

$EXPECT_DIR/expect 2>&1 > "${LOG_DIR}/${LOG_FILE}" <<EOF
set timeout 100
spawn  /usr/bin/sftp $SFTP_USER@$SFTP_HOST
log_file ${LOG_DIR}/${LOG_FILE}
expect "Connecting to $SFTP_HOST..." {
  expect {
           -re ".*assword:" {
                              send -- "$SFTP_PASS\r";
                              expect {
                                       "Password:" {
                                                       send_user "ERROR SFTP password invalid\n";
                                                       exit 1
                                                      }
                                       "sftp> " {
                                                       send "ls $tstFile\r";
                                                       expect {
                                                                "No such file or directory" {
                                                                              send_user "ERROR No such File: $tstFile\n";
                                                                              send "quit\r";
                                                                              exit 8
                                                                             }
                                                                "sftp> " {
                                                                        send "get $tstFile $DEST_DIR\r";
                                                                        expect "sftp>" {
                                                                                        send "quit\r";
                                                                                        exit
                                                                                      }
                                                                        send_user "ERROR SFTP get failed OR timed out\n";
                                                                        send "quit\r";
                                                                        exit 7
                                                                        }
                                                            }
                                                        send_user "ERROR SFTP listing failed\n";
                                                        send "quit\r";
                                                        exit 6
                                                   }
                                     }
                           }
       }
       send_user "ERROR SFTP connection failed\n"
       exit 3
}
send_user "ERROR SFTP initialization failed\n"
exit 4
EOF
SFTPEXIT=$?
}

# ----- Main -----
echo "ls" > $CMD_FILE
sftp_batch
echo "----------------"
echo "Returned $SFTPEXIT"
echo "--"
cat "${LOG_DIR}/${LOG_FILE}"
0
 

Author Comment

by:kalyan_arjun
ID: 33601057
This script works like a charm. I am trying to remove the files after it is downloaded where i am unable to do it. Could you throw some light on this?
0
 
LVL 26

Accepted Solution

by:
arober11 earned 500 total points
ID: 33601428
Not the safest method, but you could just chnage the following lines:

                                                                        expect "sftp>" {
                                                                                        send "quit\r";
                                                                                        exit
                                                                                      }
                                                                        send_user "ERROR SFTP get failed OR timed out\n";

to:
                                                                        expect "sftp>" {
                                                                                               send "rm $tstFile\r";
                                                                                               expect "sftp>" {
                                                                                                                     send "quit\r";
                                                                                                                     exit
                                                                                                                    }
                                                                                              send_user "ERROR SFTP remove failed OR timed out\n";
                                                                                              exit 5
                                                                                              }
                                                                        send_user "ERROR SFTP get failed OR timed out\n";
0
 

Author Closing Comment

by:kalyan_arjun
ID: 33708881
perfect solution and it worked like a charm...
0

Featured Post

Are You Headed to Black Hat USA 2017?

Getting ready for Black Hat next week? Kick things off with the WatchGuard Badge Challenge and test your puzzle and cipher skills. Do you have what it takes to earn our limited edition Firebox Badge? Get started today - https://crimsonthorn.net

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
Imagine a situation that you have installed SSL (http://en.wikipedia.org/wiki/Secure_Sockets_Layer) Certificate on your Cisco ASA (Cisco Adaptive Security Appliance) firewall. Installation of SSL certificate on ASA is an another topic for which you …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses

627 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question