Crontab won't run my python script and can't figure out why

I think maybe your shebang line is wrong... Needs to end with the name of the python interpreter binary which I assume from above is
#!/usr/local/lib/python2.7/python

Unless your python binary is named python2.7
Check if script is chmod x+ also

Tried changing the shebang line and no luck still. And for the record my permissions on the script are 777 for testing purposes to see if it was a permission issue. It seems not to be.

-rwxrwxrwx 1 root root   805 Aug  4 16:31 mlb_cdt_download_newv2.py

I think what I need to do is find out what interpreter it is using and make sure my paramiko/pysftp are located in the right directories. I just don't know the commands/steps on how to do that.

The very first test that I would do is run the script from command line without the shebang

so just run

python /usr/local/mlb_cdt_download_newv2.py

Select allOpen in new window

Does this work?

If yes, then type

which python

Select allOpen in new window

and put exactly this version of python in your shebang.



if which python returns for example /usr/bin/python,
then set the shebang to

#!/usr/bin/python

Select allOpen in new window

After the shebang you should have the exact name of your python executable.

( There is a trick with the shebang (#!/usr/bin/env python) , that finds the python in your search path, however this fails usually for cronjobs, so I won't go into further details for it)

if your above script does not work, then I recommend to try to indentify the correct python executable on the command line.

So if you don't manage to run the script successfully on the command line I wouldn't even try to run it in a cronjob.

Next thing I would do is to capture stdout and stderr of your cronjob.
Often the problem becomes obvious or at least you could send the output to us and we might help.

so instead of

* * * * * /usr/local/mlb_cdt_download_newv2.py > /dev/null 2>&1 

Select allOpen in new window

I would do

* * * * * /usr/local/mlb_cdt_download_newv2.py > /tmp/mlb_cdt_download_newv2.log 2>&1 

Select allOpen in new window

then wait one minute and look at the contents of the log file /tmp/mlb_cdt_download_newv2.log

As gelnodia suggested, change your logging to a file, rather than /dev/null + you'll instantly have your question answered.

The results are the following.

/usr/local/mlb_cdt_download_newv2.py: 3: /usr/local/mlb_cdt_download_newv2.py: import: not found
/usr/local/mlb_cdt_download_newv2.py: 4: /usr/local/mlb_cdt_download_newv2.py: import: not found
/usr/local/mlb_cdt_download_newv2.py: 5: /usr/local/mlb_cdt_download_newv2.py: import: not found
/usr/local/mlb_cdt_download_newv2.py: 6: /usr/local/mlb_cdt_download_newv2.py: import: not found
/usr/local/mlb_cdt_download_newv2.py: 16: /usr/local/mlb_cdt_download_newv2.py: Syntax error: "(" unexpected

And for the record running from the command line works fine. If I put the path I can run it from any directory.

Edit: So I did some research and checked out where my PYTHONPATH is for each import. So I added the absolute path but now I am getting the following error.

Location of __file__ for each import.
/usr/lib/python2.7/os.pyc
/usr/local/lib/python2.7/dist-packages/pysftp/__init__.pyc
/usr/local/lib/python2.7/dist-packages/paramiko/__init__.pyc
/usr/lib/python2.7/posixpath.pyc

#!/usr/bin/python
import os; os.path.append('/usr/lib/python2.7')
import pysftp; pysftp.path.append('/usr/local/lib/python2.7/dist-packages')
import os.path; os.path.path.append('/usr/lib/python2.7')
import paramiko; paramiko.path.append('/usr/local/lib/python2.7/dist-packages')

Select allOpen in new window

/usr/local/mlb_cdt_download_newv2.py: 3: /usr/local/mlb_cdt_download_newv2.py: Syntax error: word unexpected (expecting ")")

How do you execute the command from command line?

with

 /usr/local/mlb_cdt_download_newv2.py

Select allOpen in new window

or with

python  /usr/local/mlb_cdt_download_newv2.py

Select allOpen in new window

Somehow your first line (the 'shebang' line) is not properly detected.

That's why your script is interpretesd as shell script and not as a python script.

If this is the case and you have the program xxd installed on your PC,
could you send the first lines of the output of following command to this group?

xxd /usr/local/mlb_cdt_download_newv2.py

Select allOpen in new window

Thus your script (being interpreted as shell script) tries to execute  the command import which it doesn't find (probably you don't have imagemagick installed)
then it encounters syntax errors because of the '(' and ')' character.

Just try to call your script with

python /usr/local/mlb_cdt_download_newv2.py

Select allOpen in new window

Probably you will have a different error messages then.



I'm not sure, why you want to add '/usr/lib/python2.7' to the path.

You wrote:

import os; os.path.append('/usr/lib/python2.7')

Select allOpen in new window

If this line is really required (which I'm not 100% convinced of), then it should probably be

import sys
print("sys.path is %s" % sys.path)
sys.path.append('/usr/lib/python2.7')
print("sys.path is now %s" % sys.path)

Select allOpen in new window

to repair your first issue I propose, that you create a new file, and that you verify, that the new file is saved with unix line endings and not with DOS/Windows or Mac line endings.

Ok so long story short I just retyped the hashbang and that made it work for the most part. I read somewhere that if you copy and pasted the hashbang it might not work. That being said, I am getting an error with the hostkey/known_hosts and such. I tried using known_hosts and I get the issue "No password or key specified". When I change it to my private key I get "No hostkey for host "servername" found. Again what is weird is it works fine from the command line but doing so via the Crontab it seems to not work. I did read up on an article about pysftp 0.2.9 vs 0.2.8 and how downgrading works but it also leaves you open to attacks and is less secure. I don't know why this is so hard :(

https://github.com/Yenthe666/auto_backup/issues/47

#!/usr/bin/python
import os
import pysftp
import paramiko
from shutil import copy2
import os.path

HOST=""
USER=""
#PASSWORD=""
SUFFIX_TO_FETCH__AND_DELETE=".txt"
DESTINATION_PATH= "/mnt/sas/ftp"
DESTINATION_PATH2="/mnt/sas/ftp_ncdfs_coll"

cnopts=pysftp.CnOpts()
cnopts.hostkeys.load('/home/boro/.ssh/id_rsa')

srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts)

Select allOpen in new window

When this line is: cnopts.hostkeys.load('/home/boro/.ssh/id_rsa')
Error: No Hostkey for host found.

When this line is: cnopts.hostkeys.load('/home/boro/.ssh/known_hosts')
Error: No password or key specified.

I am at a loss at this point. Tried so much, this is getting frustrating ugh.

Cut + Paste sometimes copies unprintable characters.

In vi type :set list to see all unprintable characters.

I'm still a bit confused about why you're basically attempting to rewrite rsync in python.

A simple rsync command accomplishes what you're attempting. Off the top of my head, so might require some massaging...

rsync -av -e "ssh -i /home/boro/.ssh/id_rsa" $user@$host:$remote-path/. $local-path/.

Select allOpen in new window

Just drop this into CRON or into a bash script or whatever language you like.

@David

I don't know about python rsync but I am trying to connect via SFTP using a Python script. I suppose I would use a bash script if that is possible as well. That being said my client has it to where i can connect via SFTP but not FTP over SSH. That is why when researching I decided to use paramiko and pysftp as they were the most popular. It seems with the latest version of pysftp it's causing this issue with the keys not being read properly or something of that sorts. I am about ready to just downgrade to 0.2.8 and call it a day even though it is technically less secure.

So if I understand you correctly you say, that your script is working correctly if called manually, but not correctly if run via a cron job.

The potential differences to look at.

1.) Do you really use the same python version for both. I assume that this is fine for you, but to be sure, I propose, that you call your script explicitly with a python interpreter.

so in an interactive shell you'll type for example

/usr/bin/python /usr/local/mlb_cdt_download_newv2.py

Select allOpen in new window

and in the cronjob you write

* * * * * /usr/bin/python /usr/local/mlb_cdt_download_newv2.py > /tmp/mlb_cdt_download_newv2.log 2>&1 

Select allOpen in new window

2,compare the environment variables in your interactive shell and in your cron job.

the first variables to look at are probably
PATH
and


You can do this for example by adding following lines

import os
for name, value in sorted(os.environ.items()):
    print("export %s='%s'" % (name, value)

Select allOpen in new window

the variables, that are probably ost interesting are probably
PATH and SSH_AUTH_SOCK


It might be, that the script, that you run in your shell works only if you are used to a keyring or an ssh-agent.
In order to test this I propose, that you unset any potential environment variable offering keyrings / agents to ssh and try to run it in your shell.

Does your script still work if you do following?

example:

 unset SSH_AUTH_SOCK
 /usr/bin/python /usr/local/mlb_cdt_download_newv2.py

Select allOpen in new window

When adding that for name function I get the following results.

export MAIL='/var/mail/boro'
export OLDPWD='/home/boro'
export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games'
export PWD='/usr/local'
export SHELL='/bin/bash'
export SHLVL='1'
export SSH_CLIENT='10.249.72.158 51217 22'
export SSH_CONNECTION='10.249.72.158 51217 10.249.72.228 22'
export SSH_TTY='/dev/pts/1'
export TERM='xterm'
export USER='boro'
export XDG_RUNTIME_DIR='/run/user/1001'
export XDG_SESSION_ID='25'
export _='/usr/bin/python'

FYI the code ran properly from the shell with showing those results.

When I did the unset ssh_auth_sock my code didn't work properly.

Traceback (most recent call last):
  File "/usr/local/mlb_cdt_download_newv2.py", line 15, in <module>
    srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts, password=PASSWORD)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 143, in __init__
    self._transport.connect(**self._tconnect)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1123, in connect
    self.start_client()
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 511, in start_client
    raise SSHException('Negotiation failed.')
paramiko.ssh_exception.SSHException: Negotiation failed.

When running the crontab as you stated putted /usr/bin/python prior to calling the code it WORKED!

Log output:

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'

So it seems that SSH_AUTH_SOCK thing is what causes the problem but when I explicitly call the pythonpath in the crontab it works. If that's what it takes then fine.

Really appreciate you helping me. Your knowledge on this subject is impressive.

glad it works now :-) though I'm (to be honest) not 100% sure, why the cronjob works with explicit python path and unset SSH_AUTH_SOCKET whereas the cronjob works with explicit python path and (I assume) also an unset SSH_AUTH_SOCKET.

In case you encounter any more problems, there is one more thing that I noticed:

It seems that the cronjob is run as root whereas your interactive job was executed as user boro.

Whenever you require further debugging, compare the env vars and execute the job manually as the same user as the one, that will run the cronjob.


happy coding

Funny you mentioned other problems. So my other client it works in command line and you can see the information below:

export HOME='/home/boro'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'
export MAIL='/var/mail/root'
export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin'
export SHELL='/bin/bash'
export SUDO_COMMAND='/usr/bin/python new_ncdfs_sftp.py'
export SUDO_GID='1001'
export SUDO_UID='1001'
export SUDO_USER='boro'
export TERM='xterm'
export USER='root'
export USERNAME='root'

But when I run it on the crontab and explicitly call the /usr/bin/python I get the following error log

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'
Traceback (most recent call last):
  File "/usr/local/new_ncdfs_sftp.py", line 21, in <module>
    srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts)#, password=PASSWORD)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 142, in __init__
    self._set_authentication(password, private_key, private_key_pass)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 156, in _set_authentication
    raise CredentialException("No password or key specified.")
pysftp.exceptions.CredentialException: No password or key specified.

Crontab: * * * * * /usr/bin/python /usr/local/new_ncdfs_sftp.py  > /var/tmp/ncdfs_sftp.log 2>&1

And the code is the following in it:

HOST=""
USER=""
#PASSWORD=""
SUFFIX_TO_FETCH__AND_DELETE=".txt"
DESTINATION_PATH= "/mnt/sas/ftp"
DESTINATION_PATH2="/mnt/sas/ftp_ncdfs_coll"

cnopts=pysftp.CnOpts()
cnopts.hostkeys.load('/home/boro/.ssh/known_hosts')

for name, value in sorted(os.environ.items()):
    print("export %s='%s'" % (name, value))

srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts)#, password=PASSWORD)

Select allOpen in new window

I went and commented out password because when I tried to SSH into it I got an error saying only SFTP is allowed. Once I commented out password I could SFTP and SSH into it. Also, you get a different log file when password isn't commented out.

I get the following error log file when doing that:

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'
Traceback (most recent call last):
  File "/usr/local/new_ncdfs_sftp.py", line 22, in <module>
    srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts, password=PASSWORD)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 143, in __init__
    self._transport.connect(**self._tconnect)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1156, in connect
    self.auth_password(username, password)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1329, in auth_password
    return self.auth_handler.wait_for_response(my_event)
  File "/usr/local/lib/python2.7/dist-packages/paramiko/auth_handler.py", line 217, in wait_for_response
    raise e
paramiko.ssh_exception.BadAuthenticationType: ('Bad authentication type', [u'publickey']) (allowed_types=[u'publickey'])

For the record I can only work on this whilst at work and I am headed out of town tomorrow until next wednesday (returning to work thursday). Just an FYI so the question doesn't die and I plan to give the points accordingly obviously.

Thanks again.

do you really show the output for all environment variables.

I don't see the SSH_AUTH_SOCKET variable when you run interactively?

The SSH might not be called since I am trying to explicitly use SFTP for this client since SSH isn't allowed. Or so it seems. When you say interactively do you mean via command line? I copied and pasted everything from my putty window for the record. The logs are the output and should be empty if it ran successfully.

Anyways like I said I will be away for a week so don't linger on it too much. I plan to keep the question open though. I really do appreciate all the help you have given me. I feel like I owe you lunch or something haha. Have a good one for now. I am off.

Just wanted to give a quick update. I am back and will be trying out some new things. I will let you know how it goes. Thanks again for all of your help everyone.

OK. Good luck. so just to be sure, that there is now confusion with the entire thread and with what is working and what is not working I'd suggest you post  (once more)

one setup that is working and show us the environment variables of that user, the user name and the command to start the script and the script's output


the other setup that fails.
with the username (is it a root cronjob or a cronjob with the same user id?) and the exact line of the cronjob and its output.

Somewhre must be a diffrence.
Mostly it's related to the executing user, the env vars and the way the script is called

Ok. So here's where we stand.

Not working:
Crontab (I removed the explicit call in crontab "/usr/bin/python"):

* * * * * /usr/local/new_ncdfs_sftp.py  > /var/tmp/ncdfs_sftp.log 2>&1

Log:

/bin/sh: 1: /usr/local/new_ncdfs_sftp.py: not found

Crontab (with explicit call):

* * * * * /usr/bin/python /usr/local/new_ncdfs_sftp.py  > /var/tmp/ncdfs_sftp.log 2>&1

Log:

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'
Traceback (most recent call last):
  File "/usr/local/new_ncdfs_sftp.py", line 21, in <module>
    srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts)#, password=PASSWORD)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 142, in __init__
    self._set_authentication(password, private_key, private_key_pass)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 156, in _set_authentication
    raise CredentialException("No password or key specified.")
pysftp.exceptions.CredentialException: No password or key specified.

Code (works in command line, not in CRON tab)

#!usr/bin/python
import os
import pysftp
import paramiko
from shutil import copy2
import os.path

HOST=""
USER=""
#PASSWORD=""
SUFFIX_TO_FETCH__AND_DELETE=".txt"
DESTINATION_PATH= "/mnt/sas/ftp"
DESTINATION_PATH2="/mnt/sas/ftp_ncdfs_coll"

cnopts=pysftp.CnOpts()
cnopts.hostkeys.load('/home/boro/.ssh/known_hosts')

for name, value in sorted(os.environ.items()):
    print("export %s='%s'" % (name, value))

srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts)#, password=PASSWORD)


def do_nothing(fname):
        return " "

def fetch_and_remove(fname):
    if not fname.endswith(SUFFIX_TO_FETCH__AND_DELETE):
        return # skip files with wrong suffix
    filename = fname[11:]
    dst_fname = os.path.join(DESTINATION_PATH, filename)
    dst_dirname = os.path.dirname(dst_fname)
    if os.path.isfile(dst_fname):
        return " "
    else:
        srv.get(fname, dst_fname)
        copy2(dst_fname, DESTINATION_PATH2)
        srv.remove(fname)

srv.walktree('/tolab/sdt', fetch_and_remove, do_nothing, do_nothing)

def fetch_and_remove2(fname2):
    if not fname2.endswith(SUFFIX_TO_FETCH__AND_DELETE):
        return # skip files with wrong suffix
    filename2 = fname2[7:]
    dst_fname2 = os.path.join(DESTINATION_PATH, filename2)
    dst_dirname2 = os.path.dirname(dst_fname2)
    if os.path.isfile(dst_fname2):
        return " "
    else:
        srv.get(fname2, dst_fname2)
        srv.remove(fname2)

srv.walktree('/tolab', fetch_and_remove2, do_nothing, do_nothing)

Select allOpen in new window

Working:
Crontab (with explicit call):

* * * * * /usr/bin/python  /usr/local/mlb_cdt_download_newv2.py > /var/tmp/mlb_cdt_download_newv2.log 2>&1

Log:

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'

Code (works in command line and CRON tab)

#!/usr/bin/python
import os
import pysftp
import os.path
import paramiko

HOST=""
USER=""
PASSWORD=""
SUFFIX_TO_FETCH__AND_DELETE=".csv"
DESTINATION_PATH="/mnt/sas/ftp_mlb"
cnopts=pysftp.CnOpts()
cnopts.hostkeys.load('/home/boro/.ssh/id_rsa.txt')

srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts, password=PASSWORD)

for name, value in sorted(os.environ.items()):
        print("export %s='%s'" % (name,value))

def do_nothing(fname):
        return  " "

def fetch_and_remove(fname):
    if not fname.endswith(SUFFIX_TO_FETCH__AND_DELETE):
        return # skip files with wrong suffix
    dst_fname = os.path.join(DESTINATION_PATH, fname)
    dst_dirname = os.path.dirname(dst_fname)
    if os.path.isfile(dst_fname):
        return " "
    else:
        srv.get(fname, dst_fname)
        srv.remove(fname)

srv.walktree('.', fetch_and_remove, do_nothing, do_nothing)

Select allOpen in new window

For the first code that works in command line and not crontab the server is setup differently so I access it a bit differently. The first one is a Linux server and we have the SFTP authorized_keys file on there correctly but only SFTP via key is allowed (password has been disabled). When I change the "known_hosts" to my id_rsa.txt file like that in the second code I get the following error in my log.

Log with "id_rsa.txt instead of known_hosts":

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'
Traceback (most recent call last):
  File "/usr/local/new_ncdfs_sftp.py", line 21, in <module>
    srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts)#, password=PASSWORD)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 132, in __init__
    self._tconnect['hostkey'] = self._cnopts.get_hostkey(host)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 71, in get_hostkey
    raise SSHException("No hostkey for host %s found." % host)
paramiko.ssh_exception.SSHException: No hostkey for host scanftp.drugfreesport.com found.
Exception AttributeError: "'Connection' object has no attribute '_sftp_live'" in <bound method Connection.__del__ of <pysftp.Connection object at 0x7f7f710f0ed0>> ignored

The second code has password and key both enabled but as you can see I am access it via key. It works just fine as is so I am not touching it.

I think that's everything let me know if you need more.

So you have one script, that is working on command line as user root with the command:

/usr/bin/python  /usr/local/mlb_cdt_download_newv2.py 

Select allOpen in new window

and from  below cronjob line

* * * * * /usr/bin/python  /usr/local/mlb_cdt_download_newv2.py > /var/tmp/mlb_cdt_download_newv2.log 2>&1 

Select allOpen in new window

This script is working with password authentification. I assume, that the code would also work if you changed line 15
from

srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts, password=PASSWORD)

Select allOpen in new window

to

srv = pysftp.Connection(host=HOST, username=USER, password=PASSWORD)

Select allOpen in new window

right?

Then you have a second script, that tries to connect via a private key to one server, that has the public key corresponding to the private key in its ~/.ssh/authorized_keys file, right?

Please show me the output of the working script when being called from the commandline as root user (the same user being used by the cronjob)

 /usr/bin/python /usr/local/new_ncdfs_sftp.py 

Select allOpen in new window

Then show me the output of the cronjob
command line from the failing script

* * * * * /usr/bin/python /usr/local/new_ncdfs_sftp.py  > /var/tmp/ncdfs_sftp.log 2>&1 

Select allOpen in new window

I suspect that there is an issue with an ssh private key, that is encypted and that the command line version has access to an ssh agent and the cronjob doesn't, but I have to be able to compare the entire output of the passing and the failing scripts.

As I mentioned already before my primary suspect would be the environment variable SSH_AUTH_SOCK

can you please also verify the contents of
/home/boro/.ssh/id_rsa.txt
The name is strange.
Normally it should either be /home/boro/.ssh/id_rsa or /home/boro/.ssh/id_rsa.pub
I'd like to check, whether the contents of /home/boro/.ssh/id_rsa.txt is the contents of the private key file or of the public key file.

if it is the private file, then it should be a file containing multiple lines and start with something like

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAdKCAQEA901xhKantpa+CZ7HqFM98QSyIZihJCuK/E3FZrQVT5rdKChS
. . . 

Select allOpen in new window

Then it is the private file and you should never publish it or communicat it to anyone.

If it is an encrypted private key file, then it should start with something like

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,02F0127ACB8514C58A29F7CE467E65DD

zNBdvd3VSjzmf+a9x+RLxE13FMZjLbbkzeGH8aUNChLErxPBSVTqtkNpHtH97sZ8
. . .

Select allOpen in new window

Though such a file is password protected I still wouldn't communicate it to anyone else

or if it is a public key file it should contain only one line, something like

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJvtoVIFKkHWX37j/u7g9TnON+Fsdu1TvCvPMw+7uycs7TmkaOrXhaVX/GokSU3dG2bYGxFTcnOf3ocY/AUZmMtjeVG8IOMAaiiXco990mcyDroPuDzyyQt0nrm5+xMIkNsOmRaa3ZpS+XSIf+wJBudEKzM9HPfI8F96X5h2OOJ8nGISJmSd907/gm3HR8EDXEQzQq/fj1RrITfndwOUgQ9AszXAp6PVwAWKAmeSoWF07HxJGIPV6PNnz1kJFPkO/BDi8o192a/ag1nDbphC8NSLEGXNCjkxTsJL6MueRhVtBIzi2xjL22C2Q7naYig7T6rTQLUm8lSOoYOn8LiGhF gelonida@mypc

Select allOpen in new window

and it can be published without any issues

So what does your file contain?

Yes. When I got rid of Cnopts in the .connection function I get the following in my log.

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'

It works without it there. Again though, that server is setup differently. That particular server is a CoreFTP Windows server. The one I am having trouble with is a Linux server and it only accepts SFTP, not FTP or Password or even SSH.

please reread my previous post. I enhanced it and corrected some typos while you were already replying to it. (You're too fast ;-) )


perhaps start with checking the contents of the key file

When running the code from the command line I get the following:

boro@NewCDServerVM:/usr/local$ sudo python new_ncdfs_sftp.py
export HOME='/home/boro'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'
export MAIL='/var/mail/root'
export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin'
export SHELL='/bin/bash'
export SUDO_COMMAND='/usr/bin/python new_ncdfs_sftp.py'
export SUDO_GID='1001'
export SUDO_UID='1001'
export SUDO_USER='boro'
export TERM='xterm'
export USER='root'
export USERNAME='root'

So the reason I made a .txt file to call from was originally I had issues with my first code not recognizing known_hosts or id_rsa so I copied id_rsa to a .txt file and it was able to read that just fine. You can see that in the code that is working (why it works with a .txt and not the main file who knows). So id_rsa and id_rsa.txt is my PRIVATE key.

When I input id_rsa, id_rsa.pub, or my id_rsa.txt I get the following log:

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'
Traceback (most recent call last):
  File "/usr/local/new_ncdfs_sftp.py", line 21, in <module>
    srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts)#, password=PASSWORD)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 132, in __init__
    self._tconnect['hostkey'] = self._cnopts.get_hostkey(host)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 71, in get_hostkey
    raise SSHException("No hostkey for host %s found." % host)
paramiko.ssh_exception.SSHException: No hostkey for host scanftp.drugfreesport.com found.
Exception AttributeError: "'Connection' object has no attribute '_sftp_live'" in <bound method Connection.__del__ of <pysftp.Connection object at 0x7f6aa003aed0>> ignored

I noticed one thing.
if you have a problem with a cronjob, that is run as root, then do not use sudo.
sudo inherits too many environment variables from the calling user. so the comparison is more complicated.


You also did not use the ecact command line, that I asked you to use. (I think it doesn't make a difference in this case, but sometimes it could, so please try to replicate the exact command.

Can you please try to call your script as user root with the shell setup of the user root?

You can do this by typing

sudo su - # now you will have a root shell with root's environment varables
/usr/bin/python /usr/local/new_ncdfs_sftp.py # the exact same command as being used in the cronjob (except the redirection of stdout / stderr)

Select allOpen in new window

Will your script still run? if not, what error.


Can you also tell me whether your private key file is a private key file with or without a password. (In my previous post I showed you how to identify the difference)

Have to leave now. will check back tomorrow.

The private key has no password I can confirm that. Here is what I get when I run it using the exact line of code you recommend.

boro@NewCDServerVM:/usr/local$ sudo su
[sudo] password for boro:
root@NewCDServerVM:/usr/local# /usr/bin/python /usr/local/new_ncdfs_sftp.py
export HOME='/root'
export LANG='en_US.UTF-8'
export LESSCLOSE='/usr/bin/lesspipe %s %s'
export LESSOPEN='| /usr/bin/lesspipe %s'
export LOGNAME='root'
export LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'
export MAIL='/var/mail/root'
export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games'
export PWD='/usr/local'
export SHELL='/bin/bash'
export SHLVL='1'
export SUDO_COMMAND='/bin/su'
export SUDO_GID='1001'
export SUDO_UID='1001'
export SUDO_USER='boro'
export TERM='xterm'
export USER='root'
export USERNAME='root'
export XDG_RUNTIME_DIR='/run/user/1001'
export XDG_SESSION_ID='26'
export _='/usr/bin/python'
Traceback (most recent call last):
  File "/usr/local/new_ncdfs_sftp.py", line 21, in <module>
    srv = pysftp.Connection(host=HOST, username=USER, cnopts=cnopts)#, password=PASSWORD)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 132, in __init__
    self._tconnect['hostkey'] = self._cnopts.get_hostkey(host)
  File "/usr/local/lib/python2.7/dist-packages/pysftp/__init__.py", line 71, in get_hostkey
    raise SSHException("No hostkey for host %s found." % host)
paramiko.ssh_exception.SSHException: No hostkey for host scanftp.drugfreesport.com found.
Exception AttributeError: "'Connection' object has no attribute '_sftp_live'" in <bound method Connection.__del__ of <pysftp.Connection object at 0x7f4e80262f50>> ignored

I changed it from id_rsa, id_rsa.pub, id_rsa.txt, and known_hosts. When running from the "root" user it seems to not work like that at all. *Interesting*

I look forward to hearing from you tomorrow.

well you used

sudo su 

Select allOpen in new window

instead of

sudo su -

Select allOpen in new window

The '-' switch of su is intended to get an environment as close as possible to a login shell of the user with the given user id.

However even with the non complete interactive root account you could show something quite important.

That the interactive shell is not working.
Now the job is to get this interactive shell working and if we really understand we can make the cronjob work as well.

Ok I have done everything you asked. When I tried to SFTP into the server I got the following error but it worked after I said "yes":

root@NewCDServerVM:/home/boro/.ssh# sftp user@host
The authenticity of host 'hostname' can't be established.
ECDSA key fingerprint is hexadecimal string.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hostname' (ECDSA) to the list of known hosts.
Connected to host.
sftp>

I replaced all the "private" information with user or host and hostname but you get the idea. It says it added it to the hosts file which is what we were hoping.

These are the results after using sudo su - and running the command explicitly in the command line:

root@NewCDServerVM:/usr/local# /usr/bin/python new_ncdfs_sftp.py
export HOME='/root'
export LANG='en_US.UTF-8'
export LESSCLOSE='/usr/bin/lesspipe %s %s'
export LESSOPEN='| /usr/bin/lesspipe %s'
export LOGNAME='root'
export LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'
export MAIL='/var/mail/root'
export OLDPWD='/root'
export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
export PWD='/usr/local'
export SHELL='/bin/bash'
export SHLVL='1'
export TERM='xterm'
export USER='root'
export XDG_RUNTIME_DIR='/run/user/1001'
export XDG_SESSION_ID='26'
export _='/usr/bin/python'
root@NewCDServerVM:/usr/local#

In other words, it worked. I can try testing it on the crontab now and let you know.

Edit: HOLY it worked :)
Log file:

export HOME='/root'
export LANG='en_US.UTF-8'
export LOGNAME='root'
export PATH='/usr/bin:/bin'
export PWD='/root'
export SHELL='/bin/sh'

No errors! So it was something to do with interactive shell and root directory permissions? I am trying to follow but really just happy it plain and simple worked.

Gelonida, I feel like I should pay you or at least take you out to lunch. You saved me so much pain and agony trying to understand what the heck I was doing wrong and in the process taught me so much. Thank you so so much!

Really glad it works now.

To summarize. First thing is whenever something doesn't work with a cronjob  is to try to get your interactive environment as close as possible to the cronjob's one. It's just easier to debug interactively.
In parallel you capture a cronjob's stdout / stderr to a log file for analysis.
In case of doubt dumping environment vars can reveal some differences

As you noticed in your case, sudo is doing a lousy job for this kind of tasks. It intentionally tries to keep many environment variables from the initial (calling) user.

So for debugging I suggest to combine sudo and su.

sudo su - <username>

Select allOpen in new window

username is the name of the user, that is executing the cronjob. (Don't forget the '-')
if the user is root you can omit  <username>

The next two issues  became obvious as soon as your interactive shell failed the same way as your cron job.

They were:

private key file permissions:
---
The private keyfiles must be owned by the user who's running the ssh/sftp command and the keyfile MUST not be readable by any other user.

confirmation for connection the first time to a host:
---
For each user, ssh keeps a file named ~/.ssh/known_hosts, which memorizes the remote hosts public keys, of the hosts, that a user connected already to.
This will ensure, that you always connect to the same server than the first time (so nobody can perform spoofing attacks)

So whenever you connect the first time to a server ssh will ask you whether this is the right server. you confirm with 'yes' and next time this question won't be asked anymore.

There might be options in paramiko / pysftp to 'silently' accept new servers and add them to ~/.ssh/known_hosts, but I don't know them.


Happy coding