Python CGI Runs Fine In SSH, But Not In Browser?

Hmm,
It's quite trick to know without further info.
So I'll ask you some questions a little blindly.


Q1.) What web server is being used?
- Do you have any other cgi script written in an other language, that is working? like for example:

#!/bin/sh
cat << eot
Content-Type: text/html

Hello world
eot

Select allOpen in new window

Q2.) Do you have access to the webserver's logs? (This would definitely simplify the debugging)

Q3.) I assume you execute the script on the same machine on which the server is run. Am I right?

Q4.) How do you call the script via ssh? (let's assume you called your cgi-script mycgiscript.py)
with
python ./mycgiscript.py
or just with
./mycgiscript.py

Q5.) What's the output of following command?
which python

Q6.) what's the output of following command?
python -m cgi > /dev/null && echo $?

Wow gelonida, thank you so much for your response.  It sounds like you really know what you are doing.

Ok, I've set up the following answers to correspond with each of your questions:

A1.) I am running a Linux system using Apache version 2.2.24 and CentOS release 6.5.  I took your script and saved it as a.sh under the cgi-bin subdirectory and made it executable.  I ran it using ./a.sh and it ran fine.  I tried it on the web browser using the format (url is just an example) "http://mysite.com/cgi-bin/a.sh" and I got the same error as I did when I tried to run "http://mysite.com/cgi-bin/a.py".

A2.) I do have access to the webserver logs.  Currently the error log is showing only the following error: [Fri May 09 15:49:08 2014] [client xx.xx.xx.xx] File does not exist: /var/www/test/500.shtml (ip and directory have been changed), which indicates to me that the script is throwing a 500 error and is simply trying to access the page to display that error.

A3.) Yes, you are right.

A4.) I call it with ./mycgiscript.py

A5.) /usr/bin/python

A6.) 0

A1 indicates the problem is related to the server and not to python.

A3 to A6 look all OK, no surprises there.

A2 is not reall helpful :-(

So it could be access right problems or a problem with the apache config.
perhaps the server cannot read / execute the script or they cgi scripts are not located in the right directory

Can you check the file permissions of our cgi scripts and of the directory containing it and of all its parent directories?
Has the user running the web server enough permissions to read and execute the cgi scripts?

Could you check the apache server configuration?
You can post it. Just be sure to anonmize anything you don't want to share or to just show the extracts containing the setup of the cgi directory?

You might also try to rename our scripts to
a_sh.cgi and to a_py.cgi

I'm not reall convinced about this attempt, but some server configs insist on a .cgi suffix

Hi gelonida,

Yeah, A2 has been really frustrating for me too.  When I go ahead and install the missing 500.shtml file as I have on my other sites, all I get is a blank page and no error messages at all.  This is why I got rid of the 500.shtml in the hope that I could at least capture some errors.

I tried the file rename, and per your suspicions it made no difference.

Permissions don't appear to be a problem.  All the files and the cgi-bin directory are set to 755.

Your question regarding the users, "Has the user running the web server enough permissions to read and execute the cgi scripts?" might be a key to this problem, however.

Let me ask you (because I seriously don't know): Does the fact that the entire parent directory and all its subfolders are under one user affect the ability of the site to be seen?  I don't think so, because the user in question sets up html files and they display just fine in the browser.  All my sites are under a different user (apache), and they all display fine in the browser, however all of my sites are either html or php.  I have no python or cgi sites.  Am I wrong?

The only thing I see related to the /cgi-bin/ directory in the httpd.conf file is the following:

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin/"

</IfModule>

<Directory "/usr/local/apache/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all

</Directory>

Select allOpen in new window

Do you see an issue here?

* My bad, the search wasn't done looking.  There is also a valid ScriptAlias /cgi-bin/ for the directory where the scripts in question are located.

Not sure I understood correctly. Do you have two script aliases for /cgi-bin/ ?
This should not be the case and will not work.
Aliases are not like a  search path.  Only one of the aliases will be taken into account. ( I don't know whether the first or the last entry)

So first you should remove the ambiguity from your config file and make sure, that aliases / script aliases are unique.
You can do this be removing one of the script alias or by renaming one of them.
Example:
rename yours to
   ScriptAlias /my-cgi-bin/ "/path/to/my/cgi-scripts/"

and then change the url to http://mysite.com/my-cgi-bin/a.py"

Of course you can also (after having uniqified your aliases) copy your cgi-script to /usr/local/apache/cgi-bin/


Concerning file ownerships.
if a user tries to access any file it has to start accessing from the root directory.
So the parent directories have to be at least executable (and normally also readably) for the user apache (proably meaning read/execute permissions for others)

Hi gelonida!

Sorry for the delay in getting back to you.  There was no way I could respond today.

My httpd.conf file is super lengthy, which is why I am hesitant to post it here.  I did revisit the /cgi-bin/ script aliases and it appears each one is enveloped within a <VirtualHost></VirtualHost> tag set.  That probably makes a difference, right?

Here is the VirtualHost that affects the directory in question (changed for security):

<VirtualHost xx.xx.xx.xx:80>
    ServerName somename.domain.com
    ServerAlias www.somename.com www.somename.domain.com somename.com
    DocumentRoot /home/domain/public_html/somename
    ServerAdmin webmaster@somename.domain.com
    UseCanonicalName Off
    CustomLog /usr/local/apache/domlogs/somename.domain.com combined
    CustomLog /usr/local/apache/domlogs/somename.domain.com-bytes_log "%{%s}t %I .\n%{%s}t %O ."
    ## User domain # Needed for Cpanel::ApacheConf
    UserDir enabled domain
    <IfModule mod_suphp.c>
        suPHP_UserGroup domain domain
    </IfModule>
    <IfModule !mod_disable_suexec.c>
        <IfModule !mod_ruid2.c>
            SuexecUserGroup domain domain
        </IfModule>
    </IfModule>
    <IfModule mod_ruid2.c>
        RUidGid domain domain
    </IfModule>
    ScriptAlias /cgi-bin/ /home/domain/public_html/somename/cgi-bin/

</VirtualHost>

Select allOpen in new window

Now, one thing I can note is the ownership of /home/domain/public_html/somename and all its subdirectories is someotheruser, not domain.  Would this make a difference?

You;re right.

one script alias per virtual host is of course OK. only muultiple ones per virtual host would cause problems.

I assume, that you have the proof, that this virtrualhost section is really used, because you can access the non-cgi contents without problems, right?

To be sure. that there's no ownership problem I propose to post the output of following command

ls -lsd / /home/ /home/domain/ /home/domain/public_html/ /home/domain/public_html/somename/ /home/domain/public_html/somename/cgi-bin/

Select allOpen in new window

and also

ls -ld /home/domain/public_html/somename/cgi-bin/a.* 

Select allOpen in new window

or however your cgi-script python and shell scripts are called.

Thanks gelonida, I really appreciate the help.

I assume, that you have the proof, that this virtrualhost section is really used, because you can access the non-cgi contents without problems, right?

Yes, we can access html files through the browser with no problem, and these are files saved with someotheruser as the owner.

Here is what you requested:

# ls -lsd / /home/ /home/domain/ /home/domain/public_html/ /home/domain/public_html/somename/ /home/domain/public_html/somename/cgi-bin/
4 dr-xr-xr-x. 31 root     root     4096 May  9 17:21 /
4 drwx--x--x. 14 root     root     4096 May  7 01:32 /home/
4 drwx--x--x  21 domain   nobody   4096 May 13 01:02 /home/domain/
4 drwxr-xr-x  14 domain domain 4096 May 12 18:29 /home/domain/public_html/
4 drwxr-xr-x  17 someotheruser someotheruser 4096 May  8 23:02 /home/domain/public_html/somename/
4 drwxr-xr-x   2 someotheruser someotheruser 4096 May  9 16:50 /home/domain/public_html/somename/cgi-bin/

# ls -ld /home/domain/public_html/somename/cgi-bin/*.*
-rwxr-xr-x 1 root  root     62 May  9 15:47 /home/domain/public_html/somename/cgi-bin/a_sh.cgi
-rwxr-xr-x 1 someotheruser someotheruser 34819 May  8 22:25 /home/domain/public_html/somename/cgi-bin/cgi.py
-rw-r--r-- 1 root  root  34967 May  8 22:30 /home/domain/public_html/somename/cgi-bin/cgi.pyc
-rwxr-xr-x 1 someotheruser someotheruser 123 May  8 22:30 /home/domain/public_html/somename/cgi-bin/a.py

#

Select allOpen in new window

Hmmm, I'm slowly running out of ideas.

The top level permissions are quite restricted (user apache can not read the contents of /home or of /home/domain, but the x permission is set so it can enter the directory
this should normally cause no problem.

But you can try to make access a little more permissive (perhaps apache, python or bash don't like it.)
Try to call next command as root (for testing):

chmod a+rx /home /home/domain

Select allOpen in new window

Then restart your apache server and try to access /cgi-bin/a.cgi

What I am a little confused about is, that the cgi-bin directory is a sub directory of the Documentroot.
In my opinion this should not cause problems, but I never setup my web projects like this.
I had the cgi directory in a director parallel to he document root.

Perhaps you can try to change
the script alias to

ScriptAlias /cgi-bin/ /home/domain/public_html/somename_cgi-bin/

Select allOpen in new window

and to

mv  /home/domain/public_html/somename/cgi-bin/  /home/domain/public_html/somename_cgi-bin/

Select allOpen in new window

You should also add following in your virtualhost section

<Directory "/home/domain/public_html/somename_cgi-bin/">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all

</Directory>

Select allOpen in new window

Hi gelonida,

Thanks for getting back to me so quick.

Before I do that, I just want to make sure I can get the permissions back to where they were in case the change has no effect.

On the listing, there is something I do not understand.  Did you see how the /home/ directory permissions were listed:

4 drwx--x--x. 14 root     root     4096 May  7 01:32 /home/

Select allOpen in new window

Did you notice the dot (.) after the permissions?  That doesn't show up on all the directories.  What does that mean?

chmod go-r /home /home/domain

Select allOpen in new window

will undo the effect of m suggested  

chmod a+rx /home /home/domain

Select allOpen in new window

which could have been in the current context just a

chmod go+r /home /home/domain

Select allOpen in new window

No idea about the trailing dot, but the chmod commands, that I suggested explicitely just change single permissions and leave the rest intact

Hi gelonida,

Thanks again for all your help.

I tried making the change.  The permissions on /home and /home/domain were changed successfully to become executable, but that made no difference displaying the python or cgi files in the browser.

Your last statement

chmod a+rx /home /home/domain

Select allOpen in new window

was the correct one to get the permissions back the way they were.

Do you think it's time I ask for more help?  Maybe get some server specialists in here?

Whoa, I think you may have stumbled onto something!

I copied the files from /home/domain/public_html/somename/cgi-bin/ into another cgi-bin directory under another VirtualHost and tried to execute the files.  They gave me the same results (execute under SSH, 500 error on the web browser.)  So now I know there is something on the server that is preventing the execution of cgi-scripts in general from within VirtualHosts.

The directory /usr/local/apache/cgi-bin/ is not part of a virtualserver.  Here is the excerpt from httpd.conf where it appears:

<Directory "/">
    Options ExecCGI IncludesNOEXEC Indexes SymLinksIfOwnerMatch
    AllowOverride All
</Directory>

<Directory "/usr/local/apache/htdocs">
    Options Includes Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all

</Directory>

<Files ~ "^error_log$">
    Order allow,deny
    Deny from all

    Satisfy All
</Files>

<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all

    Satisfy All
</FilesMatch>

<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    CustomLog "logs/access_log" common

    <IfModule logio_module>
        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

    </IfModule>

</IfModule>

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin/"

</IfModule>

<Directory "/usr/local/apache/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all

</Directory>

<IfModule mod_log_config.c>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent

    CustomLog logs/access_log common

</IfModule>

<IfModule mime_module>
    TypesConfig conf/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

</IfModule>

Select allOpen in new window

Do you think I should just open another question?

Thanks gelonida for sticking in there in spite of the fact you were running out of ideas.  Your question about being able to run cgi on a different virtualserver sparked me to do some invesigation and found out this was a CPanel issue.  I needed to create a new account through WHM which would create a separate public_html folder for just that user.  Once that folder was set up, and ownership and permissions of the files and directories were normalized to the new user, everything worked like a charm.

Thanks again, and if I have any more python questions, I will probably post to this question to see if I can get your attention.

Best Regards,

Jason