python cgi script fails with apache modsecurity

I have a python cgi script that does the following:

- With a browser pointing to, an html form is loaded containing a welcome message and form fields followed by a Submit1 and a Reset button. When the user clicks on Submit1, a confirmation page is loaded, and when the user clicks on Reset, the form is reloaded with all user entries cleared. So far, so good.
- The confirmation page is a new html form that contains a "Please confirm your inputs" message with the form fields as completed by the user, followed by a Submit2 and a Reset button. On this second page, the user can make changes or leave inputs as is. The next step is to click on Submit2 to confirm or on Reset to start over.

This is where the script breaks. If the user clicks on Submit2 or on Reset, apache loads a "Forbidden / you do not have access to myscript.cgi" page in the browser. Checking the error log, modsecurity warns of an sql injection attack. (On my lab PC everything works but I've logged the IP of my lab PC into the modsecurity config file.)  

What to do? To narrow down the issue, we should focus on the Reset button behavior (why it works on the first page but fails on the second). TIA for your feedback.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

On the second page you will not have the information objects loaded, so they are being interpreted as an injection attack because the data objects from the first page are not being initialised in the second page correctly.
sara_bellumAuthor Commented:
Ok that makes sense, thanks. But I have no idea how to proceed. The path to the template_form variable is defined in a config file that is imported, and there are several functions in the script to generate this rather long form. So I tried to initialize the file that they generate with:

class initClass:
    def __init__(self):
        self.html_form = None
    def initPage(self):
        self.html_form = template_form

but nothing happens if the methods I'm using aren't nested inside the class, so I presume it's not doing anything or not being read. When my methods are nested inside the class, my script won't execute because they aren't defined (even when adding self as the first argument to each method). What to do?
What executes when the reset button is submitted?  

What form data is returned in each form case for the reset button?

It is the data in the form post that needs to be examined.  Have a look at the first page to see what it is posting, and then look at the second form to see what is different about the post.

Presumably the reset button is invoking the same function on form post for both forms.
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

sara_bellumAuthor Commented:
The form that contains the Submit2 button mirrors the start page with the Submit1 button, but the second page contains user-submitted values along with the form fields. Where form fields contain non-empty values, the key/value pairs are also saved to a list that is printed at the top of the new form. The form action is the same for both forms and is called in this way:

if __name__ == '__main__':

    print "Content-type: text/html \n"

    if 'Submit1' in form:
       # run 3 methods that:
            -  append field names and values to a list that is printed at the top of the destination form (template_form)
            -  print the html form fields with the user-submitted inputs to file (body.html)
            -  print a warning message for each required field containing an empty value:
               on my lab PC, the form reloads on submit with updated values and warning  
               message(s) until all required fields have non-null values  
       # shutil copy the header, body and footer data to the destination form
       # load the destination form via python template substitution

    elif 'Submit2' in form:
      # run 3 methods that:
         - append field names and values to a list that is printed at the top of the next page
         - print the html form fields with the user inputs to body.html, but this part of the script is not used *
      # load a confirmation page with the list of form fields and values submitted; use template substitution for this
      # insert the form values into a mysql table with column names corresponding to the form fields

    elif 'Reset' in form:
        # empty out the list and the form fields dictionary and load the start page.
           (The html source for the Reset button on both forms is identical.)

* I added a shutil copy instruction here for consistency which made no difference.
sara_bellumAuthor Commented:
I got the sequence wrong for the warning message on required fields: the form updates on each submission on my lab PC thanks to the write-copy-load sequence described.  But the warning message is not material to the problem, since it can only work when the Submit2 and Reset buttons on the destination template_form are fully functional.
Does anything get populated in the form fields for submit2?

Is the same error message present for Reset or Submit2?

Perhaps supplying the source might make things clearer.
sara_bellumAuthor Commented:
When the user completes all required fields, the template_form that is loaded populates the form fields and asks the user to confirm his inputs: in this case, the submit button name is Submit2.

If the user does not complete all required fields, the template_form that is loaded populates the form fields and warns the user that required inputs are missing: in this case, the submit button name is Submit1.

So if there are missing values for required fields,  the user must fix this first, and only then can he confirm all of of his form value inputs on the page.

Whatever the situation, the template_form is generated dynamically on each submit action and loaded with form field values as submitted by the user. The only difference between Submit1 and Submit2 form actions is the html template that is loaded.

Modsecurity does not block the first form submit action from a static html page - presumably this default python cgi behavior is supported.  But any subsequent submit action is using a dynamically generated web form as its source, with a  filename / path that is not identical to the original form that was used.  

If this is the problem I don't know how to fix it.
sara_bellumAuthor Commented:
As I suspected, if I use the dynamically generated form as the default start page instead of the static one, no submit action is accepted at all - modsecurity blocks execution of any form post from html pages generated by my script.

I'm aware that string substitution is vulnerable to sql injection, in particular this syntax:
template_form.write('<div> %s %s </div>' % (key, value)) so I changed it to
template_form.write('<div>' + key + ':  ' + value + '</div>')
but the apache modsecurity errors are the same.  

So for example a text input is built like this:
'<td>' + k +  ' <input type="text" name="' + k + '" value="' + v + '"></td>\n'
which prints the stored values to the form fields when executed from my lab PC.
 Very tedious, but I can understand and maintain this code.
If apache has correctly labeled the problem and you've really got a sql injection problem then you're not properly encoding your SQL statements and they're open to injection.  In other words, it sounds like you're building your queries dynamically by hand.  Doing something like this:

form_data = cgi.FieldStorage()
field_name = form_data.getfirst('some_field')
field_value = form_data.getfirst('some_value')

sql = "update some_table set %s = '%s'" % (field_name, field_value)
cursor = conn.execute(sql)

Open in new window

If that (or something similar) is what you're doing, you shouldn't be doing it.  You should be using placeholders in your queries.  If placeholders aren't possible and there isn't any other way to build your query other than dynamically, then you have to make sure you're escaping everything that could possibly cause a problem with your query syntax.   How and where you'd do that depends on the query that you're trying to run, the database that you're using, and the library that you're using to connect to it.  In the above example, if you were using MySQLdb to connect to you database, you should have done this to prevent an injection problem:
sql = "update some_table set %s = %%s" % (MySQLdb.escape_string(field_name))
cursor = conn.execute(sql, (field_value,))

Open in new window

In the above, I used a placeholder to escape the value and I manually escaped the fieldname-- because placeholders can only stand in for values (not syntax elements).  Anyway... if it's really a sql injection problem, then you should post the code showing us how you're building and executing your sql statements.  

I say "if" because based on some of your comments, it appears that you're also open to html script injection attacks.  So not sure which apache would really be complaining about.   Unless you've properly escaped your key and value variables beforehand, neither of these statements are safe:
"<div> %s %s </div>" % (key, value)
"<div>" + key + ": " + value + "</div>"

Open in new window

They're nowhere near as dangerous as a sql injection attack, but they're still problems and it's possible apache would complain about them.   I don't think to the point of preventing the page from executing, but who knows.  To make them safe, they need to be html escaped-- see the cgi.escape function (
"<div> %s %s </div>" % (cgi.escape(key, True), cgi.escape(value, True))

Open in new window


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
sara_bellumAuthor Commented:
Thanks clockwatcher, I know that I have not protected the script against html injection attacks, so I'll follow the link / your advice and see where it gets me.  

Since I haven't reached the point where sql injection can be an issue, I may or may not have a problem there as well.  But one step at a time.  I'm not near my lab PC at the moment, and mimicking the script behavior / environment remotely will be difficult.  Will do what I can and post my progress as soon as possible.
It also might help if you posted the full error message you're receiving in the apache log.
sara_bellumAuthor Commented:
I'm quite certain that the problem was html injection. I wrote a short script that mimics the (much longer) one I was working with and the error disappeared, so I'm concluding that the solution was the cgi.escape syntax you recommended.   Errors may reappear in the full script but that's another problem for another day.  Thanks very much, your feedback was terrific!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.