500 Internal Error

I am running WHMCS software used for billing which is built using PHP & MySQL. Recently we have been receiving some issues when it is accessed through API by our Payment gateway 2Checkout.com.  

Apache access log shows following lines which shows the server got 500 internal server error:

64.128.115.196 - - [13/Sep/2018:10:54:58 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:59 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:59 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:58 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:55:00 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:58 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:58 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:58 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:58 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:58 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:55:00 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:58 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:55:00 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:55:00 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:54:59 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:55:00 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:55:01 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"
64.128.115.196 - - [13/Sep/2018:10:55:01 +0500] "POST /modules/gateways/callback/tco.php HTTP/1.1" 500 27 "-" "Jakarta Commons-HttpClient/3.1"

Open in new window


However nothing is recorded in error_logs about this issue and timestamp. Error logs have other issues recorded for different timestamps but no detail about this one.

Software vendor is unable to help as no details about the error are found. Payment gateway isn't provided any more details on this.

Can someone help me getting more details about this 500 Internal error.

[root@clients ~]# cat /etc/redhat-release 
CentOS release 6.9 (Final)
[root@clients ~]# php -version
PHP 5.6.31 (cli) (built: Sep 14 2017 18:31:45) 
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v10.2.0, Copyright (c) 2002-2018, by ionCube Ltd.
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
[root@clients ~]# httpd -version
Server version: Apache/2.2.15 (Unix)
Server built:   Aug 15 2017 18:04:44

Open in new window

sysautomationAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

girionisCommented:
Who has the tco.php file? This is the one who should look at their logs and tell you what's wrong.
0
sysautomationAuthor Commented:
I have this file on my server.
0
girionisCommented:
And there is nothing in the logs? Have you checked all log files? (not only the error logs).
0
Acronis True Image 2019 just released!

Create a reliable backup. Make sure you always have dependable copies of your data so you can restore your entire system or individual files.

sysautomationAuthor Commented:
All I found is /var/log/httpd/suexec.log

[2018-09-13 10:54:58]: uid: (500/clients) gid: (500/clients) cmd: php5.fcgi
[2018-09-13 10:54:59]: uid: (500/clients) gid: (500/clients) cmd: php5.fcgi
[2018-09-13 10:54:59]: uid: (500/clients) gid: (500/clients) cmd: php5.fcgi
[2018-09-13 10:54:59]: uid: (500/clients) gid: (500/clients) cmd: php5.fcgi
[2018-09-13 10:54:59]: uid: (500/clients) gid: (500/clients) cmd: php5.fcgi
[2018-09-13 10:54:59]: uid: (500/clients) gid: (500/clients) cmd: php5.fcgi
0
Martyn SpencerSoftware Developer / Linux System Administrator / Managing DirectorCommented:
It's possible that PHP error logging has been turned off. This could be in the framework that your site is using or in your php.ini file. If you know the framework in use, check how it recommends you enable error logging. If you can't do this, assuming that the file /modules/gateways/callback/tco.php has no syntactical errors (because if it does, the code below may not run), add the following lines to the file:
ini_set("log_errors", 1);
ini_set("error_log", "/tmp/php-errors.log");
error_log( "Hello, errors!" );

Open in new window

Check that it is syntactically correct using an online PHP parser and you can then check /tmp/php-errors.log to see if anything is present.

If this still does not work, since you are using Apache, you could create a .htaccess file where the php module is and add the following:
php_flag log_errors on 
php_value error_log /tmp/php-errors.log

Open in new window


This may provide you with more detail. Remember to undo your changes once you have found the cause.
1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
girionisCommented:
There must be some logging somewhere, except if it is turned off. When did the error start? Does it help if you replace the fcgi with cgi?

Can you also post the virtual hosts file? I want to see the SuexecUserGroup line.
1
sysautomationAuthor Commented:
Hello

The tco.php file is encrypted using IonCube therefore I don't see I can add the given code.

I also added following lines to .htaccess file but the new logfile still doesn't show anything:

php_flag log_errors on
php_value error_log /tmp/php-errors.log

Below is the configuration:

SuexecUserGroup "#500" "#500"
ServerName clients.mydomain.com
ServerAlias www.clients.mydomain.com
ServerAlias webmail.clients.mydomain.com
ServerAlias admin.clients.mydomain.com
DocumentRoot /home/clients/public_html
ErrorLog /var/log/virtualmin/clients.mydomain.com_error_log
CustomLog /var/log/virtualmin/clients.mydomain.com_access_log combined
ScriptAlias /cgi-bin/ /home/clients/cgi-bin/
ScriptAlias /awstats/ /home/clients/cgi-bin/
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/clients/public_html>
Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
AddType application/x-httpd-php .php
AddHandler fcgid-script .php
AddHandler fcgid-script .php5
FCGIWrapper /home/clients/fcgi-bin/php5.fcgi .php
FCGIWrapper /home/clients/fcgi-bin/php5.fcgi .php5
</Directory>
<Directory /home/clients/cgi-bin>
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
</Directory>
RewriteEngine on
RewriteCond %{HTTP_HOST} =webmail.clients.mydomain.com
RewriteRule ^(.*) https://clients.mydomain.com:20000/ [R]
RewriteCond %{HTTP_HOST} =admin.clients.mydomain.com
RewriteRule ^(.*) https://clients.mydomain.com:10000/ [R]
RemoveHandler .php
RemoveHandler .php5
php_admin_value engine Off
FcgidMaxRequestLen 1073741824
<Files awstats.pl>
AuthName "clients.mydomain.com statistics"
AuthType Basic
AuthUserFile /home/clients/.awstats-htpasswd
require valid-user
</Files>
SSLEngine on
SSLCertificateFile /home/clients/ssl.cert
SSLCertificateKeyFile /home/clients/ssl.key
SSLCACertificateFile /home/clients/ssl.ca
php_value memory_limit 32M
IPCCommTimeout 601
0
Martyn SpencerSoftware Developer / Linux System Administrator / Managing DirectorCommented:
What has changed? When did it stop working? If you have not made any changes or updates, can you verify that something else has not? Have usage patterns changed?

What I find when I search is that the server error 500 in your instance may be simply a memory limit that is too low, so increasing PHP memory may help. I am hesitant to suggest this as a fix because if it has always worked as-is up until recently there would be a reason why memory would now be an issue. So, have a think about my questions and perhaps consider trying an increase to the memory limit, but do consider why it may have changed. It could point towards issues that need attention.
0
sysautomationAuthor Commented:
A new version of the PHP application WHMCS has caused this in my point of view as this is the only thing that has changed.

But this is happening with a very few percentage of cases about 2-3% others going fine. The software vendor isn't cooperating much and only asking for anything in error logs.
0
girionisCommented:
What happens if you comment out this line?

SuexecUserGroup "#500" "#500"
0
Martyn SpencerSoftware Developer / Linux System Administrator / Managing DirectorCommented:
If it is only happening on occasion, it could genuinely be a resource issue and so increasing the memory limit of PHP may well help. A newer version of a PHP application may well have a slightly different memory footprint. As a troubleshooting step, it is certainly worth considering. Have you tried following the instructions here to enable error reporting? https://docs.whmcs.com/Enabling_Error_Reporting
0
sysautomationAuthor Commented:
It was a bug in the PHP application which the vendor now admitted and fixed.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Software

From novice to tech pro — start learning today.