IIS 10 on Microsoft Server 2016 with PHP7 FastCGI invokes cron-job by executing URL serverside, results in Error 500 (probably timeout?)

Hi,
I'm running Joomla 3.8.5 on Windows server 2016 with IIS v10.0, PHP 7.2.2 (32-bit) all on an Azure VM Standard A2 v2 (2 vcpus, 4 GB memory).
On Joomla I am running JomSocial 4.5.1, which is community software. Users can amoung other things, upload video to post on site. Uploaded video's are converted with ffmpeg which I installed on my server, version version N-90143-gb6652f5100.
ffmpeg is called by a scheduled task running periodically.

When uploading no so big video's, everything works fine, video's got converted an publshed on site as they should.
When uploading bigger video's (+/- 10-50Mb) video's don't get published.

I've gone through extensive troubleshooting with JomSocial support. We can see video upload is fine.
To troubleshoot we stopped scheduled task on web server (whick invokes ffmpeg with URL) and ran URL to invoke ffmeg manually on same web server.

For small video's, you can see output of URL telling video conversion is succesfull and video is published on site.
For larger video's, there is no output and after about +/- a minute, error 500 - Internal server error is shown.
In background ffmpeg is still runing for minutes after that and when you look in the file location where converted files end up, you can see the converted video is just fine. But, the video is not published because the URL which invokes ffmpeg and published video afterwards, has died (error 500).

So ffmpeg is not the problem, because conversion was successfull. But some way, the batch-process which the URL sets off, was not able to finish.

To troubleshoot, I've upgraded the Azure VM to A8_V2 (8 CPU's, 16Gb of memory). During ffmpeg conversion on my 2 core VM, the ffmpeg process raises CPU load to 100% over a longer period. On this temporary 8 core VM, ffmpeg is also high, but not 100% and also the conversion process is much quicker. Now the URL which invokes ffmpeg shows conversion is ok and video is published. This is where I get error 500 on the 2 core VM.

So my conclusion is, that when server is in less stress, the whole conversion and publishing process works, and when server is in stress, conversion also works, but the job which should wait till conversion is complete, has died before it can finish publication of the video.

I asumed that inside IIS I could try to raise the "Ping Period" of the worker process, on which the website runs. So I raised ping from default 30 sec. to 120 sec. Made no difference, same error 500. I also disabled the ping to stop health monitoring, but also no difference.

Hope anyone has an idea how to tackle this problem. Running permanently a costly 8 core VM for only a very short moment of conversion and publication of video is absurd, since during normal load my 2 core VM is more than enough and 4 times cheaper.

Regards, Rini
RBraatAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

nociSoftware EngineerCommented:
Error 500 means the intermediate proxy is unable to connect to the backend system.
==> can the proxy server do the queries to the backend system (can you browse from there...)
if not try to fix the issues n reaching the backend first.
If the backend IS reachable then you need to check the proxy config & error logs of the webserver.
David Johnson, CD, MVPOwnerCommented:
why don't you just split the processing of the video onto another azure instance?
save the video to a azure storage location run an azure batch file to do the conversion and retrieve the video to be published.

What happens if several users upload a video at the same time?  Auto-Scaling will help here (use cpu load)
RBraatAuthor Commented:
Hi noci,
Please read my question carefully, the url runs on web servers itselve, there is no proxy.
Regards Rini
Acronis True Image 2019 just released!

Create a reliable backup. Make sure you always have dependable copies of your data so you can restore your entire system or individual files.

RBraatAuthor Commented:
Hi David,
I cannot split the process. I use commercial software JomSocisl, that is as it is. I’m not a developer. I understand splitting would have done advantages, but that is not an option.
Regards Rini
RBraatAuthor Commented:
I’m quite sure it has something to do with a timeout, limitation or threshold within IIS, which happens on slower machine with 2 cores, and not on faster machine with 8 cores, where FFmpeg finishes within some threshold or acceptable time.
I also tried settings queue length from 1000 to 5000 but that also did not help me.
Regards, Rini
David Johnson, CD, MVPOwnerCommented:
have you tried scaling your web server depending upon cpu load?
RBraatAuthor Commented:
Sorry no experience with that
Julian HansenCommented:
Your script is probably timing out - set_time_limit(1800) in your conversions script might help.

However, I wouldn't do it that way - running video conversion as part of the upload script is just wrong. The right way to do it is write the video to a queue (folder) and have a scheduled process or a watcher monitor the queue - when video's come in it processes them and then notifies when the process is done. This is the only real solution available when your site starts to scale - unless you want to bring your server to its knees and chop its head off.

There are solutions out there for managing media uploads and conversions - trying to do them through a PHP script is not the optimal approach.

I would go with either a cron job that runs every 5 min or similar OR a daemon that watches a folder for incoming files - when it sees the folder has content it spawns a process to convert it. The spawned process the communicates (database write / message) when it is done and the status. A secondary process can then take over to inform whoever needs to know that the conversion has been completed.

You might be able to get the process you have for low traffic and small files - but I don't see this as a viable long term solution.

Having been through a couple of projects that do similar things with large files I can say from experience that once your load gets above a very low limit your server grinds to a halt. Meanwhile your client is sitting watching a spinner with no idea what is happening. Much better to say Video Received - we will email you / update screen / send sms when your video is ready.
RBraatAuthor Commented:
Hi Julian,

I think you misunderstood a little bit of my explanation. On the site, visitors can upload a video. At the end of the upload a message is shown, telling them that upload is complete and that video will be converted and they will be notified when converion and publishment is done, just as you describe.
On the backend on webserver, there is a process which runs periodically, which does administrative task. Among these tasks, it converts uploaded video's if they are present. At the end of the conversion, the converted video is published and user is notified. How this periodic task is setup is described here https://documentation.jomsocial.com/wiki/Setting_Up_Cron_Job 
But I'm not running on Linux, I'm running on Windows, so I use a scheduled task whick repeats itself every 5 minutes.
So far exactly as you describe.
Small video's are processed as should and get published and user receives a notification.
Large video's however not. So I swtiched on ffmpeg loggin and noticed that conversion was fully executed and converted video is available for publication. But they wont get published.

So next thing I did was in Azure scaling up to 8 cores. Then also large video's were converted, published and user got notification. So software and installation is OK. Then I scaled down again in Azure to 2 cores, the VM I'm continuing to use in future.

Then I looked at the contents of the cron job. As you can see in former link, the cron job is nothing else than an URL (http://www.domain.com/index.php?option=com_community&task=cron) which has to be invoked on webserver. I do that with a scheduled task, which call's a powershell script. In that script I invoke the URL with command: Invoke-WebRequest 'http://www.domain.com/index.php?option=com_community&task=cron'  | Select-Object -Expand Content.
To troubleshoot I disabled this scheduled task and in browser on webserver itself is ran "http://myshadowmc.nl/index.php?option=com_community&task=cron", exactly the same URL. For smaller video's I get output stating conversion was fine. For larger files I get error 500.

So putting it all together: the cronjob just not gets finished. Video is converted (because it kicked off ffmpeg), but there was no publication and no user got informed, because the URL-job, which was in charge of that, just went dead with error 500.

In this link https://documentation.jomsocial.com/wiki/Video_Linking_%26_Upload_Requirements you can see parameters to change in php.ini to overcome timeouts, but that did not did the trick.

On IIS, I also go to FastCGI settings on webserver, selected my php-cgi,exe instance en EDIT. I've set "Activity Timeout" tot 300 and "Idle Timeout" to 300, but still get the error 500 message as output on the URL-cron-job.
Only difference is, BEFORE is changed "Activity Timeout" and "Idle Timeout", I got error 500 after about 70-80 seconds (original timeout value) and ffmpeg continues to run, now jhe URL-job keeps running during the time ffmpeg is also running. Now, after giving "Activity Timeout" and "Idle Timeout" a higher value, I get an error 500 exactly on the moment ffmpeg terminates in background becourse conversion is done. So still something times out... cannot figure out what.

Just another thing I noticed, I don't think it has something to do with issue: website (php-cgi.exe) runs as IIS worker process (as I configured it), ffmpeg however, which is invoked bij cron-url which dies out, runs as IUSR. Both IIS worker process and IUSR have read/wite persmissions in all relevant folders.

Hope this explanation makes more sense now.

Regards, Rini
RBraatAuthor Commented:
SOLVED!
I've spend some extensive time going through all PHP-code, even I'm not a developer at all. At some place I've found a "Set_Time_Limit(120), meaning the corn-script is not allowed to run for more then 2 minutes. I've changed it to a higher value, enough to convert video, publish it and notify user.
That did the trick, working now like it should.
So in the end, setting the "Activity Timeout" and "Idle Timeout" parameters of fastcgi in IIS to a higher value, as described earlier, was the most important action. Without changing that, every PHP-process will get killed after 90 seconds, it's default value.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Julian HansenCommented:
Understood - sounds like you are on top of it.
RBraatAuthor Commented:
Increasing script execution time with set_time_limit, together with fastcgi Params solved issue
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
FFmpeg

From novice to tech pro — start learning today.