How do I get rid of the error: maximum execution time of 60 minutes exceeded, when loading a json file with laravel

use the set_time_limit()  function

<?php
set_time_limit(0);  // global setting

function doStuff()
{
    set_time_limit(10);   // limit this function
    //  stuff
    set_time_limit(10);   // give ourselves another 10 seconds if we want
    //  stuff
    set_time_limit(0);    // the rest of the file can run forever
}

// ....
sleep(900);
// ....
doStuff();  // only has 10 secs to run
// ....
sleep(900);
// ....

Select allOpen in new window

Looking at your code, the following appears to be correct.

1) You're running this code using a localhost (127.0.0.1) interface.

2) This might be a true LAN machine or a WAN (public) machine where you simply are running packets through localhost.

3) And you're using port 8080.

4) Also you're... as a hidden side effect... maybe attempting to contact a Payment Gateway, like PayPal or Authorize or Stripe.

5) If #4 is correct, your request will be blocked silently forever, as all Payment Gateways... well... at least all the big players... require connections come from sites only implementing TLSv1.2 or TLSv1.3, so contact attempts from localhost will either error out with an odd error... causing in retry logic to trigger... forever... leading to an infinite loop.

6) Or... the code in your save() is running to long, which might relate to either your TCP connections or database speed.

Whatever the cause, having the source of save() + all source save() calls will be required to answer your question.

Tip: You can also figure this out by inserting trace statements in save() before + after all I/O (network + disk + SQL) which will show you instantly where time is being used up.

Because 60 seconds is a veritable eternity in TCP time, this is almost surely a network block or network I/O error or some return code error (from somewhere) triggering an infinite loop retrying TCP connections.

Hey there,

Based on what we've previously discussed, the problem is simply down to your input file size. The sample I saw has 10,000 records, so your code is running at least 20,000 individual database queries (INSERTS), plus the overhead of instantiating over 20,000 objects, and running 10,000 regular expressions.

There are optimisations you can make to your code, to cut down the execution time (remove the ternary checks on checked and interest - not needed). For the date of birth, at the DB level (Migration), either allow Nulls, or set a Default Value, and run Carbon::createFromFormat on the value. Your whole sensitive_data block can be reduced to:

Creditcard::create($contactDetail['credit_card']);

as long as you've set up mass-assignment correctly in your Model.

Those changes may give you a little performance boost, but if not enough, then you have several choices.

1. Increase the Timeout as detailed by David. This may be fine for testing, but in a Live environment, you don't want to leave your Users hanging on a non-responsive screen for more than 60 seconds !!

2. Split the file into smaller recordsets. Don't know where you're getting the data from so it may not be possible

3. Code the logic to loop over your JSON file in batches of 10 records (2000 queries), or even 100 records (200 queries).

4. Pass the processing off to a Queued Job. In Laravel, this is generally the preferred approach - in your Controller method, you just call a queued job, and then return control back to the User. The built-in scheduling will then deal with the Job in the background.

Thank you chris,

i do think creating a job is the best option.

I do have a lil issue with creating a job, its not working
I am now trying to figure out how to create one properly

this is my job file: UserDataProcess

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        //
        $contactInfo = Storage::disk('local')->exists('challenge.json') ? json_decode(Storage::disk('local')->get('challenge.json'), true) : [];
        
            $chunks = array_chunk($contactInfo, 1000);


            foreach ($chunks[0] as $contactDetail ){


                $contact = new User; // assumes you have a Model called Contact
                


                $contact->name = $contactDetail['name'];
                $contact->address = $contactDetail['address'];
                $contact->checked = (!$contactDetail['checked'] === false) ? $contactDetail['checked'] : 'false';
                $contact->description = $contactDetail['description'];
                $contact->interest = (!$contactDetail['interest'] === null) ? $contactDetail['interest'] : 'null';
                $contact->date_of_birth = (!is_null($contactDetail['date_of_birth'])) 
                ? 
                substr(preg_replace("([^0-9/])", "", $contactDetail['date_of_birth']), 0, 8) 
                : 
                new DateTime('2000-01-01');
              
                $contact->email = (!$contactDetail['email'] === null) ? $contactDetail['email'] : 'null';
                $contact->account = $contactDetail['account'];
                
                $sensitive_data = new Creditcard;
                $sensitive_data->type = $contactDetail['credit_card']['type'];
                $sensitive_data->number = $contactDetail['credit_card']['number'];
                $sensitive_data->name = $contactDetail['credit_card']['name'];
                $sensitive_data->expirationDate = $contactDetail['credit_card']['expirationDate'];


                
                $contact->save();
                $sensitive_data->save();
                $contact->creditcards()->attach($sensitive_data->id);
            }
    }
}

Select allOpen in new window

.env:
I also set the Queue_Connection to database instead of sync

QUEUE_CONNECTION=database

Select allOpen in new window

Route:

use App\Jobs\UserDataProcess;

Route::get('/test', function () {


    
    UserDataProcess::dispatch();
   
});

Select allOpen in new window

I think I suppose to see the job in the tablejobs but I ont see anything

I tried to foreach it and echo it out

echo Carbon::createFromFormat('d/m/Y', $contactDetail['date_of_birth']);

this is what I see

Hey there,

Have you already got your queue running ?

If you're not seeing the jobs in your DB, check the failed_jobs table.

While debugging, just do something really simple in your Job (create a simple text file), just to make sure it runs:

public function handle()
{
    Storage::disk('local')->put('job_test.txt', 'The Job ran successfully');
}

Select allOpen in new window

As for your Carbon Exception, that shows that the date_of_birth you passed into createFromFormat is NOT in the d/m/Y format

Hi Chris thank you for replying,

I refresh the migrations and then checked the database and didnt saw anything in the jobs table or the failed_jobs table

I tried : php artisan queue:work

But for some reason it never  executes

then I tried running: php artisan queue:work --force
in maintainance mode but also doenst executes, it keeps hanging and does nothing

OK,

Show the full code for your Job class - I'm presuming you've made this a very simple test. Also, you haven't said whether the Job Queue is actually running (while testing, make sure it's NOT running). Also show the code you're using to dispatch the Job (either a route closure or Controller method).

Route:
https://github.com/Ted-bot/Laravel_upload_json/blob/master/routes/web.php

Job:
https://github.com/Ted-bot/Laravel_upload_json/tree/master/app/Jobs

OK. Your Route doesn't fire the Job, and your Job doesn't run anycode - what exactly are you expecting to happen.

Like I've said before, create a Job that does one simple thing (like writing text to a file), and create a quick, closure based Route that dispatches the Job.

Make sure you Job Queue isn't run and the fire a request to your Route.

Thanks for the tip

I am no pro with laravel,
or else I wouldnt as at all

No worries. Regardless of Laravel, when trying to debug code,  it makes it much easier if you simplify it as much as possible ... so in your case,  a single line route closure that fires a job and a single line job that performs a basic operation. This was,  there's no other code involved that complicates the problem

I though just calling the job class and calling the dispatch is all I needed to get it working
can you point me to the right direction how to set up the job correctly

I have been watching and reading a couple of tutorials,
But I am missing something to get it right.

Excuse my unknowingness :p