wp_remote_request/cURL response timeout unless ran from command line

Problem: wp_remote_get/wp_remote_request of a specific https target fails to return a response unless ran via WP_CLI function.

$wp_remote_args => array(
    'timeout' => 5,
    'sslverify' => true,
    'blocking' => true,
    'user-agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0',
    'headers' => array(
        'Accept' => 'application/json, text/plain, */*',
        'Accept-Encoding' => 'gzip, deflate, br',
        'Accept-Language' => 'en-US,en;q=0.5',
        'Authorization' => '...',
        'Connection' => 'keep-alive',
        'Content-Type' => 'application/json; charset=UTF-8; text/plain',
        'Host' => '...',
        'Origin' => '...',
        'Referer' => '...',
wp_remote_request($url, $wp_remote_args)

Open in new window

I can only assume that something is perceived differently from the remote server when the GET request is received from standard PHP processing vs WP_CLI. Other wp_remote_request/cURL URLs are sent and received without issue.
Tim BunchAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

When it fails, is there an error message? Or does it just timeout?
Tim BunchAuthor Commented:
It simply times out, no response from the server. I should say that increasing the timeout length has no effect on the request.
Does it actually take the full 5 seconds to run the attempt, or does the request end early? So for example, if you increase the timeout to 30, does it sit there for 30 seconds, or does it simply indicate there's no response after a few seconds?

I'd hesitate to say that it has anything to do with a timeout unless the remote server / request is REALLY slow. If it finishes the function in LESS time than the timeout, then it might be failing for some other reason (but appearing incorrectly as a timeout).

I notice you don't capture the result of wp_remote_request($url, $wp_remote_args). What's the output if you var_dump it:

var_dump(wp_remote_request($url, $wp_remote_args));

You -SHOULD- get some kind of response from wp_remote_request, whether it's successful data or an instance of the WP_Error class (for failures). The error object should be a little more explicit in the problem details.

My gut says it's either an SSL verification problem or an extra header that's throwing things off, or a DNS / URL issue.
Learn SQL Server Core 2016

This course will introduce you to SQL Server Core 2016, as well as teach you about SSMS, data tools, installation, server configuration, using Management Studio, and writing and executing queries.

Tim BunchAuthor Commented:
It does take the full 5 seconds, or 30 seconds to make the attempt.

{"error":"No response from https://..."}

Open in new window

The odd thing here is that the function runs perfectly fine if initiated via command line. My thought was that it was headers or a ssl verification issue. I set ssl to not verify, and the headers are identical to the successful command line job.

The remote server must be differentiating between the two methods somehow to not return a response.
It's pretty unlikely that a server would timeout one request while allowing another, especially if they're coming from the same server and using SSL. It's not impossible, but very unlikely. Usually if a request is bad or malformed in some way, a server will simply reject it, but it will reject it using some HTTP error code or an error response of some kind, rather than timing out.

The timeout suggests that either:
1. The request isn't making it to the remote server.
2. The remote server is taking a really long time to process the request.
3. The response isn't making it BACK into your server.
4. The remote server has a firewall that is intercepting the request and not dropping it in a way that lets the client know.
5. There's some SSL protocol problems that aren't being immediately rejected.

I'd put my money on #1.

Now, WP uses cURL by default for HTTPS requests, so let's simplify this and just create a simple script that tries to replicate what you have on the command line. Can you provide the EXACT cURL command line you're using and redact nothing EXCEPT for the inner parts of the domain name and any password/key values? For example, if your request is:

curl https://www.innovationtechnologysomething.com/api/rest.php?key=2M4nyS3cret5&request=fetchpage

Then provide us with:

curl https://www.i[REDACTED]g.com/api/rest.php?key=2[REDACTED]5&request=fetchpage

Notice the first and last character of the redacted values are still there, but the request isn't exposed in any significant way.

Just to confirm, the WP instance is running from the exact same server where you're doing the command line cURL, correct? And you're not changing the URL for the command line (e.g. a test remote server vs. production remote server) ?

I'd also confirm versions. On the command line, do:
curl -V

Then create a separate PHP script with just these 2 lines:

Run both and provide the output here.
Tim BunchAuthor Commented:
This all occurs on a single EC2 instance. The CLI command uses the same PHP function to initiate the cURL operation. Nothing is different.

When I remove the authorization header, I get a response back saying it's missing. So I know my request is making it out. When I add the Authorization back in, I get a plain and simple timeout (unless the function is ran from the CLI):

{"errors":{"http_request_failed":["cURL error 28: Operation timed out after 15001 milliseconds with 313 out of 15051 bytes received"]},"error_data":[]}

Open in new window

My cURL version:

curl 7.51.0 (x86_64-redhat-linux-gnu) libcurl/7.51.0 NSS/3.28.4 zlib/1.2.8 libidn2/0.16 libpsl/0.6.2 (+libicu/50.1.2) libssh2/1.4.2
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets PSL 

Open in new window


Open in new window

It's an unusual puzzle, this one. I normally crack these things fairly quickly.
The CLI command uses the same PHP function to initiate the cURL operation.
So when you're talking about CLI, you're not running the cURL binary directly? You're running a PHP script that calls the cURL functions? If not, let's take all the overhead of WP's libraries out of the picture. Can you create a basic PHP script that has nothing but the bare minimum cURL call along?

$ch = curl_init('https://...your URL...');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERPWD, 'USERNAME_HERE:PASSWORD_HERE'); // Populates the Authorization HTTP header
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

// Just for debugging purposes
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
curl_setopt($ch, CURLOPT_VERBOSE, true);

$result = curl_exec($ch);

Open in new window

If that still gives you the timeout, then can you elaborate on the service you're calling? For example, is it some service that has to do things like database lookups and return data and if so, are you giving it a lot of parameters?
Tim BunchAuthor Commented:
The CORS configuration on the remote server was configured in such a way that caused the Origin to not be recognized by the PHP script unless ran from the CLI. The solution here is not on the requesting server's side.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Tim BunchAuthor Commented:
Resolution discovered through further testing of my own.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.