[SOLVED] CURL in PHP to SCRAPYD using schedule.json not returning anything

Issue

I have set up Scrapyd on my server and everything seems to be working fine. I can use CURL to get a list of my spiders, like so curl -u super:secret http://111.111.111.111:6800/listspiders.json?project=myproject. I can also launch spiders using curl -u super:secret http://111.111.111.111:6800/schedule.json -d project=myproject -d spider=spider1.

Now I want to do this from PHP. The first CURL command is working fine:

<?php
  $username='super';
  $password='secret';
  
  $ch = curl_init();
  
  $url = 'http://111.111.111.111:6800/listspiders.json?project=myproject';
  #$url = "http://111.111.111.111:6800/schedule.json -d project=myproject -d spider=spider1";
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
  curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
  curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
  
  $response = curl_exec($ch);
  echo 'Type ($response): '.gettype($response). "<br/>";
  echo 'Response: '.$response;

  curl_close($ch);
?>

This returns the response as expected: {"node_name": "ubuntu-s-1vcpu-512mb-10gb-fra1-01", "status": "ok", "spiders": ["spider1", "spider2"]}.

However, when I change the URL in the snippet above to $url = "http://111.111.111.111:6800/schedule.json -d project=myproject -d spider=spider1"; then I don't get any response. No error, nothing. The gettype($response) returns boolean, for what it's worth.

Again, the CURL command, when using the terminal, is working fine and returns something like {"node_name": "ubuntu-s-1vcpu-512mb-10gb-fra1-01", "status": "ok", "jobid": "6a885343fa8233a738bb9efa11ec2e94"}

Any hints on what's going on here, or better ways to trouble shoot, are greatly appreciated.

Solution

In your PHP code when you change the URL to include -d parameters, you're actually trying to send the data in a way that's specific to command-line cURL, not PHP cURL. Hope the following helps.

<?php
$username = 'super';
$password = 'secret';

$ch = curl_init();

$url = 'http://111.111.111.111:6800/schedule.json';

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query([
    'project' => 'myproject',
    'spider' => 'spider1'
]));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");

$response = curl_exec($ch);

if ($response === false) {
    echo 'Curl error: ' . curl_error($ch);
}

echo 'Type ($response): ' . gettype($response) . "<br/>";
echo 'Response: ' . $response;

curl_close($ch);
?>

Answered By - TSCAmerica.com

Answer Checked By - David Marino (WPSolving Volunteer)