[SOLVED] CURL HTTP/1.1 403 Forbidden Date

Issue

$url = "http://search.aol.com/aol/search?q=hello";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5');   
$html = curl_exec($ch);
curl_close($ch);
echo $html;

This return:

HTTP/1.1 403 Forbidden

Forbidden

You don't have permission to access /aol/search on this server.

Anything curl setting that would acceess to get the result.

Solution

Many servers will check the request headers carefully to confirm that a human is making a request (preferably from a browser agent), rather than an automated process. All caveats aside about unauthorized access to their resources, you can add some Accept headers to get a 200 response. In this case, I was able to get a 200 response using only one additional header:

curl -i \
-H 'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11' \
http://search.aol.com/aol/search?q=hello

Answered By - vaporbook

Answer Checked By - Clifford M. (WPSolving Volunteer)