Monday, February 5, 2024

[SOLVED] PHP curl HEAD request gives 404, but browser (or default curl GET request) loads page fine

February 05, 2024 amazon-s3, curl, php

Issue

I have a form that has fields for a couple of URLS. I wrote a Zend Framework validator that does a trivial preg_match to screen out ridiculous strings, and then does a curl HEAD request (CURLOPT_NOBODY) to screen out 404's and other connectivity issues. In testing I came across the mysterious return code 0 with "unknown SSL protocol error", so I added a check to accept as valid anything that gave a message with "SSL" in it, since that would suggest that the URL reached a webserver.

But one particular URL that our customers would likely use in practice redirects to an s3.amazonaws.com URL for a PDF file. In a browser, both the original URL, and the s3 URL it redirects to, display the PDF just fine. Since I used CURLOPT_FOLLOWLOCATION, I expected my validator would accept it. But instead it gave a 404. I then tried specifying the s3 URL directly, and that gave a 403(!). Thinking that possibly the 403 was triggered by the fact that I had specified a header of 'HTTP_X_REQUESTED_WITH: XMLHttpRequest', I commented out that line in the code. But it still gave a 403.

How can this happen? It seems to me that amazon s3 would have to look for HEAD requests explicitly, and deliberately issue a 404 or 403 depending on whether it came via a redirect???

I suppose I could delete the CURLOPT_NOBODY to have it send a GET request, but that seems silly since I don't care about the body.

Here is my complete code:

<?php

class Oshk_ZendX_Validate_Url {
    static $debug = true;
    // Based on https://stackoverflow.com/a/42619410/467590
    const PATTERN = '/^(https?:\/\/)?[^" ]+(\.[^" ]+)*$/';

    public static function isValid($value) {
        $STDERR = fopen("php://stderr", "w");
        $value = (string) $value;
        $matches = array();
        if (! preg_match(self::PATTERN, $value, $matches)) {
            fwrite($STDERR, sprintf("File '%s', line %d, value '%s' does not match pattern '%s'\n", __FILE__, __LINE__, $value, self::PATTERN));
            fclose($STDERR);
            return false;
        }
        if (! array_key_exists(1, $matches)) {
            $value = "https://$value";
        }
        if (self::$debug) {
            fwrite($STDERR, sprintf("File '%s', line %d, \$value = '%s', \$matches = %s", __FILE__, __LINE__, $value, print_r($matches, true)));
        }
        // URL looks well-formed. Ask curl to send a HEAD request to it
        $ch = curl_init($value);
        if ($ch === false) {
            throw new Exception("curl_init($value) failed!");
        }
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_HEADER, 0); // From https://www.php.net/manual/en/curl.examples-basic.php
        curl_setopt($ch, CURLOPT_HTTPHEADER, array('HTTP_X_REQUESTED_WITH: XMLHttpRequest'));
        curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36');
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
        curl_setopt($ch, CURLOPT_NOBODY, true);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($ch, CURLOPT_FAILONERROR, true);
        if (self::$debug) {
            curl_setopt($ch, CURLOPT_VERBOSE, true);
            curl_setopt($ch, CURLOPT_STDERR, $STDERR);
        }
        $data = curl_exec($ch);
        $msg = curl_error($ch);
        $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        if (self::$debug) {
            // https://stackoverflow.com/a/14436877/467590
            $allinfo = curl_getinfo($ch);
            fwrite($STDERR, sprintf("File '%s', line %d, \$allinfo = %s\n", __FILE__, __LINE__, print_r($allinfo, true)));
        }
        curl_close($ch);
        if (self::$debug) {
            fwrite($STDERR,  sprintf("File '%s', line %d, data = '%s'\n", __FILE__, __LINE__, substr($data, 0, 255)));
        }
        if(! strlen($data) && $status != 0 && false === strpos($msg, 'SSL')) {
            fwrite($STDERR, sprintf("File '%s', line %d, '%s' gives bad status code %d when accessed, with message '%s'\n", __FILE__, __LINE__, $value, $status, $msg));
            fclose($STDERR);
            return false;
        }
        if (self::$debug) {
            fwrite($STDERR, sprintf("File '%s', line %d, url = '%s'\n", __FILE__, __LINE__, $value));
            fwrite($STDERR, sprintf("File '%s', line %d, data = '%s'\n", __FILE__, __LINE__, substr($data, 0, 255)));
        }
        unset($data);
        if (self::$debug) {
            fwrite($STDERR, sprintf("File '%s', line %d, \$msg = '%s'\n", __FILE__, __LINE__, $msg));
            fwrite($STDERR, sprintf("File '%s', line %d, \$status = '%s'\n", __FILE__, __LINE__, $status));
            fwrite($STDERR, sprintf("File '%s', line %d, \$value = '%s'\n", __FILE__, __LINE__, $value));
        }
        if (($status >= 100 & $status < 400) || false !== strpos($msg, 'SSL')) {
            fclose($STDERR);
            return true;
        }
        fwrite($STDERR, sprintf("File '%s', line %d, '%s' gives bad status code %d when accessed, with message '%s'\n", __FILE__, __LINE__, $value, $status, $msg));
        fclose($STDERR);
        return false;
    }
}

echo var_dump(Oshk_ZendX_Validate_Url::isValid($argv[1]));

Here is the bash shell session running it with the original URL:

Here's the same thing using the s3 URL it redirects to:

Solution

I added a check to accept as valid anything that gave a message with "SSL" in it

This seems dangerous. What if the error message is "Invalid SSL certificate"?

since that would suggest that the URL reached a webserver

This true of any response -- 300, 400, 500, whatever. If your connection didn't timeout, then you've successfully connected to something, regardless of the status code. I.e., by this logic, if "reaching a webserver" is what you're validating, then only a timeout should fail.

I suppose I could delete the CURLOPT_NOBODY to have it send a GET request, but that seems silly since I don't care about the body.

You can't expect that every URL will be successfully reachable via a HEAD request, or that the results of HEAD request will always be the same as the results of a GET request.

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

Don't do this. If the verify fails, you want the request the fail, that's the whole point of SSL.

Overall, if you're not going to validate the actual content of the page, then I don't think it makes any sense to even make the request. Just validate the syntax of the URL. Otherwise, you're going to fail on things like transient network errors, maintenance downtimes, ad blockers, IP-based filtering, etc. You've got acres of code for what should just be one line:

class Oshk_ZendX_Validate_Url {
    public static function isValid(string $url): bool
    {
        return (bool) filter_var($url, FILTER_VALIDATE_URL);
    }
}

If you want to also test the connection and make sure there's a live server answering the request at the time of form submission, then the status doesn't really matter, and you can just check for a non-false return value from the HTTP wrapper via file_get_contents():

class Oshk_ZendX_Validate_Url {
    public static function isValid(string $url): bool
    {
        return filter_var($url, FILTER_VALIDATE_URL) &&
            file_get_contents($url) !== false;
    }
}

Answered By - Alex Howansky

Answer Checked By - Katrina (WPSolving Volunteer)

This Answer collected from stackoverflow and tested by PythonFixing community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Monday, February 5, 2024

[SOLVED] PHP curl HEAD request gives 404, but browser (or default curl GET request) loads page fine

Issue

Solution

Popular Posts

Labels