[SOLVED] Problem with large PHP cURL request (JSON)

Issue

We are programming a WooCommerce webshop for a customer. The goal is to sync the products from the client's new ERP system. The ERP system also is in development right now and provides a JSON interface for testing. The product import JSON contains more than 5.000 products with multiple variations right now.

I tried to do the request with postman for testing. The request took a relatively long time (approx. 8 minutes), but was successful. The response JSON file has a size of approx. 2 MB.

Now I tried to do the request with PHP but it doesn't work. After 1 hour (max_execution_time = 3600) In the server protocol I get the following error message:

"18454#0: *1484295 upstream timed out (110: Connection timed out) while reading response header from upstream"

I have set the following PHP settings:

memory_limit = 256M
post_max_size = 256M
upload_max_filesize = 256M
max_execution_time = 3600
max_input_time = 3600

The PHP code for the cURL request looks like this:

$curl = curl_init();
curl_setopt_array($curl, array(
    CURLOPT_URL => 'https://api.anqodo.com:8088/anqDataSvc/api/TofererExternal/webshopproducts?ws_id=' . $webshop_id . '&date_from=' . $date_from, // the date_from parameter can be set to limit the products to all edited products since this date. If I limit the date (e.g. 2024-01-01) the request works, because the response is smaller. But for the initial import I need all products
    CURLOPT_RETURNTRANSFER  => true,
    CURLOPT_ENCODING        => '',
    CURLOPT_MAXREDIRS       => 10,
    CURLOPT_TIMEOUT         => 3600,
    CURLOPT_FOLLOWLOCATION  => true,
    CURLOPT_CUSTOMREQUEST   => 'GET',
    CURLOPT_HTTPHEADER      => array(
        'RequestTimeClient: ' . date("Y-m-d\TH:i:s\Z"),
        'Content-Type: application/json',
        'Authorization: Bearer ' . $token // I retrieve the token in another function, which works fine
    ),
    ));

    $response = json_decode(curl_exec($curl));
    curl_close($curl);

    return $response;

I do not have much experience with such interfaces, which is why I need help. I don't really know what causes the problem. Is there a limit to the size of such a JSON response? Why does it work with Postman but not with PHP? What would be the best way to deal with this problem?

I tried to do the request with Postman, which worked.

After that I tried to to this request within my WordPress plugin, but also in a "normal" PHP file in the root path, to exclude possible issues with WordPress, which didn't work.

As mentioned before, I also tried to set the "date_from" parameter, to limit the response to all products, that were edited within the last 2 weeks or so. This worked perfectly.

Solution

function fetchProducts($webshop_id, $date_from, $token) {
    $batchSize = 100; // Number of products to fetch in each batch
    $currentPage = 1;
    $allProducts = [];

    do {
        $start = ($currentPage - 1) * $batchSize;
        $url = 'https://api.anqodo.com:8088/anqDataSvc/api/TofererExternal/webshopproducts?ws_id=' . $webshop_id . '&date_from=' . $date_from . '&start=' . $start . '&limit=' . $batchSize;

        $response = fetchBatch($url, $token);

        if (!$response || empty($response->products)) {
            // Handle error or break the loop if no more products
            break;
        }

        // Merge products from the current batch into the result array
        $allProducts = array_merge($allProducts, $response->products);

        $currentPage++;
    } while (count($response->products) == $batchSize); // Continue fetching until fewer products are returned than the batch size

    return $allProducts;
}

function fetchBatch($url, $token) {
    $curl = curl_init();
    curl_setopt_array($curl, array(
        CURLOPT_URL => $url,
        CURLOPT_RETURNTRANSFER  => true,
        CURLOPT_ENCODING        => '',
        CURLOPT_MAXREDIRS       => 10,
        CURLOPT_TIMEOUT         => 3600,
        CURLOPT_FOLLOWLOCATION  => true,
        CURLOPT_CUSTOMREQUEST   => 'GET',
        CURLOPT_HTTPHEADER      => array(
            'RequestTimeClient: ' . date("Y-m-d\TH:i:s\Z"),
            'Content-Type: application/json',
            'Authorization: Bearer ' . $token
        ),
    ));

    $response = json_decode(curl_exec($curl));
    curl_close($curl);

    return $response;
}

// Example usage:
$webshop_id = 123;
$date_from = '2024-01-01';
$token = 'your_access_token';

$products = fetchProducts($webshop_id, $date_from, $token);

Batch Processing: The fetchProducts function gets products in smaller groups (batches) by setting a starting point and a limit in the web address. The batch size is initially 100 but can be changed according to your requirements.

Loop through Batches: A do-while loop continues to get batches until the number of products received is smaller than the set batch size.

Merge Results: Products from each batch are combined into the $allProducts collection.

Separate Function: The fetchBatch function handles single requests to the ERP system.

Error Handling: Properly dealing with errors is crucial. If there's an issue or no more products are returned, the loop stops.

By using batch processing, you can handle a large dataset more effectively and decrease the risk of running into execution time limits. Adjust the batch size based on your server's capabilities and the ERP system's response times.

Answered By - Arif Rahman

Answer Checked By - Senaida (WPSolving Volunteer)