Client Concurrent Requests with Symfony: How We Passed from 40 Minutes to 10 Minutes Fetching APIs

Ever faced the challenge of processing vast amounts of API data efficiently? We recently tackled this exact problem, transforming what was a data-fetching marathon into a blazing fast sprint.

Here’s how we tackled it on a project that required indexing approximately 1 million elements. To accomplish this, we needed to fetch these items from an external API and store them in our index. This process involved making numerous API calls — we’re talking thousands of requests to retrieve all the necessary data.

The full indexing operation was designed to run as a one-shot process. While we wouldn’t need to run it frequently (we’d have delta updates for day-to-day operations), we wanted the initial indexing to be as efficient as possible. In scenarios where data recovery might be necessary, speed would be crucial.

🏃The Marathon: Sequential Requests

Our initial implementation followed a straightforward approach: crawl the API, get items from one page, process them, then move to the next page. Simple and logical, right?

The problem? It was painfully slow. The entire process took around 40 minutes to complete.

Here’s what our original code looked like conceptually:

use Symfony\Component\HttpClient\HttpClient;

$client = HttpClient::create();

$page = 1;
do {
    // Make request for current page.
    $response = $client->request('GET', "https://api.example.org/items?page={$page}");
    
    // Process response.
    $data = $response->toArray();
    $items = $data['items'];
    
    foreach ($items as $item) {
        // Index each item.
        $this->indexItem($item);
    }
    
    $page++;
} while (isset($data['next_page']));

// Total: ~40 minutes

Just like in a marathon where each mile must be completed one after another, our requests were processed sequentially:

// Perform and wait for page 1...
// Perform and wait for page 2...
// Perform and wait for page X...
// Total: X * duration of request

Each request had to complete before the next one began, creating a linear time relationship. With thousands of pages to process, the time added up quickly.

🏃💨 The Sprint: Implementing Concurrent Requests

To improve our performance, we needed a different approach — something more akin to a sprint than a marathon. This came in the form of multiplexing — a technique that allows multiple requests to be processed concurrently rather than sequentially.

In the world of HTTP requests, multiplexing allows you to send multiple requests simultaneously without waiting for previous ones to complete. Instead of running the entire course one mile at a time, it’s like having multiple runners each covering different segments simultaneously.

Symfony’s HTTP client supports this through its stream() method, which allows us to monitor multiple responses concurrently and handle them as they complete.

Here’s how we refactored our approach using Symfony’s HTTP client with multiplexing:

use Symfony\Component\HttpClient\HttpClient;

$client = HttpClient::create([
    'max_duration' => 30, // Set maximum request duration.
]);

// Function to process in batches.
function processBatch($startPage, $batchSize) {
    global $client;
    
    // Build the list of pages to fetch.
    $responses = [];
    for ($page = $startPage; $page < $startPage + $batchSize; $page++) {
        $responses[] = $client->request('GET', "https://api.example.org/items?page={$page}");
    }
    
    foreach ($client->stream($responses) as $response => $chunk) {
        $userData = $response->getInfo('user_data');

        if ($chunk->isTimeout()) {
            $io->warning('Request timeout for page '. $userData['page']);
        }

        if ($chunk->isFirst()) {
            if (200 !== $response->getStatusCode()) {
                $io->error('Error fetching data for page '. $userData['page'].' with statusCode: '.$response->getStatusCode());
                continue;
            }
        }

        if ($chunk->isLast()) {
             // ... indexing logic.
        }
    }
}

// Process the entire dataset in batches of 50 pages.
$totalPages = 20000; // 1 million items with 50 items per page.
$batchSize = 50;

for ($i = 1; $i <= $totalPages; $i += $batchSize) {
    processBatch($i, $batchSize);
    
    // Add a small delay between batches to avoid rate limiting.
    usleep(500000); // 0.5 seconds.
}

The key differences in this approach:

1. We initiate all requests upfront without waiting for responses
2. We use the stream() method to process responses as they arrive
3. The total execution time is now primarily determined by the slowest request rather than the sum of all requests

This approach gave us a dramatic performance improvement, cutting our processing time from 40 minutes to just 10 minutes.

It’s important to note that this approach works so well precisely because our data items are independent of each other. Each API page and the items it contains can be fetched and processed without depending on the results from any other page. This independence makes our scenario perfect for concurrent processing. If your data has dependencies (where processing one item requires the results of another), you might need a different strategy or a hybrid approach.

🏁 Crossing the Finish Line

By implementing concurrent requests through Symfony’s multiplexing capability, we dramatically improved our data processing efficiency, reducing the indexing time.

This approach not only saved time but also made our system more resilient and scalable. When dealing with large datasets and API interactions, the ability to process requests concurrently rather than sequentially can lead to significant performance improvements.

Remember that while concurrency improves performance, it’s important to be mindful of API rate limits and server capacity. Always implement proper error handling and consider batch processing for very large datasets.

Just as a runner chooses the right technique for the right distance, choosing the right request processing strategy can make all the difference in your application’s performance.

Have you implemented similar optimizations in your projects? I’d love to hear about your experiences in the comments !

❤️ Happy coding!

Sources

Jakub Skowron (August 2023) Handling Multiple Requests Seamlessly with Symfony Lock.
https://medium.com/@skowron.dev/handling-multiple-requests-seamlessly-with-symfony-lock-3c09e3bf6d89

The official Symfony documentation HTTP Client.
https://symfony.com/doc/current/http_client.html#multiplexing-responses

Nuno Maduro (July 2022) Chain of Responsibility. https://nunomaduro.com/speed_up_your_php_http_guzzle_requests_with_concurrency

Ali Ebrahimpour (August 2023) Efficient Concurrent Requests Handling in Laravel Using HTTP Pooling.
https://medium.com/@ali74.ebrahimpour/efficient-concurrent-requests-handling-in-laravel-using-http-pooling-a40196b551c7

Resources

Claude AI, https://claude.ai
Assisted with writing and refining my English.

DALL-E & Midjourney, https://openai.com & https://www.midjourney.com
Generated the very accurate article’s cover illustration.

This article has been first write for the Blog of Antistatique — Web Agency in Lausanne, Switzerland.
A place where I work as Senior Backend Developer. Feel free to read it here or check it out there: https://antistatique.net/en/blog

All images copyright of their respective owners.
Big thanks to @Antistatique for the review.