NodeJS – doing external calls in parallel

I am still getting my head around Node’s async model. The specific problem that I was tackling was calling a paged API endpoint. In most programming languages, you’d call the first page, wait to get the data, then call the second and so on. It turns out that there is a better way todo this in Node. Start by defining a few initial variables :

const fetch = require("node-fetch");
let currentPage = 1;
const queryUrl = '';
let data = [];

Then, create an array of all the urls you want to call. Well, this means you need to know the number of pages in advance, which can be done by doing a single call first. For simplicity I assume I know the number of pages:

let urls = [];
while(currentPage <= 5) {

Now, just do the calls. Since we are getting back promises, this code will not wait for each individual call to finish:

const promises = url => {
  try {
    console.log("Querying "+url)
    const response = await fetch(url);
    return response.json();
  catch(err) {

Finally, assemble back the data returned by each promise:

for (const promise of promises) {
  data = data.concat(await promise);

The data array would contain all the info, also sorted in the correct way no matter if the API calls responded in a different order. Neat.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s