Automatically retry on error 429

[dobesv]

It would be nice to automatically retry if we get error 429 back from the server, indicating we have sent too many requests.

This is relatively easy to implement using bottleneck, see https://github.com/SGrondin/bottleneck#retries

If retries are not wanted as a default, perhaps the hubspot client object could pass-through the relevant events from bottleneck so the user of hubspot can listen to them, and provide an example in the docs how to add appropriate error handlers to retry.

To workaround the issue, we can add our own event handler on the limiter, but it requires accessing the undocumented internals of the hubspot client:

const hubspotClient = new Hubspot({
  apiKey,
  limiter: {
    id: 'hubspot',
    maxConcurrent: 2,
    minTime: 150,
    // Clustering options
    datastore: 'ioredis',
    clearDatastore: false,
    clientOptions: redisClientOptions,
    timeout: 60000,
  },
});

const errorMaxRetries = (error: any) => {
  switch (error.statusCode) {
    case 502:
    case 503:
    case 427: {
      return 10;
    }
    case 500: {
      return 1;
    }
  }
  switch (error.code) {
    case 'ECONNRESET':
    case 'ESOCKETTIMEDOUT':
    case 'ETIMEDOUT':
      return 10;
  }
  return 0;
};
((hubspotClient as any).limiter as Bottleneck).on(
  'failed',
  async (error: any, jobInfo: any) => {
    if (jobInfo.retryCount < errorMaxRetries(error)) {
      log.warn(error, 'Retrying HubSpot request.');
      return 500;
    }
  },
);

[vkhomiv-hubspot]

@pcothenet what do you think about this? As for me looks sensible. Probably extend wrapper Bottleneck setup?