feat: limit rate of increase and decrease when hitting too many API resource errors

joolarequested to merge
limiter into main
  • Count the API errors for "not found resource", "unavailable resource" or "resource limit exceeded".
  • Slow the rate of increase or decrease calls when the count of API errors is above 10.
  • Use an exponential backoff sleep duration when slowing down the increase and decrease calls.

Example log message:

level=warning msg="too many failures, limiting request rate" duration=32s location=hel1 name=dev-docker-autoscaler operation=increase

To reproduce:

  • Start the gitlab-runner with the fleeting plugin
  • Delete the ssh key generated by the fleeting plugin.
  • See error during increase occur and rate limit starting.

Closes #30 (closed)

Merge request reports