FIX: Stop DistributedMutex burning CPU while acquiring a lock

FIX: Stop DistributedMutex burning CPU while acquiring a lock

In Discourse, this can happen when redis enters readonly mode. This commit adds an exponential backoff to the polling time to reduce CPU usage, and will throw an exception after approximately 1 minute of being unable to lock.

diff --git a/lib/mini_scheduler/distributed_mutex.rb b/lib/mini_scheduler/distributed_mutex.rb
index 4f566f6..a37f905 100644
--- a/lib/mini_scheduler/distributed_mutex.rb
+++ b/lib/mini_scheduler/distributed_mutex.rb
@@ -1,5 +1,6 @@
 module MiniScheduler
   class DistributedMutex
+    class Timeout < StandardError; end
 
     @default_redis = nil
 
@@ -18,11 +19,28 @@ module MiniScheduler
       @mutex = Mutex.new
     end
 
+    MAX_POLLING_ATTEMPTS ||= 60
+    BASE_SLEEP_DURATION ||= 0.001
+    MAX_SLEEP_DURATION ||= 1
+
     # NOTE wrapped in mutex to maintain its semantics
     def synchronize
       @mutex.lock
+
+      attempts = 0
+      sleep_duration = BASE_SLEEP_DURATION
       while !try_to_get_lock
-        sleep 0.001
+        sleep sleep_duration
+
+        # Exponential backoff
+        if sleep_duration >= MAX_SLEEP_DURATION
+          sleep_duration = MAX_SLEEP_DURATION
+        else
+          sleep_duration = sleep_duration * 2
+        end
+
+        attempts += 1
+        raise Timeout if attempts >= MAX_POLLING_ATTEMPTS
       end
 
       yield

GitHub sha: 6daa5969

1 Like