Reduce memory use of file_store.rb (#362)

Reduce memory use of file_store.rb (#362)

What

Writing to disk via File.open by default is buffered. By disabling the buffer via setting sync = true we can reduce memory allocation. It looks like no matter how small the amount of data being written is a default buffer size is used.

Impact

While making a request to CodeTriage and measuring with memory_profiler via derailed benchmarks I see:

Before:
Total allocated: 742573 bytes (6644 objects)
48741 rack-mini-profiler/lib

After:
Total allocated: 734236 bytes (6645 objects)
40477 rack-mini-profiler/lib

This change accounts for

Diff:

(742573 - 734236) / 742573.0 # => 1.1 % decrease in memory

Other Considerations

Is this slower than the buffered option?

require 'benchmark/ips'

DATA = ["1ada37891cbb5442fa56157bc451a608", nil, 634602.016795]

def sync
  File.open("tmp-sync.dump", 'w') do |f|
    f.sync = true
    Marshal.dump(DATA, f)
  end
end

def no_sync
  File.open("tmp-no-sync.dump", 'w') do |f|
    Marshal.dump(DATA, f)
  end
end

Benchmark.ips do |x|
  x.report("no-sync") { no_sync }
  x.report("sync   ") { sync }
  x.compare!
end
# Warming up --------------------------------------
#              no-sync   597.000  i/100ms
#              sync      593.000  i/100ms
# Calculating -------------------------------------
#              no-sync      6.404k (±12.2%) i/s -     31.641k in   5.026470s
#              sync         6.416k (±12.4%) i/s -     32.022k in   5.079501s

# Comparison:
#              sync   :     6416.2 i/s
#              no-sync:     6404.3 i/s - same-ish: difference falls within error

It looks like the sync is not any slower and in most of my test runs was marginally faster (though well within standard deviation so it’s impossible to make that claim in a statistically significant manner).

From 9d7bd9ba1414382780704f6a0307d5766f37973d Mon Sep 17 00:00:00 2001
From: Richard Schneeman <richard.schneeman+no-recruiters@gmail.com>
Date: Thu, 13 Sep 2018 21:19:57 -0500
Subject: [PATCH] Reduce memory use of file_store.rb (#362)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## What

Writing to disk via `File.open` by default is buffered. By disabling the buffer via setting `sync = true` we can reduce memory allocation. It looks like no matter how small the amount of data being written is a default buffer size is used.

## Impact

While making a request to CodeTriage and measuring with memory_profiler via derailed benchmarks I see:

Before:
Total allocated: 742573 bytes (6644 objects)
     48741  rack-mini-profiler/lib

After:
Total allocated: 734236 bytes (6645 objects)
     40477  rack-mini-profiler/lib

This change accounts for

Diff:

(742573 - 734236) / 742573.0 # => **1.1 % decrease in memory**

## Other Considerations

Is this slower than the buffered option?

`‍``ruby
require 'benchmark/ips'

DATA = ["1ada37891cbb5442fa56157bc451a608", nil, 634602.016795]

def sync
  File.open("tmp-sync.dump", 'w') do |f|
    f.sync = true
    Marshal.dump(DATA, f)
  end
end

def no_sync
  File.open("tmp-no-sync.dump", 'w') do |f|
    Marshal.dump(DATA, f)
  end
end

Benchmark.ips do |x|
  x.report("no-sync") { no_sync }
  x.report("sync   ") { sync }
  x.compare!
end
# Warming up --------------------------------------
#              no-sync   597.000  i/100ms
#              sync      593.000  i/100ms
# Calculating -------------------------------------
#              no-sync      6.404k (±12.2%) i/s -     31.641k in   5.026470s
#              sync         6.416k (±12.4%) i/s -     32.022k in   5.079501s

# Comparison:
#              sync   :     6416.2 i/s
#              no-sync:     6404.3 i/s - same-ish: difference falls within error
`‍``


It looks like the sync is not any slower and in most of my test runs was marginally faster (though well within standard deviation so it's impossible to make that claim in a statistically significant manner).

diff --git a/lib/mini_profiler/storage/file_store.rb b/lib/mini_profiler/storage/file_store.rb
index 9fe91d8..304a513 100644
--- a/lib/mini_profiler/storage/file_store.rb
+++ b/lib/mini_profiler/storage/file_store.rb
@@ -24,7 +24,10 @@ module Rack
         end
 
         def []=(key,val)
-          ::File.open(path(key), "wb+") {|f| f.write Marshal.dump(val)}
+          ::File.open(path(key), "wb+") do |f|
+            f.sync = true
+            f.write Marshal.dump(val)
+          end
         end
 
         private

GitHub