Allow collecting web metrics as histograms (#186)

Allow collecting web metrics as histograms (#186)

Web metrics are currently reported as summaries, which provides precise data for each controller, action, and node. However, timings from low-throughput actions will cause an average to be wildly inaccurate, which means aggregating across controllers, actions, and nodes is not currently possible.

This adds an option to the middleware which instructs the collector to use histogram metrics rather than summaries, which allows estimating aggregates using histogram_quantile.

When present, the --histogram flag causes the default collectors to use histograms instead of summaries for aggregated metrics.

  • Sidekiq, DelayedJob duration support histograms

Resolves #92.

diff --git a/README.md b/README.md
index 6b032e5..07c1b2f 100644
--- a/README.md
+++ b/README.md
@@ -27,6 +27,7 @@ To learn more see [Instrumenting Rails with Prometheus](https://samsaffron.com/a
   * [Metrics default prefix / labels](#metrics-default-prefix--labels)
   * [Client default labels](#client-default-labels)
   * [Client default host](#client-default-host)
+  * [Histogram mode](#histogram-mode)
 * [Transport concerns](#transport-concerns)
 * [JSON generation and parsing](#json-generation-and-parsing)
 * [Logging](#logging)
@@ -776,6 +777,7 @@ Usage: prometheus_exporter [options]
     -c, --collector FILE             (optional) Custom collector to run
     -a, --type-collector FILE        (optional) Custom type collectors to run in main collector
     -v, --verbose
+    -g, --histogram                  Use histogram instead of summary for aggregations
         --auth FILE                  (optional) enable basic authentication using a htpasswd FILE
         --realm REALM                (optional) Use REALM for basic authentication (default: "Prometheus Exporter")
         --unicorn-listen-address ADDRESS
@@ -846,6 +848,18 @@ http_requests_total{service="app-server-01",app_name="app-01"} 1
 
 By default, `PrometheusExporter::Client.default` connects to `localhost:9394`. If your setup requires this (e.g. when using `docker-compose`), you can change the default host and port by setting the environment variables `PROMETHEUS_EXPORTER_HOST` and `PROMETHEUS_EXPORTER_PORT`.
 
+### Histogram mode
+
+By default, the built-in collectors will report aggregations as summaries. If you need to aggregate metrics across labels, you can switch from summaries to histograms:
+
+`‍``
+$ prometheus_exporter --histogram
+`‍``
+
+In histogram mode, the same metrics will be collected but will be reported as histograms rather than summaries. This sacrifices some precision but allows aggregating metrics across actions and nodes using [`histogram_quantile`].
+
+[`histogram_quantile`]: https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile
+
 ## Transport concerns
 
 Prometheus Exporter handles transport using a simple HTTP protocol. In multi process mode we avoid needing a large number of HTTP request by using chunked encoding to send metrics. This means that a single HTTP channel can deliver 100s or even 1000s of metrics over a single HTTP session to the `/send-metrics` endpoint. All calls to `send` and `send_json` on the `PrometheusExporter::Client` class are **non-blocking** and batched.
diff --git a/bin/prometheus_exporter b/bin/prometheus_exporter
index e0e0594..387d8eb 100755
--- a/bin/prometheus_exporter
+++ b/bin/prometheus_exporter
@@ -50,6 +50,9 @@ def run
     opt.on('-v', '--verbose') do |o|
       options[:verbose] = true
     end
+    opt.on('-g', '--histogram', "Use histogram instead of summary for aggregations") do |o|
+      options[:histogram] = true
+    end
     opt.on('--auth FILE', String, "(optional) enable basic authentication using a htpasswd FILE") do |o|
       options[:auth] = o
     end
diff --git a/lib/prometheus_exporter/metric/base.rb b/lib/prometheus_exporter/metric/base.rb
index 9593193..d8220ad 100644
--- a/lib/prometheus_exporter/metric/base.rb
+++ b/lib/prometheus_exporter/metric/base.rb
@@ -5,6 +5,7 @@ module PrometheusExporter::Metric
 
     @default_prefix = nil if !defined?(@default_prefix)
     @default_labels = nil if !defined?(@default_labels)
+    @default_aggregation = nil if !defined?(@default_aggregation)
 
     # prefix applied to all metrics
     def self.default_prefix=(name)
@@ -23,6 +24,14 @@ module PrometheusExporter::Metric
       @default_labels || {}
     end
 
+    def self.default_aggregation=(aggregation)
+      @default_aggregation = aggregation
+    end
+
+    def self.default_aggregation
+      @default_aggregation ||= Summary
+    end
+
     attr_accessor :help, :name, :data
 
     def initialize(name, help)
diff --git a/lib/prometheus_exporter/server/delayed_job_collector.rb b/lib/prometheus_exporter/server/delayed_job_collector.rb
index 85c3220..854c7db 100644
--- a/lib/prometheus_exporter/server/delayed_job_collector.rb
+++ b/lib/prometheus_exporter/server/delayed_job_collector.rb
@@ -76,12 +76,12 @@ module PrometheusExporter::Server
                 "delayed_jobs_max_attempts_reached_total", "Total number of delayed jobs that reached max attempts.")
 
         @delayed_job_duration_seconds_summary =
-            PrometheusExporter::Metric::Summary.new("delayed_job_duration_seconds_summary",
-                                                    "Summary of the time it takes jobs to execute.")
+            PrometheusExporter::Metric::Base.default_aggregation.new("delayed_job_duration_seconds_summary",
+                                                                     "Summary of the time it takes jobs to execute.")
 
         @delayed_job_attempts_summary =
-            PrometheusExporter::Metric::Summary.new("delayed_job_attempts_summary",
-                                                    "Summary of the amount of attempts it takes delayed jobs to succeed.")
+            PrometheusExporter::Metric::Base.default_aggregation.new("delayed_job_attempts_summary",
+                                                                     "Summary of the amount of attempts it takes delayed jobs to succeed.")
       end
     end
   end
diff --git a/lib/prometheus_exporter/server/runner.rb b/lib/prometheus_exporter/server/runner.rb
index 619e31b..6f3f9ae 100644
--- a/lib/prometheus_exporter/server/runner.rb
+++ b/lib/prometheus_exporter/server/runner.rb
@@ -17,6 +17,7 @@ module PrometheusExporter::Server
       @prefix = nil
       @auth = nil
       @realm = nil
+      @histogram = nil
 
       options.each do |k, v|
         send("#{k}=", v) if self.class.method_defined?("#{k}=")
@@ -27,6 +28,10 @@ module PrometheusExporter::Server
       PrometheusExporter::Metric::Base.default_prefix = prefix
       PrometheusExporter::Metric::Base.default_labels = label
 
+      if histogram
+        PrometheusExporter::Metric::Base.default_aggregation = PrometheusExporter::Metric::Histogram
+      end
+
       register_type_collectors
 
       unless collector.is_a?(PrometheusExporter::Server::CollectorBase)
@@ -47,7 +52,7 @@ module PrometheusExporter::Server
     end
 
     attr_accessor :unicorn_listen_address, :unicorn_pid_file
-    attr_writer :prefix, :port, :bind, :collector_class, :type_collectors, :timeout, :verbose, :server_class, :label, :auth, :realm
+    attr_writer :prefix, :port, :bind, :collector_class, :type_collectors, :timeout, :verbose, :server_class, :label, :auth, :realm, :histogram
 
     def auth
       @auth || nil
@@ -98,6 +103,10 @@ module PrometheusExporter::Server
       @label ||= PrometheusExporter::DEFAULT_LABEL
     end
 
+    def histogram
+      @histogram || false
+    end
+
     private
 
     def register_type_collectors
diff --git a/lib/prometheus_exporter/server/sidekiq_collector.rb b/lib/prometheus_exporter/server/sidekiq_collector.rb
index c0aff88..1149444 100644
--- a/lib/prometheus_exporter/server/sidekiq_collector.rb
+++ b/lib/prometheus_exporter/server/sidekiq_collector.rb
@@ -52,7 +52,7 @@ module PrometheusExporter::Server
       if !@sidekiq_jobs_total
 
         @sidekiq_job_duration_seconds =
-        PrometheusExporter::Metric::Summary.new(
+        PrometheusExporter::Metric::Base.default_aggregation.new(
           "sidekiq_job_duration_seconds", "Total time spent in sidekiq jobs.")
 
         @sidekiq_jobs_total =
diff --git a/lib/prometheus_exporter/server/web_collector.rb b/lib/prometheus_exporter/server/web_collector.rb
index f477d8f..5f5d7a4 100644
--- a/lib/prometheus_exporter/server/web_collector.rb
+++ b/lib/prometheus_exporter/server/web_collector.rb
@@ -33,22 +33,22 @@ module PrometheusExporter::Server
           "Total HTTP requests from web app."
         )
 

[... diff too long, it was truncated ...]

GitHub sha: 38d28c04285f2f5ee8e7861cf90c8a68430f61eb

This commit appears in #186 which was approved by SamSaffron. It was merged by SamSaffron.