FEATURE: Include optimized thumbnails for topics (#9215)

FEATURE: Include optimized thumbnails for topics (#9215)

This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

  • Introduces new image_upload_id columns on the posts and topics table. This replaces the old image_url column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

  • A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

  • Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

  • Topic lists and topics now include a thumbnails key, which includes all the available images:

    "thumbnails": [
    {
      "max_width": null,
      "max_height": null,
      "url": "//example.com/original-image.png",
      "width": 1380,
      "height": 1840
    },
    {
      "max_width": 1024,
      "max_height": 1024,
      "url": "//example.com/optimized-image.png",
      "width": 768,
      "height": 1024
    }
    ]
    
  • Themes can request additional thumbnail sizes by using a modifier in their about.json file:

     "modifiers": {
       "topic_thumbnail_sizes": [
         [200, 200],
         [800, 800]
       ],
       ...
    

    Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

  • Two new raw plugin outlets are introduced, to improve the customisability of the topic list. topic-list-before-columns and topic-list-before-link

diff --git a/app/assets/javascripts/discourse/app/templates/list/topic-list-item.hbr b/app/assets/javascripts/discourse/app/templates/list/topic-list-item.hbr
index ab05872..03ea153 100644
--- a/app/assets/javascripts/discourse/app/templates/list/topic-list-item.hbr
+++ b/app/assets/javascripts/discourse/app/templates/list/topic-list-item.hbr
@@ -1,3 +1,5 @@
+{{~raw-plugin-outlet name="topic-list-before-columns"}}
+
 {{#if bulkSelectEnabled}}
   <td class="bulk-select">
     <input type="checkbox" class="bulk-select">
@@ -12,6 +14,7 @@
   at the end of the link, preventing it from line wrapping onto its own line.
 --}}
 <td class='main-link clearfix' colspan="1">
+  {{~raw-plugin-outlet name="topic-list-before-link"}}
   <span class='link-top-line'>
     {{~raw-plugin-outlet name="topic-list-before-status"}}
     {{~raw "topic-status" topic=topic}}
diff --git a/app/assets/javascripts/discourse/app/templates/mobile/list/topic-list-item.hbr b/app/assets/javascripts/discourse/app/templates/mobile/list/topic-list-item.hbr
index c126de0..6c4663b 100644
--- a/app/assets/javascripts/discourse/app/templates/mobile/list/topic-list-item.hbr
+++ b/app/assets/javascripts/discourse/app/templates/mobile/list/topic-list-item.hbr
@@ -1,4 +1,5 @@
 <td>
+  {{~raw-plugin-outlet name="topic-list-before-columns"}}
   {{~#unless expandPinned}}
   <div class='pull-left'>
     <a href="{{topic.lastPostUrl}}" data-user-card="{{topic.last_poster_username}}">{{avatar topic.lastPosterUser imageSize="large"}}</a>
@@ -14,6 +15,7 @@
           This causes the topic-post-badge to be considered the same word as "text"
           at the end of the link, preventing it from line wrapping onto its own line.
         --}}
+        {{~raw-plugin-outlet name="topic-list-before-link"}}
         <div class='main-link'>
           {{~raw-plugin-outlet name="topic-list-before-status"}}
           {{~raw "topic-status" topic=topic~}}
diff --git a/app/jobs/regular/generate_topic_thumbnails.rb b/app/jobs/regular/generate_topic_thumbnails.rb
new file mode 100644
index 0000000..a014526
--- /dev/null
+++ b/app/jobs/regular/generate_topic_thumbnails.rb
@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module Jobs
+  class GenerateTopicThumbnails < ::Jobs::Base
+    sidekiq_options queue: 'ultra_low'
+
+    def execute(args)
+      topic_id = args[:topic_id]
+      extra_sizes = args[:extra_sizes]
+
+      raise Discourse::InvalidParameters.new(:topic_id) if topic_id.blank?
+
+      topic = Topic.find(topic_id)
+      topic.generate_thumbnails!(extra_sizes: extra_sizes)
+    end
+
+  end
+end
diff --git a/app/jobs/scheduled/ensure_db_consistency.rb b/app/jobs/scheduled/ensure_db_consistency.rb
index 6d7f3fb..6c21814 100644
--- a/app/jobs/scheduled/ensure_db_consistency.rb
+++ b/app/jobs/scheduled/ensure_db_consistency.rb
@@ -22,7 +22,8 @@ module Jobs
         CategoryTagStat,
         User,
         UserAvatar,
-        Category
+        Category,
+        TopicThumbnail
       ].each do |klass|
         klass.ensure_consistency!
         measure(klass)
diff --git a/app/models/post.rb b/app/models/post.rb
index f2ee105..c2f80c8 100644
--- a/app/models/post.rb
+++ b/app/models/post.rb
@@ -51,6 +51,8 @@ class Post < ActiveRecord::Base
 
   has_many :user_actions, foreign_key: :target_post_id
 
+  belongs_to :image_upload, class_name: "Upload"
+
   validates_with PostValidator, unless: :skip_validation
 
   after_save :index_search
@@ -1062,6 +1064,10 @@ class Post < ActiveRecord::Base
     Upload.where(access_control_post_id: self.id)
   end
 
+  def image_url
+    image_upload&.url
+  end
+
   private
 
   def parse_quote_into_arguments(quote)
@@ -1144,6 +1150,7 @@ end
 #  action_code             :string
 #  image_url               :string
 #  locked_by_id            :integer
+#  image_upload_id         :bigint
 #
 # Indexes
 #
@@ -1152,6 +1159,7 @@ end
 #  idx_posts_user_id_deleted_at              (user_id) WHERE (deleted_at IS NULL)
 #  index_for_rebake_old                      (id) WHERE (((baked_version IS NULL) OR (baked_version < 2)) AND (deleted_at IS NULL))
 #  index_posts_on_id_and_baked_version       (id DESC,baked_version) WHERE (deleted_at IS NULL)
+#  index_posts_on_image_upload_id            (image_upload_id)
 #  index_posts_on_reply_to_post_number       (reply_to_post_number)
 #  index_posts_on_topic_id_and_percent_rank  (topic_id,percent_rank)
 #  index_posts_on_topic_id_and_post_number   (topic_id,post_number) UNIQUE
diff --git a/app/models/theme_modifier_set.rb b/app/models/theme_modifier_set.rb
index 99a8818..9fae7fa 100644
--- a/app/models/theme_modifier_set.rb
+++ b/app/models/theme_modifier_set.rb
@@ -12,7 +12,7 @@ class ThemeModifierSet < ActiveRecord::Base
 
   def type_validator
     ThemeModifierSet.modifiers.each do |k, config|
-      value = public_send(k)
+      value = read_attribute(k)
       next if value.nil?
 
       case config[:type]
@@ -39,7 +39,7 @@ class ThemeModifierSet < ActiveRecord::Base
   def self.resolve_modifier_for_themes(theme_ids, modifier_name)
     return nil if !(config = self.modifiers[modifier_name])
 
-    all_values = self.where(theme_id: theme_ids).where.not(modifier_name => nil).pluck(modifier_name)
+    all_values = self.where(theme_id: theme_ids).where.not(modifier_name => nil).map { |s| s.public_send(modifier_name) }
     case config[:type]
     when :boolean
       all_values.any?
@@ -50,6 +50,26 @@ class ThemeModifierSet < ActiveRecord::Base
     end
   end
 
+  def topic_thumbnail_sizes
+    array = read_attribute(:topic_thumbnail_sizes)
+
+    return if array.nil?
+
+    array.map do |dimension|
+      parts = dimension.split("x")
+      next if parts.length != 2
+      [parts[0].to_i, parts[1].to_i]
+    end.filter(&:present?)
+  end
+
+  def topic_thumbnail_sizes=(val)
+    return write_attribute(:topic_thumbnail_sizes, val) if val.nil?
+    return write_attribute(:topic_thumbnail_sizes, val) if !val.is_a?(Array)
+    return write_attribute(:topic_thumbnail_sizes, val) if !val.all? { |v| v.is_a?(Array) && v.length == 2 }
+
+    super(val.map { |dim| "#{dim[0]}x#{dim[1]}" })
+  end
+
   private
 
   # Build the list of modifiers from the DB schema.
@@ -78,11 +98,12 @@ end
 #
 # Table name: theme_modifier_sets
 #
-#  id                       :bigint           not null, primary key
-#  theme_id                 :bigint           not null
-#  serialize_topic_excerpts :boolean
-#  csp_extensions           :string           is an Array
-#  svg_icons                :string           is an Array
+#  id                               :bigint           not null, primary key
+#  theme_id                         :bigint           not null
+#  serialize_topic_excerpts         :boolean
+#  csp_extensions                   :string           is an Array
+#  svg_icons                        :string           is an Array
+#  topic_thumbnail_sizes            :string           is an Array
 #
 # Indexes
 #
diff --git a/app/models/topic.rb b/app/models/topic.rb
index aaeca3f..371f32d 100644
--- a/app/models/topic.rb
+++ b/app/models/topic.rb
@@ -28,6 +28,83 @@ class Topic < ActiveRecord::Base
     400
   end
 
+  def self.share_thumbnail_size
+    [1024, 1024]
+  end
+
+  def self.thumbnail_sizes
+    [ self.share_thumbnail_size ]
+  end
+
+  def thumbnail_job_redis_key(extra_sizes)
+    "generate_topic_thumbnail_enqueue_#{id}_#{extra_sizes.inspect}"
+  end
+
+  def filtered_topic_thumbnails(extra_sizes: [])
+    return nil unless original = image_upload
+    return nil unless original.width && original.height
+
+    thumbnail_sizes = Topic.thumbnail_sizes + extra_sizes
+    topic_thumbnails.filter { |record| thumbnail_sizes.include?([record.max_width, record.max_height]) }
+  end
+
+  def thumbnail_info(enqueue_if_missing: false, extra_sizes: [])
+    return nil unless original = image_upload
+    return nil unless original.width && original.height
+
+    infos = []
+    infos << { # Always add original
+               max_width: nil,

[... diff too long, it was truncated ...]

GitHub sha: 03818e64

1 Like

This commit appears in #9215 which was merged by davidtaylorhq.

This commit has been mentioned on Discourse Meta. There might be relevant details there:

Very nice! I read through it and it’s very clear.

This commit has been mentioned on Discourse Meta. There might be relevant details there:

https://meta.discourse.org/t/topic-list-previews/101646/1041

This commit has been mentioned on Discourse Meta. There might be relevant details there:

https://meta.discourse.org/t/thumbnail-generation-markdown-rendering-issue/152701/1