FIX: Reduce input of to_tsvector to follow limits (#13806)

FIX: Reduce input of to_tsvector to follow limits (#13806)

Long posts may have cooked fields that produce tsvectors longer than the maximum size of 1MiB (1,048,576 bytes). This commit uses just the first million characters of the scrubbed cooked text for indexing.

Reducing the size to exactly 1MB (1_048_576) is not sufficient because sometimes the output tsvector may be longer than the input and this gives us some breathing room.

diff --git a/app/services/search_indexer.rb b/app/services/search_indexer.rb
index de3553d..6fa15e4 100644
--- a/app/services/search_indexer.rb
+++ b/app/services/search_indexer.rb
@@ -120,7 +120,11 @@ class SearchIndexer
       a_weight: topic_title,
       b_weight: category_name,
       c_weight: topic_tags,
-      d_weight: scrub_html_for_search(cooked)
+      # Length of a tsvector must be less than 1_048_576 bytes.
+      # The difference between the max ouptut limit and imposed input limit
+      # accounts for the fact that sometimes the output tsvector may be
+      # slighlty longer than the input.
+      d_weight: scrub_html_for_search(cooked)[0..1_000_000]
     ) do |params|
       params["private_message"] = private_message

GitHub sha: 823c3f09d44ab89e88c4910abe36899bb23d601d

This commit appears in #13806 which was approved by ZogStriP. It was merged by udan11.