FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)

FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)

This adds a few different things to allow for direct S3 uploads using uppy. These changes are still not the default. There are hidden enable_experimental_image_uploader and enable_direct_s3_uploads settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new ExternalUploadStub model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used.

Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new generate-presigned-put endpoint in UploadsController. This generates an S3 key in the temp folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an ExternalUploadStub and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

Completing a direct S3 upload

Once the upload to S3 is done we call the new complete-external-upload route with the unique identifier of the ExternalUploadStub created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the ExternalUploadManager.

  1. If the object in S3 is too large (currently 100mb defined by ExternalUploadManager::DOWNLOAD_LIMIT) we do not download and generate the SHA1 for that file. Instead we create the Upload record via UploadCreator and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to UploadCreator have been made to accommodate this.

  2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the UppyChecksum plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal UploadCreator path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in UploadCreator we follow the same copy + delete temp path that we do for files that are too large.

  3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by UploadsController as well.

Also in this PR is some refactoring of displayErrorForUpload to handle both uppy and jquery file uploader errors.

diff --git a/app/assets/javascripts/discourse/app/components/composer-editor.js b/app/assets/javascripts/discourse/app/components/composer-editor.js
index ce73e3b..c9d5566 100644
--- a/app/assets/javascripts/discourse/app/components/composer-editor.js
+++ b/app/assets/javascripts/discourse/app/components/composer-editor.js
@@ -845,7 +845,7 @@ export default Component.extend({
         this._xhr = null;
 
         if (!userCancelled) {
-          displayErrorForUpload(data, this.siteSettings);
+          displayErrorForUpload(data, this.siteSettings, data.files[0].name);
         }
       });
     });
diff --git a/app/assets/javascripts/discourse/app/components/create-invite-uploader.js b/app/assets/javascripts/discourse/app/components/create-invite-uploader.js
index 79fe2a1..c98064d 100644
--- a/app/assets/javascripts/discourse/app/components/create-invite-uploader.js
+++ b/app/assets/javascripts/discourse/app/components/create-invite-uploader.js
@@ -75,7 +75,7 @@ export default Component.extend({
 
     $upload.on("fileuploadfail", (e, data) => {
       if (data.errorThrown !== "abort") {
-        displayErrorForUpload(data, this.siteSettings);
+        displayErrorForUpload(data, this.siteSettings, data.files[0].name);
       }
       this.reset();
     });
diff --git a/app/assets/javascripts/discourse/app/components/uppy-image-uploader.js b/app/assets/javascripts/discourse/app/components/uppy-image-uploader.js
index 7e1ecbf..7b416ca 100644
--- a/app/assets/javascripts/discourse/app/components/uppy-image-uploader.js
+++ b/app/assets/javascripts/discourse/app/components/uppy-image-uploader.js
@@ -1,4 +1,5 @@
 import Component from "@ember/component";
+import { or } from "@ember/object/computed";
 import UppyUploadMixin from "discourse/mixins/uppy-upload";
 import { ajax } from "discourse/lib/ajax";
 import discourseComputed from "discourse-common/utils/decorators";
@@ -25,6 +26,8 @@ export default Component.extend(UppyUploadMixin, {
     }
   },
 
+  uploadingOrProcessing: or("uploading", "processing"),
+
   @discourseComputed("imageUrl", "placeholderUrl")
   showingPlaceholder(imageUrl, placeholderUrl) {
     return !imageUrl && placeholderUrl;
diff --git a/app/assets/javascripts/discourse/app/lib/uploads.js b/app/assets/javascripts/discourse/app/lib/uploads.js
index 480a300..04500a0 100644
--- a/app/assets/javascripts/discourse/app/lib/uploads.js
+++ b/app/assets/javascripts/discourse/app/lib/uploads.js
@@ -1,4 +1,5 @@
 import I18n from "I18n";
+import deprecated from "discourse-common/lib/deprecated";
 import bootbox from "bootbox";
 import { isAppleDevice } from "discourse/lib/utilities";
 
@@ -279,40 +280,39 @@ export function getUploadMarkdown(upload) {
   }
 }
 
-export function displayErrorForUpload(data, siteSettings) {
+export function displayErrorForUpload(data, siteSettings, fileName) {
+  if (!fileName) {
+    deprecated(
+      "Calling displayErrorForUpload without a fileName is deprecated and will be removed in a future version."
+    );
+    fileName = data.files[0].name;
+  }
+
   if (data.jqXHR) {
     const didError = displayErrorByResponseStatus(
       data.jqXHR.status,
       data.jqXHR.responseJSON,
-      data.files[0].name,
+      fileName,
       siteSettings
     );
     if (didError) {
       return;
     }
-  } else if (data.errors && data.errors.length > 0) {
-    bootbox.alert(data.errors.join("\n"));
-    return;
-  }
-  // otherwise, display a generic error message
-  bootbox.alert(I18n.t("post.errors.upload"));
-}
-
-export function displayErrorForUppyUpload(response, fileName, siteSettings) {
-  if (response.body.errors && response.body.errors.length > 0) {
-    bootbox.alert(response.body.errors.join("\n"));
-    return;
-  } else {
+  } else if (data.body && data.status) {
     const didError = displayErrorByResponseStatus(
-      response.status,
-      response.body,
+      data.status,
+      data.body,
       fileName,
       siteSettings
     );
     if (didError) {
       return;
     }
+  } else if (data.errors && data.errors.length > 0) {
+    bootbox.alert(data.errors.join("\n"));
+    return;
   }
+
   // otherwise, display a generic error message
   bootbox.alert(I18n.t("post.errors.upload"));
 }
diff --git a/app/assets/javascripts/discourse/app/lib/uppy-checksum-plugin.js b/app/assets/javascripts/discourse/app/lib/uppy-checksum-plugin.js
new file mode 100644
index 0000000..c14e03f
--- /dev/null
+++ b/app/assets/javascripts/discourse/app/lib/uppy-checksum-plugin.js
@@ -0,0 +1,94 @@
+import { Plugin } from "@uppy/core";
+import { warn } from "@ember/debug";
+import { Promise } from "rsvp";
+
+export default class UppyChecksum extends Plugin {
+  constructor(uppy, opts) {
+    super(uppy, opts);
+    this.id = opts.id || "uppy-checksum";
+    this.capabilities = opts.capabilities;
+    this.type = "preprocessor";
+  }
+
+  canUseSubtleCrypto() {
+    if (!window.isSecureContext) {
+      this.warnPrefixed(
+        "Cannot generate cryptographic digests in an insecure context (not HTTPS)."
+      );
+      return false;
+    }
+    if (this.capabilities.isIE11) {
+      this.warnPrefixed(
+        "The required cipher suite is unavailable in Internet Explorer 11."
+      );
+      return false;
+    }
+    if (
+      !(window.crypto && window.crypto.subtle && window.crypto.subtle.digest)
+    ) {
+      this.warnPrefixed(
+        "The required cipher suite is unavailable in this browser."
+      );
+      return false;
+    }
+
+    return true;
+  }
+
+  generateChecksum(fileIds) {
+    if (!this.canUseSubtleCrypto()) {
+      return Promise.resolve();
+    }
+
+    let promises = fileIds.map((fileId) => {
+      let file = this.uppy.getFile(fileId);
+
+      this.uppy.emit("preprocess-progress", file, {
+        mode: "indeterminate",
+        message: "generating checksum",
+      });
+
+      return file.data.arrayBuffer().then((arrayBuffer) => {
+        return window.crypto.subtle
+          .digest("SHA-1", arrayBuffer)
+          .then((hash) => {
+            const hashArray = Array.from(new Uint8Array(hash));
+            const hashHex = hashArray
+              .map((b) => b.toString(16).padStart(2, "0"))
+              .join("");
+            this.uppy.setFileMeta(fileId, { sha1_checksum: hashHex });
+          })
+          .catch((err) => {
+            if (
+              err.message.toString().includes("Algorithm: Unrecognized name")
+            ) {
+              this.warnPrefixed(
+                "SHA-1 algorithm is unsupported in this browser."
+              );
+            }
+          });
+      });
+    });
+
+    const emitPreprocessCompleteForAll = () => {
+      fileIds.forEach((fileId) => {
+        const file = this.uppy.getFile(fileId);
+        this.uppy.emit("preprocess-complete", file);
+      });
+    };
+
+    return Promise.all(promises).then(emitPreprocessCompleteForAll);
+  }
+
+  warnPrefixed(message) {
+    warn(`[uppy-checksum-plugin] ${message}`);
+  }
+
+  install() {
+    this.uppy.addPreProcessor(this.generateChecksum.bind(this));
+  }
+
+  uninstall() {
+    this.uppy.removePreProcessor(this.generateChecksum.bind(this));
+  }
+}
diff --git a/app/assets/javascripts/discourse/app/mixins/upload.js b/app/assets/javascripts/discourse/app/mixins/upload.js
index b626f6c..780b375 100644
--- a/app/assets/javascripts/discourse/app/mixins/upload.js
+++ b/app/assets/javascripts/discourse/app/mixins/upload.js
@@ -108,7 +108,7 @@ export default Mixin.create({
     });
 
     $upload.on("fileuploadfail", (e, data) => {
-      displayErrorForUpload(data, this.siteSettings);
+      displayErrorForUpload(data, this.siteSettings, data.files[0].name);
       reset();
     });
   }),
diff --git a/app/assets/javascripts/discourse/app/mixins/uppy-upload.js b/app/assets/javascripts/discourse/app/mixins/uppy-upload.js
index 60a028d..0408b88 100644
--- a/app/assets/javascripts/discourse/app/mixins/uppy-upload.js
+++ b/app/assets/javascripts/discourse/app/mixins/uppy-upload.js

[... diff too long, it was truncated ...]

GitHub sha: b500949ef688277c53c396e4d7e90cdbf4918d06

This commit appears in #13787 which was approved by eviltrout and tgxworld. It was merged by martin.