Uploading lots of small files can be made significantly faster by parallelizing the
s3.put_object calls. In testing, an UPLOAD_CONCURRENCY of 10 made a large restore 10x faster. An UPLOAD_CONCURRENCY of 20 made the same restore 18x faster.
This commit is careful to parallelize as little as possible, to reduce the chance of concurrency issues. In the worker threads, no database transactions are performed. All modification of shared objects is controlled with a mutex.
Unfortunately we do not have any existing tests for the
ToS3Migration class. This change has been tested with a large site backup (120k uploads totalling 45GB)