Commit Graph

21 Commits

Author SHA1 Message Date
Marius 29b59a2c90 s3store: Use queue for parallel uploads 2021-04-27 00:34:23 +02:00
Marius 86a329cef2 Add first draft of parallel upload queue 2021-04-26 13:19:20 +02:00
Marius 8c5192c254 s3store: Move parts producer into own file 2021-04-26 12:18:09 +02:00
Marius b3bf854712 s3store: Remove test for handling unexpected EOF
The store must no longer handle unexpected EOFs since this is convered by the handler.
See the previous commits for more details
2021-04-26 12:17:20 +02:00
Marius 6d987aa226 core: Handle errors from reading request body centrally 2021-04-26 10:08:37 +02:00
Adam Jensen e85d630748
s3store: Fix data loss when using deferred upload lengths (#462) 2021-03-11 18:17:47 +01:00
Marius fdb19a7904 s3store: Temporary workaround against data loss on connection resets 2021-02-10 23:07:16 +01:00
Marius 67fd74e129 Add flag to disable calculating expensive hashes for S3 2021-01-18 13:31:07 +01:00
Marius 9831c8b4da
s3store: Increase default part size (#410)
* s3store: Increase default part size

* Update tests and add flag

* Fix minor issues
2020-08-01 14:58:31 +02:00
Adam Jensen 6662f43d01
s3Store: Concurrently write upload parts to S3 while reading from client (#402)
* Allow empty metadata values

* Make tests less fragile by allowing loose call ordering

* Add s3ChunkProducer

* Integrate s3ChunkProducer to support chunk buffering

* Remove completed chunk files inline to reduce disk space usage

* Add tests for chunk producer

* docs: Use value from Host header to forward to tusd

* Use int64 for MaxBufferedParts field

* Default to 20 buffered parts

* Rename s3ChunkProducer -> s3PartProducer

* Document s3PartProducer struct

* Clarify misleading comment

* Revert "Remove completed chunk files inline to reduce disk space usage"

This reverts commit b72a4d43d6.

* Remove redundant seek

This is already being done in s3PartProducer.

* Clean up any remaining files in the channel when we return

* Make putPart* functions responsible for cleaning up temp files

* handler: Add tests for empty metadata pairs

* Factor out cleanUpTempFile func

* Add test to ensure that temporary files get cleaned up

Co-authored-by: Jens Steinhauser <jens.steinhauser@gmail.com>
Co-authored-by: Marius <marius@transloadit.com>
2020-07-29 15:24:46 +02:00
Marius 9210fbe0fc s3store: Allow concatenation of uploads with less than 5MB 2020-04-27 12:42:56 +02:00
Marius 973a4fe066 s3store: Fix bug when completing empty upload 2020-02-23 20:26:00 +01:00
Adam Jensen 9c0e0c8f11
s3store: Add optional key prefix for metadata objects (#347)
* Add MetadataObjectPrefix field to S3Store

* Add metadataKeyWithPrefix helper function

* Use metadataKeyWithPrefix for .info and .part operations

* Add s3store tests for metadata object prefixes

* Clarify ObjectPrefix docs
2020-02-01 17:33:02 +01:00
Marius b73d634b47 core: Pass Upload interfaces to ConcatUploads 2019-09-19 12:14:25 +02:00
Marius 5a31f3230e s3store: Cache FileInfo between function calls
This lead to an 15% increase in upload speed for our tests.
2019-09-16 12:13:39 +02:00
Marius cbfde4e4b0 s3store: Pass context to AWS SDK 2019-09-15 14:33:02 +02:00
Marius 485c21d72e core: Add context to DataStores
Closes https://github.com/tus/tusd/issues/288
2019-09-15 13:43:59 +02:00
Marius d36b0ea3a9 s3store: Move to new DataStore interfaces 2019-09-10 10:05:45 +02:00
Marius a323c91a6e s3store: Add storage details 2019-08-19 10:38:08 +02:00
Marius e8fb3a431b Merge branch 'master' of github.com:tus/tusd into v1 2019-08-18 14:34:37 +02:00
Marius 42bfe35457 core: Move packages into pkg/ and internal/ folders 2019-06-11 18:23:20 +02:00