Commit Graph

12 Commits

Author SHA1 Message Date
Adam Jensen 6662f43d01
s3Store: Concurrently write upload parts to S3 while reading from client (#402)
* Allow empty metadata values

* Make tests less fragile by allowing loose call ordering

* Add s3ChunkProducer

* Integrate s3ChunkProducer to support chunk buffering

* Remove completed chunk files inline to reduce disk space usage

* Add tests for chunk producer

* docs: Use value from Host header to forward to tusd

* Use int64 for MaxBufferedParts field

* Default to 20 buffered parts

* Rename s3ChunkProducer -> s3PartProducer

* Document s3PartProducer struct

* Clarify misleading comment

* Revert "Remove completed chunk files inline to reduce disk space usage"

This reverts commit b72a4d43d6.

* Remove redundant seek

This is already being done in s3PartProducer.

* Clean up any remaining files in the channel when we return

* Make putPart* functions responsible for cleaning up temp files

* handler: Add tests for empty metadata pairs

* Factor out cleanUpTempFile func

* Add test to ensure that temporary files get cleaned up

Co-authored-by: Jens Steinhauser <jens.steinhauser@gmail.com>
Co-authored-by: Marius <marius@transloadit.com>
2020-07-29 15:24:46 +02:00
Marius 9210fbe0fc s3store: Allow concatenation of uploads with less than 5MB 2020-04-27 12:42:56 +02:00
Marius 973a4fe066 s3store: Fix bug when completing empty upload 2020-02-23 20:26:00 +01:00
Adam Jensen 9c0e0c8f11
s3store: Add optional key prefix for metadata objects (#347)
* Add MetadataObjectPrefix field to S3Store

* Add metadataKeyWithPrefix helper function

* Use metadataKeyWithPrefix for .info and .part operations

* Add s3store tests for metadata object prefixes

* Clarify ObjectPrefix docs
2020-02-01 17:33:02 +01:00
Marius b73d634b47 core: Pass Upload interfaces to ConcatUploads 2019-09-19 12:14:25 +02:00
Marius 5a31f3230e s3store: Cache FileInfo between function calls
This lead to an 15% increase in upload speed for our tests.
2019-09-16 12:13:39 +02:00
Marius cbfde4e4b0 s3store: Pass context to AWS SDK 2019-09-15 14:33:02 +02:00
Marius 485c21d72e core: Add context to DataStores
Closes https://github.com/tus/tusd/issues/288
2019-09-15 13:43:59 +02:00
Marius d36b0ea3a9 s3store: Move to new DataStore interfaces 2019-09-10 10:05:45 +02:00
Marius a323c91a6e s3store: Add storage details 2019-08-19 10:38:08 +02:00
Marius e8fb3a431b Merge branch 'master' of github.com:tus/tusd into v1 2019-08-18 14:34:37 +02:00
Marius 42bfe35457 core: Move packages into pkg/ and internal/ folders 2019-06-11 18:23:20 +02:00