Commit Graph

4 Commits

Author SHA1 Message Date
Stefan Scheidewig 9d7096fcb3 Return ReadCloser in getReader 2022-06-03 22:39:27 +02:00
Marius 8fd18364e7
s3store: Parallelize part uploads and information retrieval (#478)
* Add first draft of parallel upload queue

* s3store: Use queue for parallel uploads

* Revert "Add first draft of parallel upload queue"

This reverts commit 86a329cef2.

* Revert "s3store: Use queue for parallel uploads"

This reverts commit 29b59a2c90.

* s3store: Cache results from listing parts and checking incomplete object

* s3store: Remove debugging output`

* s3store: Make requests for fetching info concurrently

* s3store: Make parallel uploads work and tests pass

* s3store: Add semaphore package

* s3store: Add comments to semaphore package

* s3store: Encapsulate more logic into s3PartProducer

* s3store: Refactor WriteChunk

* s3store: Remove TODO

* s3store: Acquire lock before uploading

* cli: Add flag for setting concurrency limit

* s3store: One more comment
2021-05-18 10:29:18 +02:00
Marius cbfde4e4b0 s3store: Pass context to AWS SDK 2019-09-15 14:33:02 +02:00
Marius 42bfe35457 core: Move packages into pkg/ and internal/ folders 2019-06-11 18:23:20 +02:00