Commit Graph

4 Commits

Author SHA1 Message Date
Marius 1038298a79 core: Change GetReader to return io.ReadCloser (#739)
Squashed commit of the following:

commit e8b5b3751a86d86cae10e0bcf89caa481e5c3de6
Author: Marius <marius@transloadit.com>
Date:   Sun Jun 19 12:15:22 2022 +0200

    Fix generated mocks

commit 736e2e7bb6
Merge: 9d7096f 1e69d9b
Author: Stefan Scheidewig <stefan.scheidewig@staffbase.com>
Date:   Sat Jun 18 07:53:29 2022 +0200

    Merge branch 'v2' into readcloser_in_getreader

commit 9d7096fcb3
Author: Stefan Scheidewig <stefan.scheidewig@staffbase.com>
Date:   Tue May 24 14:16:01 2022 +0200

    Return ReadCloser in getReader
2022-06-19 12:18:02 +02:00
Marius 8fd18364e7
s3store: Parallelize part uploads and information retrieval (#478)
* Add first draft of parallel upload queue

* s3store: Use queue for parallel uploads

* Revert "Add first draft of parallel upload queue"

This reverts commit 86a329cef2.

* Revert "s3store: Use queue for parallel uploads"

This reverts commit 29b59a2c90.

* s3store: Cache results from listing parts and checking incomplete object

* s3store: Remove debugging output`

* s3store: Make requests for fetching info concurrently

* s3store: Make parallel uploads work and tests pass

* s3store: Add semaphore package

* s3store: Add comments to semaphore package

* s3store: Encapsulate more logic into s3PartProducer

* s3store: Refactor WriteChunk

* s3store: Remove TODO

* s3store: Acquire lock before uploading

* cli: Add flag for setting concurrency limit

* s3store: One more comment
2021-05-18 10:29:18 +02:00
Marius cbfde4e4b0 s3store: Pass context to AWS SDK 2019-09-15 14:33:02 +02:00
Marius 42bfe35457 core: Move packages into pkg/ and internal/ folders 2019-06-11 18:23:20 +02:00