Commit Graph

38 Commits

Author SHA1 Message Date
Marius 870c434485 s3store: Add metrics for demand on upload semaphore 2022-08-11 23:31:33 +02:00
Marius cab456900a Merge branch 'master' of github.com:tus/tusd into v2 2022-07-25 13:02:05 +02:00
Roberto Villalba e423e30135
s3storeL Add prefix to upload concat source (#767) 2022-07-16 14:52:53 +02:00
Marius efe8c9ce05 s3store: Add option for switching to Minio SDK 2022-07-11 13:43:09 +02:00
Marius 1038298a79 core: Change GetReader to return io.ReadCloser (#739)
Squashed commit of the following:

commit e8b5b3751a86d86cae10e0bcf89caa481e5c3de6
Author: Marius <marius@transloadit.com>
Date:   Sun Jun 19 12:15:22 2022 +0200

    Fix generated mocks

commit 736e2e7bb6
Merge: 9d7096f 1e69d9b
Author: Stefan Scheidewig <stefan.scheidewig@staffbase.com>
Date:   Sat Jun 18 07:53:29 2022 +0200

    Merge branch 'v2' into readcloser_in_getreader

commit 9d7096fcb3
Author: Stefan Scheidewig <stefan.scheidewig@staffbase.com>
Date:   Tue May 24 14:16:01 2022 +0200

    Return ReadCloser in getReader
2022-06-19 12:18:02 +02:00
Marius c0f2026e96 s3store: Implement temporary support for buffering in memory 2022-04-10 21:13:07 +02:00
Marius 12c10bf62f
v2: Rework hooks system (#516)
* ci: Remove plugin hook handler

* Rework error type from interface to struct

* Avoid writing to http.ResponseWriter directly

* Allow hooks to modify response

* Add example for HTTP hooks using Python

* Implement new plugin system using Hashicorp/go-plugin

* Enable returning partial HTTPResponses

* Remove some (unnecessary) error handling

* Forward stdout and stderr from plugin to tusd

* docs: Update examples

* cli: Update filehooks to new system

* cli: Renovate gRPC hooks

* docs: Correct casing of gRPC

* misc: Documentation, better examples, and code structure
2022-03-02 00:36:49 +01:00
Marius a05c090d05 Merge branch 'master' into v2 2022-02-23 16:03:41 +01:00
Anatoly 6828cbdce1
s3store: stricter character replacement regexp for x-amz-metadata (#598)
* stricter character replacement rule for x-amz-metadata

* misspell fixed
2021-12-30 17:17:45 +01:00
Marius b2273d4153 core: Add error constants
/cc @kvz
2021-10-25 11:54:02 +02:00
Marius 387b04a2e2 Merge branch 'master' of github.com:tus/tusd into v2 2021-10-15 22:05:01 +02:00
benito 16a3747ec4
s3store: Accept alternate error response from DigitalOcean Spaces (#507)
* fix digitalocean spaces fetch

* remove log

* remove log

* remove info.Size check

* Add comment

* Delete .gitignore

Co-authored-by: Marius <marius.kleidl@gmail.com>
2021-10-15 21:50:17 +02:00
Marius 675e767ee6 Merge branch 'master' into v2 2021-07-01 18:49:09 +02:00
Marius 781324986d s3store: Return 400 instead of 500 if upload cannot be streamed
Closes https://github.com/tus/tusd/issues/490
2021-07-01 18:48:58 +02:00
Marius ccdfe8e604 cli: Add flag to customize the S3 part buffer size 2021-06-21 13:17:36 +02:00
Marius 946539c3b9 cli: Add option to expose Go's pprof 2021-05-28 13:26:13 +02:00
Marius f4314dd360 s3store: Do not register metrics to default registry 2021-05-24 23:45:54 +02:00
Marius 0f24a80ea5 s3store: Expose metrics about request durations 2021-05-24 12:00:20 +02:00
Marius ce54ff8b1f fixup! s3store: Parallelize part uploads and information retrieval (#478) 2021-05-23 12:54:53 +02:00
Marius 8fd18364e7
s3store: Parallelize part uploads and information retrieval (#478)
* Add first draft of parallel upload queue

* s3store: Use queue for parallel uploads

* Revert "Add first draft of parallel upload queue"

This reverts commit 86a329cef2.

* Revert "s3store: Use queue for parallel uploads"

This reverts commit 29b59a2c90.

* s3store: Cache results from listing parts and checking incomplete object

* s3store: Remove debugging output`

* s3store: Make requests for fetching info concurrently

* s3store: Make parallel uploads work and tests pass

* s3store: Add semaphore package

* s3store: Add comments to semaphore package

* s3store: Encapsulate more logic into s3PartProducer

* s3store: Refactor WriteChunk

* s3store: Remove TODO

* s3store: Acquire lock before uploading

* cli: Add flag for setting concurrency limit

* s3store: One more comment
2021-05-18 10:29:18 +02:00
Marius 8c5192c254 s3store: Move parts producer into own file 2021-04-26 12:18:09 +02:00
Marius 6d987aa226 core: Handle errors from reading request body centrally 2021-04-26 10:08:37 +02:00
Adam Jensen e85d630748
s3store: Fix data loss when using deferred upload lengths (#462) 2021-03-11 18:17:47 +01:00
Marius fdb19a7904 s3store: Temporary workaround against data loss on connection resets 2021-02-10 23:07:16 +01:00
Marius 67fd74e129 Add flag to disable calculating expensive hashes for S3 2021-01-18 13:31:07 +01:00
Marius 9831c8b4da
s3store: Increase default part size (#410)
* s3store: Increase default part size

* Update tests and add flag

* Fix minor issues
2020-08-01 14:58:31 +02:00
Adam Jensen 6662f43d01
s3Store: Concurrently write upload parts to S3 while reading from client (#402)
* Allow empty metadata values

* Make tests less fragile by allowing loose call ordering

* Add s3ChunkProducer

* Integrate s3ChunkProducer to support chunk buffering

* Remove completed chunk files inline to reduce disk space usage

* Add tests for chunk producer

* docs: Use value from Host header to forward to tusd

* Use int64 for MaxBufferedParts field

* Default to 20 buffered parts

* Rename s3ChunkProducer -> s3PartProducer

* Document s3PartProducer struct

* Clarify misleading comment

* Revert "Remove completed chunk files inline to reduce disk space usage"

This reverts commit b72a4d43d6.

* Remove redundant seek

This is already being done in s3PartProducer.

* Clean up any remaining files in the channel when we return

* Make putPart* functions responsible for cleaning up temp files

* handler: Add tests for empty metadata pairs

* Factor out cleanUpTempFile func

* Add test to ensure that temporary files get cleaned up

Co-authored-by: Jens Steinhauser <jens.steinhauser@gmail.com>
Co-authored-by: Marius <marius@transloadit.com>
2020-07-29 15:24:46 +02:00
Marius 9210fbe0fc s3store: Allow concatenation of uploads with less than 5MB 2020-04-27 12:42:56 +02:00
Marius 973a4fe066 s3store: Fix bug when completing empty upload 2020-02-23 20:26:00 +01:00
Adam Jensen 9c0e0c8f11
s3store: Add optional key prefix for metadata objects (#347)
* Add MetadataObjectPrefix field to S3Store

* Add metadataKeyWithPrefix helper function

* Use metadataKeyWithPrefix for .info and .part operations

* Add s3store tests for metadata object prefixes

* Clarify ObjectPrefix docs
2020-02-01 17:33:02 +01:00
Marius b73d634b47 core: Pass Upload interfaces to ConcatUploads 2019-09-19 12:14:25 +02:00
Marius 5a31f3230e s3store: Cache FileInfo between function calls
This lead to an 15% increase in upload speed for our tests.
2019-09-16 12:13:39 +02:00
Marius cbfde4e4b0 s3store: Pass context to AWS SDK 2019-09-15 14:33:02 +02:00
Marius 485c21d72e core: Add context to DataStores
Closes https://github.com/tus/tusd/issues/288
2019-09-15 13:43:59 +02:00
Marius d36b0ea3a9 s3store: Move to new DataStore interfaces 2019-09-10 10:05:45 +02:00
Marius a323c91a6e s3store: Add storage details 2019-08-19 10:38:08 +02:00
Marius e8fb3a431b Merge branch 'master' of github.com:tus/tusd into v1 2019-08-18 14:34:37 +02:00
Marius 42bfe35457 core: Move packages into pkg/ and internal/ folders 2019-06-11 18:23:20 +02:00