Commit Graph

1262 Commits

Author SHA1 Message Date
benito 16a3747ec4
s3store: Accept alternate error response from DigitalOcean Spaces (#507)
* fix digitalocean spaces fetch

* remove log

* remove log

* remove info.Size check

* Add comment

* Delete .gitignore

Co-authored-by: Marius <marius.kleidl@gmail.com>
2021-10-15 21:50:17 +02:00
Marius dd44267c08
ci: Upgrade Go version for tests and releases 2021-10-13 22:05:19 +02:00
Marius 5614cecb60
misc: Specify Go version for Heroku
See https://github.com/tus/tusd/issues/509
2021-10-13 21:52:30 +02:00
Marius d973f9a81a misc: Increase Go version in go.mod
The current version Go 1.12 is too old, so Heroku does not compile tusd anymore: https://github.com/tus/tusd/runs/3886723478
We use Go 1.16 and not 1.17 because we always support the two latest major releases.
2021-10-13 21:40:39 +02:00
CharlyBr 484e506cd3
misc: Ignore tusd binary (#505) 2021-10-13 21:33:49 +02:00
Marius 0ad435b4c8 core: Fix wrong offset in upload progress notifications
Closes https://github.com/tus/tusd/issues/500
2021-10-13 21:08:09 +02:00
Ole-Martin Bratteng 1b11885823
azurestore: Add implementation
* Add azure-storage-blob-go dependency

* Implement Azure BlobStorage store

* Add AzureStore Mock test

* Refactor Blob interfaces to use uppercase fields

* Refactor and remove the Create function
When getting the offset, and we get the status code BlobNotFound, we can say the offset is 0, and start from the beginning

* Update the mock

* Refactor error checking of GetOffset to actually check the service code

* Begin testing azurestore

* Write more tests

* New feature allows to set access type on new containers and blob access tier

* Write more docs

* Upgrade azure-storage-blob-go to v0.13.0

* Remove AzError, not needed

* Update link to container access type information

* Remove ?toc from link in comments

* Remove trailing spaces from workflow

* Run tests with go1.15 and 1.16

* Don't fail fast
This lets all other tests complete, and makes it easier to see if it's just a one-off fail, or on different OSes and versions

* Remove darwin 386 from `build_all.sh` script
Removed in go1.15 https://github.com/golang/go/issues/37610

* Update go version in `Dockerfile`

* Compile for Apple Silicone (darwin arm64)
Only go1.16 supports it
2021-07-29 01:14:50 +02:00
Marius 5be2afa2f8 Merge branch 'master' of github.com:tus/tusd 2021-07-01 18:49:42 +02:00
Marius 675e767ee6 Merge branch 'master' into v2 2021-07-01 18:49:09 +02:00
Marius 781324986d s3store: Return 400 instead of 500 if upload cannot be streamed
Closes https://github.com/tus/tusd/issues/490
2021-07-01 18:48:58 +02:00
Marius 934265dd38 Merge branch 'master' of github.com:tus/tusd into v2 2021-07-01 18:15:47 +02:00
Marius ccdfe8e604 cli: Add flag to customize the S3 part buffer size 2021-06-21 13:17:36 +02:00
Marius 36f12b1d18 s3store: Fix failing tests due to missing argument 2021-06-21 13:15:21 +02:00
Juanjo Rodriguez b50c2936a1
cli: Change timestamp to microseconds precision in log (#489) 2021-06-14 14:16:39 +02:00
Marius 946539c3b9 cli: Add option to expose Go's pprof 2021-05-28 13:26:13 +02:00
Marius f4314dd360 s3store: Do not register metrics to default registry 2021-05-24 23:45:54 +02:00
Marius 0f24a80ea5 s3store: Expose metrics about request durations 2021-05-24 12:00:20 +02:00
Marius ce54ff8b1f fixup! s3store: Parallelize part uploads and information retrieval (#478) 2021-05-23 12:54:53 +02:00
Marius 578731ab0b cli: Add metric for number of hook invocations 2021-05-20 21:38:19 +02:00
Marius 8fd18364e7
s3store: Parallelize part uploads and information retrieval (#478)
* Add first draft of parallel upload queue

* s3store: Use queue for parallel uploads

* Revert "Add first draft of parallel upload queue"

This reverts commit 86a329cef2.

* Revert "s3store: Use queue for parallel uploads"

This reverts commit 29b59a2c90.

* s3store: Cache results from listing parts and checking incomplete object

* s3store: Remove debugging output`

* s3store: Make requests for fetching info concurrently

* s3store: Make parallel uploads work and tests pass

* s3store: Add semaphore package

* s3store: Add comments to semaphore package

* s3store: Encapsulate more logic into s3PartProducer

* s3store: Refactor WriteChunk

* s3store: Remove TODO

* s3store: Acquire lock before uploading

* cli: Add flag for setting concurrency limit

* s3store: One more comment
2021-05-18 10:29:18 +02:00
Marius d560c4e753
ci: Update Docker image to Go 1.13
- The Go version now matches the one used for the GHA CI jobs
- The Docker builds failed because a dependency changed and Go 1.12 was not able to use the version from go.sum (see https://hub.docker.com/repository/registry-1.docker.io/tusproject/tusd/builds/913de821-d650-454b-9f05-107862caedf3)
2021-05-17 00:08:36 +02:00
Márk Sági-Kazár f1b183022b
docs: Mention the helm chart in the installation instructions (#475) 2021-05-05 10:35:02 +02:00
Marius 8c5192c254 s3store: Move parts producer into own file 2021-04-26 12:18:09 +02:00
Marius b3bf854712 s3store: Remove test for handling unexpected EOF
The store must no longer handle unexpected EOFs since this is convered by the handler.
See the previous commits for more details
2021-04-26 12:17:20 +02:00
Marius 27957bd22b Merge branch 'master' of github.com:tus/tusd 2021-04-26 10:08:58 +02:00
Marius 6d987aa226 core: Handle errors from reading request body centrally 2021-04-26 10:08:37 +02:00
Marius 97602c3d62 filestore: Handle os.ErrNotExist not in core handler 2021-04-25 23:17:42 +02:00
Márk Sági-Kazár 36ecc269ae
misc: Update alpine version in Dockerfile (#470) 2021-04-06 00:15:45 +02:00
Marius 8031aabb7e cli: Add flag to disable SSL for S3 2021-03-29 15:12:17 +02:00
Ole-Martin Bratteng 56723a5ea6
ci: Compile tusd for arm64 (#461) 2021-03-11 18:20:14 +01:00
Adam Jensen e85d630748
s3store: Fix data loss when using deferred upload lengths (#462) 2021-03-11 18:17:47 +01:00
Marius d152c5bbf8 Merge branch 'master' of github.com:tus/tusd 2021-02-10 23:07:38 +01:00
Marius fdb19a7904 s3store: Temporary workaround against data loss on connection resets 2021-02-10 23:07:16 +01:00
Abdelhadi Khiati 734c630069 Update for build step also, uses a stable version of go 2021-01-18 13:46:09 +01:00
Abdelhadi Khiati 8b81859a80 Use setup-go@v2 to avoid old commands. 2021-01-18 13:41:07 +01:00
Marius 67fd74e129 Add flag to disable calculating expensive hashes for S3 2021-01-18 13:31:07 +01:00
Ahmed J 0822c0ac43
handler: Use correct separation symbol for parsing Forwarded header (#433)
* fix forwarded header separator

* added tests

* fix tests

* updated test

* Reuse existing test for Forwarded header

Co-authored-by: Ahmed J <ahjabir@earthlink.iq>
Co-authored-by: Marius <marius@transloadit.com>
2020-10-12 12:36:55 +02:00
Ruben Garcia f863189009
docs: Mention support for HTTPS hooks (#430)
* Clarify that HTTPS is supported in hooks.

* Fix markdown.

* Fix Blocking hooks section.
2020-10-10 18:21:33 +02:00
Marius 9b934ebca7 Remove double spaces 2020-10-02 12:45:16 +02:00
Marius e5a79650cf Use new URL for public tus demo server 2020-10-02 12:44:12 +02:00
Joey Coleman f4d1b7c443
cli: Allow accepting requests over HTTPS (#423)
* Add TLS support to tusd

* Adds `-tls-certificate`, `-tls-key`, and `-tls-mode` flags
* Alter printed URL to reflect protocol in use
* For non-TLS, do an eary-exit if http.Serve() returns
* Configure TLS for the following modes
    * TLSv1.3-only
    * (default) TLSv1.3+TLSv1.2, with recommended ciphersuites from Mozilla + matching RSA key transport modes
    * TLSv1.2 only, with only 256-bit AES ciphersuites
* All modes disable HTTP/2, given that it’s not supported in non-TLS mode
* Update documentation

* * Remove RSA-based key transport ciphersuites as they don’t support forward secrecy

* Improve the TLS/HTTPS example in the usage documentation

Signed-off-by: Joey Coleman <joey.coleman@kirasystems.com>

* Update docs further to a) record that the key file must be unencrypted, and b) clean up the RSA-based ciphersuite comments.
2020-09-29 15:19:10 +02:00
Ifedapo .A. Olarewaju ed85c526cb
build: remove k8s related build (#426)
* build: remove k8s related build

* build: there's more
2020-09-11 20:47:45 +02:00
Ifedapo .A. Olarewaju 2c7735e697
Heroku deployment (#425)
* add heroku deployment

* build: remove duplicate step
2020-09-11 15:20:18 +02:00
Kevin McGee 90e208a286
cli: Add flag to enable S3 transfer acceleration (#411)
* Enable S3 transfer acceleration in SDK

* Format better

* Setup for flag changes

* Place feature behind new flag

* Fix a docs issue

Co-authored-by: Cloud User <centos@ip-10-0-0-184.us-west-2.compute.internal>
2020-08-01 15:07:03 +02:00
Márk Sági-Kazár 3a344c5e81
memorylocker: Fix function names in documentation (#412) 2020-08-01 15:02:46 +02:00
Márk Sági-Kazár ec9c56bb03
docs: Fix typo (#413) 2020-08-01 15:01:18 +02:00
Marius 9831c8b4da
s3store: Increase default part size (#410)
* s3store: Increase default part size

* Update tests and add flag

* Fix minor issues
2020-08-01 14:58:31 +02:00
Adam Jensen 6662f43d01
s3Store: Concurrently write upload parts to S3 while reading from client (#402)
* Allow empty metadata values

* Make tests less fragile by allowing loose call ordering

* Add s3ChunkProducer

* Integrate s3ChunkProducer to support chunk buffering

* Remove completed chunk files inline to reduce disk space usage

* Add tests for chunk producer

* docs: Use value from Host header to forward to tusd

* Use int64 for MaxBufferedParts field

* Default to 20 buffered parts

* Rename s3ChunkProducer -> s3PartProducer

* Document s3PartProducer struct

* Clarify misleading comment

* Revert "Remove completed chunk files inline to reduce disk space usage"

This reverts commit b72a4d43d6.

* Remove redundant seek

This is already being done in s3PartProducer.

* Clean up any remaining files in the channel when we return

* Make putPart* functions responsible for cleaning up temp files

* handler: Add tests for empty metadata pairs

* Factor out cleanUpTempFile func

* Add test to ensure that temporary files get cleaned up

Co-authored-by: Jens Steinhauser <jens.steinhauser@gmail.com>
Co-authored-by: Marius <marius@transloadit.com>
2020-07-29 15:24:46 +02:00
Marius 26b84bcb1c handler: Add tests for empty metadata pairs 2020-07-15 17:30:17 +02:00
Marius 59c3d42f8f Merge branch 'metadata' of https://github.com/JenSte/tusd 2020-07-15 17:27:37 +02:00