Commit Graph

703 Commits

Author SHA1 Message Date
Acconut fa58f91bb4 Store data on S3 even for io.ErrUnexpectedEOF
Previously, some data would be discarded if the user pauseed the upload.
Pausing causes the connection to be interrupted which makes Go's
net/http return an io.ErrUnexpectedEOF.
Before https://github.com/tus/tusd/pull/219 from @acj this would
not be noticed since data is discarded anyway. However, after this
PR, every byte can now be saved on S3 even if we don't have enough
data for a multipart upload part.
2019-01-28 21:41:50 +01:00
kiloreux 5919636d6f Switch master.tus.io storage to AWS S3 (#232)
* Added reference to s3 secrets

* Change s3

* Adjust indentation
2019-01-17 12:06:05 +01:00
Acconut 3b745ad85d Expose number of failed hook executions 2019-01-17 09:52:25 +01:00
Marius 7eb95254a5
Cleanup Travis CI instructions (#231) 2019-01-14 16:19:11 +01:00
Marius 5bc494a989
Install new Docker version on CI (#230) 2019-01-14 13:10:18 +01:00
Acconut 05f79b33ac Only test the prometheuscollector on supported versions 2019-01-14 00:22:12 +01:00
Adam Jensen 204044e4b9 Handle "NotFound" error code from HeadObject (#228)
* Handle "NotFound" error code from HeadObject

This accommodates third party implementations of the S3 interface, such as Minio, that may return a different error string.

* Check NotFound error string in test

This also fixes an incorrect return value.
2019-01-14 00:16:40 +01:00
Acconut daa2250530 Add CA certificates to final Docker image
Fixes https://github.com/tus/tusd/issues/227
2019-01-14 00:08:03 +01:00
Adam Jensen 33d12533e7 S3Store: Add support for creation-defer-length extension (#219)
* Add HeadObject function to S3API

* Regenerate S3API mock

* Include incomplete part size in the offset

* Add CRUD functions for managing incomplete parts

* Account for incomplete parts in S3Store's Terminate

* Account for incomplete parts in S3Store's WriteChunk

* Factor out writeInfo function

* Declare support for deferred length in S3Store

* Add test for S3Store's DeclareLength

* Adapt S3Store tests to new implementation

* Add PutObjectInputMatcher test helper

* Add test for prepending incomplete parts

* Add GetInfo test for incomplete parts

* Update S3Store docs

* Consistently handle NoSuchKey errors from S3

* Handle both 403 and 404 responses from HeadObject

If the IAM role doesn't have permission to list the contents of the bucket, then HEAD requests will return 403 for nonexistent objects.
2019-01-05 09:02:42 +01:00
Rio Kierkels 36526ef03d Switch to a multistage build for the container image (#225)
This makes it easier to reason about tusd's run environment. It also
significantly lowers the docker image size from 280MB for the current 0.11.0
image to only 26MB. Also not having the entire build environment in the
shipped images makes it quite attractive to use this image as a base
for other images. Those could contain some custom hooks for example.
2019-01-04 22:40:22 +01:00
Marius f46d660073
Merge pull request #220 from lucab85/20181121
Fix typos
2018-11-23 11:38:38 +00:00
Luca Berton 0b2312020f
Fix typos 2018-11-21 21:43:55 +01:00
Marius d81a8a088d
Add information about metrics endpoint to README 2018-11-19 17:28:11 +00:00
Adam Jensen 27c9c4aab5 Support prefix on S3 object names (#213)
* Add ObjectPrefix field to S3Store

* Integrate S3ObjectPrefix with Flags

* Account for S3ObjectPrefix flag in CreateComposer

* Account for ObjectPrefix in S3Store operations

* Add test for configuring an S3Store with ObjectPrefix
2018-11-13 00:14:37 +00:00
Anders Chen a4a6ef5a1c etcd3locker: Update formatting of docs; timeout -> ttl substitution (#216)
* Update formatting of docs; timeout -> ttl substitution

* Add link to etcd3 locker in main README.md
2018-11-10 23:33:06 +00:00
Tom Hofman 3549a75e8c Added Upload-Concat to Access-Control-Allow-Headers (#215)
* Added Upload-Concat to Access-Control-Allow-Headers for enabling browsers to make use of the concatenation extension

* Unit test update
2018-11-10 20:10:38 +00:00
Anders Chen 9af87d50ce Add etcd3 locker (#202)
* First pass at etcd3 locker

* Add etcd packages to vendor ignore attribute

* Properly exclude etcd3locker from earlier versions of golang

* Etcd test to only run on go1.9 and go1.10

* Do not export internal etcd3locker methods

* context.TODO -> context.Background

* Return errors instead of logging

* Enforce a 1.5s timeout on acquiring a lease from etcd

* Allow etcd3 concurreny.NewSession manage KeepAlive connection

- Introduce LockerOptions to allow a custom session TTL (default: 60s); etcd3Locker can be initialized with NewWithLockerOptions if one wants granular control over TTL and prefix used for etcd3 keys
- Keep NewWithPrefix backwards compatible by calling NewWithLockerOptions

* Add comment on locker_test.go regarding testing the session KeepAlive

* Re-export main type

* Try to address missing github.com/gogo/protobuf/gogoproto package

* Update etcd package import to use go.etcd.io/etcd/clientv3

* Use forked go-etcd-harness for testing

* Add more extensive package overview / docs

* go fmt

* Fix test script

* Add downloaded etcd binary to path
2018-11-10 20:05:44 +00:00
Acconut 1b756f0239 Use OS-specific path separator for hooks
Fixes https://github.com/tus/tusd/issues/206
2018-09-25 13:49:06 +01:00
Marius 6a474ce89e
Add FAQ entry about using tusd in a VM 2018-08-21 15:55:11 +02:00
Marius cbaba850a7
Merge pull request #199 from vimeo/gcs-getinfo-parallel
Add support for concurrent GetObjectSize requests
2018-07-30 22:39:20 +02:00
Tom Berger 9406b5e516 Update tests to pass context and expect GetObjectSize in any order 2018-07-25 12:51:38 -04:00
Tom Berger 82e140d317 Pass GCS object params directly to goroutine 2018-07-25 12:51:07 -04:00
Tom Berger a1273df6cd Pass contexts to GCSService. Add support for concurrent GetObjectSize requests in GetInfo 2018-07-25 10:25:59 -04:00
Marius 93d83ebdd8
Merge pull request #198 from ruv/patch-1
Add documentation about HTTP Hooks status code
2018-07-16 17:51:03 +02:00
ruv de50d26620
Add documentation about HTTP Hooks status code
HTTP Status Code meaning was missed.
Source code reference: 311cdf253e/cmd/tusd/cli/hooks.go (L137)
2018-07-16 06:11:14 +03:00
Marius 311cdf253e
Fix environment checks for tests 2018-07-14 22:12:21 +02:00
Marius 07852c5b1b
Skip consul tests on Go 1.9 2018-07-14 22:08:16 +02:00
Abdelhadi Khiati 284ee974bc Remove cleaning job 2018-07-11 00:25:14 +01:00
Abdelhadi Khiati c5f39a3205 Keep it stateless 2018-07-10 23:46:50 +01:00
Abdelhadi Khiati e527a9b0aa Run 2 instances of tusd 2018-07-10 23:37:46 +01:00
Abdelhadi Khiati d8cbc9ce6b Update tusd to use gcs 2018-07-10 23:21:57 +01:00
Abdelhadi Khiati 4c5a99ffed Upgrade to cert-manager 2018-07-07 18:07:57 +01:00
Abdelhadi Khiati fd2687779f Support scaling based on memory usage also @kvz 2018-07-07 18:07:57 +01:00
Marius f17a39352c
Merge pull request #195 from egalpin/update-upload-length-logic
Updates uploadLength logic in PatchFile
2018-06-22 19:50:50 +02:00
egalpin c42ba74f10 Updates uploadLength logic in PatchFile 2018-06-22 11:35:57 -04:00
Acconut 4a26d90584 A note about shared config file to README 2018-06-21 22:36:30 +02:00
Mark Armendariz d416990829 Allow AWS-SDK to handle gathering credentials.
This change enables ec2 instance assumed roles, environment variables, and shared configs as is standard for the aws-sdk.
2018-06-14 13:58:57 -05:00
Marius 23177d4a49
Merge pull request #191 from acj/acj/upload-defer-length-followup
Add CORS support for Upload-Defer-Length header
2018-06-14 18:58:01 +02:00
Adam Jensen 7f11b92fab Refactor Upload-Length handling to use early returns 2018-06-13 20:21:49 -04:00
Adam Jensen 7fd0139355 Add Upload-Defer-Length to access-control-{allow,expose}-headers 2018-06-11 17:58:08 -04:00
Abdelhadi Khiati ca36347384 Change cluster name 2018-06-11 21:31:00 +01:00
Abdelhadi Khiati 2a20201dba Fixed deletion script @Acconut 2018-06-11 21:25:10 +01:00
Abdelhadi Khiati 3e054562ea Added job for file deletion everyday @kvz 2018-06-11 01:18:11 +01:00
Marius d9d0f7c4e7
Merge pull request #182 from acj/acj/add-upload-defer-length-support
Implement Upload-Defer-Length Extension
2018-06-04 20:24:15 +02:00
Adam Jensen c605926ff8 If a partial upload has a deferred size, then it's not finished 2018-06-03 16:03:28 -04:00
Adam Jensen 750d4d5eb6 Specify MaxSize in PATCH tests that refer to it 2018-06-03 16:03:28 -04:00
Adam Jensen a74c46816d Remove creation-defer-length declaration from tests that don't use it 2018-06-03 16:03:28 -04:00
Adam Jensen 8c3ceebcfc If size is deferred, then constrain how much of the request body to read 2018-06-03 16:03:27 -04:00
Adam Jensen a366b00e3e Return appropriate errors if upload-length is misused in PATCH requests 2018-06-03 12:57:40 -04:00
Adam Jensen 22fdd3935b Return upload length error if it's too small or too large 2018-06-03 12:57:40 -04:00