Commit Graph

730 Commits

Author SHA1 Message Date
Marius fd2926cf60 chore: Deploy tusd on Go 1.12 2019-05-20 22:53:39 +02:00
Marius 12914032e5 s3store: Exclude aws-sdk-go examples from build
These examples are not compatible with old Go versions.
See https://travis-ci.org/tus/tusd/jobs/535001180#L508
2019-05-20 22:52:15 +02:00
Marius 774cd3800d etcd3locker: Do not run tests on Go < 1.11 2019-05-20 22:38:53 +02:00
Marius 0dd8efd606 filestore: Do not error out on unexpected EOF 2019-05-16 00:03:14 +02:00
Marius 6de723ec8b core: Do not emit post-receive hook when the offset didn't change 2019-05-15 23:57:32 +02:00
Rémy Sanchez 966711019b Create CLI option to bind to UNIX socket (#265) 2019-05-12 22:12:28 +01:00
kiloreux 9d693c93a3 Require it to not run on preemptible 2019-05-08 04:06:26 +01:00
kiloreux cce12757d4 Remove request for 2GB of memory from machine and the requirements to run on a preemptible machine @kvz @Acconut 2019-05-08 04:02:32 +01:00
Marius 12f102abf6
docs: add note about hook extensions on Windows
See https://github.com/tus/tusd/issues/267
2019-04-24 22:48:56 +02:00
Marius acc99bfecf
Only run a single tusd instance at master.tus.io 2019-04-11 21:25:00 +02:00
Marius c02acb8fdf
Decrease read timeout to 6sec for master.tus.io 2019-04-11 19:49:19 +02:00
Marius 2c19a92316
ci: Update helm installation script
In the hope that it will fix the `curl: (3) Illegal characters found in URL` error inside the deployment step:
https://travis-ci.org/tus/tusd/jobs/518477872#L7357
2019-04-10 22:53:35 +02:00
Marius fbbd768a0d consullocker: Update tests to use new import path 2019-04-10 22:28:10 +02:00
Anders Chen b6dc60d2dc docs: Add tusd-dynamo-locker link to README.md (#263) 2019-04-10 21:31:55 +02:00
Marius 589189ba67
docs: Stress the trailing slash in README 2019-04-09 21:47:28 +02:00
Marius 25f64ee00a
Extend FAQ entry about lockfile errors 2019-04-01 22:26:45 +02:00
Rija Ménagé 82bd6c796a docker: Enable the sample hook files to execute without error (#256)
- Include jq package in the Dockerfile for parsing JSON
- Use sh instead of bash which is not available in Alpine Linux
2019-03-30 18:16:53 +01:00
Marius 56acce823d
misc: Add templates for GitHub issues 2019-03-24 19:28:09 +01:00
Kirill Shvakov 495068b5a4 core: Remove unnecessary goroutines (#253)
* deleted unnecessary goroutines

* remove unnecessary goroutines

* restore NotifyTerminatedUploads

* fix merge

* restore NotifyTerminatedUploads

fix merge

revert upload progress event
2019-03-21 20:04:01 +01:00
Acconut b1a657049e limitedstore: Remove unused package fmt 2019-03-15 22:01:43 +01:00
Acconut 0116e4e15b limitedstore: Claim back space from already deleted uploads
Fixes https://github.com/tus/tusd/issues/249
2019-03-15 21:57:47 +01:00
Marius 1612a8ca46
Add FAQ entry about disabling downloads 2019-03-15 21:23:46 +01:00
Acconut c0651e77fc gcsstore: Do not ignore errors when writing an object
Previously, any error returned from w.Close() would be ignored. This is
a problem since this function is responsible for returning any error
that occurred during the actual GCS transaction.

This fix also uncovered an issue in the corresponding test routine.

Furthermore, a "404 Not Found" error when writing an object is now
interpreted as if the bucket does not exist to ease debugging
(see https://github.com/tus/tusd/issues/241).

/cc @tab1293
2019-03-10 17:01:29 +01:00
Adam Jensen 5e06fc54b0 s3store: Use DeleteObject and GetObject to fix IAM issues for incomplete parts (#233)
* Use GetObject instead of HeadObject to locate incomplete part

This avoids confusion around the errors that are returned by HeadObject, especially when the request has limited permissions for the bucket.

* Remove unused HeadObject function

* Add DeleteObject to S3API interface

* Use DeleteObject to remove .part objects

* Update tests
2019-02-23 21:24:38 +01:00
oliverpool 0baa9eaf48 Forward status code and body from pre-create HTTP hook (#236)
* Forward status code from Pre-create http hook

Fix #170

* Allow the returned body to difer from the logged error

* Update HTTP Hooks documentation
2019-02-12 22:45:08 +01:00
Jonas Thelemann df77e34fed Correct "around" Spelling (#242) 2019-02-12 22:11:11 +01:00
Anders Chen 057bb9e2c7 Docs: consullocker/etcd3locker - correct UnlockUpload method docs (#237) 2019-02-04 22:33:17 +01:00
Acconut fa58f91bb4 Store data on S3 even for io.ErrUnexpectedEOF
Previously, some data would be discarded if the user pauseed the upload.
Pausing causes the connection to be interrupted which makes Go's
net/http return an io.ErrUnexpectedEOF.
Before https://github.com/tus/tusd/pull/219 from @acj this would
not be noticed since data is discarded anyway. However, after this
PR, every byte can now be saved on S3 even if we don't have enough
data for a multipart upload part.
2019-01-28 21:41:50 +01:00
kiloreux 5919636d6f Switch master.tus.io storage to AWS S3 (#232)
* Added reference to s3 secrets

* Change s3

* Adjust indentation
2019-01-17 12:06:05 +01:00
Acconut 3b745ad85d Expose number of failed hook executions 2019-01-17 09:52:25 +01:00
Marius 7eb95254a5
Cleanup Travis CI instructions (#231) 2019-01-14 16:19:11 +01:00
Marius 5bc494a989
Install new Docker version on CI (#230) 2019-01-14 13:10:18 +01:00
Acconut 05f79b33ac Only test the prometheuscollector on supported versions 2019-01-14 00:22:12 +01:00
Adam Jensen 204044e4b9 Handle "NotFound" error code from HeadObject (#228)
* Handle "NotFound" error code from HeadObject

This accommodates third party implementations of the S3 interface, such as Minio, that may return a different error string.

* Check NotFound error string in test

This also fixes an incorrect return value.
2019-01-14 00:16:40 +01:00
Acconut daa2250530 Add CA certificates to final Docker image
Fixes https://github.com/tus/tusd/issues/227
2019-01-14 00:08:03 +01:00
Adam Jensen 33d12533e7 S3Store: Add support for creation-defer-length extension (#219)
* Add HeadObject function to S3API

* Regenerate S3API mock

* Include incomplete part size in the offset

* Add CRUD functions for managing incomplete parts

* Account for incomplete parts in S3Store's Terminate

* Account for incomplete parts in S3Store's WriteChunk

* Factor out writeInfo function

* Declare support for deferred length in S3Store

* Add test for S3Store's DeclareLength

* Adapt S3Store tests to new implementation

* Add PutObjectInputMatcher test helper

* Add test for prepending incomplete parts

* Add GetInfo test for incomplete parts

* Update S3Store docs

* Consistently handle NoSuchKey errors from S3

* Handle both 403 and 404 responses from HeadObject

If the IAM role doesn't have permission to list the contents of the bucket, then HEAD requests will return 403 for nonexistent objects.
2019-01-05 09:02:42 +01:00
Rio Kierkels 36526ef03d Switch to a multistage build for the container image (#225)
This makes it easier to reason about tusd's run environment. It also
significantly lowers the docker image size from 280MB for the current 0.11.0
image to only 26MB. Also not having the entire build environment in the
shipped images makes it quite attractive to use this image as a base
for other images. Those could contain some custom hooks for example.
2019-01-04 22:40:22 +01:00
Marius f46d660073
Merge pull request #220 from lucab85/20181121
Fix typos
2018-11-23 11:38:38 +00:00
Luca Berton 0b2312020f
Fix typos 2018-11-21 21:43:55 +01:00
Marius d81a8a088d
Add information about metrics endpoint to README 2018-11-19 17:28:11 +00:00
Adam Jensen 27c9c4aab5 Support prefix on S3 object names (#213)
* Add ObjectPrefix field to S3Store

* Integrate S3ObjectPrefix with Flags

* Account for S3ObjectPrefix flag in CreateComposer

* Account for ObjectPrefix in S3Store operations

* Add test for configuring an S3Store with ObjectPrefix
2018-11-13 00:14:37 +00:00
Anders Chen a4a6ef5a1c etcd3locker: Update formatting of docs; timeout -> ttl substitution (#216)
* Update formatting of docs; timeout -> ttl substitution

* Add link to etcd3 locker in main README.md
2018-11-10 23:33:06 +00:00
Tom Hofman 3549a75e8c Added Upload-Concat to Access-Control-Allow-Headers (#215)
* Added Upload-Concat to Access-Control-Allow-Headers for enabling browsers to make use of the concatenation extension

* Unit test update
2018-11-10 20:10:38 +00:00
Anders Chen 9af87d50ce Add etcd3 locker (#202)
* First pass at etcd3 locker

* Add etcd packages to vendor ignore attribute

* Properly exclude etcd3locker from earlier versions of golang

* Etcd test to only run on go1.9 and go1.10

* Do not export internal etcd3locker methods

* context.TODO -> context.Background

* Return errors instead of logging

* Enforce a 1.5s timeout on acquiring a lease from etcd

* Allow etcd3 concurreny.NewSession manage KeepAlive connection

- Introduce LockerOptions to allow a custom session TTL (default: 60s); etcd3Locker can be initialized with NewWithLockerOptions if one wants granular control over TTL and prefix used for etcd3 keys
- Keep NewWithPrefix backwards compatible by calling NewWithLockerOptions

* Add comment on locker_test.go regarding testing the session KeepAlive

* Re-export main type

* Try to address missing github.com/gogo/protobuf/gogoproto package

* Update etcd package import to use go.etcd.io/etcd/clientv3

* Use forked go-etcd-harness for testing

* Add more extensive package overview / docs

* go fmt

* Fix test script

* Add downloaded etcd binary to path
2018-11-10 20:05:44 +00:00
Acconut 1b756f0239 Use OS-specific path separator for hooks
Fixes https://github.com/tus/tusd/issues/206
2018-09-25 13:49:06 +01:00
Marius 6a474ce89e
Add FAQ entry about using tusd in a VM 2018-08-21 15:55:11 +02:00
Marius cbaba850a7
Merge pull request #199 from vimeo/gcs-getinfo-parallel
Add support for concurrent GetObjectSize requests
2018-07-30 22:39:20 +02:00
Tom Berger 9406b5e516 Update tests to pass context and expect GetObjectSize in any order 2018-07-25 12:51:38 -04:00
Tom Berger 82e140d317 Pass GCS object params directly to goroutine 2018-07-25 12:51:07 -04:00
Tom Berger a1273df6cd Pass contexts to GCSService. Add support for concurrent GetObjectSize requests in GetInfo 2018-07-25 10:25:59 -04:00