Previously, any error returned from w.Close() would be ignored. This is
a problem since this function is responsible for returning any error
that occurred during the actual GCS transaction.
This fix also uncovered an issue in the corresponding test routine.
Furthermore, a "404 Not Found" error when writing an object is now
interpreted as if the bucket does not exist to ease debugging
(see https://github.com/tus/tusd/issues/241).
/cc @tab1293
* Use GetObject instead of HeadObject to locate incomplete part
This avoids confusion around the errors that are returned by HeadObject, especially when the request has limited permissions for the bucket.
* Remove unused HeadObject function
* Add DeleteObject to S3API interface
* Use DeleteObject to remove .part objects
* Update tests
Previously, some data would be discarded if the user pauseed the upload.
Pausing causes the connection to be interrupted which makes Go's
net/http return an io.ErrUnexpectedEOF.
Before https://github.com/tus/tusd/pull/219 from @acj this would
not be noticed since data is discarded anyway. However, after this
PR, every byte can now be saved on S3 even if we don't have enough
data for a multipart upload part.
* Handle "NotFound" error code from HeadObject
This accommodates third party implementations of the S3 interface, such as Minio, that may return a different error string.
* Check NotFound error string in test
This also fixes an incorrect return value.
* Add HeadObject function to S3API
* Regenerate S3API mock
* Include incomplete part size in the offset
* Add CRUD functions for managing incomplete parts
* Account for incomplete parts in S3Store's Terminate
* Account for incomplete parts in S3Store's WriteChunk
* Factor out writeInfo function
* Declare support for deferred length in S3Store
* Add test for S3Store's DeclareLength
* Adapt S3Store tests to new implementation
* Add PutObjectInputMatcher test helper
* Add test for prepending incomplete parts
* Add GetInfo test for incomplete parts
* Update S3Store docs
* Consistently handle NoSuchKey errors from S3
* Handle both 403 and 404 responses from HeadObject
If the IAM role doesn't have permission to list the contents of the bucket, then HEAD requests will return 403 for nonexistent objects.
This makes it easier to reason about tusd's run environment. It also
significantly lowers the docker image size from 280MB for the current 0.11.0
image to only 26MB. Also not having the entire build environment in the
shipped images makes it quite attractive to use this image as a base
for other images. Those could contain some custom hooks for example.
* Add ObjectPrefix field to S3Store
* Integrate S3ObjectPrefix with Flags
* Account for S3ObjectPrefix flag in CreateComposer
* Account for ObjectPrefix in S3Store operations
* Add test for configuring an S3Store with ObjectPrefix
ridhozhr
Marius
Rémy Sanchez
kiloreux
Marius
Anders Chen
Rija Ménagé
Kirill Shvakov
Adam Jensen
oliverpool
Jonas Thelemann
Anders Chen
kiloreux
Rio Kierkels
Luca Berton