Commit Graph

27 Commits

Author SHA1 Message Date
Adam Jensen 33d12533e7 S3Store: Add support for creation-defer-length extension (#219)
* Add HeadObject function to S3API

* Regenerate S3API mock

* Include incomplete part size in the offset

* Add CRUD functions for managing incomplete parts

* Account for incomplete parts in S3Store's Terminate

* Account for incomplete parts in S3Store's WriteChunk

* Factor out writeInfo function

* Declare support for deferred length in S3Store

* Add test for S3Store's DeclareLength

* Adapt S3Store tests to new implementation

* Add PutObjectInputMatcher test helper

* Add test for prepending incomplete parts

* Add GetInfo test for incomplete parts

* Update S3Store docs

* Consistently handle NoSuchKey errors from S3

* Handle both 403 and 404 responses from HeadObject

If the IAM role doesn't have permission to list the contents of the bucket, then HEAD requests will return 403 for nonexistent objects.
2019-01-05 09:02:42 +01:00
Adam Jensen 27c9c4aab5 Support prefix on S3 object names (#213)
* Add ObjectPrefix field to S3Store

* Integrate S3ObjectPrefix with Flags

* Account for S3ObjectPrefix flag in CreateComposer

* Account for ObjectPrefix in S3Store operations

* Add test for configuring an S3Store with ObjectPrefix
2018-11-13 00:14:37 +00:00
Adam Jensen 5252c98cd7 Avoid incorrectly returning errors when upload length is deferred 2018-06-03 12:57:39 -04:00
Adam Jensen 1ab5231643 Add basic Upload-Defer-Length header handling 2018-06-03 12:57:39 -04:00
Adam Jensen 450a606334 Fix a couple of typos 2018-04-24 11:53:36 -04:00
Adam Jensen e276384a48
Remove misleading comment
The comment mentions 6, but the relevant sizes are 5MB, 5GB, and 5TB. I think it's preferable to let the code speak for itself so that the comment doesn't become stale.
2018-04-24 11:47:29 -04:00
Markus Kienast 2df5d11672 s3store: calculate part size based on upload size
See https://github.com/tus/tusd/issues/149 and
https://github.com/tus/tusd/pull/150 for more details.

Squashed commit of the following:

commit 78312ab26ea7ee664038e5b5d362bd534bfe0e37
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:49:48 2017 +0200

    Correct error assertions for exceeding max part size

commit 9350712c0a46651e6a7a91d8819307ba4b08ec7e
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:44:28 2017 +0200

    Make CalcOptimalPartSize unexported

commit 593f3b2d37d16c51f229572c1d6b39fc2a234079
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:38:46 2017 +0200

    Add more output for debugging tests

commit b7193bfe67b535c9b9dd441610b41af11fe4538f
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:35:48 2017 +0200

    Extract size assertions into own function

commit 7521de23194652519fbbf3d61a41ef0b44b005fa
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:26:48 2017 +0200

    Move tests for CalcPartSize into own file

commit 6c483de7710cc119c870271ccad629c98c15c9a3
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:13:02 2017 +0200

    Use same assertions in AllUploadSizes test

commit 7b0290a07e7def09ea8ed982e7817a2ea7cd468a
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 18:30:02 2017 +0200

    Split negative test case from TestCalcOptimalPartSize into own test

commit 79c0a20d7bc71b494bc0824ad2aa8879b0c2900b
Merge: 5240f9b 997961f
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 17:32:31 2017 +0200

    Merge branch 'f-s3-part-size' of https://github.com/flaneurtv/tusd into flaneurtv-f-s3-part-size

commit 997961ff5c
Author: Markus Kienast <mark@rickkiste.at>
Date:   Fri Sep 1 00:59:38 2017 +0200

    TestNewUploadLargerMaxObjectSize

commit 0831bd79f8
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 31 23:08:03 2017 +0200

    fmt.Sprintf removed, range from 0 - MaxObjectSize+1

commit 1be7081524
Author: Markus Kienast <mark@rickkiste.at>
Date:   Tue Aug 29 10:23:50 2017 +0200

    turn off debug mode

commit be9a9bec10
Author: Markus Kienast <mark@rickkiste.at>
Date:   Tue Aug 29 10:12:20 2017 +0200

    moved MaxObjectSize check to NewUpload, refined tests

    * moved MaxObjectSize check to NewUpload
    * removed MaxObjectSize check from CalcOptimalPartSize
    * switched to assert in tests
    * added TestAllPartSizes, excluded in short mode

    TODO: TestNewUploadLargerMaxObjectSize needs to fail if MaxObjectSize > size

commit 7c22847a45
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:55:07 2017 +0200

    adding debug code to TestCalcOptimalPartSize

commit 5240f9b549000fac34be79ddfbe6e82404387f6b
Merge: 63c011e 5b116e7
Author: Marius <maerious@gmail.com>
Date:   Sat Aug 26 12:50:51 2017 +0200

    Merge branch 'f-s3-part-size' of https://github.com/flaneurtv/tusd into flaneurtv-f-s3-part-size

commit 63c011ef768db42e99004df921c2b9e5c4776fd2
Author: Marius <maerious@gmail.com>
Date:   Sat Aug 26 12:50:45 2017 +0200

    Format s3store_test

commit 5b116e7087
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:24:22 2017 +0200

    restructuring tests to accommodate optimalPartSize of 0

commit 93134a5696
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:03:18 2017 +0200

    moving MaxObjectSize check to top

commit 68e6bb8c41
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 02:31:27 2017 +0200

    enhance readability, comments and errors

commit 8831a98c34
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 24 02:27:57 2017 +0200

    separated partsize calc and error handling

commit f059acc7cc
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 24 01:29:26 2017 +0200

    fixed edge cases; pre-cleanup

commit e2e3b9ffe4
Author: Markus Kienast <mark@rickkiste.at>
Date:   Wed Aug 23 13:28:59 2017 +0200

    added error, when size > MaxObjectSize; additional case in algorithm + tests; go fmt

commit 381d3326cb
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 17 16:32:25 2017 +0200

    calculating PartSize based on size of upload

    simplified algorithm, respect MaxObjectSize, updated tests, go fmt

commit 1ad6187d6d
Author: koenvo <info@koenvossen.nl>
Date:   Thu Aug 17 21:31:37 2017 +0200

    Take IsTruncated field of S3 ListParts API response into account (#148)

    * Take IsTruncated field of S3 ListParts API response into account

    * Rename s3store.ListParts to ListAllParts

    * Use proper formatting + make listAllParts private + test listAllParts through TestGetInfo

    * Update TestFinishUpload to also test paged ListParts response
2017-09-03 10:57:06 +02:00
Markus Kienast 028ba57206 Squashed commit of the following:
commit 5a268dbafb9318b888142931ea27a1af10b9a8e7
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 11:47:26 2017 +0200

    Remove manual assignment of upload ID in S3Store

commit a37e149090ee7fd5f170d24ccc33b8af9ae18fae
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 11:42:00 2017 +0200

    Format Go code

commit 6643a9be62
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sun Jul 16 17:08:24 2017 +0200

    fixed ID value in .info; adjusted tests; fixed assert(expected, received) swap
2017-07-19 11:54:26 +02:00
Marius d48e4ac577 Correct when S3Store write loop is ended 2017-03-23 12:06:15 +01:00
Marius 37fe99b37d Define own, minimal S3API interface 2016-12-20 17:13:02 +01:00
Marius 513fe9fcf3 Correct linting issue and misspellings 2016-09-27 22:10:16 +02:00
Marius 1bf1c942e0 Update mocked S3API implementation for tests 2016-09-19 19:30:56 +02:00
Marius 3eefdbe2c6 Update mocked S3API implementation for tests 2016-08-26 17:14:33 +02:00
Marius 015ae39d18 Replace non-ASCII chars in metadata for multipart uploads with question mark
AWS does not handle non-ASCII encoded values for metadata values well since
they are transported in HTTP header values which, by specification, should
only contain ASCII characters. If you still supply AWS with, for example,
UTF-8 encoded strings it will reject the request due to mismatching
signatures. Our solution is to replace these characters with question
marks.
2016-07-06 16:25:06 +02:00
Marius 2f436ab7e3 Update mocked S3 API for tests 2016-05-24 17:32:21 +02:00
Marius 199487bdf6 Set correct ID in S3Store.GetInfo 2016-03-26 18:23:37 +01:00
Marius be8fd83d0d Document functions for handling StoreComposers 2016-03-11 20:17:43 +01:00
Marius 21ae1c45a7 Add initial draft of StoreComposr 2016-02-21 23:25:35 +01:00
Marius 11b92b91c2 Provide better URL to S3 consistency models 2016-02-11 12:23:46 +01:00
Marius bbf706622c Implement Concatentation extension for S3Store 2016-02-03 21:18:21 +01:00
Marius bfde73ff89 Test interface implementations for data stores 2016-01-19 22:37:05 +01:00
Marius 29047eddc2 Add support for meta data in S3Store 2016-01-19 21:39:24 +01:00
Marius 101648e382 Add documentation about S3Store 2016-01-17 15:30:19 +01:00
Marius 810fa89494 Add support for upload termination in S3Store 2016-01-16 16:12:37 +01:00
Marius cb0f8dac6a Add isAwsError helper method 2016-01-12 22:21:05 +01:00
Marius 2073521776 Add tests for S3Store 2016-01-05 18:21:53 +01:00
Marius 063d4a2442 Add first draft for s3Store 2015-12-08 22:26:37 +01:00