Commit Graph

22 Commits

Author SHA1 Message Date
Adam Jensen e276384a48
Remove misleading comment
The comment mentions 6, but the relevant sizes are 5MB, 5GB, and 5TB. I think it's preferable to let the code speak for itself so that the comment doesn't become stale.
2018-04-24 11:47:29 -04:00
Markus Kienast 2df5d11672 s3store: calculate part size based on upload size
See https://github.com/tus/tusd/issues/149 and
https://github.com/tus/tusd/pull/150 for more details.

Squashed commit of the following:

commit 78312ab26ea7ee664038e5b5d362bd534bfe0e37
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:49:48 2017 +0200

    Correct error assertions for exceeding max part size

commit 9350712c0a46651e6a7a91d8819307ba4b08ec7e
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:44:28 2017 +0200

    Make CalcOptimalPartSize unexported

commit 593f3b2d37d16c51f229572c1d6b39fc2a234079
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:38:46 2017 +0200

    Add more output for debugging tests

commit b7193bfe67b535c9b9dd441610b41af11fe4538f
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:35:48 2017 +0200

    Extract size assertions into own function

commit 7521de23194652519fbbf3d61a41ef0b44b005fa
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:26:48 2017 +0200

    Move tests for CalcPartSize into own file

commit 6c483de7710cc119c870271ccad629c98c15c9a3
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:13:02 2017 +0200

    Use same assertions in AllUploadSizes test

commit 7b0290a07e7def09ea8ed982e7817a2ea7cd468a
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 18:30:02 2017 +0200

    Split negative test case from TestCalcOptimalPartSize into own test

commit 79c0a20d7bc71b494bc0824ad2aa8879b0c2900b
Merge: 5240f9b 997961f
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 17:32:31 2017 +0200

    Merge branch 'f-s3-part-size' of https://github.com/flaneurtv/tusd into flaneurtv-f-s3-part-size

commit 997961ff5c
Author: Markus Kienast <mark@rickkiste.at>
Date:   Fri Sep 1 00:59:38 2017 +0200

    TestNewUploadLargerMaxObjectSize

commit 0831bd79f8
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 31 23:08:03 2017 +0200

    fmt.Sprintf removed, range from 0 - MaxObjectSize+1

commit 1be7081524
Author: Markus Kienast <mark@rickkiste.at>
Date:   Tue Aug 29 10:23:50 2017 +0200

    turn off debug mode

commit be9a9bec10
Author: Markus Kienast <mark@rickkiste.at>
Date:   Tue Aug 29 10:12:20 2017 +0200

    moved MaxObjectSize check to NewUpload, refined tests

    * moved MaxObjectSize check to NewUpload
    * removed MaxObjectSize check from CalcOptimalPartSize
    * switched to assert in tests
    * added TestAllPartSizes, excluded in short mode

    TODO: TestNewUploadLargerMaxObjectSize needs to fail if MaxObjectSize > size

commit 7c22847a45
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:55:07 2017 +0200

    adding debug code to TestCalcOptimalPartSize

commit 5240f9b549000fac34be79ddfbe6e82404387f6b
Merge: 63c011e 5b116e7
Author: Marius <maerious@gmail.com>
Date:   Sat Aug 26 12:50:51 2017 +0200

    Merge branch 'f-s3-part-size' of https://github.com/flaneurtv/tusd into flaneurtv-f-s3-part-size

commit 63c011ef768db42e99004df921c2b9e5c4776fd2
Author: Marius <maerious@gmail.com>
Date:   Sat Aug 26 12:50:45 2017 +0200

    Format s3store_test

commit 5b116e7087
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:24:22 2017 +0200

    restructuring tests to accommodate optimalPartSize of 0

commit 93134a5696
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:03:18 2017 +0200

    moving MaxObjectSize check to top

commit 68e6bb8c41
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 02:31:27 2017 +0200

    enhance readability, comments and errors

commit 8831a98c34
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 24 02:27:57 2017 +0200

    separated partsize calc and error handling

commit f059acc7cc
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 24 01:29:26 2017 +0200

    fixed edge cases; pre-cleanup

commit e2e3b9ffe4
Author: Markus Kienast <mark@rickkiste.at>
Date:   Wed Aug 23 13:28:59 2017 +0200

    added error, when size > MaxObjectSize; additional case in algorithm + tests; go fmt

commit 381d3326cb
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 17 16:32:25 2017 +0200

    calculating PartSize based on size of upload

    simplified algorithm, respect MaxObjectSize, updated tests, go fmt

commit 1ad6187d6d
Author: koenvo <info@koenvossen.nl>
Date:   Thu Aug 17 21:31:37 2017 +0200

    Take IsTruncated field of S3 ListParts API response into account (#148)

    * Take IsTruncated field of S3 ListParts API response into account

    * Rename s3store.ListParts to ListAllParts

    * Use proper formatting + make listAllParts private + test listAllParts through TestGetInfo

    * Update TestFinishUpload to also test paged ListParts response
2017-09-03 10:57:06 +02:00
Markus Kienast 028ba57206 Squashed commit of the following:
commit 5a268dbafb9318b888142931ea27a1af10b9a8e7
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 11:47:26 2017 +0200

    Remove manual assignment of upload ID in S3Store

commit a37e149090ee7fd5f170d24ccc33b8af9ae18fae
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 11:42:00 2017 +0200

    Format Go code

commit 6643a9be62
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sun Jul 16 17:08:24 2017 +0200

    fixed ID value in .info; adjusted tests; fixed assert(expected, received) swap
2017-07-19 11:54:26 +02:00
Marius d48e4ac577 Correct when S3Store write loop is ended 2017-03-23 12:06:15 +01:00
Marius 37fe99b37d Define own, minimal S3API interface 2016-12-20 17:13:02 +01:00
Marius 513fe9fcf3 Correct linting issue and misspellings 2016-09-27 22:10:16 +02:00
Marius 1bf1c942e0 Update mocked S3API implementation for tests 2016-09-19 19:30:56 +02:00
Marius 3eefdbe2c6 Update mocked S3API implementation for tests 2016-08-26 17:14:33 +02:00
Marius 015ae39d18 Replace non-ASCII chars in metadata for multipart uploads with question mark
AWS does not handle non-ASCII encoded values for metadata values well since
they are transported in HTTP header values which, by specification, should
only contain ASCII characters. If you still supply AWS with, for example,
UTF-8 encoded strings it will reject the request due to mismatching
signatures. Our solution is to replace these characters with question
marks.
2016-07-06 16:25:06 +02:00
Marius 2f436ab7e3 Update mocked S3 API for tests 2016-05-24 17:32:21 +02:00
Marius 199487bdf6 Set correct ID in S3Store.GetInfo 2016-03-26 18:23:37 +01:00
Marius be8fd83d0d Document functions for handling StoreComposers 2016-03-11 20:17:43 +01:00
Marius 21ae1c45a7 Add initial draft of StoreComposr 2016-02-21 23:25:35 +01:00
Marius 11b92b91c2 Provide better URL to S3 consistency models 2016-02-11 12:23:46 +01:00
Marius bbf706622c Implement Concatentation extension for S3Store 2016-02-03 21:18:21 +01:00
Marius bfde73ff89 Test interface implementations for data stores 2016-01-19 22:37:05 +01:00
Marius 29047eddc2 Add support for meta data in S3Store 2016-01-19 21:39:24 +01:00
Marius 101648e382 Add documentation about S3Store 2016-01-17 15:30:19 +01:00
Marius 810fa89494 Add support for upload termination in S3Store 2016-01-16 16:12:37 +01:00
Marius cb0f8dac6a Add isAwsError helper method 2016-01-12 22:21:05 +01:00
Marius 2073521776 Add tests for S3Store 2016-01-05 18:21:53 +01:00
Marius 063d4a2442 Add first draft for s3Store 2015-12-08 22:26:37 +01:00