Commit Graph

558 Commits

Author SHA1 Message Date
Markus Kienast 2df5d11672 s3store: calculate part size based on upload size
See https://github.com/tus/tusd/issues/149 and
https://github.com/tus/tusd/pull/150 for more details.

Squashed commit of the following:

commit 78312ab26ea7ee664038e5b5d362bd534bfe0e37
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:49:48 2017 +0200

    Correct error assertions for exceeding max part size

commit 9350712c0a46651e6a7a91d8819307ba4b08ec7e
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:44:28 2017 +0200

    Make CalcOptimalPartSize unexported

commit 593f3b2d37d16c51f229572c1d6b39fc2a234079
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:38:46 2017 +0200

    Add more output for debugging tests

commit b7193bfe67b535c9b9dd441610b41af11fe4538f
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:35:48 2017 +0200

    Extract size assertions into own function

commit 7521de23194652519fbbf3d61a41ef0b44b005fa
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:26:48 2017 +0200

    Move tests for CalcPartSize into own file

commit 6c483de7710cc119c870271ccad629c98c15c9a3
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 19:13:02 2017 +0200

    Use same assertions in AllUploadSizes test

commit 7b0290a07e7def09ea8ed982e7817a2ea7cd468a
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 18:30:02 2017 +0200

    Split negative test case from TestCalcOptimalPartSize into own test

commit 79c0a20d7bc71b494bc0824ad2aa8879b0c2900b
Merge: 5240f9b 997961f
Author: Marius <maerious@gmail.com>
Date:   Fri Sep 1 17:32:31 2017 +0200

    Merge branch 'f-s3-part-size' of https://github.com/flaneurtv/tusd into flaneurtv-f-s3-part-size

commit 997961ff5c
Author: Markus Kienast <mark@rickkiste.at>
Date:   Fri Sep 1 00:59:38 2017 +0200

    TestNewUploadLargerMaxObjectSize

commit 0831bd79f8
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 31 23:08:03 2017 +0200

    fmt.Sprintf removed, range from 0 - MaxObjectSize+1

commit 1be7081524
Author: Markus Kienast <mark@rickkiste.at>
Date:   Tue Aug 29 10:23:50 2017 +0200

    turn off debug mode

commit be9a9bec10
Author: Markus Kienast <mark@rickkiste.at>
Date:   Tue Aug 29 10:12:20 2017 +0200

    moved MaxObjectSize check to NewUpload, refined tests

    * moved MaxObjectSize check to NewUpload
    * removed MaxObjectSize check from CalcOptimalPartSize
    * switched to assert in tests
    * added TestAllPartSizes, excluded in short mode

    TODO: TestNewUploadLargerMaxObjectSize needs to fail if MaxObjectSize > size

commit 7c22847a45
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:55:07 2017 +0200

    adding debug code to TestCalcOptimalPartSize

commit 5240f9b549000fac34be79ddfbe6e82404387f6b
Merge: 63c011e 5b116e7
Author: Marius <maerious@gmail.com>
Date:   Sat Aug 26 12:50:51 2017 +0200

    Merge branch 'f-s3-part-size' of https://github.com/flaneurtv/tusd into flaneurtv-f-s3-part-size

commit 63c011ef768db42e99004df921c2b9e5c4776fd2
Author: Marius <maerious@gmail.com>
Date:   Sat Aug 26 12:50:45 2017 +0200

    Format s3store_test

commit 5b116e7087
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:24:22 2017 +0200

    restructuring tests to accommodate optimalPartSize of 0

commit 93134a5696
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 12:03:18 2017 +0200

    moving MaxObjectSize check to top

commit 68e6bb8c41
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sat Aug 26 02:31:27 2017 +0200

    enhance readability, comments and errors

commit 8831a98c34
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 24 02:27:57 2017 +0200

    separated partsize calc and error handling

commit f059acc7cc
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 24 01:29:26 2017 +0200

    fixed edge cases; pre-cleanup

commit e2e3b9ffe4
Author: Markus Kienast <mark@rickkiste.at>
Date:   Wed Aug 23 13:28:59 2017 +0200

    added error, when size > MaxObjectSize; additional case in algorithm + tests; go fmt

commit 381d3326cb
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Aug 17 16:32:25 2017 +0200

    calculating PartSize based on size of upload

    simplified algorithm, respect MaxObjectSize, updated tests, go fmt

commit 1ad6187d6d
Author: koenvo <info@koenvossen.nl>
Date:   Thu Aug 17 21:31:37 2017 +0200

    Take IsTruncated field of S3 ListParts API response into account (#148)

    * Take IsTruncated field of S3 ListParts API response into account

    * Rename s3store.ListParts to ListAllParts

    * Use proper formatting + make listAllParts private + test listAllParts through TestGetInfo

    * Update TestFinishUpload to also test paged ListParts response
2017-09-03 10:57:06 +02:00
Marius a51f5994bb Squashed commit of the following:
commit d6ba86b7c300122af52e5be71b5fade4bb88d77c
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 17:44:56 2017 +0200

    Add documentation for post-create

commit 1d1605f98036e6cfd7e2695cf2e4f8510a220326
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 13:21:01 2017 +0200

    Remove duplicated log for created uploads

commit 1f5c6d22bb62e1251297db9b10f78f6db363759d
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 13:08:11 2017 +0200

    Rename UploadsCreated to CreatedUploads to match other names

commit 5c67ea7548548986c4bce79b7eea311a02a4c541
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 13:07:14 2017 +0200

    Remove unused variable from hook script

commit e21d7f5824ce533d80d74e75835ddc33de2184fb
Merge: 028ba57 447aa9c
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 12:47:52 2017 +0200

    Merge branch 'f-post-create-hook' of https://github.com/flaneurtv/tusd into flaneurtv-f-post-create-hook

commit 447aa9c485
Author: Markus Kienast <mark@rickkiste.at>
Date:   Thu Jul 13 15:57:07 2017 +0200

    added post-create hook

    added test for handler.UploadCreated

    go fmt .
2017-07-19 17:45:16 +02:00
Markus Kienast 028ba57206 Squashed commit of the following:
commit 5a268dbafb9318b888142931ea27a1af10b9a8e7
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 11:47:26 2017 +0200

    Remove manual assignment of upload ID in S3Store

commit a37e149090ee7fd5f170d24ccc33b8af9ae18fae
Author: Marius <maerious@gmail.com>
Date:   Wed Jul 19 11:42:00 2017 +0200

    Format Go code

commit 6643a9be62
Author: Markus Kienast <mark@rickkiste.at>
Date:   Sun Jul 16 17:08:24 2017 +0200

    fixed ID value in .info; adjusted tests; fixed assert(expected, received) swap
2017-07-19 11:54:26 +02:00
FLANEUR ba0f004df7 Add Dockerfile (#129)
* working alpine Dockerfile usable for local and autobuild

* removed volumes
2017-05-29 23:21:53 +02:00
Marius 73763acdaa Update apache2.conf 2017-05-22 22:11:43 +02:00
Marius ac110932a4 Add link to Apache example configuration. 2017-05-20 22:38:35 +02:00
Marius 5472f66fd8 Create apache2.conf 2017-05-20 22:36:04 +02:00
Peixian Wang a7950fdadb Add HTTP hooks
Closes https://github.com/tus/tusd/pull/120
2017-04-14 16:29:18 +02:00
Marius 535e0dff59 Use new consul test server functions 2017-04-12 12:11:11 +02:00
Marius 377aee0562 Update testing dependencies 2017-04-12 12:10:24 +02:00
Marius 3918ed2a05 Merge branch 'master' of github.com:tus/tusd 2017-04-12 11:54:45 +02:00
Matthew Wratt 5600cf2d6e Allow override of default options response 2017-04-12 11:53:15 +02:00
Marius 91dea49fc7 Add explicit notice about no cookies in hooks
In regards to https://github.com/tus/tusd/pull/58#issuecomment-290379041
2017-03-30 15:17:41 +02:00
Marius b2a8ae8c85 Correct metrics types
/cc @kvz
2017-03-30 14:51:02 +02:00
Marius 473489ce06 Exlude consullocker from tests temporarily 2017-03-23 14:06:00 +01:00
Marius cc1df8e9ed Remove second unlock attempt from tests
Unlocking an already unlocked lock is not guaranteed to succeed
2017-03-23 12:23:05 +01:00
Marius d48e4ac577 Correct when S3Store write loop is ended 2017-03-23 12:06:15 +01:00
Marius 7f26757a40 Update lockfile dependency 2017-03-03 11:16:56 +01:00
Marius 8a00983980 Merge branch 'master' of github.com:tus/tusd 2017-03-01 19:43:57 +01:00
Marius c8b8703bb1 Move network timeout handling back in UnroutedHandler 2017-03-01 19:43:37 +01:00
Mischa ter Smitten 6ab114d0e0 Make use of fqdn and timezone role (#115) 2017-03-01 14:14:56 +01:00
oliverpool 74fca8c6c3 Squashed commit of the following:
commit d214ad5c92073cb754c70ab73e6cf229cba01c72
Author: Marius <maerious@gmail.com>
Date:   Tue Feb 28 20:38:47 2017 +0100

    Minor code and comment cleanups

commit 2826227404296d98a2d83519efaa754cc07a47b1
Merge: db47c89 1a58d6e
Author: Marius <maerious@gmail.com>
Date:   Tue Feb 28 20:13:15 2017 +0100

    Merge branch 'metrics_race_condition' of https://github.com/oliverpool/tusd into oliverpool-metrics_race_condition

commit 1a58d6e084
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Tue Feb 28 19:14:21 2017 +0100

    prometheus metrics typo

commit db47c8976489917a8afc53878335ab66d73fe60d
Merge: 168942b 12054be
Author: Marius <maerious@gmail.com>
Date:   Tue Feb 28 17:36:47 2017 +0100

    Merge branch 'metrics_race_condition' of https://github.com/oliverpool/tusd into oliverpool-metrics_race_condition

commit 12054be4e7
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Tue Feb 28 12:07:43 2017 +0100

    add go vet on travis

commit 16e95d2d91
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Tue Feb 28 11:58:58 2017 +0100

    add one go vet on appveyor

commit 13bc64517b
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Tue Feb 28 11:57:56 2017 +0100

    use even less pointers

commit db7130f18a
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Mon Feb 27 22:12:45 2017 +0100

    use pointer only when needed

commit 647f7390c6
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Thu Feb 23 16:29:01 2017 +0100

    Prevent lock malfunction

    literal copies lock value from metrics

commit 1ce196ed35
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 22 12:22:19 2017 +0100

    handle read timeout simplification in metrics

commit 338017c9f4
Merge: c378bc9 45a9278
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 22 12:21:42 2017 +0100

    Merge branch 'master' into metrics_race_condition

commit c378bc9798
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 22 11:51:56 2017 +0100

    consistent naming

commit e2b0050b8d
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 22 11:44:05 2017 +0100

    Use a simplfied error structure for error counting

commit 3da4095c08
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Mon Feb 20 16:38:27 2017 +0100

    split ptr retrieval and incrementation

commit ebd6873cbb
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Sun Feb 5 22:01:34 2017 +0100

    Useless initialization

commit 691224cdbf
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Thu Feb 2 08:25:55 2017 +0100

    increment the pointer

commit 3d9395e1df
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Thu Feb 2 08:25:42 2017 +0100

    Simplify lock logic

commit d7a4619080
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 1 08:37:46 2017 +0100

    Prevent race condition on errorstotal increment

    The HTTPError interface is used to have detailed metrics (by code as
    well)
2017-02-28 20:40:44 +01:00
Marius 168942bfb6 Remove Prometheus client from vendors dependencies
Closes https://github.com/tus/tusd/issues/114
2017-02-28 17:17:13 +01:00
Marius 3c19f8ebdf Use tag pointing at head as version 2017-02-22 12:49:20 +01:00
Marius 45a9278c74 Include channel testing in goroutine fore NotifyUploadProgress test 2017-02-21 23:33:59 +01:00
Marius 9ad88c0424 Include Go 1.8 release into tested versions 2017-02-21 23:27:23 +01:00
Marius 9a3667b8ab Add documentation about post-receive event 2017-02-21 23:24:28 +01:00
Marius 73b273d4bd Add documentation for NotifyUploadProgress 2017-02-21 23:17:07 +01:00
Marius 68b6b20688 Add test for NotifiyUploadProgress 2017-02-21 23:00:51 +01:00
Marius f4fdc4164a Merge branch 'master' of github.com:tus/tusd 2017-02-20 18:32:35 +01:00
Marius ca95bf275a Exclude too detailed information from timeout error 2017-02-20 17:14:58 +01:00
Kevin van Zonneveld 1f176099c7 Upgrade Frey to v0.3.31 /cc @tersmitten (#113) 2017-02-17 14:48:54 +01:00
Kevin van Zonneveld f1d2d1c60f Upgrade Frey to v0.3.30 /cc @tersmitten (#112) 2017-02-17 10:56:08 +01:00
Mischa ter Smitten 918240fe32 Fix incorrect install npm user (#111)
* Fix incorrect install_npm_user

* Fix incorrect install_npm_user
2017-02-16 13:44:24 +01:00
Mischa ter Smitten f8c2c4401e Fix incorrect install_npm_user (#110) 2017-02-16 12:53:09 +01:00
Mischa ter Smitten 30cd41836d Improved .gitignore for Frey using Docker (#109) 2017-02-16 09:51:48 +01:00
Marius 3dd28fa028 Exclude certain blacklisted ciphers from Nginx 2017-02-06 15:32:37 +01:00
Marius 9be1ece381 Merge branch 'master' of github.com:tus/tusd 2017-02-06 15:12:03 +01:00
Marius e9255c63b9 Make minor improvements to memorylocker internals 2017-02-06 15:10:43 +01:00
Mischa ter Smitten 63014c78cc Cleanup Frey file (#104) 2017-02-01 10:46:36 +01:00
Mischa ter Smitten d016e08918 Remove unused dependencies (#103)
Using unattended-upgrades
2017-02-01 10:29:52 +01:00
Marius 5491b8ff12 Merge branch 'master' of github.com:tus/tusd 2017-01-31 17:08:47 +01:00
Marius 7710fd79fb Correct parsing Upload-Concat for final uplaods
Addresses https://github.com/tus/tus-resumable-upload-protocol/issues/123
2017-01-31 16:58:31 +01:00
Kevin van Zonneveld 8784d54045 Upgrade Frey to v0.3.28 /cc @tersmitten (#102) 2017-01-31 14:44:03 +01:00
Mischa ter Smitten 98009f87c1 Update Frey variables (#100)
Needed for future role updates and cleanup
2017-01-31 13:37:43 +01:00
Mischa ter Smitten 3052090f72 Update roles (#101) 2017-01-31 13:37:18 +01:00
Marius 029191c863 Include Golang binaries in PATH 2017-01-27 14:27:02 +01:00
Marius 198c72c479 Install consul(1) before running tests on AppVeyor 2017-01-27 14:22:53 +01:00
Marius b05382eced Correct constant for post-receive hook name 2017-01-26 23:16:22 +01:00
Marius f50f03fe6f Allow data store to set the status code for errors
Closes https://github.com/tus/tusd/issues/77
2017-01-26 23:15:05 +01:00