LumeWeb
/
tusd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,257 Commits 7 Branches 68 Tags 22 MiB
Commit Graph

114 Commits

Author SHA1 Message Date
Marius 781324986d s3store: Return 400 instead of 500 if upload cannot be streamed 
Closes https://github.com/tus/tusd/issues/490
 2021-07-01 18:48:58 +02:00
Marius ccdfe8e604 cli: Add flag to customize the S3 part buffer size 2021-06-21 13:17:36 +02:00
Marius 36f12b1d18 s3store: Fix failing tests due to missing argument 2021-06-21 13:15:21 +02:00
Marius 946539c3b9 cli: Add option to expose Go's pprof 2021-05-28 13:26:13 +02:00
Marius f4314dd360 s3store: Do not register metrics to default registry 2021-05-24 23:45:54 +02:00
Marius 0f24a80ea5 s3store: Expose metrics about request durations 2021-05-24 12:00:20 +02:00
Marius ce54ff8b1f fixup! s3store: Parallelize part uploads and information retrieval (#478) 2021-05-23 12:54:53 +02:00
Marius 8fd18364e7
s3store: Parallelize part uploads and information retrieval (#478) 
* Add first draft of parallel upload queue

* s3store: Use queue for parallel uploads

* Revert "Add first draft of parallel upload queue"

This reverts commit 86a329cef2.

* Revert "s3store: Use queue for parallel uploads"

This reverts commit 29b59a2c90.

* s3store: Cache results from listing parts and checking incomplete object

* s3store: Remove debugging output`

* s3store: Make requests for fetching info concurrently

* s3store: Make parallel uploads work and tests pass

* s3store: Add semaphore package

* s3store: Add comments to semaphore package

* s3store: Encapsulate more logic into s3PartProducer

* s3store: Refactor WriteChunk

* s3store: Remove TODO

* s3store: Acquire lock before uploading

* cli: Add flag for setting concurrency limit

* s3store: One more comment
 2021-05-18 10:29:18 +02:00
Marius 8c5192c254 s3store: Move parts producer into own file 2021-04-26 12:18:09 +02:00
Marius b3bf854712 s3store: Remove test for handling unexpected EOF 
The store must no longer handle unexpected EOFs since this is convered by the handler.
See the previous commits for more details
 2021-04-26 12:17:20 +02:00
Marius 6d987aa226 core: Handle errors from reading request body centrally 2021-04-26 10:08:37 +02:00
Marius 97602c3d62 filestore: Handle os.ErrNotExist not in core handler 2021-04-25 23:17:42 +02:00
Adam Jensen e85d630748
s3store: Fix data loss when using deferred upload lengths (#462) 2021-03-11 18:17:47 +01:00
Marius fdb19a7904 s3store: Temporary workaround against data loss on connection resets 2021-02-10 23:07:16 +01:00
Marius 67fd74e129 Add flag to disable calculating expensive hashes for S3 2021-01-18 13:31:07 +01:00
Ahmed J 0822c0ac43
handler: Use correct separation symbol for parsing Forwarded header (#433) 
* fix forwarded header separator

* added tests

* fix tests

* updated test

* Reuse existing test for Forwarded header

Co-authored-by: Ahmed J <ahjabir@earthlink.iq>
Co-authored-by: Marius <marius@transloadit.com>
 2020-10-12 12:36:55 +02:00
Márk Sági-Kazár 3a344c5e81
memorylocker: Fix function names in documentation (#412) 2020-08-01 15:02:46 +02:00
Marius 9831c8b4da
s3store: Increase default part size (#410) 
* s3store: Increase default part size

* Update tests and add flag

* Fix minor issues
 2020-08-01 14:58:31 +02:00
Adam Jensen 6662f43d01
s3Store: Concurrently write upload parts to S3 while reading from client (#402) 
* Allow empty metadata values

* Make tests less fragile by allowing loose call ordering

* Add s3ChunkProducer

* Integrate s3ChunkProducer to support chunk buffering

* Remove completed chunk files inline to reduce disk space usage

* Add tests for chunk producer

* docs: Use value from Host header to forward to tusd

* Use int64 for MaxBufferedParts field

* Default to 20 buffered parts

* Rename s3ChunkProducer -> s3PartProducer

* Document s3PartProducer struct

* Clarify misleading comment

* Revert "Remove completed chunk files inline to reduce disk space usage"

This reverts commit b72a4d43d6.

* Remove redundant seek

This is already being done in s3PartProducer.

* Clean up any remaining files in the channel when we return

* Make putPart* functions responsible for cleaning up temp files

* handler: Add tests for empty metadata pairs

* Factor out cleanUpTempFile func

* Add test to ensure that temporary files get cleaned up

Co-authored-by: Jens Steinhauser <jens.steinhauser@gmail.com>
Co-authored-by: Marius <marius@transloadit.com>
 2020-07-29 15:24:46 +02:00
Marius 26b84bcb1c handler: Add tests for empty metadata pairs 2020-07-15 17:30:17 +02:00
Jens Steinhauser 87045a8fbf Allow empty metadata values 2020-06-20 13:24:07 +02:00
Hamish Forbes fdf168fbb6
cli: Add pre-finish hook (#382) 
* core: add new synchronous event: pre-finish

* docs: pre-finish hook event

* docs: Added information about CORS to the FAQ. (#384)

* Added information about CORS to the FAQ.

* docs: Expand explanation of CORS

Co-authored-by: Marius <marius@transloadit.com>

* cli: add header forwarding in HTTP hooks (#371)

* cli: add header forwarding in HTTP hooks

* docs: Reword header forwarding flag, add documentation

* Don't enable pre-finish hooks by default

* Enable pre-finish hooks for plugins

* docs: default enabled hooks

* Rename callback

Co-authored-by: josh-marshall-jax <52457971+josh-marshall-jax@users.noreply.github.com>
Co-authored-by: Marius <marius@transloadit.com>
 2020-05-15 17:27:09 +02:00
Marius 9210fbe0fc s3store: Allow concatenation of uploads with less than 5MB 2020-04-27 12:42:56 +02:00
Marius cfebf1778e cli: Fix build time error of gRPC hooks 2020-04-06 13:46:58 +02:00
Marius 936f465cc3 Merge branch 'master' of github.com:tus/tusd 2020-04-06 12:24:05 +02:00
Marius 286d2d1e23 handler: Add request ID to log output 2020-04-06 12:20:57 +02:00
dmmakita 43bd4b0ccb
core: Allow Authorization header in CORS (#355) 
* Update cors_test.go

Adding Allow Authorization Header

* Update unrouted_handler.go

Adding Allow Authorization Header to CORS
 2020-02-24 20:47:54 +01:00
Marius 973a4fe066 s3store: Fix bug when completing empty upload 2020-02-23 20:26:00 +01:00
Iñigo 8ef7648713
cli: add gRPC hooks (#316) 
* add grpc hook

* add retry/backoff params
make streaming RPC call

* Update cmd/tusd/cli/flags.go

Co-Authored-By: Márk Sági-Kazár <sagikazarmark@users.noreply.github.com>

* move one time grpc configuration to `Setup`

* remove stream grpc

Co-authored-by: Márk Sági-Kazár <sagikazarmark@users.noreply.github.com>
 2020-02-06 17:35:37 +01:00
Adam Jensen 9c0e0c8f11
s3store: Add optional key prefix for metadata objects (#347) 
* Add MetadataObjectPrefix field to S3Store

* Add metadataKeyWithPrefix helper function

* Use metadataKeyWithPrefix for .info and .part operations

* Add s3store tests for metadata object prefixes

* Clarify ObjectPrefix docs
 2020-02-01 17:33:02 +01:00
Marius 790d6ea849 handler: Allow method overriding in CORS requests 2019-11-15 21:06:29 +01:00
oliverpool e138fc3e9e filestore: Remove useless defer (#318) 
Since the `file` is not used after the creation, its closing can be done without defer (and the err can be checked).
 2019-10-23 18:51:36 +02:00
Ankit Pokhrel 898f3fe72a gcsstore: Remove unnecessary typecasts and unused mocks (#313) 2019-10-09 14:44:13 +02:00
Ankit Pokhrel 321acd1821 core, gcsstore: Update deprecated methods (#312) 2019-10-08 10:30:19 +02:00
Marius a045b8c4fc core: Allow numbers in mime types 
See https://community.transloadit.com/t/tus-server-not-returning-the-correct-content-type-for-video-or-audio-file/15076
 2019-10-07 11:28:13 +02:00
Marius 57be489b8b fixup! core: Allow writing data to storage after request is cancelled 2019-09-29 20:12:50 +02:00
Marius 65d2f1c2ef core: Allow writing data to storage after request is cancelled 2019-09-29 20:03:18 +02:00
Marius b73d634b47 core: Pass Upload interfaces to ConcatUploads 2019-09-19 12:14:25 +02:00
Marius 6b21772107 core: Provide HTTP request details to hooks 
Closes https://github.com/tus/tusd/issues/185
 2019-09-19 11:15:48 +02:00
Marius 5a31f3230e s3store: Cache FileInfo between function calls 
This lead to an 15% increase in upload speed for our tests.
 2019-09-16 12:13:39 +02:00
Marius cbfde4e4b0 s3store: Pass context to AWS SDK 2019-09-15 14:33:02 +02:00
Marius 485c21d72e core: Add context to DataStores 
Closes https://github.com/tus/tusd/issues/288
 2019-09-15 13:43:59 +02:00
Marius db18dbb70f etcd3locker: Move to separate repository 2019-09-12 21:42:54 +02:00
Marius 7d639b930d core: Replace old LockerDataStore with new Locker 2019-09-12 12:37:43 +02:00
Marius 92826b171d etc3dlocker: Implement new Locker interface 2019-09-11 12:03:39 +02:00
Marius 65072acb79 memorylocker: Implement new locker interface 2019-09-11 10:28:38 +02:00
Marius 8e1dce1dcb filelocker: Add tests 2019-09-10 16:19:49 +02:00
Marius 241c458184 cli: Print out available tus extensions 2019-09-10 15:27:47 +02:00
Marius d01d878e29 gcsstore: Adjust to new DataStore interfaces 2019-09-10 11:09:07 +02:00
Marius d36b0ea3a9 s3store: Move to new DataStore interfaces 2019-09-10 10:05:45 +02:00
Marius 12114b1ae8 Merge branch 'master' of github.com:tus/tusd into v1 2019-09-09 13:37:53 +02:00
Marius 1389d86547 core: Add back composer_test 2019-08-26 20:41:00 +02:00
Marius 2e688d5d38 filestore: Implement new interfaces 2019-08-26 11:41:52 +02:00
Marius 5004d3ca4d core: Refactor DataStore 2019-08-25 22:10:55 +02:00
Marius 5acc586800 WIP on DataStore refactor 2019-08-24 15:14:51 +02:00
Marius aa0280d004 core: Remove deprecated DataStore option 2019-08-20 16:16:05 +02:00
Marius 647c65a125 filestore: Drop the .bin suffix 
Closes https://github.com/tus/tusd/issues/210
 2019-08-19 13:03:42 +02:00
Marius f290aa36d7 gcsstore: Add storage details 2019-08-19 11:08:24 +02:00
Marius a323c91a6e s3store: Add storage details 2019-08-19 10:38:08 +02:00
Marius c0c3a4d1d0 filestore: Add information about saved upload 2019-08-19 09:29:56 +02:00
Marius e8fb3a431b Merge branch 'master' of github.com:tus/tusd into v1 2019-08-18 14:34:37 +02:00
Marius 226638bcc4 etcd3locker: Update import path for etcd3 2019-06-16 17:06:17 +02:00
Marius 0f70b98b0f memorylocker: Remove deprecated constructor for MemoryLocker 2019-06-11 18:32:00 +02:00
Marius 42bfe35457 core: Move packages into pkg/ and internal/ folders 2019-06-11 18:23:20 +02:00