Commit Graph

76 Commits

Author SHA1 Message Date
genofire 306a031953
filestore: keep ID passed to NewUpload, if available (#820)
* fix(filestore): keep info.ID

* fix
2022-10-02 00:25:40 +02:00
Shun 7e6cd89114
cli: Change timestamp to microseconds in log (#794)
Co-authored-by: 内藤峻 <naito.shun@nttcom.co.jp>
2022-09-09 18:34:18 +02:00
Marius f898cd2b25 Revert "handler: Avoid calling WriteChunk if no data is available (#746)"
This reverts commit b5cbafdbd5.

See https://github.com/tus/tusd/issues/798
2022-09-09 18:21:09 +02:00
Roberto Villalba e423e30135
s3storeL Add prefix to upload concat source (#767) 2022-07-16 14:52:53 +02:00
Walter Pesce 884cf6ecc1
handler: Fix typo preventing video/ogg from being inlined (#758)
fixed typo to allow video/ogg content-disposition inline.
fixed comment typos.
2022-07-01 23:29:20 +02:00
André Fontenele b5cbafdbd5
handler: Avoid calling WriteChunk if no data is available (#746)
* fix: avoiding `writeChunk` no-op

* fix: formatting

* fix: added return statement

* fix: preserving side effects

* refactor: smaller change
2022-06-17 14:04:26 +02:00
Marius 63830b35d1 handler: Send pre-finish hook before post-finish
Fixes https://github.com/tus/tusd/issues/702
2022-06-17 13:05:38 +02:00
Tom Berger ebf767b2e9
gcsstore: Avoid panic when composing with no passed objects (#360) 2022-05-04 19:18:41 +02:00
Ole-Martin Bratteng 3feef174fd
handler, cli: Add a flag to disable downloads and termination of uploads (#668)
* Add a flag to disable downloads of uploaded files

* Add a flag to disable deletion of uploaded files

* Fix spelling error

* Rename `DisableDelete` to `DisableTermination`

* Also rename CLI flag

* Conditionally build the `Access-Control-Allow-Methods` headers

* Update tests with new order of allowed methods header

* Add test for checking conditional `Access-Control-Allow-Methods` header
2022-03-28 23:43:35 +02:00
Ole-Martin Bratteng 35dbca86f1
Fix leakage of info when no azure blob is found (#664)
* Create `isAzureError` function

* Throw 404 is info file is not found

* Use `isAzureError` in GetOffset function

* Make `GetUpload` match others without named returns

* Follow code style for `isAzureError` not returning but setting `err` to value

* Return not found when the blob is not found
This will also be triggered when the blob is being uploaded but not yet committed

* Return not found if blob is not found when getting offset

* Don't return error in GetUpload if GetOffset returns `handler.ErrNotFound`
2022-03-17 17:29:15 +01:00
Anatoly 6828cbdce1
s3store: stricter character replacement regexp for x-amz-metadata (#598)
* stricter character replacement rule for x-amz-metadata

* misspell fixed
2021-12-30 17:17:45 +01:00
Marius fa09e71262
deps: Update dependencies for GCSStore (#535)
* build(deps): bump google.golang.org/grpc from 1.28.0 to 1.41.0

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.28.0 to 1.41.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.28.0...v1.41.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump google.golang.org/api from 0.6.0 to 0.59.0

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.6.0 to 0.59.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.6.0...v0.59.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Regenerate GCS API mock

* update test

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-10-24 10:58:03 +02:00
Ole-Martin Bratteng 511ebaed0e
s3store: relax test to prevent failure from race conditions (#529)
* fix: if a race condition occur, we check for it in the test

* fix: windows sometimes only has 1 file

* Trigger test runs

* Trigger test runs

* Trigger test runs

* Trigger test runs

* Trigger test runs
2021-10-23 23:28:33 +02:00
Ole-Martin Bratteng 2d27132938
chore: update azure-storage-blob-go to v0.14.0 (#527)
* chore: update azure-storage-blob-go to v0.14.0

* fix: convert the block size from int32 to int64
2021-10-21 14:28:58 +02:00
Marius 84faa14987 core: Allow non-tus HEAD requests, Add Length to HEAD responses
See https://github.com/tus/tusd/pull/480

Squashed commit of the following:

commit 7439fd84a6103afdedaf94701a65ce4376789380
Author: Marius <marius@transloadit.com>
Date:   Mon Oct 18 00:27:12 2021 +0200

    Docs and test

commit 16d9dc67e8c8eefc328b1ce12d7e7ca01a49f9f6
Merge: bae0ffb bea5183
Author: Marius <marius@transloadit.com>
Date:   Mon Oct 18 00:23:13 2021 +0200

    Merge branch 'head_header_check' of https://github.com/s3rius/tusd into s3rius-head_header_check

commit bea5183ec3
Author: Pavel Kirilin <win10@list.ru>
Date:   Thu May 20 19:53:36 2021 +0400

    Fixed "Tus-Resumable" header check for HEAD request.

    Signed-off-by: Pavel Kirilin <win10@list.ru>
2021-10-18 00:29:13 +02:00
Tinco Andringa bae0ffb5e5
gcsstore: Allow object prefix to contain underscores (#495)
* Only determine object type based on name after last separator

* modify test to keep in mind directory prefixes with underscores in them

* update documentation to reflect support of underscores in gcs object prefix
2021-10-17 23:48:53 +02:00
benito 16a3747ec4
s3store: Accept alternate error response from DigitalOcean Spaces (#507)
* fix digitalocean spaces fetch

* remove log

* remove log

* remove info.Size check

* Add comment

* Delete .gitignore

Co-authored-by: Marius <marius.kleidl@gmail.com>
2021-10-15 21:50:17 +02:00
Marius 0ad435b4c8 core: Fix wrong offset in upload progress notifications
Closes https://github.com/tus/tusd/issues/500
2021-10-13 21:08:09 +02:00
Ole-Martin Bratteng 1b11885823
azurestore: Add implementation
* Add azure-storage-blob-go dependency

* Implement Azure BlobStorage store

* Add AzureStore Mock test

* Refactor Blob interfaces to use uppercase fields

* Refactor and remove the Create function
When getting the offset, and we get the status code BlobNotFound, we can say the offset is 0, and start from the beginning

* Update the mock

* Refactor error checking of GetOffset to actually check the service code

* Begin testing azurestore

* Write more tests

* New feature allows to set access type on new containers and blob access tier

* Write more docs

* Upgrade azure-storage-blob-go to v0.13.0

* Remove AzError, not needed

* Update link to container access type information

* Remove ?toc from link in comments

* Remove trailing spaces from workflow

* Run tests with go1.15 and 1.16

* Don't fail fast
This lets all other tests complete, and makes it easier to see if it's just a one-off fail, or on different OSes and versions

* Remove darwin 386 from `build_all.sh` script
Removed in go1.15 https://github.com/golang/go/issues/37610

* Update go version in `Dockerfile`

* Compile for Apple Silicone (darwin arm64)
Only go1.16 supports it
2021-07-29 01:14:50 +02:00
Marius 781324986d s3store: Return 400 instead of 500 if upload cannot be streamed
Closes https://github.com/tus/tusd/issues/490
2021-07-01 18:48:58 +02:00
Marius 8c5192c254 s3store: Move parts producer into own file 2021-04-26 12:18:09 +02:00
Marius b3bf854712 s3store: Remove test for handling unexpected EOF
The store must no longer handle unexpected EOFs since this is convered by the handler.
See the previous commits for more details
2021-04-26 12:17:20 +02:00
Marius 6d987aa226 core: Handle errors from reading request body centrally 2021-04-26 10:08:37 +02:00
Marius 97602c3d62 filestore: Handle os.ErrNotExist not in core handler 2021-04-25 23:17:42 +02:00
Adam Jensen e85d630748
s3store: Fix data loss when using deferred upload lengths (#462) 2021-03-11 18:17:47 +01:00
Marius fdb19a7904 s3store: Temporary workaround against data loss on connection resets 2021-02-10 23:07:16 +01:00
Marius 67fd74e129 Add flag to disable calculating expensive hashes for S3 2021-01-18 13:31:07 +01:00
Ahmed J 0822c0ac43
handler: Use correct separation symbol for parsing Forwarded header (#433)
* fix forwarded header separator

* added tests

* fix tests

* updated test

* Reuse existing test for Forwarded header

Co-authored-by: Ahmed J <ahjabir@earthlink.iq>
Co-authored-by: Marius <marius@transloadit.com>
2020-10-12 12:36:55 +02:00
Márk Sági-Kazár 3a344c5e81
memorylocker: Fix function names in documentation (#412) 2020-08-01 15:02:46 +02:00
Marius 9831c8b4da
s3store: Increase default part size (#410)
* s3store: Increase default part size

* Update tests and add flag

* Fix minor issues
2020-08-01 14:58:31 +02:00
Adam Jensen 6662f43d01
s3Store: Concurrently write upload parts to S3 while reading from client (#402)
* Allow empty metadata values

* Make tests less fragile by allowing loose call ordering

* Add s3ChunkProducer

* Integrate s3ChunkProducer to support chunk buffering

* Remove completed chunk files inline to reduce disk space usage

* Add tests for chunk producer

* docs: Use value from Host header to forward to tusd

* Use int64 for MaxBufferedParts field

* Default to 20 buffered parts

* Rename s3ChunkProducer -> s3PartProducer

* Document s3PartProducer struct

* Clarify misleading comment

* Revert "Remove completed chunk files inline to reduce disk space usage"

This reverts commit b72a4d43d6.

* Remove redundant seek

This is already being done in s3PartProducer.

* Clean up any remaining files in the channel when we return

* Make putPart* functions responsible for cleaning up temp files

* handler: Add tests for empty metadata pairs

* Factor out cleanUpTempFile func

* Add test to ensure that temporary files get cleaned up

Co-authored-by: Jens Steinhauser <jens.steinhauser@gmail.com>
Co-authored-by: Marius <marius@transloadit.com>
2020-07-29 15:24:46 +02:00
Marius 26b84bcb1c handler: Add tests for empty metadata pairs 2020-07-15 17:30:17 +02:00
Jens Steinhauser 87045a8fbf Allow empty metadata values 2020-06-20 13:24:07 +02:00
Hamish Forbes fdf168fbb6
cli: Add pre-finish hook (#382)
* core: add new synchronous event: pre-finish

* docs: pre-finish hook event

* docs: Added information about CORS to the FAQ. (#384)

* Added information about CORS to the FAQ.

* docs: Expand explanation of CORS

Co-authored-by: Marius <marius@transloadit.com>

* cli: add header forwarding in HTTP hooks (#371)

* cli: add header forwarding in HTTP hooks

* docs: Reword header forwarding flag, add documentation

* Don't enable pre-finish hooks by default

* Enable pre-finish hooks for plugins

* docs: default enabled hooks

* Rename callback

Co-authored-by: josh-marshall-jax <52457971+josh-marshall-jax@users.noreply.github.com>
Co-authored-by: Marius <marius@transloadit.com>
2020-05-15 17:27:09 +02:00
Marius 9210fbe0fc s3store: Allow concatenation of uploads with less than 5MB 2020-04-27 12:42:56 +02:00
Marius cfebf1778e cli: Fix build time error of gRPC hooks 2020-04-06 13:46:58 +02:00
Marius 936f465cc3 Merge branch 'master' of github.com:tus/tusd 2020-04-06 12:24:05 +02:00
Marius 286d2d1e23 handler: Add request ID to log output 2020-04-06 12:20:57 +02:00
dmmakita 43bd4b0ccb
core: Allow Authorization header in CORS (#355)
* Update cors_test.go

Adding Allow Authorization Header

* Update unrouted_handler.go

Adding Allow Authorization Header to CORS
2020-02-24 20:47:54 +01:00
Marius 973a4fe066 s3store: Fix bug when completing empty upload 2020-02-23 20:26:00 +01:00
Iñigo 8ef7648713
cli: add gRPC hooks (#316)
* add grpc hook

* add retry/backoff params
make streaming RPC call

* Update cmd/tusd/cli/flags.go

Co-Authored-By: Márk Sági-Kazár <sagikazarmark@users.noreply.github.com>

* move one time grpc configuration to `Setup`

* remove stream grpc

Co-authored-by: Márk Sági-Kazár <sagikazarmark@users.noreply.github.com>
2020-02-06 17:35:37 +01:00
Adam Jensen 9c0e0c8f11
s3store: Add optional key prefix for metadata objects (#347)
* Add MetadataObjectPrefix field to S3Store

* Add metadataKeyWithPrefix helper function

* Use metadataKeyWithPrefix for .info and .part operations

* Add s3store tests for metadata object prefixes

* Clarify ObjectPrefix docs
2020-02-01 17:33:02 +01:00
Marius 790d6ea849 handler: Allow method overriding in CORS requests 2019-11-15 21:06:29 +01:00
oliverpool e138fc3e9e filestore: Remove useless defer (#318)
Since the `file` is not used after the creation, its closing can be done without defer (and the err can be checked).
2019-10-23 18:51:36 +02:00
Ankit Pokhrel 898f3fe72a gcsstore: Remove unnecessary typecasts and unused mocks (#313) 2019-10-09 14:44:13 +02:00
Ankit Pokhrel 321acd1821 core, gcsstore: Update deprecated methods (#312) 2019-10-08 10:30:19 +02:00
Marius a045b8c4fc core: Allow numbers in mime types
See https://community.transloadit.com/t/tus-server-not-returning-the-correct-content-type-for-video-or-audio-file/15076
2019-10-07 11:28:13 +02:00
Marius 57be489b8b fixup! core: Allow writing data to storage after request is cancelled 2019-09-29 20:12:50 +02:00
Marius 65d2f1c2ef core: Allow writing data to storage after request is cancelled 2019-09-29 20:03:18 +02:00
Marius b73d634b47 core: Pass Upload interfaces to ConcatUploads 2019-09-19 12:14:25 +02:00