Reference server implementation in Go of tus: the open protocol for resumable file uploads
Go to file
oliverpool 74fca8c6c3 Squashed commit of the following:
commit d214ad5c92073cb754c70ab73e6cf229cba01c72
Author: Marius <maerious@gmail.com>
Date:   Tue Feb 28 20:38:47 2017 +0100

    Minor code and comment cleanups

commit 2826227404296d98a2d83519efaa754cc07a47b1
Merge: db47c89 1a58d6e
Author: Marius <maerious@gmail.com>
Date:   Tue Feb 28 20:13:15 2017 +0100

    Merge branch 'metrics_race_condition' of https://github.com/oliverpool/tusd into oliverpool-metrics_race_condition

commit 1a58d6e084
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Tue Feb 28 19:14:21 2017 +0100

    prometheus metrics typo

commit db47c8976489917a8afc53878335ab66d73fe60d
Merge: 168942b 12054be
Author: Marius <maerious@gmail.com>
Date:   Tue Feb 28 17:36:47 2017 +0100

    Merge branch 'metrics_race_condition' of https://github.com/oliverpool/tusd into oliverpool-metrics_race_condition

commit 12054be4e7
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Tue Feb 28 12:07:43 2017 +0100

    add go vet on travis

commit 16e95d2d91
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Tue Feb 28 11:58:58 2017 +0100

    add one go vet on appveyor

commit 13bc64517b
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Tue Feb 28 11:57:56 2017 +0100

    use even less pointers

commit db7130f18a
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Mon Feb 27 22:12:45 2017 +0100

    use pointer only when needed

commit 647f7390c6
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Thu Feb 23 16:29:01 2017 +0100

    Prevent lock malfunction

    literal copies lock value from metrics

commit 1ce196ed35
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 22 12:22:19 2017 +0100

    handle read timeout simplification in metrics

commit 338017c9f4
Merge: c378bc9 45a9278
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 22 12:21:42 2017 +0100

    Merge branch 'master' into metrics_race_condition

commit c378bc9798
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 22 11:51:56 2017 +0100

    consistent naming

commit e2b0050b8d
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 22 11:44:05 2017 +0100

    Use a simplfied error structure for error counting

commit 3da4095c08
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Mon Feb 20 16:38:27 2017 +0100

    split ptr retrieval and incrementation

commit ebd6873cbb
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Sun Feb 5 22:01:34 2017 +0100

    Useless initialization

commit 691224cdbf
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Thu Feb 2 08:25:55 2017 +0100

    increment the pointer

commit 3d9395e1df
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Thu Feb 2 08:25:42 2017 +0100

    Simplify lock logic

commit d7a4619080
Author: oliverpool <oliverpool@hotmail.fr>
Date:   Wed Feb 1 08:37:46 2017 +0100

    Prevent race condition on errorstotal increment

    The HTTPError interface is used to have detailed metrics (by code as
    well)
2017-02-28 20:40:44 +01:00
.hooks Add experimental post-receive hook 2017-01-26 22:34:14 +01:00
.infra Upgrade Frey to v0.3.31 /cc @tersmitten (#113) 2017-02-17 14:48:54 +01:00
.scripts Squashed commit of the following: 2017-02-28 20:40:44 +01:00
cmd/tusd Correct constant for post-receive hook name 2017-01-26 23:16:22 +01:00
consullocker Document functions for handling StoreComposers 2016-03-11 20:17:43 +01:00
docs Add documentation about post-receive event 2017-02-21 23:24:28 +01:00
filestore Return detailed error when directory is missing 2017-01-26 21:46:06 +01:00
limitedstore Format code in limitedstore 2016-10-13 16:46:03 +02:00
memorylocker Make minor improvements to memorylocker internals 2017-02-06 15:10:43 +01:00
prometheuscollector Squashed commit of the following: 2017-02-28 20:40:44 +01:00
s3store Define own, minimal S3API interface 2016-12-20 17:13:02 +01:00
uid Correct linting issue and misspellings 2016-09-27 22:10:16 +02:00
vendor Remove Prometheus client from vendors dependencies 2017-02-28 17:17:13 +01:00
.gitignore Improved .gitignore for Frey using Docker (#109) 2017-02-16 09:51:48 +01:00
.travis.yml Include Go 1.8 release into tested versions 2017-02-21 23:27:23 +01:00
LICENSE.txt Update year 2017-01-19 21:20:12 +01:00
README.md Add entry about authentication to FAQ 2017-01-18 11:22:47 +01:00
appveyor.yml Squashed commit of the following: 2017-02-28 20:40:44 +01:00
composer.go Document functions for handling StoreComposers 2016-03-11 20:17:43 +01:00
composer.mgo Document functions for handling StoreComposers 2016-03-11 20:17:43 +01:00
composer_test.go Use memorylocker in example for composer 2016-09-30 01:14:52 +02:00
concat_test.go Correct parsing Upload-Concat for final uplaods 2017-01-31 16:58:31 +01:00
config.go Add documentation for NotifyUploadProgress 2017-02-21 23:17:07 +01:00
config_test.go Add tests for Config structure 2016-03-11 20:46:34 +01:00
cors_test.go Refactor remaining tests to subtest style 2016-10-13 18:08:34 +02:00
datastore.go Correct linting issue and misspellings 2016-09-27 22:10:16 +02:00
doc.go Add documentation about using tusd 2016-03-04 22:13:13 +01:00
get_test.go Add test for accessing unimplemented methods 2016-10-13 18:48:46 +02:00
handler.go Embed UnroutedHandler into Handler 2016-03-12 22:01:12 +01:00
handler_mock_test.go Add test for FinisherDataStore 2016-10-13 18:18:18 +02:00
head_test.go Add tests for locking in various situations 2016-10-13 18:33:33 +02:00
log.go Add proper, formatted logging 2016-09-23 21:21:38 +02:00
metrics.go Squashed commit of the following: 2017-02-28 20:40:44 +01:00
options_test.go Refactor remaining tests to subtest style 2016-10-13 18:08:34 +02:00
patch_test.go Include channel testing in goroutine fore NotifyUploadProgress test 2017-02-21 23:33:59 +01:00
post_test.go Correct parsing Upload-Concat for final uplaods 2017-01-31 16:58:31 +01:00
subtest_go17_test.go Enable subtests for environments prior to Go 1.7 2016-10-13 12:29:13 +02:00
subtest_test.go Improve output of subtests when emulated 2016-10-13 12:59:11 +02:00
terminate_test.go Add test for accessing unimplemented methods 2016-10-13 18:48:46 +02:00
unrouted_handler.go Squashed commit of the following: 2017-02-28 20:40:44 +01:00
utils_test.go Add comments to test utilities 2016-10-14 11:52:51 +02:00

README.md

tusd

Tus logo

tus is a protocol based on HTTP for resumable file uploads. Resumable means that an upload can be interrupted at any moment and can be resumed without re-uploading the previous data again. An interruption may happen willingly, if the user wants to pause, or by accident in case of an network issue or server outage.

tusd is the official reference implementation of the tus resumable upload protocol. The protocol specifies a flexible method to upload files to remote servers using HTTP. The special feature is the ability to pause and resume uploads at any moment allowing to continue seamlessly after e.g. network interruptions.

Protocol version: 1.0.0

Getting started

You can download ready-to-use packages including binaries for OS X, Linux and Windows in various formats of the latest release.

Compile from source

Requirements:

  • Go (1.5 or newer)

Running tusd from source:

Clone the git repository and cd into it.

git clone git@github.com:tus/tusd.git
cd tusd

Now you can run tusd:

go run cmd/tusd/main.go

Using tusd manually

Besides from running tusd using the provided binary, you can embed it into your own Go program:

package main

import (
	"fmt"
	"net/http"

	"github.com/tus/tusd"
	"github.com/tus/tusd/filestore"
)

func main() {
	// Create a new FileStore instance which is responsible for
	// storing the uploaded file on disk in the specified directory.
	// This path _must_ exist before tusd will store uploads in it.
	// If you want to save them on a different medium, for example
	// a remote FTP server, you can implement your own storage backend
	// by implementing the tusd.DataStore interface.
	store := filestore.FileStore{
		Path: "./uploads",
	}

	// A storage backend for tusd may consist of multiple different parts which
	// handle upload creation, locking, termination and so on. The composer is a
	// place where all those seperated pieces are joined together. In this example
	// we only use the file store but you may plug in multiple.
	composer := tusd.NewStoreComposer()
	store.UseIn(composer)

	// Create a new HTTP handler for the tusd server by providing a configuration.
	// The StoreComposer property must be set to allow the handler to function.
	handler, err := tusd.NewHandler(tusd.Config{
		BasePath:      "/files/",
		StoreComposer: composer,
	})
	if err != nil {
		panic(fmt.Errorf("Unable to create handler: %s", err))
	}

	// Right now, nothing has happened since we need to start the HTTP server on
	// our own. In the end, tusd will start listening on and accept request at
	// http://localhost:8080/files
	http.Handle("/files/", http.StripPrefix("/files/", handler))
	err = http.ListenAndServe(":8080", nil)
	if err != nil {
		panic(fmt.Errorf("Unable to listen: %s", err))
	}
}

Please consult the online documentation for more details about tusd's APIs and its sub-packages.

Implementing own storages

The tusd server is built to be as flexible as possible and to allow the use of different upload storage mechanisms. By default the tusd binary includes filestore which will save every upload to a specific directory on disk.

If you have different requirements, you can build your own storage backend which will save the files to S3, a remote FTP server or similar. Doing so is as simple as implementing the tusd.DataStore interface and using the new struct in the configuration object. Please consult the documentation about detailed information about the required methods.

Packages

This repository does not only contain the HTTP server's code but also other useful tools:

  • s3store: A storage backend using AWS S3
  • filestore: A storage backend using the local file system
  • memorylocker: An in-memory locker for handling concurrent uploads
  • consullocker: A locker using the distributed Consul service
  • limitedstore: A storage wrapper limiting the total used space for uploads

Running the testsuite

Build Status Build status

go test -v ./...

FAQ

How can I access tusd using HTTPS?

The tusd binary, once executed, listens on the provided port for only non-encrypted HTTP requests and does not accept HTTPS connections. This decision has been made to limit the functionality inside this repository which has to be developed, tested and maintained. If you want to send requests to tusd in a secure fashion - what we absolutely encourage, we recommend you to utilize a reverse proxy in front of tusd which accepts incoming HTTPS connections and forwards them to tusd using plain HTTP. More information about this topic, including sample configurations for Nginx and Apache, can be found in issue #86.

Can I run tusd behind a reverse proxy?

Yes, it is absolutely possible to do so. Firstly, you should execute the tusd binary using the -behind-proxy flag indicating it to pay attention to special headers which are only relevent when used in conjunction with a proxy. Furthermore, there are addtional details which should be kept in mind, depending on the used software:

  • Disable request buffering. Nginx, for example, reads the entire incoming HTTP request, including its body, before sending it to the backend, by default. This behavior defeats the purpose of resumability where an upload is processed while it's being transfered. Therefore, such as feature should be disabled.

  • Adjust maximum request size. Some proxies have default values for how big a request may be in order to protect your services. Be sure to check these settings to match the requirements of your application.

  • Forward hostname and scheme. If the proxy rewrites the request URL, the tusd server does not know the original URL which was used to reach the proxy. This behavior can lead to situations, where tusd returns a redirect to a URL which can not be reached by the client. To avoid this confusion, you can explicitly tell tusd which hostname and scheme to use by supplying the X-Forwarded-Host and X-Forwarded-Proto headers.

Explicit examples for the above points can be found in the Nginx configuration which is used to power the master.tus.io instace.

Can I run custom verification/authentication checks before an upload begins?

Yes, this is made possible by the hook system inside the tusd binary. It enables custom routines to be executed when certain events occurs, such as a new upload being created which can be handled by the pre-create hook. Inside the corresponding hook file, you can run your own validations against the provided upload metadata to determine whether the action is actually allowed or should be rejected by tusd. Please have a look at the corresponding documentation for a more detailed explanation.

License

This project is licensed under the MIT license, see LICENSE.txt.