Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scheme option to bind to both HTTP and HTTPS #1215

Merged
merged 8 commits into from
Aug 1, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 23 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,33 +7,33 @@ fake-gcs-server provides an emulator for Google Cloud Storage API. It can be
used as a library in Go projects and/or as a standalone binary/Docker image.

The library is available inside the package
[``github.com/fsouza/fake-gcs-server/fakestorage``](https://pkg.go.dev/github.com/fsouza/fake-gcs-server/fakestorage?tab=doc)
[`github.com/fsouza/fake-gcs-server/fakestorage`](https://pkg.go.dev/github.com/fsouza/fake-gcs-server/fakestorage?tab=doc)
and can be used from within test suites in Go package. The emulator is
available as a binary that can be built manually, downloaded from the [releases
page](https://github.com/fsouza/fake-gcs-server/releases) or pulled from Docker
Hub ([``docker pull
fsouza/fake-gcs-server``](https://hub.docker.com/r/fsouza/fake-gcs-server)).
Hub ([`docker pull
fsouza/fake-gcs-server`](https://hub.docker.com/r/fsouza/fake-gcs-server)).

## Using the emulator in Docker

You can stub/mock Google Cloud Storage as a standalone server (like the datastore/pubsub emulators)
which is ideal for integration tests and/or tests in other languages you may want to run the
``fake-gcs-server`` inside a Docker container:
`fake-gcs-server` inside a Docker container:

```shell
docker run -d --name fake-gcs-server -p 4443:4443 fsouza/fake-gcs-server
```

### Preload data

In case you want to preload some data in ``fake-gcs-server`` just mount a
folder in the container at ``/data``:
In case you want to preload some data in `fake-gcs-server` just mount a
folder in the container at `/data`:

```shell
docker run -d --name fake-gcs-server -p 4443:4443 -v ${PWD}/examples/data:/data fsouza/fake-gcs-server
```

Where the content of ``${PWD}/examples/data`` is something like:
Where the content of `${PWD}/examples/data` is something like:

```
.
Expand All @@ -51,13 +51,14 @@ curl --insecure https://0.0.0.0:4443/storage/v1/b/sample-bucket/o
{"kind":"storage#objects","items":[{"kind":"storage#object","name":"some_file.txt","id":"sample-bucket/some_file.txt","bucket":"sample-bucket","size":"33"}],"prefixes":[]}
```

This will result in one bucket called ``sample-bucket`` containing one object called ``some_file.txt``.
This will result in one bucket called `sample-bucket` containing one object called `some_file.txt`.

### Running with HTTP

fake-gcs-server defaults to HTTPS, but it can also be used with HTTP. The flag
`-scheme` can be used to specify the protocol. For example, the previous
example could be changed to pass `-scheme http`:
`-scheme` can be used to specify the protocol.
The binding port will be `-port` (defaults to `4443`).
For example, the previous example could be changed to pass `-scheme http`:

```shell
docker run -d --name fake-gcs-server -p 4443:4443 -v ${PWD}/examples/data:/data fsouza/fake-gcs-server -scheme http
Expand All @@ -74,6 +75,16 @@ curl http://0.0.0.0:4443/storage/v1/b/sample-bucket/o
{"kind":"storage#objects","items":[{"kind":"storage#object","name":"some_file.txt","id":"sample-bucket/some_file.txt","bucket":"sample-bucket","size":"33"}],"prefixes":[]}
```

### Running with both HTTPS and HTTP

To start both HTTPS and HTTP servers, pass `-scheme both`.
HTTPS will bind to `-port` (defaults to `4443`) and HTTP will bind to `-port-http` (defaults to `8000`).
For example, the previous example could be changed to pass `-scheme both`:

```shell
docker run -d --name fake-gcs-server -p 4443:4443 -p 8000:8000 -v ${PWD}/examples/data:/data fsouza/fake-gcs-server -scheme both
```

### Using with signed URLs

It is possible to use fake-gcs-server with signed URLs, although with a few caveats:
Expand All @@ -97,11 +108,11 @@ docker run --rm fsouza/fake-gcs-server -help
## Client library examples

For examples using SDK from multiple languages, check out the
[``examples``](/examples/) directory.
[`examples`](/examples/) directory.

### Building the image locally

You may use ``docker build`` to build the image locally instead of pulling it
You may use `docker build` to build the image locally instead of pulling it
from Docker Hub:

```shell
Expand Down
23 changes: 16 additions & 7 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ type Config struct {
Seed string
Host string
Port uint
PortHTTP uint
CertificateLocation string
PrivateKeyLocation string

Expand Down Expand Up @@ -65,11 +66,12 @@ func Load(args []string) (Config, error) {
fs.StringVar(&cfg.fsRoot, "filesystem-root", "/storage", "filesystem root (required for the filesystem backend). folder will be created if it doesn't exist")
fs.StringVar(&cfg.publicHost, "public-host", "storage.googleapis.com", "Optional URL for public host")
fs.StringVar(&cfg.externalURL, "external-url", "", "optional external URL, returned in the Location header for uploads. Defaults to the address where the server is running")
fs.StringVar(&cfg.Scheme, "scheme", "https", "using http or https")
fs.StringVar(&cfg.Scheme, "scheme", "https", "using http or https or both")
fs.StringVar(&cfg.Host, "host", "0.0.0.0", "host to bind to")
fs.StringVar(&cfg.Seed, "data", "", "where to load data from (provided that the directory exists)")
fs.StringVar(&allowedCORSHeaders, "cors-headers", "", "comma separated list of headers to add to the CORS allowlist")
fs.UintVar(&cfg.Port, "port", 4443, "port to bind to")
fs.UintVar(&cfg.Port, "port", 4443, "port to bind https or http to, according to scheme, and for https if scheme is 'both'")
fs.UintVar(&cfg.PortHTTP, "port-http", 8000, "used only when scheme is 'both' as port to bind http to")
fs.StringVar(&cfg.event.pubsubProjectID, "event.pubsub-project-id", "", "project ID containing the pubsub topic")
fs.StringVar(&cfg.event.pubsubTopic, "event.pubsub-topic", "", "pubsub topic name to publish events on")
fs.StringVar(&cfg.event.bucket, "event.bucket", "", "if not empty, only objects in this bucket will generate trigger events")
Expand Down Expand Up @@ -111,12 +113,15 @@ func (c *Config) validate() error {
if c.backend == filesystemBackend && c.fsRoot == "" {
return fmt.Errorf("backend %q requires the filesystem-root to be defined", c.backend)
}
if c.Scheme != "http" && c.Scheme != "https" {
return fmt.Errorf(`invalid scheme %s, must be either "http"" or "https"`, c.Scheme)
if c.Scheme != "http" && c.Scheme != "https" && c.Scheme != "both" {
return fmt.Errorf(`invalid scheme %s, must be either "http"", "https" or "both"`, c.Scheme)
}
if c.Port > math.MaxUint16 {
return fmt.Errorf("port %d is too high, maximum value is %d", c.Port, math.MaxUint16)
}
if c.PortHTTP > math.MaxUint16 {
return fmt.Errorf("port-http %d is too high, maximum value is %d", c.PortHTTP, math.MaxUint16)
}

return c.event.validate()
}
Expand Down Expand Up @@ -148,7 +153,7 @@ func (c *EventConfig) validate() error {
return nil
}

func (c *Config) ToFakeGcsOptions() fakestorage.Options {
func (c *Config) ToFakeGcsOptions(scheme string) fakestorage.Options {
storageRoot := c.fsRoot
if c.backend == memoryBackend {
storageRoot = ""
Expand All @@ -173,14 +178,18 @@ func (c *Config) ToFakeGcsOptions() fakestorage.Options {
}
}
}
port := c.Port
if c.Scheme == "both" && scheme == "http" {
port = c.PortHTTP
}
logger := logrus.New()
logger.SetLevel(c.LogLevel)
opts := fakestorage.Options{
StorageRoot: storageRoot,
Seed: c.Seed,
Scheme: c.Scheme,
Scheme: scheme,
Host: c.Host,
Port: uint16(c.Port),
Port: uint16(port),
PublicHost: c.publicHost,
ExternalURL: c.externalURL,
AllowedCORSHeaders: c.allowedCORSHeaders,
Expand Down
28 changes: 25 additions & 3 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,9 @@ func TestLoadConfig(t *testing.T) {
"-cors-headers", "X-Goog-Meta-Uploader",
"-host", "127.0.0.1",
"-port", "443",
"-port-http", "80",
"-data", "/var/gcs",
"-scheme", "http",
"-scheme", "both",
"-event.pubsub-project-id", "test-project",
"-event.pubsub-topic", "gcs-events",
"-event.object-prefix", "uploads/",
Expand All @@ -50,7 +51,8 @@ func TestLoadConfig(t *testing.T) {
allowedCORSHeaders: []string{"X-Goog-Meta-Uploader"},
Host: "127.0.0.1",
Port: 443,
Scheme: "http",
PortHTTP: 80,
Scheme: "both",
event: EventConfig{
pubsubProjectID: "test-project",
pubsubTopic: "gcs-events",
Expand All @@ -72,6 +74,7 @@ func TestLoadConfig(t *testing.T) {
allowedCORSHeaders: nil,
Host: "0.0.0.0",
Port: 4443,
PortHTTP: 8000,
Scheme: "https",
event: EventConfig{
list: []string{"finalize"},
Expand All @@ -85,11 +88,26 @@ func TestLoadConfig(t *testing.T) {
args: []string{"-port", "not-a-number"},
expectErr: true,
},
{
name: "invalid port-http value type",
args: []string{"-port-http", "not-a-number"},
expectErr: true,
},
{
name: "invalid port value",
args: []string{"-port", "65536"},
expectErr: true,
},
{
name: "invalid port-http value",
args: []string{"-port-http", "65536"},
expectErr: true,
},
{
name: "invalid scheme value",
args: []string{"-scheme", "wrong-scheme-value"},
expectErr: true,
},
{
name: "invalid backend",
args: []string{"-backend", "in-memory"},
Expand Down Expand Up @@ -154,6 +172,7 @@ func TestToFakeGcsOptions(t *testing.T) {
externalURL: "https://myhost.example.com:8443",
Host: "0.0.0.0",
Port: 443,
Scheme: "https",
event: EventConfig{
pubsubProjectID: "test-project",
pubsubTopic: "gcs-events",
Expand All @@ -169,6 +188,7 @@ func TestToFakeGcsOptions(t *testing.T) {
ExternalURL: "https://myhost.example.com:8443",
Host: "0.0.0.0",
Port: 443,
Scheme: "https",
EventOptions: notification.EventManagerOptions{
ProjectID: "test-project",
TopicName: "gcs-events",
Expand All @@ -193,13 +213,15 @@ func TestToFakeGcsOptions(t *testing.T) {
externalURL: "https://myhost.example.com:8443",
Host: "0.0.0.0",
Port: 443,
Scheme: "https",
},
fakestorage.Options{
StorageRoot: "",
PublicHost: "127.0.0.1.nip.io:8443",
ExternalURL: "https://myhost.example.com:8443",
Host: "0.0.0.0",
Port: 443,
Scheme: "https",
NoListener: true,
},
},
Expand All @@ -209,7 +231,7 @@ func TestToFakeGcsOptions(t *testing.T) {
test := test
t.Run(test.name, func(t *testing.T) {
t.Parallel()
opts := test.config.ToFakeGcsOptions()
opts := test.config.ToFakeGcsOptions(test.config.Scheme)
ignWriter := cmpopts.IgnoreFields(fakestorage.Options{}, "Writer")
if diff := cmp.Diff(opts, test.expected, ignWriter); diff != "" {
t.Errorf("wrong set of options returned\nwant %#v\ngot %#v\ndiff: %v", test.expected, opts, diff)
Expand Down
83 changes: 56 additions & 27 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,27 +24,16 @@ import (
"github.com/sirupsen/logrus"
)

func main() {
cfg, err := config.Load(os.Args[1:])
if err == flag.ErrHelp {
return
}
if err != nil {
log.Fatal(err)
}

logger := logrus.New()
logger.SetLevel(cfg.LogLevel)

opts := cfg.ToFakeGcsOptions()
func createListener(logger *logrus.Logger, cfg *config.Config, scheme string) (net.Listener, *fakestorage.Options) {
opts := cfg.ToFakeGcsOptions(scheme)

addr := fmt.Sprintf("%s:%d", cfg.Host, cfg.Port)
addr := fmt.Sprintf("%s:%d", opts.Host, opts.Port)
listener, err := net.Listen("tcp", addr)
if err != nil {
log.Fatal(err)
raz-amir marked this conversation as resolved.
Show resolved Hide resolved
}

if cfg.Scheme == "https" {
if opts.Scheme == "https" {
var tlsConfig *tls.Config
if opts.CertificateLocation != "" && opts.PrivateKeyLocation != "" {
cert, err := tls.LoadX509KeyPair(opts.CertificateLocation, opts.PrivateKeyLocation)
Expand All @@ -62,26 +51,66 @@ func main() {
listener = tls.NewListener(listener, tlsConfig)
}

return listener, &opts
}

func startServer(logger *logrus.Logger, cfg *config.Config) {
type listenerAndOpts struct {
listener net.Listener
opts *fakestorage.Options
}

listenersAndOpts := make([]listenerAndOpts, 2)

if cfg.Scheme != "both" {
listener, opts := createListener(logger, cfg, cfg.Scheme)
listenersAndOpts[0] = listenerAndOpts{listener, opts}
} else {
listener, opts := createListener(logger, cfg, "http")
listenersAndOpts[0] = listenerAndOpts{listener, opts}
listener, opts = createListener(logger, cfg, "https")
listenersAndOpts[1] = listenerAndOpts{listener, opts}
}

addMimeTypes()

httpServer, err := fakestorage.NewServerWithOptions(opts)
httpServer, err := fakestorage.NewServerWithOptions(*listenersAndOpts[1].opts)
if err != nil {
logger.WithError(err).Fatal("couldn't start the server")
}

grpcServer := grpc.NewServerWithBackend(httpServer.Backend())
go func() {
http.Serve(listener, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.ProtoMajor == 2 && strings.HasPrefix(
r.Header.Get("Content-Type"), "application/grpc") {
grpcServer.ServeHTTP(w, r)
} else {
httpServer.HTTPHandler().ServeHTTP(w, r)
}
}))
}()

logger.Infof("server started at %s://%s:%d", cfg.Scheme, cfg.Host, cfg.Port)
for _, listenerAndOpts := range listenersAndOpts {
go func(listener net.Listener) {
http.Serve(listener, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.ProtoMajor == 2 && strings.HasPrefix(
r.Header.Get("Content-Type"), "application/grpc") {
grpcServer.ServeHTTP(w, r)
} else {
httpServer.HTTPHandler().ServeHTTP(w, r)
}
}))
}(listenerAndOpts.listener)

logger.Infof("server started at %s://%s:%d",
listenerAndOpts.opts.Scheme, listenerAndOpts.opts.Host, listenerAndOpts.opts.Port)
}
}

func main() {
cfg, err := config.Load(os.Args[1:])
if err == flag.ErrHelp {
return
}
if err != nil {
log.Fatal(err)
}

logger := logrus.New()
logger.SetLevel(cfg.LogLevel)

startServer(logger, &cfg)

ch := make(chan os.Signal, 1)
signal.Notify(ch, os.Interrupt, syscall.SIGTERM)
Expand Down
Loading