Simplify unit tests runner

[pracucci]

Currently, unit tests are run via tools/test script. This script looks overcomplicated for what we need right now and the following command should be enough:

go test -tags netgo -timeout 20m -race -covermode=atomic -count=1 ./cmd/... ./pkg/... ./tools/...

However, if I run go test ./pkg/... some ring KV tests for etcd fail while if I run go test on each individual package (like tools/test) does it works. I spent some time investigating and I haven't understood the root cause yet.

Below you can find the issues I've seen.

Panic

panic: invalid Go type int for field go_etcd_io.etcd.etcdserver.etcdserverpb.loggablePutRequest.value_size

goroutine 419 [running]:
google.golang.org/protobuf/internal/impl.newSingularConverter(0x21606e0, 0x1d9d2e0, 0x21605c0, 0xc0005420e0, 0x214dc01, 0xc000a89700)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/convert.go:143 +0xf17
google.golang.org/protobuf/internal/impl.NewConverter(0x21606e0, 0x1d9d2e0, 0x21605c0, 0xc0005420e0, 0x203000, 0xc000b38240)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/convert.go:60 +0xdc
google.golang.org/protobuf/internal/impl.fieldInfoForScalar(0x21605c0, 0xc0005420e0, 0x1d8b86f, 0x9, 0x0, 0x0, 0x21606e0, 0x1d9d2e0, 0x1d8b87a, 0x2e, ...)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/message_reflect_field.go:234 +0x115
google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc(0xc00022fa40, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xc000b38150, 0xc000b38180, 0xc000b381b0, 0xc000b381e0)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go:67 +0x97d
google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs(0xc00022fa40, 0x21606e0, 0x1efd080, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xc000b38150, 0xc000b38180, 0xc000b381b0, ...)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go:36 +0x65
google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce(0xc00022fa40)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/message.go:90 +0x192
google.golang.org/protobuf/internal/impl.(*MessageInfo).init(...)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/message.go:72
google.golang.org/protobuf/internal/impl.(*messageReflectWrapper).Has(0xc0004f8ff0, 0x21605c0, 0xc000542000, 0xc000542000)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/message_reflect_gen.go:185 +0xf9
github.com/golang/protobuf/proto.(*textWriter).writeMessage(0xc000b38120, 0x215a8e0, 0xc0004f8ff0, 0x0, 0x0)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/github.com/golang/protobuf/proto/text_encode.go:278 +0x935
github.com/golang/protobuf/proto.(*TextMarshaler).marshal(0x283c562, 0x213e7e0, 0xc000b38090, 0x0, 0x0, 0xc0004f8de0, 0x0, 0x1)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/github.com/golang/protobuf/proto/text_encode.go:86 +0x190
github.com/golang/protobuf/proto.(*TextMarshaler).Text(...)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/github.com/golang/protobuf/proto/text_encode.go:44
github.com/golang/protobuf/proto.CompactTextString(...)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/github.com/golang/protobuf/proto/text_encode.go:106
go.etcd.io/etcd/etcdserver/etcdserverpb.(*loggablePutRequest).String(...)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal_stringer.go:182
go.etcd.io/etcd/etcdserver/etcdserverpb.(*requestOpStringer).String(0xc000ace8f8, 0x0, 0x0)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal_stringer.go:124 +0x21d
go.etcd.io/etcd/etcdserver/etcdserverpb.(*txnRequestStringer).String(0xc000acea00, 0xc0004927a0, 0x17)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal_stringer.go:98 +0x45b
go.etcd.io/etcd/etcdserver/etcdserverpb.(*InternalRaftStringer).String(0xc00048c418, 0x26c299fd0, 0x28b1ca0)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal_stringer.go:67 +0x125
go.etcd.io/etcd/etcdserver.warnOfExpensiveGenericRequest(0xc0004d5c80, 0xbfec5c450c3c85c8, 0x26c299fd0, 0x28b1ca0, 0x2120900, 0xc00048c418, 0x0, 0x0, 0xc00012ecd0, 0x7, ...)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/util.go:168 +0x488
go.etcd.io/etcd/etcdserver.warnOfExpensiveRequest(0xc0004d5c80, 0xbfec5c450c3c85c8, 0x26c299fd0, 0x28b1ca0, 0x2120900, 0xc00048c418, 0x213e5a0, 0xc00053d530, 0x0, 0x0)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/util.go:111 +0xcc
go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply.func1(0xc0005d72a0, 0xc000530540, 0xc00053d260, 0xbfec5c450c3c85c8, 0x26c299fd0, 0x28b1ca0)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/apply.go:118 +0x110
go.etcd.io/etcd/etcdserver.(*applierV3backend).Apply(0xc0005d72a0, 0xc000530540, 0xc00053d260)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/apply.go:179 +0x305
go.etcd.io/etcd/etcdserver.(*authApplierV3).Apply(0xc0014d7630, 0xc000530540, 0x0)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/apply_auth.go:60 +0xd2
go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntryNormal(0xc000431080, 0xc000acf4c0)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/server.go:2223 +0x1f8
go.etcd.io/etcd/etcdserver.(*EtcdServer).apply(0xc000431080, 0xc00099e9d8, 0x1, 0x1, 0xc0004c7f40, 0x1ffb690, 0xc000057000, 0xc000acf628)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/server.go:2137 +0x599
go.etcd.io/etcd/etcdserver.(*EtcdServer).applyEntries(0xc000431080, 0xc0004c7f40, 0xc000491400)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/server.go:1389 +0xd7
go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll(0xc000431080, 0xc0004c7f40, 0xc000491400)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/server.go:1113 +0x88
go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8(0x2143e20, 0xc00027ef00)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/etcdserver/server.go:1058 +0x3c
go.etcd.io/etcd/pkg/schedule.(*fifo).run(0xc000ac0840)
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/pkg/schedule/schedule.go:157 +0xf3
created by go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler
	/Users/marco/workspace/src/github.com/cortexproject/cortex/vendor/go.etcd.io/etcd/pkg/schedule/schedule.go:70 +0x13b
FAIL	github.com/cortexproject/cortex/pkg/ring/kv	11.330s

Could be related to this recent change in etcd: etcd-io/etcd#12000

Flaky TestWatchPrefix

--- FAIL: TestWatchPrefix (4.92s)
    --- FAIL: TestWatchPrefix/etcd (3.80s)
        kv_test.go:228: key test/79 has incorrect value 0
        kv_test.go:228: key test/80 has incorrect value 0
        kv_test.go:228: key test/81 has incorrect value 0
        kv_test.go:228: key test/82 has incorrect value 0
        kv_test.go:228: key test/83 has incorrect value 0
        kv_test.go:228: key test/84 has incorrect value 0
        kv_test.go:228: key test/85 has incorrect value 0
        kv_test.go:228: key test/86 has incorrect value 0
        kv_test.go:228: key test/87 has incorrect value 0
        kv_test.go:228: key test/88 has incorrect value 0
        kv_test.go:228: key test/89 has incorrect value 0
        kv_test.go:228: key test/90 has incorrect value 0
        kv_test.go:228: key test/91 has incorrect value 0
        kv_test.go:228: key test/92 has incorrect value 0
        kv_test.go:228: key test/93 has incorrect value 0
        kv_test.go:228: key test/94 has incorrect value 0
        kv_test.go:228: key test/95 has incorrect value 0
        kv_test.go:228: key test/96 has incorrect value 0
        kv_test.go:228: key test/97 has incorrect value 0
        kv_test.go:228: key test/98 has incorrect value 0
        kv_test.go:228: key test/99 has incorrect value 0

Could be caused by totalTestTimeout not large enough?

[pstibrany]

I'd suggest using go test ./... (plus other parameters), which would also discover integration tests without correct build tags:

go test -tags netgo -timeout 20m -race -covermode=atomic -count=1 ./...

[pstibrany]

Could be caused by totalTestTimeout not large enough?

go test compiles tests for each package separately, and runs them with the -timeout, so this timeout is per package.

[pstibrany]

Could be caused by totalTestTimeout not large enough?

go test compiles tests for each package separately, and runs them with the -timeout, so this timeout is per package.

I guess you mean totalTestTimeout from kv_test.go :) Sorry for confusion.

[pracucci]

No, not closed by #3767. Only the change to totalTestTimeout has been done so far.

[pstibrany]

No, not closed by #3767. Only the change to totalTestTimeout has been done so far.

Correct. Sorry about that.