Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the last decoder allocation #142

Merged
merged 4 commits into from
Jul 28, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions zstd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ For now, a high speed (fastest) and medium-fast (default) compressor has been im
The "Fastest" compression ratio is roughly equivalent to zstd level 1.
The "Default" compression ration is roughly equivalent to zstd level 3 (default).

In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode. The compression ratio compared to stdlib is around level 3, but usually 3x as fast.
In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode.
The compression ratio compared to stdlib is around level 3, but usually 3x as fast.

Compared to cgo zstd, the speed is around level 3 (default), but compression slightly worse, between level 1&2.

Expand Down Expand Up @@ -217,7 +218,8 @@ silesia.tar zstd 3 211947520 66793301 1377 146.79

As part of the development process a *Snappy* -> *Zstandard* converter was also built.

This can convert a *framed* [Snappy Stream](https://godoc.org/github.com/golang/snappy#Writer) to a zstd stream. Note that a single block is not framed.
This can convert a *framed* [Snappy Stream](https://godoc.org/github.com/golang/snappy#Writer) to a zstd stream.
Note that a single block is not framed.

Conversion is done by converting the stream directly from Snappy without intermediate full decoding.
Therefore the compression ratio is much less than what can be done by a full decompression
Expand Down
8 changes: 4 additions & 4 deletions zstd/decoder.go
Original file line number Diff line number Diff line change
Expand Up @@ -281,17 +281,17 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
}
d.decoders <- block
frame.rawInput = nil
frame.bBuf = nil
d.frames <- frame
}()
frame.bBuf = input
if cap(dst) == 0 {
// Allocate 1MB by default if nothing is provided.
dst = make([]byte, 0, 1<<20)
}

// Allocation here:
br := byteBuf(input)
for {
err := frame.reset(&br)
err := frame.reset(&frame.bBuf)
if err == io.EOF {
return dst, nil
}
Expand All @@ -313,7 +313,7 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
if err != nil {
return dst, err
}
if len(br) == 0 {
if len(frame.bBuf) == 0 {
break
}
}
Expand Down
3 changes: 3 additions & 0 deletions zstd/framedec.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ type frameDec struct {

rawInput byteBuffer

// Byte buffer that can be reused for small input blocks.
bBuf byteBuf

// asyncRunning indicates whether the async routine processes input on 'decoding'.
asyncRunning bool
asyncRunningMu sync.Mutex
Expand Down