Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deflate: Faster/efficient low level encoders #105

Merged
merged 62 commits into from
Oct 10, 2019
Merged

deflate: Faster/efficient low level encoders #105

merged 62 commits into from
Oct 10, 2019

Commits on May 22, 2019

  1. deflate: Gap hashing and skip encoding 

    Refactor level 1+2 to skip more checks, but do intermediate hashes and extend matches backwards.

Same or better compression at same or better speed.
    @klauspost
    klauspost committed May 22, 2019
    Configuration menu
    Copy the full SHA
    b1cb937 View commit details
    Browse the repository at this point in the history

  2. Skip hash for the last part of blocks.

    @klauspost
    klauspost committed May 22, 2019
    Configuration menu
    Copy the full SHA
    844f232 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2019

  1. Tweak further.

    @klauspost
    klauspost committed May 23, 2019
    Configuration menu
    Copy the full SHA
    0e829b6 View commit details
    Browse the repository at this point in the history

  2. Optimize deflate bit writing. 

    Mostly avoiding bounds checks.
    @klauspost
    klauspost committed May 23, 2019
    Configuration menu
    Copy the full SHA
    7728ed2 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'deflate-optimize-bitwriting' into deflate-skip-lower-le… 

    …vels
    @klauspost
    klauspost committed May 23, 2019
    Configuration menu
    Copy the full SHA
    721e794 View commit details
    Browse the repository at this point in the history

  4. Avoid a lot of bounds checks for byte buffer.

    @klauspost
    klauspost committed May 23, 2019
    Configuration menu
    Copy the full SHA
    f1182b1 View commit details
    Browse the repository at this point in the history

  5. Don't check error value unless there are sideeffects.

    @klauspost
    klauspost committed May 23, 2019
    Configuration menu
    Copy the full SHA
    df7a94e View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'remotes/klauspost/deflate-optimize-bitw… 

    …riting' into deflate-skip-lower-levels
    @klauspost
    klauspost committed May 23, 2019
    Configuration menu
    Copy the full SHA
    a13e71f View commit details
    Browse the repository at this point in the history

  7. Avoid more bounds checks.

    @klauspost
    klauspost committed May 23, 2019
    Configuration menu
    Copy the full SHA
    06a10a9 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2019

  1. Remove more bounds checks

    @klauspost
    klauspost committed May 24, 2019
    Configuration menu
    Copy the full SHA
    c9f0cde View commit details
    Browse the repository at this point in the history

  2. Remove final bounds checks from indexer+writer.

    @klauspost
    klauspost committed May 24, 2019
    Configuration menu
    Copy the full SHA
    1e6f5a8 View commit details
    Browse the repository at this point in the history

  3. fmt it

    @klauspost
    klauspost committed May 24, 2019
    Configuration menu
    Copy the full SHA
    d81e961 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'deflate-optimize-bitwriting' into deflate-skip-lower-le… 

    …vels
    @klauspost
    klauspost committed May 24, 2019
    Configuration menu
    Copy the full SHA
    8be34c6 View commit details
    Browse the repository at this point in the history

Commits on May 26, 2019

  1. Merge branch 'master' into deflate-skip-lower-levels

    @klauspost
    klauspost committed May 26, 2019
    Configuration menu
    Copy the full SHA
    0fd006b View commit details
    Browse the repository at this point in the history

Commits on May 27, 2019

  1. Check 8 bytes/loop for matches.

    @klauspost
    klauspost committed May 27, 2019
    Configuration menu
    Copy the full SHA
    aec1bd8 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2019

  1. Try 4 byte match before doing 8 bytes.

    @klauspost
    klauspost committed May 28, 2019
    Configuration menu
    Copy the full SHA
    80fae85 View commit details
    Browse the repository at this point in the history

  2. fmt

    @klauspost
    klauspost committed May 28, 2019
    Configuration menu
    Copy the full SHA
    87fe04d View commit details
    Browse the repository at this point in the history

  3. Remove unused code.

    @klauspost
    klauspost committed May 28, 2019
    Configuration menu
    Copy the full SHA
    e5a97db View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2019

  1. Simplify bounds checks

    @klauspost
    klauspost committed Jun 2, 2019
    Configuration menu
    Copy the full SHA
    ac656c7 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2019

  1. Rename to fastEnc

    @klauspost
    klauspost committed Jun 10, 2019
    Configuration menu
    Copy the full SHA
    3d74e17 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into deflate-skip-lower-levels 

    # Conflicts:
#	flate/deflate.go
    @klauspost
    klauspost committed Jun 10, 2019
    Configuration menu
    Copy the full SHA
    7f5ca4f View commit details
    Browse the repository at this point in the history

  3. Use flat history

    @klauspost
    klauspost committed Jun 10, 2019
    Configuration menu
    Copy the full SHA
    4c2da43 View commit details
    Browse the repository at this point in the history

  4. Move Level 2 -> 1, add new level 2.

    @klauspost
    klauspost committed Jun 10, 2019
    Configuration menu
    Copy the full SHA
    bc78eb2 View commit details
    Browse the repository at this point in the history

  5. Add new level 4 with dual hashing.

    @klauspost
    klauspost committed Jun 10, 2019
    Configuration menu
    Copy the full SHA
    970835a View commit details
    Browse the repository at this point in the history

  6. Add dedicated level 5 compression.

    @klauspost
    klauspost committed Jun 10, 2019
    Configuration menu
    Copy the full SHA
    0a41eb6 View commit details
    Browse the repository at this point in the history

  7. Add a decent level 6

    @klauspost
    klauspost committed Jun 10, 2019
    Configuration menu
    Copy the full SHA
    a2a0ea3 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2019

  1. Move encoders to separate files. 

    Fix level 6 problem.
    @klauspost
    klauspost committed Jun 12, 2019
    Configuration menu
    Copy the full SHA
    6a6c230 View commit details
    Browse the repository at this point in the history

  2. Support dictionaries for levels 1-6.

    @klauspost
    klauspost committed Jun 12, 2019
    Configuration menu
    Copy the full SHA
    b34654e View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2019

  1. Do token indexing as they are added.

    @klauspost
    klauspost committed Jun 13, 2019
    Configuration menu
    Copy the full SHA
    05d211c View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2019

  1. Fix size estimation function.

    @klauspost
    klauspost committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    2736b3e View commit details
    Browse the repository at this point in the history

  2. Move bit flushing to separate function. At least 5-8 faster due to in… 

    …lining.
    @klauspost
    klauspost committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    c325151 View commit details
    Browse the repository at this point in the history

  3. Tweak AddMatch for inlining.

    @klauspost
    klauspost committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    d7af83e View commit details
    Browse the repository at this point in the history

  4. Re-add bounds check avoidance. Credit to @egonelbre :)

    @klauspost
    klauspost committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    f3292d2 View commit details
    Browse the repository at this point in the history

  5. Use check without shift for offsetCode - @egonelbre strikes again!

    @klauspost
    klauspost committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    9f407f5 View commit details
    Browse the repository at this point in the history

  6. Completely remove unneeded check. offset length can only be up to 15 … 

    …bits.
    @klauspost
    klauspost committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    3cc1639 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2019

  1. reorder fields and use extra tables (#125)

    @egonelbre @klauspost
    egonelbre authored and klauspost committed Jun 15, 2019
    Configuration menu
    Copy the full SHA
    1786eb3 View commit details
    Browse the repository at this point in the history

  2. Add test/benchmark for EstimatedBits

    @klauspost
    klauspost committed Jun 15, 2019
    Configuration menu
    Copy the full SHA
    7455284 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2019

  1. Add huffman table reuse.

    @klauspost
    klauspost committed Jun 20, 2019
    Configuration menu
    Copy the full SHA
    2caf33c View commit details
    Browse the repository at this point in the history

  2. Manually inline code writing.

    @klauspost
    klauspost committed Jun 20, 2019
    Configuration menu
    Copy the full SHA
    1259b8e View commit details
    Browse the repository at this point in the history

  3. Fix huffman code generation.

    @klauspost
    klauspost committed Jun 20, 2019
    Configuration menu
    Copy the full SHA
    ae21d9c View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2019

  1. Don't add maxbits for no-present values.

    @klauspost
    klauspost committed Jun 21, 2019
    Configuration menu
    Copy the full SHA
    09fbddc View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2019

  1. * Add bufferReset as constant 

    * Add inflate debug printing
    @klauspost
    klauspost committed Jun 25, 2019
    Configuration menu
    Copy the full SHA
    3a3bce5 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2019

  1. Fix token writing.

    @klauspost
    klauspost committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    818e7a3 View commit details
    Browse the repository at this point in the history

  2. Fix typo in level 6.

    @klauspost
    klauspost committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    40e8dba View commit details
    Browse the repository at this point in the history

  3. Remove table masks.

    @klauspost
    klauspost committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    a29415d View commit details
    Browse the repository at this point in the history

  4. Update bit writer tests files.

    @klauspost
    klauspost committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    6d9dac0 View commit details
    Browse the repository at this point in the history

  5. Remove assembler completely.

    @klauspost
    klauspost committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    9875bfb View commit details
    Browse the repository at this point in the history

  6. Fix buffer reset size.

    @klauspost
    klauspost committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    af86d6c View commit details
    Browse the repository at this point in the history

  7. Fix level 4 table reset.

    @klauspost
    klauspost committed Aug 5, 2019
    Configuration menu
    Copy the full SHA
    e15b88c View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2019

  1. Minor compression tweaks.

    @klauspost
    klauspost committed Aug 6, 2019
    Configuration menu
    Copy the full SHA
    9bb3e90 View commit details
    Browse the repository at this point in the history

  2. Tweak level 3+4.

    @klauspost
    klauspost committed Aug 6, 2019
    Configuration menu
    Copy the full SHA
    2843ac6 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2019

  1. Cleanup.

    @klauspost
    klauspost committed Aug 15, 2019
    Configuration menu
    Copy the full SHA
    fe5f2ab View commit details
    Browse the repository at this point in the history

  2. Emit EOB when asked for sync.

    @klauspost
    klauspost committed Aug 15, 2019
    Configuration menu
    Copy the full SHA
    9992353 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2019

  1. Don't limit match length searches. 

    Instead add a token emitter that can emit arbitrary lengths.
    @klauspost
    klauspost committed Aug 18, 2019
    Configuration menu
    Copy the full SHA
    682a962 View commit details
    Browse the repository at this point in the history

  2. Tweak skiplog.

    @klauspost
    klauspost committed Aug 18, 2019
    Configuration menu
    Copy the full SHA
    f3de74d View commit details
    Browse the repository at this point in the history

  3. Don't create historgram on no compression.

    @klauspost
    klauspost committed Aug 18, 2019
    Configuration menu
    Copy the full SHA
    e390b13 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2019

  1. Merge branch 'master' into deflate-skip-lower-levels

    @klauspost
    klauspost committed Aug 26, 2019
    Configuration menu
    Copy the full SHA
    fce35b4 View commit details
    Browse the repository at this point in the history

  2. Tweaks.

    @klauspost
    klauspost committed Aug 26, 2019
    Configuration menu
    Copy the full SHA
    7125ab1 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2019

  1. Merge branch 'master' into deflate-skip-lower-levels

    @klauspost
    klauspost committed Aug 29, 2019
    Configuration menu
    Copy the full SHA
    5b74cc8 View commit details
    Browse the repository at this point in the history

  2. Documentation.

    @klauspost
    klauspost committed Aug 29, 2019
    Configuration menu
    Copy the full SHA
    e038cca View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2019

  1. Merge branch 'master' into deflate-skip-lower-levels

    @klauspost
    klauspost committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    931f7f3 View commit details
    Browse the repository at this point in the history

  2. Update README.md

    @klauspost
    klauspost committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    d110fb0 View commit details
    Browse the repository at this point in the history