Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add vparquet4 encoding #3368

Merged
merged 38 commits into from
May 7, 2024
Merged

Add vparquet4 encoding #3368

merged 38 commits into from
May 7, 2024

Commits on Feb 8, 2024

  1. Copy vparquet3 to vparquet4 folder

    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    444a6fc View commit details
    Browse the repository at this point in the history

  2. Create initial vParquet4 encoding 

    At this point vParquet3 and vParquet4 are identical
    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    31140a4 View commit details
    Browse the repository at this point in the history

  3. vParquet4: handle unsupported / dropped attributes (grafana#3123) 

    * Count dropped attributes and store them in separate column

* Remove current support for arrays and kv-lists
    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    9775501 View commit details
    Browse the repository at this point in the history

  4. Store per-trace service statistics in vparquet4 blocks (grafana#2941) 

    * Store per-trace service statistics in vparquet4 blocks

* Update pkg/traceql/combine.go

* Update modules/frontend/search_progress.go

* Update pkg/traceql/engine.go

* Pre-allocate finalSpanset.ServiceStats

* drop MaxServiceStats limit

---------

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
Co-authored-by: A. Stoewer <adrian@stoewer.me>
    @andreasgerstmayr @stoewer
    andreasgerstmayr and stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    41dc8eb View commit details
    Browse the repository at this point in the history

  5. Add support for attributes with array values (grafana#3221) 

    * Fix skipping of dropped attributes

The attribute dropping introduced in grafana#3123 wans't implemented
correctly

* Implement single values as arrays and introduce type column

* Add support for array attributes

* Tests more array values and fix attribute conversion

* Replace individual pointer helper functions

* Use a combination of snappy and delta for ValueType

* Vendor module google/go-cmp/cmp/cmpopts

* Use all types of array attributes in TestTraceToParquet

* Improve TestFieldsAreCleared
    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    30e79e1 View commit details
    Browse the repository at this point in the history

  6. links and events schema changes (grafana#3163) 

    * links and events schema changes

* update tests

* update test

* lint

* fixed tests after rebase

* lint

* addressed comments

* lint version

* reduce span to just start time for events conversion
    @ie-pham @stoewer
    ie-pham authored and stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    794fb93 View commit details
    Browse the repository at this point in the history

  7. Add attributes to instrumentation scope (grafana#3322) 

    * Add attributes to instrumentation scope

Attributes are not filled as they are not supported in current proto

* Populate instrumentation scope in test data

* Use delta encoding for dropped attributes count columns

* More consistent naming
    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    23a260e View commit details
    Browse the repository at this point in the history

  8. Fix typo in vParquet4 event name enconding (grafana#3336)

    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    a2e10bb View commit details
    Browse the repository at this point in the history

  9. Precalculate and reuse the vParquet4 schema before opening blocks

    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    9024d10 View commit details
    Browse the repository at this point in the history

  10. Address linter warnings

    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    06521ed View commit details
    Browse the repository at this point in the history

  11. Convert block in vparquet4/test-data directory

    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    1f46601 View commit details
    Browse the repository at this point in the history

  12. CHANGELOG.md

    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    580d50f View commit details
    Browse the repository at this point in the history

  13. Update test cases in BenchmarkBackendBlockTraceQL 

    The test use values are tuned to current blocks from ops in
order to have correct matching / non-matching queries
    @stoewer
    stoewer committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    16432c9 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. Unsupported attribute values are no longer dropped

    @stoewer
    stoewer committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    156ba47 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into add-vparquet4-encoding

    @stoewer
    stoewer committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    1f6708b View commit details
    Browse the repository at this point in the history

  3. Backport: Update traceql metrics to use the trace-level timestamp col… 

    …umns
    @stoewer
    stoewer committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    8ec97c9 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. Add ServiceStats data to existing TraceQL engine tests (#2) 

    Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
    @andreasgerstmayr
    andreasgerstmayr authored Feb 16, 2024
    Configuration menu
    Copy the full SHA
    91b1236 View commit details
    Browse the repository at this point in the history

  2. Fix combining service stats by using max() (#1) 

    * Fix combining service stats by using max()

* update comment

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
    @andreasgerstmayr
    andreasgerstmayr authored Feb 16, 2024
    Configuration menu
    Copy the full SHA
    9393edf View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. Recalculate ServiceStats in combiner by moving it to separate function ( 

    #3)

* Recalculate ServiceStats in combiner by moving it to assignNestedSetModelBounds()
* add additional tests

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
    @andreasgerstmayr
    andreasgerstmayr authored Apr 10, 2024
    Configuration menu
    Copy the full SHA
    d79cea6 View commit details
    Browse the repository at this point in the history

  2. Move serviceStats iterator to second pass (#4) 

    * Move serviceStats iterator to second pass

* fix nil map assignment

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
    @andreasgerstmayr
    andreasgerstmayr authored Apr 10, 2024
    Configuration menu
    Copy the full SHA
    9ab5514 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. Merge remote-tracking branch 'upstream/main' into add-vparquet4-encoding

    @stoewer
    stoewer committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    9bef5e8 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. Backport: Better trace ID sharding (grafana#3399)

    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    c89e606 View commit details
    Browse the repository at this point in the history

  2. Backport: Add string interning to SyncIterator (grafana#3411)

    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    9b81152 View commit details
    Browse the repository at this point in the history

  3. Backport: Traceql better pooling (grafana#3432)

    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    66a54e8 View commit details
    Browse the repository at this point in the history

  4. Backport: New (unsafe) query hints (grafana#3396)

    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    ff97bb8 View commit details
    Browse the repository at this point in the history

  5. Backport: Redo 8-byte trace ID sharding to reduce number of row groups 

     read (grafana#3499)
    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    dc10143 View commit details
    Browse the repository at this point in the history

  6. Backport: TraceQL nested set intrinsics (grafana#3497)

    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    0587418 View commit details
    Browse the repository at this point in the history

  7. Backport: Parquetquery iterator less pooling (grafana#3508)

    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    ad936e3 View commit details
    Browse the repository at this point in the history

  8. Remove searchProgress handler

    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    665b765 View commit details
    Browse the repository at this point in the history

  9. Fix tests: Unset service stats in TraceSearchMeta

    @stoewer
    stoewer committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    3627f3d View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. Backport: Fix pool leak when iter exhausted and async pages for bette… 

    …r throughput (grafana#3579)
    @stoewer
    stoewer committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    0c139e9 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2024

  1. make ServiceStats an intrinsic (#5) 

    * make ServiceStats an intrinsic

* skip allocating ServiceStats in rebatchIterator if already allocated

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
    @andreasgerstmayr
    andreasgerstmayr authored Apr 28, 2024
    Configuration menu
    Copy the full SHA
    19dbeab View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Replace ValueType column with IsArray column

    @stoewer
    stoewer committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    2ef85b3 View commit details
    Browse the repository at this point in the history

  2. Skip service stats map allocation when none are present

    @stoewer
    stoewer committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    d290290 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. Fix performance regression in existing parquet encodings 

    The regression was introduced by making ServiceStats an intrinsic
    @stoewer
    stoewer committed May 3, 2024
    Configuration menu
    Copy the full SHA
    a8abcb5 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Merge remote-tracking branch 'upstream/main' into add-vparquet4-encoding

    @stoewer
    stoewer committed May 6, 2024
    Configuration menu
    Copy the full SHA
    8108e03 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into add-vparquet4-encoding

    @stoewer
    stoewer committed May 6, 2024
    Configuration menu
    Copy the full SHA
    393b2a7 View commit details
    Browse the repository at this point in the history

  3. Use same queries in BackendBlockTraceQL 

    Use the same queries in vp3 and vp4
    @stoewer
    stoewer committed May 6, 2024
    Configuration menu
    Copy the full SHA
    8c71ae5 View commit details
    Browse the repository at this point in the history