v2.4.0

Breaking Changes

• Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• Deprecating vParquet v1 #3377 (@ie-pham)
  vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
• Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
  Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Changes

• [CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
• [CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
• [CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
• [CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
• [CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• [CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
• [CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

• [FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
• [FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
• [FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
  BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Enhancements

• [ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
• [ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
• [ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
• [ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
• [ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
• [ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
• [ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
• [ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
• [ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
• [ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
• [ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
• [ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
• [ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
• [ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
• [ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
• [ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
• [ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
• [ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
• [ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303#3303 (@zalegrala)
• [ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
• [ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
• [ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
• [ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)
• [ENHANCEMENT] Config: Adds query-frontend.log-query-request-headers to enable logging of request headers in query logs. #3383 (@jmichalek132)

Bugfixes

• [BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
• [BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
• [BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
• [BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
• [BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
• [BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
• [BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
• [BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
• [BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
• [BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
• [BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
• [BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
• [BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
• [BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
• [BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)
• [BUGFIX] Fix some instances where spanmetrics histograms could be inconsistent #3412 (@mdisibio)

v2.4.0-rc.0

Breaking Changes

• Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• Deprecating vParquet v1 #3377 (@ie-pham)
  vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
• Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
  Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Changes

• [CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
• [CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
• [CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
• [CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
• [CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• [CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
• [CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

• [FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
• [FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
• [FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
  BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Enhancements

• [ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
• [ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
• [ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
• [ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
• [ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
• [ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
• [ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
• [ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
• [ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
• [ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
• [ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
• [ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
• [ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
• [ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
• [ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
• [ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
• [ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
• [ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
• [ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303#3303 (@zalegrala)
• [ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
• [ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
• [ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
• [ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)

Bugfixes

• [BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
• [BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
• [BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
• [BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
• [BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
• [BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
• [BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
• [BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
• [BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
• [BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
• [BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
• [BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
• [BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
• [BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
• [BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)

v2.3.1

Bugfixes

• [BUGFIX] Include statusMessage intrinsic attribute in tag search. #3084 (@rcrowe)
• [BUGFIX] Fix compactor ignore configured S3 headers #3149 (@Batkilin)
• [BUGFIX] Readd session token to s3 credentials. #3144 (@farodin91)

v2.3.0

Breaking Changes

• Removed deprecated config option: distributor.log_received_traces. Use the distributor.log_received_spans config block instead.
• Removed frontend metrics tempo_query_frontend_queries_total{op="searchtags|metrics"}

Changes

• [CHANGE] Update Go to 1.21 #2486 (@zalegrala)
• [CHANGE] Moved the tempo_ingester_traces_created_total metric to be incremented when a trace is cut to the wal #2884 (@joe-elliott)
• [CHANGE] Upgrade from deprecated azure-storage-blob-go SDK to azure-sdk-for-go #2835 (@LasseHels)
• [CHANGE] Metrics summary API validate the requested time range #2902 (@mdisibio)
• [CHANGE] Restructure Azure backends into versioned backends. Introduce use_v2_sdk config option for switching. #2952 (@zalegrala)
  v1: azure-storage-blob-go original (now deprecated) SDK
  v2: azure-sdk-for-go
• [CHANGE] Adjust trace size estimation to better honor row group size settings. #3038 (@joe-elliott)
• [CHANGE] Update alpine image version to 3.18. #3046 (@joe-elliott)
• [CHANGE] Overrides module refactor #2688 (@mapno)
  Added new defaults block to the overrides' module. Overrides change to indented syntax.
  Old config:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

New config:

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]

Features

• [FEATURE] New TraceQL structural operators ancestor (<<), parent (<) #2877 (@kousikmitra)
• [FEATURE] Add the /api/status/buildinfo endpoint #2702 (@fabrizio-grafana)
• [FEATURE] New encoding vParquet3 with support for dedicated attribute columns (@mapno, @stoewer) #2649
• [FEATURE] Add filtering support to Generic Forwarding #2742 (@Blinkuu)
• [FEATURE] Add cli command to print out summary of large traces #2775 (@ie-pham)
• [FEATURE] Added not structural operators to TraceQL: !>, !<, and !~ #2993 (@joe-elliott)

Enhancements

• [ENHANCEMENT] Make metrics-generator ingestion slack per tenant #2589 (@ie-pham)
• [ENHANCEMENT] Support quoted attribute name in TraceQL #3004 (@kousikmitra)
• [ENHANCEMENT] Add support for searching by span status message using statusMessage keyword #2848 (@kousikmitra)
• [ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup #2697 (@mdisibio)
• [ENHANCEMENT] Assert ingestion rate limits as early as possible #2640 (@mghildiy)
• [ENHANCEMENT] Add several metrics-generator fields to user-configurable overrides #2711 (@kvrhdn)
• [ENHANCEMENT] Update /api/metrics/summary to correctly handle missing attributes and improve performance of TraceQL select() queries. #2765 (@mdisibio)
• [ENHANCEMENT] Tempo CLI command to convert from vParquet2 -> 3. #2828 (@joe-elliott)
• [ENHANCEMENT] Add TempoUserConfigurableOverridesReloadFailing alert #2784 (@kvrhdn)
• [ENHANCEMENT] Add RootSpanName and RootServiceName to log about discarded spans #2816 (@MarcinGinszt)
• [ENHANCEMENT] Add UserID to log message about rate limiting #2850 (@lshippy)
• [ENHANCEMENT] Requests to Azure Blob Storage will now be retried once instead of zero times #2835 (@LasseHels)
• [ENHANCEMENT] Add span metrics filter policies to user-configurable overrides #2906 (@rlankfo)
• [ENHANCEMENT] Add collection-interval to metrics-generator config in user-configurable overrides #2899 (@rlankfo)
• [ENHANCEMENT] Enforce max trace size on the trace by id path. #2935 (@joe-elliott)
• [ENHANCEMENT] Add target_info_excluded_dimensions to user-config api #2945 (@ie-pham)
• [ENHANCEMENT] User-configurable overrides: add scope query parameter to return merged overrides for tenant #2915 #3018 (@kvrhdn)
• [ENHANCEMENT] Add histogram buckets to metrics-generator config in user-configurable overrides #2928 (@mar4uk)
• [ENHANCEMENT] Adds websocket support for search streaming. #2971 (@joe-elliott)
• [ENHANCEMENT] Add new config block to distributors to produce debug metrics. #3008 (@joe-elliott)
  Breaking Change Removed deprecated config option: distributor.log_received_traces
• [ENHANCEMENT] added a metrics generator config option to enable/disable X-Scope-OrgID headers on remote write. #2974 (@vineetjp)
• [ENHANCEMENT] Correctly return RetryInfo to Otel Collector/Grafana Agent on ResourceExhausted. This allows the agents to honor their own retry settings. #3019 (@joe-elliott)

Bugfixes

• [BUGFIX] Unescape tag names #2894 (@fabrizio-grafana)
• [BUGFIX] Load defaults for the internal server #3041 (@rubenvp8510)
• [BUGFIX] Fix pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions #3012 (@electron0zero)
• [BUGFIX] Fix panic in metrics summary api #2738 (@mdisibio)
• [BUGFIX] Fix rare deadlock when uploading blocks to Azure Blob Storage #2129 (@LasseHels)
• [BUGFIX] Only search ingester blocks that fall within the request time range. #2783 (@joe-elliott)
• [BUGFIX] Align tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. #2840 (@joe-elliott)
  This query will now correctly tell you %age of requests that are within SLO:

  sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op) 
  /
  sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
  

  BREAKING CHANGE Removed: tempo_query_frontend_queries_total{op="searchtags|metrics"}.
• [BUGFIX] To support blob storage in Azure Stack Hub as backend. #2853 (@chlislb)
• [BUGFIX] Respect spss on GRPC streaming. #2971 (@joe-elliott)
• [BUGFIX] Moved empty root span substitution from querier to query-frontend. #2671 (@galalen)
• [BUGFIX] Correctly propagate ingester errors on the query path #2935 (@joe-elliott)
• [BUGFIX] Fix issue where ingester doesn't stop query after timeout #3031 (@mdisibio)
• [BUGFIX] Fix cases where empty filter {} wouldn't return expected results #2498 (@mdisibio)
• [BUGFIX] Reorder S3 credential chain and upgrade minio-go. native_aws_auth_enabled is deprecated #3006 (@ekristen, @mapno)

v2.2.4

Changes

• [CHANGE] Update alpine image version to 3.18 to patch CVE-2022-48174. #3046 (@joe-elliott)
• [CHANGE] Bump Jaeger query docker image to 1.50.0 #2998 (@pavolloffay)

v2.3.0-rc.0

Breaking Changes

• Removed deprecated config option: distributor.log_received_traces. Use the distributor.log_received_spans config block instead.
• Removed frontend metrics tempo_query_frontend_queries_total{op="searchtags|metrics"}

Changes

• [CHANGE] Update Go to 1.21 #2486 (@zalegrala)
• [CHANGE] Make metrics-generator ingestion slack per tenant #2589 (@ie-pham)
• [CHANGE] Moved the tempo_ingester_traces_created_total metric to be incremented when a trace is cut to the wal #2884 (@joe-elliott)
• [CHANGE] Upgrade from deprecated azure-storage-blob-go SDK to azure-sdk-for-go #2835 (@LasseHels)
• [CHANGE] Metrics summary API validate the requested time range #2902 (@mdisibio)
• [CHANGE] Restructure Azure backends into versioned backends. Introduce use_v2_sdk config option for switching. #2952 (@zalegrala)
  v1: azure-storage-blob-go original (now deprecated) SDK
  v2: azure-sdk-for-go
• [CHANGE] Adjust trace size estimation to better honor row group size settings. #3038 (@joe-elliott)
• [CHANGE] Update alpine image version to 3.18. #3046 (@joe-elliott)
• [CHANGE] Overrides module refactor #2688 (@mapno)
  Added new defaults block to the overrides' module. Overrides change to indented syntax.
  Old config:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

New config:

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]

Features

• [FEATURE] New TraceQL structural operators ancestor (<<), parent (<) #2877 (@kousikmitra)
• [FEATURE] Add the /api/status/buildinfo endpoint #2702 (@fabrizio-grafana)
• [FEATURE] New encoding vParquet3 with support for dedicated attribute columns (@mapno, @stoewer) #2649
• [FEATURE] Add filtering support to Generic Forwarding #2742 (@Blinkuu)
• [FEATURE] Add cli command to print out summary of large traces #2775 (@ie-pham)
• [FEATURE] Added not structural operators to TraceQL: !>, !<, and !~ #2993 (@joe-elliott)

Enhancements

• [ENHANCEMENT] Support quoted attribute name in TraceQL #3004 (@kousikmitra)
• [ENHANCEMENT] Unescape tag names #2894 (@fabrizio-grafana)
• [ENHANCEMENT] Add support for searching by span status message using statusMessage keyword #2848 (@kousikmitra)
• [ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup #2697 (@mdisibio)
• [ENHANCEMENT] Assert ingestion rate limits as early as possible #2640 (@mghildiy)
• [ENHANCEMENT] Add several metrics-generator fields to user-configurable overrides #2711 (@kvrhdn)
• [ENHANCEMENT] Update /api/metrics/summary to correctly handle missing attributes and improve performance of TraceQL select() queries. #2765 (@mdisibio)
• [ENHANCEMENT] Tempo CLI command to convert from vParquet2 -> 3. #2828 (@joe-elliott)
• [ENHANCEMENT] Add TempoUserConfigurableOverridesReloadFailing alert #2784 (@kvrhdn)
• [ENHANCEMENT] Add RootSpanName and RootServiceName to log about discarded spans #2816 (@MarcinGinszt)
• [ENHANCEMENT] Add UserID to log message about rate limiting #2850 (@lshippy)
• [ENHANCEMENT] Requests to Azure Blob Storage will now be retried once instead of zero times #2835 (@LasseHels)
• [ENHANCEMENT] Add span metrics filter policies to user-configurable overrides #2906 (@rlankfo)
• [ENHANCEMENT] Add collection-interval to metrics-generator config in user-configurable overrides #2899 (@rlankfo)
• [ENHANCEMENT] Enforce max trace size on the trace by id path. #2935 (@joe-elliott)
• [ENHANCEMENT] Add target_info_excluded_dimensions to user-config api #2945 (@ie-pham)
• [ENHANCEMENT] User-configurable overrides: add scope query parameter to return merged overrides for tenant #2915 #3018 (@kvrhdn)
• [ENHANCEMENT] Add histogram buckets to metrics-generator config in user-configurable overrides #2928 (@mar4uk)
• [ENHANCEMENT] Adds websocket support for search streaming. #2971 (@joe-elliott)
• [ENHANCEMENT] Add new config block to distributors to produce debug metrics. #3008 (@joe-elliott)
  Breaking Change Removed deprecated config option: distributor.log_received_traces
• [ENHANCEMENT] added a metrics generator config option to enable/disable X-Scope-OrgID headers on remote write. #2974 (@vineetjp)
• [ENHANCEMENT] Correctly return RetryInfo to Otel Collector/Grafana Agent on ResourceExhausted. This allows the agents to honor their own retry
  settings. #3019 (@joe-elliott)

Bugfixes

• [BUGFIX] Load defaults for the internal server #3041 (@rubenvp8510)
• [BUGFIX] Fix pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions #3012 (@electron0zero)
• [BUGFIX] Fix panic in metrics summary api #2738 (@mdisibio)
• [BUGFIX] Fix rare deadlock when uploading blocks to Azure Blob Storage #2129 (@LasseHels)
• [BUGFIX] Only search ingester blocks that fall within the request time range. #2783 (@joe-elliott)
• [BUGFIX] Align tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. #2840 (@joe-elliott)
  This query will now correctly tell you %age of requests that are within SLO:

  sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op) 
  /
  sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
  

  BREAKING CHANGE Removed: tempo_query_frontend_queries_total{op="searchtags|metrics"}.
• [BUGFIX] To support blob storage in Azure Stack Hub as backend. #2853 (@chlislb)
• [BUGFIX] Respect spss on GRPC streaming. #2971 (@joe-elliott)
• [BUGFIX] Moved empty root span substitution from querier to query-frontend. #2671 (@galalen)
• [BUGFIX] Correctly propagate ingester errors on the query path #2935 (@joe-elliott)
• [BUGFIX] Fix issue where ingester doesn't stop query after timeout #3031 (@mdisibio)
• [BUGFIX] Fix cases where empty filter {} wouldn't return expected results #2498 (@mdisibio)
• [BUGFIX] Reorder S3 credential chain and upgrade minio-go. native_aws_auth_enabled is deprecated #3006 (@ekristen, @mapno)

v2.2.3

Bugfixes

• Fix S3 credentials providers configuration #2889 (@mapno)

v2.2.2

Bugfixes

[BUGFIX] Fix node role auth IDMSv1 #2760 (@coufalja)

v2.2.1

Bugfixes

[BUGFIX] Fix incorrect metrics for index failures #2781 (@zalegrala)
[BUGFIX] Fix panic in the metrics-generator when using multiple tenants with default overrides #2786 (@kvrhdn)
[BUGFIX] Restore tenant_header_key removed in #2414. #2795 (@joe-elliott)
[BUGFIX] Disable streaming over http by default. #2803 (@joe-elliott)

v2.2.0

Breaking Changes

• [CHANGE] Breaking Change Remove support tolerate_failed_blocks. #2416 (@joe-elliott)
  Removed config option:

  query_frontend:
    tolerate_failed_blocks: <int>
  

• [CHANGE] Breaking Change Rename s3.insecure_skip_verify #2407 (@zalegrala)

storage:
  trace:
    s3:
      insecure_skip_verify: true   // renamed to tls_insecure_skip_verify

• [CHANGE] Breaking Change Convert metrics generator from deployment to a statefulset in jsonnet. Refer to the PR for seamless migration instructions. #2533 #2467 (@zalegrala)

Changes

• [CHANGE] Make vParquet2 the default block format #2526 (@stoewer)
• [CHANGE] Change log level of two compactor messages from debug to info. #2443 (@DylanGuedes)
• [CHANGE] Remove tenant_header_key option from tempo-query config #2414 (@kousikmitra)
• [CHANGE] Prefix service graph extra dimensions labels with server_ and client_ if enable_client_server_prefix is enabled #2335 (@domasx2)
• [CHANGE] Ignore context canceled errors in the queriers #2440 (@joe-elliott)
• [CHANGE] Start flush queue worker after wal replay and block rediscovery #2456 (@ie-pham)
• [CHANGE] Update Go to 1.20.4 #2486 (@ie-pham)

Features

• [FEATURE] New experimental API to derive on-demand RED metrics grouped by any attribute, and new metrics generator processor #2368 #2418 #2424 #2442 #2480 #2481 #2501 #2579 #2582 (@mdisibio @zalegrala)
• [FEATURE] New TraceQL structural operators descendant (>>), child (>), and sibling (~) #2625 #2660 (@mdisibio)
• [FEATURE] Add user-configurable overrides module #2543 (@electron0zero @kvrhdn)
• [FEATURE] Add support for q query param in /api/v2/search/<tag.name>/values to filter results based on a TraceQL query #2253 (@mapno)
  To make use of filtering, configure autocomplete_filtering_enabled.
• [FEATURE] Add support for by() and coalesce() to TraceQL. #2490
• [FEATURE] Add a GRPC streaming endpoint for traceql search #2366 (@joe-elliott)
• [FEATURE] Add select() to TraceQL #2494 (@joe-elliott)
• [FEATURE] Add traceDuration, rootName and rootServiceName intrinsics to TraceQL #2503 (@joe-elliott)

Enhancements

• [ENHANCEMENT] Add support for query batching between frontend and queriers to improve throughput #2677 (@joe-elliott)
• [ENHANCEMENT] Add initial RBAC support for serverless backend queries, limited to Google CloudRun #2487 (@modulitos)
• [ENHANCEMENT] Add capability to flush all remaining traces to backend when ingester is stopped #2538(@mghildiy)
• [ENHANCEMENT] Fill parent ID column and nested set columns #2487 (@stoewer)
• [ENHANCEMENT] Add metrics generator config option to allow customizable ring port #2399 (@mdisibio)
• [ENHANCEMENT] Improve performance of TraceQL regex #2484 (@mdisibio)
• [ENHANCEMENT] log client ip to help identify which client is no org id #2436 (@Komey)
• [ENHANCEMENT] Add spss parameter to /api/search/tags to configure the spans per span set in response#2308 (@stoewer)
• [ENHANCEMENT] Continue polling tenants on error with configurable threshold #2540 (@mdisibio)
• [ENHANCEMENT] Fully skip over parquet row groups with no matches in the column dictionaries #2676 (@mdisibio)
• [ENHANCEMENT] Add prefix configuration option for s3 #2362 (@kousikmitra)
• [ENHANCEMENT] Add support to filter using negated regex operator !~ #2410 (@kousikmitra)
• [ENHANCEMENT] Add prefix configuration option to storage.trace.azure and storage.trace.gcs #2386 (@kousikmitra)
• [ENHANCEMENT] Add prefix configuration option to storage.trace.s3 #2362 (@kousikmitra)
• [ENHANCEMENT] Add support for concurrent_shards under trace_by_id #2416 (@joe-elliott)

  query_frontend:
    trace_by_id:
      concurrent_shards: 3
  

• [ENHANCEMENT] Enable cross cluster querying by adding two config options. #2598 (@joe-elliott)

  querier:
    secondary_ingester_ring: <string>
  metrics_generator:
    override_ring_key: <string>
  

• [ENHANCEMENT] Add scope parameter to /api/search/tags #2282 (@joe-elliott)
  Create new endpoint /api/v2/search/tags that returns all tags organized by scope.
• [ENHANCEMENT] Ability to toggle off latency or count metrics in metrics-generator #2070 (@AlexDHoffer)
• [ENHANCEMENT] Extend /flush to support flushing a single tenant #2260 (@kvrhdn)
• [ENHANCEMENT] Add override to limit number of blocks inspected in tag value search #2358 (@mapno)
• [ENHANCEMENT] New synchronous read mode for vParquet and vParquet2 #2165 #2535 (@mdisibio)
• [ENHANCEMENT] Add option to override metrics-generator ring port #2399 (@mdisibio)
• [ENHANCEMENT] Add support for IPv6 #1555 (@zalegrala)
• [ENHANCEMENT] Add span filtering to spanmetrics processor #2274 (@zalegrala)
• [ENHANCEMENT] Add ability to detect virtual nodes in the servicegraph processor #2365 (@mapno)
• [ENHANCEMENT] Introduce overrides.Interface to decouple implementation from usage #2482 (@kvrhdn)
• [ENHANCEMENT] Improve TraceQL throughput by asynchronously creating jobs #2530 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix Search SLO by routing tags to a new handler. #2468 (@electron0zero)
• [BUGFIX] tempodb integer divide by zero error #2167 (@kroksys)
• [BUGFIX] metrics-generator: ensure Prometheus will scale up shards when remote write is lagging behind #2463 (@kvrhdn)
• [BUGFIX] Fixes issue where matches and other spanset level attributes were not persisted to the TraceQL results. #2490
• [BUGFIX] Fixes issue where ingester search could occasionally fail with file does not exist error #2534 (@mdisibio)
• [BUGFIX] Tempo failed to find meta.json path after adding prefix in S3/GCS/Azure configuration. #2585 (@WildCatFish)
• [BUGFIX] Delay logging config warnings until the logger has been initialized #2645 (@kvrhdn)
• [BUGFIX] Fix issue where metrics-generator was setting wrong labels for traces_target_info #2546 (@ie-pham)