feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

evenyag · 2025-01-21T08:55:02Z

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

This PR bumps a lot of crates:

datafusion 2464703c84c400a09cc59277018813f0e797bb4e (After 43.0.0)
sqllparser 0.52.x
prost 0.13
hyper 1.x
tonic 0.12
tower 0.5
axum 0.8
arrow 53
parquet 53

There are too many changes due to many breaking changes in these crates. Here are some of them:

Implements Debug for plans and related structs.
Uses SessionStateBuilder to build SessionState
Removes the usage of HandleErrorLayer.
Implements From<arrow interval> for interval types.
Adds a new method new_with_count() to RecordBatch to allow constructing an empty RecordBatch that only contains row count.
DFLogicalSubstraitConvertor doesn't require catalog list now.
Supports Some(Vec::new()) projection in mito.
mito uses min_bytes_opt, max_bytes_opt, min_opt, max_opt.
ColumnSchema will ignore the metadata of time index if it doesn't have timestamp type. (For SQL CAST).
Some protobuf structs support Copy now.
DfContextProviderAdapter implements all missing methods.
Uses SortExpr and the new function API of datafusion.
Uses Bytes to replace RawBody in axum handlers.
Refactors all middleware for gRPC server and HTTP server.
Updates all sqlness results.
udf functions with zero arguements need Nullary signature now

Breaking changes

id and tag are SQL keywords now

Notes

The HTTP server doesn't support setting the TCP keepalive timeout. We may need to check if it has regressions.
DatafusionQueryEngine::optimize_physical_plan() doesn't optimize the physical plan now, as the planner already optimize it.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.
This PR requires documentation updates.
API changes are backward compatible.
Schema or data changes are backward compatible.

Signed-off-by: Ruihang Xia <[email protected]>

* remove handle_error handler * refactor timeout layer * serve axum

* boxed() * use TokioIo

* feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning

src/common/datasource/src/file_format/csv.rs

src/common/grpc/src/channel_manager.rs

src/common/meta/src/state_store.rs

src/mito2/src/read/projection.rs

waynexia · 2025-01-21T12:44:06Z

src/common/runtime/src/runtime.rs

+        let runtime = if cfg!(debug_assertions) {
+            // Set the stack size to 8MB for the thread so it wouldn't overflow on large stack usage in debug mode
+            // This is necessary to avoid stack overflow while running sqlness.
+            // https://github.com/rust-lang/rust/issues/34283
+            builder
+                .thread_stack_size(8 * 1024 * 1024)
+                .build()
+                .context(BuildRuntimeSnafu)?
+        } else {
+            builder.build().context(BuildRuntimeSnafu)?
+        };


wondering who is overflowing

You can check this backtrace. Maybe the LogicalPlan consumes the stack a lot and the rewrite() call finally overflow.
sqlness-overflow.txt

src/mito2/src/cache/file_cache.rs

src/sql/src/util.rs

tests-integration/src/otlp.rs

src/common/meta/src/state_store.rs

src/flow/src/expr/relation/func.rs

src/flow/src/df_optimizer.rs

src/flow/src/transform/aggr.rs

src/flow/src/transform/literal.rs

src/operator/src/statement/admin.rs

src/query/src/datafusion.rs

Co-authored-by: jeremyhi <[email protected]>

src/catalog/src/system_schema/information_schema/cluster_info.rs

src/common/function/src/scalars/date/date_add.rs

src/flow/src/expr/func.rs

killme2008 · 2025-01-22T08:09:51Z

src/flow/src/transform.rs

@@ -179,6 +187,7 @@ mod test {

    pub fn create_test_ctx() -> FlownodeContext {
        let mut tri_map = IdToNameMap::new();
+        // deprecated: use `numbers_with_ts` instead since this table has no timestamp column


Leave a fixme here @discord9 , don't forget to update the tests.

Leave a fixme here @discord9 , don't forget to update the tests.

will fix this in a separate PR once this get merged since it' unrelated to update datafusion

src/flow/src/transform/literal.rs

src/mito2/src/read/projection.rs

src/mito2/src/sst/parquet/format.rs

killme2008 · 2025-01-22T08:21:31Z

src/promql/src/extension_plan/scalar_calculate.rs

@@ -195,6 +195,37 @@ impl ScalarCalculate {
    }
 }

+impl PartialOrd for ScalarCalculate {


NonBlocking: maybe we can add some macros to help implement PartialOrd for plans.

Signed-off-by: Ruihang Xia <[email protected]>

tests/cases/standalone/common/view/view.result

tests/cases/standalone/common/flow/flow_null.sql

src/servers/Cargo.toml

waynexia · 2025-01-22T12:32:13Z

src/sql/src/statements/transform/type_alias.rs

+  tint TINYINT,
  sint SMALLINT,
  i INT,
-  bint INT64,
+  bint BIGINT,
  v VARCHAR,
  f FLOAT,
-  d FLOAT64,
+  d DOUBLE,


Not sure if we have some document about these that need to be updated

Maybe we don't have to do that. This is the expected behavior implemented in #2331

We forgot to fix this when we updated the sqlparser. You can find out some types have alias but some do not in this test. Now this PR fixes this and removes the TODO. cc @killme2008

src/sql/src/statements/transform/type_alias.rs

waynexia

I've reviewed them and hereby cast my vote 🫡

src/query/src/range_select/plan.rs

evenyag · 2025-01-22T13:18:58Z

rustc may panic...
https://github.com/GreptimeTeam/greptimedb/actions/runs/12908415598/job/35994232050

I'll retry that test.

fengjiachun

Rest LGTM

waynexia and others added 30 commits December 6, 2024 16:33

change dep

2db8107

Signed-off-by: Ruihang Xia <[email protected]>

feat: adapt to arrow's interval array

db9dfba

chore: fix compile errors in datatypes crate

c757feb

chore: fix api crate compiler errors

181f28e

chore: fix compiler errors in common-grpc

75f9ea2

chore: fix common-datasource errors

46bc65e

chore: fix deprecated code in common-datasource

4514db9

fix promql and physical plan related

ce73058

Signed-off-by: Ruihang Xia <[email protected]>

wip: upgrading network deps

8dad368

Signed-off-by: Ruihang Xia <[email protected]>

block on updating sqlparser

8f1c760

upgrade sqlparser

3845086

Signed-off-by: Ruihang Xia <[email protected]>

adapt new df's trait requirements

3047af4

Signed-off-by: Ruihang Xia <[email protected]>

chore: fix compiler errors in mito2

9caf689

chore: fix common-function crate errors

5986565

chore: fix catalog errors

761ff28

change import path

93213dd

Signed-off-by: Ruihang Xia <[email protected]>

chore: fix some errors in query crate

55add17

chore: fix some errors in query crate

069fff0

aggr expr and some other tiny fixes

176645b

Signed-off-by: Ruihang Xia <[email protected]>

chore: fix expr related errors in query crate

885bc52

chore: fix query serializer and admin command

2b1caa6

chore: fix grpc services

0207c37

feat: axum serve

9c575d8

chore: fix http server

b1a3490

* remove handle_error handler * refactor timeout layer * serve axum

chore: fix flow aggr functions

91abacf

chore: fix flow

ba6442f

feat: fix errors in meta-srv

04871e2

* boxed() * use TokioIo

feat!: Remove script crate and python feature (#5321)

b68f727

* feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning

chore: fix servers tests compiler errors

124115e

feat: fix tests-integration errors

c6e3452

waynexia reviewed Jan 21, 2025

View reviewed changes

src/common/datasource/src/file_format/csv.rs Show resolved Hide resolved

src/common/grpc/src/channel_manager.rs Show resolved Hide resolved

waynexia reviewed Jan 21, 2025

View reviewed changes

src/mito2/src/cache/file_cache.rs Show resolved Hide resolved

src/sql/src/util.rs Outdated Show resolved Hide resolved

tests-integration/src/otlp.rs Outdated Show resolved Hide resolved

tests-integration/src/otlp.rs Outdated Show resolved Hide resolved

evenyag added 2 commits January 21, 2025 21:21

fix: return error if sql option is not kv

68ba6df

chore: do not use ..default::default()

5826579

fengjiachun reviewed Jan 22, 2025

View reviewed changes

src/operator/src/statement/admin.rs Outdated Show resolved Hide resolved

src/query/src/datafusion.rs Show resolved Hide resolved

discord9 and others added 2 commits January 22, 2025 15:39

chore: per review

b58d2f8

chore: update error message in BuildAdminFunctionArgsSnafu

feeb3a8

Co-authored-by: jeremyhi <[email protected]>

killme2008 reviewed Jan 22, 2025

View reviewed changes

discord9 and others added 2 commits January 22, 2025 16:41

refactor: typed precision

3aed891

update sqlness view case

257e8dd

Signed-off-by: Ruihang Xia <[email protected]>

killme2008 reviewed Jan 22, 2025

View reviewed changes

tests/cases/standalone/common/view/view.result Outdated Show resolved Hide resolved

chore: flow per review

500a8c7

waynexia reviewed Jan 22, 2025

View reviewed changes

tests/cases/standalone/common/flow/flow_null.sql Show resolved Hide resolved

evenyag added 2 commits January 22, 2025 17:38

chore: add example in comment

bad35dc

chore: warn if parquet stats of timestamp is not INT64

30a78a9

waynexia reviewed Jan 22, 2025

View reviewed changes

src/servers/Cargo.toml Outdated Show resolved Hide resolved

evenyag force-pushed the bump-df-202412 branch from de9e55b to 30a78a9 Compare January 22, 2025 10:50

evenyag added 3 commits January 22, 2025 18:52

style: add a newline before derive to make the comment more clear

aaacdbd

test: update sqlness result

5b9d412

chore: Merge branch 'main' into bump-df-202412

5e6b7e8

waynexia reviewed Jan 22, 2025

View reviewed changes

src/sql/src/statements/transform/type_alias.rs Show resolved Hide resolved

fix: flow from substrait

74b34aa

waynexia approved these changes Jan 22, 2025

View reviewed changes

src/query/src/range_select/plan.rs Outdated Show resolved Hide resolved

evenyag added 2 commits January 22, 2025 21:22

chore: change update_range_context log to debug level

fbbbfee

chore: move axum-extra axum-macros to workspace

3a28240

fengjiachun approved these changes Jan 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

evenyag commented Jan 21, 2025 •

edited by waynexia

Loading

waynexia Jan 21, 2025

evenyag Jan 21, 2025

killme2008 Jan 22, 2025

discord9 Jan 22, 2025

killme2008 Jan 22, 2025

waynexia Jan 22, 2025

evenyag Jan 22, 2025

waynexia left a comment

evenyag commented Jan 22, 2025

fengjiachun left a comment

feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

Are you sure you want to change the base?

feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

Conversation

evenyag commented Jan 21, 2025 • edited by waynexia Loading

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Breaking changes

Notes

PR Checklist

waynexia Jan 21, 2025

Choose a reason for hiding this comment

evenyag Jan 21, 2025

Choose a reason for hiding this comment

killme2008 Jan 22, 2025

Choose a reason for hiding this comment

discord9 Jan 22, 2025

Choose a reason for hiding this comment

killme2008 Jan 22, 2025

Choose a reason for hiding this comment

waynexia Jan 22, 2025

Choose a reason for hiding this comment

evenyag Jan 22, 2025

Choose a reason for hiding this comment

waynexia left a comment

Choose a reason for hiding this comment

evenyag commented Jan 22, 2025

fengjiachun left a comment

Choose a reason for hiding this comment

evenyag commented Jan 21, 2025 •

edited by waynexia

Loading