You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There is a limitation when using Entity::insert_many with PostgreSQL when inserting a large number of rows, resulting in a panic!:
[2023-12-11T15:53:31Z INFO sqlx::query] summary="SELECT col1, col2, ..., colN FROM table rows_affected=16854 rows_returned=16854 elapsed=487.267501ms
thread 'main' panicked at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-postgres-0.7.3/src/message/parse.rs:31:13:
assertion failed: self.param_types.len() <= (u16::MAX as usize)
stack backtrace:
0: rust_begin_unwind
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5
1: core::panicking::panic_fmt
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14
2: core::panicking::panic
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:127:5
3: <sqlx_postgres::message::parse::Parse as sqlx_core::io::encode::Encode>::encode_with::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-postgres-0.7.3/src/message/parse.rs:31:13
4: <alloc::vec::Vec<u8> as sqlx_postgres::io::buf_mut::PgBufMutExt>::put_length_prefixed
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-postgres-0.7.3/src/io/buf_mut.rs:25:9
5: <sqlx_postgres::message::parse::Parse as sqlx_core::io::encode::Encode>::encode_with
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-postgres-0.7.3/src/message/parse.rs:25:9
6: sqlx_core::net::socket::buffered::BufferedSocket<S>::write_with
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-core-0.7.3/src/net/socket/buffered.rs:85:9
7: sqlx_core::net::socket::buffered::BufferedSocket<S>::write
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-core-0.7.3/src/net/socket/buffered.rs:78:9
8: sqlx_postgres::connection::executor::prepare::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-postgres-0.7.3/src/connection/executor.rs:50:5
9: sqlx_postgres::connection::executor::<impl sqlx_postgres::connection::PgConnection>::get_or_prepare::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-postgres-0.7.3/src/connection/executor.rs:177:66
10: sqlx_postgres::connection::executor::<impl sqlx_postgres::connection::PgConnection>::run::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-postgres-0.7.3/src/connection/executor.rs:214:18
11: sqlx_postgres::connection::executor::<impl sqlx_core::executor::Executor for &mut sqlx_postgres::connection::PgConnection>::fetch_many::{{closure}}::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-postgres-0.7.3/src/connection/executor.rs:377:71
12: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/future/future.rs:125:9
13: <futures_util::future::future::fuse::Fuse<Fut> as core::future::future::Future>::poll
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/future/future/fuse.rs:84:26
14: futures_util::future::future::FutureExt::poll_unpin
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/future/future/mod.rs:558:9
15: <sqlx_core::ext::async_stream::TryAsyncStream<T> as futures_core::stream::Stream>::poll_next
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-core-0.7.3/src/ext/async_stream.rs:106:15
16: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.29/src/stream.rs:120:9
17: <S as futures_core::stream::TryStream>::try_poll_next
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.29/src/stream.rs:196:9
18: <futures_util::stream::try_stream::try_filter_map::TryFilterMap<St,Fut,F> as futures_core::stream::Stream>::poll_next
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/stream/try_stream/try_filter_map.rs:76:47
19: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.29/src/stream.rs:120:9
20: <S as futures_core::stream::TryStream>::try_poll_next
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.29/src/stream.rs:196:9
21: <futures_util::stream::try_stream::try_collect::TryCollect<St,C> as core::future::future::Future>::poll
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/stream/try_stream/try_collect.rs:46:26
22: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/future/future.rs:125:9
23: sqlx_core::query::Query<DB,A>::fetch_all::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-core-0.7.3/src/query.rs:204:34
24: <sea_orm::database::transaction::DatabaseTransaction as sea_orm::database::connection::ConnectionTrait>::query_all::{{closure}}::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sea-orm-0.12.7/src/database/transaction.rs:455:26
25: <sea_orm::database::transaction::DatabaseTransaction as sea_orm::database::connection::ConnectionTrait>::query_all::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sea-orm-0.12.7/src/database/transaction.rs:430:5
26: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/future/future.rs:125:9
27: sea_orm::executor::insert::exec_insert::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sea-orm-0.12.7/src/executor/insert.rs:228:52
28: migrate::main::{{closure}}
at ./migrate/src/main.rs:46:78
29: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/park.rs:282:63
30: tokio::runtime::coop::with_budget
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/coop.rs:107:5
31: tokio::runtime::coop::budget
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/coop.rs:73:5
32: tokio::runtime::park::CachedParkThread::block_on
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/park.rs:282:31
33: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/context/blocking.rs:66:9
34: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
35: tokio::runtime::context::runtime::enter_runtime
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/context/runtime.rs:65:16
36: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
37: tokio::runtime::runtime::Runtime::block_on
at /home/nickb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/runtime.rs:350:45
38: migrate::main
at ./migrate/src/main.rs:50:5
39: core::ops::function::FnOnce::call_once
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
I feel that it may be the responsibility of the TryInsert::exec functions to transparently chunk the data for insert, or perhaps the TryInsert::add function to fail early if the underlying driver cannot support more than self.columns.len() * self.values.len() in a prepared statement.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
There is a limitation when using
Entity::insert_many
with PostgreSQL when inserting a large number of rows, resulting in apanic!
:The offending
panic!
is defined here:https://github.com/launchbadge/sqlx/blob/main/sqlx-postgres/src/message/parse.rs#L30-L31
I feel that it may be the responsibility of the
TryInsert::exec
functions to transparently chunk the data for insert, or perhaps theTryInsert::add
function to fail early if the underlying driver cannot support more thanself.columns.len() * self.values.len()
in a prepared statement.Beta Was this translation helpful? Give feedback.
All reactions