You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I don't know the right API for this, but a scan already partitions the file into splits. When column splits don't align, we end up computing filters and projections over chunked arrays.
Currently the chunked compute functions preserve chunking, which adds lots of overhead to the splits which we know are reasonably sized.
Either we should never preserve chunking? Or we should set some thread local options that we can configure during the scan.
The text was updated successfully, but these errors were encountered:
I don't know the right API for this, but a scan already partitions the file into splits. When column splits don't align, we end up computing filters and projections over chunked arrays.
Currently the chunked compute functions preserve chunking, which adds lots of overhead to the splits which we know are reasonably sized.
Either we should never preserve chunking? Or we should set some thread local options that we can configure during the scan.
The text was updated successfully, but these errors were encountered: