Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Add support for lz4raw compression #126

Open
Dev-iL opened this issue Jan 12, 2025 · 0 comments
Open

[FEAT] Add support for lz4raw compression #126

Dev-iL opened this issue Jan 12, 2025 · 0 comments
Labels
enhancement New feature or request parquet-dotnet-bug

Comments

@Dev-iL
Copy link

Dev-iL commented Jan 12, 2025

I tried opening a parquet file created in pandas using LZ4 compression, and the viewer failed to open it.

See error inside

---------------------------
Lz4Raw not supported
---------------------------
Something went wrong (CTRL+C to copy):

System.NotSupportedException: Lz4Raw not supported

   at Parquet.File.Compressor.ToCodec(CompressionMethod method)

   at Parquet.File.Compressor.Decompress(CompressionMethod method, ReadOnlySpan`1 input, Int32 outLength)

   at Parquet.File.DataColumnReader.ReadPageDataAsync(PageHeader ph)

   at Parquet.File.DataColumnReader.ReadDictionaryPage(PageHeader ph, PackedColumn pc)

   at Parquet.File.DataColumnReader.ReadAsync(CancellationToken cancellationToken)

   at ParquetViewer.Engine.ParquetEngine.ReadPrimitiveField(DataTableLite dataTable, ParquetRowGroupReader groupReader, Int32 rowBeginIndex, ParquetSchemaElement field, Int64 skipRecords, Int64 readRecords, Boolean isFirstColumn, CancellationToken cancellationToken, IProgress`1 progress)

   at ParquetViewer.Engine.ParquetEngine.ProcessRowGroup(DataTableLite dataTable, ParquetRowGroupReader groupReader, Int64 skipRecords, Int64 readRecords, CancellationToken cancellationToken, IProgress`1 progress)

   at ParquetViewer.Engine.ParquetEngine.PopulateDataTable(DataTableLite dataTable, ParquetReader parquetReader, Int64 offset, Int64 recordCount, CancellationToken cancellationToken, IProgress`1 progress)

   at ParquetViewer.Engine.ParquetEngine.ReadRowsAsync(List`1 selectedFields, Int32 offset, Int32 recordCount, CancellationToken cancellationToken, IProgress`1 progress)

   at ParquetViewer.MainForm.<>c__DisplayClass33_0.<<LoadFileToGridview>b__1>d.MoveNext()

--- End of stack trace from previous location ---

   at ParquetViewer.MainForm.LoadFileToGridview()

   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_0(Object state)

   at InvokeStub_SendOrPostCallback.Invoke(Object, Span`1)

   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
---------------------------
OK   
---------------------------

Describe the feature you'd like to be added to Parquet Viewer
LZ4 compression is quite common in certain domains, and it would be great if the viewer could support such files.

Share why this feature would be a good addition to the utility
Will improve usability.

@Dev-iL Dev-iL added the enhancement New feature or request label Jan 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request parquet-dotnet-bug
Projects
None yet
Development

No branches or pull requests

2 participants