Core: Parsing and Writing Tests for V3 Metadata #11947
Conversation
Parameterize TestTableMetadata
# Conflicts: # core/src/test/java/org/apache/iceberg/TestTableMetadata.java
| @@ -949,6 +972,8 @@ private Builder(int formatVersion) { | |||
| this.schemasById = Maps.newHashMap(); | |||
| this.specsById = Maps.newHashMap(); | |||
| this.sortOrdersById = Maps.newHashMap(); | |||
| this.rowLinageEnabled = false; | |||
| this.nextRowId = -1L; | |||
There was a problem hiding this comment.
Theoretically, next-row-id can start from any integer, including negative ones. I am thinking may be in the implementation we could make it starts from 0 to simplify the implementation and make it more intuitive.
|
cc @RussellSpitzer @flyrain |
| @@ -240,6 +242,13 @@ public static void toJson(TableMetadata metadata, JsonGenerator generator) throw | |||
| } | |||
| generator.writeEndArray(); | |||
|
|
|||
| if (metadata.formatVersion() == 3) { | |||
| generator.writeBooleanField(ROW_LINEAGE, metadata.rowLinageEnabled()); | |||
| if (metadata.nextRowId() > -1L) { | |||
There was a problem hiding this comment.
Should we check if the row lineage is enable as well as a defensive programming practice? Or we have to carry on the next row id no matter whether row lineage is enabled or not? cc @RussellSpitzer
There was a problem hiding this comment.
Thanks for reviewing this. Based on offline sync with Russell, I removed row-lineage related changes from this PR. Row-lineage related metadata changes will be included in #11948. Sorry for the confusion
| return new TableMetadataBuilder(formatVersion); | ||
| } | ||
|
|
||
| public static class TableMetadataBuilder { |
There was a problem hiding this comment.
I added test-only builders for TableMetadata and BaseSnapshot. This shall reduce the amount of code changes in tests when adding new fields.
cc: @RussellSpitzer
Fixes #10764
MetadataTestUtils.TableMetadataBuilderandMetadataTestUtils.BaseSnapshotBuilderto simply test metadata and snapshot build. This will reduce the amount of test code changes when adding new fields to metadata/snapshot.Not included:
unknownV3 schema data type (Tracked by: Support UnknownType for V3 Schema #11732 )source-idsfor V3 partition spec (Tracked by: Handling of Source Id => Source Ids #10762)