Conversation
| let table_creation = TableCreation::builder() | ||
| .name(name.clone()) | ||
| .schema(iceberg_schema) | ||
| .format_version(format_version) |
There was a problem hiding this comment.
Before this change existing tests where rightfully failing as we used to create a V2 table with a NS Timestamp column:
https://github.com/apache/iceberg-rust/actions/runs/22522306915/job/65248930667
This new logic determines the min format version required and uses that - but at least V2. Thus we switch now to V3 for ns timestamps.
|
@CTTY @liurenjie1024 @Xuanwo this would be ready for review! |
CTTY
left a comment
CTTY
left a comment
There was a problem hiding this comment.
Thanks for the feature! Just took a look.
Also the test seems to be failing
| custom_attributes: Default::default(), | ||
| }, | ||
| ]; | ||
| let mut schema = avro_record_schema(VARIANT_LOGICAL_TYPE, fields)?; |
There was a problem hiding this comment.
I don't think we can use a static string for every record. what if there are multiple variant columns, would the record name conflict?
| // field ID resolves to Type::Variant and record all their sub-fields so | ||
| // the second filter_leaves can include them directly. | ||
| let mut variant_sub_fields: HashMap<FieldRef, i32> = HashMap::new(); | ||
| for top_field in fields.iter() { |
There was a problem hiding this comment.
What would happen if variant is nested within another type?
| default: None, | ||
| custom_attributes: Default::default(), | ||
| }, | ||
| ]; |
| Err(Error::new( | ||
| ErrorKind::FeatureUnsupported, | ||
| "Conversion from VariantType is not supported for Glue", | ||
| )) |
There was a problem hiding this comment.
we can just return "variant".to_string(), on glue it would look like below:
{
"data": {
"unknown": "variant"
}
}
2 participants
Which issue does this PR close?
Variant Support.
Arrow value support is currently missing as I am unsure how we want to extend
LiteralWhat changes are included in this PR?
Core: Variant Type
crates/iceberg/src/spec/datatypes.rs— newVarianttypecrates/iceberg/src/spec/values/literal.rs—Variantliteral valuecrates/iceberg/src/spec/schema/— visitor, index, pruning, mod, id reassigner all handleVariantcrates/iceberg/src/spec/table_metadata.rs— metadata supportAvro
crates/iceberg/src/avro/schema.rs— read/writeVariantin AvroArrow
crates/iceberg/src/arrow/schema.rs— mapVariantto Arrow typecrates/iceberg/src/arrow/reader.rs— readVariantfrom Arrowcrates/iceberg/src/arrow/value.rs— Arrow value conversioncaching_delete_file_loader.rsandnan_val_cnt_visitor.rsParquet
crates/iceberg/src/writer/file_writer/parquet_writer.rs— writeVariantcolumnsTests & Dev
crates/integration_tests/tests/read_variant.rs— new integration test for reading Variant datadev/spark/provision.py— Spark provisioning to generate Variant test dataAre these changes tested?
Sure! Even integration tested :)