-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[c++] Integrate SOMAColumn
: Arrow adapter methods, part 1
#3405
base: main
Are you sure you want to change the base?
[c++] Integrate SOMAColumn
: Arrow adapter methods, part 1
#3405
Conversation
SOMAColumn
in Arrow adapter methods [WIP]SOMAColumn
in Arrow adapter methods Part 2
SOMAColumn
in Arrow adapter methods Part 2SOMAColumn
in Arrow adapter methods Part 2
SOMAColumn
in Arrow adapter methods Part 2SOMAColumn
in Arrow adapter methods, part 2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you explain what the Skip
and Take
are for and document it? It looks like Take
is the index of the column to retrieve and Skip
is relevant only for geometry columns (where it's always 2)?
Also is there a way to use std::variant
or a templated type instead of std::any
or would that make things too complicated?
/** | ||
* Return a copy of the data in a specified column of an arrow table. | ||
* Complex column types are supported. The for each sub column are an | ||
* std::array<T, 2> casted as an std::any object. | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/** | |
* Return a copy of the data in a specified column of an arrow table. | |
* Complex column types are supported. The for each sub column are an | |
* std::array<T, 2> casted as an std::any object. | |
*/ | |
/** | |
* Return a copy of the data in a specified column of an arrow table. | |
* Complex column types are supported. The type for each sub column is | |
* an std::array<T, 2> casted as an std::any object. | |
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Skip
and Take
are used in 2 places with 2 specific sets on values (either Skip=3
and Take=2
or Skip=0
and Take=2
) and are independent of the geometry column. Their usage is to extract specific subranges of ArrowArray data and they come in handy during ArrowSchema -> TileDBSchema
where the arrow array provided has 5 values per dimension and we only need the last 2 to set the current domain.
As to using std::variant
, adding more SOMAColumn
types would require changing multiple variants. The use of std::any
here is to enable runtime polymorphism and indirectly introduces a runtime type check (via any_cast
, make_any
) between the templated function and the actual dimension type. std::variant
can provide all the above it is just a different style I am open to discuss further.
804c87e
to
a58361a
Compare
b1bd03c
to
5485141
Compare
…nt domain checks, replace vector with span when selecting points
…nt domain checks, replace vector with span when selecting points
…nt domain checks, replace vector with span when selecting points
5485141
to
0e69ed7
Compare
SOMAColumn
in Arrow adapter methods, part 2SOMAColumn
: Arrow adapter methods, part 1
0e69ed7
to
d6d6187
Compare
8daf17e
to
a426c7a
Compare
This PR replaces the Arrow schema to TileDB schema transformation to use the
SOMAColumn
create methods.Also there are a set of new data converters from arrow arrays to
std::array
for simplification.This migration also enforces a current domain restriction for string dimensions to libtiledbsoma in addition to the restriction being present only on the R and Python APIs.