💫 Categorical dtypes

[AlenkaF]

Adapt the Dataframe protocol implementation for Vaex to work with categorical columns.

• Research how Vaex handles categorical dtypes
• Use the knowladge to adapt the interchange protocol
• Add a roundtrip test

[AlenkaF]

In Vaex there are two ways of constructing a categorical column:

1. using categorize() method on Vaex dataframe
2. convert Arrow Dictionary to Vaex dataframe

For now the implementation works for the first possibility and gives an error for the second.
There are also two possibilities one can use categorize method:

• applying list of labels
• using min and max values of the data (in this case values in the dataframe are the labels)

See more in the research Notebook.

There is also some confusion in the categorical columns generated by categorize method. I created an example Notebook and send it to Vaex team (Maatren and Jovan) for comments and posted a PR. The PR will probably get solved by Maartens work and I will have to adapt expressions.codes in the protocol implementation.

• post a PR to Vaex with test erroring when converting Arrow Dictionary dtype
• resolve the error for Arrow Dictionary
• implement expressions.codes from Maartens PR

[AlenkaF]

The only thing left that needs to be done is to implement expressions.codes from Maartens PR. Otherwise the protocol now works for categorical dtypes.

[AlenkaF]

One question on categorical dtypes came up while working on categorical metadata - #10 (comment):

What needed to be done also is to separately define the dtype for categorical columns (in Vaex dtype of a categorical column is the dtype of data itself). I am not sure if the default is correct:

(_DtypeKind.CATEGORICAL, 64, 'u', '=')

vaex-df-api-implementation/notebook_research/vaex_implementation_v11.py

Lines 288 to 292 in 1002192

	# Categorical
	# If it is internal, kind is categorical (23)
	# If it is external (call from_dataframe) must give data dtype
	if self._col.df.is_category(self._col):
	return (_DtypeKind.CATEGORICAL, 64, 'u', '=') # what should be the default??

Ralf pointed me to an open issue in the general dataframe-api:

data-apis/dataframe-api#49 (comment)
data-apis/dataframe-api#46 (comment)