Execution fails when data contains missing values

[rabsr]

As per exception message and comment, missing value is not handled only for categorical features.

But, it fails even when there are any missing values for numerical data. The check, used to find missing value for categorical features and aborts execution, does not consider feature type and check for missing values in all features irrespective of their type.

To handle the issue of missing values, we should update implementation for method _check_and_encode_features and follows

• removing missing value check
• Use category_encoders.OrdinaEncoder as this can handle missing values for category encoding, until missing values for categorical data type is not available in scikit-learn.
• Avoid reordering feature type

[mfeurer]

Thanks a lot for raising this issue and providing a fix @rabsr. Could you please also provide a failing example so we can reproduce this issue?

Also, I just had a brief look at your PR, too. While it potentially solves the issue, I'm leaning to not merge it as it adds a dependency on a 3rd-party package for ML, which we are trying to keep out as much as possible (as we're not even keeping up with scikit-learn yet). Anyway, once you posted the example we'll have a look.

Finally, why is reordering the feature type actually an issue?

[rabsr]

@mfeurer I have used OpenML dataset, data id 15. It has only numerical values and no categorical values. I changed the data to pandas dataframe. Following is the code

import autosklearn.classification
import pandas as pd
import sklearn.datasets
import sklearn.metrics

X, y = sklearn.datasets.fetch_openml(data_id=15, return_X_y=True, as_frame=False)

X = pd.DataFrame(data=X)
y = pd.DataFrame(y, dtype='category')

X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(
    X, y, test_size=0.5, random_state=3)

cls = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=120,
    per_run_time_limit=30,
)
cls.fit(X_train, y_train, X_test, y_test)
predictions = cls.predict(X_test)
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions))

I got the following error
Traceback (most recent call last): File "example_pandas_train_test.py", line 22, in <module> cls.fit(X_train, y_train, X_test, y_test) File "/git/opensource/auto-sklearn/autosklearn/estimators.py", line 587, in fit dataset_name=dataset_name, File "/git/opensource/auto-sklearn/autosklearn/estimators.py", line 346, in fit self.automl_.fit(load_models=True, **kwargs) File "/git/opensource/auto-sklearn/autosklearn/automl.py", line 1177, in fit load_models=load_models, File "/git/opensource/auto-sklearn/autosklearn/automl.py", line 331, in fit X, y = self.InputValidator.validate(X, y) File "/git/opensource/auto-sklearn/autosklearn/data/validation.py", line 70, in validate X = self.validate_features(X) File "/git/opensource/auto-sklearn/autosklearn/data/validation.py", line 118, in validate_features X = self._check_and_encode_features(X) File "/git/opensource/auto-sklearn/autosklearn/data/validation.py", line 311, in _check_and_encode_features raise ValueError("Categorical features in a dataframe cannot contain " ValueError: Categorical features in a dataframe cannot contain missing/NaN values. The OrdinalEncoder used by Auto-sklearn cannot handle this yet (due to a limitation on scikit-learn being addressed via: https://github.com/scikit-learn/scikit-learn/issues/17123)

Finally, why is reordering the feature type actually an issue?

The feature type ordering was required because ColumnTransformer was used which reorders the data. Now columns orders do not change as I have removed the column transformer. Thus feature type reordering is not required.

[mfeurer]

Thanks for the example, we'll have a look into this.

Regarding the underlying issue (scikit-learn/scikit-learn#17123), I think it would be best to solve the issue in scikit-learn instead of importing a 3rd party library here. Maybe you can offer your help in either scikit-learn/scikit-learn#17123 or scikit-learn/scikit-learn#15796

[mfeurer]

Fixed via #1010.