AutoML API model incompatible with AutoML CLI?

[famschopman]

When I train and save a model with the ModelBuilder from Visual Studio, and inspect the model that is loaded to predict I get a TrainSchema with all columns, and the last column is the Features column.

When I train and save a model with the AutoML API, it received an exception where it tries to find a SamplingKeyColumn. After inspection AutoML has saved the model with an additional column just before the Features column. This is different compared to the ModelBuilder behavior.

So I have to extend me model with the following lines to make sure I can run a prediction (I have 34 columns in my dataset).

        [ColumnName("SamplingKeyColumn"), LoadColumn(35)]
        public float SamplingKeyColumn { get; set; }

Is this expected behavior? It feels very inconsistent and quite confusing behavior. I couldn't find any documentation about this either.

[justinormont]

Sounds like an issue w/ our CodeGen in the CLI. If a SamplingKeyColumn is provided to the API, we should be creating one in the output data class.

/cc @srsaggam, @JakeRadMSFT

[famschopman]

Ok, good to hear. For the time being I will have to use two different model definitions.

1: One to load the IDataView, without a SamplingKeyColumn
2: One to load the model from disk, with a SamplingKeyColumn

If you add the SamplingKeyColumn to the initial model definition AutoML tries to mitigate that and you will end up with another one, drumroll so don't even think about it 🤐

Could not find input column 'temp_SamplingKeyColumn_000'
Parameter name: inputSchema

[acrigney]

This really needs to be fixed ASAP please.

[LittleLittleCloud]

Seems like a bug in AutoML.Net
@justinormont is this one been fixed or not?

[justinormont]

@justinormont is this one been fixed or not?

I don't think it's been fixed.

[JakeRadMSFT]

@LittleLittleCloud Can you look into this?

[antoniovs1029]

So I believe this is actually an issue on ML.NET's mlContext.Data.TrainTestSplit() and mlContext.Data.CrossValidationSplit() as explained here:
dotnet/machinelearning#5256 (comment)

But I'm unsure if the OP of this issue here ( @famschopman ) used these methods? Can you still provide a full repro of your issue? It would be very helpful, @famschopman !! 😄

My suspicion is that when the user says "When I train and save a model with the AutoML API", it means that they used code similar to the one on dotnet/machinelearning#5256 (comment) , where TrainTestSplit() or CrossValidationSplit() was actually used before calling AutoML? ... It is also not clear from this issue description exactly where/when the exception was thrown? In the other issue I've linked, it was while creating a PredictionEngine, whereas no PredictionEngine is mentioned here, and it would be interesting to know this information to find other possible places where this issue might surface.

[beccamc]

@LittleLittleCloud Is this fixed?

[famschopman]

I am not sure if I still have the code laying around to help you with a repro, but if needed just ping me and I will try to find. I will be picking up ML.Net soon again. First have to finish some other software first ... 😘

[luisquintanilla]

@LittleLittleCloud since this references the old AutoML API and presumably the older version of the ML.NET CLI, I think we're okay to close unless it continues to be an issue.

[LittleLittleCloud]

Close since this issue is related to old AutoML API, feel free to reopen it if you have questions