-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Using PFI with AutoML, possible? #3972
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I'm interested in a solution to this also. It seems like a good way to reduce the number of features if you can identify which features are important. |
@daholste: Do you think this simply needs to be cast into the right type which has Possibly related comic: https://blog.toggl.com/build-horse-programming/ |
First and foremost, I love that comic, @justinormont +1, the C# segment of the comic feels apropos. If you inspect the model in the debugger GUI, you should be able to navigate to the last transformer. Thru casting C# objects as you see them in the debugger, you could write lines of C# code that correspond to the navigation in the GUI Of course, this is terribly hacky. Off-hand, I'm not aware of an officially supported / less hacky way to do this. It could be a great area of focus for future development |
The following cast lets me access the LastTransformer, however I cannot use it for PFI until I provide a better type for predictor. Debugging I can see it is of type Microsoft.ML.Data.RegressionPredictionTransformer<Microsoft.ML.IPredictorProducing> but I am unable to cast to that because Microsoft.ML.IPredictorProducing is not visible, so it seems like we're still stuck.
The following compile error is produced.
|
See my analysis on #3976 as well. These two issues feel like they are the same thing. |
The only thing that was needed to make this build and run was to add the RunDetail<BinaryClassificationMetrics> bestRun = experimentResult.BestRun;
TransformerChain<ITransformer> trainedModel = (TransformerChain <ITransformer>) bestRun.Model;
var predictions = trainedModel.Transform(testData);
var linearPredictor = (ISingleFeaturePredictionTransformer<object>)trainedModel.LastTransformer;
var permutationMetrics = mlContext.BinaryClassification.PermutationFeatureImportance(
linearPredictor, predictions, permutationCount: 30); PS: There was a bug (#4517) when running PFI particularly with Binary classification models, so even after getting this running, if AutoML had returned a non-calibrated binary model, then running PFI would have thrown an exception. This bug got fixed on #4587 , which got included in ML.NET 1.5.0-preview2 and 1.5.0, so that is fixed.
The problem described there got fixed on #4262 and #4292. Still, that problem wasn't really causing this problem, as the solution I mentioned above would have worked even then. The problem you refer to is not being able to cast a model loaded from disk to their actual type (e.g. |
Playing with AutoML and so far having much fun with it.
I have a trained model and now trying to retrieve the feature weights. None of the objects returned expose a LastTransformer object that I need to
Code snippet:
Then I want to get the PFI information and I get stuck. There appears no way to get the LastTransformer object from the trainedModel.
Hope someone can help me with some guidance.
The text was updated successfully, but these errors were encountered: