Skip to content

Keeping data loader in the pipeline while saving model. #268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Keeping data loader in the pipeline while saving model. #268

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
21d2b48
Merge pull request #1 from dotnet/master
zeahmed May 23, 2018
405ffcf
Merge remote-tracking branch 'upstream/master'
zeahmed May 24, 2018
966b091
Merge remote-tracking branch 'upstream/master'
zeahmed May 29, 2018
0a8ec57
Merge remote-tracking branch 'upstream/master'
zeahmed May 30, 2018
b115e7e
Keeping data loader in the pipeline while saving model.
zeahmed May 30, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions src/Microsoft.ML.Core/Data/ITransformModel.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,10 @@ public interface ITransformModel
/// <returns>The transform model as an <see cref="IRowToRowMapper"/>. If not all transforms
/// in the pipeline are <see cref="IRowToRowMapper"/> then it returns null.</returns>
IRowToRowMapper AsRowToRowMapper(IExceptionContext ectx);

/// <summary>
/// Get the loader information from the model.
/// </summary>
IDataView GetLoader();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @zeahmed , thanks for beginning to look at this.

I don't consider this an appropriate addition to ITransformModel. Transform models are for, well, transforms. :) There's also absolutely no reason to believe that a loader will be involved at all, since many usages of source data are programmatic.

The problem with the current API is that it did not account for the presence of a loader-model at all, and this change does not seem to address that core issue.

}
}
7 changes: 4 additions & 3 deletions src/Microsoft.ML.Data/EntryPoints/ScoreModel.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,9 @@ public static Output MakeScoringTransform(IHostEnvironment env, ModelInput input

IPredictor predictor;
RoleMappedData data;
var emptyData = new EmptyDataView(host, input.PredictorModel.TransformModel.InputSchema);
input.PredictorModel.PrepareData(host, emptyData, out data, out predictor);
var loader = input.PredictorModel.TransformModel.GetLoader();
var dataview = loader ?? new EmptyDataView(host, input.PredictorModel.TransformModel.InputSchema);
input.PredictorModel.PrepareData(host, dataview, out data, out predictor);

IDataView scoredPipe;
using (var ch = host.Start("Creating scoring pipeline"))
Expand All @@ -147,7 +148,7 @@ public static Output MakeScoringTransform(IHostEnvironment env, ModelInput input
return new Output
{
ScoredData = scoredPipe,
ScoringTransform = new TransformModel(host, scoredPipe, emptyData)
ScoringTransform = new TransformModel(host, scoredPipe, dataview)
};
}
}
Expand Down
18 changes: 16 additions & 2 deletions src/Microsoft.ML.Data/EntryPoints/TransformModel.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ public TransformModel(IHostEnvironment env, IDataView result, IDataView input)
env.CheckValue(result, nameof(result));
env.CheckValue(input, nameof(input));

var root = new EmptyDataView(env, input.Schema);
var root = input is IDataLoader ? input : new EmptyDataView(env, input.Schema);
_schemaRoot = root.Schema;
_chain = ApplyTransformUtils.ApplyAllTransformsToData(env, result, root, input);
}
Expand Down Expand Up @@ -171,7 +171,7 @@ public void Save(IHostEnvironment env, Stream stream)
using (var rep = RepositoryWriter.CreateNew(stream, ch))
{
ch.Trace("Saving root schema and transformations");
TrainUtils.SaveDataPipe(env, rep, _chain, blankLoader: true);
TrainUtils.SaveDataPipe(env, rep, _chain, blankLoader: false);
rep.Commit();
}
ch.Done();
Expand All @@ -186,6 +186,20 @@ public IRowToRowMapper AsRowToRowMapper(IExceptionContext ectx)
: null;
}

public IDataView GetLoader()
{
// Find the root schema.
for (IDataView view = _chain; ;)
{
var xf = view as IDataTransform;
if (xf == null)
{
return view;
}
view = xf.Source;
}
}

private sealed class CompositeRowToRowMapper : IRowToRowMapper
{
private readonly IDataView _chain;
Expand Down