Skip to content

Added Microsoft.ML.Benchmarks Project #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
May 15, 2018
Merged

Added Microsoft.ML.Benchmarks Project #62

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
1088d68
Benchmark
KrzysztofCwalina May 7, 2018
0f5ce0c
Changed to .NET Core app
KrzysztofCwalina May 8, 2018
1647ef5
Added Accuracy Reporting
KrzysztofCwalina May 8, 2018
2b163cf
fixed build
KrzysztofCwalina May 8, 2018
335ead1
Feedback from Gleb
KrzysztofCwalina May 9, 2018
cf144ab
Added batch prediction tests
KrzysztofCwalina May 14, 2018
63fcb8a
Resolved conflicts the sln file
KrzysztofCwalina May 14, 2018
8db49d0
Merge branch 'master' into Bench
KrzysztofCwalina May 14, 2018
0a281b2
Renamed the new file to match type name
KrzysztofCwalina May 14, 2018
b9b91e8
Removed duplicated method
KrzysztofCwalina May 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added Accuracy Reporting
  • Loading branch information
@KrzysztofCwalina
KrzysztofCwalina committed May 8, 2018
commit 1647ef59d4b33c582786eecda570bfea65fa7ca8
5 changes: 5 additions & 0 deletions test/Microsoft.ML.Benchmarks/Microsoft.ML.Benchmarks.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@
<TargetFramework>netcoreapp2.0</TargetFramework>
<StartupObject>Microsoft.ML.Benchmarks.Program</StartupObject>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this used for? I've never seen this in new SDK-style projects.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I remove it, it tells me there is more than one entry point and so the one real one must be specified. Possibly one of the BDN dlls has a Main method? @adamsitnik?

</PropertyGroup>
<ItemGroup>
<Compile Remove="BenchmarkDotNet.Artifacts\**" />
<EmbeddedResource Remove="BenchmarkDotNet.Artifacts\**" />
<None Remove="BenchmarkDotNet.Artifacts\**" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.10.14" />
</ItemGroup>
Expand Down
42 changes: 42 additions & 0 deletions test/Microsoft.ML.Benchmarks/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,17 @@
using BenchmarkDotNet.Diagnosers;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Toolchains.CsProj;
using BenchmarkDotNet.Toolchains.InProcess;
using System;
using System.IO;
using Microsoft.ML.Models;
using Microsoft.ML.Runtime.Api;
using Microsoft.ML.Trainers;
using Microsoft.ML.Transforms;
using Microsoft.ML.Benchmarks;

namespace Microsoft.ML.Benchmarks
{
Expand All @@ -30,6 +38,7 @@ private static IConfig CreateClrVsCoreConfig()
var config = DefaultConfig.Instance.With(
Job.ShortRun.
With(InProcessToolchain.Instance)).
With(new ClassificationMetricsColumn("AccuracyMacro", "Macro-average accuracy of the model")).
With(MemoryDiagnoser.Default);
return config;
}
Expand All @@ -45,4 +54,37 @@ static Program()
_dataRoot = Path.Combine(rootDir, "test", "data");
}
}


public class ClassificationMetricsColumn : IColumn
{
string _metricName;
string _legend;

public ClassificationMetricsColumn(string metricName, string legend)
{
_metricName = metricName;
_legend = legend;
}

public string ColumnName => _metricName;
public string Id => _metricName;
public string Legend => _legend;
public bool IsNumeric => true;
public bool IsDefault(Summary summary, Benchmark benchmark) => true;
public bool IsAvailable(Summary summary) => true;
public bool AlwaysShow => true;
public ColumnCategory Category => ColumnCategory.Custom;
public int PriorityInCategory => 1;
public UnitType UnitType => UnitType.Dimensionless;

public string GetValue(Summary summary, Benchmark benchmark, ISummaryStyle style)
{
var property = typeof(ClassificationMetrics).GetProperty(_metricName);
return property.GetValue(TrainPredictionBench.s_metrics).ToString();
}
public string GetValue(Summary summary, Benchmark benchmark) => GetValue(summary, benchmark, null);

public override string ToString() => ColumnName;
}
}
18 changes: 16 additions & 2 deletions test/Microsoft.ML.Benchmarks/TrainPredictionBench.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,28 @@
// See the LICENSE file in the project root for more information.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using Microsoft.ML.Models;
using Microsoft.ML.Runtime.Api;
using Microsoft.ML.Trainers;
using Microsoft.ML.Transforms;


namespace Microsoft.ML.Benchmarks
{
[KeepBenchmarkFiles]
public class TrainPredictionBench
Copy link
Contributor

@glebuk glebuk May 9, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TrainPredictionBench [](start = 17, length = 20)

Split benchmarks by task (regression, binary, etc). You already have classification metrics as a field. Perhaps have a base class and then subclasses for each task. #Closed

Copy link
Member Author

@KrzysztofCwalina KrzysztofCwalina May 9, 2018 •
edited by glebuk
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed the test to StochasticDualCoordinateAscentClassifierBench #Closed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the file name be updated too?

In reply to: 187092340 [](ancestors = 187092340)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please update the file name to match the class name?

In reply to: 187243814 [](ancestors = 187243814,187092340)

{
internal static ClassificationMetrics s_metrics;
Copy link
Contributor

@glebuk glebuk May 9, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nternal static ClassificationMetrics s_metrics; [](start = 9, length = 47)

Would also be awesome to figure out how to minimize the amount of code per dataset. so that you can just add a dataset and then not have to change much else
Also, it should be good to be able to quickly and easily substitute different learners.
Basically the dimensions for tests are:
Training:
Task:
Learner
Hyper-parameters{ detault, sweep}
Dataset
Subset by size (like 10%, 50%, 100% of rows)
Inference:
Pipeline load time
Time per prediction #Pending

Copy link
Member Author

@KrzysztofCwalina KrzysztofCwalina May 9, 2018 •
edited by glebuk
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is a good idea, but I think such generalizations are best done when actually trying to develop the second, third, etc. test. i.e. I will try to do it when I am adding more tests. #Closed

static PredictionModel<IrisData, IrisPrediction> s_trainedModel;

[GlobalCleanup]
public void Accuracy()
{
var dataPath = Program.GetDataPath("iris.txt");
var testData = new TextLoader<IrisData>(dataPath, useHeader: true, separator: "tab");
var evaluator = new ClassificationEvaluator();
s_metrics = evaluator.Evaluate(s_trainedModel, testData);
}

[Benchmark]
public void Iris()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible for us to mark a bunch of integration tests as benchmarks in this way?

Copy link
Member Author

@KrzysztofCwalina KrzysztofCwalina May 8, 2018 •
edited by glebuk
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They would need to be a part of an exe project (like this one). But maybe we can try to expose the integration tests in public APIs (of the test project) and then just run them from a benchmark stub in this project. @adamsitnik? #Pending

Copy link
Contributor

@glebuk glebuk May 10, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now there are at least two possible ways to do this: Create those items as unit tests (comment above) OR, use a general-purpose command line tool to execute specific pipelines.
For the second scenario - command line tool:
It seems that such command tool would be great to have in contexts outside of benchmarking. Does it really make sense to create a benchmarking-only tool?
Why not simply port a MaML.EXE and create a bunch of *.RSP files to define each benchnarks? That would save us a lot of time developing this AND would give users a CLI for ML.NET
Opened issue #108 to track this.

In reply to: 186589537 [](ancestors = 186589537)

{
Expand Down Expand Up @@ -50,6 +62,8 @@ public void Iris()
PetalLength = 1.2f,
PetalWidth = 4.4f,
});

s_trainedModel = model;
}

public class IrisData
Expand Down