Skip to content

Added RankingEvaluatorOptions and removed the truncation limit. #4081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 58 commits into from
Oct 10, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
8749a10
Merge pull request #1 from dotnet/master
harishsk Jun 20, 2019
fe25bf6
Fixed build errors resulting from upgrade to VS2019 compilers
harishsk Jun 21, 2019
cb446be
Added additional message describing the previous fix
harishsk Jun 21, 2019
b5ee220
Merge pull request #2 from dotnet/master
harishsk Jun 21, 2019
b9a7471
Merge pull request #3 from dotnet/master
harishsk Jun 24, 2019
80e238d
Merge pull request #4 from dotnet/master
harishsk Jun 27, 2019
2ef424d
Merge pull request #5 from dotnet/master
harishsk Jul 9, 2019
3958f01
Merge pull request #6 from dotnet/master
harishsk Aug 7, 2019
56d4595
Fixed #3993
harishsk Aug 7, 2019
00bc7ef
Merge pull request #7 from dotnet/master
harishsk Aug 14, 2019
d0462f1
Merge pull request #8 from dotnet/master
harishsk Aug 16, 2019
87cefbc
Merge branch 'master' into bugfix_3993
harishsk Aug 16, 2019
c3a908b
Reverted previous change and added a separate class to control evalua…
harishsk Aug 21, 2019
c0a430a
Merge pull request #9 from dotnet/master
harishsk Aug 21, 2019
0b55903
Syncing upstream fork (#10)
harishsk Aug 30, 2019
56983d5
Syncing upstream fork (#11)
harishsk Aug 30, 2019
3382d1d
Merge pull request #13 from dotnet/master
harishsk Sep 6, 2019
8ca5d01
Merge remote-tracking branch 'upstream/master'
harishsk Sep 10, 2019
4ac459e
Added unit test for ranking evaluation with options
harishsk Sep 11, 2019
8f20ea4
Merge remote-tracking branch 'upstream/master'
harishsk Sep 11, 2019
f9f9e1d
Changed visibility of OutputGroupSummary to internal until we expose …
harishsk Sep 11, 2019
21cb8f3
Added a unit test for maml ranking evaluation
harishsk Sep 12, 2019
138f201
Merge remote-tracking branch 'upstream/master'
harishsk Sep 13, 2019
55e3460
Merge remote-tracking branch 'upstream/master'
harishsk Sep 13, 2019
e43bba3
Merge remote-tracking branch 'upstream/master'
harishsk Sep 16, 2019
421d713
Merge branch 'master' of ssh://github.com/harishsk/machinelearning
harishsk Sep 16, 2019
4f4f81c
Merge remote-tracking branch 'upstream/master'
harishsk Sep 17, 2019
89082a5
Merge branch 'master' into bugfix_3993
harishsk Sep 17, 2019
f167af8
Merge branch 'master' of ssh://github.com/harishsk/machinelearning
harishsk Sep 17, 2019
0d4d34f
getting rid of maxTruncationLevel
Lynx1820 Sep 18, 2019
6cd2f15
removed unnecessary imports
Lynx1820 Sep 18, 2019
1424ab3
removed old comment
Lynx1820 Sep 18, 2019
3ee03ca
removed old comment
Lynx1820 Sep 18, 2019
34b7a91
Merge remote-tracking branch 'upstream/master'
harishsk Sep 19, 2019
5539127
Merge remote-tracking branch 'upstream/master'
harishsk Sep 19, 2019
02053a6
adding a maml baseline test
Lynx1820 Sep 23, 2019
35ad3c0
reformatted maml test and moved to where all the other maml tests are
Lynx1820 Sep 25, 2019
0eb3e2b
Reverted back some spacing. Accidentally reverted some changes in Eva…
Lynx1820 Sep 25, 2019
a3291b1
added more relaxed precision
Lynx1820 Sep 25, 2019
37af437
Merge remote-tracking branch 'upstream/master'
harishsk Sep 26, 2019
68f1f35
baseline precision set 1
Lynx1820 Sep 26, 2019
5b90a34
added more helpful debugging comment
Lynx1820 Sep 26, 2019
0efe238
temp
Lynx1820 Sep 27, 2019
b6584aa
Merge remote-tracking branch 'upstream/master'
harishsk Sep 27, 2019
7d47832
Merge branch 'master' of ssh://github.com/harishsk/machinelearning
harishsk Sep 27, 2019
0e99776
Merge branch 'master' into bugfix_3993
harishsk Sep 27, 2019
20a4490
Revert "temp"
Lynx1820 Sep 30, 2019
0d111f4
Revert "added more relaxed precision"
Lynx1820 Sep 30, 2019
72d1a4d
using fastRankRanking to test RankingEvaluator instead of lightgbm
Lynx1820 Sep 30, 2019
ea9ebed
Merge branch 'bugfix_3993' of git://github.com/harishsk/machinelearni…
Lynx1820 Sep 30, 2019
013be4f
reverted some indenting and LightGBM imports
Lynx1820 Sep 30, 2019
d2ae365
removed the stratification flag to get real numbers
Lynx1820 Oct 7, 2019
a9e6db8
testcase change added
Lynx1820 Oct 7, 2019
5855f99
Added the Bestfriend attribute back
Lynx1820 Oct 7, 2019
724bb12
Changed implementation of discount map computation to minimize mutabl…
harishsk Oct 9, 2019
d009f55
Added doc strings
harishsk Oct 9, 2019
8f7b6cd
Fixed bug in creation of fixed discount map
harishsk Oct 9, 2019
30d56a0
Merge remote-tracking branch 'upstream/master' into bugfix_3993
harishsk Oct 10, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
adding a maml baseline test
  • Loading branch information
Lynx1820 committed Sep 23, 2019
commit 02053a6bb40e65c317e6faf343fe8ac2084bf552
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#@ TextLoader{
#@ header+
#@ sep=tab
#@ col=NDCG:R8:0-9
#@ col=DCG:R8:10-19
#@ col={name={Fold Index} type=TX src=20}
#@ }
NDCG@1 NDCG@2 NDCG@3 NDCG@4 NDCG@5 NDCG@6 NDCG@7 NDCG@8 NDCG@9 NDCG@10 DCG@1 DCG@2 DCG@3 DCG@4 DCG@5 DCG@6 DCG@7 DCG@8 DCG@9 DCG@10 Fold Index
0.47412491562447456 0.4719357410981887 0.47543961782034327 0.48089345697106028 0.48776440826942524 0.49343209460325765 0.49879601065292078 0.50419344126796073 0.50933717558985403 0.51452470414727791 9.7472027801286938 14.182900214917305 17.149757037474153 19.508035827118427 21.532449245734401 23.266676880999444 24.808797414614865 26.196851963285635 27.463201584793453 28.628843617965828 Average
0.001592283158454337 0.0011332986839019653 0.0004153153408829224 0.00054118433592205883 0.00017552254806811829 6.3737418432819389E-05 0.00068125927701142685 0.00029595376508887149 5.828811104620942E-05 5.8705420456141367E-05 0.032268867133279942 0.05073621600715323 0.027242433849222782 0.0085466158360746735 0.018045238326612048 0.044258533338137479 0.045093838642232098 0.061173773545056688 0.073040019424083974 0.07607796723154947 Standard Deviation
0.47571719878292618 0.47306903978207127 0.47585493316122129 0.48143464130699215 0.48793993081758941 0.49349583202184949 0.49947726992992481 0.50448939503299484 0.50939546370071986 0.51458340956802318 9.7794716472619427 14.233636430924651 17.176999471321892 19.516582442956842 21.514404007406711 23.222418347660543 24.763703575971959 26.135678189739878 27.390161565369567 28.55276565073514 Fold 0
0.47253263246602295 0.47080244241430619 0.47502430247946525 0.48035227263512842 0.48758888572126102 0.49336835718466576 0.49811475137591676 0.50389748750292673 0.5092788874789882 0.51446599872653265 9.7149339129954448 14.132163998909959 17.122514603626417 19.499489211280007 21.550494484062092 23.310935414338346 24.853891253257768 26.258025736831392 27.536241604217338 28.704921585196519 Fold 1
14 changes: 9 additions & 5 deletions test/Microsoft.ML.Functional.Tests/Evaluation.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System.IO;
using Microsoft.ML.Data;
using Microsoft.ML.Functional.Tests.Datasets;
using Microsoft.ML.RunTests;
using Microsoft.ML.TestFramework;
using Microsoft.ML.TestFramework.Attributes;
using Microsoft.ML.Tools;
using Microsoft.ML.Trainers;
Expand All @@ -15,7 +15,7 @@

namespace Microsoft.ML.Functional.Tests
{
public class Evaluation : BaseTestClass
public class Evaluation : BaseTestBaseline
{
public Evaluation(ITestOutputHelper output): base(output)
{
Expand Down Expand Up @@ -219,23 +219,27 @@ public void TrainAndEvaluateRankingWithOptions()
Common.AssertMetrics(metrics);
}


}

[Fact]
public void EvaluateRankingWithMaml()
{
string _mslrWeb10k_Train = GetDataPath(TestDatasets.MSLRWeb.trainFilename);

string filename = "CommandTrainRanking-summary2.txt";
string filepath = GetOutputPath(filename);
var subDir = Path.Combine("..", "..", "BaselineOutput", "Common", "Command");
string cmd = @"CV tr=LightGBMRanking" +
" eval=RankingEvaluator{t=10}" +
" k=2" +
" loader=TextLoader{col=Label:R4:0 col=GroupId:TX:1 col=Features:R4:2-138 header=+}" +
" data=" + _mslrWeb10k_Train +
" xf = HashTransform{col=GroupId}" +
" xf = NAHandleTransform{col=Features}";
" xf = NAHandleTransform{col=Features}" +
" summaryFilename=" + filepath;

Assert.Equal(0, Maml.Main(new[] { cmd }));

CheckEquality(subDir,filename);
}

/// <summary>
Expand Down