ML.NET Asks [Tracking]

[JakeRadMSFT]

ML.NET

P0.5

• Object Detection Support

  • Description: Unlike image classification in ML.NET, which provides the high-level API for image classification training via transfer learning, object detection only supports consumption of ONNX and TF models. Thus, we'd like to add support for object detection training. This will also help keep parity with Azure and Local scenarios in Model Builder, since we are currently building out the Azure object detection experience.

• Update Image Classification API so we can ship GPU training

  • Description: Cuda 10.0 is not supported by NVidia anymore. The installer installs a driver that has an invalid signature, which installs a bad graphics driver. Our instructions take the user down a bad path to failure. There are several options we can explore to fix this issue:
  • Option A: We can rebuild TF targeting Cuda 10.1 and use the same version of TF.
  • Option B: We can upgrade our TF version to one that is already targeting Cuda 10.1
  • Option C: Change to PyTorch low-level implementation, which supports Cuda 10.1

P1

• Change Multi-Class Model Output to provide all labels and scores or provide helper

P2

• SoftMax Transform
  • We're currently using a custom mapping
• Normalizer Transform
  • We're currently using a custom mapping
  • I think ML.NET has one but it doesn't do what we need.
• Pixel Format Converter Transform ( 0-255, 0-255, 0-255 ) ->(0-1, 0-1, 0-1)
  • We're currently using a custom mapping

alternatively, if the following operator is supported by ExpressionTransformer, the first three custom transform can be replaced.

• Sum ( For SoftMax Transform, the expression that is not supported is x => exp(x)/sum(exp(x))
• index slice/index selection (for Normalizer Transform), the expression is

x => with(
 x[: , 0] = (x[:,0]-mean1)/var1;
 x[: , 1] = (x[:,1]-mean2)/var2;
 x[: , 2] = (x[:,2]-mean3)/var3;
x)

• Image Resize that doesn't depend on System.Drawing
  • System.Drawing isn't supported by UWP apps

Customer Hit Issues

P0

• Fix Text Loader - In Progress
  • Description: ML.NET text loader does not currently handle all CSV and TSV values for strings in quotes or new lines / escaped characters.
  • Example datasets that fail to parse due to text loader issues: jigsaw, Airbnb

AutoML.NET

P0

• Add AutoML.NET Support for all supported ML.NET Scenarios

  • Time Series Forecasting
  • Anomaly Detection
  • Ranking

• Improve Text Classification

  • Description: Currently, ML.NET only tries the default FeaturizeText transformer which doesn't work that well with free-form strings (it's ok with categories). This can be improved if AutoML.NET tries out multiple text featurizers to find the best one for the user's data and scenario.

P0.5

• Add AutoML.NET Support for
  • Object Detection

P1

• Save/Restore AutoML State
  • Add More Time
  • Resume Training
• Shows trainer's sweeping parameters when iterating.
• Improve Recommendation P1
  • Only tries MatrixFactorization and that type requires complete retraining for new items/users.
• Improve Image Classification P1
  • Only tries one DNN Type.
• Improve Time Series Forecasting (Once added) P1
  • Rolling origin cross validation
    - Time Series machinelearning#978
    - Rolling Cross-validation for Time-series machinelearning#1026

Customer Hit Issues

P2

• Inspect AutoML parameter choices (Inspect AutoML parameter choices #688)
• A dropdown list to choose what OptimizingMetric to use in AutoML GUI (Feature: A dropdown list to choose what OptimizingMetric to use in AutoML GUI #695)
• [AutoML,CLI,ModelBuilder] Exploration Time vs Epochs + Loss/AUC (Enhancement) ([AutoML,CLI,ModelBuilder] Exploration Time vs Epochs + Loss/AUC (Enhancement) #699)

[gvashishtha]

@JakeRadMSFT can you or @briacht add priorities for AutoML.NET asks? I tried to add priorities I could find in the Azure DevOps epic, but there were some missing.

[JakeRadMSFT]

@gvashishtha done

[JakeRadMSFT]

Things to investigate before adding to asks:

• Easier way to Save/Load custom mappings or ways to avoid them
  • ONNX Custom Mapping? Limit to ONNX serializable operations so there isn't a DLL dependency.
  • Offer simple base operations like divide/multiply/resize to avoid the need for custom mappings.
• Blazor Support
• Add additional metrics to match Azure AutoML
  • Forecasting: MAPE, MASE_single_grain, SMAPE.
  • Multi-Class: AUC

[justinormont]

Here's a few feature engineering techniques that could be added as AutoML experts.

Useful feature engineering techniques: (only light ordering)

• Per row Feature Selection: [text]

  xf=CSharp{in=Label out=LabelR4:R4 code={O.LabelR4 = ( I.Label == 0 ? Single.NaN : I.Label - 1 );}} 
  xf=TrainScore{scorer=wtf tr=FastForestRegression{shuffleLabels=+ nl=20 iter=100} feat=Features 
  lab=LabelR4}
  xf=Concat{col=Features:FeatureContributions}

• TargetEncoding w/ a linear model on text ngrams: [text]

  xf=TrainScore{tr=OVA{p=AveragedPerceptron{iter=10}} feat=FeaturesText lab=Label}

• WordEmbeddings + ngrams: [text]

  xf=TextTransform{col=FeaturesText:text tokens=+ wordExtractor=NGramExtractorTransform{ngram=2} charExtractor=NGramExtractorTransform{ngram=3}}
  xf=WordEmbeddingsTransform{col=FeaturesWordEmbeddings:FeaturesText_TransformedText model=$WORDEMBEDDING1$}
  xf=Concat{col=Features:FeaturesText,FeaturesWordEmbeddings} 

• TreeFeat w/ Leaves: [text, numeric, categorical]

  xf=TreeFeat{tr=FastForestRegression{shuffleLabels=+ nl=20 iter=100}}
  xf=Concat{col=Features:Leaves} 

• KMeans distance: [numeric, categorical]

  xf=TrainScore{tr=KMeansPlusPlus{k=10 init=Random} feat=FeaturesNumeric} 

• Binning + Merging + OneHot: [numeric]

  xf=BinNormalizer{col=XBin:X col=YBin:Y bins=512}
  xf=Expression{col=FeaturesLatLng2DBin:XBin,YBin expr={(x,y) : concat(text(x), "_", text(y))}}
  xf=CategoricalTransform{col=FeaturesLatLng2DBin}

• Dracula: [categorical]

  xf=DraculaTransform{col=VendorIDDracula:VendorID col=PaymentTypeDracula:PaymentType col=StoreAndFwdFlagDracula:StoreAndFwdFlag label=Label}
  xf=Concat{col=FeaturesDraculaOnCategorical:VendorIDDracula,PaymentTypeDracula,StoreAndFwdFlagDracula}

  DRACuLa is called CountTargetEncoder in ML․NET, though the PR is still pending.

• Binary Words Exists or Not: [text]

  xf=TextTransform{col=FeaturesText:text punc=- num=- wordExtractor=NGramExtractorTransform{ngram=1} charExtractor={} norm=None}
  xf=Expression{col=FeaturesTextUniqWords:FeaturesText expr={x : min(x, 1)}}
  xf=CategoricalTransform{col=Features:FeaturesTextUniqWords kind=Bag termCol=0,1 sort=Value} 

• DateTimeTransform w/ polar transfomation: [date]

  xf=CSharp{in=Dates out=DateWeekday:R4 out=DateMonth:R4 out=DateHour:R4 out=DatePartOfDay:R4 out=DateCosTimeOfDay:R4 out=DateSinTimeOfDay:R4 out=DateCosTimeOfWeek:R4 out=DateSinTimeOfWeek:R4 out=DateCosTimeOfMonth:R4 out=DateSinTimeOfMonth:R4 out=DateCosTimeOfYear:R4 out=DateSinTimeOfYear:R4 code={DateTime dt; var badDate = !DateTime.TryParse(I.Dates, out dt); var percentOfDay = (badDate ? Single.NaN : (float)(new TimeSpan(dt.Hour, dt.Minute, 0).TotalMinutes) / 1440); var percentOfWeek = (badDate ? Single.NaN : ((float)dt.DayOfWeek + percentOfDay) / 7f); var percentOfMonth = (badDate ? Single.NaN : (float)(dt.Day + percentOfDay) / (float)DateTime.DaysInMonth(dt.Year, dt.Month)); var percentOfYear = (badDate ? Single.NaN : (float)((dt.DayOfYear + percentOfDay) / (new DateTime(dt.Year+1, 1, 1 ) - new DateTime(dt.Year, 1, 1 )).TotalDays)); O.DateWeekday = (badDate ? Single.NaN : (float)dt.DayOfWeek); O.DateMonth = (badDate ? Single.NaN : (float)dt.Month); O.DateHour = (badDate ? Single.NaN : dt.Hour); O.DatePartOfDay = (badDate ? Single.NaN : (float)Math.Round(dt.AddHours(-3).Hour/6f)); O.DateCosTimeOfDay = (float)Math.Cos(2*Math.PI*percentOfDay); O.DateSinTimeOfDay = (float)Math.Sin(2*Math.PI*percentOfDay); O.DateCosTimeOfWeek = (float)Math.Cos(2*Math.PI*percentOfWeek); O.DateSinTimeOfWeek = (float)Math.Sin(2*Math.PI*percentOfWeek); O.DateCosTimeOfMonth = (float)Math.Cos(2*Math.PI*percentOfMonth); O.DateSinTimeOfMonth = (float)Math.Sin(2*Math.PI*percentOfMonth); O.DateCosTimeOfYear = (float)Math.Cos(2*Math.PI*percentOfYear); O.DateSinTimeOfYear = (float)Math.Sin(2*Math.PI*percentOfYear);}} 
  
  xf=CategoricalTransform{col=DateWeekdayCat:DateWeekday col=DateMonthCat:DateMonth col=DateHourCat:DateHourcol=DatePartOfDayCat:DatePartOfDay}
  xf=Concat{col=FeaturesDate:DateWeekday,DateMonth,DateHour,DatePartOfDay,DateCosTimeOfDay,DateSinTimeOfDay,DateCosTimeOfWeek,DateSinTimeOfWeek,DateCosTimeOfMonth,DateSinTimeOfMonth,DateCosTimeOfYear,DateSinTimeOfYear,DateWeekdayCat,DateMonthCat,DateHourCat,DatePartOfDayCat} 

  ML․NET has a DateTime transform, which will do most of this. Main missing component is the polar transforms.

• TrainScore to alternative labels: [all types]

  xf=TrainScore{tr=LightGBMMulticlass lab=AltLabel}
  WordEmbeddings + TrainScore-NN (or LightGBM): _[text]_
  xf=TextTransform{col=Tokens:text tokens=+ wordExtractor={} charExtractor={}}
  xf=WordEmbeddingsTransform{col=FeaturesWordEmbeddings:Tokens_TransformedText model=$WORDEMBEDDING1$}
  xf=TrainScore{tr=MultiClassNeuralNetwork{hidden=$K1$} feat=FeaturesWordEmbeddings} 
  xf=Concat{col=Features:Score} 

• Learner Feature Selection FastForestRegression: [all types]

  xf=CSharp{in=Label out=LabelR4:R4 code={O.LabelR4 = ( I.Label == 0 ? Single.NaN : I.Label - 1 );}}
  xf=LearnerFeatureSelectionTransform{topk=1000 f=FastForestRegression{shuffleLabels=+ iter=100} lab=LabelR4} 

• Text Statistics (small -- 5 stats): [text]

  xf=CSharp{in=text out=length:R4 out=vowelCount:R4 out=consonantCount:R4 out=numberCount:R4 out=underscoreCount:R4 out=letterCount:R4 out=startsWithVowel:R4 out=endsInVowel:R4 out=endsInVowelNumber:R4 out=maxRepeatingChar:R4 out=maxRepeatingVowel:R4 out=lowerCaseCount:R4 out=upperCaseCount:R4 out=upperCasePercent:R4 out=letterPercent:R4 out=numberPercent:R4 out=longestRepeatingChar:R4 out=longestRepeatingVowel:R4 initFile=FeaturesStringStatistics.init.cs codeFile=FeaturesStringStatistics.body.cs}
  xf=Concat{col=Features:length,maxRepeatingChar,upperCasePercent,letterPercent,numberPercent}

  Referenced C# files:
  FeaturesStringStatistics.init.cs.txt
  FeaturesStringStatistics.body.cs.txt

• Text Statistics (18 stats): [text]

  xf=CSharp{in=text out=length:R4 out=vowelCount:R4 out=consonantCount:R4 out=numberCount:R4 out=underscoreCount:R4 out=letterCount:R4 out=startsWithVowel:R4 out=endsInVowel:R4 out=endsInVowelNumber:R4 out=maxRepeatingChar:R4 out=maxRepeatingVowel:R4 out=lowerCaseCount:R4 out=upperCaseCount:R4 out=upperCasePercent:R4 out=letterPercent:R4 out=numberPercent:R4 out=longestRepeatingChar:R4 out=longestRepeatingVowel:R4 initFile=FeaturesStringStatistics.init.cs codeFile=FeaturesStringStatistics.body.cs} 
  xf=Concat{col=Features:length,vowelCount,consonantCount,numberCount,underscoreCount,letterCount,startsWithVowel,endsInVowel,endsInVowelNumber,maxRepeatingChar,maxRepeatingVowel,lowerCaseCount,upperCaseCount,upperCasePercent,letterPercent,numberPercent,longestRepeatingChar,longestRepeatingVowel}

These are written in ML․NET's MAML language. We'd have to translate to the Estimator API.

Specifically to address first improving text dataset, see techniques labeled text, or all types. The first step would be word embeddings.

[gvashishtha]

@justinormont can you help me understand what the impact of these improvements would be and how to prioritize among them? What would we get if we only implemented a subset of the [text] ones? Or just the date ones?

[justinormont]

@gvashishtha

Prioritization

The main of the engineering work is re-enabling sweeping of featurization techniques. After this, adding these additional experts is pretty simple. I would just start w/ the Word Embeddings perhaps also the TargetEncoding w/ a linear model.

For the date one, we can try sweeping over having/not-having the DateTime featurizer, though I expect we'll find it's more useful to simply always have it enabled, when we see a DateTime column.

Data science validation

The data science work is demonstrating that most users get to a better model in human centric time frames. Basically measuring how good of a model we can produce in 15min, a lunch break, an afternoon, come back tomorrow across a variety of datasets.

Expanding the search space causes the convergence rate to decrease; it'll take longer to find a good model as there are extra dimensions to search through.

Impact

The impact depends on the dataset.

Quick example w/ the DogBreeds_vs_Fruits_NLP.tsv.txt dataset:

Baseline: (bigram+trichar)
0.902582 +/- 0.012348 accuracy
  
With bigram+trichar + wordembedding (fastText300D):
0.970735 +/ 0.009212 accuracy

Moves from 9.7% incorrect guesses to 2.9% -- removes 70% of the incorrect predictions. Though at the cost of runtime. Note: I'm cherry picking as I know word embeddings work well on the dataset. Side note: we own and have CELA permission to redistribute this dataset, perhaps can use as a demo.

[mstfbl]

Hi @JakeRadMSFT and team,

This is related to the P0.5 - "Update Image Classification API so we can ship GPU training" ask. This task of updating to using CUDA v10.1 is not currently possible with the current NuGet package we use for GPU TensorFlow training (SciSharp.TensorFlow.Redist-Windows/Linux-GPU). The highest version of this NuGet package uses TensorFlow-GPU v1.14.1/v1.15.1, which targets CUDA v10.0, not v10.1. This is the result for when CUDA v10.1 is installed with the current NuGet packages, where it's trying to find CUDA v10.0 dlls:

tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found

TensorFlow-GPU v2.1.0 and above do support CUDA v10.1. If our TensorFlow-GPU NuGet packages (SciSharp.TensorFlow.Redist-Windows/Linux-GPU) can be updated to use TensorFlow-GPU v.2.1.0, then we can update to using CUDA 10.1. Else, we would need another solution.