Forecasting model framework for time series. #1900

codemzs · 2018-12-18T01:14:40Z

This PR introduces forecasting framework/interface for time series. It allows the following:

Train forecasting model from an IDataView.
Update the model with new observation using an IDataView.
Forecast values up to a certain horizon (with confidence intervals).
Checkpoint the model to disk.
Load a model from disk.

fixes #929
fixes #3151

shauheen

Not addressing an issue.

codemzs · 2018-12-18T04:40:50Z

@shauheen this PR addresses forecasting capability requested in issue #929 #Resolved

shauheen · 2018-12-18T04:43:31Z

@codemzs That issue has not been triaged and does not have a clear design. #Resolved

It is not triaged for 1.1 release.

test/Microsoft.ML.TimeSeries.Tests/TimeSeriesDirectApi.cs

ganik · 2019-05-30T05:17:21Z

    /// Train a forecasting model from an <see cref="IDataView"/>.

Is there an entrypoint for this? This would be needed for NimbusML #Resolved

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:1631 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

ganik · 2019-05-30T05:21:42Z

    public float[] Forecast(int horizon) => _modeler.Forecast(horizon);

There should be API that does not only single point forecast but also gives area forecast based on confidence interval provided #Resolved

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:1649 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

artidoro · 2019-05-30T16:51:40Z

    public enum RankSelectionMethod

Can you add comments for public fields? #Resolved

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:31 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

artidoro · 2019-05-30T16:52:20Z

Same here if you need these to be public, can you add documentation? #Resolved

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:39 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

artidoro · 2019-05-30T16:54:41Z

    public AdaptiveSingularSpectrumSequenceForecastingModeler(IHostEnvironment env, int trainSize, int seriesLength, int windowSize, Single discountFactor = 1,

We have moved away from having public constructors. All our transforms/trainers are found through MLContext. We should follow the same pattern here I believe. #Resolved

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:83 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

src/Microsoft.ML.TimeSeries/Forecast.cs

codemzs · 2019-05-31T16:53:38Z

    /// Train a forecasting model from an <see cref="IDataView"/>.

We spoke offline and agreed there is a separate work item for entrypoints in time series,

In reply to: 497205979 [](ancestors = 497205979)

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:1631 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

codemzs · 2019-05-31T21:07:24Z

    public float[] Forecast(int horizon) => _modeler.Forecast(horizon);

Agreed but currently we don't have confidence intervals. I can check with Saeed and see if we can add later.

In reply to: 497206778 [](ancestors = 497206778)

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:1649 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

ganik · 2019-05-31T21:17:20Z

    public AdaptiveSingularSpectrumSequenceForecastingModeler(IHostEnvironment env, int trainSize, int seriesLength, int windowSize, Single discountFactor = 1,

+1

In reply to: 497401572 [](ancestors = 497401572)

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:83 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

ganik · 2019-05-31T21:18:31Z

    public float[] Forecast(int horizon) => _modeler.Forecast(horizon);

see this method ComputeForecastIntervals in the modeler class

In reply to: 497860420 [](ancestors = 497860420,497206778)

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:1649 in 5a29c42. [](commit_id = 5a29c42, deletion_comment = False)

yaeldekel · 2019-05-31T21:47:48Z

    internal sealed class AdaptiveSingularSpectrumSequenceModeler : SequenceModelerBase<Single, Single>

Can it be private? #Resolved

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:135 in 47e5f22. [](commit_id = 47e5f22, deletion_comment = False)

codemzs · 2019-05-31T22:10:27Z

    internal sealed class AdaptiveSingularSpectrumSequenceModeler : SequenceModelerBase<Single, Single>

It can't because it is used in SSABase class as well.

In reply to: 497870543 [](ancestors = 497870543)

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:135 in 47e5f22. [](commit_id = 47e5f22, deletion_comment = False)

…e from master.

codemzs · 2019-06-03T20:22:38Z

    /// </summary>

We discussed offline and decided to open an issue for this.

In reply to: 498346346 [](ancestors = 498346346)

Refers to: src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs:36 in 2950dfa. [](commit_id = 2950dfa, deletion_comment = False)

CESARDELATORRE · 2019-06-03T20:30:00Z

Question: Why is this 'Time Series Forecasting' algorithm/component not aligned to the rest of trainers in ML.NET?

Here, with 'Time Series Forecasting' you do the following:

You don't provide an specific trainer to the pipeline but run: ml.Forecasting.AdaptiveSingularSpectrumSequenceModeler()
You don't create a Prediction Engine when scoring but call 'model.Forecast()'
You don't call .Transform() when predicting in bulk, instead you call model.Forecast()
Also, saving and loading models into a .ZIP file is different (singular specific API implementation):
ml.Model.SaveForecastingModel() and ml.Model.LoadForecastingModel

I think this 'singularity' in the API might cause some issues..
For instance, you cannot use the Microsoft.Extensions.ML PredictionEnginePool with this 'Time Series Forecasting' component or any other typical ML.NET code. This API is completely different.

Is it possible to shape its API on a similar way that other algorithms/trainers are in ML.NET?

codemzs · 2019-06-03T20:33:40Z

@CESARDELATORRE The question you should ask yourself is if time series forecasting behaves the same way as other trainers and transforms? I'll be happy to explain the rationale behind design choices and we have a month to make any changes if needed before we make this GA.

CESARDELATORRE · 2019-06-03T20:53:14Z

Yes please, I'd like you to explain the rationale behind the API design here. If possible the internal implementation shouldn't surface to the external API interfaces. For instance, why do we have a different API interface for Loading/Saving to .ZIP file?: ml.Model.SaveForecastingModel() and ml.Model.LoadForecastingModel(). Let's meet to chat and discuss about it, ok? :)

codemzs · 2019-06-03T20:55:15Z

@CESARDELATORRE sounds good.

rdzidziguri · 2019-06-04T10:41:03Z

@CESARDELATORRE I found some API designs relatively questionable as well please see #3151 (comment), but as long as this is highly requested feature and we needed it a lot, for now, we are good to use this approach while keeping in mind that API surface might change in the future.

eerhardt · 2019-06-04T16:29:41Z

I agree that the new APIs introduced in this PR are drastically different (such as using different terms/words for what appear to be the same operations) from the existing ML.NET framework.

@codemzs - can you add a design document into the docs folder that describes why these forecasting APIs cannot fit into the existing framework of estimator, tranformer, data #581?

eerhardt · 2019-06-04T17:22:49Z

src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs

        public struct GrowthRatio
        {
            private int _timeSpan;
            private Double _growth;

+            /// <summary>
+            /// Time span of growth ratio. Must be strictly positive.
+            /// </summary>
            public int TimeSpan


Why use an int here? Why not use a TimeSpan type, which avoids any confusion about what unit this int is? seconds? Minutes? days?

eerhardt · 2019-06-04T17:24:55Z

src/Microsoft.ML.TimeSeries/AdaptiveSingularSpectrumSequenceModeler.cs

-
+        /// <summary>
+        /// Growth ratio. Defined as Growth^(1/TimeSpan).
+        /// </summary>
        public struct GrowthRatio


Is using a mutable struct here the best API? Should we instead use a readonly struct? Or possibly make this a class.

shauheen previously requested changes Dec 18, 2018

View reviewed changes

codemzs requested a review from Zruty0 December 18, 2018 04:41

codemzs added the time series label Dec 18, 2018

shauheen removed the time series label Dec 18, 2018

codemzs closed this Dec 18, 2018

codemzs reopened this May 14, 2019

codemzs force-pushed the forecast branch from 6fed1b1 to 5a29c42 Compare May 29, 2019 22:28

codemzs requested review from artidoro, wschin and ganik and removed request for Zruty0 May 29, 2019 22:30

ganik reviewed May 30, 2019

View reviewed changes

test/Microsoft.ML.TimeSeries.Tests/TimeSeriesDirectApi.cs Outdated Show resolved Hide resolved

yaeldekel reviewed May 30, 2019

View reviewed changes

src/Microsoft.ML.TimeSeries/Forecast.cs Outdated Show resolved Hide resolved

yaeldekel reviewed May 30, 2019

View reviewed changes

src/Microsoft.ML.TimeSeries/Forecast.cs Show resolved Hide resolved

yaeldekel reviewed May 30, 2019

View reviewed changes

src/Microsoft.ML.TimeSeries/Forecast.cs Outdated Show resolved Hide resolved

codemzs force-pushed the forecast branch from 47e5f22 to 0a810d7 Compare May 31, 2019 23:04

codemzs added 18 commits June 3, 2019 11:45

Forecasting interface with a unit-test.

b10477a

Comments.

9348ba9

cleanup.

a53d743

cleanup.

04e7193

Internalize forecasting model and expose bare minimum needed and merg…

b2319fd

…e from master.

PR feedback.

feb85e5

PR feedback.

f57b2a3

PR feedback.

a34d8ea

Comments.

0d20e36

Samples.

c9811ed

Samples.

1394477

Comments.

a6fd250

Fix sample output.

850bb00

clean up.

273c0f8

clean up.

0d6feb4

docs and cleanup.

c435ef0

typo in doc.

9a1a840

PR feedback.

3afefde

codemzs force-pushed the forecast branch from 2950dfa to 3afefde Compare June 3, 2019 20:22

codemzs merged commit 380f167 into dotnet:master Jun 3, 2019

eerhardt reviewed Jun 4, 2019

View reviewed changes

ghost locked as resolved and limited conversation to collaborators Mar 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Forecasting model framework for time series. #1900

Forecasting model framework for time series. #1900

codemzs commented Dec 18, 2018 •

edited

Loading

shauheen left a comment

codemzs commented Dec 18, 2018 •

edited

Loading

shauheen commented Dec 18, 2018 •

edited by codemzs

Loading

ganik commented May 30, 2019 •

edited by codemzs

Loading

ganik commented May 30, 2019 •

edited by codemzs

Loading

artidoro commented May 30, 2019 •

edited by codemzs

Loading

artidoro commented May 30, 2019 •

edited by codemzs

Loading

artidoro commented May 30, 2019 •

edited by codemzs

Loading

codemzs commented May 31, 2019

codemzs commented May 31, 2019

ganik commented May 31, 2019

ganik commented May 31, 2019

yaeldekel commented May 31, 2019 •

edited by codemzs

Loading

codemzs commented May 31, 2019

codemzs commented Jun 3, 2019

CESARDELATORRE commented Jun 3, 2019

codemzs commented Jun 3, 2019 •

edited

Loading

CESARDELATORRE commented Jun 3, 2019 •

edited

Loading

codemzs commented Jun 3, 2019

rdzidziguri commented Jun 4, 2019

eerhardt commented Jun 4, 2019

eerhardt Jun 4, 2019

eerhardt Jun 4, 2019

Forecasting model framework for time series. #1900

Forecasting model framework for time series. #1900

Conversation

codemzs commented Dec 18, 2018 • edited Loading

shauheen left a comment

Choose a reason for hiding this comment

codemzs commented Dec 18, 2018 • edited Loading

shauheen commented Dec 18, 2018 • edited by codemzs Loading

ganik commented May 30, 2019 • edited by codemzs Loading

ganik commented May 30, 2019 • edited by codemzs Loading

artidoro commented May 30, 2019 • edited by codemzs Loading

artidoro commented May 30, 2019 • edited by codemzs Loading

artidoro commented May 30, 2019 • edited by codemzs Loading

codemzs commented May 31, 2019

codemzs commented May 31, 2019

ganik commented May 31, 2019

ganik commented May 31, 2019

yaeldekel commented May 31, 2019 • edited by codemzs Loading

codemzs commented May 31, 2019

codemzs commented Jun 3, 2019

CESARDELATORRE commented Jun 3, 2019

codemzs commented Jun 3, 2019 • edited Loading

CESARDELATORRE commented Jun 3, 2019 • edited Loading

codemzs commented Jun 3, 2019

rdzidziguri commented Jun 4, 2019

eerhardt commented Jun 4, 2019

eerhardt Jun 4, 2019

Choose a reason for hiding this comment

eerhardt Jun 4, 2019

Choose a reason for hiding this comment

codemzs commented Dec 18, 2018 •

edited

Loading

codemzs commented Dec 18, 2018 •

edited

Loading

shauheen commented Dec 18, 2018 •

edited by codemzs

Loading

ganik commented May 30, 2019 •

edited by codemzs

Loading

ganik commented May 30, 2019 •

edited by codemzs

Loading

artidoro commented May 30, 2019 •

edited by codemzs

Loading

artidoro commented May 30, 2019 •

edited by codemzs

Loading

artidoro commented May 30, 2019 •

edited by codemzs

Loading

yaeldekel commented May 31, 2019 •

edited by codemzs

Loading

codemzs commented Jun 3, 2019 •

edited

Loading

CESARDELATORRE commented Jun 3, 2019 •

edited

Loading