Allow user to overwrite unknown shapes loaded from ONNX model #3963

wschin · 2019-07-03T16:24:09Z

This PR adds an extra argument to ApplyOnnxModel functions.

        /// <param name="shapeDictionary">ONNX shape should be used to over those loaded from <paramref name="modelFile"/>.</param>

If a key-value pair, (tensor_name, [1, 2, 3]) exists, the tensor_name's shape loaded from ONNX model file will be replaced by [1, 2, 3]. It's allows user to replace unknown shapes in their applications.

Fix #3932.

With this PR, FasterRCNN mentioned in #3932 can be run as the example below.

        private class RcnnInput
        {
            [ColumnName("image")]
            [VectorType(3, 224, 224)]
            public float[] Input { get; set; }
        }

        private class RcnnOutput
        {
            [ColumnName("6379")]
            public float[] Boxes { get; set; }

            [ColumnName("6381")]
            public long[] Label { get; set; } 

            [ColumnName("6383")]
            public float[] Score { get; set; }
        }

        static public IEnumerable<float> GenerateRandomVector(int size)
        {
            var rnd = new Random(0);

            for (int i = 0; i < size; ++i)
                yield return rnd.NextSingle();
        }

        /// <summary>
        /// A test to check if dynamic shape works.
        /// </summary>
        [OnnxFact]
        public void TestOnnxTransformDynamicShape()
        {
            var modelFile = Path.Combine(Directory.GetCurrentDirectory(), "fasterRCNN", "faster_rcnn_R_50_FPN_1x.onnx");

            var dataPoints = new RcnnInput[] {
                new RcnnInput() { Input = GenerateRandomVector(3 * 224 * 224).ToArray(), }
            };

            var shapeDictionary = new Dictionary<string, int[]>() { { "image", new int[] { 3, 224, 224 } } };

            var dataView = ML.Data.LoadFromEnumerable(dataPoints);
            var transformedDataView = ML.Transforms.ApplyOnnxModel(new[] { "6379", "6381", "6383" }, new[] { "image" }, modelFile,
                shapeDictionary: shapeDictionary).Fit(dataView).Transform(dataView);

            // Convert IDataView to IEnumerable<ZipMapOutput> and then inspect the values.
            var transformedDataPoints = ML.Data.CreateEnumerable<RcnnOutput>(transformedDataView, false).ToList();

            for (int i = 0; i < transformedDataPoints.Count; ++i)
            {
                Assert.NotNull(transformedDataPoints[i].Boxes);
                Assert.NotNull(transformedDataPoints[i].Label);
                Assert.NotNull(transformedDataPoints[i].Score);
            }
        }

wschin · 2019-07-03T16:28:47Z

src/Microsoft.ML.OnnxTransformer/OnnxTransform.cs

@@ -226,36 +328,17 @@ private static IRowMapper Create(IHostEnvironment env, ModelLoadContext ctx, Dat
        /// <param name="modelFile">Model file path.</param>
        /// <param name="gpuDeviceId">Optional GPU device ID to run execution on. Null for CPU.</param>
        /// <param name="fallbackToCpu">If GPU error, raise exception or fallback to CPU.</param>
-        internal OnnxTransformer(IHostEnvironment env, string modelFile, int? gpuDeviceId = null, bool fallbackToCpu = false)


This ctor is not used at all, so I removed it. #Resolved

Add a command line test Remove unused code

artidoro · 2019-07-09T17:58:17Z

src/Microsoft.ML.OnnxTransformer/OnnxCatalog.cs

-        => new OnnxScoringEstimator(CatalogUtils.GetEnvironment(catalog), modelFile, gpuDeviceId, fallbackToCpu);
+            bool fallbackToCpu = false,
+            IDictionary<string, int[]> shapeDictionary = null)
+        => new OnnxScoringEstimator(CatalogUtils.GetEnvironment(catalog), modelFile, gpuDeviceId, fallbackToCpu,


I think this should be a breaking change. I am afraid API compat is not active for this assembly. I will open an issue about it and fix it in the meantime.
Unfortunately the signature of the function changes if you add another parameter. The only way to do this change without being a breaking change is to add a new overload with the new parameter. #Resolved

There is already a PR on reactivating API compat, could you approve it and rebase afterwards?

In reply to: 301718819 [](ancestors = 301718819)

Sure. Thanks.

In reply to: 301768225 [](ancestors = 301768225,301718819)

artidoro · 2019-07-09T18:11:16Z

    private protected override void SaveModel(ModelSaveContext ctx)

Why are we not saving the ShapeDictionary here?
It seems like we will lose that information. #Resolved

Refers to: src/Microsoft.ML.OnnxTransformer/OnnxTransform.cs:307 in ced8c42. [](commit_id = ced8c42, deletion_comment = False)

artidoro · 2019-07-09T18:12:35Z

test/Microsoft.ML.OnnxTransformerTest/OnnxTransformTests.cs

+        public void TestOnnxTransformWithCustomShapes()
+        {
+            // The loaded model has input shape [-1, 3] and output shape [-1].
+            var modelFile = Path.Combine(Directory.GetCurrentDirectory(), "unknowndimensions", "test_unknowndimensions_float.onnx");


test_unknowndimensions_float [](start = 96, length = 28)

nit: Could we use CSharp style for file names?
If that was the original name of the model, maybe it's fine like this. #ByDesign

It's in another repo and I feel it looks just fine.

In reply to: 301724967 [](ancestors = 301724967)

artidoro · 2019-07-09T18:15:09Z

test/Microsoft.ML.OnnxTransformerTest/OnnxTransformTests.cs

+        public void TestOnnxTransformWithCustomShapes()
+        {
+            // The loaded model has input shape [-1, 3] and output shape [-1].
+            var modelFile = Path.Combine(Directory.GetCurrentDirectory(), "unknowndimensions", "test_unknowndimensions_float.onnx");


test_unknowndimensions_float [](start = 96, length = 28)

Could you add a link to the description of the onnx model this test is using? #Resolved

There is no such a description for those internal ONNX models but I can add a link to its source.

In reply to: 301726035 [](ancestors = 301726035)

artidoro · 2019-07-09T18:15:48Z

test/Microsoft.ML.OnnxTransformerTest/OnnxTransformTests.cs

+        /// </summary>
+        /// <param name="shapeDictionary">Dictionary of tensor shapes. Keys are tensor names
+        /// while values the associated shapes.</param>
+        private void TryModwlWithCustomShapesHelper(IDictionary<string, int[]> shapeDictionary)


TryModwlWithCustomShapesHelper [](start = 21, length = 30)

I think there might be a small typo: Modwl -> Model #Resolved

wschin · 2019-07-10T01:16:13Z

    private protected override void SaveModel(ModelSaveContext ctx)

Fixed and added a test.

In reply to: 509748317 [](ancestors = 509748317)

Refers to: src/Microsoft.ML.OnnxTransformer/OnnxTransform.cs:307 in ced8c42. [](commit_id = ced8c42, deletion_comment = False)

artidoro · 2019-07-16T18:06:03Z

            verWrittenCur: 0x00010002,

Should we change the version number since we are changing the binary format of how it is saved?

Refers to: src/Microsoft.ML.OnnxTransformer/OnnxTransform.cs:155 in ab6424c. [](commit_id = ab6424c, deletion_comment = False)

artidoro · 2019-07-16T18:07:45Z

It would be very nice to have in a comment the layout of the binary format like we do in other transformers.
It's not a blocking issue though I think it would be very nice to have :)
If you do add it, you could write it in the load method above too.

Refers to: src/Microsoft.ML.OnnxTransformer/OnnxTransform.cs:327 in ab6424c. [](commit_id = ab6424c, deletion_comment = False)

codemzs

wschin commented Jul 3, 2019

View reviewed changes

wschin force-pushed the add-shape branch from ad10591 to 201a674 Compare July 3, 2019 19:53

Allow user to specify ONNX shape

ced8c42

Add a command line test Remove unused code

wschin force-pushed the add-shape branch from 201a674 to ced8c42 Compare July 3, 2019 20:06

wschin requested review from jignparm, artidoro and codemzs July 3, 2019 22:16

wschin self-assigned this Jul 3, 2019

artidoro reviewed Jul 9, 2019

View reviewed changes

Address comments

ab6424c

Merge branch 'master' into add-shape

5b923e1

codemzs approved these changes Aug 6, 2019

View reviewed changes

codemzs merged commit a6aaaf8 into dotnet:master Aug 6, 2019

mdslack mentioned this pull request Sep 7, 2019

Error in Microsoft.ML.Extensions (Attempted to divide by zero.) when running in Azure #4171

Closed

antoniovs1029 mentioned this pull request Jul 8, 2020

Opaque disruptive handling of variable axes of ONNX-models #5293

Closed

ankitbko mentioned this pull request Nov 28, 2021

Allow to overwrite ONNX Model Shape and multiple input and outputs microsoft/psi#208

Open

ghost locked as resolved and limited conversation to collaborators Mar 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow user to overwrite unknown shapes loaded from ONNX model #3963

Allow user to overwrite unknown shapes loaded from ONNX model #3963

wschin commented Jul 3, 2019 •

edited

Loading

wschin Jul 3, 2019 •

edited

Loading

artidoro Jul 9, 2019 •

edited by wschin

Loading

artidoro Jul 9, 2019

wschin Jul 10, 2019

artidoro commented Jul 9, 2019 •

edited by wschin

Loading

artidoro Jul 9, 2019 •

edited by wschin

Loading

wschin Jul 10, 2019

artidoro Jul 9, 2019 •

edited by wschin

Loading

wschin Jul 10, 2019

artidoro Jul 9, 2019 •

edited by wschin

Loading

wschin commented Jul 10, 2019

artidoro commented Jul 16, 2019

artidoro commented Jul 16, 2019

codemzs left a comment

Allow user to overwrite unknown shapes loaded from ONNX model #3963

Allow user to overwrite unknown shapes loaded from ONNX model #3963

Conversation

wschin commented Jul 3, 2019 • edited Loading

wschin Jul 3, 2019 • edited Loading

Choose a reason for hiding this comment

artidoro Jul 9, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

artidoro Jul 9, 2019

Choose a reason for hiding this comment

wschin Jul 10, 2019

Choose a reason for hiding this comment

artidoro commented Jul 9, 2019 • edited by wschin Loading

artidoro Jul 9, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

wschin Jul 10, 2019

Choose a reason for hiding this comment

artidoro Jul 9, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

wschin Jul 10, 2019

Choose a reason for hiding this comment

artidoro Jul 9, 2019 • edited by wschin Loading

Choose a reason for hiding this comment

wschin commented Jul 10, 2019

artidoro commented Jul 16, 2019

artidoro commented Jul 16, 2019

codemzs left a comment

Choose a reason for hiding this comment

wschin commented Jul 3, 2019 •

edited

Loading

wschin Jul 3, 2019 •

edited

Loading

artidoro Jul 9, 2019 •

edited by wschin

Loading

artidoro commented Jul 9, 2019 •

edited by wschin

Loading

artidoro Jul 9, 2019 •

edited by wschin

Loading

artidoro Jul 9, 2019 •

edited by wschin

Loading

artidoro Jul 9, 2019 •

edited by wschin

Loading