Speed-up bitmap operations on images. Fixes #5856

[darth-vader-lg]

Speed-up operations using raw access instead of very slow functions GetPixel and SetPixel.
Associated issue: #5856

I made this changes to make more fast the operations on the images, as explained in the issue #5856.
My approach is substituting all the very slow GetPixel and SetPixel functions with a raw access to the images data.

My test code is this:

[TensorFlowFact]
public void TensorFlowTransformObjectDetectionTest()
{
    // Saved model
    var modelLocation = @"D:\ObjectDetection\carp\TensorFlow\exported-model-SSD-MobileNET-v2-320x320\saved_model";
    // Create the estimators pipe
    var pipe = 
        _mlContext.Transforms.LoadImages(
            inputColumnName: "ImagePath",
            outputColumnName: "Image",
            imageFolder: "")
        .Append(_mlContext.Transforms.ResizeImages(
            inputColumnName: "Image",
            outputColumnName: "ResizedImage",
            imageWidth: 300,
            imageHeight: 300,
            resizing: ImageResizingEstimator.ResizingKind.Fill))
        .Append(_mlContext.Transforms.ExtractPixels(
            inputColumnName: "ResizedImage",
            outputColumnName: "serving_default_input_tensor:0",
            interleavePixelColors: true,
            outputAsFloatArray: false))
        .Append(_mlContext.Model.LoadTensorFlowModel(modelLocation).ScoreTensorFlowModel(
            inputColumnNames: new[] { "serving_default_input_tensor:0" },
            outputColumnNames: new[]
            {
                "StatefulPartitionedCall:1" /* detection_boxes */,
                "StatefulPartitionedCall:2" /* detection_classes */,
                "StatefulPartitionedCall:4" /* detection_scores */
            }));

    // Collect all the path of the images in the test directory
    var imagesLocation = @"D:\ObjectDetection\carp\TensorFlow\images\test";
    var images =
        Directory.GetFiles(imagesLocation).Where(file => new[] { ".jpg", ".jfif" }
        .Any(ext => Path.GetExtension(file).ToLower() == ext))
        .Select(file => new { ImagePath = file })
        .ToArray();

    // Create the transformer
    var data = _mlContext.Data.LoadFromEnumerable(images.Take(0));
    var model = pipe.Fit(data);

    // Test n times the inference on the collected images
    for (int i = 0, nImage = 0; i < 1000; i++, nImage = (nImage + 1) % images.Length)
        model.Transform(_mlContext.Data.LoadFromEnumerable(new[] { images[nImage] })).Preview();
}

but unluckily I cannot insert in the commit because it access to my local data (model and images).

The result of this changes is the improvement of speed dealing with images as described in my issue #5856.
If they are associated with my other PR #5848 / Issue #5847 they can improve the all the TF object detection system by ~396% (of course, measured on my device; it could be less or more in other devices).

Without optimizations (current)

With only image raw access optimization (Issue #5856 / PR #5857)

With TensorFlow and image raw access optimization (Issue #5847 / PR #5848 and Issue #5856 / PR #5857)

[darth-vader-lg]

Hello @michaelgsharp,

Sorry me, but I'm having some problems with the repository CI Checks.
https://dev.azure.com/dnceng/public/_build/results?buildId=1199249&view=results

It seems to be caused by an Internal error.
https://dev.azure.com/dnceng/public/_build/results?buildId=1199249&view=logs&j=9d6f93fc-7103-540d-abb9-b79bad552b0d&t=ae083334-c4b4-421c-90e4-01f3a1d3463d

Are you maybe doing some maintenance to the system now? Or there is something that I need to do with this PR?

Kind Regards.

[justinormont]

@darth-vader-lg: I restarted the CI tests. Hopefully it was a transient problem.

[codecov]

Codecov Report

Merging #5857 (6938294) into main (ff01708) will increase coverage by 0.00%.
The diff coverage is 89.06%.

@@           Coverage Diff           @@
##             main    #5857   +/-   ##
=======================================
  Coverage   68.35%   68.35%           
=======================================
  Files        1134     1134           
  Lines      241910   241942   +32     
  Branches    25289    25292    +3     
=======================================
+ Hits       165347   165373   +26     
- Misses      69919    69923    +4     
- Partials     6644     6646    +2     

Flag	Coverage Δ
Debug	68.35% <89.06%> (+<0.01%)	⬆️
production	62.92% <89.06%> (+<0.01%)	⬆️
test	89.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...Microsoft.ML.ImageAnalytics/ImagePixelExtractor.cs	72.62% <81.08%> (+1.22%)	⬆️
...rosoft.ML.ImageAnalytics/VectorToImageTransform.cs	76.54% <100.00%> (+1.12%)	⬆️
....ML.AutoML/PipelineSuggesters/PipelineSuggester.cs	78.74% <0.00%> (-6.30%)	⬇️
...soft.ML.Transforms/Text/WordEmbeddingsExtractor.cs	85.74% <0.00%> (-1.14%)	⬇️
src/Microsoft.ML.AutoML/Sweepers/Parameters.cs	84.74% <0.00%> (-0.85%)	⬇️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs	90.41% <0.00%> (+0.63%)	⬆️
...rosoft.ML.AutoML/ColumnInference/TextFileSample.cs	62.25% <0.00%> (+2.64%)	⬆️

[darth-vader-lg]

This is the final version of the PR.
The last commit is aimed to fix a wrong behaviour: some tests concerning the images management didn’t pass in the Ubuntu and the Centos builds.
They passed on Windows and MacOS.
Really speaking they also didn’t have to pass in Windows and MacOS because I did a mistake defining the lock mode at the call of the Image.LockBits function, at this point.
It must be ImageLockMode.WriteOnly and not ImageLockMode.ReadOnly, cose it’s a write operation.
For some strange reasons it functioned anyway on Windows and MacOS builds; it could be due to some bugs in the Net Core framework. Maybe it needs some investigation by the .NET team.

[michaelgsharp]

@darth-vader-lg this one also looks good. Just checking on a couple of things and I'll get this approved/merged in. If not tonight it will be tomorrow.

[michaelgsharp]

Looks good to me. Thanks again for taking the time to do this!

@darth-vader-lg How did you learn all your optimization stuff? That is something I have been really interested in (in general, not specifically towards this project) but am not entirely sure where to start looking.

[darth-vader-lg]

Looks good to me. Thanks again for taking the time to do this!

@darth-vader-lg How did you learn all your optimization stuff? That is something I have been really interested in (in general, not specifically towards this project) but am not entirely sure where to start looking.

@michaelgsharp As I told You in the last PR, it’s just a professional mental disease contracted after working for more than 30 years in the automation field, PIDs and position control loops.

if ((you are not sitting on an I7 processor) &&
   (you need to calculate in real time all together: trajectories, S-Curve velocity profiles, accelerations  and their derivatives, interpolations, maintaining the path’s error of 9 or more axes inside 0.01mm) &&
   (the loop’s time you are allowed to have is 1ms or less to achieve accurate control results) &&
   (you must to do this continuously in these little time frames))
{
   Optimization and tricks are one of the most important things, believe me;
}

By the way, It was quite simple to do these few improvements, also because the framework is well written and tidy enough.
It was more harder for me to build and put together ML.NET, Tensorflow.NET, TensorFlow 2.5.0, PyTorch, Yolo, etc..., with their different languages, environments, bugs and join all in a more big-all-inclusive stable framework in which they communicate with each other. Friendly to install and to use. Without the need of always wasting the time doing manual configurations, setups and so on.

I’m quite relatively new to this field but you better know how all these frameworks are still evolving and how the situation is such a mess, still now. But the current market is going for sure toward this direction plus/additionally the always needed "fast development and delivering timeline" to meet the competition requirements.

In any case, if You need something, have ideas or proposals about your projects, feel free to write me an email.

Have a nice day.