ImgMath updates #87

acardona · 2020-07-27T16:14:21Z

A few updates to ImgMath, including the power operation as recently added to imglib2 core, and convenience methods for viewing operations as RandomAccessibleInterval (rather than computing them), and for multithreaded computing thanks to LoopBuilder.

acardona · 2020-07-27T16:20:28Z

The travis build fails because it isn't using the latest imglib2.

side by side with the RealType one. It is at least twice as fast.

… ArrayImg or into Img of the same kind as one of the input images, with the additional ability to define a Type for computing.

…tion without actually running anything. So, instead of copying the result into an image, merely return it as a recipe for generating the result at any one sample, with random access. Works great because computations are pixel-wise.

…Builder and the asRandomAccessibleInterval view of the computations.

… anew to contain the result of the pixel-wise computations.

which are more readable than EQ, NEQ, GT and LT (which are otherwise the exact same thing).

We want to call the new pow(...) method on a RealType.

…nterface to NumericType.

Not based on NumericType math as it lacks these functions. Uses RealType.setReal and RealType.getRealDouble(), which is the standard approach as used by e.g. net.imglib2.util.Util.genericRealTypeConverter.

ctrueden · 2020-07-29T22:38:02Z

The travis build fails because it isn't using the latest imglib2.

An excellent illustration of why reproducible builds are vital.

I have force pushed this branch to address the issue. Here are the steps I followed:

I ran check-branch.sh (available here) to see which commits compile with passing tests:

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (master) »
git checkout io-package
Branch 'io-package' set up to track remote branch 'io-package' from 'origin'.
Switched to a new branch 'io-package'

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (io-package) »
check-branch.sh
0000 e5d0f014e4bfc01bec9bd02bf4624ed19696dceb SUCCESS 16
0001 bc1bba8d189a6be51124297d4277134b96d6c0dc SUCCESS 13
0002 bb29d7d0643ddacd8ab377b14f73216e47df6f70 SUCCESS 14
0003 94426d5978a719eb67e493cdb44de8db807860a8 FAILURE 6
0004 8c54855d5fbaf6e2ce2e4f5f523f6f78b89f0cf5 FAILURE 5
0005 86e50a651ce6a70c6fdbddc11ab560eca942dc7c FAILURE 6
0006 d410e997ef5caa17a564e64729c31cd1acb853e6 FAILURE 7
0007 32893f1d14d4ebc8f799126cdfb452ea760bf6e0 FAILURE 8
0008 6d3da1c187c714bb426a656de60143188a6d9c3f FAILURE 8
0009 641a5458149f53a53b70d2014c3780e3b8e5a095 FAILURE 9

What went wrong?

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (io-package) »
grep -A9999 'BUILD FAILURE' tmp/0003-94426d5978a719eb67e493cdb44de8db807860a8
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.654 s
[INFO] Finished at: 2020-07-29T11:41:39-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) on project imglib2-algorithm: Compilation failure: Compilation failure:
[ERROR] /Volumes/code/imglib/imglib2-algorithm/src/main/java/net/imglib2/algorithm/math/Compute.java:[10,19] cannot find symbol
[ERROR]   symbol:   class AbstractWrappedPositionableLocalizable
[ERROR]   location: package net.imglib2
[ERROR] /Volumes/code/imglib/imglib2-algorithm/src/main/java/net/imglib2/algorithm/math/Compute.java:[287,92] cannot find symbol
[ERROR]   symbol:   class AbstractWrappedPositionableLocalizable
[ERROR]   location: class net.imglib2.algorithm.math.Compute
[ERROR] /Volumes/code/imglib/imglib2-algorithm/src/main/java/net/imglib2/algorithm/math/Compute.java:[287,17] RandomAccessCompute is not abstract and does not override abstract method getLongPosition(int) in net.imglib2.Localizable
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

So 94426d5 introduces changes relying on a newer imglib2. Before this commit can be introduced in the history, we need to update imglib2-algorithm to a new-enough imglib2.

I updated imglib2-algorithm's master branch to a new-enough imglib2. I edited the pom.xml to extend the newest bill of materials:

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (master*) »
git diff
diff --git a/pom.xml b/pom.xml
index 713b696..80df03f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -5,7 +5,7 @@
        <parent>
                <groupId>org.scijava</groupId>
                <artifactId>pom-scijava</artifactId>
-               <version>27.0.1</version>
+               <version>29.0.0</version>
                <relativePath />
        </parent>

@@ -200,8 +200,6 @@ Jean-Yves Tinevez and Michael Zinsmaier.</license.copyrightOwners>

                <!-- NB: Deploy releases to the SciJava Maven repository. -->
                <releaseProfiles>deploy-to-scijava</releaseProfiles>
-
-               <imglib2-roi.version>0.9.0</imglib2-roi.version>
        </properties>

        <repositories>

The relevant commit is 1096528.

Note the removal of the imglib2-roi version pin—without this, we force a too-old version of imglib2-roi that breaks some tests.

So, which components are upgraded by this change? I checked using mvn dependency:list before and after, and diffed the result. I have a script automating this approach (available here):

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (master) »
dep-changes
...
-[INFO]    net.imglib2:imglib2-realtransform:jar:2.2.1:compile
-[INFO]    net.imglib2:imglib2-roi:jar:0.9.0:compile
-[INFO]    net.imglib2:imglib2:jar:5.6.3:compile
+[INFO]    net.imglib2:imglib2-realtransform:jar:3.0.0:compile
+[INFO]    net.imglib2:imglib2-roi:jar:0.10.2:compile
+[INFO]    net.imglib2:imglib2:jar:5.9.2:compile
...

I.e.:

imglib2-realtransform: 2.2.1 -> 3.0.0
imglib2-roi: 0.9.0 -> 0.10.2
imglib2: 5.6.3 -> 5.9.2

So 👍 on the version updates. But do the tests still pass? One mvn clean install later: BUILD SUCCESS! So all good.

I rebased the io-package branch over the new master containing the pom-scijava update:

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (master) »
git checkout io-package                                                  #  0 {2020-07-29 12:14:17}
Switched to branch 'io-package'
Your branch is up to date with 'origin/io-package'.
: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (io-package) »
git rebase master                                                        #  0 {2020-07-29 12:14:21}
Successfully rebased and updated refs/heads/io-package.

I ran check-branch.sh again to see if there are still failing commits:

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (io-package) »
check-branch.sh
0000 1a233eb7a66e1c98c95ec2165a60d303c1cff441 SUCCESS 20
0001 46d8051545f93659c66909a468600f655bd1dedc SUCCESS 17
0002 6cdf2165c1e15d9dc2671d468c904e925f88f8ed SUCCESS 19
0003 4834861ab156d28203d5a178d5e5ad98121357b2 SUCCESS 18
0004 c56b2d5812710983813985e7dd399aea3f0f0c96 SUCCESS 24
0005 1b1059549d2202745a61a9a3bd4e240d04651f10 SUCCESS 22
0006 f83380f0b99bd805e254bd8c05e64e7434464578 SUCCESS 20
0007 8c932dc339378e0d6d2ff232e5b61f4e554198be FAILURE 7
0008 60a60a463e7298e805bbf11ae8f4f51217ff392a FAILURE 9
0009 0d2a9ff0e760aab70c7f8bf9f0e5182dd298ad44 FAILURE 7

OK, we got farther... what's wrong now?

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (io-package) »
grep -A9999 'BUILD FAILURE' tmp/0007-8c932dc339378e0d6d2ff232e5b61f4e554198be
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.397 s
[INFO] Finished at: 2020-07-29T16:25:24-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project imglib2-algorithm: Compilation failure: Compilation failure:
[ERROR] /Volumes/code/imglib/imglib2-algorithm/src/main/java/net/imglib2/algorithm/math/execution/Power.java:[26,27] cannot find symbol
[ERROR]   symbol:   method pow(O)
[ERROR]   location: variable scrap of type O
[ERROR] /Volumes/code/imglib/imglib2-algorithm/src/main/java/net/imglib2/algorithm/math/execution/Power.java:[34,27] cannot find symbol
[ERROR]   symbol:   method pow(O)
[ERROR]   location: variable scrap of type O
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Hmm, RealType does not have or inherit any pow(...) method in imglib2 5.9.2! I see now that this functionality was added with the recent PR imglib/imglib2#287, but not yet released.

I published imglib2 5.10.0 using the release-version.sh script (available here).
I inserted commit f1c25c2 bumping the imglib2 version to the newly released 5.10.0.
I tested the branch again:

: curtis@sirius /Volumes/code/imglib/imglib2-algorithm (io-package) »
check-branch.sh
0000 1a233eb7a66e1c98c95ec2165a60d303c1cff441 SUCCESS 18
0001 46d8051545f93659c66909a468600f655bd1dedc SUCCESS 18
0002 6cdf2165c1e15d9dc2671d468c904e925f88f8ed SUCCESS 19
0003 4834861ab156d28203d5a178d5e5ad98121357b2 SUCCESS 18
0004 c56b2d5812710983813985e7dd399aea3f0f0c96 SUCCESS 19
0005 1b1059549d2202745a61a9a3bd4e240d04651f10 SUCCESS 20
0006 f83380f0b99bd805e254bd8c05e64e7434464578 SUCCESS 23
0007 f1c25c2a1b7af8644432630df74ceda2f645fecd SUCCESS 19
0008 5c82415c4f47c3ada182225712fdd22e3759a9ad SUCCESS 20
0009 da706e788aa2fd4b8cbba6fc392e453b06666b8c SUCCESS 20
0010 cc919c6b5a4b6e5d0eed9e09ef1ac2b00a9d5bf4 SUCCESS 22

🎉

I force-pushed the io-package branch to include this work.

I expect that the Travis build will now fail again, due to failure to download imglib2, due to problem with maven.scijava.org that started happening within the past couple of weeks. I'm looking into it.

acardona · 2020-07-30T21:33:28Z

Thanks @ctrueden, this is very helpful. Also, not something I'd reasonably expect of anybody else! Hope you can find what's up with maven.scijava.org.

Reproducible builds are indeed fantastic. Here, the issue is instead about compiling the lastest imglib2-algorithm with the latest imglib2, both unreleased.

acardona · 2020-08-06T14:20:57Z

Hi @ctrueden, did you find the issue with maven.scijava.org, and can this be merged?

ctrueden · 2020-08-06T17:15:35Z

@acardona Yes! The Maven server is working, and this PR build is passing. Merging now!

acardona · 2020-08-06T21:00:09Z

Thanks very much!

acardona changed the title ~~Io package~~ ImgMath updates Jul 27, 2020

acardona and others added 11 commits July 29, 2020 12:14

ImgMath: an implementation in plain native double rather than RealType,

1a233eb

side by side with the RealType one. It is at least twice as fast.

ImgMath Util: no need for LinkedList, List suffices as argument type.

46d8051

ImgMath: Compute implements methods for trivially storing result into…

6cdf216

… ArrayImg or into Img of the same kind as one of the input images, with the additional ability to define a Type for computing.

ImgMath: Compute implements parallel processing methods based on Loop…

c56b2d5

…Builder and the asRandomAccessibleInterval view of the computations.

ImgMath: static methods for computing into an image that gets created…

1b10595

… anew to contain the result of the pixel-wise computations.

ImgMath: static methods equal, notEqual, greaterThan and lessThan

f83380f

which are more readable than EQ, NEQ, GT and LT (which are otherwise the exact same thing).

POM: pin imglib2 version to 5.10.0

f1c25c2

We want to call the new pow(...) method on a RealType.

ImgMath: add pow as an operation, following the addition of the Pow i…

5c82415

…nterface to NumericType.

ImgMath: added documentation to parallel execution methods.

da706e7

ImgMath: new math operations Log and Exp.

cc919c6

Not based on NumericType math as it lacks these functions. Uses RealType.setReal and RealType.getRealDouble(), which is the standard approach as used by e.g. net.imglib2.util.Util.genericRealTypeConverter.

ctrueden force-pushed the io-package branch from 641a545 to cc919c6 Compare July 29, 2020 22:37

ctrueden merged commit e2fa1fe into master Aug 6, 2020

ctrueden deleted the io-package branch August 6, 2020 17:15

ctrueden mentioned this pull request Aug 6, 2020

ImgMath blockread and offset #88

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ImgMath updates #87

ImgMath updates #87

Uh oh!

acardona commented Jul 27, 2020

Uh oh!

acardona commented Jul 27, 2020

Uh oh!

ctrueden commented Jul 29, 2020 •

edited

Loading

Uh oh!

acardona commented Jul 30, 2020

Uh oh!

acardona commented Aug 6, 2020

Uh oh!

ctrueden commented Aug 6, 2020

Uh oh!

acardona commented Aug 6, 2020

Uh oh!

Uh oh!

ImgMath updates #87

ImgMath updates #87

Uh oh!

Conversation

acardona commented Jul 27, 2020

Uh oh!

acardona commented Jul 27, 2020

Uh oh!

ctrueden commented Jul 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

acardona commented Jul 30, 2020

Uh oh!

acardona commented Aug 6, 2020

Uh oh!

ctrueden commented Aug 6, 2020

Uh oh!

acardona commented Aug 6, 2020

Uh oh!

Uh oh!

ctrueden commented Jul 29, 2020 •

edited

Loading