Node v22 LTS upgrade

[spwoodcock]

Continuation from #127 (anticipates merge of or replaces PR)

Fixes #116

Attempted to continue upgrading past Node v16.
It wasn't too tricky & things don't seem to be broken.

Notes:

• rimraf removed support for require / es6 imports are needed, max v4.4.1 possible rimraf no longer supports require and needs ES6 imports. But the lib is no longer needed, as in node 16+ fs.rm() does everything we need.
• busboy is old & unmaintained - may need to be removed in future?
  • I updated to avoid breaking changes 'field' and 'file' event handler parameters described here: Changes in v1.0.0 mscdex/busboy#266
• minimist is also pretty old and unmaintained - may also need removal in future?

[pierotofy]

Hey thanks for the PR @spwoodcock 👋

It's great you're attempting to update, but please don't assume you can just upgrade Node version and a few packages without introducing errors 🙏 . You need to throughly test every single code branch/endpoint (including the autoscaling mechanism) manually because of changes to node-libcurl (among other packages that might have changed behavior between releases). There's also been changes to how unhandled promises are caught in newer versions (in the current version, they are ignored with a warning message, in the latest version the node process exits with code 1 !!)

Also, I can't build this on M1 due to:

#13 13.66 npm error node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v127 ABI, glibc) (falling back to source compile with node-gyp)
#13 13.66 npm error node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/JCMais/node-libcurl/releases/download/v4.1.0/node_libcurl-v4.1.0-node-v127-linux-arm64-glibc.tar.gz

[spwoodcock]

Hey thanks for the PR @spwoodcock 👋

It's great you're attempting to update, but please don't assume you can just upgrade Node version and a few packages without introducing errors 🙏 . You need to throughly test every single code branch/endpoint (including the autoscaling mechanism) manually because of changes to node-libcurl (among other packages that might have changed behavior between releases). There's also been changes to how unhandled promises are caught in newer versions (in the current version, they are ignored with a warning message, in the latest version the node process exits with code 1 !!)

Also, I can't build this on M1 due to:

#13 13.66 npm error node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v127 ABI, glibc) (falling back to source compile with node-gyp)
#13 13.66 npm error node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/JCMais/node-libcurl/releases/download/v4.1.0/node_libcurl-v4.1.0-node-v127-linux-arm64-glibc.tar.gz

I fully agree, but as there aren't any automated tests in this lib it needs some good manual testing as you say.

Apologies I haven't got around to that yet - I was hoping for a bit of feedback before diving into something like that - I should have probably marked this as draft 😅

I'm sure we can fix the M1 issues 👍

[spwoodcock]

• Fixed the node-libcurl compile during docker build on linux/arm64 by having all required build dependencies (a pre-compiled binary is not available for that arch).
• Added the start of a test config, which runs through a standard NodeODM processing job:
  • Uses 3 drone images that I reduced down in size significantly using imagemagick and exiftool:

    # significantly compress image
    magick DJI_20240716094137_0134_D.JPG \
    -sampling-factor 4:2:0 \
    -strip \
    -quality 1 \
    -interlace Plane \
    -gaussian-blur 0.05 \
    -colorspace RGB \
    test-photo-1.jpg
    
     # ensure final image has required EXIF data
    exiftool -TagsFromFile DJI_20240716094137_0134_D.JPG -overwrite_original test-photo-1.jpg

  • I thought these images of rice paddies from Bali were nice to test with, as they have such clear demarcations between bank and field.
  • The final ortho obviously looks like crap, but OpenSfM is able to patch the photos together nicely.
    
  • I wonder if there was ever any progress on the generation of synthetic test data for this sort of thing Add synthetic datasets for testing ODMdata#8 (the images I processed aren't synthetic, but seem to be pretty decent for this sort of testing).
• Running the tests first requires the new ClusterODM contaner to be built: docker build . -t opendronemap/clusterodm:2.0.0
• Everything seems to work from the ClusterODM --> NodeODM side. It takes ~1 min to run the job.
• Next up would be looking at how to test / mock the autoscaling capabilities.

Note

There is one point where this test is a bit flaky and I need to debug - it mostly works, but sometimes the responseJson for the task polling is undefined.

[pierotofy]

NodeODM has a test mode, check https://github.com/OpenDroneMap/NodeODM/blob/master/config.js#L39-L44 which might be useful for your case here.