Skip to content

Add out-of-tree Pyodide builds in CI for numcodecs #529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
db77ad5
Git-ignore pytest cache folders
agriyakhetarpal Apr 24, 2024
22cede6
Refine outdated sections for Contributing guide
agriyakhetarpal Apr 24, 2024
3c2b0be
Add patch to disable threading and multiprocessing
agriyakhetarpal Apr 24, 2024
a8c956f
Add patch to embed missing POSIX syscall headers
agriyakhetarpal Apr 24, 2024
8769743
Add initial workflow to run + debug Pyodide builds
agriyakhetarpal Apr 24, 2024
28674c3
Disable AVX2 and SSE2 instructions
agriyakhetarpal Apr 24, 2024
f6df488
Use separate build and test jobs and share wheel
agriyakhetarpal Apr 24, 2024
d678be8
Add patch that allows loading `.npy` files
agriyakhetarpal Apr 24, 2024
6ce4127
Apply patch for Emscripten to load `.npy` files
agriyakhetarpal Apr 24, 2024
199928d
Add a constant to check if platform is WASM
agriyakhetarpal Apr 24, 2024
1f5e7b7
Use `is_wasm` to disable multiprocessing tests
agriyakhetarpal Apr 24, 2024
da909f8
Apply `.npy` files patch again when running tests
agriyakhetarpal Apr 24, 2024
a54c555
Remove patch that is unneeded at build time
agriyakhetarpal May 21, 2024
494dd2f
Merge branch 'main' into setup-emscripten-ci
agriyakhetarpal Nov 7, 2024
a7446c8
Delete patch that's not needed now
agriyakhetarpal Nov 7, 2024
9271054
Ignore Pyodide cross-build environment files
agriyakhetarpal Nov 7, 2024
764f34e
Revamp workflow and clean up changes
agriyakhetarpal Nov 7, 2024
d41f75f
Fix linter errors and style failures
agriyakhetarpal Nov 7, 2024
ee39d1e
Catch and disable `mutex` assignment
agriyakhetarpal Nov 7, 2024
731c6b2
Fix patch that disables multiprocessing, threading
agriyakhetarpal Nov 7, 2024
0616dd5
Don't hardcode `zlib` paths in patch details
agriyakhetarpal Nov 7, 2024
1aa838b
Install test + runtime extras, disable pytest cache
agriyakhetarpal Nov 7, 2024
0638a91
Fix shell expansion in installation command
agriyakhetarpal Nov 7, 2024
53f2096
Temporary: skip tests that require crc32c
agriyakhetarpal Nov 8, 2024
7133abb
Merge branch 'main' into setup-emscripten-ci
agriyakhetarpal Nov 8, 2024
bc9b6dc
Temporary: skip `crc32c` dependency until available
agriyakhetarpal Nov 8, 2024
5a4e234
Update ci-emscripten.yaml
agriyakhetarpal Nov 8, 2024
4ce9e4d
Fix zarr installation order to collect more tests
agriyakhetarpal Nov 8, 2024
d6c43ef
Force install zarr version 3 for CI
agriyakhetarpal Nov 8, 2024
55a2690
Skip a test that requires threads
agriyakhetarpal Nov 8, 2024
9a0482b
Temporary: skip Zarr's async functionality tests
agriyakhetarpal Nov 8, 2024
bf42491
Guard import of `multiprocessing`
agriyakhetarpal Nov 8, 2024
9924e8e
Revert "Guard import of `multiprocessing`"
agriyakhetarpal Nov 8, 2024
0847241
Bump verbosity when running test suite
agriyakhetarpal Nov 8, 2024
b6713ca
Skip codec entrypoint test that requires processes
agriyakhetarpal Nov 8, 2024
a590785
Let `pytest` inherit config for doctests
agriyakhetarpal Nov 8, 2024
3e1a968
Revert "Let `pytest` inherit config for doctests"
agriyakhetarpal Nov 8, 2024
e4f6670
Ignore a doctest that uses threads within the file
agriyakhetarpal Nov 8, 2024
4c348f2
Merge `main`
agriyakhetarpal Jan 1, 2025
dacbd09
Fix skip reason error message for entrypoint test
agriyakhetarpal Jan 1, 2025
6caf076
`crc32c` >=2.7 is now available with Pyodide 0.27.0
agriyakhetarpal Jan 1, 2025
85252f4
Bump to Pyodide version 0.27.0
agriyakhetarpal Jan 1, 2025
6b8d139
Clean up changes
agriyakhetarpal Jan 1, 2025
8aee545
Merge branch 'main' into setup-emscripten-ci
agriyakhetarpal Feb 18, 2025
c3e346a
Bring back code-style quoting
agriyakhetarpal Feb 18, 2025
2e0e759
Update to Pyodide 0.27.3, add `pcodec`
agriyakhetarpal Feb 26, 2025
4e6f2b6
Fix drop in coverage
agriyakhetarpal Feb 27, 2025
1cf2c6b
Fix another drop in coverage
agriyakhetarpal Mar 1, 2025
6d06a8b
Merge branch 'main' into setup-emscripten-ci
agriyakhetarpal Mar 3, 2025
1d92260
Don't add `pthread` for Emscripten
agriyakhetarpal Mar 3, 2025
c807e08
Get version properly using `fetch-tags: true`
agriyakhetarpal Mar 3, 2025
3b82940
Add CHANGELOG entry for Pyodide builds
agriyakhetarpal Mar 3, 2025
9fc8add
Wait for Git 2.48.1 for using `fetch-tags: true`
agriyakhetarpal Mar 3, 2025
d8f70b4
Merge main
agriyakhetarpal Apr 15, 2025
3821a31
Merge main
agriyakhetarpal May 22, 2025
df17500
Ensure reliability of builds and tests
agriyakhetarpal May 22, 2025
902fdf3
`zfpy` is now available in Pyodide
agriyakhetarpal May 22, 2025
89e8b04
`zarr3` should also work now
agriyakhetarpal May 22, 2025
ae74d0b
Ignoring Pyodide xbuildenv is no longer needed
agriyakhetarpal May 22, 2025
34f1fc2
Bump to Pyodide 0.27.6, and Node 22
agriyakhetarpal May 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add patch to embed missing POSIX syscall headers
  • Loading branch information
agriyakhetarpal committed May 21, 2024
commit a8c956fd67687c481d3fdc7a6395ef3b5bee9e5a
49 changes: 49 additions & 0 deletions tools/ci/patches/0002-add-missing-unistd-headers.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
This patch adds missing headers in vendored zlib as done in numcodecs.js. This file
is attributed to the Pyodide developers and can be viewed at the upstream
Pyodide repository at the following link;

https://github.com/pyodide/pyodide/blob/d32e376013d8977b66c6aa828042b1fee8047aea/packages/numcodecs/patches/fixzlib.patch


diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c b/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
index fae202e..80606a6 100644
--- a/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
+++ b/c-blosc/internal-complibs/zlib-1.2.13/gzlib.c
@@ -2,7 +2,7 @@
* Copyright (C) 2004, 2010, 2011, 2012, 2013 Mark Adler
* For conditions of distribution and use, see copyright notice in zlib.h
*/
-
+#include <unistd.h>
#include "gzguts.h"

#if defined(_WIN32) && !defined(__BORLANDC__)

diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzread.c b/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
index bf4538e..afe6acd 100644
--- a/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
+++ b/c-blosc/internal-complibs/zlib-1.2.13/gzread.c
@@ -2,7 +2,7 @@
* Copyright (C) 2004, 2005, 2010, 2011, 2012, 2013 Mark Adler
* For conditions of distribution and use, see copyright notice in zlib.h
*/
-
+#include <unistd.h>
#include "gzguts.h"

/* Local functions */


diff --git a/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c b/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
index aa767fb..a87676f 100644
--- a/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
+++ b/c-blosc/internal-complibs/zlib-1.2.13/gzwrite.c
@@ -2,7 +2,7 @@
* Copyright (C) 2004, 2005, 2010, 2011, 2012, 2013 Mark Adler
* For conditions of distribution and use, see copyright notice in zlib.h
*/
-
+#include <unistd.h>
#include "gzguts.h"

/* Local functions */