Skip to content

Commit 1244396

Browse files
ownEyesownEyes
committed
restructure envs & julia&R modified
1 parent 84912ef commit 1244396

File tree

11 files changed

+319
-56
lines changed

11 files changed

+319
-56
lines changed

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
[![Binder badge](https://static.mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/jupyter/docker-stacks/main?urlpath=lab/tree/README.ipynb "Launch a quay.io/jupyter/base-notebook container on mybinder.org")
99

1010
**🔷Some images in this repo are modified by [ownEyes](https://github.com/ownEyes) to use Conda instead of mamba, all packages are installed to a new environment instead of the base env. The purpose of such changes is for the installation of Kears3 with different backend support.**
11+
**🔷Conda envs now have been restructured.**
1112

1213
Jupyter Docker Stacks are a set of ready-to-run [Docker images](https://quay.io/organization/jupyter) containing Jupyter applications and interactive computing tools.
1314
You can use a stack image to do any of the following (and more):

‎images/base-notebook/Dockerfile

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,9 @@ RUN apt-get update --yes && \
2626
# than one unique instance of some command with a unique set of arguments,
2727
# we use `run-one-constantly` to support the `RESTARTABLE` option
2828
run-one && \
29-
apt-get clean && rm -rf /var/lib/apt/lists/*
29+
apt-get clean && rm -rf /var/lib/apt/lists/* && \
30+
sed -i '/^# Execute the command!/i \
31+
if [[ "${cmd[0]}" == "jupyter" ]]; then\n source /opt/conda/bin/activate jupyter\nfi\n' /usr/local/bin/start.sh
3032

3133
USER ${NB_UID}
3234

@@ -37,17 +39,23 @@ USER ${NB_UID}
3739
# Do all this in a single RUN command to avoid duplicating all of the
3840
# files across image layers when the permissions change
3941
WORKDIR /tmp
40-
RUN conda create -n keras-tf && \
41-
conda run -n keras-tf conda install -y \
42+
RUN conda create -n jupyter && \
43+
# conda run -n jupyter conda install -y \
44+
source /opt/conda/bin/activate jupyter && \
45+
conda install -y \
4246
jupyter_core \
4347
jupyterlab \
4448
notebook \
4549
jupyterhub \
4650
nbclassic && \
47-
conda run -n keras-tf jupyter server --generate-config && \
48-
conda run -n keras-tf conda clean --all -f -y && \
49-
conda run -n keras-tf npm cache clean --force && \
50-
conda run -n keras-tf jupyter lab clean && \
51+
# conda run -n jupyter jupyter server --generate-config && \
52+
# conda run -n jupyter conda clean --all -f -y && \
53+
# conda run -n jupyter npm cache clean --force && \
54+
# conda run -n jupyter jupyter lab clean && \
55+
jupyter server --generate-config && \
56+
conda clean --all -f -y && \
57+
npm cache clean --force && \
58+
jupyter lab clean && \
5159
# conda create --name keras-torch --clone keras-tf && \
5260
rm -rf "/home/${NB_USER}/.cache/yarn" && \
5361
fix-permissions "${CONDA_DIR}" && \

‎images/base-notebook/start-notebook.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
22
# Shim to emit warning and call start-notebook.py
33
echo "WARNING: Use start-notebook.py instead"
4-
4+
source activate jupyter
55
exec /usr/local/bin/start-notebook.py "$@"

‎images/base-notebook/start-singleuser.sh

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
#!/bin/bash
22
# Shim to emit warning and call start-singleuser.py
33
echo "WARNING: Use start-singleuser.py instead"
4-
4+
# Activate the Conda environment
5+
source activate jupyter
56
exec /usr/local/bin/start-singleuser.py "$@"

‎images/datascience-notebook/Dockerfile

Lines changed: 74 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,63 +1,99 @@
11
# Copyright (c) Jupyter Development Team.
22
# Distributed under the terms of the Modified BSD License.
3-
ARG REGISTRY=quay.io
4-
ARG OWNER=jupyter
5-
ARG BASE_CONTAINER=$REGISTRY/$OWNER/scipy-notebook
3+
# ARG REGISTRY=quay.io
4+
# ARG OWNER=jupyter
5+
# ARG BASE_CONTAINER=$REGISTRY/$OWNER/scipy-notebook
6+
# FROM $BASE_CONTAINER
7+
8+
# LABEL maintainer="Jupyter Project <[email protected]>"
9+
10+
ARG REGISTRY=docker.io
11+
ARG OWNER=yifengdev
12+
ARG BASE_IMAGE=scipy-jupyter
13+
ARG BASE_CONTAINER=$REGISTRY/$OWNER/$BASE_IMAGE
614
FROM $BASE_CONTAINER
715

8-
LABEL maintainer="Jupyter Project <jupyter@googlegroups.com>"
16+
LABEL maintainer="Liu <932048437@qq.com>"
917

1018
# Fix: https://github.com/hadolint/hadolint/wiki/DL4006
1119
# Fix: https://github.com/koalaman/shellcheck/wiki/SC3014
1220
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
1321

22+
USER ${NB_UID}
23+
24+
# Add setup scripts that may be used by downstream images or inherited images
25+
COPY setup-scripts/ /opt/setup-scripts/
26+
27+
# R packages including IRKernel which gets installed globally.
28+
# r-e1071: dependency of the caret R package
29+
RUN conda create -n r-env && \
30+
# conda run -n jupyter conda install -y \
31+
source /opt/conda/bin/activate r-env && \
32+
conda install --yes \
33+
r-base \
34+
r-caret \
35+
r-crayon \
36+
r-devtools \
37+
r-e1071 \
38+
r-forecast \
39+
r-hexbin \
40+
r-htmltools \
41+
r-htmlwidgets \
42+
r-irkernel \
43+
r-nycflights13 \
44+
r-randomforest \
45+
r-rcurl \
46+
r-rmarkdown \
47+
r-rodbc \
48+
r-rsqlite \
49+
r-shiny \
50+
r-tidymodels \
51+
r-tidyverse \
52+
rpy2 \
53+
unixodbc && \
54+
conda clean --all -f -y && \
55+
ln -s /opt/conda/envs/jupyter/bin/jupyter /opt/conda/envs/r-env/bin/jupyter && \
56+
fix-permissions "${CONDA_DIR}" && \
57+
fix-permissions "/home/${NB_USER}"
58+
59+
# Add an R mimetype option to specify how the plot returns from R to the browser
60+
COPY --chown=${NB_UID}:${NB_GID} Rprofile.site /opt/conda/envs/r-env/lib/R/etc/
61+
1462
USER root
1563

64+
# Julia dependencies
65+
# install Julia packages in /opt/julia instead of ${HOME}
66+
ENV JULIA_DEPOT_PATH=/opt/conda/envs/julia-env/share/julia \
67+
JULIA_PKGDIR=/opt/conda/envs/julia-env/share/julia
68+
1669
# R pre-requisites
1770
RUN apt-get update --yes && \
1871
apt-get install --yes --no-install-recommends \
1972
fonts-dejavu \
2073
gfortran \
2174
gcc && \
22-
apt-get clean && rm -rf /var/lib/apt/lists/*
23-
24-
# Julia dependencies
25-
# install Julia packages in /opt/julia instead of ${HOME}
26-
ENV JULIA_DEPOT_PATH=/opt/julia \
27-
JULIA_PKGDIR=/opt/julia
75+
apt-get clean && rm -rf /var/lib/apt/lists/* && \
76+
conda create -n julia-env && \
77+
source /opt/conda/bin/activate julia-env && \
78+
conda clean --all -f -y && \
79+
/opt/setup-scripts/setup_julia.py && \
80+
fix-permissions "${CONDA_DIR}" && \
81+
fix-permissions "/home/${NB_USER}"
2882

83+
RUN source /opt/conda/bin/activate r-env && \
84+
R --slave -e "IRkernel::installspec(name = 'r-env', displayname = 'R (r-env)', user = FALSE)" && \
85+
mv /usr/local/share/jupyter/kernels/r-env /opt/conda/envs/jupyter/share/jupyter/kernels && \
86+
rm -rf /usr/local/share/jupyter && \
87+
rm -rf /root/.local && \
88+
chown -R ${NB_UID}:${NB_GID} "${CONDA_DIR}/envs/jupyter/share/jupyter/kernels/" && \
89+
fix-permissions "${CONDA_DIR}" && \
90+
fix-permissions "/home/${NB_USER}" "${CONDA_DIR}/envs/jupyter/share/jupyter/kernels/"
2991
# Setup Julia
30-
RUN /opt/setup-scripts/setup_julia.py
3192

3293
USER ${NB_UID}
3394

3495
# Setup IJulia kernel & other packages
3596
RUN /opt/setup-scripts/setup-julia-packages.bash
3697

37-
# R packages including IRKernel which gets installed globally.
38-
# r-e1071: dependency of the caret R package
39-
RUN mamba install --yes \
40-
'r-base' \
41-
'r-caret' \
42-
'r-crayon' \
43-
'r-devtools' \
44-
'r-e1071' \
45-
'r-forecast' \
46-
'r-hexbin' \
47-
'r-htmltools' \
48-
'r-htmlwidgets' \
49-
'r-irkernel' \
50-
'r-nycflights13' \
51-
'r-randomforest' \
52-
'r-rcurl' \
53-
'r-rmarkdown' \
54-
'r-rodbc' \
55-
'r-rsqlite' \
56-
'r-shiny' \
57-
'r-tidymodels' \
58-
'r-tidyverse' \
59-
'rpy2' \
60-
'unixodbc' && \
61-
mamba clean --all -f -y && \
62-
fix-permissions "${CONDA_DIR}" && \
63-
fix-permissions "/home/${NB_USER}"
98+
99+

‎images/datascience-notebook/Rprofile.site

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Add R mimetype to specify how the plot returns from R to the browser.
2+
# https://notebook.community/andrie/jupyter-notebook-samples/Changing%20R%20plot%20options%20in%20Jupyter
3+
4+
options(jupyter.plot_mimetypes = c('text/plain', 'image/png', 'image/jpeg', 'image/svg+xml', 'application/pdf'))

‎images/datascience-notebook/setup-scripts/setup-julia-packages.bash

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
#!/bin/bash
2+
set -exuo pipefail
3+
# Requirements:
4+
# - Run as a non-root user
5+
# - The JULIA_PKGDIR environment variable is set
6+
# - Julia is already set up, with the setup-julia.bash command
7+
source /opt/conda/bin/activate julia-env
8+
9+
# If we don't specify what CPUs the precompilation should be done for, it's
10+
# *only* done for the target of the host doing the compilation. When the
11+
# container runs on a host that's the same architecture, but a *different*
12+
# generation of CPU than what the build host was, the precompilation is useless
13+
# and Julia takes a long long time to start up. This specific multitarget comes
14+
# from https://github.com/JuliaCI/julia-buildkite/blob/70bde73f6cb17d4381b62236fc2d96b1c7acbba7/utilities/build_envs.sh#L20-L76,
15+
# and may need to be updated as new CPU generations come out.
16+
# If the architecture the container runs on is different,
17+
# precompilation may still have to be re-done on first startup - but this
18+
# *should* catch most of the issues. See
19+
# https://github.com/jupyter/docker-stacks/issues/2015 for more information
20+
if [ "$(uname -m)" == "x86_64" ]; then
21+
# See https://github.com/JuliaCI/julia-buildkite/blob/70bde73f6cb17d4381b62236fc2d96b1c7acbba7/utilities/build_envs.sh#L24
22+
# for an explanation of these options
23+
export JULIA_CPU_TARGET="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)"
24+
elif [ "$(uname -m)" == "aarch64" ]; then
25+
# See https://github.com/JuliaCI/julia-buildkite/blob/70bde73f6cb17d4381b62236fc2d96b1c7acbba7/utilities/build_envs.sh#L54
26+
# for an explanation of these options
27+
export JULIA_CPU_TARGET="generic;cortex-a57;thunderx2t99;carmel"
28+
fi
29+
30+
# Install base Julia packages
31+
julia -e '
32+
import Pkg;
33+
Pkg.update();
34+
Pkg.add([
35+
"HDF5",
36+
"IJulia",
37+
"Pluto"
38+
]);
39+
Pkg.precompile();
40+
'
41+
42+
# Move the kernelspec out of ${HOME} to the system share location.
43+
# Avoids problems with runtime UID change not taking effect properly
44+
# on the .local folder in the jovyan home dir.
45+
mv "${HOME}/.local/share/jupyter/kernels/julia"* "${CONDA_DIR}/envs/jupyter/share/jupyter/kernels/"
46+
chmod -R go+rx "${CONDA_DIR}/envs/jupyter/share/jupyter/kernels/"
47+
rm -rf "${HOME}/.local"
48+
fix-permissions "${JULIA_PKGDIR}" "${CONDA_DIR}/envs/jupyter/share/jupyter/kernels/"
49+
50+
# Install jupyter-pluto-proxy to get Pluto to work on JupyterHub
51+
source /opt/conda/bin/activate jupyter
52+
conda install --yes \
53+
'jupyter-pluto-proxy' && \
54+
conda clean --all -f -y && \
55+
fix-permissions "${CONDA_DIR}" && \
56+
fix-permissions "/home/${NB_USER}"

0 commit comments

Comments
 (0)