Merge pull request #222 from aws-samples/feature/cartopy

Add GEOS bootstrap action

Author: dacort

geos/README.md

@@ -0,0 +1,46 @@
+# GEOS Installation on EMR
+
+[GEOS](https://trac.osgeo.org/geos/) is a popular library for Geospatial analysis and is used by Python libraries like Shapely and GeoPandas for manipulating geographic data.
+
+Let's see how to install another Python package, [Cartopy](https://scitools.org.uk/cartopy/docs/latest/index.html), for geospatial data processing.
+
+## Bootstrap Action
+
+Our bootstrap action needs to perform two main functions:
+
+1. Install the GEOS library itself
+2. Install our Python packages
+
+Given that Cartopy requires GEOS 3.7.2 or greater, we unfortunately need to build it from source.
+
+This also requires that we build proj from source as well as sqlite3 (instructions here: https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html).
+
+Once those are installed, we can `pip3 install cartopy` with a minor caveat that your sudo shell needs `/usr/local/bin` in its path because that's where the `geos-config` binary is.
+
+## Running
+
+Upload `install-geos.sh` from this repository to an S3 bucket and use `aws emr create-cluster`!
+
+```bash
+S3_BUCKET=dcortesi-demo-code-us-west-2
+AWS_REGION=us-west-2
+
+aws s3 cp geos/install-geos.sh s3://${S3_BUCKET}/code/bootstrap/geos/
+aws emr create-cluster \
+ --name "emr-cartopy" \
+ --region ${AWS_REGION} \
+ --bootstrap-actions Path="s3://${S3_BUCKET}/code/bootstrap/geos/install-geos.sh"  \
+ --log-uri "s3n://${S3_BUCKET}/logs/emr/" \
+ --release-label "emr-6.10.0" \
+ --use-default-roles \
+ --applications Name=Spark Name=Livy Name=JupyterEnterpriseGateway \
+ --instance-fleets '[{"Name":"Primary","InstanceFleetType":"MASTER","TargetOnDemandCapacity":1,"TargetSpotCapacity":0,"InstanceTypeConfigs":[{"InstanceType":"c5a.2xlarge"},{"InstanceType":"m5a.2xlarge"},{"InstanceType":"r5a.2xlarge"}]},{"Name":"Core","InstanceFleetType":"CORE","TargetOnDemandCapacity":0,"TargetSpotCapacity":1,"InstanceTypeConfigs":[{"InstanceType":"c5a.2xlarge"},{"InstanceType":"m5a.2xlarge"},{"InstanceType":"r5a.2xlarge"}],"LaunchSpecifications":{"OnDemandSpecification":{"AllocationStrategy":"lowest-price"},"SpotSpecification":{"TimeoutDurationMinutes":10,"TimeoutAction":"SWITCH_TO_ON_DEMAND","AllocationStrategy":"capacity-optimized"}}}]' \
+ --scale-down-behavior "TERMINATE_AT_TASK_COMPLETION" \
+ --auto-termination-policy '{"IdleTimeout":14400}'
+```
+
+The cluster will take about 20 minutes to boot up due to needing to compile several projects from source.
+
+## Docker
+
+Another option is building a Docker container containing the necessary dependencies. 

geos/install-geos.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+
+set -e
+
+# Install build dependencies
+sudo yum install -y automake \
+    bzip2 \
+    cmake3 \
+    curl-devel \
+    expect \
+    gcc \
+    gcc-c++ \
+    gzip \
+    libtiff-devel \
+    make \
+    python3-devel \
+    tar
+
+# Install GEOS
+curl -O https://download.osgeo.org/geos/geos-3.7.5.tar.bz2
+tar xjvf geos-3.7.5.tar.bz2
+cd geos-3.7.5
+./configure && make && sudo make install
+cd ..
+rm -rf geos-3.7.5
+
+# Install sqlite>=3.11 (needed for proj)
+curl -O https://www.sqlite.org/src/tarball/sqlite.tar.gz
+tar xzf sqlite.tar.gz
+cd sqlite/
+export CFLAGS="-DSQLITE_ENABLE_FTS3 \
+    -DSQLITE_ENABLE_FTS3_PARENTHESIS \
+    -DSQLITE_ENABLE_FTS4 \
+    -DSQLITE_ENABLE_FTS5 \
+    -DSQLITE_ENABLE_JSON1 \
+    -DSQLITE_ENABLE_LOAD_EXTENSION \
+    -DSQLITE_ENABLE_RTREE \
+    -DSQLITE_ENABLE_STAT4 \
+    -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \
+    -DSQLITE_SOUNDEX \
+    -DSQLITE_TEMP_STORE=3 \
+    -DSQLITE_USE_URI \
+    -O2 \
+    -fPIC"
+export PREFIX="/usr/local"
+LIBS="-lm" ./configure --disable-tcl --enable-shared --enable-tempstore=always --prefix="$PREFIX"
+make
+sudo make install
+cd ..
+rm -rf sqlite
+
+# Install proj
+curl -O https://download.osgeo.org/proj/proj-9.2.1.tar.gz
+tar xzvf proj-9.2.1.tar.gz
+cd proj-9.2.1
+mkdir build
+cd build
+cmake3 -DSQLITE3_INCLUDE_DIR=/usr/local/include/ -DSQLITE3_LIBRARY=/usr/local/lib/libsqlite3.so ..
+cmake3 --build .
+sudo cmake3 --build . --target install
+cd ../../
+rm -rf proj-9.2.1
+
+# We need to let Python know where libproj and libgeos are installed
+echo -e '/usr/local/lib/\n/usr/local/lib64/' | sudo tee -a /etc/ld.so.conf.d/proj-x86_64.conf > /dev/null
+sudo ldconfig
+
+# Now install cartopy
+# The path=$path portion adds /usr/local/bin to the sudo environment path so that cartopy can find geos
+# shapely==1.8.5 is needed due to https://github.com/SciTools/cartopy/issues/2076
+sudo env "PATH=$PATH" pip3 install shapely==1.8.5 cartopy