Neurocontainers

• 1: Contribute Containers
• 2: CVMFS
• 3: Datalad
• 4: Docker
• 5: Google Colab
• 6: Singularity
• 7: Windows 11 and Windows Subsystem for Linux

1 - Contribute Containers

To make contributing containers easier, we developed an interactive container build system. If you are very familiar with Git and building containers you can also follow the manual process, which you can find documented here: https://www.neurodesk.org/developers/new_tools/add_tool/

1) Open an issue to get access to the interactive container build system

https://github.com/NeuroDesk/neurocontainers/issues/new

• describe which container you would like to add
• wait for a reply on your issue that your account has been setup

2) Access the container build system

https://labtokyo.neurodesk.org/

• authenticate with your github account
• select a CPU session or a GPU session (if your container requires a GPU)
• open a Neurodesktop session

3) Run the interactive build process

• open a Terminal session
• run:

cd ~
git clone https://github.com/NeuroDesk/neurocontainers.git
cd neurocontainers/interactive_builder/
./run_interactive_builder.sh

• Follow the instructions of the interactive build tool. After a couple of seconds where the base image gets updated you should see a “root@neurodesk-builder:~$>” shell. Now run the commands to get your tool to work.
• Once the tool works, hit CTRL-D or type “exit”
• Then answer more questions in the build tool

4) Submit the generated build.sh and Readme.md file as attachements to your issue

• once completed, download the build.sh and README.md file and submit them as attachements to your github issue

2 - CVMFS

Install CVMFS

First you need to install CVMFS. Follow the official instructions here: https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html#getting-the-software

one example for Ubuntu in Windows Subsystem for Linux (WSL) could look like this:

sudo apt-get install lsb-release
wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb
sudo dpkg -i cvmfs-release-latest_all.deb
rm -f cvmfs-release-latest_all.deb
sudo apt-get update
sudo apt-get build-essential
sudo apt-get install cvmfs

Configure CVMFS

Once installed create the keys and configure the servers used:

sudo mkdir -p /etc/cvmfs/keys/ardc.edu.au/


echo "-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwUPEmxDp217SAtZxaBep
Bi2TQcLoh5AJ//HSIz68ypjOGFjwExGlHb95Frhu1SpcH5OASbV+jJ60oEBLi3sD
qA6rGYt9kVi90lWvEjQnhBkPb0uWcp1gNqQAUocybCzHvoiG3fUzAe259CrK09qR
pX8sZhgK3eHlfx4ycyMiIQeg66AHlgVCJ2fKa6fl1vnh6adJEPULmn6vZnevvUke
I6U1VcYTKm5dPMrOlY/fGimKlyWvivzVv1laa5TAR2Dt4CfdQncOz+rkXmWjLjkD
87WMiTgtKybsmMLb2yCGSgLSArlSWhbMA0MaZSzAwE9PJKCCMvTANo5644zc8jBe
NQIDAQAB
-----END PUBLIC KEY-----" | sudo tee /etc/cvmfs/keys/ardc.edu.au/neurodesk.ardc.edu.au.pub

echo "CVMFS_USE_GEOAPI=yes" | sudo tee /etc/cvmfs/config.d/neurodesk.ardc.edu.au.conf

echo 'CVMFS_SERVER_URL="http://cvmfs.neurodesk.org/cvmfs/@fqrn@;http://cvmfs-brisbane.neurodesk.org/cvmfs/@fqrn@;http://cvmfs-sydney.neurodesk.org/cvmfs/@fqrn@;http://cvmfs-frankfurt.neurodesk.org/cvmfs/@fqrn@;http://cvmfs-zurich.neurodesk.org/cvmfs/@fqrn@;http://cvmfs-toronto.neurodesk.org/cvmfs/@fqrn@;http://cvmfs-ashburn.neurodesk.org/cvmfs/@fqrn@;http://cvmfs.neurodesk.org/cvmfs/@fqrn@"' | sudo tee -a /etc/cvmfs/config.d/neurodesk.ardc.edu.au.conf 

echo 'CVMFS_KEYS_DIR="/etc/cvmfs/keys/ardc.edu.au/"' | sudo tee -a /etc/cvmfs/config.d/neurodesk.ardc.edu.au.conf

echo "CVMFS_HTTP_PROXY=DIRECT" | sudo tee  /etc/cvmfs/default.local
echo "CVMFS_QUOTA_LIMIT=5000" | sudo tee -a  /etc/cvmfs/default.local

sudo cvmfs_config setup

For WSL users

It is required to run this for each new new WSL session:

sudo cvmfs_config wsl2_start

Test if the connection works:

sudo cvmfs_config chksetup

ls /cvmfs/neurodesk.ardc.edu.au

sudo cvmfs_talk -i neurodesk.ardc.edu.au host probe
sudo cvmfs_talk -i neurodesk.ardc.edu.au host info

cvmfs_config stat -v neurodesk.ardc.edu.au

For Ubuntu 22.04 users

If configuring CVMFS returns the following error:

Error: failed to load cvmfs library, tried: './libcvmfs_fuse3_stub.so' '/usr/lib/libcvmfs_fuse3_stub.so' '/usr/lib64/libcvmfs_fuse3_stub.so' './libcvmfs_fuse_stub.so' '/usr/lib/libcvmfs_fuse_stub.so' '/usr/lib64/libcvmfs_fuse_stub.so'
./libcvmfs_fuse3_stub.so: cannot open shared object file: No such file or directory
/usr/lib/libcvmfs_fuse3_stub.so: cannot open shared object file: No such file or directory
/usr/lib64/libcvmfs_fuse3_stub.so: cannot open shared object file: No such file or directory
./libcvmfs_fuse_stub.so: cannot open shared object file: No such file or directory
libcrypto.so.1.1: cannot open shared object file: No such file or directory
/usr/lib64/libcvmfs_fuse_stub.so: cannot open shared object file: No such file or directory


Failed to read CernVM-FS configuration

A temporary workaround is:

wget https://mirror.umd.edu/ubuntu/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb
dpkg -i libssl1.1_1.1.1f-1ubuntu2.15_amd64.deb

Install singularity/apptainer

e.g. for Ubuntu/Debian:

export VERSION=1.18.3 OS=linux ARCH=amd64 && \
    wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
    sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
    rm go$VERSION.$OS-$ARCH.tar.gz

echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
    echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
    source ~/.bashrc

go get -d github.com/sylabs/singularity

export VERSION=v3.10.0 # or another tag or branch if you like && \
    cd $GOPATH/src/github.com/sylabs/singularity && \
    git fetch && \
    git checkout $VERSION # omit this command to install the latest bleeding edge code from master

export VERSION=3.10.0 && # adjust this as necessary \
    mkdir -p $GOPATH/src/github.com/sylabs && \
    cd $GOPATH/src/github.com/sylabs && \
    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-ce-${VERSION}.tar.gz && \
    tar -xzf singularity-ce-${VERSION}.tar.gz && \789
    cd ./singularity-ce-${VERSION} && \
    ./mconfig --without-seccomp --without-conmon

./mconfig --without-seccomp --without-conmon && \
    make -C ./builddir && \
    sudo make -C ./builddir install

export PATH="/usr/local/singularity/bin:${PATH}"

Use of Neurodesk CVMFS containers

The containers are now available in /cvmfs/neurodesk.ardc.edu.au/containers/ and can be started with:

singularity shell /cvmfs/neurodesk.ardc.edu.au/containers/itksnap_3.8.0_20201208/itksnap_3.8.0_20201208.simg

make sure that SINGULARITY_BINDPATH include the directories you want to work with:

export SINGULARITY_BINDPATH='/cvmfs,/mnt,/home'

For WSL users

The homedirectory might not be supported. Avoid mounting it with

singularity shell --no-home /cvmfs/neurodesk.ardc.edu.au/containers/itksnap_3.8.0_20201208/itksnap_3.8.0_20201208.simg

or configure permanently:

sudo vi /etc/singularity/singularity.conf

set

mount home = no

Install module system

sudo yum install lmod

or

sudo apt install lmod

Use of containers in the module system

Configuration for module system

Create a the new file /usr/share/module.sh with the content:

# system-wide profile.modules                                          #
# Initialize modules for all sh-derivative shells                      #
#----------------------------------------------------------------------#
trap "" 1 2 3

case "$0" in
    -bash|bash|*/bash) . /usr/share/lmod/6.6/init/bash ;;
       -ksh|ksh|*/ksh) . /usr/share/lmod/6.6/init/ksh ;;
       -zsh|zsh|*/zsh) . /usr/share/lmod/6.6/init/zsh ;;
          -sh|sh|*/sh) . /usr/share/lmod/6.6/init/sh ;;
                    *) . /usr/share/lmod/6.6/init/sh ;;  # default for scripts
esac

trap - 1 2 3

Make the module system usable in the shell

Add the following lines to your ~/.bashrc file:

if [ -f '/usr/share/module.sh' ]; then source /usr/share/module.sh; fi

if [ -d /cvmfs/neurodesk.ardc.edu.au/neurodesk-modules ]; then
        # export MODULEPATH="/cvmfs/neurodesk.ardc.edu.au/neurodesk-modules"
        module use /cvmfs/neurodesk.ardc.edu.au/neurodesk-modules/*
else
        export MODULEPATH="/neurodesktop-storage/containers/modules"              
        module use $MODULEPATH
        export CVMFS_DISABLE=true
fi

if [ -f '/usr/share/module.sh' ]; then
        echo 'Run "ml av" to see which tools are available - use "ml <tool>" to use them in this shell.'
        if [ -v "$CVMFS_DISABLE" ]; then
                if [ ! -d $MODULEPATH ]; then
                        echo 'Neurodesk tools not yet downloaded. Choose tools to install from the Application menu.'
                fi
        fi
fi

Restart the current shell or run

source ~/.bashrc

Use of containers in the module system

export SINGULARITY_BINDPATH='/cvmfs,/mnt,/home'
module use /cvmfs/neurodesk.ardc.edu.au/neurodesk-modules/*
ml fsl
fslmaths

3 - Datalad

4 - Docker

Our containers are automatically built in https://github.com/NeuroDesk/neurocontainers/ and hosted on dockerhub or on github

Pull Docker containers

e.g. for a julia container docker

docker pull vnmd/julia_1.6.1

build singularity image from dockerhub

singularity build julia_1.6.1.simg docker://vnmd/julia_1.6.1

Replace julia_1.6.1 with your selected application. You can find the available containers here: https://www.neurodesk.org/applications/

5 - Google Colab

Open a notebook in Google Colab and run the following commands to setup the Neurodesk environment:

import os
os.environ["LD_PRELOAD"] = "";
os.environ["APPTAINER_BINDPATH"] = "/content"
os.environ["MPLCONFIGDIR"] = "/content/matplotlib-mpldir"
os.environ["LMOD_CMD"] = "/usr/share/lmod/lmod/libexec/lmod"

!curl -J -O https://raw.githubusercontent.com/NeuroDesk/neurocommand/main/googlecolab_setup.sh
!chmod +x googlecolab_setup.sh
!./googlecolab_setup.sh

os.environ["MODULEPATH"] = ':'.join(map(str, list(map(lambda x: os.path.join(os.path.abspath('/cvmfs/neurodesk.ardc.edu.au/neurodesk-modules/'), x),os.listdir('/cvmfs/neurodesk.ardc.edu.au/neurodesk-modules/')))))

Once this setup is completed you can list Neurodesk applications like this:

import lmod
await lmod.avail()

and use applications like this:

await lmod.load('fsl/6.0.4')
!bet

This notebook demonstrates how to use all Neurodesk applications in Google Colab: https://colab.research.google.com/drive/1g5cnZxj1llRaHmOs4xSglqsXnFkQYuol?usp=sharing

This is a google colab notebook that shows how to integrate with google drive and it contains an example how to run fMRIprep in google colab: https://colab.research.google.com/drive/11wVBkjNvrzo2TkUAILtWnPumAeFAfqkl?usp=sharing

6 - Singularity

Our docker containers are converted to singularity containers and stored on Object storage.

Download Singularity Containers

First get an overview which containers are available as Singularity containers: https://github.com/NeuroDesk/neurodesk/blob/master/cvmfs/log.txt

curl -s https://raw.githubusercontent.com/NeuroDesk/neurodesk/master/cvmfs/log.txt

assign the container name to a variable:

export container=itksnap_3.8.0_20201208

Then download the containers. An easy way is to use CURL (e.g. downloading from the US location):

curl -X GET https://objectstorage.us-ashburn-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/${container}.simg -O

or from australia

curl -X GET https://objectstorage.ap-sydney-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/${container}.simg -O

or from europe

curl -X GET https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/${container}.simg -O

A faster way is pulling from multiple storage locations at once using aria2:

aria2c https://objectstorage.us-ashburn-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/${container}.simg https://objectstorage.eu-frankfurt-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/${container}.simg https://objectstorage.ap-sydney-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/${container}.simg 

Transparent Singularity

The singularity containers can be also be used in combination with our Transparent Singularity Tool tool, that wraps the executables inside a container to make them easily available for pipelines. More information can be found here:

one example to do this is:

curl -s https://raw.githubusercontent.com/NeuroDesk/neurodesk/master/cvmfs/log.txt
export container=itksnap_3.8.0_20201208
git clone https://github.com/NeuroDesk/transparent-singularity ${container}
cd ${container}
./run_transparent_singularity.sh ${container}

7 - Windows 11 and Windows Subsystem for Linux

1. Install WSL

Follow the instructions to enable Windows Subsystem for Linux 2 in Windows 11: https://docs.microsoft.com/en-us/windows/wsl/install

2. Configure CVMFS, Singularity and LMOD (only needs to be done once)

Install build tools

sudo apt update
sudo apt install make gcc

Install singularity

export SINGULARITY_VERSION=3.9.3 VERSION=1.17.2 OS=linux ARCH=amd64
wget -q https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz 
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz 
rm go$VERSION.$OS-$ARCH.tar.gz 
export GOPATH=${HOME}/go 
export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin 
mkdir -p $GOPATH/src/github.com/sylabs 
cd $GOPATH/src/github.com/sylabs 
wget -q https://github.com/sylabs/singularity/releases/download/v${SINGULARITY_VERSION}/singularity-ce-${SINGULARITY_VERSION}.tar.gz 
tar -xzvf singularity-ce-${SINGULARITY_VERSION}.tar.gz 
cd singularity-ce-${SINGULARITY_VERSION} 
./mconfig --prefix=/usr/local/singularity 
make -C builddir 
sudo make -C builddir install 
cd .. 
sudo rm -rf singularity-ce-${SINGULARITY_VERSION} 
sudo rm -rf /usr/local/go $GOPATH

Setup Bindpaths for Singularity (e.g. in .bashrc)

export PATH="/usr/local/singularity/bin:${PATH}"
export SINGULARITY_BINDPATH='/cvmfs,/mnt,/home'

CVMFS

Follow the instructions here: https://www.neurodesk.org/docs/neurocontainers/cvmfs/

LMOD

sudo apt install lmod

3. Use Neurodesk containers

sudo cvmfs_config wsl2_start

module use /cvmfs/neurodesk.ardc.edu.au/neurodesk-modules/*

Example usage of fsleyes:

ml fsl
fsleyes

List the available programs:

ml av