This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Developer Documentation

Documentation for Developers

Video tutorial

See below for a 57 minute tutorial video from ReproNim 2023

1 - Contributors

This section acknowledges the contributions made to the project.

If you contributed to the project please list yourself here with a description of your contribution. We try to update this page based on the git commit history:

Steffen Bollmann

  • https://github.com/stebo85
  • funding: Oracle Cloud (114k AUD), ECR Knowledge Exchange & Translation Fund (42k AUD), ARDC (CI for $566k AUD)
  • system architecture
  • CVMFS container deployment
  • initial desktop container prototype
  • container build scripts
  • application containers (afni, aslprep, code, convert3D, freesurfer, hdbet, minc mriqc, romeo, spm12, tgvqsm, bart, fatsegnet, fsl, itksnap, lcmodel, mritools, niistat, qsmxt, root, slicer, trackvis, ants, cat12, conn, diffusiontoolkit, gimp, mricrogl, mrtrix3, rstudio, slicersalt, surfice, vesselapp, bidstools, clearswi, connectomeworkbench, dsistudio, fmriprep, julia, mrtrix3tissue, palm rabies, spinalcordtoolbox)
  • migrating container recipes and bugfixes to neurodocker upstream (fsl, ants)
  • documentation
  • tutorials (QSM, SWI, Unwrapping, lcmodel, freesurfer)
  • google colab support
  • outreach (e.g. Mastodon, talks at conferences, youtube videos)

Aswin Narayanan

  • https://github.com/aswinnarayanan
  • funding: ARDC (CI for $566k AUD)
  • Neurocontainer devops
  • Neurodesktop development
  • Neurocommand installer rewrite
  • Neurodesk Play & Kubernetes implementation
  • Jupyter notebook support
  • Hugo website build and documentation

Angela Renton

  • https://github.com/air2310
  • Tutorials (MNE-Python, Tutorial template)
  • graphics for website (layer diagram)
  • documentation
  • user testing
  • neurodesk paper lead author

Thuy Dao

  • https://github.com/iishiishii
  • Application search tool with lunr
  • platform independent GUI application (Neurodeskapp)
  • application container development (civet)
  • documentation (github workflow)
  • automated building and testing of example documentation from jupyter notebooks
  • development of interactive container building tool

Oren Civier

  • https://github.com/civier
  • Funding: ARDC Australian Electrophysiology Data Analytics PlaTform (AEDAPT) (CI for $566k AUD) - contributing to initial conceptualisation, EOI writeup, scope of project, proposal writeup, teaming up with the Australian Imaging Service (AIS), recruiting collaborators, assisting collaborators with case studies
  • Design: leading the Virtual Neuro Machine (VNM) hackathon project in the 2020 OHBM BrainHack, where the first version of Neurodesktop was developed
  • Development: allowing Neurodesk containers running in Neurodesktop to access sshfs mounts
  • Development: template for container recipe documentation
  • Development: software application containers - developer (bidscoin, MATLAB; in progress: MMVT)
  • Development: software application containers - facilitator (Fieldtrip, running arbitrary scripts using compiled MATLAB containers; in progress: SOVABIDS)
  • Documentation: for developers (contribution to “add tools”, configuring Github)
  • Documentation: for users (copy and paste troubleshooting, accessing storage, installation on different platforms, using ARDC Virtual Desktop Service, screenshots)
  • Documentation: Neurodesk’s original logo
  • User testing: HPC, NECTAR, Mac, Linux, ARDC Virtual Desktop Service
  • User testing: VNC and RDP interfaces, including multiple concurrent users
  • Outreach: providing assistance to nodes of the Australian National Imaging Facility with installing/using Neurodesk
  • Administration: one of Neurodesk/AEDAPT representatives in ARCOS, AIS and NECTAR Interactive Analytics committees and working groups
  • Papers: co-author Neurodesk manuscript (in preparation; contribution to initial outline, input on first draft), co-author proceedings of the OHBM Brainhack 2021 (to be published in Aperture)

Thomas Shaw

  • https://github.com/thomshaw92
  • Win, Mac, Linux startup scripts
  • initial transparent singularity prototype
  • application container development (LASHiS, ASHS)
  • user testing

Tom Johnstone

Martin Grignard

David White

Akshaiy Narayanan

Kelly Garner

Paris Lyons

  • design of Neurodesk Logo
  • project management of AEDAPT project

Ashley Stewart

  • https://github.com/astewartau
  • application container development (qsmxt)
  • presentation of neurodesk at OHBM Brainhack 2022 and OHBM educational course 2022

Lars Kasper

Judy D Zhu

Korbinian Eckstein

Stefanie Evas

Xincheng Ye

Fernanda Ribeiro

Jeryn Chang

Sin Kim

Jakub Kaczmarzyk

Alan Hockings

Aditya Garg

  • application container development (hdbet)

Kexin Lou

Renzo Huber

Monika Doerig

  • fsl nipype fMRI example

Marcel Zwiers

Steering Committee members without code contributions:

  • Ryan Sullivan, University of Sydney, Key User, Steering Committee
  • Thomas Close, University of Sydney, Key User, Scientific/Subject Expert Advisory Board
  • Wojtek Goscinski, Monash University, Steering Committee, Technical Advisory Board
  • Tony Hannan, Florey Institute of Neuroscience and Mental Health, Scientific/Subject Expert Advisory Board
  • Gary Egan, Monash University, Steering Committee
  • Paul Sowman, Macquarie University, Key User, Scientific/Subject Expert Advisory Board
  • Marta Garrido, University of Melbourne, Key User, Scientific/Subject Expert Advisory Board
  • Patrick Johnston, Queensland University of Technology, Key User, Scientific/Subject Expert Advisory Board
  • Aina Puce, Indiana University, Key User, Scientific/Subject Expert Advisory Board
  • Franco Pestilli, Indiana University, Technical Advisory Board
  • Levin Kuhlmann, Monash University, Key User, Scientific/Subject Expert Advisory Board
  • Gershon Spitz, Monash Epworth Rehabilitation Research Centre, Key User, Scientific/Subject Expert Advisory Board
  • David Abbott, Florey Institute of Neuroscience and Mental Health, Key User, Scientific/Subject Expert Advisory Board
  • Megan Campbell, The University of Newcastle, Key User, Scientific/Subject Expert Advisory Board
  • Nigel Rogasch, University of Adelaide, Key User, Scientific/Subject Expert Advisory Board
  • Will Woods, Swinburne University of Technology, Key User
  • Satrajit Ghosh, Massachusetts Institute of Technology, Provision of advice only

2 - Architecture

The architecture of the Neurodesk ecosystem

2.1 - Neurodesk Architecture

The architecture of the Neurodesk ecosystem

architecture

Neurodesktop is a compact Docker container with a browser-accessible virtual desktop that allows you develop and implement data analysis, pre-equipped with basic fMRI and EEG analysis tools. To get started, see: Neurodesktop (Github)

  • docker container with interface modifications
  • contains tools necessary to manage workflows in sub-containers: vscode, git
  • CI: builds docker image and tests if it runs; tests if CVMFS servers are OK before deployment
  • CD: pushes images to github & docker registry

Neurocommand:

Neurocommand offers the option to install and manage multiple distinct containers for more advanced users who prefer a command-line interface. Neurocommand is the recommended interface for users seeking to use Neurodesk in high performance computing (HPC) environments.

To get started, see: Neurocommand (Github)

  • script to install and manage multiple containers using transparent singularity on any linux system
  • this repo also handles the creation of menu entries in a general form applicable to different desktop environments
  • this repo can be re-used in other projects like CVL and when installing it on bare-metal workstations
  • CI: tests if containers can be installed
  • CD: this repo checks if containers requested in apps.json file are available on object storage and if not converts the singularity containers based on the docker containers and uploads them to object storage

transparent-singularity:

transparent-singularity offers seamless access to applications installed in neurodesktop and neurocommand, treating containerised software as native installations.

More info: transparent-singularity (Github)

  • script to install neuro-sub-containers, installers are called by neurocommand
  • this repo provides a way of using our containers on HPCs for large scale processing of the pipelines (including the support of SLURM and other job schedulers)
  • CI: test if exposing of binaries from container works

Neurocontainers:

neurocontainers contains scripts for building sub-containers for neuroimaging data-analysis software. These containers can be used alongside neurocommand or transparent-singularity.

To get started, see: Neurocontainers (Github)

  • build scripts for neuro-sub-containers
  • CI: building and testing of containers
  • CD: pushing containers to github and dockerhub registry

Neurodocker:

Neurodocker is a command-line program that generates custom Dockerfiles and Singularity recipes for neuroimaging and minifies existing containers.

More info: Github

  • fork of neurodocker project
  • provides recipes for our containers built
  • we are providing pull requests back of recipes
  • CI: handled by neurodocker - testing of generating container recipes

2.2 - Neurodesktop Release Process

A description of what to do to create new release of our Neurodesktop

Neurodesktop:

  1. Check if the last automated build ran OK: https://github.com/NeuroDesk/neurodesktop/actions
  2. Run this build date and test if everything is ok and no regression happened
  3. Check what changes where made since the last release: https://github.com/NeuroDesk/neurodesktop/commits/main
  4. Summarize the main changes and copy this to the Release History: https://www.neurodesk.org/docs/overview/release-history/
  5. Change the version of the latest desktop in https://github.com/NeuroDesk/neurodesk.github.io/blob/main/data/neurodesktop.toml
  6. Commit all changes
  7. Tweet a quick summary of the changes and announce new version: https://masto.ai/@Neurodesk

Neurodesk App:

Release process

Follow these steps to create a new release of the Neurodesk App.

  1. If there’s new version of Neurodesktop image, Github Action will PR with updated jupyter_neurodesk_version in neurodesktop.toml file. Double-check and merge this PR.

  2. Create a new release on GitHub as pre-release. Set the release tag to the value of target application version and prefix it with v (for example v1.0.0 for Neurodesk App version 1.0.0). Enter release title and release notes. Release needs to stay as pre-release for GitHub Actions to be able to attach installers to the release.

  3. Make sure that application is building, installing and running properly.

  4. In the main branch, create a branch preferably with the name release-v<new-version>. Add a commit with the version changes in package.json file. This is necessary for GitHub Actions to be able to attach installers to the release. (for example "version": "1.0.0").

  5. GitHub Actions will automatically create installers for each platform (Linux, macOS, Windows) and upload them as release assets. Assets will be uploaded only if a release of type pre-release with tag matching the Neurodesk App’s version with a v prefix is found. For example, if the Neurodesk App version in the PR is 1.0.0, the installers will be uploaded to a release that is flagged as pre-release and has a tag v1.0.0. New commits to this branch will overwrite the installer assets of the release.

  6. Once all the changes are complete, and installers are uploaded to the release then publish the release.

Update MacOS certificate

Follow these step-by-step instructions to generate and export the required Macos certificate for Neurodesk App release.

  1. Launch the “Keychain Access” application on your Mac, go to “Certificate Assistant.”
  2. Request Certificate from Certificate Authority: Within “Certificate Assistant,” select “Request a Certificate from a Certificate Authority.”
  3. Follow the URL to access the Apple Developer website: https://developer.apple.com/account/resources/certificates/add, upload the generated certificate.
  4. After uploading the certificate, download the resulting file provided by the Apple Developer website.
  5. Import the Certificate in Keychain.
  6. Right-click the imported certificate in “Keychain Access.”, choose “Export” and save it in .p12 format.
  7. Convert the .p12 file to Base64 using the following command: openssl base64 -in neurodesk_certificate.p12

2.3 - Neurodesktop Dev

Testing the latest dev version of Neurodesktop

Building neurodesktop-dev

Dev builds can be triggered by Neurodesk admins from https://github.com/NeuroDesk/neurodesktop/actions/workflows/build-neurodesktop-dev.yml

Running latest neurodesktop-dev

Linux

docker pull vnmd/neurodesktop-dev:latest
sudo docker run \
  --shm-size=1gb -it --cap-add SYS_ADMIN \
  --security-opt apparmor:unconfined --device=/dev/fuse \
  --name neurodesktop-dev \
  -v ~/neurodesktop-storage:/neurodesktop-storage \
  -e NB_UID="$(id -u)" -e NB_GID="$(id -g)" \
  -p 8888:8888 -e NEURODESKTOP_VERSION=dev \
  vnmd/neurodesktop-dev:latest

Windows

docker pull vnmd/neurodesktop-dev:latest
docker run --shm-size=1gb -it --cap-add SYS_ADMIN --security-opt apparmor:unconfined --device=/dev/fuse --name neurodesktop -v C:/neurodesktop-storage:/neurodesktop-storage -p 8888:8888 -e NEURODESKTOP_VERSION=dev vnmd/neurodesktop-dev:latest

2.4 - Transparent Singularity

For more advanced users who wish to use Transparent Singularity directly

Transparent singularity is here https://github.com/NeuroDesk/transparent-singularity/

This project allows to use singularity containers transparently on HPCs, so that an application inside the container can be used without adjusting any scripts or pipelines (e.g. nipype).

Important: add bind points to .bashrc before executing this script

This script expects that you have adjusted the Singularity Bindpoints in your .bashrc, e.g.:

export SINGULARITY_BINDPATH="/gpfs1/,/QRISdata,/data"

This gives you a list of all tested images available in neurodesk:

https://github.com/NeuroDesk/neurocommand/blob/main/cvmfs/log.txt

curl -s https://raw.githubusercontent.com/NeuroDesk/neurocommand/main/cvmfs/log.txt

Clone repo into a folder with the intended image name

git clone https://github.com/NeuroDesk/transparent-singularity convert3d_1.0.0_20210104

Install

This will create scripts for every binary in the container located in the $DEPLOY_PATH inside the container. It will also create activate and deactivate scripts and module files for lmod (https://lmod.readthedocs.io/en/latest)

cd convert3d_1.0.0_20210104
./run_transparent_singularity.sh convert3d_1.0.0_20210104

Options for Transparent singularity:

  • --storage - this option can be used to force a download from docker, e.g.: --storage docker
  • --container - this option can be used to explicitly define the container name to be downloaded
  • --unpack - this will unpack the singularity container so it can be used on systems that do not allow to open simg / sif files for security reasons, e.g.: --unpack true
  • --singularity-opts - this will be passed on to the singularity call, e.g.: --singularity-opts '--bind /cvmfs'

Use in module system LMOD

Add the module folder path to $MODULEPATH

Manual activation and deactivation (in case module system is not available). This will add the paths to the .bashrc

Activate

source activate_convert3d_1.0.0_20210104.sh

Deactivate

source deactivate_convert3d_1.0.0_20210104.sif.sh

Uninstall container and cleanup

./ts_uninstall.sh

2.5 - Neurodesk CVMFS

How to interact with our CVMFS service.

2.5.1 - Setup CVMFS Proxy

Setup CVMFS Proxy server

If you want more speed in a region one way could be to setup another Stratum 1 server or a proxy. We currently don’t run any proxy servers but it would be important for using it on a cluster.

docker run --shm-size=1gb -it --privileged --user=root --name neurodesktop `
-v C:/neurodesktop-storage:/neurodesktop-storage -p 8888:8888 `
-e NEURODESKTOP_VERSION=2024-01-12 `
vnmd/neurodesktop:2024-01-12

Setup a CVMFS proxy server

sudo yum install -y squid

Open the squid.confand use the following configuration

sudo vi /etc/squid/squid.conf
# List of local IP addresses (separate IPs and/or CIDR notation) allowed to access your local proxy
#acl local_nodes src YOUR_CLIENT_IPS

# Destination domains that are allowed
#acl stratum_ones dstdomain .YOURDOMAIN.ORG
#acl stratum_ones dstdom_regex YOUR_REGEX
acl stratum_ones dst 140.238.211.92

# Squid port
http_port 3128

# Deny access to anything which is not part of our stratum_ones ACL.
http_access deny !stratum_ones

# Only allow access from our local machines
#http_access allow local_nodes
http_access allow localhost

# Finally, deny all other access to this proxy
http_access deny all

minimum_expiry_time 0
maximum_object_size 1024 MB

cache_mem 128 MB
maximum_object_size_in_memory 128 KB
# 5 GB disk cache
cache_dir ufs /var/spool/squid 5000 16 256
sudo squid -k parse
sudo systemctl start squid
sudo systemctl enable squid
sudo systemctl status squid
sudo systemctl restart squid

2.5.2 - CVMFS architecture

CVMFS architecture

We store our singularity containers unpacked on CVMFS. We tried the DUCC tool in the beginning, but it was causing too many issues with dockerhub and we were rate limited. The script to unpack our singularity containers is here: https://github.com/NeuroDesk/neurocommand/blob/main/cvmfs/sync_containers_to_cvmfs.sh

It gets called by a cronjob on the CVMFS Stratum 0 server and relies on the log.txt file being updated via an action in the neurocommand repository (https://github.com/NeuroDesk/neurocommand/blob/main/.github/workflows/upload_containers_simg.sh)

The Stratum 1 servers then pull this repo from Stratum 0 and our desktops mount these repos (configured here: https://github.com/NeuroDesk/neurodesktop/blob/main/Dockerfile)

The startup script (https://github.com/NeuroDesk/neurodesktop/blob/main/config/jupyter/before_notebook.sh) sets up CVMFS and tests which server is fastest during the container startup.

This can also be done manually:

sudo cvmfs_talk -i neurodesk.ardc.edu.au host info
sudo cvmfs_talk -i neurodesk.ardc.edu.au host probe
cvmfs_config stat -v neurodesk.ardc.edu.au

2.5.3 - Setup Stratum 0 server

Host a Stratum 0 server

Setup a Stratum 0 server:

Setup Storage

(would object storage be better? -> see comment below under next iteration ideas)

lsblk -l
sudo mkfs.ext4 /dev/vdb
sudo mkdir /storage
sudo mount /dev/vdb /storage/ -t auto
sudo chown ec2-user /storage/
sudo chmod a+rwx /storage/
sudo vi /etc/fstab
/dev/vdb  /storage    auto    defaults,nofail   0  2

Setup server

sudo yum install vim htop gcc git screen
sudo timedatectl set-timezone Australia/Brisbane

sudo yum install -y https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest.noarch.rpm
sudo yum install -y cvmfs cvmfs-server

sudo systemctl enable httpd
sudo systemctl restart httpd

# sudo systemctl stop firewalld

# restore keys:
sudo mkdir /etc/cvmfs/keys/incoming
sudo chmod a+rwx /etc/cvmfs/keys/incoming
cd connections/cvmfs_keys/
scp neuro* ec2-user@203.101.226.164:/etc/cvmfs/keys/incoming
sudo mv /etc/cvmfs/keys/incoming/* /etc/cvmfs/keys/

#backup keys: 
#mkdir cvmfs_keys
#scp opc@158.101.127.61:/etc/cvmfs/keys/neuro* .

sudo cvmfs_server mkfs -o $USER neurodesk.ardc.edu.au

cd /storage
sudo mkdir -p cvmfs-storage/srv/
cd /srv/
sudo mv cvmfs/ /storage/cvmfs-storage/srv/
sudo ln -s /storage/cvmfs-storage/srv/cvmfs/

cd /var/spool
sudo mkdir /storage/spool
sudo mv cvmfs/ /storage/spool/
sudo ln -s  /storage/spool/cvmfs .

cvmfs_server transaction neurodesk.ardc.edu.au

cvmfs_server publish neurodesk.ardc.edu.au
sudo vi /etc/cron.d/cvmfs_resign
0 11 * * 1 root /usr/bin/cvmfs_server resign neurodesk.ardc.edu.au
cat /etc/cvmfs/keys/neurodesk.ardc.edu.au.pub
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuV9JBs9uXBR83qUs7AiE
nSQfvh6VCdNigVzOfRMol5cXsYq3cFy/Vn1Nt+7SGpDTQArQieZo4eWC9ww2oLq0
vY1pWyAms3Y4i+IUmMbwNifDU4GQ1KN9u4zl9Peun2YQCLE7mjC0ZLQtLM7Q0Z8h
NwP8jRJTN+u8mRKzkyxfSMLscVMKhm2pAwnT1zB9i3bzVV+FSnidXq8rnnzNHMgv
tfqx1h0gVyTeodToeFeGG5vq69wGZlwEwBJWVRGzzr+a8dWNBFMJ1HxamrBEBW4P
AxOKGHmQHTGbo+tdV/K6ZxZ2Ry+PVedNmbON/EPaGlI8Vd0fascACfByqqeUEhAB
dQIDAQAB
-----END PUBLIC KEY-----

Next iteration of this:

use object storage?

  • current implementation uses block storage, but this makes increasing the volume size a bit more work
  • we couldn’t get object storage to work on Oracle as it assumes AWS S3

Optimize settings for repositories for Container Images

from the CVMFS documentation: Repositories containing Linux container image contents (that is: container root file systems) should use overlayfs as a union file system and have the following configuration:

CVMFS_INCLUDE_XATTRS=true
CVMFS_VIRTUAL_DIR=true

Extended attributes of files, such as file capabilities and SElinux attributes, are recorded. And previous file system revisions can be accessed from the clients.

Currently not used

We tested the DUCC tool in the beginning, but it was leading to too many docker pulls and we therefore replaced it with our own script: https://github.com/NeuroDesk/neurocommand/blob/main/cvmfs/sync_containers_to_cvmfs.sh

This is the old DUCC setup

sudo yum install cvmfs-ducc.x86_64
sudo -i
dnf install -y yum-utils 
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install docker-ce docker-ce-cli containerd.io
systemctl enable docker
systemctl start docker
docker version
docker info

# leave root mode

sudo groupadd docker
sudo usermod -aG docker $USER
sudo chown root:docker /var/run/docker.sock
newgrp docker


vi convert_appsjson_to_wishlist.sh
export DUCC_DOCKER_REGISTRY_PASS=configure_secret_password_here_and_dont_push_to_github
cd neurodesk
git pull
./gen_cvmfs_wishlist.sh
cvmfs_ducc convert recipe_neurodesk_auto.yaml
cd ..


chmod +x convert_appsjson_to_wishlist.sh

git clone https://github.com/NeuroDesk/neurodesk/

# setup cron job
sudo vi /etc/cron.d/cvmfs_dockerpull
*/5 * * * * opc cd ~ && bash /home/opc/convert_appsjson_to_wishlist.sh



#vi recipe.yaml

##version: 1
#user: vnmd
#cvmfs_repo: neurodesk.ardc.edu.au
#output_format: '$(scheme)://$(registry)/vnmd/thin_$(image)'
#input:
#- 'https://registry.hub.docker.com/vnmd/tgvqsm_1.0.0:20210119'
#- 'https://registry.hub.docker.com/vnmd/itksnap_3.8.0:20201208'


#cvmfs_ducc convert recipe_neurodesk.yaml
#cvmfs_ducc convert recipe_unpacked.yaml

2.5.4 - Setup Stratum 1 server

Host a Stratum 1 server

The stratum 1 servers for the desktop are configured here: https://github.com/NeuroDesk/neurodesktop/blob/main/Dockerfile

If you want more speed in a region one way could be to setup another Stratum 1 server or a proxy.

Setup a Stratum 1 server:

sudo yum install -y https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest.noarch.rpm
sudo yum install -y cvmfs-server squid
sudo yum install -y python3-mod_wsgi 

sudo sed -i 's/Listen 80/Listen 127.0.0.1:8080/' /etc/httpd/conf/httpd.conf

set +H
echo "http_port 80 accel" | sudo tee /etc/squid/squid.conf
echo "http_port 8000 accel" | sudo tee -a /etc/squid/squid.conf
echo "http_access allow all" | sudo tee -a /etc/squid/squid.conf
echo "cache_peer 127.0.0.1 parent 8080 0 no-query originserver" | sudo tee -a /etc/squid/squid.conf
echo "acl CVMFSAPI urlpath_regex ^/cvmfs/[^/]*/api/" | sudo tee -a /etc/squid/squid.conf
echo "cache deny !CVMFSAPI" | sudo tee -a /etc/squid/squid.conf
echo "cache_mem 128 MB" | sudo tee -a /etc/squid/squid.conf

sudo systemctl start httpd
sudo systemctl start squid
sudo systemctl enable httpd
sudo systemctl enable squid

echo 'CVMFS_GEO_LICENSE_KEY=kGepdzqbAP4fjf5X' | sudo tee -a /etc/cvmfs/server.local
sudo chmod 600 /etc/cvmfs/server.local

sudo mkdir -p /etc/cvmfs/keys/ardc.edu.au/

echo "-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwUPEmxDp217SAtZxaBep
Bi2TQcLoh5AJ//HSIz68ypjOGFjwExGlHb95Frhu1SpcH5OASbV+jJ60oEBLi3sD
qA6rGYt9kVi90lWvEjQnhBkPb0uWcp1gNqQAUocybCzHvoiG3fUzAe259CrK09qR
pX8sZhgK3eHlfx4ycyMiIQeg66AHlgVCJ2fKa6fl1vnh6adJEPULmn6vZnevvUke
I6U1VcYTKm5dPMrOlY/fGimKlyWvivzVv1laa5TAR2Dt4CfdQncOz+rkXmWjLjkD
87WMiTgtKybsmMLb2yCGSgLSArlSWhbMA0MaZSzAwE9PJKCCMvTANo5644zc8jBe
NQIDAQAB
-----END PUBLIC KEY-----" | sudo tee /etc/cvmfs/keys/ardc.edu.au/neurodesk.ardc.edu.au.pub


sudo cvmfs_server add-replica -o $USER http://203.101.226.164/cvmfs/neurodesk.ardc.edu.au /etc/cvmfs/keys/ardc.edu.au

# CVMFS will store everything in /srv/cvmfs so make sure there is enough space or create a symlink to a bigger storage volume
# e.g.:
<!-- cd /storage
sudo mkdir -p cvmfs-storage/srv/
cd /srv/
sudo mv cvmfs/ /storage/cvmfs-storage/srv/
sudo ln -s /storage/cvmfs-storage/srv/cvmfs/ -->


sudo cvmfs_server snapshot neurodesk.ardc.edu.au


echo "/var/log/cvmfs/*.log {
    weekly
    missingok
    notifempty
}" | sudo tee /etc/logrotate.d/cvmfs


echo '*/5 * * * * root output=$(/usr/bin/cvmfs_server snapshot -a -i 2>&1) || echo "$output" ' | sudo tee /etc/cron.d/cvmfs_stratum1_snapshot

sudo yum install iptables
sudo iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8000

sudo systemctl disable firewalld 
sudo systemctl stop firewalld 
# make sure that port 80 is open in the real firewall

sudo cvmfs_server update-geodb

2.6 - Project Roadmap

This page lists ideas we have planned for Neurodesk - If any of these things sound exciting to you, get in touch and we help you to become a contributor.

All things we are currently working on and are planning to do are listed here: https://github.com/orgs/NeuroDesk/projects/9/views/4

The larger themes and subthemes are:

Streamlining container build and release process

Improving the workflow of how users can add new applications to Neurodesk

Adding new applications to Neurodesk requires multiple steps and back-and-forth between contributors and maintainers. We are aiming to simplify this process by developing an interactive workflow based on our current interactive container builder and the existing github workflows.

Some issues in this theme are:

Standardizing the container deployment

Currently, deploying the application containers happens through a connection of various custom scripts distributed across various repositories (apps.json in neurocommand repository, neurocontainers, transparent singularity). We would like to adopt community standard tools, like SHPC, that can perform some of these tasks. The goal is to remove duplication of effort and maintenance.

Some issues in this theme are:

Reuse and citability of containers

Currently, there is no good way of describing and citing the individual software containers. We want to increase the reusability and citability of the software containers.

Some issues in this theme are:

Improving user experience

Improving documentation

We would love to have more tutorials and examples that help people perform Neuroimaging analyses in Neurodesk. When we developed our current documentation system (https://www.neurodesk.org/tutorials-examples/), we wanted to develop an interactive documentation system that ensures that examples always work correctly because they are automatically tested. We have a first proof-of-concept that runs jupyter notebooks and converts them to a website: https://www.neurodesk.org/example-notebooks/intro.html - but currently errors are not flagged automatically and it needs manual checking.

Some issues in this theme are:

Facilitating the use of Neurodesk in teaching and workshops

Neurodesk is a great fit for teaching Neuroimaging methods. Currently, however, it’s not easy to run a custom Neurodesk instance for a larger group. We would like to make it easier for users to deploy Neurodesk for classes and workshops with a shared data storage location and we would love to support advanced features for cost saving (e.g. autoscaling, support of ARM processors) on various cloud providers (e.g. Google Cloud, Amazon, Azure, OpenStack, OpenShift).

Support of advanced workflows

Deeper Integration of containers and jupyter notebook system

We want to integrate the software containers deeper into the jupyter notebook system. Ideally, it is possible to use a jupyter kernel from within a software container.

Some issues in this theme are:

Support of scheduling workflows

Currently, all Neurodesk work is entirely interactive. We want to add a way of scheduling workflows and jobs so that larger computations can be managed efficiently.

Some issues in this theme are:

3 - Documentation

How to edit the documentation

3.1 - Local Hugo Docsy

How to edit the documentation

Local Hugo Docsy in Linux and WSL2

https://github.com/NeuroDesk/neurodesk.github.io/blob/main/CONTRIBUTING.md

Local Hugo Docsy in Windows

Clone repository

Using SSH

git clone --recurse-submodules git@github.com:NeuroDesk/neurodesk.github.io.git

or Https:

git clone --recurse-submodules https://github.com/NeuroDesk/neurodesk.github.io.git

If you cloned without –recurse-submodules

Run the following command to pull submodules

git submodule update --init --recursive --remote

Download Hugo binary

Hugo releases are on https://github.com/gohugoio/hugo/releases

Download latest version of hugo extended

e.g. for windows: https://github.com/gohugoio/hugo/releases/download/v0.88.1/hugo_extended_0.88.1_Windows-64bit.zip

Start local hugo server

Extract hugo binary (hugo.exe) to your neurodesk.github.io dir

Run server for windows: .\hugo.exe server --disableFastRender

Once started, dev website will be accessible via http://localhost:1313

Update docsy theme submodule

git submodule update --remote
git add themes/
git commit -m "Updating theme submodule"
git push origin main

4 - How to add new tools

How to add new tools to neurodesk

Guiding principles

To decide if a tool should be packaged in a Neurocontainers or be installed in the Neurodesktop container, we are currently following these guiding principles:

1) Neurodesk is a Platform, Not a Package Manager: We don’t distribute tools that can be easily installed via standard package managers.

2) Multiple versions of tools: Neurodesk supports the use of multiple versions of a tool in parallel via lmod. If a tool doesn’t support this, follow this instruction to package it in Neurocontainers.

3) Inter-Container Tool Linking: Neurodesk is designed to facilitate the linking of tools from different containers, such as workflow managers like nipype or nextflow. Therefore, if a tool is needed to coordinate various container-tools, create an issue to have it installed directly in the Neurodesktop container.

Examples:

easy installcoordinates containerssmall in sizelatest version is okuseful to most usersConclusion
gityesyesyesyesyesneurodesktop
lmodnoyesyesyesyesneurodesktop
itksnapyesnoyesyesyesneurocontainer
convert3Dyesnoyesnononeurocontainer
fslnonononononeurocontainer

Adding new tools to Neurocontainers:

Follow either of these instructions to add new tools

via interactive container builder

Or

via manual container builder

4.1 - Interactive Build

How to contribute a new container.

1) Open an issue to get access to the interactive container build system

https://github.com/NeuroDesk/neurocontainers/issues/new

  • describe which container you would like to add
  • wait for a reply on your issue that your account has been setup

2) Access the container build system

https://labtokyo.neurodesk.org/

  • authenticate with your github account
  • select a CPU session or a GPU session (if your container requires a GPU)
  • open a Neurodesktop session

3) Run the interactive build process

  • open a Terminal session
  • run:
cd ~
git clone https://github.com/NeuroDesk/neurocontainers.git
cd neurocontainers/interactive_builder/
git pull
./run_interactive_builder.sh
  • Follow the instructions of the interactive build tool. After a couple of seconds where the base image gets updated you should see a “root@neurodesk-builder:~$>” shell. Now run the commands to get your tool to work.
  • Once the tool works, hit CTRL-D or type “exit”
  • Then answer more questions in the build tool

4) Submit the generated build.sh and Readme.md file as attachments to your issue

  • once completed, download the build.sh and README.md file and submit them as attachments to your github issue

4.2 - Manual Build

How to contribute a new container.

Adding new recipes

Refer to neurodocker for more information on neurodocker recipes.

Build container

Environment Requirements

Install Neurodocker

Neurodocker is the dependency we use to build containers.

  1. (optional) Sync upstream repository:
    If you have the permissions to do so: Press “Fetch upstream” in https://github.com/NeuroDesk/neurodocker to check if our fork of Neurodocker is already up-to-date. Otherwise, open an issue in https://github.com/NeuroDesk/neurocontainers/issues, requesting to pull-in latest changes from Neurodocker upstream into our fork of Neurodocker. One of the admins will attend the issue and perform the operation.
  2. (optional) Add a new neurodocker tool:
    If relevant to your project, add an option to neurodocker that installs new software (https://github.com/NeuroDesk/neurodocker) and create a pull request to neurodocker’s main repository (add new tool in a branch!).
  3. Clone our fork of Neurodocker:
    git clone https://github.com/NeuroDesk/neurodocker/
    
  4. Install neurodocker:
    cd neurodocker  
    python -m pip install .
    cd ..
    
  5. Append line to .bashrc for adding the path:
    echo 'export PATH=${PATH}:${HOME}/.local/bin' >> ${HOME}/.bashrc
    
  6. Close the terminal, and reopen it for the updated PATH to take effect

Fork the Neurocontainers repository

  • Fork neurocontainers and setup github actions.

Create a new app

  1. Copy the directory template and rename to NEWAPP in neurocontainers/recipes (NEWAPP being the name of the application to be displayed in Neurodesk’s menu; notice it shouldn’t have any special characters):

    cd neurocontainers/recipes
    cp -R template NEWAPP
    
  2. Create your Container Files:
    Modify build.sh in neurocontainers/recipes/NEWAPP to build your application and update README.md (make sure the version is correct in the README!). Notice that the example build script in the template has instructions to build a container for datalad, that may or may not suite your exact needs

    cd NEWAPP
    (edit build.sh as required)
    (edit README.md as required)
    

    Upload your application to object storage first if needed, so you can then download it in build.sh (ask for instructions about this if you don’t know the key, and never share it anywhere public!)

  3. Building containers

    Any NEWAPP under the recipes/ directory are built and pushed automatically via github actions

  4. Build and test the container locally

    1. run the build script with the debug flag:

      cd recipes/NEWAPP
      chmod +x build.sh
      ./build.sh -ds
      

      NOTICE: the README.md file will automatically be updated to reflect the version of the tool given in the build.sh script. For this to work, leave “toolVersion” in the README and do not remove this or alter.

    2. test running some commands within the container that should be available in your local docker container repository.

      For example, to open an interactive shell in a container (with the home folder /root bound to /root on host), you may run:

      sudo docker run -it -v /root:/root --entrypoint /bin/bash NEWAPP_VERSION:TAG
      

      with VERSION being the version of the app, and TAG the version tag of the container (run ‘sudo docker image list’ to find the tag)

    3. if your application requires a Matlab Runtime and you get an error about shared library “libmwlaunchermain.so” not found, check which version of the runtime was installed by the build script

  5. Update changes in local git repository

    git add .github/workflows/NEWAPP.yml recipes/NEWAPP/test.sh recipes/NEWAPP/build.sh recipes/NEWAPP/README.md
    git config user.email "the email that you use for github"
    git config user.name "your name"
    git commit
    

Push the new or updated app to Neurocontainers

Prerequisite

Generate git personal access token (if you don’t have one already)

  1. Browse to https://github.com/
  2. Log into your account
  3. Press on your picture in upper right corner → Setting → Developer Settings → Personal Access Token
  4. Press on “generate personal access token”
  5. Write something in “Notes” (doesn’t matter what, it’s for your own use)
  6. Check “repo”
  7. Check “Workflow”
  8. Press “Generate Token” at the bottom
  9. Copy the token displayed to somewhere safe, as you will have to user it later

Verify that user has write permission to /neurocommand/local

  1. If not, run sudo chmod a+w /neurocommand/local

Step by step guide

  1. Test the container locally, and if successful push repo to trigger the automatic build on GitHub. When asked for your Github password, please provide the personal access token obtained in the previous stage.

    git pull
    git push
    
  2. Go to https://github.com/neurodesk/neurocontainers/actions. Check that the most recent workflow run in the list terminated successfully (green). Otherwise, click on it, click on “build docker”, and the line that caused the error will be highlighted

  3. Find your new package under https://github.com/orgs/NeuroDesk/packages?repo_name=neurocontainers
    Enter the name of the package in the search box, and verify that the full package name shows up in the format toolName_toolVersion

  4. Obtain buildDate by clicking on the full package name that came up in the search. The build date will be the newest date shown under Recent tagged image versions

  5. If updating an app, use toolName delete the locally installed container of the old app version or old app build:

    rm -R /neurocommand/local/containers/toolName_*/
    rm -R /neurocommand/local/containers/modules/toolName/
    
  6. Use toolName, toolVersion and buildDate from the previous two steps to manually download the package by typing the following in a terminal open in Neurodesktop

    bash /neurocommand/local/fetch_and_run.sh toolName toolVersion buildDate
    (when you see the "Singularity>" prompt, type exit and ENTER)
    ml toolName/toolVersion
    

    For example: If the full package name that comes up in the step 11 is itksnap_3.8.0, and the newest date under Recent tagged image versions is 20210322

    The command to use in a terminal open in Neurodesktop is:

    bash /neurocommand/local/fetch_and_run.sh itksnap 3.8.0 20210322
    (when you see the "Singularity>" prompt, type exit and ENTER)
     ml toolName/toolVersion
    
  7. Test the new container. Run some commands, to see all is good
    If the container doesn’t work yet, it’s sometimes useful to try and troubleshoot it and install missing libraries. This can be achieved by running it in a writable mode with fakeroot enabled:

    SINGULARITY_BINDPATH=''; singularity shell --writable --fakeroot /neurodesktop-storage/containers/toolName_toolVersion_buildDate/toolName_toolVersion_buildDate.simg
    
  8. Fork https://github.com/NeuroDesk/neurocommand/ to your Github account

  9. Edit an entry for your package in your fork of neurocommand/blob/main/neurodesk/apps.json based on one of the other entries (generating one menu item for opening a terminal inside the containers, and one menu item for the GUI, if relevant). Notice that in the json file, the version field should contain the buildDate rather than the toolVersion !!! toolVersion should be included instead in the text of the menu entry itself, e.g., “fsl 6.0.3”. Also notice that whereas categories appear in the Neurodesktop menu in start case (first letter of each word capitalized), in the json files they are sentence case (all letters lower case).

  10. Include an icon file in your fork of neurocommand/neurodesk/icons

  11. Send a pull request from your fork of neurocommand to https://github.com/NeuroDesk/neurocommand/

  12. When the pull request is merged by Neurodesk admins, it will trigger an action to build the singularity container, distribute it in all object storage locations and on CVMFS, and it will update the menus in the desktop image on the next daily build.

  13. Wait at least 24 hours

  14. Download and run the daily build of Neurodesktop to check that your app can be launched from the start menu and works properly:

    sudo docker pull vnmd/neurodesktop:latest && sudo docker run --shm-size=1gb -it --privileged --user=root --name neurodesktop -v ~/neurodesktop-storage:/neurodesktop-storage -e HOST_UID="$(id -u)"  -e HOST_GID="$(id -g)" -p 8888:8888 -e NEURODESKTOP_VERSION=latest vnmd/neurodesktop:latest
    
  15. Open an issue in https://github.com/NeuroDesk/neurocontainers/issues notifying that your app appears in the start menu and tested. The app will be included in the next release of Neurodesktop, and will be mentioned in the public announcement that accompanies the release. If the app is not in the start menu or not working as expected based on your earlier testing, open an issue as well, and report it.

  16. If somebody wants to use the application before the next release of Neurodesktop is out, you can instruct them to use the command in step 14 above instead of the default commands given in the user install instructions.

  17. Consider contributing a tutorial about the new tool in this instruction

4.3 - Menu entries

Menu entries in neurodesktop

As we want to propose several versions of the tools, each piece of software should have its own submenu under VNM Neuroimaging. To do so, you first have to add a submenu to menus/vnm-applications.menu by adding:

<!-- [[Tool Name]] submenu -->
<Menu>
    <Name>[[Tool Name]]</Name>
    <Directory>vnm-[[tool-name]].directory</Directory>
    <Include>
        <And>
            <Category>[[Tool-Name]]</Category>
        </And>
    </Include>
</Menu> <!-- End [[Tool Name]] -->

The following table shows the formatting rules to follow:

PlaceholderRuleExample
[[Tool name]]Capitalized, spacesITK snap
[[tool-name]]Lower case, no spaces (use - instead)itk-snap or itksnap
[[Tool-name]]Capitalized, no spaces (use - instead)ITK-snap

Next, we have to create the submenu itself as we referenced it by vnm-[[tool-name]].directory. To do so, create the file menus/submenus/vnm-[[tool-name]].directory and add the following information inside:

[Desktop Entry]
Name=[[Tool Name]]
Comment=[[Tool Name]]
Icon=/home/neuro/.config/lxpanel/LXDE/icons/[[icon-name]].png
Type=Directory

If a specific icon is available in the menus/icons directory, replace [[icon-name]] by its name. Otherwise, use vnm.

Create the application

Finally, we have to create the actual application by creating the file menus/applications/vnm-[[tool-name]]-[[0.0.0]].desktop. The name of this file must contain the version of the tool (once again to allow multiple versions to live inside the same directory). Add the following description to this file:

[Desktop Entry]
Name=[[Tool Name]] [[0.0.0]] [[(Install only)]]
GenericName=[[Tool Name]] [[0.0.0]]
Comment=The description of what clicking on this application does. # This will be the tooltip of the application.
Exec=The command used to run the application.
Icon=/home/neuro/.config/lxpanel/LXDE/icons/[[icon-name]].png
Type=Application
Categories=[[Tool-name]]
Terminal=true # or false

The important part here is the value of Exec. If the tool is in the form of a singularity image, you should run the following command:

bash /usr/share/fetch_and_run.sh [[tool-name]] [[0.0.0]] [[YYYYMMDD]] [[cmd]] [[args]]

What fetch_and_run.sh does is check if the image is already installed as a module. If not, it checks whether it can be installed or not (return 1 if not possible). After that, it installs the image as a module. If [[cmd]] is specified, once the image is installed, it runs the command by giving the arguments from [[args]]. Here are two examples for FreeSurfer and FreeView. This first one only installs the image as a module:

bash /usr/share/fetch_and_run.sh freesurfer 6.0.1 20200506

And this does the same but runs FreeView afterward:

bash /usr/share/fetch_and_run.sh freesurfer 6.0.1 20200506 freeview

The resulting .desktop file corresponding to FreeView contains:

[Desktop Entry]
Name=FreeView 6.0.1
GenericName=FreeView 6.0.1
Comment=Start FreeView 6.0.1
Exec=bash /usr/share/fetch_and_run.sh freesurfer 6.0.1 20200506 freeview
Icon=/home/neuro/.config/lxpanel/LXDE/icons/run.png
Type=Application
Categories=FreeSurfer
Terminal=true