This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

How to add new tools

How to add new tools to neurodesk

Guiding principles

To decide if a tool should be packaged in a Neurocontainers or be installed in the Neurodesktop container, we are currently following these guiding principles:

1) Neurodesk is a Platform, Not a Package Manager: We don’t distribute tools that can be easily installed via standard package managers.

2) Multiple versions of tools: Neurodesk supports the use of multiple versions of a tool in parallel via lmod. If a tool doesn’t support this, follow this instruction to package it in Neurocontainers.

3) Inter-Container Tool Linking: Neurodesk is designed to facilitate the linking of tools from different containers, such as workflow managers like nipype or nextflow. Therefore, if a tool is needed to coordinate various container-tools, create an issue to have it installed directly in the Neurodesktop container.

Examples:

easy installcoordinates containerssmall in sizelatest version is okuseful to most usersConclusion
gityesyesyesyesyesneurodesktop
lmodnoyesyesyesyesneurodesktop
itksnapyesnoyesyesyesneurocontainer
convert3Dyesnoyesnononeurocontainer
fslnonononononeurocontainer

Adding new tools to Neurocontainers:

Follow either of these instructions to add new tools

via interactive container builder

Or

via manual container builder

1 - Interactive Build

How to contribute a new container.

1) Run the interactive build process in Neurodesk

  • open a Terminal session
  • run:
cd ~
git clone https://github.com/NeuroDesk/neurocontainers.git
cd neurocontainers/interactive_builder/
git pull
./run_interactive_builder.sh
  • Follow the instructions of the interactive build tool. After a couple of seconds when the base image gets updated, you should see a “root@neurodesk-builder:~$>” shell. Now run the commands to get your tool to work.
  • Once the tool works, hit CTRL-D or type “exit”
  • Then answer more questions in the build tool

2) Submit the generated build.sh and Readme.md file as attachments to your issue

  • once completed, download the build.sh and README.md file and submit them as attachments to a new github issue

2 - Manual Build

How to contribute a new container.

Adding new recipes

Refer to neurodocker for more information on neurodocker recipes.

Build container

Environment Requirements

Install Neurodocker

Neurodocker is the dependency we use to build containers.

  1. (optional) Sync upstream repository:
    If you have the permissions to do so: Press “Fetch upstream” in https://github.com/NeuroDesk/neurodocker to check if our fork of Neurodocker is already up-to-date. Otherwise, open an issue in https://github.com/NeuroDesk/neurocontainers/issues, requesting to pull-in latest changes from Neurodocker upstream into our fork of Neurodocker. One of the admins will attend the issue and perform the operation.
  2. (optional) Add a new neurodocker tool:
    If relevant to your project, add an option to neurodocker that installs new software (https://github.com/NeuroDesk/neurodocker) and create a pull request to neurodocker’s main repository (add new tool in a branch!).
  3. Clone our fork of Neurodocker:
    git clone https://github.com/NeuroDesk/neurodocker/
    
  4. Install neurodocker:
    cd neurodocker  
    python -m pip install .
    cd ..
    
  5. Append line to .bashrc for adding the path:
    echo 'export PATH=${PATH}:${HOME}/.local/bin' >> ${HOME}/.bashrc
    
  6. Close the terminal, and reopen it for the updated PATH to take effect

Fork the Neurocontainers repository

  • Fork neurocontainers and setup github actions.

Create a new app

  1. Copy the directory template and rename to NEWAPP in neurocontainers/recipes (NEWAPP being the name of the application to be displayed in Neurodesk’s menu; notice it shouldn’t have any special characters):

    cd neurocontainers/recipes
    cp -R template NEWAPP
    
  2. Create your Container Files:
    Modify build.sh in neurocontainers/recipes/NEWAPP to build your application and update README.md (make sure the version is correct in the README!). Notice that the example build script in the template has instructions to build a container for datalad, that may or may not suite your exact needs

    cd NEWAPP
    (edit build.sh as required)
    (edit README.md as required)
    

    Upload your application to object storage first if needed, so you can then download it in build.sh (ask for instructions about this if you don’t know the key, and never share it anywhere public!)

  3. Building containers

    Any NEWAPP under the recipes/ directory are built and pushed automatically via github actions

  4. Build and test the container locally

    1. run the build script with the debug flag:

      cd recipes/NEWAPP
      chmod +x build.sh
      ./build.sh -ds
      

      NOTICE: the README.md file will automatically be updated to reflect the version of the tool given in the build.sh script. For this to work, leave “toolVersion” in the README and do not remove this or alter.

    2. test running some commands within the container that should be available in your local docker container repository.

      For example, to open an interactive shell in a container (with the home folder /root bound to /root on host), you may run:

      sudo docker run -it -v /root:/root --entrypoint /bin/bash NEWAPP_VERSION:TAG
      

      with VERSION being the version of the app, and TAG the version tag of the container (run ‘sudo docker image list’ to find the tag)

    3. if your application requires a Matlab Runtime and you get an error about shared library “libmwlaunchermain.so” not found, check which version of the runtime was installed by the build script

  5. Update changes in local git repository

    git add .github/workflows/NEWAPP.yml recipes/NEWAPP/test.sh recipes/NEWAPP/build.sh recipes/NEWAPP/README.md
    git config user.email "the email that you use for github"
    git config user.name "your name"
    git commit
    

Push the new or updated app to Neurocontainers

Prerequisite

Generate git personal access token (if you don’t have one already)

  1. Browse to https://github.com/
  2. Log into your account
  3. Press on your picture in upper right corner → Setting → Developer Settings → Personal Access Token
  4. Press on “generate personal access token”
  5. Write something in “Notes” (doesn’t matter what, it’s for your own use)
  6. Check “repo”
  7. Check “Workflow”
  8. Press “Generate Token” at the bottom
  9. Copy the token displayed to somewhere safe, as you will have to user it later

Verify that user has write permission to /neurocommand/local

  1. If not, run sudo chmod a+w /neurocommand/local

Step by step guide

  1. Test the container locally, and if successful push repo to trigger the automatic build on GitHub. When asked for your Github password, please provide the personal access token obtained in the previous stage.

    git pull
    git push
    
  2. Go to https://github.com/neurodesk/neurocontainers/actions. Check that the most recent workflow run in the list terminated successfully (green). Otherwise, click on it, click on “build docker”, and the line that caused the error will be highlighted

  3. Find your new package under https://github.com/orgs/NeuroDesk/packages?repo_name=neurocontainers
    Enter the name of the package in the search box, and verify that the full package name shows up in the format toolName_toolVersion

  4. Obtain buildDate by clicking on the full package name that came up in the search. The build date will be the newest date shown under Recent tagged image versions

  5. If updating an app, use toolName delete the locally installed container of the old app version or old app build:

    rm -R /neurocommand/local/containers/toolName_*/
    rm -R /neurocommand/local/containers/modules/toolName/
    
  6. Use toolName, toolVersion and buildDate from the previous two steps to manually download the package by typing the following in a terminal open in Neurodesktop

    bash /neurocommand/local/fetch_and_run.sh toolName toolVersion buildDate
    (when you see the "Singularity>" prompt, type exit and ENTER)
    ml toolName/toolVersion
    

    For example: If the full package name that comes up in the step 11 is itksnap_3.8.0, and the newest date under Recent tagged image versions is 20210322

    The command to use in a terminal open in Neurodesktop is:

    bash /neurocommand/local/fetch_and_run.sh itksnap 3.8.0 20210322
    (when you see the "Singularity>" prompt, type exit and ENTER)
     ml toolName/toolVersion
    
  7. Test the new container. Run some commands, to see all is good
    If the container doesn’t work yet, it’s sometimes useful to try and troubleshoot it and install missing libraries. This can be achieved by running it in a writable mode with fakeroot enabled:

    SINGULARITY_BINDPATH=''; singularity shell --writable --fakeroot /neurodesktop-storage/containers/toolName_toolVersion_buildDate/toolName_toolVersion_buildDate.simg
    
  8. Fork https://github.com/NeuroDesk/neurocommand/ to your Github account

  9. Edit an entry for your package in your fork of neurocommand/blob/main/neurodesk/apps.json based on one of the other entries (generating one menu item for opening a terminal inside the containers, and one menu item for the GUI, if relevant). Notice that in the json file, the version field should contain the buildDate rather than the toolVersion !!! toolVersion should be included instead in the text of the menu entry itself, e.g., “fsl 6.0.3”. Also notice that whereas categories appear in the Neurodesktop menu in start case (first letter of each word capitalized), in the json files they are sentence case (all letters lower case).

  10. Include an icon file in your fork of neurocommand/neurodesk/icons

  11. Send a pull request from your fork of neurocommand to https://github.com/NeuroDesk/neurocommand/

  12. When the pull request is merged by Neurodesk admins, it will trigger an action to build the singularity container, distribute it in all object storage locations and on CVMFS, and it will update the menus in the desktop image on the next daily build.

  13. Wait at least 24 hours

  14. Download and run the daily build of Neurodesktop to check that your app can be launched from the start menu and works properly:

    sudo docker pull vnmd/neurodesktop:latest && sudo docker run --shm-size=1gb -it --privileged --user=root --name neurodesktop -v ~/neurodesktop-storage:/neurodesktop-storage -e HOST_UID="$(id -u)"  -e HOST_GID="$(id -g)" -p 8888:8888 -e NEURODESKTOP_VERSION=latest vnmd/neurodesktop:latest
    
  15. Open an issue in https://github.com/NeuroDesk/neurocontainers/issues notifying that your app appears in the start menu and tested. The app will be included in the next release of Neurodesktop, and will be mentioned in the public announcement that accompanies the release. If the app is not in the start menu or not working as expected based on your earlier testing, open an issue as well, and report it.

  16. If somebody wants to use the application before the next release of Neurodesktop is out, you can instruct them to use the command in step 14 above instead of the default commands given in the user install instructions.

  17. Consider contributing a tutorial about the new tool in this instruction

3 - Menu entries

Menu entries in neurodesktop

As we want to propose several versions of the tools, each piece of software should have its own submenu under VNM Neuroimaging. To do so, you first have to add a submenu to menus/vnm-applications.menu by adding:

<!-- [[Tool Name]] submenu -->
<Menu>
    <Name>[[Tool Name]]</Name>
    <Directory>vnm-[[tool-name]].directory</Directory>
    <Include>
        <And>
            <Category>[[Tool-Name]]</Category>
        </And>
    </Include>
</Menu> <!-- End [[Tool Name]] -->

The following table shows the formatting rules to follow:

PlaceholderRuleExample
[[Tool name]]Capitalized, spacesITK snap
[[tool-name]]Lower case, no spaces (use - instead)itk-snap or itksnap
[[Tool-name]]Capitalized, no spaces (use - instead)ITK-snap

Next, we have to create the submenu itself as we referenced it by vnm-[[tool-name]].directory. To do so, create the file menus/submenus/vnm-[[tool-name]].directory and add the following information inside:

[Desktop Entry]
Name=[[Tool Name]]
Comment=[[Tool Name]]
Icon=/home/neuro/.config/lxpanel/LXDE/icons/[[icon-name]].png
Type=Directory

If a specific icon is available in the menus/icons directory, replace [[icon-name]] by its name. Otherwise, use vnm.

Create the application

Finally, we have to create the actual application by creating the file menus/applications/vnm-[[tool-name]]-[[0.0.0]].desktop. The name of this file must contain the version of the tool (once again to allow multiple versions to live inside the same directory). Add the following description to this file:

[Desktop Entry]
Name=[[Tool Name]] [[0.0.0]] [[(Install only)]]
GenericName=[[Tool Name]] [[0.0.0]]
Comment=The description of what clicking on this application does. # This will be the tooltip of the application.
Exec=The command used to run the application.
Icon=/home/neuro/.config/lxpanel/LXDE/icons/[[icon-name]].png
Type=Application
Categories=[[Tool-name]]
Terminal=true # or false

The important part here is the value of Exec. If the tool is in the form of a singularity image, you should run the following command:

bash /usr/share/fetch_and_run.sh [[tool-name]] [[0.0.0]] [[YYYYMMDD]] [[cmd]] [[args]]

What fetch_and_run.sh does is check if the image is already installed as a module. If not, it checks whether it can be installed or not (return 1 if not possible). After that, it installs the image as a module. If [[cmd]] is specified, once the image is installed, it runs the command by giving the arguments from [[args]]. Here are two examples for FreeSurfer and FreeView. This first one only installs the image as a module:

bash /usr/share/fetch_and_run.sh freesurfer 6.0.1 20200506

And this does the same but runs FreeView afterward:

bash /usr/share/fetch_and_run.sh freesurfer 6.0.1 20200506 freeview

The resulting .desktop file corresponding to FreeView contains:

[Desktop Entry]
Name=FreeView 6.0.1
GenericName=FreeView 6.0.1
Comment=Start FreeView 6.0.1
Exec=bash /usr/share/fetch_and_run.sh freesurfer 6.0.1 20200506 freeview
Icon=/home/neuro/.config/lxpanel/LXDE/icons/run.png
Type=Application
Categories=FreeSurfer
Terminal=true