Gateways Tutorial

Objective: learn the basics of the Apache Airavata Django Portal and how to make both simple and complex customizations to the user interface.

Prerequisites

Tutorial attendees should have:

• a laptop on which to write Python code
• Git client
• Docker desktop for Windows and macOS or Docker Engine for Linux

!!! note “Special note for Windows Home users”

If you have Windows Home installed, you'll need to either use a **remote
Docker host** (recommended) or take some extra steps to setup WSL2
(Windows Subsystem for Linux 2). The following are some special
instructions to help you with these options:

1. (**Recommended**) For the in person session of the tutorial, you will
have the option of using a **remote Docker host** provided to you. See
[Appendix: Setting up Windows Home for a remote Docker
host](#appendix-setting-up-windows-for-a-remote-docker-host) for more
details.
2. (**Advanced**) [Install Docker Desktop on Windows Home with WSL2
enabled](https://docs.docker.com/docker-for-windows/install-windows-home/).
See the link for more information.

Outline

• Introduction
• Presentation: Overview of Airavata and Django Portal
  • History of the Airavata UI and how did we get here
• Hands on: run a basic computational experiment in the Django portal
• Tutorial exercise: customize the input user interface for an application
• Tutorial exercise: Create a custom output viewer for an output file
• Tutorial exercise: Create a custom Django app
  • use the AiravataAPI JavaScript library for utilizing the backend Airavata API
  • develop a simple custom user interface for setting up and visualizing computational experiments

Hands on: run a Gaussian computational experiment in the Django portal

Log into testdrive.airavata.org

First, you‘ll need a user account. For the in person tutorial we’ll have a set of pre-created usernames and passwords to use. If you are unable to attend the in person tutorial or would otherwise like to create your own account, go to the Create Account page and select Sign in with existing institution credentials. This will take you to the CILogon institution selection page. If you don't find your institution listed here, go back to the Create Account page and fill out the form to create an account with a username, password, etc.

Once you have an account, log into the Airavata Testdrive portal.

After you‘ve logged in, an administrator can grant you access to run the Gaussian application. During the tutorial we’ll grant you access right away and let you know. If you're at the in person tutorial and using a pre-created username and password, you should already have all of the necessary authorizations.

When you log in for the first time you will see a list of applications that are available in this science gateway. Applications that you are not able to run are greyed out but the other ones you can run. Once you are granted access, refresh the page and you should now see that you the Gaussian16 application is not greyed out.

Submit a test job

From the dashboard, click on the Gaussian16 application. The page title is Create a New Experiment.

Here you can change the Experiment Name, add a description or select a different project if you have multiple projects.

We'll focus on the Application Inputs for this hands-on. The Gaussian application requires one input, an Input-File. The following is a preconfigured Gaussian input file. Download this to your local computer and then click the Browse button to upload the file:

• npentane12diol.inp

You can click on View File to take a quick look at the file.

Now we'll select what account to charge and where to run this job. The Allocation field should already have Default Gateway Profile selected. Under Compute Resource make sure you select comet.sdsc.edu.

Then click Save and Launch.

You should then be taken to the Experiment Summary page which will update as the job progresses. When the job finishes you'll be able to download the .log file which is the primary output file of the gaussian application.

We'll come back to this experiment later in the tutorial.

Tutorial exercise: customize the input user interface for an application

For this exercise we‘ll define an application based on the Computational Systems Biology Group’s eFindSite drug-binding site detection software. We'll use this application to demonstrate how to customize the user interface used for application inputs.

Basic application configuration

1. In the portal, click on the dropdown menu at the top right (currently Workspace is likely selected) and select Settings.

2. You should see the Application Catalog. Click on the New Application button.
3. For Application Name provide eFindSite-<your username>. Appending your username will allow you to distinguish your version of eFindSite from other users.
4. Click Save.
5. Click on the Interface tab.
6. This application has 4 command line inputs. We'll add them now. To add the first one, click on Add application input and provide the following information:
   • Name: Target ID
   • Type: STRING (which is the default)
   • Application Argument: -i
   • User Friendly Description: 3-10 alphanumerical characters.
   • Required: True
   • Required on Command Line: True

7. Add the next three application inputs in the same way, using the values in the table below:

Name	Type	Application Argument	Required	Required on Command Line
Target Structure	URI	-s	True	True
Screening libraries	STRING	-l	False	True
Visualization scripts	STRING	-v	False	True

(In Airavata, files are represented as URIs. When an application input has type URI it means that a file is needed for that input. From a UI point of view, this essentially means that the user will be able to upload a file for inputs of type URI.)

Normally we would also define the output files for this application, but for this exercise we are only interested in exploring the options available in customizing the application inputs and we won‘t actually run this application. We need to register a deployment to be able to invoke this application. An application deployment includes the details of how and where an application is installed on a compute resource. Since we won’t actually run this application, we'll just create a dummy deployment so that we can invoke it from the Workspace Dashboard.

8. Click Save at the bottom of the screen.
9. Click on the Deployments tab.
10. Click on the New Deployment button. Select the mike.hpc.lsu.edu compute resource in the drop down list and click OK.
11. For the Application Executable Path, provide the value /usr/bin/true. This is the only required field.
12. Click Save at the bottom of the screen.
13. Use the top level menu to go back to the Workspace. You should see your eFindSite application listed there.
14. Click on your eFindSite application.

If you see a form with the inputs that we registered for the application (Target ID, etc.) then you have successfully registered the application interface.

Improving the application input user interface

There are a few things to point out now:

• the Screening libraries and Visualization scripts only accept specific values. For example, one of the allowed values for Screening libraries is screen_drugbank
• the Target ID input takes a string value, but only certain characters (alphanumeric) are allowed and the string value has a minimum and maximum allowed length.

We can make this user interface more user friendly by providing more guidance in the application inputs' user interface. For the Screening libraries and Visualization scripts we‘ll provide a list of labeled checkboxes for the user to select. For the Target ID we’ll provide validation feedback that verifies that the given value has an allowed length and only allowed characters.

1. Go back to Settings and in the Application Catalog click on your eFindSite application.
2. Click on the Interface tab.
3. For Target ID, in the Advanced Input Field Modification Metadata box, add the following JSON configuration:

{
    "editor": {
        "validations": [
            {
                "type": "min-length",
                "value": 3
            },
            {
                "type": "max-length",
                "value": 10
            },
            {
                "message": "Target ID may only contain alphanumeric characters and underscores.",
                "type": "regex",
                "value": "^[a-zA-Z0-9_]+$"
            }
        ],
        "ui-component-id": "string-input-editor"
    }
}

It should look something like this:

This JSON configuration customizes the input editor in two ways:

• it adds 3 validations: min-length, max-length and a regex
• it sets the UI component of the input editor to be the string-input-editor (which is also the default)

4. Likewise for Screening Libraries, set the Advanced Input Field Modification Metadata to:

{
    "editor": {
        "ui-component-id": "checkbox-input-editor",
        "config": {
            "options": [
                {
                    "text": "BindingDB",
                    "value": "screen_bindingdb"
                },
                {
                    "text": "ChEMBL (non-redundant, TC<0.8)",
                    "value": "screen_chembl_nr"
                },
                {
                    "text": "DrugBank",
                    "value": "screen_drugbank"
                },
                {
                    "text": "KEGG Compound",
                    "value": "screen_keggcomp"
                },
                {
                    "text": "KEGG Drug",
                    "value": "screen_keggdrug"
                },
                {
                    "text": "NCI-Open",
                    "value": "screen_nciopen"
                },
                {
                    "text": "RCSB PDB",
                    "value": "screen_rcsbpdb"
                },
                {
                    "text": "ZINC12 (non-redundant, TC<0.7)",
                    "value": "screen_zinc12_nr"
                }
            ]
        }
    }
}

This JSON configuration specifies a different UI component to use as the input editor, the checkbox-input-editor. It also provides a list of text/value pairs for the checkboxes; the values are what will be provided to the application as command line arguments.

5. Similarly for the Visualization scripts, provide the following JSON configuration:

{
    "editor": {
        "ui-component-id": "checkbox-input-editor",
        "config": {
            "options": [
                {
                    "text": "VMD",
                    "value": "visual_vmd"
                },
                {
                    "text": "PyMOL",
                    "value": "visual_pymol"
                },
                {
                    "text": "ChimeraX",
                    "value": "visual_chimerax"
                }
            ]
        }
    }
}

6. Click Save at the bottom of the page.
7. Now, go back to the Workspace and on the Dashboard click on your eFindSite application. The application inputs form should now reflect your changes.
8. Try typing an invalid character (for example, #) in Target ID. Also try typing in more than 10 alphanumeric characters. When an invalid value is provided the validation feedback informs the user of the problem so that the user can correct it.

Additional application input customizations

Other UI components are available:

• textarea
• radio buttons
• dropdown

We're working to provide a way for custom input editors to be added by the community, especially domain specific input editors. For example, a ball and stick molecule editor or a map view for selecting a bounding box of a region of interest.

Also you can define dependencies between application inputs and show or hide inputs based on the values of other inputs.

Tutorial exercise: Create a custom output viewer for an output file

By default, the Django portal provides a very simple view for output files that allows users to download the file to their local machine. However, it is possible to provide additional custom views for output files. Examples include:

• image (visualization)
• link (perhaps to another web application that can visualize the file)
• chart
• parameterized notebook

To be able to create a custom output viewer we‘ll need to write some Python code. First, we’ll get a local version of the Django portal running which we'll use as a development environment.

Setup local Django portal development environment

To run the Django portal locally we‘ll start it as a Docker container. Another option, which we won’t cover in this tutorial, is to check out the code and run the portal locally as a Python process (see the airavata-django-portal README if you are interested).

1. Make sure you have Docker installed.
2. Run the following to create a Docker container called gateways19-tutorial.

cd $HOME
git clone https://github.com/machristie/gateways19-tutorial.git
cd gateways19-tutorial
docker run -d --name gateways19-tutorial -p 8000:8000 -v "$PWD:/extensions" -v "$PWD/settings_local.py:/code/django_airavata/settings_local.py" machristie/airavata-django-portal

!!! note “For remote Docker host users”

If you are using a remote Docker host (for example, you have Windows Home
and can't install Docker Desktop), make sure you run the above commands
on the remote Docker host. That means you need to
`ssh USERNAME@IP_ADDRESS` to the remote host first. See [Appendix on
running on a remote Docker
host](#appendix-setting-up-windows-for-a-remote-docker-host) for more
information on setting up the SSH connection. You can run the remaining
`docker` commands on your own computer, but this `docker run` command
must be run on the remote Docker host so that the tutorial files can be
mounted into it.

!!! note

You can also build the Docker image from scratch, which you might want to
do if the Docker image is out-dated. To do that run the following:

    cd /tmp/
    git clone https://github.com/apache/airavata-django-portal.git
    cd airavata-django-portal
    docker build -t airavata-django-portal .

Now you can `airavata-django-portal` instead of
`machristie/airavata-django-portal` in the `docker run` command above.

3. Run the following to load the default set of CMS pages:

docker exec gateways19-tutorial python manage.py load_cms_data new_default_theme

---

Go to http://localhost:8000, click on Login in, enter your username and password. On the dashboard you should see the your experiments listed on the right hand side.

Setup the custom output viewer package

1. We‘ve defined a custom output view provider, called GaussianEigenvaluesViewProvider, in output_views.py. Open $HOME/gateways19-tutorial/gateways19_tutorial/output_views.py in your editor and we’ll look at how it is implemented. First we add some imports

import io
import os

import numpy as np
from matplotlib.figure import Figure

from cclib.parser import ccopen

BASE_DIR = os.path.dirname(os.path.abspath(__file__))

2. Next we define the GaussianEigenvaluesViewProvider class, set some metadata attributes on the class. We set it's display_type to image and give it a name:

class GaussianEigenvaluesViewProvider:
    display_type = 'image'
    name = "Gaussian Eigenvalues"

3. Now we implement the generate_data function. This function should return a dictionary with values that are expected for this display_type. For a display type of image, the required return values are image which should be a bytes array or file-like object with the image bytes and mime-type which should be the image‘s mime type. Here’s the generate_data function:

    def generate_data(self, request, experiment_output, experiment, output_file=None):
        # Parse output_file
        output_text = io.TextIOWrapper(output_file)
        gaussian = ccopen(output_text)
        data = gaussian.parse()
        data.listify()
        homo_eigenvalues = None
        lumo_eigenvalues = None
        if hasattr(data, 'homos') and hasattr(data, 'moenergies'):
            homos = data.homos[0] + 1
            moenergies = data.moenergies[0]
            if homos > 9 and len(moenergies) >= homos:
                homo_eigenvalues = [data.moenergies[0][homos - 1 - i] for i in range(1, 10)]
            if homos + 9 <= len(moenergies):
                lumo_eigenvalues = [data.moenergies[0][homos + i] for i in range(1, 10)]

        # Create plot
        fig = Figure()
        if homo_eigenvalues and lumo_eigenvalues:
            fig.suptitle("Eigenvalues")
            ax = fig.subplots(2, 1)
            ax[0].plot(range(1, 10), homo_eigenvalues, label='Homo')
            ax[0].set_ylabel('eV')
            ax[0].legend()
            ax[1].plot(range(1, 10), lumo_eigenvalues, label='Lumo')
            ax[1].set_ylabel('eV')
            ax[1].legend()
        else:
            ax = fig.subplots()
            ax.text(0.5, 0.5, "No applicable data", horizontalalignment='center',
                verticalalignment='center', transform=ax.transAxes)

        # Export plot as image buffer
        buffer = io.BytesIO()
        fig.savefig(buffer, format='png')
        image_bytes = buffer.getvalue()
        buffer.close()

        # return dictionary with image data
        return {
            'image': image_bytes,
            'mime-type': 'image/png'
        }

This plots the eigenvalues of molecular orbital energies calculated by Gaussian. cclib is a Python computational chemistry library which is used to read the molecular orbital energies. Then matplotlib is used to create two plots of these values. Finally, the plots are exported as a PNG image that is returns as a buffer of bytes.

4. Altogether, the output_views.py file should have the following contents:

import io
import os

import numpy as np
from matplotlib.figure import Figure

from cclib.parser import ccopen

BASE_DIR = os.path.dirname(os.path.abspath(__file__))

class GaussianEigenvaluesViewProvider:
    display_type = 'image'
    name = "Gaussian Eigenvalues"

    def generate_data(self, request, experiment_output, experiment, output_file=None):

        # Parse output_file
        output_text = io.TextIOWrapper(output_file)
        gaussian = ccopen(output_text)
        data = gaussian.parse()
        data.listify()
        homo_eigenvalues = None
        lumo_eigenvalues = None
        if hasattr(data, 'homos') and hasattr(data, 'moenergies'):
            homos = data.homos[0] + 1
            moenergies = data.moenergies[0]
            if homos > 9 and len(moenergies) >= homos:
                homo_eigenvalues = [data.moenergies[0][homos - 1 - i] for i in range(1, 10)]
            if homos + 9 <= len(moenergies):
                lumo_eigenvalues = [data.moenergies[0][homos + i] for i in range(1, 10)]

        # Create plot
        fig = Figure()
        if homo_eigenvalues and lumo_eigenvalues:
            fig.suptitle("Eigenvalues")
            ax = fig.subplots(2, 1)
            ax[0].plot(range(1, 10), homo_eigenvalues, label='Homo')
            ax[0].set_ylabel('eV')
            ax[0].legend()
            ax[1].plot(range(1, 10), lumo_eigenvalues, label='Lumo')
            ax[1].set_ylabel('eV')
            ax[1].legend()
        else:
            ax = fig.subplots()
            ax.text(0.5, 0.5, "No applicable data", horizontalalignment='center',
                verticalalignment='center', transform=ax.transAxes)

        # Export plot as image buffer
        buffer = io.BytesIO()
        fig.savefig(buffer, format='png')
        image_bytes = buffer.getvalue()
        buffer.close()

        # return dictionary with image data
        return {
            'image': image_bytes,
            'mime-type': 'image/png'
        }

5. Now we need to register our output view provider with the package metadata so that the Django Portal will be able to discover it. We add the following lines to the entry_points parameter in the $HOME/gateways19-tutorial/setup.py file:

setuptools.setup(
# ...
    entry_points="""
[airavata.output_view_providers]
gaussian-eigenvalues-plot = gateways19_tutorial.output_views:GaussianEigenvaluesViewProvider
""",
)

gaussian-eigenvalues-plot is the output view provider id. gateways19_tutorial.output_views is the module in which the GaussianEigenvaluesViewProvider output view provider class is found.

6. Now we need to install the gateways19-tutorial package into the Django portal's virtual environment.

docker exec -w /extensions gateways19-tutorial pip install -r requirements.txt
docker exec -w /extensions gateways19-tutorial python setup.py develop
docker exec gateways19-tutorial touch /code/django_airavata/wsgi.py

These commands:

1. install our package's dependencies,
2. install the package into the container's Python environment, and
3. touches the wsgi.py to trigger a reload of the Django portal dev server.

Use the GaussianEigenvaluesViewProvider with the Gaussian log output file

Back in the Django Portal, we'll make sure the application interface for Gaussian is configured to add the GaussianEigenvaluesViewProvider as an additional output view of the file.

1. Log into your local Django Portal instance at http://localhost:8000.
2. In the menu at the top, select Settings.
3. Click on the Gaussian16 application.
4. Click on the Interface tab.
5. Scroll down to the Output Field: Gaussian-Application-Output.
6. Verify that the following is in the Metadata section:

{
    "output-view-providers": ["gaussian-eigenvalues-plot"]
}

It should look something like this:

7. Go back to the Workspace using the menu at the top.
8. Select your Gaussian16 experiment from the right sidebar.
9. For the .log output file there should be a dropdown menu allowing you to select an alternate view. Select Gaussian Eigenvalues. Now you should see the image generated by the custom output view provider.

(Optional) Interactive parameter

In additional to producing static visualizations, output view providers can declare interactive parameters that can be manipulated by the user. We can add a simple boolean interactive parameter to toggle the display of the matplotlib grid as an example.

1. Open $HOME/gateways19-tutorial/gateways19_tutorial/output_views.py. Change the generate_data function so that it has an additional show_grid parameter with a default value of False:

    def generate_data(self, request, experiment_output, experiment, output_file=None, show_grid=False):

2. Add the following .grid() lines to the matplotlib code:

...
            fig.suptitle("Eigenvalues")
            ax = fig.subplots(2, 1)
            ax[0].plot(range(1, 10), homo_eigenvalues, label='Homo')
            ax[0].set_ylabel('eV')
            ax[0].legend()
            ax[0].grid(show_grid)
            ax[1].plot(range(1, 10), lumo_eigenvalues, label='Lumo')
            ax[1].set_ylabel('eV')
            ax[1].legend()
            ax[1].grid(show_grid)
...

3. Change the resulting dictionary to have the special interactive property and declare the show_grid parameter:

...
        # return dictionary with image data
        return {
            'image': image_bytes,
            'mime-type': 'image/png'
            'interactive': [
                {'name': 'show_grid', 'value': show_grid}
            ]
        }

This will provider the user with a checkbox for manipulating the show_grid parameter. Every time the user changes it, the GaussianEigenvaluesViewProvider will be again invoked. It should look something like the following:

There are several more interactive parameter types and additional options. You can learn more about them in the custom output view provider documentation.

Tutorial exercise: Create a custom Django app

In this tutorial exercise we'll create a fully custom user interface that lives within the Django Portal.

What we're going to build is a very simple user interface that will:

• allow a user to pick a greeting in one of several languages
• submit a simple echo job to a batch scheduler to echo that greeting
• display the echoed greeting by displaying the STDOUT file produced by the job

This is an intentionally simple example to demonstrate the general principle of using custom REST APIs and UI to setup, execute and post-process/visualize the output of a computational experiment.

We've already registered the Echo application with the portal, meaning we registered its interface and on which compute resource it is deployed.

A Django application or app is a Python package that may include Django views, url mappings, models, etc. It‘s a way of creating a kind of plug-in that integrates with a Django server. We’ll create this custom user interface by developing a Django app that uses the Django framework as well as the Airavata Django Portal REST APIs and JS library.

Setting up the Django app

To start, we'll just create a simple “Hello World” page for the Django app and get it properly registered with the local Django Portal instance.

1. In the gateways19-tutorial directory, open $HOME/gateways19-tutorial/gateways19_tutorial/templates/gateways19_tutorial/hello.html. Some of the HTML view is commented out. The following is the uncommented content:

{% extends 'base.html' %}

{% block content %}
<div class="main-content-wrapper">
    <main class="main-content">
        <div class="container-fluid">
            <h1>Hello World</h1>
        </div>
    </main>
</div>
{% endblock content %}

2. Open the file $HOME/gateways19-tutorial/gateways19_tutorial/apps.py:

from django.apps import AppConfig


class Gateways19TutorialAppConfig(AppConfig):
    name = 'gateways19_tutorial'
    label = name
    verbose_name = "Gateways Tutorial"
    fa_icon_class = "fa-comment"

This the main metadata for this custom Django app. Besides the normal metadata that the Django framework expects, this also defines a display name (verbose_name) and an icon (fa_icon_class) to use for this custom app.

3. Open the file $HOME/gateways19-tutorial/gateways19_tutorial/views.py:

from django.shortcuts import render
from django.contrib.auth.decorators import login_required


@login_required
def hello_world(request):
    return render(request, "gateways19_tutorial/hello.html")

This view will simply display the template created in the previous step.

4. Open the file $HOME/gateways19-tutorial/gateways19_tutorial/urls.py:

from django.conf.urls import url, include

from . import views

app_name = 'gateways19_tutorial'
urlpatterns = [
    url(r'^hello/', views.hello_world, name="home"),
]

This maps the /hello/ URL to the hello_world view.

5. We've created the necessary code for our Django app to display the hello world page, but now we need to add some metadata to this Python package so that the Django Portal knows about this Django app. In $HOME/gateways19-tutorial/setup.py, we add the following [airavata.djangoapp] metadata to the entry_points section:

setuptools.setup(
# ...
    entry_points="""
[airavata.output_view_providers]
gaussian-eigenvalues-plot = gateways19_tutorial.output_views:GaussianEigenvaluesViewProvider
[airavata.djangoapp]
gateways19_tutorial = gateways19_tutorial.apps:Gateways19TutorialAppConfig
""",
)

---

Now you should be able to log into the portal locally and see Gateways Tutorial in the drop down menu in the header (click on Workspace then you should see it in that menu).

Adding a list of “Hello” greetings

Now we'll create a REST endpoint in our custom Django app that will return greetings in several languages.

1. In the $HOME/gateways19-tutorial/gateways19_tutorial/views.py file, we add the following import:

from django.http import JsonResponse

2. Also we add the following view:

@login_required
def languages(request):
    return JsonResponse({'languages': [{
        'lang': 'French',
        'greeting': 'bonjour',
    }, {
        'lang': 'German',
        'greeting': 'guten tag'
    }, {
        'lang': 'Hindi',
        'greeting': 'namaste'
    }, {
        'lang': 'Japanese',
        'greeting': 'konnichiwa'
    }, {
        'lang': 'Swahili',
        'greeting': 'jambo'
    }, {
        'lang': 'Turkish',
        'greeting': 'merhaba'
    }]})

3. In $HOME/gateways19-tutorial/gateways19_tutorial/urls.py we add a url mapping for the languages view:

urlpatterns = [
    url(r'^hello/', views.hello_world, name="home"),
    url(r'^languages/', views.languages, name="languages"),
]

4. In $HOME/gateways19-tutorial/gateways19_tutorial/templates/gateways19_tutorial/hello.html, uncomment the comment that starts <!-- Adding a list of "Hello" greetings on line 11 and ends on line 21. That is, just delete lines 11 and 21. This adds a <select> element to the template which will be used to display the greeting options:

...
<h1>Hello World</h1>

<div class="card">
    <div class="card-header">Run "echo" for different languages</div>
    <div class="card-body">
        <select id="greeting-select"></select>
        <button id="run-button" class="btn btn-primary">Run</button>
    </div>
</div>
...

5. We also add the {% load static %} directive and then a scripts block to the end of hello.html. This will load the AiravataAPI JavaScript library which has utilities for interacting with the Django portal‘s REST API (which can also be used for custom developed REST endpoints) and model classes for Airavata’s data models. The utils.FetchUtils is used to load the languages REST endpoint.

{% extends 'base.html' %}

{% load static %}

...
{% endblock content %}

{% block scripts %}
<script src="{% static 'django_airavata_api/dist/airavata-api.js' %}"></script>
<script>
    const { models, services, session, utils } = AiravataAPI;

    utils.FetchUtils.get("/gateways19_tutorial/languages/").then(data => {
        data.languages.forEach(language => {
            $("#greeting-select").append(
                `<option value="${language.greeting}">
                    ${language.lang} - "${language.greeting}"
                 </option>`
            );
        });
    });
</script>
{% endblock scripts %}

Now when you view the custom app at http://localhost:8000/gateways19_tutorial/hello/ you should see a dropdown of greetings in several languages, like so:

Displaying a list of recent experiments

Now we‘ll use the AiravataAPI library to load the user’s recent experiments.

1. In $HOME/gateways19-tutorial/gateways19_tutorial/templates/gateways19_tutorial/hello.html, uncomment the comment that begins with <!-- Displaying a list of recent experiments on line 21 or so and ends on line 45. This adds table to display recent experiments to the bottom of hello.html:

...
            <div class="card">
                <div class="card-header">
                    Run "echo" for different languages
                </div>
                <div class="card-body">
                    <select id="greeting-select"></select>
                    <button id="run-button" class="btn btn-primary">Run</button>
                </div>
            </div>
            <div class="card">
                <div class="card-header">
                    Experiments
                </div>
                <div class="card-body">
                    <button id="refresh-button" class="btn btn-secondary">Refresh</button>
                    <table class="table">
                        <thead>
                            <tr>
                                <th scope="col">Name</th>
                                <th scope="col">Application</th>
                                <th scope="col">Creation Time</th>
                                <th scope="col">Status</th>
                                <th scope="col">Output</th>
                            </tr>
                        </thead>
                        <tbody id="experiment-list">
                        </tbody>
                    </table>
                </div>
            </div>
        </div>
    </main>
</div>
{% endblock content %}

2. Now we‘ll use the ExperimentSearchService to load the user’s most recent 5 Echo experiments and display them in the table. We add the following to the end of the scripts block in hello.html:

// ...
    const appInterfaceId = "Echo_23d67491-1bef-47bd-a0f5-faf069e09773";

    function loadExperiments() {

        return services.ExperimentSearchService
            .list({limit: 5,
                [models.ExperimentSearchFields.USER_NAME.name]: session.Session.username,
                [models.ExperimentSearchFields.APPLICATION_ID.name]: appInterfaceId,
            })
            .then(data => {
                $('#experiment-list').empty();
                data.results.forEach((exp, index) => {
                    $('#experiment-list').append(
                    `<tr>
                        <td>${exp.name}</td>
                        <td>${exp.executionId}</td>
                        <td>${exp.creationTime}</td>
                        <td>${exp.experimentStatus.name}</td>
                        <td id="output_${index}"></td>
                    </tr>`);
                });
        });
    }

    loadExperiments();
    $("#refresh-button").click(loadExperiments);

</script>

{% endblock scripts %}

The user interface should now look something like:

Submitting an Echo job

Now we'll use AiravataAPI to submit an Echo job.

1. In $HOME/gateways19-tutorial/gateways19_tutorial/templates/gateways19_tutorial/hello.html we add a click handler to the Run button that gets the selected greeting value:

$("#run-button").click((e) => {
    const greeting = $("#greeting-select").val();
});

2. There are a couple key pieces of information that are needed to submit a computational experiment. We can use the REST API to find these. The application we want to use is called Echo and it has id Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8. We can browse the API for this application using: https://testdrive.airavata.org/api/applications/Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8/. First, we need the Application Interface for the application, which defines the inputs and outputs of the application. We can get its id by following the link to applicationInterface: https://testdrive.airavata.org/api/applications/Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8/application_interface/. We'll create an Experiment instance from the Application Interface definition:

const loadAppInterface = services.ApplicationInterfaceService.retrieve({
    lookup: appInterfaceId,
});

3. Second, we need to know where and how the application is deployed. We could let the user then pick where they want to run this application. For this exercise we're going to hard code the resource and the application deployment that will be used for executing the application, but we still need the application deployment information so we can get default values for the application that can be used when submitting the job to that scheduler. The application deployment id we get from https://testdrive.airavata.org/api/applications/Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8/application_deployments/.

const appDeploymentId =
    "example-vc.jetstream-cloud.org_Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8";
const loadQueues = services.ApplicationDeploymentService.getQueues({
    lookup: appDeploymentId,
});

4. We also need to know a few other pieces of information, like the id of the compute resource which https://testdrive.airavata.org/api/applications/Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8/application_deployments/ also provides with the value computeHostId. The queue name we can get from following the link from the deployment to the queues: https://testdrive.airavata.org/api/application-deployments/example-vc.jetstream-cloud.org_Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8/queues/. Here we see that the queueName is cloud. We also need the account to use to submit the job and that is specified via a “Group Resource Profile”. https://testdrive.airavata.org/api/group-resource-profiles/ lists profiles you have access to and the compute resources each profile can use for job submission. We‘ll use the tutorial reservation one. Finally, experiments are organized by projects so we’ll also load the user‘s most recently used project which is stored in the user’s WorkspacePreferences:

const resourceHostId =
    "example-vc.jetstream-cloud.org_794fd026-101a-46af-8868-5d7a86f813a1";
const queueName = "cloud";
const groupResourceProfileId = "fc245311-a7d1-41af-b8ae-a4142989c9a1";
const loadWorkspacePrefs = services.WorkspacePreferencesService.get();

5. Once we have all of this information we can then create an Experiment object then save and launch it. Here's the complete click handler:

$("#run-button").click((e) => {
    const greeting = $("#greeting-select").val();
    const loadAppInterface = services.ApplicationInterfaceService.retrieve({
        lookup: appInterfaceId,
    });
    const appDeploymentId =
        "example-vc.jetstream-cloud.org_Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8";
    const loadQueues = services.ApplicationDeploymentService.getQueues({
        lookup: appDeploymentId,
    });
    const resourceHostId =
        "example-vc.jetstream-cloud.org_794fd026-101a-46af-8868-5d7a86f813a1";
    const queueName = "cloud";
    const groupResourceProfileId = "fc245311-a7d1-41af-b8ae-a4142989c9a1";
    const loadWorkspacePrefs = services.WorkspacePreferencesService.get();
    Promise.all([loadAppInterface, loadWorkspacePrefs, loadQueues])
        .then(([appInterface, workspacePrefs, queues]) => {
            const experiment = appInterface.createExperiment();
            experiment.experimentName = "Echo " + greeting;
            experiment.projectId = workspacePrefs.most_recent_project_id;
            const cloudQueue = queues.find((q) => q.queueName === queueName);
            experiment.userConfigurationData.groupResourceProfileId = groupResourceProfileId;
            experiment.userConfigurationData.computationalResourceScheduling.resourceHostId = resourceHostId;
            experiment.userConfigurationData.computationalResourceScheduling.totalCPUCount =
                cloudQueue.defaultCPUCount;
            experiment.userConfigurationData.computationalResourceScheduling.nodeCount =
                cloudQueue.defaultNodeCount;
            experiment.userConfigurationData.computationalResourceScheduling.wallTimeLimit =
                cloudQueue.defaultWalltime;
            experiment.userConfigurationData.computationalResourceScheduling.queueName = queueName;
            // Copy the selected greeting to the value of the first input
            experiment.experimentInputs[0].value = greeting;

            return services.ExperimentService.create({ data: experiment });
        })
        .then((exp) => {
            return services.ExperimentService.launch({
                lookup: exp.experimentId,
            });
        });
});

Now that we can launch the experiment we can go ahead and give it a try.

You can also try this out in the production deployment at https://testdrive.airavata.org/gateways19_tutorial/hello/.

Displaying the experiment output

Instead of simply reporting the status of the job we would also like to do something with the output. The STDOUT of the Echo job has a format like the following:

bonjour

We'll read the STDOUT file and display that in our experiment listing table.

1. What we need to do is get the identifier for the experiment's STDOUT file. In Airavata, this identifier is called the Data Product ID. Once we have that we can get the DataProduct object which has the files metadata, including a downloadURL. For each exp we can use the FullExperimentService to get these details like so:

if (exp.experimentStatus === models.ExperimentState.COMPLETED) {
    services.FullExperimentService.retrieve({ lookup: exp.experimentId }).then(
        (fullDetails) => {
            const stdoutDataProductId = fullDetails.experiment.experimentOutputs.find(
                (o) => o.name === "Echo-STDOUT"
            ).value;
            const stdoutDataProduct = fullDetails.outputDataProducts.find(
                (dp) => dp.productUri === stdoutDataProductId
            );
            if (stdoutDataProduct && stdoutDataProduct.downloadURL) {
                return fetch(stdoutDataProduct.downloadURL, {
                    credentials: "same-origin",
                }).then((result) => result.text());
            }
        }
    );
}

2. Then we'll simply display the value in the table.

if (exp.experimentStatus === models.ExperimentState.COMPLETED) {
    services.FullExperimentService.retrieve({ lookup: exp.experimentId })
        .then((fullDetails) => {
            const stdoutDataProductId = fullDetails.experiment.experimentOutputs.find(
                (o) => o.name === "Echo-STDOUT"
            ).value;
            const stdoutDataProduct = fullDetails.outputDataProducts.find(
                (dp) => dp.productUri === stdoutDataProductId
            );
            if (stdoutDataProduct && stdoutDataProduct.downloadURL) {
                return fetch(stdoutDataProduct.downloadURL, {
                    credentials: "same-origin",
                }).then((result) => result.text());
            }
        })
        .then((text) => {
            $(`#output_${index}`).text(text);
        });
}

3. To enable this, remove the comment starting with /* Displaying the experiment output on line 88 and ending on line 113. Here's the update to the loadExperiments function:

function loadExperiments() {
    return services.ExperimentSearchService.list({
        limit: 5,
        [models.ExperimentSearchFields.USER_NAME.name]:
            session.Session.username,
        [models.ExperimentSearchFields.APPLICATION_ID.name]: appInterfaceId,
    }).then((data) => {
        $("#experiment-list").empty();
        data.results.forEach((exp, index) => {
            $("#experiment-list").append(
                `<tr>
                            <td>${exp.name}</td>
                            <td>${exp.executionId}</td>
                            <td>${exp.creationTime}</td>
                            <td>${exp.experimentStatus.name}</td>
                            <td id="output_${index}"></td>
                        </tr>`
            );
            // If experiment has finished, load full details, then parse the stdout file
            if (exp.experimentStatus === models.ExperimentState.COMPLETED) {
                services.FullExperimentService.retrieve({
                    lookup: exp.experimentId,
                })
                    .then((fullDetails) => {
                        const stdoutDataProductId = fullDetails.experiment.experimentOutputs.find(
                            (o) => o.name === "Echo-STDOUT"
                        ).value;
                        const stdoutDataProduct = fullDetails.outputDataProducts.find(
                            (dp) => dp.productUri === stdoutDataProductId
                        );
                        if (
                            stdoutDataProduct &&
                            stdoutDataProduct.downloadURL
                        ) {
                            return fetch(stdoutDataProduct.downloadURL, {
                                credentials: "same-origin",
                            }).then((result) => result.text());
                        }
                    })
                    .then((text) => {
                        $(`#output_${index}`).text(text);
                    });
            }
        });
    });
}

Resources

You can browser the final version of the gateways19-tutorial code at https://github.com/machristie/gateways19-tutorial/tree/solution. If you get stuck at some point with the tutorial you can skip to the solution by running the following git command in your gateways19-tutorial repo:

cd $HOME/gateways19-tutorial
git reset --hard origin/solution

Airavata API

The Django portal provides a REST API bridge to the backend Airavata API. So it's helpful to look at what is available in the backend Airavata API. See the Apache Airavata API docs

To see what is in the AiravataAPI JavaScript library, take a look at it's index.js file in the airavata-django-portal repo.

Airavata Gateway Hosting

SciGaP provides free Airavata Gateways hosting services. Log in or create an account at scigap.org to request gateway hosting.

Appendix: Setting up Windows for a remote Docker host

Installing Visual Studio Code

The tutorial code needs to be on the remote Docker host so that it can be mounted into the Django portal container. To modify the files as required by the tutorial you'll need to either SSH into the remote Docker host and edit the files there with a terminal editor, like Vim, or you can use Visual Studio Code to edit the files remotely with the SSH extension.

To use Visual Studio Code, install it from https://code.visualstudio.com/. Once you have it installed, start it. In the menu, go to View > Extensions. In the Extensions search, type ssh and click on Install for the Remote - SSH extension. Once it is installed, click on the green box in the lower left hand corner and select Remote-SSH: Connect to Host..., or, type Ctrl-Shift-P and type Remote-SSH and select Remote-SSH: Connect to Host.... Select linux as the remote platform type.

Once you are connected, go to View > Explorer. Click on the Open Folder button. Select the folder that contains the tutorial code and click OK. You should now see the tutorial files in the Explorer on the left hand side. Now you can edit the tutorial files from your local computer and they will be immediately reflected in the Django portal container.

SSH configuration

To connect via SSH to the remote Docker host, you'll need an SSH key pair. If you are setting up the remote Docker host, you can create the key pair yourself and copy the public portion to the ~/.ssh/authorized_keys file under the user account on the remote Docker host. For the in person tutorial session this will have already been setup and you will be provided with the private key.

1. Create a directory in your home directory (i.e. C:\Users\<username>\) called .ssh.
2. Copy the private key file into the .ssh directory.
3. Create (or if provided, copy) a config file in the .ssh directory. The contents of the file should be:

Host IP_ADDRESS
    IdentityFile ~\.ssh\PRIVATE_KEY

Where IP_ADDRESS should be replaced with the IP address or hostname of the remote Docker host (for example: 149.165.168.201). And PRIVATE_KEY should be replaced with the name of your private key file that you copied in step 2.

Test that SSH is working by opening a Powershell command prompt (in Visual Studio Code open a new terminal with Ctrl+Shift+`) and running the following:

ssh username@IP_ADDRESS

You should see something like the following:

PS C:\Users\testuser> ssh user1@149.165.157.132
The authenticity of host '149.165.157.132 (149.165.157.132)' can't be established.
ECDSA key fingerprint is SHA256:RG86D7KwCZNQtFOAfEc4TZ4V0stn1RyGrj5I+v7SHxU.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '149.165.157.132' (ECDSA) to the list of known hosts.
Last login: Thu Aug 20 14:31:41 2020 from 156-56-142-122.dhcp-bl.indiana.edu
[user1@django-tutorial ~]$

If you get the “authenticity of host ... can't be established”, as shown above, enter yes at the prompt to proceed. You shouldn't be prompted for a password since the private key will be used for authentication. If you are prompted for a password, double check that the config file is correct.

Connecting to Docker

To connect to Docker you'll need the Docker client. Download https://github.com/StefanScherer/docker-cli-builder/releases/download/19.03.12/docker.exe and copy it to C:\Windows (or anywhere else that is on your PATH, or in the current directory).

Next, at a Powershell command prompt set the environment variable DOCKER_HOST to the SSH username and IP address of the remote Docker host.

$env:DOCKER_HOST = "ssh://USERNAME@IP_ADDRESS"

But replace USERNAME with your username on the remote Docker host and the IP_ADDRESS with the IP address or domain name of the remote Docker host. For example:

$env:DOCKER_HOST = "ssh://train01@149.165.170.99"

Now run the following to test that the connection is working:

docker ps

You'll use this command prompt window during the tutorial to run docker commands.

!!! note “Note for legacy Command Prompt users”

If you are using `CMD`, the legacy command prompt program then you need a
slightly different way of specifying the DOCKER_HOST environment
variable. The form is

```
set DOCKER_HOST=ssh://USERNAME@IP_ADDRESS
```

SSH Tunnel

For some of the tutorial instructions you'll be asked to load the Django portal in your browser at http://localhost:8000. However, your Django portal container is running on a remote Docker host, not your local computer. To make it appear that the Django portal is running locally, create an SSH tunnel to forward port 8000 to the remote Docker host. In a separate command prompt (in Visual Studio Code open a new terminal with Ctrl+Shift+`), run the following:

ssh -L 8000:localhost:8000 USERNAME@IP_ADDRESS