Objective: learn the basics of the Apache Airavata Django Portal and how to make both simple and complex customizations to the user interface.
Tutorial attendees should have:
!!! note “Special note for Windows Home users”
If you have Windows Home installed, you'll need to either use a **remote Docker host** (recommended) or take some extra steps to setup WSL2 (Windows Subsystem for Linux 2). The following are some special instructions to help you with these options: 1. (**Recommended**) For the in person session of the tutorial, you will have the option of using a **remote Docker host** provided to you. See [Appendix: Setting up Windows Home for a remote Docker host](#appendix-setting-up-windows-for-a-remote-docker-host) for more details. 2. (**Advanced**) [Install Docker Desktop on Windows Home with WSL2 enabled](https://docs.docker.com/docker-for-windows/install-windows-home/). See the link for more information.
AiravataAPI
JavaScript library for utilizing the backend Airavata APIFirst, you‘ll need a user account. For the in person tutorial we’ll have a set of pre-created usernames and passwords to use. If you are unable to attend the in person tutorial or would otherwise like to create your own account, go to the Create Account page and select Sign in with existing institution credentials. This will take you to the CILogon institution selection page. If you don't find your institution listed here, go back to the Create Account page and fill out the form to create an account with a username, password, etc.
Once you have an account, log into the Airavata Testdrive portal.
After you‘ve logged in, an administrator can grant you access to run the Gaussian application. During the tutorial we’ll grant you access right away and let you know. If you're at the in person tutorial and using a pre-created username and password, you should already have all of the necessary authorizations.
When you log in for the first time you will see a list of applications that are available in this science gateway. Applications that you are not able to run are greyed out but the other ones you can run. Once you are granted access, refresh the page and you should now see that you the Gaussian16 application is not greyed out.
From the dashboard, click on the Gaussian16 application. The page title is Create a New Experiment.
Here you can change the Experiment Name, add a description or select a different project if you have multiple projects.
We'll focus on the Application Inputs for this hands-on. The Gaussian application requires one input, an Input-File. The following is a preconfigured Gaussian input file. Download this to your local computer and then click the Browse button to upload the file:
You can click on View File to take a quick look at the file.
Now we'll select what account to charge and where to run this job. The Allocation field should already have Default Gateway Profile selected. Under Compute Resource make sure you select comet.sdsc.edu.
Then click Save and Launch.
You should then be taken to the Experiment Summary page which will update as the job progresses. When the job finishes you'll be able to download the .log
file which is the primary output file of the gaussian application.
We'll come back to this experiment later in the tutorial.
For this exercise we‘ll define an application based on the Computational Systems Biology Group’s eFindSite drug-binding site detection software. We'll use this application to demonstrate how to customize the user interface used for application inputs.
eFindSite-<your username>
. Appending your username will allow you to distinguish your version of eFindSite from other users.Target ID
-i
3-10 alphanumerical characters.
True
True
Name | Type | Application Argument | Required | Required on Command Line |
---|---|---|---|---|
Target Structure | URI | -s | True | True |
Screening libraries | STRING | -l | False | True |
Visualization scripts | STRING | -v | False | True |
(In Airavata, files are represented as URIs. When an application input has type URI it means that a file is needed for that input. From a UI point of view, this essentially means that the user will be able to upload a file for inputs of type URI.)
Normally we would also define the output files for this application, but for this exercise we are only interested in exploring the options available in customizing the application inputs and we won‘t actually run this application. We need to register a deployment to be able to invoke this application. An application deployment includes the details of how and where an application is installed on a compute resource. Since we won’t actually run this application, we'll just create a dummy deployment so that we can invoke it from the Workspace Dashboard.
/usr/bin/true
. This is the only required field.If you see a form with the inputs that we registered for the application (Target ID, etc.) then you have successfully registered the application interface.
There are a few things to point out now:
screen_drugbank
We can make this user interface more user friendly by providing more guidance in the application inputs' user interface. For the Screening libraries and Visualization scripts we‘ll provide a list of labeled checkboxes for the user to select. For the Target ID we’ll provide validation feedback that verifies that the given value has an allowed length and only allowed characters.
{ "editor": { "validations": [ { "type": "min-length", "value": 3 }, { "type": "max-length", "value": 10 }, { "message": "Target ID may only contain alphanumeric characters and underscores.", "type": "regex", "value": "^[a-zA-Z0-9_]+$" } ], "ui-component-id": "string-input-editor" } }
It should look something like this:
This JSON configuration customizes the input editor in two ways:
string-input-editor
(which is also the default){ "editor": { "ui-component-id": "checkbox-input-editor", "config": { "options": [ { "text": "BindingDB", "value": "screen_bindingdb" }, { "text": "ChEMBL (non-redundant, TC<0.8)", "value": "screen_chembl_nr" }, { "text": "DrugBank", "value": "screen_drugbank" }, { "text": "KEGG Compound", "value": "screen_keggcomp" }, { "text": "KEGG Drug", "value": "screen_keggdrug" }, { "text": "NCI-Open", "value": "screen_nciopen" }, { "text": "RCSB PDB", "value": "screen_rcsbpdb" }, { "text": "ZINC12 (non-redundant, TC<0.7)", "value": "screen_zinc12_nr" } ] } } }
This JSON configuration specifies a different UI component to use as the input editor, the checkbox-input-editor
. It also provides a list of text/value pairs for the checkboxes; the values are what will be provided to the application as command line arguments.
{ "editor": { "ui-component-id": "checkbox-input-editor", "config": { "options": [ { "text": "VMD", "value": "visual_vmd" }, { "text": "PyMOL", "value": "visual_pymol" }, { "text": "ChimeraX", "value": "visual_chimerax" } ] } } }
#
) in Target ID. Also try typing in more than 10 alphanumeric characters. When an invalid value is provided the validation feedback informs the user of the problem so that the user can correct it.Other UI components are available:
We're working to provide a way for custom input editors to be added by the community, especially domain specific input editors. For example, a ball and stick molecule editor or a map view for selecting a bounding box of a region of interest.
Also you can define dependencies between application inputs and show or hide inputs based on the values of other inputs.
By default, the Django portal provides a very simple view for output files that allows users to download the file to their local machine. However, it is possible to provide additional custom views for output files. Examples include:
To be able to create a custom output viewer we‘ll need to write some Python code. First, we’ll get a local version of the Django portal running which we'll use as a development environment.
To run the Django portal locally we‘ll start it as a Docker container. Another option, which we won’t cover in this tutorial, is to check out the code and run the portal locally as a Python process (see the airavata-django-portal README if you are interested).
cd $HOME git clone https://github.com/machristie/gateways19-tutorial.git cd gateways19-tutorial docker run -d --name gateways19-tutorial -p 8000:8000 -v $PWD:/extensions -v $PWD/settings_local.py:/code/django_airavata/settings_local.py machristie/airavata-django-portal
!!! note “For remote Docker host users”
If you are using a remote Docker host (for example, you have Windows Home and can't install Docker Desktop), make sure you run the above commands on the remote Docker host. That means you need to `ssh USERNAME@IP_ADDRESS` to the remote host first. See [Appendix on running on a remote Docker host](#appendix-setting-up-windows-for-a-remote-docker-host) for more information on setting up the SSH connection. You can run the remaining `docker` commands on your own computer, but this `docker run` command must be run on the remote Docker host so that the tutorial files can be mounted into it.
!!! note
You can also build the Docker image from scratch, which you might want to do if the Docker image is out-dated. To do that run the following: cd /tmp/ git clone https://github.com/apache/airavata-django-portal.git cd airavata-django-portal docker build -t airavata-django-portal . Now you can `airavata-django-portal` instead of `machristie/airavata-django-portal` in the `docker run` command above.
docker exec gateways19-tutorial python manage.py load_cms_data new_default_theme
Go to http://localhost:8000, click on Login in, enter your username and password. On the dashboard you should see the your experiments listed on the right hand side.
output_views.py
. Open $HOME/gateways19-tutorial/gateways19_tutorial/output_views.py
in your editor and we’ll look at how it is implemented. First we add some importsimport io import os import numpy as np from matplotlib.figure import Figure from cclib.parser import ccopen BASE_DIR = os.path.dirname(os.path.abspath(__file__))
display_type
to image and give it a name:class GaussianEigenvaluesViewProvider: display_type = 'image' name = "Gaussian Eigenvalues"
generate_data
function. This function should return a dictionary with values that are expected for this display_type
. For a display type of image, the required return values are image which should be a bytes array or file-like object with the image bytes and mime-type which should be the image‘s mime type. Here’s the generate_data
function:def generate_data(self, request, experiment_output, experiment, output_file=None): # Parse output_file output_text = io.TextIOWrapper(output_file) gaussian = ccopen(output_text) data = gaussian.parse() data.listify() homo_eigenvalues = None lumo_eigenvalues = None if hasattr(data, 'homos') and hasattr(data, 'moenergies'): homos = data.homos[0] + 1 moenergies = data.moenergies[0] if homos > 9 and len(moenergies) >= homos: homo_eigenvalues = [data.moenergies[0][homos - 1 - i] for i in range(1, 10)] if homos + 9 <= len(moenergies): lumo_eigenvalues = [data.moenergies[0][homos + i] for i in range(1, 10)] # Create plot fig = Figure() if homo_eigenvalues and lumo_eigenvalues: fig.suptitle("Eigenvalues") ax = fig.subplots(2, 1) ax[0].plot(range(1, 10), homo_eigenvalues, label='Homo') ax[0].set_ylabel('eV') ax[0].legend() ax[1].plot(range(1, 10), lumo_eigenvalues, label='Lumo') ax[1].set_ylabel('eV') ax[1].legend() else: ax = fig.subplots() ax.text(0.5, 0.5, "No applicable data", horizontalalignment='center', verticalalignment='center', transform=ax.transAxes) # Export plot as image buffer buffer = io.BytesIO() fig.savefig(buffer, format='png') image_bytes = buffer.getvalue() buffer.close() # return dictionary with image data return { 'image': image_bytes, 'mime-type': 'image/png' }
This plots the eigenvalues of molecular orbital energies calculated by Gaussian. cclib
is a Python computational chemistry library which is used to read the molecular orbital energies. Then matplotlib
is used to create two plots of these values. Finally, the plots are exported as a PNG image that is returns as a buffer of bytes.
import io import os import numpy as np from matplotlib.figure import Figure from cclib.parser import ccopen BASE_DIR = os.path.dirname(os.path.abspath(__file__)) class GaussianEigenvaluesViewProvider: display_type = 'image' name = "Gaussian Eigenvalues" def generate_data(self, request, experiment_output, experiment, output_file=None): # Parse output_file output_text = io.TextIOWrapper(output_file) gaussian = ccopen(output_text) data = gaussian.parse() data.listify() homo_eigenvalues = None lumo_eigenvalues = None if hasattr(data, 'homos') and hasattr(data, 'moenergies'): homos = data.homos[0] + 1 moenergies = data.moenergies[0] if homos > 9 and len(moenergies) >= homos: homo_eigenvalues = [data.moenergies[0][homos - 1 - i] for i in range(1, 10)] if homos + 9 <= len(moenergies): lumo_eigenvalues = [data.moenergies[0][homos + i] for i in range(1, 10)] # Create plot fig = Figure() if homo_eigenvalues and lumo_eigenvalues: fig.suptitle("Eigenvalues") ax = fig.subplots(2, 1) ax[0].plot(range(1, 10), homo_eigenvalues, label='Homo') ax[0].set_ylabel('eV') ax[0].legend() ax[1].plot(range(1, 10), lumo_eigenvalues, label='Lumo') ax[1].set_ylabel('eV') ax[1].legend() else: ax = fig.subplots() ax.text(0.5, 0.5, "No applicable data", horizontalalignment='center', verticalalignment='center', transform=ax.transAxes) # Export plot as image buffer buffer = io.BytesIO() fig.savefig(buffer, format='png') image_bytes = buffer.getvalue() buffer.close() # return dictionary with image data return { 'image': image_bytes, 'mime-type': 'image/png' }
entry_points
parameter in the $HOME/gateways19-tutorial/setup.py
file:setuptools.setup( # ... entry_points=""" [airavata.output_view_providers] gaussian-eigenvalues-plot = gateways19_tutorial.output_views:GaussianEigenvaluesViewProvider """, )
gaussian-eigenvalues-plot
is the output view provider id. gateways19_tutorial.output_views
is the module in which the GaussianEigenvaluesViewProvider
output view provider class is found.
docker exec -w /extensions gateways19-tutorial pip install -r requirements.txt docker exec -w /extensions gateways19-tutorial python setup.py develop docker exec gateways19-tutorial touch /code/django_airavata/wsgi.py
These commands:
Back in the Django Portal, we'll make sure the application interface for Gaussian is configured to add the GaussianEigenvaluesViewProvider as an additional output view of the file.
{ "output-view-providers": ["gaussian-eigenvalues-plot"] }
It should look something like this:
In additional to producing static visualizations, output view providers can declare interactive parameters that can be manipulated by the user. We can add a simple boolean interactive parameter to toggle the display of the matplotlib grid as an example.
$HOME/gateways19-tutorial/gateways19_tutorial/output_views.py
. Change the generate_data
function so that it has an additional show_grid
parameter with a default value of False
:def generate_data(self, request, experiment_output, experiment, output_file=None, show_grid=False):
.grid()
lines to the matplotlib code:... fig.suptitle("Eigenvalues") ax = fig.subplots(2, 1) ax[0].plot(range(1, 10), homo_eigenvalues, label='Homo') ax[0].set_ylabel('eV') ax[0].legend() ax[0].grid(show_grid) ax[1].plot(range(1, 10), lumo_eigenvalues, label='Lumo') ax[1].set_ylabel('eV') ax[1].legend() ax[1].grid(show_grid) ...
interactive
property and declare the show_grid
parameter:... # return dictionary with image data return { 'image': image_bytes, 'mime-type': 'image/png' 'interactive': [ {'name': 'show_grid', 'value': show_grid} ] }
This will provider the user with a checkbox for manipulating the show_grid parameter. Every time the user changes it, the GaussianEigenvaluesViewProvider will be again invoked. It should look something like the following:
There are several more interactive parameter types and additional options. You can learn more about them in the custom output view provider documentation.
In this tutorial exercise we'll create a fully custom user interface that lives within the Django Portal.
What we're going to build is a very simple user interface that will:
This is an intentionally simple example to demonstrate the general principle of using custom REST APIs and UI to setup, execute and post-process/visualize the output of a computational experiment.
We've already registered the Echo application with the portal, meaning we registered its interface and on which compute resource it is deployed.
A Django application or app is a Python package that may include Django views, url mappings, models, etc. It‘s a way of creating a kind of plug-in that integrates with a Django server. We’ll create this custom user interface by developing a Django app that uses the Django framework as well as the Airavata Django Portal REST APIs and JS library.
To start, we'll just create a simple “Hello World” page for the Django app and get it properly registered with the local Django Portal instance.
gateways19-tutorial
directory, open $HOME/gateways19-tutorial/gateways19_tutorial/templates/gateways19_tutorial/hello.html
. Some of the HTML view is commented out. The following is the uncommented content:{% extends 'base.html' %} {% block content %} <div class="main-content-wrapper"> <main class="main-content"> <div class="container-fluid"> <h1>Hello World</h1> </div> </main> </div> {% endblock content %}
$HOME/gateways19-tutorial/gateways19_tutorial/apps.py
:from django.apps import AppConfig class Gateways19TutorialAppConfig(AppConfig): name = 'gateways19_tutorial' label = name verbose_name = "Gateways Tutorial" fa_icon_class = "fa-comment"
This the main metadata for this custom Django app. Besides the normal metadata that the Django framework expects, this also defines a display name (verbose_name
) and an icon (fa_icon_class
) to use for this custom app.
$HOME/gateways19-tutorial/gateways19_tutorial/views.py
:from django.shortcuts import render from django.contrib.auth.decorators import login_required @login_required def hello_world(request): return render(request, "gateways19_tutorial/hello.html")
This view will simply display the template created in the previous step.
$HOME/gateways19-tutorial/gateways19_tutorial/urls.py
:from django.conf.urls import url, include from . import views app_name = 'gateways19_tutorial' urlpatterns = [ url(r'^hello/', views.hello_world, name="home"), ]
This maps the /hello/
URL to the hello_world
view.
$HOME/gateways19-tutorial/setup.py
, we add the following [airavata.djangoapp]
metadata to the entry_points section:setuptools.setup( # ... entry_points=""" [airavata.output_view_providers] gaussian-eigenvalues-plot = gateways19_tutorial.output_views:GaussianEigenvaluesViewProvider [airavata.djangoapp] gateways19_tutorial = gateways19_tutorial.apps:Gateways19TutorialAppConfig """, )
Now you should be able to log into the portal locally and see Gateways Tutorial in the drop down menu in the header (click on Workspace then you should see it in that menu).
Now we'll create a REST endpoint in our custom Django app that will return greetings in several languages.
$HOME/gateways19-tutorial/gateways19_tutorial/views.py
file, we add the following import:from django.http import JsonResponse
@login_required def languages(request): return JsonResponse({'languages': [{ 'lang': 'French', 'greeting': 'bonjour', }, { 'lang': 'German', 'greeting': 'guten tag' }, { 'lang': 'Hindi', 'greeting': 'namaste' }, { 'lang': 'Japanese', 'greeting': 'konnichiwa' }, { 'lang': 'Swahili', 'greeting': 'jambo' }, { 'lang': 'Turkish', 'greeting': 'merhaba' }]})
$HOME/gateways19-tutorial/gateways19_tutorial/urls.py
we add a url mapping for the languages
view:urlpatterns = [ url(r'^hello/', views.hello_world, name="home"), url(r'^languages/', views.languages, name="languages"), ]
$HOME/gateways19-tutorial/gateways19_tutorial/templates/gateways19_tutorial/hello.html
, uncomment the comment that starts <!-- Adding a list of "Hello" greetings
on line 11 and ends on line 21. That is, just delete lines 11 and 21. This adds a <select>
element to the template which will be used to display the greeting options:... <h1>Hello World</h1> <div class="card"> <div class="card-header">Run "echo" for different languages</div> <div class="card-body"> <select id="greeting-select"></select> <button id="run-button" class="btn btn-primary">Run</button> </div> </div> ...
{% load static %}
directive and then a scripts
block to the end of hello.html
. This will load the AiravataAPI JavaScript library which has utilities for interacting with the Django portal‘s REST API (which can also be used for custom developed REST endpoints) and model classes for Airavata’s data models. The utils.FetchUtils
is used to load the languages REST endpoint.{% extends 'base.html' %} {% load static %} ... {% endblock content %} {% block scripts %} <script src="{% static 'django_airavata_api/dist/airavata-api.js' %}"></script> <script> const { models, services, session, utils } = AiravataAPI; utils.FetchUtils.get("/gateways19_tutorial/languages/").then(data => { data.languages.forEach(language => { $("#greeting-select").append( `<option value="${language.greeting}"> ${language.lang} - "${language.greeting}" </option>` ); }); }); </script> {% endblock scripts %}
Now when you view the custom app at http://localhost:8000/gateways19_tutorial/hello/ you should see a dropdown of greetings in several languages, like so:
Now we‘ll use the AiravataAPI
library to load the user’s recent experiments.
$HOME/gateways19-tutorial/gateways19_tutorial/templates/gateways19_tutorial/hello.html
, uncomment the comment that begins with <!-- Displaying a list of recent experiments
on line 21 or so and ends on line 45. This adds table to display recent experiments to the bottom of hello.html
:... <div class="card"> <div class="card-header"> Run "echo" for different languages </div> <div class="card-body"> <select id="greeting-select"></select> <button id="run-button" class="btn btn-primary">Run</button> </div> </div> <div class="card"> <div class="card-header"> Experiments </div> <div class="card-body"> <button id="refresh-button" class="btn btn-secondary">Refresh</button> <table class="table"> <thead> <tr> <th scope="col">Name</th> <th scope="col">Application</th> <th scope="col">Creation Time</th> <th scope="col">Status</th> <th scope="col">Output</th> </tr> </thead> <tbody id="experiment-list"> </tbody> </table> </div> </div> </div> </main> </div> {% endblock content %}
hello.html
:// ... const appInterfaceId = "Echo_23d67491-1bef-47bd-a0f5-faf069e09773"; function loadExperiments() { return services.ExperimentSearchService .list({limit: 5, [models.ExperimentSearchFields.USER_NAME.name]: session.Session.username, [models.ExperimentSearchFields.APPLICATION_ID.name]: appInterfaceId, }) .then(data => { $('#experiment-list').empty(); data.results.forEach((exp, index) => { $('#experiment-list').append( `<tr> <td>${exp.name}</td> <td>${exp.executionId}</td> <td>${exp.creationTime}</td> <td>${exp.experimentStatus.name}</td> <td id="output_${index}"></td> </tr>`); }); }); } loadExperiments(); $("#refresh-button").click(loadExperiments); </script> {% endblock scripts %}
The user interface should now look something like:
Now we'll use AiravataAPI
to submit an Echo job.
$HOME/gateways19-tutorial/gateways19_tutorial/templates/gateways19_tutorial/hello.html
we add a click handler to the Run button that gets the selected greeting value:$("#run-button").click((e) => { const greeting = $("#greeting-select").val(); });
Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8
. We can browse the API for this application using: https://testdrive.airavata.org/api/applications/Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8/. First, we need the Application Interface for the application, which defines the inputs and outputs of the application. We can get its id by following the link to applicationInterface
: https://testdrive.airavata.org/api/applications/Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8/application_interface/. We'll create an Experiment instance from the Application Interface definition:const loadAppInterface = services.ApplicationInterfaceService.retrieve({ lookup: appInterfaceId, });
const appDeploymentId = "example-vc.jetstream-cloud.org_Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8"; const loadQueues = services.ApplicationDeploymentService.getQueues({ lookup: appDeploymentId, });
computeHostId
. The queue name we can get from following the link from the deployment to the queues: https://testdrive.airavata.org/api/application-deployments/example-vc.jetstream-cloud.org_Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8/queues/. Here we see that the queueName
is cloud
. We also need the account to use to submit the job and that is specified via a “Group Resource Profile”. https://testdrive.airavata.org/api/group-resource-profiles/ lists profiles you have access to and the compute resources each profile can use for job submission. We‘ll use the tutorial reservation one. Finally, experiments are organized by projects so we’ll also load the user‘s most recently used project which is stored in the user’s WorkspacePreferences:const resourceHostId = "example-vc.jetstream-cloud.org_794fd026-101a-46af-8868-5d7a86f813a1"; const queueName = "cloud"; const groupResourceProfileId = "fc245311-a7d1-41af-b8ae-a4142989c9a1"; const loadWorkspacePrefs = services.WorkspacePreferencesService.get();
Experiment
object then save and launch it. Here's the complete click handler:$("#run-button").click((e) => { const greeting = $("#greeting-select").val(); const loadAppInterface = services.ApplicationInterfaceService.retrieve({ lookup: appInterfaceId, }); const appDeploymentId = "example-vc.jetstream-cloud.org_Echo_37eb38ac-74c8-4aa4-a037-c656ab5bc6b8"; const loadQueues = services.ApplicationDeploymentService.getQueues({ lookup: appDeploymentId, }); const resourceHostId = "example-vc.jetstream-cloud.org_794fd026-101a-46af-8868-5d7a86f813a1"; const queueName = "cloud"; const groupResourceProfileId = "fc245311-a7d1-41af-b8ae-a4142989c9a1"; const loadWorkspacePrefs = services.WorkspacePreferencesService.get(); Promise.all([loadAppInterface, loadWorkspacePrefs, loadQueues]) .then(([appInterface, workspacePrefs, queues]) => { const experiment = appInterface.createExperiment(); experiment.experimentName = "Echo " + greeting; experiment.projectId = workspacePrefs.most_recent_project_id; const cloudQueue = queues.find((q) => q.queueName === queueName); experiment.userConfigurationData.groupResourceProfileId = groupResourceProfileId; experiment.userConfigurationData.computationalResourceScheduling.resourceHostId = resourceHostId; experiment.userConfigurationData.computationalResourceScheduling.totalCPUCount = cloudQueue.defaultCPUCount; experiment.userConfigurationData.computationalResourceScheduling.nodeCount = cloudQueue.defaultNodeCount; experiment.userConfigurationData.computationalResourceScheduling.wallTimeLimit = cloudQueue.defaultWalltime; experiment.userConfigurationData.computationalResourceScheduling.queueName = queueName; // Copy the selected greeting to the value of the first input experiment.experimentInputs[0].value = greeting; return services.ExperimentService.create({ data: experiment }); }) .then((exp) => { return services.ExperimentService.launch({ lookup: exp.experimentId, }); }); });
Now that we can launch the experiment we can go ahead and give it a try.
You can also try this out in the production deployment at https://testdrive.airavata.org/gateways19_tutorial/hello/.
Instead of simply reporting the status of the job we would also like to do something with the output. The STDOUT of the Echo job has a format like the following:
bonjour
We'll read the STDOUT file and display that in our experiment listing table.
downloadURL
. For each exp
we can use the FullExperimentService
to get these details like so:if (exp.experimentStatus === models.ExperimentState.COMPLETED) { services.FullExperimentService.retrieve({ lookup: exp.experimentId }).then( (fullDetails) => { const stdoutDataProductId = fullDetails.experiment.experimentOutputs.find( (o) => o.name === "Echo-STDOUT" ).value; const stdoutDataProduct = fullDetails.outputDataProducts.find( (dp) => dp.productUri === stdoutDataProductId ); if (stdoutDataProduct && stdoutDataProduct.downloadURL) { return fetch(stdoutDataProduct.downloadURL, { credentials: "same-origin", }).then((result) => result.text()); } } ); }
if (exp.experimentStatus === models.ExperimentState.COMPLETED) { services.FullExperimentService.retrieve({ lookup: exp.experimentId }) .then((fullDetails) => { const stdoutDataProductId = fullDetails.experiment.experimentOutputs.find( (o) => o.name === "Echo-STDOUT" ).value; const stdoutDataProduct = fullDetails.outputDataProducts.find( (dp) => dp.productUri === stdoutDataProductId ); if (stdoutDataProduct && stdoutDataProduct.downloadURL) { return fetch(stdoutDataProduct.downloadURL, { credentials: "same-origin", }).then((result) => result.text()); } }) .then((text) => { $(`#output_${index}`).text(text); }); }
/* Displaying the experiment output
on line 88 and ending on line 113. Here's the update to the loadExperiments
function:function loadExperiments() { return services.ExperimentSearchService.list({ limit: 5, [models.ExperimentSearchFields.USER_NAME.name]: session.Session.username, [models.ExperimentSearchFields.APPLICATION_ID.name]: appInterfaceId, }).then((data) => { $("#experiment-list").empty(); data.results.forEach((exp, index) => { $("#experiment-list").append( `<tr> <td>${exp.name}</td> <td>${exp.executionId}</td> <td>${exp.creationTime}</td> <td>${exp.experimentStatus.name}</td> <td id="output_${index}"></td> </tr>` ); // If experiment has finished, load full details, then parse the stdout file if (exp.experimentStatus === models.ExperimentState.COMPLETED) { services.FullExperimentService.retrieve({ lookup: exp.experimentId, }) .then((fullDetails) => { const stdoutDataProductId = fullDetails.experiment.experimentOutputs.find( (o) => o.name === "Echo-STDOUT" ).value; const stdoutDataProduct = fullDetails.outputDataProducts.find( (dp) => dp.productUri === stdoutDataProductId ); if ( stdoutDataProduct && stdoutDataProduct.downloadURL ) { return fetch(stdoutDataProduct.downloadURL, { credentials: "same-origin", }).then((result) => result.text()); } }) .then((text) => { $(`#output_${index}`).text(text); }); } }); }); }
You can browser the final version of the gateways19-tutorial code at https://github.com/machristie/gateways19-tutorial/tree/solution. If you get stuck at some point with the tutorial you can skip to the solution by running the following git command in your gateways19-tutorial repo:
cd $HOME/gateways19-tutorial git reset --hard origin/solution
The Django portal provides a REST API bridge to the backend Airavata API. So it's helpful to look at what is available in the backend Airavata API. See the Apache Airavata API docs
To see what is in the AiravataAPI
JavaScript library, take a look at it's index.js file in the airavata-django-portal repo.
SciGaP provides free Airavata Gateways hosting services. Log in or create an account at scigap.org to request gateway hosting.
The tutorial code needs to be on the remote Docker host so that it can be mounted into the Django portal container. To modify the files as required by the tutorial you'll need to either SSH into the remote Docker host and edit the files there with a terminal editor, like Vim, or you can use Visual Studio Code to edit the files remotely with the SSH extension.
To use Visual Studio Code, install it from https://code.visualstudio.com/. Once you have it installed, start it. In the menu, go to View > Extensions. In the Extensions search, type ssh
and click on Install for the Remote - SSH extension. Once it is installed, click on the green box in the lower left hand corner and select Remote-SSH: Connect to Host..., or, type Ctrl-Shift-P
and type Remote-SSH
and select Remote-SSH: Connect to Host.... Select linux as the remote platform type.
Once you are connected, go to View > Explorer. Click on the Open Folder button. Select the folder that contains the tutorial code and click OK. You should now see the tutorial files in the Explorer on the left hand side. Now you can edit the tutorial files from your local computer and they will be immediately reflected in the Django portal container.
To connect via SSH to the remote Docker host, you'll need an SSH key pair. If you are setting up the remote Docker host, you can create the key pair yourself and copy the public portion to the ~/.ssh/authorized_keys
file under the user account on the remote Docker host. For the in person tutorial session this will have already been setup and you will be provided with the private key.
C:\Users\<username>\
) called .ssh
..ssh
directory.config
file in the .ssh
directory. The contents of the file should be:Host IP_ADDRESS IdentityFile ~\.ssh\PRIVATE_KEY
Where IP_ADDRESS should be replaced with the IP address or hostname of the remote Docker host (for example: 149.165.168.201
). And PRIVATE_KEY should be replaced with the name of your private key file that you copied in step 2.
Test that SSH is working by opening a Powershell command prompt (in Visual Studio Code open a new terminal with Ctrl+Shift+`) and running the following:
ssh username@IP_ADDRESS
You should see something like the following:
PS C:\Users\testuser> ssh user1@149.165.157.132 The authenticity of host '149.165.157.132 (149.165.157.132)' can't be established. ECDSA key fingerprint is SHA256:RG86D7KwCZNQtFOAfEc4TZ4V0stn1RyGrj5I+v7SHxU. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '149.165.157.132' (ECDSA) to the list of known hosts. Last login: Thu Aug 20 14:31:41 2020 from 156-56-142-122.dhcp-bl.indiana.edu [user1@django-tutorial ~]$
If you get the “authenticity of host ... can't be established”, as shown above, enter yes
at the prompt to proceed. You shouldn't be prompted for a password since the private key will be used for authentication. If you are prompted for a password, double check that the config
file is correct.
To connect to Docker you'll need the Docker client. Download https://github.com/StefanScherer/docker-cli-builder/releases/download/19.03.12/docker.exe and copy it to C:\Windows
(or anywhere else that is on your PATH, or in the current directory).
Next, at a Powershell command prompt set the environment variable DOCKER_HOST
to the SSH username and IP address of the remote Docker host.
$env:DOCKER_HOST = "ssh://USERNAME@IP_ADDRESS"
But replace USERNAME with your username on the remote Docker host and the IP_ADDRESS with the IP address or domain name of the remote Docker host. For example:
$env:DOCKER_HOST = "ssh://train01@149.165.170.99"
Now run the following to test that the connection is working:
docker ps
You'll use this command prompt window during the tutorial to run docker commands.
!!! note “Note for legacy Command Prompt users”
If you are using `CMD`, the legacy command prompt program then you need a slightly different way of specifying the DOCKER_HOST environment variable. The form is ``` set DOCKER_HOST=ssh://USERNAME@IP_ADDRESS ```
For some of the tutorial instructions you'll be asked to load the Django portal in your browser at http://localhost:8000. However, your Django portal container is running on a remote Docker host, not your local computer. To make it appear that the Django portal is running locally, create an SSH tunnel to forward port 8000 to the remote Docker host. In a separate command prompt (in Visual Studio Code open a new terminal with Ctrl+Shift+`), run the following:
ssh -L 8000:localhost:8000 USERNAME@IP_ADDRESS