DLAB-000 fixed merge conflicts
diff --git a/DISCLAIMER b/DISCLAIMER
new file mode 100644
index 0000000..90bc650
--- /dev/null
+++ b/DISCLAIMER
@@ -0,0 +1,4 @@
+Apache DLab is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by The Apache Incubator.
+Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications,
+and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status
+is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
\ No newline at end of file
diff --git a/NOTICE b/NOTICE
index 1c4ead4..5dca626 100644
--- a/NOTICE
+++ b/NOTICE
@@ -1,4 +1,8 @@
+Apache DLab
+Copyright 2018-2019 The Apache Software Foundation
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
=======================================================================
Dlab Subcomponents:
diff --git a/README.md b/README.md
index 1924d90..36b1075 100644
--- a/README.md
+++ b/README.md
@@ -299,7 +299,7 @@
To build SSN node, following steps should be executed:
-1. Clone Git repository and make sure that all [pre-requisites](#Pre-requisites) are installed.
+1. Clone Git repository and make sure that all following [pre-requisites](#Pre-requisites) are installed.
2. Go to *dlab* directory.
3. Execute following script:
```
@@ -366,7 +366,7 @@
To build SSN node, following steps should be executed:
-1. Clone Git repository and make sure that all [pre-requisites](#Pre-requisites) are installed
+1. Clone Git repository and make sure that all following [pre-requisites](#Pre-requisites) are installed
2. Go to *dlab* directory
3. To have working billing functionality please review Billing configuration note and use proper parameters for SSN node deployment
4. To use Data Lake Store please review Azure Data Lake usage pre-requisites note and use proper parameters for SSN node deployment
@@ -449,7 +449,7 @@
To build SSN node, following steps should be executed:
-1. Clone Git repository and make sure that all [pre-requisites](#Pre-requisites) are installed.
+1. Clone Git repository and make sure that all following [pre-requisites](#Pre-requisites) are installed.
2. Go to *dlab* directory.
3. Execute following script:
```
@@ -1771,16 +1771,8 @@
In order to start development of Front-end Web UI part of DLab - Git repository should be cloned and the following packages should be installed:
- Git 1.7 or higher
-- Maven 3.3 or higher
- Python 2.7 with library Fabric v1.14.0
-- Mongo DB 3.0 or higher
- Docker 1.12 - Infrastructure provisioning
-- Java Development Kit 8 – Back-end
-- Node.js 6.x or 7.x - WebUI
-- Angular CLI v1.0.0-rc.1 or higher - WebUI
-- TypeScript v2.0 or higher - WebUI
-- Angular2 v2.4 – WebUI
-- Development IDE (Eclipse or Intellij IDEA)
## Java back-end services <a name="Java_back_end_services"></a>
@@ -2326,10 +2318,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Jupyter notebook 5.2.0",
+ "template_name": "Jupyter notebook 5.7.4",
"description": "Base image with jupyter node creation routines",
"environment_type": "exploratory",
- "version": "jupyter_notebook-5.2.0",
+ "version": "jupyter_notebook-5.7.4",
"vendor": "Azure"
}
]
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index afd287f..fbbb57b 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,56 +1,74 @@
# DLab is Self-service, Fail-safe Exploratory Environment for Collaborative Data Science Workflow
-## New features in v2.0
+## New features in v2.1
**All Cloud platforms:**
-- added possibility to go to links (Notebook UI, ungit, Tensorboard) without opening tunnel to Edge on AWS
-- implemented environment management page
-- added possibility to generate a key during key reuploading
+- implemented tuning Apache Spark standalone cluster and local spark configurations from WEB UI (except for Apache Zeppelin)
+- added a reminder after user logged in notifying that corresponding resources are about to be stopped/terminated
+- implemented SSN load monitor: CPU, Memory, HDD
+- updated versions of installed software:
+ * Jupyter 5.7.4
+ * RStudio 1.1.463
+ * Apache Zeppelin 0.8.0
+ * Apache Spark 2.3.2 for standalone cluster
+ * Scala 2.12.8
+ * CNTK 2.3.1
+ * Keras 2.1.6 (except for DeepLearning - 2.0.8)
+ * MXNET 1.3.1
+ * Theano 1.0.3
+ * ungit 1.4.36
**AWS:**
-- added new "RStudio with TensorFlow" template on AWS
-- Data Engine/Data Engine Service job tracker URL is displayed on Web UI on AWS
-- added possibility to use AWS default reporting as a source for DLAB billing
+- implemented tuning Data Engine Service from WEB UI (except for Apache Zeppelin)
+- added support of new version of Data Engine Service (AWS EMR) 5.19.0
+**MS azure and AWS:**
+- implemented ability to manage total billing quota for DLab as well as billing quota per user
-## Improvements in v2.0
+## Improvements in v2.1
**All Cloud platforms:**
-- it is now possible to separately configure start and stop schedules for analytical resources
-- added shell interpreter for Zeppelin
+- added ability to configure instance size/shape (CPU, RAM) from DLab UI for different user groups
+- added possibility to install Java dependencies from DLab UI
+- added alternative way to access analytical notebooks just by clicking on notebook's direct URL.
+ * added LDAP authorization in Squid (user should provide his LDAP credentials when accessing notebooks/Data Engine/Data Engine Service via browser)
+- improved error handling for various scenarios on UI side
+- added support of installing DLab into two VPCs
+**MS Azure:**
+- it is now possible to install DLab only with private IP’s
+
+## Bug fixes in v2.1
**AWS:**
-- optimized starting/stopping duration of Data Engine service
+- fixed pricing retrieval logic to optimize RAM usage on SSN for small instances
-**MS Azure and AWS:**
-- DLab's billing report, now indicates costs associated with any mounted storage of analytical tool
-
-
-## Bug fixes in v2.0
-**AWS:**
-- when computational resource name is part of a name of any other computational resource - it will correspondingly affected during stop/terminate actions (e.g. stopping EMR1 will stop EMR11, terminating EMR1 will terminate EMR11)
-
-**GCP:**
-- fixed occasionally reproducible problem: failure in Notebook stopping/starting and failure in creation of computational resources (it was not possible to automatically determine credentials)
-
-
-## Known issues in v2.0
+## Known issues in v2.1
**All Cloud platforms:**
-- remote kernel list for Data Engine is not updated after stop/start Data Engine
+- remote kernel list for Data Engine is not updated after stop/start Data Engine
- following links can be opened via tunnel for Data Engine/Data Engine: service: worker/application ID, application detail UI, event timeline, logs for Data Engine
+- if Apache Zeppelin is created from AMI with different instance shape, spark memory size is the same as in created AMI.
+- sparklyr library (r package) can not be installed on RStudio, RStudio with TensorFlow notebooks
+- Spark default configuration for Apache Zeppelin can not be changed from DLab UI. Currently it can be done directly through Apache Zeppelin interpreter menu.
+For more details please refer for Apache Zeppelin official documentation: https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/overview.html
+- shell interpreter for Apache Zeppelin is missed for some instance shapes
+- executor memory is not allocated depending on notebook instance shape for local spark
+
**AWS**
- can not open master application URL on resource manager page, issue known for Data Engine Service v.5.12.0
+- java library installation fails on DLab UI on Data Engine Service in case when it is installed together with libraries from other groups.
**GCP:**
- storage permissions aren't differentiated by users via Dataproc permissions (all users have R/W access to other users buckets)
- Data Engine Service creation is failing after environment has been recreated
- It is temporarily not possible to run playbooks using remote kernel of Data Engine (dependencies issue)
+- DeepLearning creation fails
**Microsoft Azure:**
- creation of Zeppelin from custom image fails on the step when cluster kernels are removing
-- start Notebook by scheduler does not work when Data Lake is enabled
+- start Notebook by scheduler does not work when Data Lake is enabled
+- playbook running on Apache Zeppelin fails due to impossible connection to blob via wasbs protocol
-## Known issues caused by cloud provider limitations in v2.0
+## Known issues caused by cloud provider limitations in v2.1
**Microsoft Azure:**
- resource name length should not exceed 80 chars
diff --git a/USER_GUIDE.md b/USER_GUIDE.md
index 0ed42e4..4a221a9 100644
--- a/USER_GUIDE.md
+++ b/USER_GUIDE.md
@@ -76,6 +76,16 @@
To stop working with DLab - click on Log Out link at the top right corner of DLab.
+After login user will see warning in case of exceeding quota or close to this limit.
+
+<p align="center" class="facebox-popup">
+ <img src="doc/exceeded quota.png" alt="Exceeded quota" width="400">
+</p>
+
+<p align="center" class="facebox-popup">
+ <img src="doc/close to limit.png" alt="Close to limit" width="400">
+</p>
+
----------------------------------
# Setup a Gateway/Edge node <a name="setup_edge_node"></a>
@@ -136,6 +146,8 @@
\* Please refer to official documentation from Amazon that will help you understand what [instance shapes](https://aws.amazon.com/ec2/instance-types/) would be most preferable in your particular DLAB setup. Also, you can use [AWS calculator](https://calculator.s3.amazonaws.com/index.html) to roughly estimate the cost of your environment.
+You can override the default configurations for local spark. The configuration object is referenced as a JSON file. To tune spark configuration check off "Spark configurations" check box and insert JSON format in text box.
+
After you Select the template, fill in the Name and choose needed instance shape - you need to click on "Create" button for your instance to start creating. Corresponding record will show up in your dashboard:
@@ -147,7 +159,7 @@
When you click on the name of your Analytical tool in the dashboard – analytical tool popup will show up:
<p align="center">
- <img src="doc/notebook_info.png" alt="Notebook info" width="520">
+ <img src="doc/notebook_info.png" alt="Notebook info" width="574">
</p>
In the header you will see version of analytical tool, its status and shape.
@@ -160,9 +172,13 @@
- Shared bucket for all users
- Bucket that has been provisioned for your needs
-To access analytical tool Web UI – you can use the first two links. In this case you don't need to open tunnel for Edge node and set SOCKS proxy.
+To access analytical tool Web UI you proceed with one of the options:
-Also you can use links via tunnel in order to access analytical tool Web UI. In this case you need to configure SOCKS proxy and open tunnel for Edge node. Please follow the steps described on “Read instruction how to create the tunnel” page to configure SOCKS proxy for Windows/MAC/Linux machines.
+- use direct URL's to access notebooks (your access will be established via reverse proxy, so you don't need to have Edge node tunnel up and running)
+- SOCKS proxy based URL's to access notebooks (via tunnel to Edge node)
+
+If you use direct urls you don't need to open tunnel for Edge node and set SOCKS proxy.
+If you use indirect urls you need to configure SOCKS proxy and open tunnel for Edge node. Please follow the steps described on “Read instruction how to create the tunnel” page to configure SOCKS proxy for Windows/MAC/Linux machines. “Read instruction how to create the tunnel” is available on DLab notebook popup.
### Manage libraries <a name="manage_libraries"></a>
@@ -226,6 +242,7 @@
</p>
Before clicking "Create" button you should choose the image from "Select AMI" and fill in the "Name" and "Instance shape".
+
--------------------------
## Stop Notebook server <a name="notebook_stop"></a>
@@ -284,11 +301,11 @@
This picture shows menu for creating Computational resource for AWS:
<p align="center">
- <img src="doc/emr_create.png" alt="Create Computational resource on AWS" width="480">
+ <img src="doc/emr_create.png" alt="Create Computational resource on AWS" width="760">
</p>
You can override the default configurations for applications by supplying a configuration object for applications when you create a cluster (this functionality is only available for Amazon EMR cluster ). The configuration object is referenced as a JSON file.
-To tune computational resource configuration check off "Configurations" check box and insert JSON format in text box:
+To tune computational resource configuration check off "Cluster configurations" check box and insert JSON format in text box:
<p align="center">
<img src="doc/emr_create_configuration.png" alt="Create Custom Computational resource on AWS" width="760">
@@ -296,7 +313,7 @@
This picture shows menu for creating Computational resource for Azure:
<p align="center">
- <img src="doc/dataengine_creating_menu.png" alt="Create Computational resource on Azure" width="480">
+ <img src="doc/dataengine_creating_menu.png" alt="Create Computational resource on Azure" width="760">
</p>
If you click on "Create" button Computational resource creation will kick off. You will see corresponding record on DLab Web UI in status **Creating**:
@@ -308,7 +325,7 @@
Clicking on Computational resource name in DLab dashboard will open Computational resource details popup:
<p align="center">
- <img src="doc/emr_info.png" alt="Computational resource info" width="400">
+ <img src="doc/emr_info.png" alt="Computational resource info" width="480">
</p>
Also you can go to computational resource master UI via link "Apache Spark Master' or "EMR Master" (this functionality is only available for AWS cloud).
@@ -372,7 +389,7 @@
When you click on the button "Git credentials" – following popup will show up:
<p align="center">
- <img src="doc/git_creds_window.png" alt="Git_creds_window" width="600">
+ <img src="doc/git_creds_window.png" alt="Git_creds_window" width="760">
</p>
In this window you need to add:
@@ -387,7 +404,7 @@
Clicking on "Apply changes" button, your credentials will be sent to all running instances with analytical tools. It takes a few seconds for changes to be applied.
<p align="center">
- <img src="doc/git_creds_window2.png" alt="Git_creds_window1" width="600">
+ <img src="doc/git_creds_window2.png" alt="Git_creds_window1" width="760">
</p>
On this tab you can also edit your credentials (click on pen icon) or delete (click on bin icon).
@@ -449,13 +466,13 @@
-To Stop Edge Node please click on actions icon on Health Status page and hit Stop.
+To Stop Edge Node please click on actions icon on Health Status page and hit "Stop".
<p align="center">
<img src="doc/edge_stop.png" alt="EDGE stop" width="150">
</p>
-Confirm you want to stop Edge node by clicking Yes:
+Confirm you want to stop Edge node by clicking "Yes":
<p align="center">
<img src="doc/edge_stop_confirm.png" alt="EDGE stop confirm" width="400">
@@ -468,7 +485,7 @@
Administrator can use backup functionality. In order to do it click Backup button. "Backup options" popup will show-up. You can choose a preferable option to be backed up.
<p align="center">
- <img src="doc/backup_options.png" alt="Backup options" width="480">
+ <img src="doc/backup_options.png" alt="Backup options" width="400">
</p>
Confirm you want to do backup by clicking "Apply".
@@ -491,6 +508,8 @@
<img src="doc/manage_env_confirm.png" alt="Manage environment confirm" width="550">
</p>
+Administrator can manage total billing quota for DLab as well as billing quota per user(s).To do this enter appropriate number in text box(es) per user(s) or/and total budget. Hit "Apply" button.
+
### Manage roles <a name="manage_roles"></a>
Administrator can choose what instance shape(s) and notebook(s) can be allowed for certain group(s) or user(s).
@@ -648,6 +667,12 @@
Once any scheduler is set up, notebook/spark cluster will be started/stopped automatically.
Please also note that if notebook is configured to be stopped, all running data engines assosiated with it will be stopped (for spark cluster) or terminated (for data engine serice) with notebook.
+After login user will be notified that corresponding resources are about to be stopped/terminated in some time.
+
+<p align="center">
+ <img src="doc/scheduler reminder.png" alt="Scheduler reminder" width="400">
+</p>
+
# Key reupload <a name="key_reupload"></a>
In case when user private key was corrupted, lost etc. DLAB provide a possibility to reupload user public key.
It can be done on manage environment page using ACTIONS menu on edge instance:
diff --git a/build.properties b/build.properties
index 6e130a7..ccfa9a6 100644
--- a/build.properties
+++ b/build.properties
@@ -1 +1,13 @@
-dlab.version=2.0
\ No newline at end of file
+# Copyright (c) 2018, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+dlab.version=2.1
\ No newline at end of file
diff --git a/doc/close to limit.png b/doc/close to limit.png
new file mode 100644
index 0000000..ad6fd0a
--- /dev/null
+++ b/doc/close to limit.png
Binary files differ
diff --git a/doc/create_notebook_from_ami.png b/doc/create_notebook_from_ami.png
index 072b0c2..7e4453e 100644
--- a/doc/create_notebook_from_ami.png
+++ b/doc/create_notebook_from_ami.png
Binary files differ
diff --git a/doc/dataengine_creating_menu.png b/doc/dataengine_creating_menu.png
index 0ef0460..e5294f0 100644
--- a/doc/dataengine_creating_menu.png
+++ b/doc/dataengine_creating_menu.png
Binary files differ
diff --git a/doc/delete_group.png b/doc/delete_group.png
index 8a82442..d5c38e3 100644
--- a/doc/delete_group.png
+++ b/doc/delete_group.png
Binary files differ
diff --git a/doc/emr_create.png b/doc/emr_create.png
index 37370f4..2e74bb0 100644
--- a/doc/emr_create.png
+++ b/doc/emr_create.png
Binary files differ
diff --git a/doc/emr_create_configuration.png b/doc/emr_create_configuration.png
index e30ef21..61c1a2a 100644
--- a/doc/emr_create_configuration.png
+++ b/doc/emr_create_configuration.png
Binary files differ
diff --git a/doc/emr_info.png b/doc/emr_info.png
index a1fc4f9..860cb74 100644
--- a/doc/emr_info.png
+++ b/doc/emr_info.png
Binary files differ
diff --git a/doc/exceeded quota.png b/doc/exceeded quota.png
new file mode 100644
index 0000000..4ba7a18
--- /dev/null
+++ b/doc/exceeded quota.png
Binary files differ
diff --git a/doc/manage_environment.png b/doc/manage_environment.png
index 18970fc..ead01e1 100644
--- a/doc/manage_environment.png
+++ b/doc/manage_environment.png
Binary files differ
diff --git a/doc/manage_role.png b/doc/manage_role.png
index 3fa96f9..152cf7c 100644
--- a/doc/manage_role.png
+++ b/doc/manage_role.png
Binary files differ
diff --git a/doc/notebook_create.png b/doc/notebook_create.png
index d2827e8..18a674b 100644
--- a/doc/notebook_create.png
+++ b/doc/notebook_create.png
Binary files differ
diff --git a/doc/notebook_info.png b/doc/notebook_info.png
index 62965c8..4cc01a2 100644
--- a/doc/notebook_info.png
+++ b/doc/notebook_info.png
Binary files differ
diff --git a/doc/scheduler reminder.png b/doc/scheduler reminder.png
new file mode 100644
index 0000000..d62bebc
--- /dev/null
+++ b/doc/scheduler reminder.png
Binary files differ
diff --git a/infrastructure-provisioning/scripts/deploy_dlab.py b/infrastructure-provisioning/scripts/deploy_dlab.py
index 201f7ff..f34f30e 100644
--- a/infrastructure-provisioning/scripts/deploy_dlab.py
+++ b/infrastructure-provisioning/scripts/deploy_dlab.py
@@ -26,6 +26,7 @@
parser.add_argument('--conf_network_type', type=str, default='', help='Define in which network DLab will be deployed. '
'Possible options: public|private')
parser.add_argument('--conf_vpc_cidr', type=str, default='', help='CIDR of VPC')
+parser.add_argument('--conf_vpc2_cidr', type=str, default='', help='CIDR of secondary VPC')
parser.add_argument('--conf_allowed_ip_cidr', type=str, default='', help='Comma-separated CIDR of IPs which will have '
'access to SSN')
parser.add_argument('--conf_user_subnets_range', type=str, default='', help='Range of subnets which will be using for '
@@ -59,7 +60,6 @@
parser.add_argument('--azure_vpc_name', type=str, default='', help='Azure VPC Name')
parser.add_argument('--gcp_vpc_name', type=str, default='', help='GCP VPC Name')
parser.add_argument('--aws_subnet_id', type=str, default='', help='AWS Subnet ID')
-parser.add_argument('--aws_subnet2_id', type=str, default='', help='AWS Subnet ID')
parser.add_argument('--azure_subnet_name', type=str, default='', help='Azure Subnet Name')
parser.add_argument('--gcp_subnet_name', type=str, default='', help='GCP Subnet Name')
parser.add_argument('--aws_security_groups_ids', type=str, default='', help='One of more comma-separated Security '
@@ -94,6 +94,8 @@
'validation(true|false).')
parser.add_argument('--azure_oauth2_enabled', type=str, default='false', help='Using OAuth2 for logging in DLab')
parser.add_argument('--azure_region_info', type=str, default='', help='Azure region info')
+parser.add_argument('--azure_source_vpc_name', type=str, default='', help='Azure VPC source Name')
+parser.add_argument('--azure_source_resource_group_name', type=str, default='', help='Azure source resource group')
parser.add_argument('--gcp_project_id', type=str, default='', help='The project ID in Google Cloud Platform')
parser.add_argument('--gcp_service_account_path', type=str, default='', help='The project ID in Google Cloud Platform')
parser.add_argument('--dlab_id', type=str, default="'user:user:tag'", help='Column name in report file that contains '
@@ -109,11 +111,13 @@
parser.add_argument('--cost', type=str, default='BlendedCost', help='Column name in report file that contains cost tag')
parser.add_argument('--resource_id', type=str, default='ResourceId', help='Column name in report file that contains '
'dlab resource id tag')
-parser.add_argument('--ldap_hostname', type=str, default='', help='Ldap instance hostname')
-parser.add_argument('--ldap_dn', type=str, default='', help='Ldap distinguished name (dc=example,dc=com)')
-parser.add_argument('--ldap_ou', type=str, default='ou=People', help='Ldap organisation unit (ou=People)')
-parser.add_argument('--ldap_service_username', type=str, default='', help='Ldap admin user name')
-parser.add_argument('--ldap_service_password', type=str, default='', help='Ldap password for admin user')
+parser.add_argument('--ldap_hostname', type=str, default='localhost', help='Ldap instance hostname')
+parser.add_argument('--ldap_dn', type=str, default='dc=example,dc=com',
+ help='Ldap distinguished name')
+parser.add_argument('--ldap_ou', type=str, default='ou=People', help='Ldap organisation unit')
+parser.add_argument('--ldap_service_username', type=str, default='cn=service-user', help='Ldap service user name')
+parser.add_argument('--ldap_service_password', type=str, default='service-user-password',
+ help='Ldap password for admin user')
parser.add_argument('--tags', type=str, default='Operation,ItemDescription', help='Column name in report file that '
'contains tags')
parser.add_argument('--action', required=True, type=str, default='', choices=['build', 'deploy', 'create', 'terminate'],
diff --git a/infrastructure-provisioning/src/base/scripts/install_prerequisites.py b/infrastructure-provisioning/src/base/scripts/install_prerequisites.py
index 7ad6554..3e646a4 100644
--- a/infrastructure-provisioning/src/base/scripts/install_prerequisites.py
+++ b/infrastructure-provisioning/src/base/scripts/install_prerequisites.py
@@ -60,13 +60,13 @@
change_pkg_repos()
create_china_pip_conf_file()
+ print("Updating hosts file")
+ update_hosts_file(args.user)
+
print("Updating repositories and installing requested tools.")
- if not ensure_pkg(args.user):
- sys.exit(1)
+ ensure_pkg(args.user)
print("Installing python packages: {}".format(args.pip_packages))
- if not ensure_pip(args.pip_packages):
- sys.exit(1)
+ ensure_pip(args.pip_packages)
- sys.exit(0)
diff --git a/infrastructure-provisioning/src/edge/templates/conf.d/proxy.conf b/infrastructure-provisioning/src/edge/templates/conf.d/proxy.conf
index 5dee3b4..7f0d42f 100644
--- a/infrastructure-provisioning/src/edge/templates/conf.d/proxy.conf
+++ b/infrastructure-provisioning/src/edge/templates/conf.d/proxy.conf
@@ -1,3 +1,20 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
server {
listen 80;
server_name EDGE_IP;
diff --git a/infrastructure-provisioning/src/edge/templates/locations/emr.conf b/infrastructure-provisioning/src/edge/templates/locations/emr.conf
index 88d88d4..cbaf963 100644
--- a/infrastructure-provisioning/src/edge/templates/locations/emr.conf
+++ b/infrastructure-provisioning/src/edge/templates/locations/emr.conf
@@ -1,3 +1,20 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
location /{{ CLUSTER_NAME }}/ {
rewrite ^/{{ CLUSTER_NAME }}/{{ CLUSTER_NAME }}/(.*)$ /$1 break;
rewrite ^/{{ CLUSTER_NAME }}/(.*)$ /$1 break;
diff --git a/infrastructure-provisioning/src/edge/templates/locations/jupyter.conf b/infrastructure-provisioning/src/edge/templates/locations/jupyter.conf
index eac56be..665db2b 100644
--- a/infrastructure-provisioning/src/edge/templates/locations/jupyter.conf
+++ b/infrastructure-provisioning/src/edge/templates/locations/jupyter.conf
@@ -1,3 +1,20 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
location ~* /{{ NAME }}/.* {
proxy_pass http://{{ IP }}:8888;
proxy_set_header Host $http_host;
diff --git a/infrastructure-provisioning/src/edge/templates/locations/rstudio.conf b/infrastructure-provisioning/src/edge/templates/locations/rstudio.conf
index 9fd7b3e..a025dc9 100644
--- a/infrastructure-provisioning/src/edge/templates/locations/rstudio.conf
+++ b/infrastructure-provisioning/src/edge/templates/locations/rstudio.conf
@@ -1,3 +1,20 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
location /{{ NAME }}/ {
rewrite ^/{{ NAME }}/(.*)$ /$1 break;
proxy_pass http://{{ IP }}:8787;
diff --git a/infrastructure-provisioning/src/edge/templates/locations/spark.conf b/infrastructure-provisioning/src/edge/templates/locations/spark.conf
index 6b15620..f8be5c0 100644
--- a/infrastructure-provisioning/src/edge/templates/locations/spark.conf
+++ b/infrastructure-provisioning/src/edge/templates/locations/spark.conf
@@ -1,3 +1,20 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
location /{{ CLUSTER_NAME }}/ {
rewrite ^/{{ CLUSTER_NAME }}/(.*)$ /$1 break;
proxy_pass http://{{ MASTER_IP }}:8080;
diff --git a/infrastructure-provisioning/src/edge/templates/locations/tensor.conf b/infrastructure-provisioning/src/edge/templates/locations/tensor.conf
index 96bee2a..af11b2c 100644
--- a/infrastructure-provisioning/src/edge/templates/locations/tensor.conf
+++ b/infrastructure-provisioning/src/edge/templates/locations/tensor.conf
@@ -1,4 +1,20 @@
-
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
location /{{ NAME }}-tensor/ {
rewrite ^/{{ NAME }}-tensor/(.*)$ /$1 break;
proxy_pass http://{{ IP }}:6006;
diff --git a/infrastructure-provisioning/src/edge/templates/locations/ungit.conf b/infrastructure-provisioning/src/edge/templates/locations/ungit.conf
index af31911..da9a9da 100644
--- a/infrastructure-provisioning/src/edge/templates/locations/ungit.conf
+++ b/infrastructure-provisioning/src/edge/templates/locations/ungit.conf
@@ -1,4 +1,20 @@
-
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
location ~* /{{ NAME }}-ungit/.* {
proxy_pass http://{{ IP }}:8085;
proxy_set_header Host $http_host;
diff --git a/infrastructure-provisioning/src/edge/templates/locations/zeppelin.conf b/infrastructure-provisioning/src/edge/templates/locations/zeppelin.conf
index 61537d5..c33b0c6 100644
--- a/infrastructure-provisioning/src/edge/templates/locations/zeppelin.conf
+++ b/infrastructure-provisioning/src/edge/templates/locations/zeppelin.conf
@@ -1,3 +1,20 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
location /{{ NAME }}/ {
rewrite ^/{{ NAME }}/(.*)$ /$1 break;
proxy_pass http://{{ IP }}:8080;
diff --git a/infrastructure-provisioning/src/edge/templates/nginx.conf b/infrastructure-provisioning/src/edge/templates/nginx.conf
index 09ba937..37efcde 100644
--- a/infrastructure-provisioning/src/edge/templates/nginx.conf
+++ b/infrastructure-provisioning/src/edge/templates/nginx.conf
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
diff --git a/infrastructure-provisioning/src/edge/templates/nginx_debian b/infrastructure-provisioning/src/edge/templates/nginx_debian
index 6205c5d..1f69184 100644
--- a/infrastructure-provisioning/src/edge/templates/nginx_debian
+++ b/infrastructure-provisioning/src/edge/templates/nginx_debian
@@ -1,4 +1,21 @@
#!/bin/sh
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
### BEGIN INIT INFO
# Provides: nginx
diff --git a/infrastructure-provisioning/src/edge/templates/nginx_redhat b/infrastructure-provisioning/src/edge/templates/nginx_redhat
index 67c260d..31a4ea3 100644
--- a/infrastructure-provisioning/src/edge/templates/nginx_redhat
+++ b/infrastructure-provisioning/src/edge/templates/nginx_redhat
@@ -1,6 +1,23 @@
#!/bin/sh
+# *****************************************************************************
#
-# nginx - this script starts and stops the nginx daemin
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+#
+# nginx - this script starts and stops the nginx daemon
#
# chkconfig: - 85 15
# description: Nginx is an HTTP(S) server, HTTP(S) reverse \
diff --git a/infrastructure-provisioning/src/edge/templates/squid.conf b/infrastructure-provisioning/src/edge/templates/squid.conf
index 613638c..b8930b4 100644
--- a/infrastructure-provisioning/src/edge/templates/squid.conf
+++ b/infrastructure-provisioning/src/edge/templates/squid.conf
@@ -1,3 +1,20 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
auth_param basic program LDAP_AUTH_PATH -b "LDAP_DN" -D "LDAP_SERVICE_USERNAME,LDAP_DN" -w LDAP_SERVICE_PASSWORD -f uid=%s LDAP_HOST
acl DLab_user_src_subnet src PROXY_SUBNET
diff --git a/infrastructure-provisioning/src/general/conf/dlab.ini b/infrastructure-provisioning/src/general/conf/dlab.ini
index 93a4298..43dc86e 100644
--- a/infrastructure-provisioning/src/general/conf/dlab.ini
+++ b/infrastructure-provisioning/src/general/conf/dlab.ini
@@ -1,3 +1,21 @@
+## *****************************************************************************
+##
+## Copyright (c) 2016, EPAM SYSTEMS INC
+##
+## Licensed under the Apache License, Version 2.0 (the "License");
+## you may not use this file except in compliance with the License.
+## You may obtain a copy of the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS,
+## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+## See the License for the specific language governing permissions and
+## limitations under the License.
+##
+## ******************************************************************************
+
#---- List of all dlab parameters (commented ones are passing from UI/Jenkins) ----#
@@ -55,8 +73,6 @@
# security_groups_ids =
### Id of the subnet for SSN and EDGE provisioning
# subnet_id =
-### Id of the subnet for notebooks and compute engines
-# subnet2_id =
### Id of the vpc for whole dlab provisioning
# vpc_id =
### Id of the secondary vpc for notebooks and compute engines
@@ -191,19 +207,19 @@
### Size of the additional volume for notebook instance
disk_size = 30
### Version of Apache Spark to be installed on notebook
-spark_version = 2.2.1
+spark_version = 2.3.2
### Version of Apache Hadoop to be installed on notebook
hadoop_version = 2.7
### Version of Jupyter to be installed on notebook
-jupyter_version = 5.2.0
+jupyter_version = 5.7.4
### Version of TensorFlow to be installed on notebook
tensorflow_version = 1.8.0
### Version of Zeppelin to be installed on notebook
-zeppelin_version = 0.7.3
+zeppelin_version = 0.8.0
### Version of Rstudio to be installed on notebook
-rstudio_version = 1.1.383
+rstudio_version = 1.1.463
### Version of Scala to be installed on notebook
-scala_version = 2.11.8
+scala_version = 2.12.8
### Version of Livy top be installed on notebook
livy_version = 0.3.0
### If it is true, Livy will be used on Zeppelin notebook
@@ -219,13 +235,14 @@
### Cmake version for DeepLearning notebook
cmake_version = 3.11.3
### CNTK library version for DeepLearning notebook
-cntk_version = 2.1
+### All releases 2.4+ officially only support Ubuntu 16.04.
+cntk_version = 2.3.1
### MXNet library version for DeepLearning notebook
-mxnet_version = 1.0.0
+mxnet_version = 1.3.1
### Keras library version for Tensor/DeepLearning notebook
keras_version = 2.1.6
### Theano library version for Tensor/DeepLearning notebook
-theano_version = 0.9.0
+theano_version = 1.0.3
### Version of CUDA
cuda_version = 9.0
### Name of CUDA file
@@ -240,7 +257,7 @@
tornado_version = 4.5.3
ipykernel_version = 4.8.2
### Version of ungit if previous needed. Use latest as default.
-ungit_version = 1.4.31
+ungit_version = 1.4.36
### Numpy version
numpy_version = 1.14.3
### Apache Ivy version
diff --git a/infrastructure-provisioning/src/general/files/aws/dataengine-service_description.json b/infrastructure-provisioning/src/general/files/aws/dataengine-service_description.json
index b46f5d0..7caee2a 100644
--- a/infrastructure-provisioning/src/general/files/aws/dataengine-service_description.json
+++ b/infrastructure-provisioning/src/general/files/aws/dataengine-service_description.json
@@ -24,7 +24,7 @@
},
"templates":
[
- {"version":"emr-5.6.0", "applications": [{"Name":"Hadoop", "Version": "2.7.3"}, {"Name":"Spark", "Version": "2.1.1"}, {"Name":"Hive", "Version": "2.1.1"}]},
- {"version":"emr-5.12.0", "applications": [{"Name":"Hadoop", "Version": "2.8.3"}, {"Name":"Spark", "Version": "2.2.1"}, {"Name":"Hive", "Version": "2.3.2"}]}
+ {"version":"emr-5.12.0", "applications": [{"Name":"Hadoop", "Version": "2.8.3"}, {"Name":"Spark", "Version": "2.2.1"}, {"Name":"Hive", "Version": "2.3.2"}]},
+ {"version":"emr-5.19.0", "applications": [{"Name":"Hadoop", "Version": "2.8.5"}, {"Name":"Spark", "Version": "2.3.2"}, {"Name":"Hive", "Version": "2.3.3"}]}
]
}
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/files/aws/deeplearning_description.json b/infrastructure-provisioning/src/general/files/aws/deeplearning_description.json
index e8fe00a..da6f79d 100644
--- a/infrastructure-provisioning/src/general/files/aws/deeplearning_description.json
+++ b/infrastructure-provisioning/src/general/files/aws/deeplearning_description.json
@@ -8,10 +8,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Deep Learning 1.9",
+ "template_name": "Deep Learning 2.1",
"description": "Base image with Deep Learning and Jupyter",
"environment_type": "exploratory",
- "version": "deeplearning-1.9",
+ "version": "deeplearning-2.1",
"vendor": "AWS"
}
]
diff --git a/infrastructure-provisioning/src/general/files/aws/jupyter_description.json b/infrastructure-provisioning/src/general/files/aws/jupyter_description.json
index 1d59285..9b14a9f 100644
--- a/infrastructure-provisioning/src/general/files/aws/jupyter_description.json
+++ b/infrastructure-provisioning/src/general/files/aws/jupyter_description.json
@@ -19,10 +19,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Jupyter notebook 5.2.0",
+ "template_name": "Jupyter notebook 5.7.4",
"description": "Base image with Jupyter node creation routines",
"environment_type": "exploratory",
- "version": "jupyter_notebook-5.2.0",
+ "version": "jupyter_notebook-5.7.4",
"vendor": "AWS"
}
]
diff --git a/infrastructure-provisioning/src/general/files/aws/rstudio_description.json b/infrastructure-provisioning/src/general/files/aws/rstudio_description.json
index 618e165..e23e275 100644
--- a/infrastructure-provisioning/src/general/files/aws/rstudio_description.json
+++ b/infrastructure-provisioning/src/general/files/aws/rstudio_description.json
@@ -19,10 +19,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "RStudio 1.1.383",
+ "template_name": "RStudio 1.1.463",
"description": "Base image with RStudio node creation routines",
"environment_type": "exploratory",
- "version": "RStudio-1.1.383",
+ "version": "RStudio-1.1.463",
"vendor": "AWS"
}
]
diff --git a/infrastructure-provisioning/src/general/files/aws/tensor-rstudio_description.json b/infrastructure-provisioning/src/general/files/aws/tensor-rstudio_description.json
index f7f4c76..b838f67 100644
--- a/infrastructure-provisioning/src/general/files/aws/tensor-rstudio_description.json
+++ b/infrastructure-provisioning/src/general/files/aws/tensor-rstudio_description.json
@@ -8,7 +8,7 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Rstudio with TensorFlow 1.8.0",
+ "template_name": "RStudio with TensorFlow 1.8.0",
"description": "Base image with TensorFlow and RStudio node creation routines",
"environment_type": "exploratory",
"version": "tensorflow_gpu-1.8.0",
diff --git a/infrastructure-provisioning/src/general/files/aws/zeppelin_description.json b/infrastructure-provisioning/src/general/files/aws/zeppelin_description.json
index 5b59395..31cb86d 100644
--- a/infrastructure-provisioning/src/general/files/aws/zeppelin_description.json
+++ b/infrastructure-provisioning/src/general/files/aws/zeppelin_description.json
@@ -19,10 +19,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Apache Zeppelin 0.7.3",
+ "template_name": "Apache Zeppelin 0.8.0",
"description": "Base image with Apache Zeppelin node creation routines",
"environment_type": "exploratory",
- "version": "zeppelin-0.7.3",
+ "version": "zeppelin-0.8.0",
"vendor": "AWS"
}
]
diff --git a/infrastructure-provisioning/src/general/files/azure/deeplearning_description.json b/infrastructure-provisioning/src/general/files/azure/deeplearning_description.json
index b9fdaef..704ba7d 100644
--- a/infrastructure-provisioning/src/general/files/azure/deeplearning_description.json
+++ b/infrastructure-provisioning/src/general/files/azure/deeplearning_description.json
@@ -8,10 +8,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Deep Learning 1.9",
+ "template_name": "Deep Learning 2.1",
"description": "Base image with Deep Learning and Jupyter",
"environment_type": "exploratory",
- "version": "deeplearning-1.9",
+ "version": "deeplearning-2.1",
"vendor": "Azure"
}
]
diff --git a/infrastructure-provisioning/src/general/files/azure/jupyter_description.json b/infrastructure-provisioning/src/general/files/azure/jupyter_description.json
index 24bc1ed..638531e 100644
--- a/infrastructure-provisioning/src/general/files/azure/jupyter_description.json
+++ b/infrastructure-provisioning/src/general/files/azure/jupyter_description.json
@@ -15,10 +15,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Jupyter notebook 5.2.0",
+ "template_name": "Jupyter notebook 5.7.4",
"description": "Base image with Jupyter node creation routines",
"environment_type": "exploratory",
- "version": "jupyter_notebook-5.2.0",
+ "version": "jupyter_notebook-5.7.4",
"vendor": "Azure"
}
]
diff --git a/infrastructure-provisioning/src/general/files/azure/rstudio_description.json b/infrastructure-provisioning/src/general/files/azure/rstudio_description.json
index 74fa5f1..8457197 100644
--- a/infrastructure-provisioning/src/general/files/azure/rstudio_description.json
+++ b/infrastructure-provisioning/src/general/files/azure/rstudio_description.json
@@ -15,10 +15,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "RStudio 1.1.383",
+ "template_name": "RStudio 1.1.463",
"description": "Base image with RStudio node creation routines",
"environment_type": "exploratory",
- "version": "RStudio-1.1.383",
+ "version": "RStudio-1.1.463",
"vendor": "Azure"
}
]
diff --git a/infrastructure-provisioning/src/general/files/azure/zeppelin_description.json b/infrastructure-provisioning/src/general/files/azure/zeppelin_description.json
index 74fee0b..499d3ff 100644
--- a/infrastructure-provisioning/src/general/files/azure/zeppelin_description.json
+++ b/infrastructure-provisioning/src/general/files/azure/zeppelin_description.json
@@ -15,10 +15,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Apache Zeppelin 0.7.3",
+ "template_name": "Apache Zeppelin 0.8.0",
"description": "Base image with Apache Zeppelin node creation routines",
"environment_type": "exploratory",
- "version": "zeppelin-0.7.2",
+ "version": "zeppelin-0.8.0",
"vendor": "Azure"
}
]
diff --git a/infrastructure-provisioning/src/general/files/gcp/deeplearning_description.json b/infrastructure-provisioning/src/general/files/gcp/deeplearning_description.json
index 9e0bfb6..ceaebc2 100644
--- a/infrastructure-provisioning/src/general/files/gcp/deeplearning_description.json
+++ b/infrastructure-provisioning/src/general/files/gcp/deeplearning_description.json
@@ -10,10 +10,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Deep Learning 1.9",
+ "template_name": "Deep Learning 2.1",
"description": "Base image with Deep Learning and Jupyter",
"environment_type": "exploratory",
- "version": "deeplearning-1.9",
+ "version": "deeplearning-2.1",
"vendor": "GCP"
}
]
diff --git a/infrastructure-provisioning/src/general/files/gcp/jupyter_description.json b/infrastructure-provisioning/src/general/files/gcp/jupyter_description.json
index 4812532..7a18b6a 100644
--- a/infrastructure-provisioning/src/general/files/gcp/jupyter_description.json
+++ b/infrastructure-provisioning/src/general/files/gcp/jupyter_description.json
@@ -24,10 +24,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Jupyter notebook 5.2.0",
+ "template_name": "Jupyter notebook 5.7.4",
"description": "Base image with jupyter node creation routines",
"environment_type": "exploratory",
- "version": "jupyter_notebook-5.2.0",
+ "version": "jupyter_notebook-5.7.4",
"vendor": "GCP"
}
]
diff --git a/infrastructure-provisioning/src/general/files/gcp/rstudio_description.json b/infrastructure-provisioning/src/general/files/gcp/rstudio_description.json
index 7962c78..95d83fa 100644
--- a/infrastructure-provisioning/src/general/files/gcp/rstudio_description.json
+++ b/infrastructure-provisioning/src/general/files/gcp/rstudio_description.json
@@ -24,10 +24,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "RStudio 1.1.383",
+ "template_name": "RStudio 1.1.463",
"description": "Base image with rstudio node creation routines",
"environment_type": "exploratory",
- "version": "RStudio-1.1.383",
+ "version": "RStudio-1.1.463",
"vendor": "GCP"
}
]
diff --git a/infrastructure-provisioning/src/general/files/gcp/zeppelin_description.json b/infrastructure-provisioning/src/general/files/gcp/zeppelin_description.json
index 5cb1aaf..ccb152a 100644
--- a/infrastructure-provisioning/src/general/files/gcp/zeppelin_description.json
+++ b/infrastructure-provisioning/src/general/files/gcp/zeppelin_description.json
@@ -24,10 +24,10 @@
"exploratory_environment_versions" :
[
{
- "template_name": "Apache Zeppelin 0.7.3",
+ "template_name": "Apache Zeppelin 0.8.0",
"description": "Base image with Apache Zeppelin node creation routines",
"environment_type": "exploratory",
- "version": "zeppelin-0.7.2",
+ "version": "zeppelin-0.8.0",
"vendor": "GCP"
}
]
diff --git a/infrastructure-provisioning/src/general/files/os/debian/sources.list b/infrastructure-provisioning/src/general/files/os/debian/sources.list
index c363815..b35a23e 100644
--- a/infrastructure-provisioning/src/general/files/os/debian/sources.list
+++ b/infrastructure-provisioning/src/general/files/os/debian/sources.list
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
diff --git a/infrastructure-provisioning/src/general/files/os/ivysettings.xml b/infrastructure-provisioning/src/general/files/os/ivysettings.xml
index 8ba5b7e..44be1c0 100644
--- a/infrastructure-provisioning/src/general/files/os/ivysettings.xml
+++ b/infrastructure-provisioning/src/general/files/os/ivysettings.xml
@@ -1,3 +1,20 @@
+<!--
+
+Copyright (c) 2016, EPAM SYSTEMS INC
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
<ivysettings>
<settings defaultResolver="maven"/>
diff --git a/infrastructure-provisioning/src/general/files/os/redhat/sources.list b/infrastructure-provisioning/src/general/files/os/redhat/sources.list
index 6a278f8..36e2bde 100644
--- a/infrastructure-provisioning/src/general/files/os/redhat/sources.list
+++ b/infrastructure-provisioning/src/general/files/os/redhat/sources.list
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
diff --git a/infrastructure-provisioning/src/general/lib/aws/__init__.py b/infrastructure-provisioning/src/general/lib/aws/__init__.py
index e69de29..7c64801 100644
--- a/infrastructure-provisioning/src/general/lib/aws/__init__.py
+++ b/infrastructure-provisioning/src/general/lib/aws/__init__.py
@@ -0,0 +1,17 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
diff --git a/infrastructure-provisioning/src/general/lib/aws/actions_lib.py b/infrastructure-provisioning/src/general/lib/aws/actions_lib.py
index 5765c0d..ea32cdb 100644
--- a/infrastructure-provisioning/src/general/lib/aws/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/aws/actions_lib.py
@@ -1002,7 +1002,7 @@
else:
print("There are no peering connections to delete")
else:
- print("There are no peering connections to delete")
+ print("There are no peering connections to delete because duo vpc option is disabled")
except Exception as err:
logging.info("Unable to remove peering connection: " + str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
append_result(str({"error": "Unable to remove peering connection", "error_message": str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout)}))
@@ -1094,7 +1094,7 @@
traceback.print_exc(file=sys.stdout)
-def remove_kernels(emr_name, tag_name, nb_tag_value, ssh_user, key_path, emr_version):
+def remove_kernels(emr_name, tag_name, nb_tag_value, ssh_user, key_path, emr_version, computational_name=''):
try:
ec2 = boto3.resource('ec2')
inst = ec2.instances.filter(
@@ -1150,7 +1150,7 @@
sudo('sleep 5')
sudo('rm -rf /home/{}/.ensure_dir/dataengine-service_{}_interpreter_ensured'.format(ssh_user, emr_name))
if exists('/home/{}/.ensure_dir/rstudio_dataengine-service_ensured'.format(ssh_user)):
- dlab.fab.remove_rstudio_dataengines_kernel(emr_name, ssh_user)
+ dlab.fab.remove_rstudio_dataengines_kernel(computational_name, ssh_user)
sudo('rm -rf /opt/' + emr_version + '/' + emr_name + '/')
print("Notebook's {} kernels were removed".format(env.hosts))
else:
@@ -1453,7 +1453,7 @@
endpoint_url = "https://s3.{}.amazonaws.com.cn".format(args.region)
else:
endpoint_url = 'https://s3-' + args.region + '.amazonaws.com'
- local("""bash -c 'echo "spark.hadoop.fs.s3a.endpoint """ + endpoint_url + """" >> """ + spark_def_path + """'""")
+ local("""bash -c 'echo "spark.hadoop.fs.s3a.endpoint """ + endpoint_url + """ " >> """ + spark_def_path + """'""")
local('echo "spark.hadoop.fs.s3a.server-side-encryption-algorithm AES256" >> {}'.format(spark_def_path))
@@ -1540,24 +1540,26 @@
python_version = f.read()
python_version = python_version[0:5]
livy_port = ''
- livy_path = '/opt/' + emr_version + '/' + cluster_name + '/livy/'
- spark_libs = "/opt/" + emr_version + "/jars/usr/share/aws/aws-java-sdk/aws-java-sdk-core*.jar /opt/" + \
- emr_version + "/jars/usr/lib/hadoop/hadoop-aws*.jar /opt/" + emr_version + \
- "/jars/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-*.jar /opt/" + emr_version + \
- "/jars/usr/lib/hadoop-lzo/lib/hadoop-lzo-*.jar"
+ livy_path = '/opt/{0}/{1}/livy/'.format(emr_version, cluster_name)
+ spark_libs = "/opt/{0}/jars/usr/share/aws/aws-java-sdk/aws-java-sdk-core*.jar /opt/{0}/jars/usr/lib/hadoop/hadoop-aws*.jar /opt/" + \
+ "{0}/jars/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-*.jar /opt/{0}" + \
+ "/jars/usr/lib/hadoop-lzo/lib/hadoop-lzo-*.jar".format(emr_version)
+ #fix due to: Multiple py4j files found under ..../spark/python/lib
+ #py4j-0.10.7-src.zip still in folder. Versions may varies.
+ local('rm /opt/{0}/{1}/spark/python/lib/py4j-src.zip'.format(emr_version, cluster_name))
+
local('echo \"Configuring emr path for Zeppelin\"')
- local('sed -i \"s/^export SPARK_HOME.*/export SPARK_HOME=\/opt\/' + emr_version + '\/' +
- cluster_name + '\/spark/\" /opt/zeppelin/conf/zeppelin-env.sh')
- local('sed -i \"s/^export HADOOP_CONF_DIR.*/export HADOOP_CONF_DIR=\/opt\/' + emr_version + '\/' +
- cluster_name + '\/conf/\" /opt/' + emr_version + '/' + cluster_name +
- '/spark/conf/spark-env.sh')
- local('echo \"spark.jars $(ls ' + spark_libs + ' | tr \'\\n\' \',\')\" >> /opt/' + emr_version + '/' +
- cluster_name + '/spark/conf/spark-defaults.conf')
- local('sed -i "/spark.executorEnv.PYTHONPATH/d" /opt/' + emr_version + '/' + cluster_name +
- '/spark/conf/spark-defaults.conf')
- local('sed -i "/spark.yarn.dist.files/d" /opt/' + emr_version + '/' + cluster_name +
- '/spark/conf/spark-defaults.conf')
- local('sudo chown ' + os_user + ':' + os_user + ' -R /opt/zeppelin/')
+ local('sed -i \"s/^export SPARK_HOME.*/export SPARK_HOME=\/opt\/{0}\/{1}\/spark/\" /opt/zeppelin/conf/zeppelin-env.sh'.
+ format(emr_version, cluster_name))
+ local('sed -i "s/^export HADOOP_CONF_DIR.*/export HADOOP_CONF_DIR=' + \
+ '\/opt\/{0}\/{1}\/conf/" /opt/{0}/{1}/spark/conf/spark-env.sh'.format(emr_version, cluster_name))
+ local('echo \"spark.jars $(ls {0} | tr \'\\n\' \',\')\" >> /opt/{1}/{2}/spark/conf/spark-defaults.conf'
+ .format(spark_libs, emr_version, cluster_name))
+ local('sed -i "/spark.executorEnv.PYTHONPATH/d" /opt/{0}/{1}/spark/conf/spark-defaults.conf'
+ .format(emr_version, cluster_name))
+ local('sed -i "/spark.yarn.dist.files/d" /opt/{0}/{1}/spark/conf/spark-defaults.conf'
+ .format(emr_version, cluster_name))
+ local('sudo chown {0}:{0} -R /opt/zeppelin/'.format(os_user))
local('sudo systemctl daemon-reload')
local('sudo service zeppelin-notebook stop')
local('sudo service zeppelin-notebook start')
@@ -1579,15 +1581,13 @@
port_number_found = True
else:
default_port += 1
- local('sudo echo "livy.server.port = ' + str(livy_port) + '" >> ' + livy_path + 'conf/livy.conf')
- local('sudo echo "livy.spark.master = yarn" >> ' + livy_path + 'conf/livy.conf')
- if os.path.exists(livy_path + 'conf/spark-blacklist.conf'):
- local('sudo sed -i "s/^/#/g" ' + livy_path + 'conf/spark-blacklist.conf')
- local(''' sudo echo "export SPARK_HOME=''' + spark_dir + '''" >> ''' + livy_path + '''conf/livy-env.sh''')
- local(''' sudo echo "export HADOOP_CONF_DIR=''' + yarn_dir + '''" >> ''' + livy_path +
- '''conf/livy-env.sh''')
- local(''' sudo echo "export PYSPARK3_PYTHON=python''' + python_version[0:3] + '''" >> ''' +
- livy_path + '''conf/livy-env.sh''')
+ local('sudo echo "livy.server.port = {0}" >> {1}conf/livy.conf'.format(str(livy_port), livy_path))
+ local('sudo echo "livy.spark.master = yarn" >> {}conf/livy.conf'.format(livy_path))
+ if os.path.exists('{}conf/spark-blacklist.conf'.format(livy_path)):
+ local('sudo sed -i "s/^/#/g" {}conf/spark-blacklist.conf'.format(livy_path))
+ local(''' sudo echo "export SPARK_HOME={0}" >> {1}conf/livy-env.sh'''.format(spark_dir, livy_path))
+ local(''' sudo echo "export HADOOP_CONF_DIR={0}" >> {1}conf/livy-env.sh'''.format(yarn_dir, livy_path))
+ local(''' sudo echo "export PYSPARK3_PYTHON=python{0}" >> {1}conf/livy-env.sh'''.format(python_version[0:3], livy_path))
template_file = "/tmp/dataengine-service_interpreter.json"
fr = open(template_file, 'r+')
text = fr.read()
@@ -1605,16 +1605,16 @@
break
except:
local('sleep 5')
- local('sudo cp /opt/livy-server-cluster.service /etc/systemd/system/livy-server-' + str(livy_port) +
- '.service')
- local("sudo sed -i 's|OS_USER|" + os_user + "|' /etc/systemd/system/livy-server-" + str(livy_port) +
- '.service')
- local("sudo sed -i 's|LIVY_PATH|" + livy_path + "|' /etc/systemd/system/livy-server-" + str(livy_port)
- + '.service')
- local('sudo chmod 644 /etc/systemd/system/livy-server-' + str(livy_port) + '.service')
+ local('sudo cp /opt/livy-server-cluster.service /etc/systemd/system/livy-server-{}.service'
+ .format(str(livy_port)))
+ local("sudo sed -i 's|OS_USER|{0}|' /etc/systemd/system/livy-server-{1}.service"
+ .format(os_user, str(livy_port)))
+ local("sudo sed -i 's|LIVY_PATH|{0}|' /etc/systemd/system/livy-server-{1}.service"
+ .format(livy_path, str(livy_port)))
+ local('sudo chmod 644 /etc/systemd/system/livy-server-{}.service'.format(str(livy_port)))
local("sudo systemctl daemon-reload")
- local("sudo systemctl enable livy-server-" + str(livy_port))
- local('sudo systemctl start livy-server-' + str(livy_port))
+ local("sudo systemctl enable livy-server-{}".format(str(livy_port)))
+ local('sudo systemctl start livy-server-{}'.format(str(livy_port)))
else:
template_file = "/tmp/dataengine-service_interpreter.json"
p_versions = ["2", python_version[:3]]
@@ -1627,7 +1627,7 @@
text = text.replace('PYTHONVER_SHORT', p_version[:1])
text = text.replace('ENDPOINTURL', endpoint_url)
text = text.replace('DATAENGINE-SERVICE_VERSION', emr_version)
- tmp_file = "/tmp/emr_spark_py" + p_version + "_interpreter.json"
+ tmp_file = "/tmp/emr_spark_py{}_interpreter.json".format(p_version)
fw = open(tmp_file, 'w')
fw.write(text)
fw.close()
@@ -1729,7 +1729,7 @@
sudo('sleep 5')
sudo('rm -rf /home/{}/.ensure_dir/dataengine_{}_interpreter_ensured'.format(os_user, cluster_name))
if exists('/home/{}/.ensure_dir/rstudio_dataengine_ensured'.format(os_user)):
- dlab.fab.remove_rstudio_dataengines_kernel(cluster_name, os_user)
+ dlab.fab.remove_rstudio_dataengines_kernel(os.environ['computational_name'], os_user)
sudo('rm -rf /opt/' + cluster_name + '/')
print("Notebook's {} kernels were removed".format(env.hosts))
except Exception as err:
diff --git a/infrastructure-provisioning/src/general/lib/azure/__init__.py b/infrastructure-provisioning/src/general/lib/azure/__init__.py
index e69de29..452a116 100644
--- a/infrastructure-provisioning/src/general/lib/azure/__init__.py
+++ b/infrastructure-provisioning/src/general/lib/azure/__init__.py
@@ -0,0 +1,17 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
index 71c5fa9..4176bbb 100644
--- a/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/actions_lib.py
@@ -120,6 +120,33 @@
file=sys.stdout)}))
traceback.print_exc(file=sys.stdout)
+ def create_virtual_network_peerings(self, resource_group_name,
+ virtual_network_name,
+ virtual_network_peering_name,
+ vnet_id
+ ):
+ try:
+ result = self.network_client.virtual_network_peerings.create_or_update(
+ resource_group_name,
+ virtual_network_name,
+ virtual_network_peering_name,
+ {
+ "allow_virtual_network_access": True,
+ "allow_forwarded_traffic": True,
+ "remote_virtual_network": {
+ "id": vnet_id
+ }
+ }
+ ).wait(60)
+ return result
+ except Exception as err:
+ logging.info(
+ "Unable to create Virtual Network peering: " + str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
+ append_result(str({"error": "Unable to create Virtual Network peering",
+ "error_message": str(err) + "\n Traceback: " + traceback.print_exc(
+ file=sys.stdout)}))
+ traceback.print_exc(file=sys.stdout)
+
def remove_vpc(self, resource_group_name, vpc_name):
try:
result = self.network_client.virtual_networks.delete(
@@ -966,7 +993,7 @@
sudo('sleep 5')
sudo('rm -rf /home/{}/.ensure_dir/dataengine_{}_interpreter_ensured'.format(os_user, cluster_name))
if exists('/home/{}/.ensure_dir/rstudio_dataengine_ensured'.format(os_user)):
- dlab.fab.remove_rstudio_dataengines_kernel(cluster_name, os_user)
+ dlab.fab.remove_rstudio_dataengines_kernel(os.environ['computational_name'], os_user)
sudo('rm -rf /opt/' + cluster_name + '/')
print("Notebook's {} kernels were removed".format(env.hosts))
except Exception as err:
diff --git a/infrastructure-provisioning/src/general/lib/azure/meta_lib.py b/infrastructure-provisioning/src/general/lib/azure/meta_lib.py
index bcb2512..5fa7aed 100644
--- a/infrastructure-provisioning/src/general/lib/azure/meta_lib.py
+++ b/infrastructure-provisioning/src/general/lib/azure/meta_lib.py
@@ -411,6 +411,21 @@
file=sys.stdout)}))
traceback.print_exc(file=sys.stdout)
+ def get_network_peering_status(self, resource_group_name,
+ virtual_network_name,
+ virtual_network_peering_name):
+ try:
+ result = self.network_client.virtual_network_peerings.get(resource_group_name,
+ virtual_network_name,
+ virtual_network_peering_name)
+ return result.peering_state
+ except Exception as err:
+ logging.info(
+ "Unable to get peering status: " + str(err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout))
+ append_result(str({"error": "Unable to get peering status", "error_message": str(
+ err) + "\n Traceback: " + traceback.print_exc(file=sys.stdout)}))
+ traceback.print_exc(file=sys.stdout)
+
def get_disk(self, resource_group_name, disk_name):
try:
result = self.compute_client.disks.get(resource_group_name, disk_name)
@@ -587,8 +602,9 @@
node_list = []
resource_group_name = os.environ['azure_resource_group_name']
for node in AzureMeta().compute_client.virtual_machines.list(resource_group_name):
- if cluster_name == node.tags["Name"]:
- node_list.append(node.name)
+ if "Name" in node.tags:
+ if cluster_name == node.tags["Name"]:
+ node_list.append(node.name)
result = len(node_list)
return result
except Exception as err:
diff --git a/infrastructure-provisioning/src/general/lib/gcp/__init__.py b/infrastructure-provisioning/src/general/lib/gcp/__init__.py
index e69de29..452a116 100644
--- a/infrastructure-provisioning/src/general/lib/gcp/__init__.py
+++ b/infrastructure-provisioning/src/general/lib/gcp/__init__.py
@@ -0,0 +1,17 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/lib/gcp/actions_lib.py b/infrastructure-provisioning/src/general/lib/gcp/actions_lib.py
index af7a8f0..6fa366e 100644
--- a/infrastructure-provisioning/src/general/lib/gcp/actions_lib.py
+++ b/infrastructure-provisioning/src/general/lib/gcp/actions_lib.py
@@ -964,7 +964,7 @@
local(""" sudo bash -c " sed -i 's/STANDALONE_SPARK_MASTER_HOST.*/STANDALONE_SPARK_MASTER_HOST={0}-m/g' {1}" """.format(args.cluster_name, spark_def_path))
local(""" sudo bash -c " sed -i 's|/hadoop_gcs_connector_metadata_cache|/tmp/hadoop_gcs_connector_metadata_cache|g' /opt/{0}/{1}/conf/core-site.xml" """.format(args.dataproc_version, args.cluster_name))
- def remove_kernels(self, notebook_name, dataproc_name, dataproc_version, ssh_user, key_path):
+ def remove_kernels(self, notebook_name, dataproc_name, dataproc_version, ssh_user, key_path, computational_name):
try:
notebook_ip = meta_lib.GCPMeta().get_private_ip_address(notebook_name)
env.hosts = "{}".format(notebook_ip)
@@ -1011,7 +1011,7 @@
sudo('sleep 5')
sudo('rm -rf /home/{}/.ensure_dir/dataengine-service_{}_interpreter_ensured'.format(ssh_user, dataproc_name))
if exists('/home/{}/.ensure_dir/rstudio_dataengine-service_ensured'.format(ssh_user)):
- dlab.fab.remove_rstudio_dataengines_kernel(dataproc_name, ssh_user)
+ dlab.fab.remove_rstudio_dataengines_kernel(computational_name, ssh_user)
sudo('rm -rf /opt/{0}/{1}/'.format(dataproc_version, dataproc_name))
print("Notebook's {} kernels were removed".format(env.hosts))
except Exception as err:
@@ -1316,7 +1316,7 @@
sudo('sleep 5')
sudo('rm -rf /home/{}/.ensure_dir/dataengine_{}_interpreter_ensured'.format(os_user, cluster_name))
if exists('/home/{}/.ensure_dir/rstudio_dataengine_ensured'.format(os_user)):
- dlab.fab.remove_rstudio_dataengines_kernel(cluster_name, os_user)
+ dlab.fab.remove_rstudio_dataengines_kernel(os.environ['computational_name'], os_user)
sudo('rm -rf /opt/' + cluster_name + '/')
print("Notebook's {} kernels were removed".format(env.hosts))
except Exception as err:
diff --git a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
index 26015d5..67cae89 100644
--- a/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
+++ b/infrastructure-provisioning/src/general/lib/gcp/meta_lib.py
@@ -502,6 +502,23 @@
data.append(host)
return data
+ def get_cluster(self, cluster_name):
+ try:
+ request = self.dataproc.projects().regions().clusters().get(projectId=self.project,
+ region=os.environ['gcp_region'],
+ clusterName=cluster_name)
+ result = request.execute()
+ return result
+ except Exception as err:
+ logging.info(
+ "Unable to get Dataproc: " + str(err) + "\n Traceback: " + traceback.print_exc(
+ file=sys.stdout))
+ append_result(str({"error": "Unable to get Dataproc",
+ "error_message": str(err) + "\n Traceback: " + traceback.print_exc(
+ file=sys.stdout)}))
+ traceback.print_exc(file=sys.stdout)
+ return ''
+
def get_dataproc_job_status(self, job_id):
request = self.dataproc.projects().regions().jobs().get(projectId=self.project,
region=os.environ['gcp_region'],
diff --git a/infrastructure-provisioning/src/general/lib/os/debian/common_lib.py b/infrastructure-provisioning/src/general/lib/os/debian/common_lib.py
index 8f7d7f4..4cd8675 100644
--- a/infrastructure-provisioning/src/general/lib/os/debian/common_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/debian/common_lib.py
@@ -36,9 +36,8 @@
sudo('unattended-upgrades -v')
sudo('export LC_ALL=C')
sudo('touch /home/{}/.ensure_dir/pkg_upgraded'.format(user))
- return True
except:
- return False
+ sys.exit(1)
def renew_gpg_key():
diff --git a/infrastructure-provisioning/src/general/lib/os/fab.py b/infrastructure-provisioning/src/general/lib/os/fab.py
index 88eac65..5be66d4 100644
--- a/infrastructure-provisioning/src/general/lib/os/fab.py
+++ b/infrastructure-provisioning/src/general/lib/os/fab.py
@@ -39,9 +39,8 @@
sudo('pip install -UI pip=={} --no-cache-dir'.format(os.environ['conf_pip_version']))
sudo('pip install -U {} --no-cache-dir'.format(requisites))
sudo('touch /home/{}/.ensure_dir/pip_path_added'.format(os.environ['conf_os_user']))
- return True
except:
- return False
+ sys.exit(1)
def dataengine_dir_prepare(cluster_dir):
@@ -148,17 +147,17 @@
sudo('pip2 install jupyter --no-cache-dir')
sudo('pip3.5 install notebook=={} --no-cache-dir'.format(jupyter_version))
sudo('pip3.5 install jupyter --no-cache-dir')
- sudo('rm -rf ' + jupyter_conf_file)
- run('jupyter notebook --generate-config --config ' + jupyter_conf_file)
+ sudo('rm -rf {}'.format(jupyter_conf_file))
+ run('jupyter notebook --generate-config --config {}'.format(jupyter_conf_file))
with cd('/home/{}'.format(os_user)):
run('mkdir -p ~/.jupyter/custom/')
run('echo "#notebook-container { width: auto; }" > ~/.jupyter/custom/custom.css')
- sudo('echo "c.NotebookApp.ip = \'*\'" >> ' + jupyter_conf_file)
+ sudo('echo "c.NotebookApp.ip = \'0.0.0.0\'" >> {}'.format(jupyter_conf_file))
sudo('echo "c.NotebookApp.base_url = \'/{0}/\'" >> {1}'.format(exploratory_name, jupyter_conf_file))
- sudo('echo c.NotebookApp.open_browser = False >> ' + jupyter_conf_file)
- sudo('echo \'c.NotebookApp.cookie_secret = b"' + id_generator() + '"\' >> ' + jupyter_conf_file)
- sudo('''echo "c.NotebookApp.token = u''" >> ''' + jupyter_conf_file)
- sudo('echo \'c.KernelSpecManager.ensure_native_kernel = False\' >> ' + jupyter_conf_file)
+ sudo('echo c.NotebookApp.open_browser = False >> {}'.format(jupyter_conf_file))
+ sudo('echo \'c.NotebookApp.cookie_secret = b"{0}"\' >> {1}'.format(id_generator(), jupyter_conf_file))
+ sudo('''echo "c.NotebookApp.token = u''" >> {}'''.format(jupyter_conf_file))
+ sudo('echo \'c.KernelSpecManager.ensure_native_kernel = False\' >> {}'.format(jupyter_conf_file))
put(templates_dir + 'jupyter-notebook.service', '/tmp/jupyter-notebook.service')
sudo("chmod 644 /tmp/jupyter-notebook.service")
if os.environ['application'] == 'tensor':
@@ -278,9 +277,9 @@
sudo('echo -e "\tKexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256" >> /etc/ssh/ssh_config')
sudo('echo -e "\tCiphers aes256-gcm@openssh.com,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr" >> /etc/ssh/ssh_config')
try:
- sudo('service ssh restart')
+ sudo('service ssh reload')
except:
- sudo('service sshd restart')
+ sudo('service sshd reload')
except Exception as err:
traceback.print_exc()
print('Failed to ensure ciphers: ', str(err))
@@ -680,3 +679,13 @@
except Exception as err:
print('Failed to update Zeppelin interpreters', str(err))
sys.exit(1)
+
+
+def update_hosts_file(os_user):
+ try:
+ if not exists('/home/{}/.ensure_dir/hosts_file_updated'.format(os_user)):
+ sudo('sed -i "s/^127.0.0.1 localhost/127.0.0.1 localhost localhost.localdomain/g" /etc/hosts')
+ sudo('touch /home/{}/.ensure_dir/hosts_file_updated'.format(os_user))
+ except Exception as err:
+ print('Failed to update hosts file', str(err))
+ sys.exit(1)
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/lib/os/redhat/common_lib.py b/infrastructure-provisioning/src/general/lib/os/redhat/common_lib.py
index 10dfa87..1184c42 100644
--- a/infrastructure-provisioning/src/general/lib/os/redhat/common_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/redhat/common_lib.py
@@ -20,6 +20,7 @@
from fabric.api import *
from fabric.contrib.files import exists
+import sys
def ensure_pkg(user, requisites='git vim gcc python-devel openssl-devel nmap libffi libffi-devel unzip'):
@@ -50,9 +51,8 @@
sudo('export LC_ALL=C')
sudo('yum -y install ' + requisites)
sudo('touch /home/{}/.ensure_dir/pkg_upgraded'.format(user))
- return True
except:
- return False
+ sys.exit(1)
def change_pkg_repos():
diff --git a/infrastructure-provisioning/src/general/lib/os/redhat/notebook_lib.py b/infrastructure-provisioning/src/general/lib/os/redhat/notebook_lib.py
index 5a84001..54d6fa6 100644
--- a/infrastructure-provisioning/src/general/lib/os/redhat/notebook_lib.py
+++ b/infrastructure-provisioning/src/general/lib/os/redhat/notebook_lib.py
@@ -334,7 +334,8 @@
def install_nodejs(os_user):
if not exists('/home/{}/.ensure_dir/nodejs_ensured'.format(os_user)):
- sudo('yum install -y npm nodejs')
+ sudo('curl -sL https://rpm.nodesource.com/setup_6.x | sudo -E bash -')
+ sudo('yum install -y nodejs')
sudo('touch /home/{}/.ensure_dir/nodejs_ensured'.format(os_user))
diff --git a/infrastructure-provisioning/src/general/scripts/aws/common_stop_notebook.py b/infrastructure-provisioning/src/general/scripts/aws/common_stop_notebook.py
index ebe9891..5986eb9 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/common_stop_notebook.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/common_stop_notebook.py
@@ -39,16 +39,20 @@
clusters_list = get_emr_list(nb_tag_value, 'Value')
if clusters_list:
for cluster_id in clusters_list:
+ computational_name = ''
client = boto3.client('emr')
cluster = client.describe_cluster(ClusterId=cluster_id)
cluster = cluster.get("Cluster")
emr_name = cluster.get('Name')
emr_version = cluster.get('ReleaseLabel')
+ for tag in cluster.get('Tags'):
+ if tag.get('Key') == 'ComputationalName':
+ computational_name = tag.get('Value')
s3_cleanup(bucket_name, emr_name, os.environ['edge_user_name'])
print("The bucket {} has been cleaned successfully".format(bucket_name))
terminate_emr(cluster_id)
print("The EMR cluster {} has been terminated successfully".format(emr_name))
- remove_kernels(emr_name, tag_name, nb_tag_value, ssh_user, key_path, emr_version)
+ remove_kernels(emr_name, tag_name, nb_tag_value, ssh_user, key_path, emr_version, computational_name)
print("{} kernels have been removed from notebook successfully".format(emr_name))
else:
print("There are no EMR clusters to terminate.")
diff --git a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_create.py b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_create.py
index 8e1fe4c..7ca4e46 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_create.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_create.py
@@ -198,6 +198,7 @@
str(err) + "\n Traceback: " +
traceback.print_exc(file=sys.stdout))
+
def get_instance_by_ip(ip):
try:
ec2 = boto3.resource('ec2')
diff --git a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_prepare.py
index c5ee58d..5012008 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_prepare.py
@@ -90,7 +90,8 @@
emr_conf['tags'] = 'Name={0}-{1}-des-{2}-{3},' \
'{0}-Tag={0}-{1}-des-{2}-{3},' \
'Notebook={4},' \
- 'State=not-configured'\
+ 'State=not-configured,' \
+ 'ComputationalName={3}' \
.format(emr_conf['service_base_name'],
os.environ['edge_user_name'],
emr_conf['exploratory_name'],
diff --git a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_terminate.py b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_terminate.py
index e45c4cb..561f6e3 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_terminate.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/dataengine-service_terminate.py
@@ -32,18 +32,22 @@
clusters_list = get_emr_list(emr_name, 'Value')
if clusters_list:
for cluster_id in clusters_list:
+ computational_name = ''
client = boto3.client('emr')
cluster = client.describe_cluster(ClusterId=cluster_id)
cluster = cluster.get("Cluster")
emr_name = cluster.get('Name')
emr_version = cluster.get('ReleaseLabel')
+ for tag in cluster.get('Tags'):
+ if tag.get('Key') == 'ComputationalName':
+ computational_name = tag.get('Value')
s3_cleanup(bucket_name, emr_name, os.environ['edge_user_name'])
print("The bucket {} has been cleaned successfully".format(bucket_name))
terminate_emr(cluster_id)
print("The EMR cluster {} has been terminated successfully".format(emr_name))
print("Removing EMR kernels from notebook")
remove_kernels(emr_name, tag_name, nb_tag_value, ssh_user, key_path,
- emr_version)
+ emr_version, computational_name)
else:
print("There are no EMR clusters to terminate.")
except:
diff --git a/infrastructure-provisioning/src/general/scripts/aws/edge_prepare.py b/infrastructure-provisioning/src/general/scripts/aws/edge_prepare.py
index ecd1333..a91a0de 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/edge_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/edge_prepare.py
@@ -252,7 +252,7 @@
"PrefixListIds": [],
"FromPort": 4040,
"IpRanges": [{"CidrIp": edge_conf['private_subnet_cidr']}],
- "ToPort": 4040, "IpProtocol": "tcp", "UserIdGroupPairs": []
+ "ToPort": 4045, "IpProtocol": "tcp", "UserIdGroupPairs": []
},
{
"PrefixListIds": [],
diff --git a/infrastructure-provisioning/src/general/scripts/aws/rstudio_dataengine-service_create_configs.py b/infrastructure-provisioning/src/general/scripts/aws/rstudio_dataengine-service_create_configs.py
index 56d2544..782a1f0 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/rstudio_dataengine-service_create_configs.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/rstudio_dataengine-service_create_configs.py
@@ -46,9 +46,9 @@
parser.add_argument('--os_user', type=str, default='')
args = parser.parse_args()
-emr_dir = '/opt/' + args.emr_version + '/jars/'
-spark_dir = '/opt/' + args.emr_version + '/' + args.cluster_name + '/spark/'
-yarn_dir = '/opt/' + args.emr_version + '/' + args.cluster_name + '/conf/'
+emr_dir = '/opt/{}/jars/'.format(args.emr_version)
+spark_dir = '/opt/{0}/{1}/spark/'.format(args.emr_version, args.cluster_name)
+yarn_dir = '/opt/{0}/{1}/conf/'.format(args.emr_version, args.cluster_name)
def configure_rstudio():
@@ -61,6 +61,9 @@
local('echo \'HADOOP_CONF_DIR="' + yarn_dir + '"\' >> /home/' + args.os_user + '/.Renviron')
local("sed -i 's/^master/#master/' /home/" + args.os_user + "/.Rprofile")
local('''R -e "source('/home/{}/.Rprofile')"'''.format(args.os_user))
+ #fix emr 5.19 problem with warnings in rstudio because of bug in AWS configuration
+ if args.emr_version == "emr-5.19.0":
+ local("sed -i '/DRFA/s/^/#/' " + spark_dir + "conf/log4j.properties")
local('touch /home/' + args.os_user + '/.ensure_dir/rstudio_dataengine-service_ensured')
except Exception as err:
print('Error: {0}'.format(err))
@@ -76,6 +79,9 @@
local('echo \'YARN_CONF_DIR="' + yarn_dir + '"\' >> /home/' + args.os_user + '/.Renviron')
local('echo \'HADOOP_CONF_DIR="' + yarn_dir + '"\' >> /home/' + args.os_user + '/.Renviron')
local('''R -e "source('/home/{}/.Rprofile')"'''.format(args.os_user))
+ #fix emr 5.19 problem with warnings in rstudio because of bug in AWS configuration
+ if args.emr_version == "emr-5.19.0":
+ local("sed -i '/DRFA/s/^/#/' " + spark_dir + "conf/log4j.properties")
except Exception as err:
print('Error: {0}'.format(err))
sys.exit(1)
diff --git a/infrastructure-provisioning/src/general/scripts/aws/ssn_configure.py b/infrastructure-provisioning/src/general/scripts/aws/ssn_configure.py
index 5f7f631..73dbe10 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/ssn_configure.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/ssn_configure.py
@@ -92,13 +92,6 @@
os.environ['aws_peering_id'] = get_peering_by_tag(tag_name, service_base_name)
pre_defined_peering = True
try:
- if os.environ['conf_duo_vpc_enable'] == 'true' and not os.environ['aws_subnet2_id']:
- raise KeyError
- except KeyError:
- tag = {"Key": tag2_name, "Value": "{}-subnet".format(service_base_name)}
- os.environ['aws_subnet2_id'] = get_subnet_by_tag(tag, True)
- pre_defined_subnet2 = True
- try:
if os.environ['aws_security_groups_ids'] == '':
raise KeyError
except KeyError:
@@ -296,7 +289,7 @@
if os.environ['conf_duo_vpc_enable'] == 'true':
secondary_parameters = {
"aws_notebook_vpc_id": os.environ['aws_vpc2_id'],
- "aws_notebook_subnet_id": os.environ['aws_subnet2_id'],
+ "aws_notebook_subnet_id": os.environ['aws_subnet_id'],
"aws_peering_id": os.environ['aws_peering_id']
}
else:
diff --git a/infrastructure-provisioning/src/general/scripts/aws/ssn_prepare.py b/infrastructure-provisioning/src/general/scripts/aws/ssn_prepare.py
index 3a1468b..e3ee361 100644
--- a/infrastructure-provisioning/src/general/scripts/aws/ssn_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/aws/ssn_prepare.py
@@ -36,7 +36,6 @@
pre_defined_subnet = False
pre_defined_sg = False
pre_defined_vpc2 = False
- pre_defined_subnet2 = False
try:
logging.info('[CREATE AWS CONFIG FILE]')
print('[CREATE AWS CONFIG FILE]')
@@ -161,48 +160,15 @@
sys.exit(1)
try:
- if os.environ['conf_duo_vpc_enable'] == 'true' and not os.environ['aws_subnet2_id']:
- raise KeyError
- except KeyError:
- try:
- pre_defined_subnet2 = True
- logging.info('[CREATE SECONDARY SUBNET]')
- print('[CREATE SECONDARY SUBNET]')
- params = "--vpc_id {} --username {} --infra_tag_name {} --infra_tag_value {} --prefix {} --ssn {}".format(os.environ['aws_vpc2_id'], 'ssn', tag2_name, service_base_name, '20', True)
- try:
- local("~/scripts/{}.py {}".format('common_create_subnet', params))
- except:
- traceback.print_exc()
- raise Exception
- with open('/tmp/ssn_subnet_id', 'r') as f:
- os.environ['aws_subnet2_id'] = f.read()
- create_product_tag(os.environ['aws_subnet2_id'])
- except Exception as err:
- print('Error: {0}'.format(err))
- append_result("Failed to create Subnet.", str(err))
- if pre_defined_vpc:
- remove_route_tables(tag_name, True)
- try:
- remove_subnets(service_base_name + "-subnet")
- except:
- print("Subnet hasn't been created.")
- remove_vpc(os.environ['aws_vpc_id'])
- if pre_defined_vpc2:
- remove_route_tables(tag2_name, True)
- remove_vpc(os.environ['aws_vpc2_id'])
- sys.exit(1)
-
- try:
- if os.environ['conf_duo_vpc_enable'] == 'true' and os.environ['aws_subnet2_id'] and os.environ['aws_subnet_id']:
+ if os.environ['conf_duo_vpc_enable'] == 'true' and os.environ['aws_vpc_id'] and os.environ['aws_vpc2_id']:
raise KeyError
except KeyError:
try:
logging.info('[CREATE PEERING CONNECTION]')
print('[CREATE PEERING CONNECTION]')
os.environ['aws_peering_id'] = create_peering_connection(os.environ['aws_vpc_id'], os.environ['aws_vpc2_id'], service_base_name)
- create_route_by_id(os.environ['aws_subnet2_id'], os.environ['aws_vpc2_id'], os.environ['aws_peering_id'], get_cidr_by_vpc(os.environ['aws_vpc_id']))
- create_route_by_id(os.environ['aws_subnet_id'], os.environ['aws_vpc_id'], os.environ['aws_peering_id'], get_cidr_by_vpc(os.environ['aws_vpc2_id']))
print('PEERING CONNECTION ID:' + os.environ['aws_peering_id'])
+ create_route_by_id(os.environ['aws_subnet_id'], os.environ['aws_vpc_id'], os.environ['aws_peering_id'], get_cidr_by_vpc(os.environ['aws_vpc2_id']))
except Exception as err:
print('Error: {0}'.format(err))
append_result("Failed to create peering connection.", str(err))
diff --git a/infrastructure-provisioning/src/general/scripts/azure/edge_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/edge_prepare.py
index 48f513d..3a71d8b 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/edge_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/edge_prepare.py
@@ -161,17 +161,6 @@
"direction": "Inbound"
},
{
- "name": "in-4",
- "protocol": "*",
- "source_port_range": "*",
- "destination_port_range": "*",
- "source_address_prefix": "*",
- "destination_address_prefix": "*",
- "access": "Deny",
- "priority": 200,
- "direction": "Inbound"
- },
- {
"name": "out-1",
"protocol": "Tcp",
"source_port_range": "*",
@@ -296,7 +285,7 @@
"name": "out-12",
"protocol": "Tcp",
"source_port_range": "*",
- "destination_port_range": "4040",
+ "destination_port_range": "4040-4045",
"source_address_prefix": "*",
"destination_address_prefix": edge_conf['private_subnet_cidr'],
"access": "Allow",
diff --git a/infrastructure-provisioning/src/general/scripts/azure/ssn_create_peering.py b/infrastructure-provisioning/src/general/scripts/azure/ssn_create_peering.py
new file mode 100644
index 0000000..47b9076
--- /dev/null
+++ b/infrastructure-provisioning/src/general/scripts/azure/ssn_create_peering.py
@@ -0,0 +1,62 @@
+#!/usr/bin/python
+
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
+import argparse
+from dlab.actions_lib import *
+from dlab.meta_lib import *
+import sys, time
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--source_resource_group_name', type=str, default='')
+parser.add_argument('--destination_resource_group_name', type=str, default='')
+parser.add_argument('--source_virtual_network_name', type=str, default='')
+parser.add_argument('--destination_virtual_network_name', type=str, default='')
+args = parser.parse_args()
+
+if __name__ == "__main__":
+ try:
+ source_virtual_network_peering_name = '{}_to_{}'.format(args.source_virtual_network_name, args.destination_virtual_network_name)
+ destination_virtual_network_peering_name = '{}_to_{}'.format(args.destination_virtual_network_name, args.source_virtual_network_name)
+
+ destination_vnet_id = AzureMeta().get_vpc(
+ args.destination_resource_group_name,
+ args.destination_virtual_network_name,
+ ).id
+
+ source_vnet_id = AzureMeta().get_vpc(
+ args.source_resource_group_name,
+ args.source_virtual_network_name,
+ ).id
+
+ print("Creating Virtual Network peering {} and {}".format(source_virtual_network_peering_name, destination_virtual_network_peering_name))
+ AzureActions().create_virtual_network_peerings(
+ args.source_resource_group_name,
+ args.source_virtual_network_name,
+ source_virtual_network_peering_name,
+ destination_vnet_id)
+ AzureActions().create_virtual_network_peerings(
+ args.destination_resource_group_name,
+ args.destination_virtual_network_name,
+ destination_virtual_network_peering_name,
+ source_vnet_id)
+ time.sleep(250)
+ except Exception as err:
+ print("Error creating vpc peering: " + str(err))
+ sys.exit(1)
diff --git a/infrastructure-provisioning/src/general/scripts/azure/ssn_prepare.py b/infrastructure-provisioning/src/general/scripts/azure/ssn_prepare.py
index e43d1ac..9e49443 100644
--- a/infrastructure-provisioning/src/general/scripts/azure/ssn_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/azure/ssn_prepare.py
@@ -40,11 +40,17 @@
print('[DERIVING NAMES]')
ssn_conf = dict()
+ # Verify vpc deployment
+ if os.environ['conf_network_type'] == 'private' and os.environ.get('azure_vpc_name') == None and os.environ.get('azure_source_vpc_name') == None:
+ raise Exception('Not possible to deploy private environment without predefined vpc or without source vpc')
+ if os.environ['conf_network_type'] == 'private' and os.environ.get('azure_resource_group_name') == None and os.environ.get('azure_source_resource_group_name') == None:
+ raise Exception('Not possible to deploy private environment without predefined resource_group_name or source_group_name')
# We need to cut service_base_name to 12 symbols do to the Azure Name length limitation
ssn_conf['service_base_name'] = os.environ['conf_service_base_name'] = replace_multi_symbols(
os.environ['conf_service_base_name'].replace('_', '-')[:12], '-', True)
# Check azure predefined resources
ssn_conf['resource_group_name'] = os.environ.get('azure_resource_group_name', ssn_conf['service_base_name'])
+ ssn_conf['source_resource_group_name'] = os.environ.get('azure_source_resource_group_name', ssn_conf['resource_group_name'])
ssn_conf['vpc_name'] = os.environ.get('azure_vpc_name', '{}-vpc'.format(ssn_conf['service_base_name']))
ssn_conf['subnet_name'] = os.environ.get('azure_subnet_name', '{}-ssn-subnet'.format(ssn_conf['service_base_name']))
ssn_conf['security_group_name'] = os.environ.get('azure_security_group_name', '{}-sg'.format(ssn_conf['service_base_name']))
@@ -120,31 +126,7 @@
print("Resources hasn't been removed: " + str(err))
append_result("Failed to create VPC. Exception: " + str(err))
sys.exit(1)
-
- try:
- if 'azure_subnet_name' in os.environ:
- logging.info('VPC predefined')
- print('Peering')
- else:
- logging.info('[CREATING Peering]')
- print("[CREATING Peering]")
- params = "--resource_group_name {} --vpc_name {} --region {} --vpc_cidr {} --subnet_name {} --prefix {}".\
- format(ssn_conf['resource_group_name'], ssn_conf['vpc_name'], ssn_conf['region'],
- ssn_conf['vpc_cidr'], ssn_conf['subnet_name'], ssn_conf['subnet_prefix'])
- local("~/scripts/{}.py {}".format('common_create_subnet', params))
- except Exception as err:
- traceback.print_exc()
- print('Error creating Subnet: ' + str(err))
- try:
- if 'azure_resource_group_name' not in os.environ:
- AzureActions().remove_resource_group(ssn_conf['service_base_name'], ssn_conf['region'])
- if 'azure_vpc_name' not in os.environ:
- AzureActions().remove_vpc(ssn_conf['resource_group_name'], ssn_conf['vpc_name'])
- except Exception as err:
- print("Resources hasn't been removed: " + str(err))
- append_result("Failed to create Subnet. Exception: " + str(err))
- sys.exit(1)
-
+
try:
if 'azure_subnet_name' in os.environ:
logging.info('Subnet predefined')
@@ -168,6 +150,27 @@
print("Resources hasn't been removed: " + str(err))
append_result("Failed to create Subnet. Exception: " + str(err))
sys.exit(1)
+
+ try:
+ if 'azure_vpc_name' not in os.environ and os.environ['conf_network_type'] == 'private':
+ logging.info('[CREATING VPC PEERING]')
+ print("[CREATING VPC PEERING]")
+ params = "--source_resource_group_name {} --destination_resource_group_name {} " \
+ "--source_virtual_network_name {} --destination_virtual_network_name {}".format(ssn_conf['source_resource_group_name'],
+ ssn_conf['resource_group_name'], os.environ['azure_source_vpc_name'], ssn_conf['vpc_name'])
+ local("~/scripts/{}.py {}".format('ssn_create_peering', params))
+ except Exception as err:
+ traceback.print_exc()
+ print('Error creating VPC peering: ' + str(err))
+ try:
+ if 'azure_resource_group_name' not in os.environ:
+ AzureActions().remove_resource_group(ssn_conf['service_base_name'], ssn_conf['region'])
+ if 'azure_vpc_name' not in os.environ:
+ AzureActions().remove_vpc(ssn_conf['resource_group_name'], ssn_conf['vpc_name'])
+ except Exception as err:
+ print("Resources hasn't been removed: " + str(err))
+ append_result("Failed to create VPC peering. Exception: " + str(err))
+ sys.exit(1)
try:
if 'azure_security_group_name' in os.environ:
diff --git a/infrastructure-provisioning/src/general/scripts/gcp/common_notebook_configure_dataengine-service.py b/infrastructure-provisioning/src/general/scripts/gcp/common_notebook_configure_dataengine-service.py
index 9868227..c91168e 100644
--- a/infrastructure-provisioning/src/general/scripts/gcp/common_notebook_configure_dataengine-service.py
+++ b/infrastructure-provisioning/src/general/scripts/gcp/common_notebook_configure_dataengine-service.py
@@ -59,7 +59,8 @@
"sbn": notebook_config['service_base_name'],
"user": notebook_config['edge_user_name'],
"notebook_name": os.environ['notebook_instance_name'],
- "product": "dlab"
+ "product": "dlab",
+ "computational_name": (os.environ['computational_name']).lower().replace('_', '-')
}
try:
diff --git a/infrastructure-provisioning/src/general/scripts/gcp/common_stop_notebook.py b/infrastructure-provisioning/src/general/scripts/gcp/common_stop_notebook.py
index d7f652a..297b4c6 100644
--- a/infrastructure-provisioning/src/general/scripts/gcp/common_stop_notebook.py
+++ b/infrastructure-provisioning/src/general/scripts/gcp/common_stop_notebook.py
@@ -39,12 +39,15 @@
clusters_list = meta_lib.GCPMeta().get_dataproc_list(labels)
if clusters_list:
for cluster_name in clusters_list:
+ computational_name = meta_lib.GCPMeta().get_cluster(cluster_name).get('labels').get(
+ 'computational_name')
cluster = meta_lib.GCPMeta().get_list_cluster_statuses([cluster_name])
actions_lib.GCPActions().bucket_cleanup(bucket_name, user_name, cluster_name)
print('The bucket {} has been cleaned successfully'.format(bucket_name))
actions_lib.GCPActions().delete_dataproc_cluster(cluster_name, region)
print('The Dataproc cluster {} has been terminated successfully'.format(cluster_name))
- actions_lib.GCPActions().remove_kernels(instance_name, cluster_name, cluster[0]['version'], ssh_user, key_path)
+ actions_lib.GCPActions().remove_kernels(instance_name, cluster_name, cluster[0]['version'], ssh_user,
+ key_path, computational_name)
else:
print("There are no Dataproc clusters to terminate.")
except Exception as err:
diff --git a/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_prepare.py b/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_prepare.py
index bf366bd..197920b 100644
--- a/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_prepare.py
@@ -73,7 +73,8 @@
"sbn": dataproc_conf['service_base_name'],
"user": dataproc_conf['edge_user_name'],
"notebook_name": os.environ['notebook_instance_name'],
- "product": "dlab"
+ "product": "dlab",
+ "computational_name": dataproc_conf['computational_name']
}
dataproc_conf['dataproc_service_account_name'] = '{0}-{1}-ps'.format(dataproc_conf['service_base_name'],
dataproc_conf['edge_user_name'])
diff --git a/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_terminate.py b/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_terminate.py
index 9022892..2f40406 100644
--- a/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_terminate.py
+++ b/infrastructure-provisioning/src/general/scripts/gcp/dataengine-service_terminate.py
@@ -32,11 +32,13 @@
try:
cluster = meta_lib.GCPMeta().get_list_cluster_statuses([dataproc_name])
if cluster[0]['status'] == 'running':
+ computational_name = meta_lib.GCPMeta().get_cluster(dataproc_name).get('labels').get('computational_name')
actions_lib.GCPActions().bucket_cleanup(bucket_name, os.environ['edge_user_name'], dataproc_name)
print('The bucket {} has been cleaned successfully'.format(bucket_name))
actions_lib.GCPActions().delete_dataproc_cluster(dataproc_name, os.environ['gcp_region'])
print('The Dataproc cluster {} has been terminated successfully'.format(dataproc_name))
- actions_lib.GCPActions().remove_kernels(notebook_name, dataproc_name, cluster[0]['version'], ssh_user, key_path)
+ actions_lib.GCPActions().remove_kernels(notebook_name, dataproc_name, cluster[0]['version'], ssh_user,
+ key_path, computational_name)
else:
print("There are no Dataproc clusters to terminate.")
except Exception as err:
diff --git a/infrastructure-provisioning/src/general/scripts/gcp/edge_prepare.py b/infrastructure-provisioning/src/general/scripts/gcp/edge_prepare.py
index b7202a9..614e3b3 100644
--- a/infrastructure-provisioning/src/general/scripts/gcp/edge_prepare.py
+++ b/infrastructure-provisioning/src/general/scripts/gcp/edge_prepare.py
@@ -235,7 +235,7 @@
{
'IPProtocol': 'tcp',
'ports': ['22', '8888', '8080', '8787', '6006', '20888', '8088', '18080', '50070', '8085', '8081',
- '4040']
+ '4040-4045']
}
]
egress_rule['allowed'] = rules
diff --git a/infrastructure-provisioning/src/general/scripts/os/zeppelin_install_dataengine_kernels.py b/infrastructure-provisioning/src/general/scripts/os/zeppelin_install_dataengine_kernels.py
index f40e450..aab5f58 100644
--- a/infrastructure-provisioning/src/general/scripts/os/zeppelin_install_dataengine_kernels.py
+++ b/infrastructure-provisioning/src/general/scripts/os/zeppelin_install_dataengine_kernels.py
@@ -49,6 +49,7 @@
put(templates_dir + 'notebook_spark-defaults_local.conf', '/tmp/{}/notebook_spark-defaults_local.conf'.format(args.cluster_name))
spark_master_ip = args.spark_master.split('//')[1].split(':')[0]
spark_memory = get_spark_memory(True, args.os_user, spark_master_ip, keyfile)
+ run('sed -i "s|EXECUTOR_MEMORY|{}m|g " /tmp/{}/dataengine_interpreter.json'.format(spark_memory, args.cluster_name))
run('echo "spark.executor.memory {0}m" >> /tmp/{1}/notebook_spark-defaults_local.conf'.format(spark_memory, args.cluster_name))
if not exists('/usr/local/bin/zeppelin_dataengine_create_configs.py'):
put(scripts_dir + 'zeppelin_dataengine_create_configs.py', '/usr/local/bin/zeppelin_dataengine_create_configs.py', use_sudo=True)
diff --git a/infrastructure-provisioning/src/general/templates/aws/dataengine-service_interpreter_livy.json b/infrastructure-provisioning/src/general/templates/aws/dataengine-service_interpreter_livy.json
index ff91326..b4bece1 100644
--- a/infrastructure-provisioning/src/general/templates/aws/dataengine-service_interpreter_livy.json
+++ b/infrastructure-provisioning/src/general/templates/aws/dataengine-service_interpreter_livy.json
@@ -2,25 +2,111 @@
"name": "CLUSTER_NAME",
"group": "livy",
"properties": {
- "zeppelin.livy.concurrentSQL": "false",
- "livy.spark.home":"SPARK_HOME",
- "livy.spark.hadoop.fs.s3a.endpoint": "ENDPOINTURL",
- "zeppelin.interpreter.localRepo": "/opt/zeppelin/local-repo/2CAC665T6",
- "zeppelin.livy.url": "http://localhost:LIVY_PORT",
- "livy.spark.hadoop.fs.s3a.buffer.dir": "/tmp",
- "zeppelin.spark.useHiveContext": "true",
- "livy.spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "livy.spark.hadoop.fs.s3a.fast.upload": "true",
- "zeppelin.R.image.width": "100%",
- "zeppelin.R.render.options": "out.format \u003d \u0027html\u0027, comment \u003d NA, echo \u003d FALSE, results \u003d \u0027asis\u0027, message \u003d F, warning \u003d F",
- "zeppelin.R.cmd": "R",
- "zeppelin.R.knitr": "true"
+ "zeppelin.livy.concurrentSQL": {
+ "propertyName": "zeppelin.livy.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "livy.spark.home": {
+ "propertyName": "livy.spark.home",
+ "value": "SPARK_HOME",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "livy.spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.interpreter.localRepo": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/zeppelin/local-repo/2CAC665T6",
+ "type": "string"
+ },
+ "zeppelin.livy.url": {
+ "propertyName": "zeppelin.livy.url",
+ "value": "http://localhost:LIVY_PORT",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.buffer.dir": {
+ "propertyName": "livy.spark.hadoop.fs.s3a.buffer.dir",
+ "value": "/tmp",
+ "type": "string"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.app.name": {
+ "propertyName": "livy.spark.app.name",
+ "value": "Zeppelin",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "type": "number"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit": {
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.localrepo": {
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.fast.upload": {
+ "propertyName": "livy.spark.hadoop.fs.s3a.fast.upload",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.R.image.width": {
+ "envName": "ZEPPELIN_R_IMAGE_WIDTH",
+ "propertyName": "zeppelin.R.image.width",
+ "value": "100%",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.render.options": {
+ "envName": "ZEPPELIN_R_RENDER_OPTIONS",
+ "propertyName": "zeppelin.R.render.options",
+ "value": "out.format \u003d \u0027html\u0027, comment \u003d NA, echo \u003d FALSE, results \u003d \u0027asis\u0027, message \u003d F, warning \u003d F",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.cmd": {
+ "envName": "ZEPPELIN_R_CMD",
+ "propertyName": "zeppelin.R.cmd",
+ "value": "R",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.knitr": {
+ "envName": "ZEPPELIN_R_KNITR",
+ "propertyName": "zeppelin.R.knitr",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/aws/dataengine-service_interpreter_spark.json b/infrastructure-provisioning/src/general/templates/aws/dataengine-service_interpreter_spark.json
index 4bf1379..4ba5737 100644
--- a/infrastructure-provisioning/src/general/templates/aws/dataengine-service_interpreter_spark.json
+++ b/infrastructure-provisioning/src/general/templates/aws/dataengine-service_interpreter_spark.json
@@ -2,25 +2,139 @@
"name": "CLUSTERNAME_pyPYTHONVER_SHORT",
"group": "spark",
"properties": {
- "args": "",
- "zeppelin.spark.useHiveContext": "true",
- "spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "master": "yarn",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.pyspark.python": "pythonPYTHONVERSION",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "spark.home": "SPARK_HOME",
- "spark.hadoop.fs.s3a.fast.upload": "true",
- "spark.hadoop.fs.s3a.endpoint": "ENDPOINTURL",
- "zeppelin.R.image.width": "100%",
- "zeppelin.R.render.options": "out.format \u003d \u0027html\u0027, comment \u003d NA, echo \u003d FALSE, results \u003d \u0027asis\u0027, message \u003d F, warning \u003d F",
- "zeppelin.R.cmd": "R",
- "zeppelin.R.knitr": "true"
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "envName": null,
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "yarn",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "spark.submit.deployMode":{
+ "envName": "spark submit deploy mode",
+ "propertyName": "spark.submit.deployMode",
+ "value": "client",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "pythonPYTHONVERSION",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "spark.home": {
+ "envName": "Spark Home",
+ "propertyName": "spark.home",
+ "value": "SPARK_HOME",
+ "description": "",
+ "type": "string"
+ },
+ "spark.hadoop.fs.s3a.fast.upload": {
+ "envName": "SPARK_HADOOP_FS_S3A_FAST_UPLOAD",
+ "propertyName": "spark.hadoop.fs.s3a.fast.upload",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.image.width": {
+ "envName": "ZEPPELIN_R_IMAGE_WIDTH",
+ "propertyName": "zeppelin.R.image.width",
+ "value": "100%",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.render.options": {
+ "envName": "ZEPPELIN_R_RENDER_OPTIONS",
+ "propertyName": "zeppelin.R.render.options",
+ "value": "out.format \u003d \u0027html\u0027, comment \u003d NA, echo \u003d FALSE, results \u003d \u0027asis\u0027, message \u003d F, warning \u003d F",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.cmd": {
+ "envName": "ZEPPELIN_R_CMD",
+ "propertyName": "zeppelin.R.cmd",
+ "value": "R",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.knitr": {
+ "envName": "ZEPPELIN_R_KNITR",
+ "propertyName": "zeppelin.R.knitr",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ }
},
"dependencies": [],
"option": {
diff --git a/infrastructure-provisioning/src/general/templates/aws/interpreter_livy.json b/infrastructure-provisioning/src/general/templates/aws/interpreter_livy.json
index 031e9d1..4fa46d6 100644
--- a/infrastructure-provisioning/src/general/templates/aws/interpreter_livy.json
+++ b/infrastructure-provisioning/src/general/templates/aws/interpreter_livy.json
@@ -5,22 +5,97 @@
"name": "local_interpreter",
"group": "livy",
"properties": {
- "zeppelin.livy.concurrentSQL": "false",
- "livy.spark.master": "local[*]",
- "livy.spark.home":"/opt/spark/",
- "livy.spark.hadoop.fs.s3a.endpoint": "ENDPOINTURL",
- "zeppelin.interpreter.localRepo": "/opt/zeppelin/local-repo/2CAC665T6",
- "zeppelin.livy.url": "http://localhost:LIVY_PORT",
- "livy.spark.hadoop.fs.s3a.buffer.dir": "/tmp",
- "zeppelin.spark.useHiveContext": "true",
- "livy.spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "livy.spark.hadoop.fs.s3a.fast.upload": "true"
+ "zeppelin.livy.concurrentSQL": {
+ "propertyName": "zeppelin.livy.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "livy.spark.master":{
+ "envName": "Master",
+ "propertyName": "master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "livy.spark.home": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/spark/",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "livy.spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.interpreter.localRepo": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/zeppelin/local-repo/2CAC665T6",
+ "type": "string"
+ },
+ "zeppelin.livy.url": {
+ "propertyName": "zeppelin.livy.url",
+ "value": "http://localhost:LIVY_PORT",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.buffer.dir": {
+ "propertyName": "livy.spark.hadoop.fs.s3a.buffer.dir",
+ "value": "/tmp",
+ "type": "string"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.app.name": {
+ "propertyName": "livy.spark.app.name",
+ "value": "Zeppelin",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "type": "number"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit": {
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.localrepo": {
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.fast.upload": {
+ "propertyName": "livy.spark.hadoop.fs.s3a.fast.upload",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "livy.spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/aws/interpreter_spark.json b/infrastructure-provisioning/src/general/templates/aws/interpreter_spark.json
index 9fb328b..0d354cd 100644
--- a/infrastructure-provisioning/src/general/templates/aws/interpreter_spark.json
+++ b/infrastructure-provisioning/src/general/templates/aws/interpreter_spark.json
@@ -5,21 +5,96 @@
"name": "local_interpreter_python2",
"group": "spark",
"properties": {
- "spark.executor.memory": "",
- "args": "",
- "zeppelin.spark.printREPLOutput": "true",
- "spark.cores.max": "",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.useHiveContext": "true",
- "zeppelin.pyspark.python": "python",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.spark.maxResult": "1000",
- "master": "local[*]",
- "spark.app.name": "Zeppelin",
- "spark.hadoop.fs.s3a.endpoint": "ENDPOINTURL"
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "python",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
@@ -44,21 +119,96 @@
"name": "local_interpreter_python3",
"group": "spark",
"properties": {
- "spark.executor.memory": "",
- "args": "",
- "zeppelin.spark.printREPLOutput": "true",
- "spark.cores.max": "",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.useHiveContext": "true",
- "zeppelin.pyspark.python": "python3.5",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.spark.maxResult": "1000",
- "master": "local[*]",
- "spark.app.name": "Zeppelin",
- "spark.hadoop.fs.s3a.endpoint": "ENDPOINTURL"
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "python3.5",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/aws/jenkins_jobs/reupload_ssh_key/config.xml b/infrastructure-provisioning/src/general/templates/aws/jenkins_jobs/reupload_ssh_key/config.xml
index 3608edf..cdf3db0 100644
--- a/infrastructure-provisioning/src/general/templates/aws/jenkins_jobs/reupload_ssh_key/config.xml
+++ b/infrastructure-provisioning/src/general/templates/aws/jenkins_jobs/reupload_ssh_key/config.xml
@@ -1,4 +1,21 @@
<?xml version='1.1' encoding='UTF-8'?>
+<!--
+
+Copyright (c) 2016, EPAM SYSTEMS INC
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
<project>
<actions/>
<description>Reupload user key</description>
diff --git a/infrastructure-provisioning/src/general/templates/azure/interpreter_livy.json b/infrastructure-provisioning/src/general/templates/azure/interpreter_livy.json
index 031e9d1..4fa46d6 100644
--- a/infrastructure-provisioning/src/general/templates/azure/interpreter_livy.json
+++ b/infrastructure-provisioning/src/general/templates/azure/interpreter_livy.json
@@ -5,22 +5,97 @@
"name": "local_interpreter",
"group": "livy",
"properties": {
- "zeppelin.livy.concurrentSQL": "false",
- "livy.spark.master": "local[*]",
- "livy.spark.home":"/opt/spark/",
- "livy.spark.hadoop.fs.s3a.endpoint": "ENDPOINTURL",
- "zeppelin.interpreter.localRepo": "/opt/zeppelin/local-repo/2CAC665T6",
- "zeppelin.livy.url": "http://localhost:LIVY_PORT",
- "livy.spark.hadoop.fs.s3a.buffer.dir": "/tmp",
- "zeppelin.spark.useHiveContext": "true",
- "livy.spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "livy.spark.hadoop.fs.s3a.fast.upload": "true"
+ "zeppelin.livy.concurrentSQL": {
+ "propertyName": "zeppelin.livy.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "livy.spark.master":{
+ "envName": "Master",
+ "propertyName": "master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "livy.spark.home": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/spark/",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "livy.spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.interpreter.localRepo": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/zeppelin/local-repo/2CAC665T6",
+ "type": "string"
+ },
+ "zeppelin.livy.url": {
+ "propertyName": "zeppelin.livy.url",
+ "value": "http://localhost:LIVY_PORT",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.buffer.dir": {
+ "propertyName": "livy.spark.hadoop.fs.s3a.buffer.dir",
+ "value": "/tmp",
+ "type": "string"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.app.name": {
+ "propertyName": "livy.spark.app.name",
+ "value": "Zeppelin",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "type": "number"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit": {
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.localrepo": {
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.fast.upload": {
+ "propertyName": "livy.spark.hadoop.fs.s3a.fast.upload",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "livy.spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/azure/interpreter_spark.json b/infrastructure-provisioning/src/general/templates/azure/interpreter_spark.json
index 9fb328b..0d354cd 100644
--- a/infrastructure-provisioning/src/general/templates/azure/interpreter_spark.json
+++ b/infrastructure-provisioning/src/general/templates/azure/interpreter_spark.json
@@ -5,21 +5,96 @@
"name": "local_interpreter_python2",
"group": "spark",
"properties": {
- "spark.executor.memory": "",
- "args": "",
- "zeppelin.spark.printREPLOutput": "true",
- "spark.cores.max": "",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.useHiveContext": "true",
- "zeppelin.pyspark.python": "python",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.spark.maxResult": "1000",
- "master": "local[*]",
- "spark.app.name": "Zeppelin",
- "spark.hadoop.fs.s3a.endpoint": "ENDPOINTURL"
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "python",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
@@ -44,21 +119,96 @@
"name": "local_interpreter_python3",
"group": "spark",
"properties": {
- "spark.executor.memory": "",
- "args": "",
- "zeppelin.spark.printREPLOutput": "true",
- "spark.cores.max": "",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.useHiveContext": "true",
- "zeppelin.pyspark.python": "python3.5",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.spark.maxResult": "1000",
- "master": "local[*]",
- "spark.app.name": "Zeppelin",
- "spark.hadoop.fs.s3a.endpoint": "ENDPOINTURL"
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "python3.5",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/gcp/dataengine-service_interpreter_livy.json b/infrastructure-provisioning/src/general/templates/gcp/dataengine-service_interpreter_livy.json
index 75eacfa..f2386b6 100644
--- a/infrastructure-provisioning/src/general/templates/gcp/dataengine-service_interpreter_livy.json
+++ b/infrastructure-provisioning/src/general/templates/gcp/dataengine-service_interpreter_livy.json
@@ -2,22 +2,94 @@
"name": "CLUSTER_NAME",
"group": "livy",
"properties": {
- "zeppelin.livy.concurrentSQL": "false",
- "livy.spark.home":"SPARK_HOME",
- "zeppelin.interpreter.localRepo": "/opt/zeppelin/local-repo/2CAC665T6",
- "zeppelin.livy.url": "http://localhost:LIVY_PORT",
- "zeppelin.spark.useHiveContext": "true",
- "livy.spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "zeppelin.R.image.width": "100%",
- "zeppelin.R.render.options": "out.format \u003d \u0027html\u0027, comment \u003d NA, echo \u003d FALSE, results \u003d \u0027asis\u0027, message \u003d F, warning \u003d F",
- "zeppelin.R.cmd": "R",
- "zeppelin.R.knitr": "true"
+ "zeppelin.livy.concurrentSQL": {
+ "propertyName": "zeppelin.livy.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "livy.spark.home": {
+ "propertyName": "livy.spark.home",
+ "value": "SPARK_HOME",
+ "type": "string"
+ },
+ "zeppelin.interpreter.localRepo": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/zeppelin/local-repo/2CAC665T6",
+ "type": "string"
+ },
+ "zeppelin.livy.url": {
+ "propertyName": "zeppelin.livy.url",
+ "value": "http://localhost:LIVY_PORT",
+ "type": "string"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.app.name": {
+ "propertyName": "livy.spark.app.name",
+ "value": "Zeppelin",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "type": "number"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit": {
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.localrepo": {
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "type": "string"
+ },
+ "zeppelin.R.image.width": {
+ "envName": "ZEPPELIN_R_IMAGE_WIDTH",
+ "propertyName": "zeppelin.R.image.width",
+ "value": "100%",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.render.options": {
+ "envName": "ZEPPELIN_R_RENDER_OPTIONS",
+ "propertyName": "zeppelin.R.render.options",
+ "value": "out.format \u003d \u0027html\u0027, comment \u003d NA, echo \u003d FALSE, results \u003d \u0027asis\u0027, message \u003d F, warning \u003d F",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.cmd": {
+ "envName": "ZEPPELIN_R_CMD",
+ "propertyName": "zeppelin.R.cmd",
+ "value": "R",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.knitr": {
+ "envName": "ZEPPELIN_R_KNITR",
+ "propertyName": "zeppelin.R.knitr",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/gcp/dataengine-service_interpreter_spark.json b/infrastructure-provisioning/src/general/templates/gcp/dataengine-service_interpreter_spark.json
index e79b21b..6f4cf31 100644
--- a/infrastructure-provisioning/src/general/templates/gcp/dataengine-service_interpreter_spark.json
+++ b/infrastructure-provisioning/src/general/templates/gcp/dataengine-service_interpreter_spark.json
@@ -2,23 +2,125 @@
"name": "CLUSTERNAME_pyPYTHONVER_SHORT",
"group": "spark",
"properties": {
- "args": "",
- "zeppelin.spark.useHiveContext": "true",
- "spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "master": "yarn",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.pyspark.python": "pythonPYTHONVERSION",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "spark.home": "SPARK_HOME",
- "zeppelin.R.image.width": "100%",
- "zeppelin.R.render.options": "out.format \u003d \u0027html\u0027, comment \u003d NA, echo \u003d FALSE, results \u003d \u0027asis\u0027, message \u003d F, warning \u003d F",
- "zeppelin.R.cmd": "R",
- "zeppelin.R.knitr": "true"
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "envName": null,
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "yarn",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "spark.submit.deployMode":{
+ "envName": "spark submit deploy mode",
+ "propertyName": "spark.submit.deployMode",
+ "value": "client",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "pythonPYTHONVERSION",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "spark.home": {
+ "envName": "Spark Home",
+ "propertyName": "spark.home",
+ "value": "SPARK_HOME",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.image.width": {
+ "envName": "ZEPPELIN_R_IMAGE_WIDTH",
+ "propertyName": "zeppelin.R.image.width",
+ "value": "100%",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.render.options": {
+ "envName": "ZEPPELIN_R_RENDER_OPTIONS",
+ "propertyName": "zeppelin.R.render.options",
+ "value": "out.format \u003d \u0027html\u0027, comment \u003d NA, echo \u003d FALSE, results \u003d \u0027asis\u0027, message \u003d F, warning \u003d F",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.cmd": {
+ "envName": "ZEPPELIN_R_CMD",
+ "propertyName": "zeppelin.R.cmd",
+ "value": "R",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.R.knitr": {
+ "envName": "ZEPPELIN_R_KNITR",
+ "propertyName": "zeppelin.R.knitr",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ }
},
"dependencies": [],
"option": {
diff --git a/infrastructure-provisioning/src/general/templates/gcp/interpreter_livy.json b/infrastructure-provisioning/src/general/templates/gcp/interpreter_livy.json
index 518f1df..d027e3d 100644
--- a/infrastructure-provisioning/src/general/templates/gcp/interpreter_livy.json
+++ b/infrastructure-provisioning/src/general/templates/gcp/interpreter_livy.json
@@ -5,19 +5,80 @@
"name": "local_interpreter",
"group": "livy",
"properties": {
- "zeppelin.livy.concurrentSQL": "false",
- "livy.spark.master": "local[*]",
- "livy.spark.home":"/opt/spark/",
- "zeppelin.interpreter.localRepo": "/opt/zeppelin/local-repo/2CAC665T6",
- "zeppelin.livy.url": "http://localhost:LIVY_PORT",
- "zeppelin.spark.useHiveContext": "true",
- "livy.spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;"
+ "zeppelin.livy.concurrentSQL": {
+ "propertyName": "zeppelin.livy.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "livy.spark.master":{
+ "envName": "Master",
+ "propertyName": "master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "livy.spark.home": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/spark/",
+ "type": "string"
+ },
+ "zeppelin.interpreter.localRepo": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/zeppelin/local-repo/2CAC665T6",
+ "type": "string"
+ },
+ "zeppelin.livy.url": {
+ "propertyName": "zeppelin.livy.url",
+ "value": "http://localhost:LIVY_PORT",
+ "type": "string"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.app.name": {
+ "propertyName": "livy.spark.app.name",
+ "value": "Zeppelin",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "type": "number"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit": {
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.localrepo": {
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "type": "string"
+ },
+ "livy.spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "livy.spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/gcp/interpreter_spark.json b/infrastructure-provisioning/src/general/templates/gcp/interpreter_spark.json
index b25752a..0d342c0 100644
--- a/infrastructure-provisioning/src/general/templates/gcp/interpreter_spark.json
+++ b/infrastructure-provisioning/src/general/templates/gcp/interpreter_spark.json
@@ -5,20 +5,89 @@
"name": "local_interpreter_python2",
"group": "spark",
"properties": {
- "spark.executor.memory": "",
- "args": "",
- "zeppelin.spark.printREPLOutput": "true",
- "spark.cores.max": "",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.useHiveContext": "true",
- "zeppelin.pyspark.python": "python",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.spark.maxResult": "1000",
- "master": "local[*]",
- "spark.app.name": "Zeppelin"
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "python",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
@@ -43,20 +112,96 @@
"name": "local_interpreter_python3",
"group": "spark",
"properties": {
- "spark.executor.memory": "",
- "args": "",
- "zeppelin.spark.printREPLOutput": "true",
- "spark.cores.max": "",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.useHiveContext": "true",
- "zeppelin.pyspark.python": "python3.5",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.spark.maxResult": "1000",
- "master": "local[*]",
- "spark.app.name": "Zeppelin"
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "python3.5",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "local[*]",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "spark.hadoop.fs.s3a.endpoint": {
+ "envName": "SPARK_HADOOP_FS_S3A_ENDPOINT",
+ "propertyName": "spark.hadoop.fs.s3a.endpoint",
+ "value": "ENDPOINTURL",
+ "description": "",
+ "type": "string"
+ },
+ "spark.driver.memory": {
+ "envName": "MEMORY_DRIVER",
+ "propertyName": "spark.driver.memory",
+ "value": "DRIVER_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/os/dataengine_interpreter_livy.json b/infrastructure-provisioning/src/general/templates/os/dataengine_interpreter_livy.json
index 543298e..1eebe07 100644
--- a/infrastructure-provisioning/src/general/templates/os/dataengine_interpreter_livy.json
+++ b/infrastructure-provisioning/src/general/templates/os/dataengine_interpreter_livy.json
@@ -2,20 +2,83 @@
"name": "CLUSTER_NAME",
"group": "livy",
"properties": {
- "zeppelin.livy.concurrentSQL": "false",
- "livy.spark.home":"SPARK_HOME",
- "zeppelin.interpreter.localRepo": "/opt/zeppelin/local-repo/2CAC665T6",
- "zeppelin.livy.url": "http://localhost:LIVY_PORT",
- "livy.spark.hadoop.fs.s3a.buffer.dir": "/tmp",
- "zeppelin.spark.useHiveContext": "true",
- "livy.spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "livy.spark.hadoop.fs.s3a.fast.upload": "true"
+ "zeppelin.livy.concurrentSQL": {
+ "propertyName": "zeppelin.livy.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "livy.spark.home": {
+ "propertyName": "livy.spark.home",
+ "value": "SPARK_HOME",
+ "type": "string"
+ },
+ "zeppelin.interpreter.localRepo": {
+ "propertyName": "livy.spark.home",
+ "value": "/opt/zeppelin/local-repo/2CAC665T6",
+ "type": "string"
+ },
+ "zeppelin.livy.url": {
+ "propertyName": "zeppelin.livy.url",
+ "value": "http://localhost:LIVY_PORT",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.buffer.dir": {
+ "propertyName": "livy.spark.hadoop.fs.s3a.buffer.dir",
+ "value": "/tmp",
+ "type": "string"
+ },
+ "zeppelin.spark.useHiveContext": {
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.app.name": {
+ "propertyName": "livy.spark.app.name",
+ "value": "Zeppelin",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "type": "number"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit": {
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "zeppelin.dep.localrepo": {
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "type": "string"
+ },
+ "livy.spark.hadoop.fs.s3a.fast.upload": {
+ "propertyName": "livy.spark.hadoop.fs.s3a.fast.upload",
+ "value": "true",
+ "type": "checkbox"
+ },
+ "livy.spark.executor.memory": {
+ "envName": "MEMORY_PER_EXECUTOR",
+ "propertyName": "livy.spark.executor.memory",
+ "value": "EXECUTOR_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/os/dataengine_interpreter_spark.json b/infrastructure-provisioning/src/general/templates/os/dataengine_interpreter_spark.json
index c09b9f4..f3bcebb 100644
--- a/infrastructure-provisioning/src/general/templates/os/dataengine_interpreter_spark.json
+++ b/infrastructure-provisioning/src/general/templates/os/dataengine_interpreter_spark.json
@@ -2,20 +2,104 @@
"name": "CLUSTERNAME_pyPYTHONVER_SHORT",
"group": "spark",
"properties": {
- "args": "",
- "zeppelin.spark.useHiveContext": "true",
- "spark.app.name": "Zeppelin",
- "zeppelin.spark.printREPLOutput": "true",
- "zeppelin.spark.maxResult": "1000",
- "master": "MASTER",
- "zeppelin.spark.concurrentSQL": "false",
- "zeppelin.spark.sql.stacktrace": "false",
- "zeppelin.spark.importImplicit": "true",
- "zeppelin.pyspark.python": "pythonPYTHONVERSION",
- "zeppelin.dep.localrepo": "local-repo",
- "zeppelin.dep.additionalRemoteRepository": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
- "spark.home": "SPARK_HOME",
- "spark.hadoop.fs.s3a.fast.upload": "true"
+ "zeppelin.spark.useHiveContext": {
+ "envName": "ZEPPELIN_SPARK_USEHIVECONTEXT",
+ "propertyName": "zeppelin.spark.useHiveContext",
+ "value": "true",
+ "description": "Use HiveContext instead of SQLContext if it is true.",
+ "type": "checkbox"
+ },
+ "spark.app.name": {
+ "envName": "SPARK_APP_NAME",
+ "propertyName": "spark.app.name",
+ "value": "Zeppelin",
+ "description": "The name of spark application.",
+ "type": "string"
+ },
+ "zeppelin.spark.printREPLOutput": {
+ "envName": null,
+ "propertyName": "zeppelin.spark.printREPLOutput",
+ "value": "true",
+ "description": "Print REPL output",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.maxResult": {
+ "envName": "ZEPPELIN_SPARK_MAXRESULT",
+ "propertyName": "zeppelin.spark.maxResult",
+ "value": "1000",
+ "description": "Max number of Spark SQL result to display.",
+ "type": "number"
+ },
+ "master":{
+ "envName": "Master",
+ "propertyName": "spark.master",
+ "value": "MASTER",
+ "description": "Spark master uri. ex) spark://masterhost:7077",
+ "type": "string"
+ },
+ "zeppelin.spark.concurrentSQL": {
+ "envName": "ZEPPELIN_SPARK_CONCURRENTSQL",
+ "propertyName": "zeppelin.spark.concurrentSQL",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.sql.stacktrace": {
+ "envName": "ZEPPELIN_SPARK_SQL_STACKTRACE",
+ "propertyName": "zeppelin.spark.sql.stacktrace",
+ "value": "false",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.spark.importImplicit":{
+ "envName": "ZEPPELIN_SPARK_IMPORTIMPLICIT",
+ "propertyName": "zeppelin.spark.importImplicit",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "zeppelin.pyspark.python": {
+ "envName": "ZEPPELIN_PYSPARK_PYTHON",
+ "propertyName": "zeppelin.pyspark.python",
+ "value": "pythonPYTHONVERSION",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.localrepo": {
+ "envName": "ZEPPELIN_DEP_LOCALREPO",
+ "propertyName": "zeppelin.dep.localrepo",
+ "value": "local-repo",
+ "description": "",
+ "type": "string"
+ },
+ "zeppelin.dep.additionalRemoteRepository": {
+ "envName": "ZEPPELIN_DEP_ADDITIONALREMOTEREPOSITORY",
+ "propertyName": "zeppelin.dep.additionalRemoteRepository",
+ "value": "spark-packages,http://dl.bintray.com/spark-packages/maven,false;",
+ "description": "",
+ "type": "string"
+ },
+ "spark.home": {
+ "envName": "Spark Home",
+ "propertyName": "spark.home",
+ "value": "SPARK_HOME",
+ "description": "",
+ "type": "string"
+ },
+ "spark.hadoop.fs.s3a.fast.upload": {
+ "envName": "SPARK_HADOOP_FS_S3A_FAST_UPLOAD",
+ "propertyName": "spark.hadoop.fs.s3a.fast.upload",
+ "value": "true",
+ "description": "",
+ "type": "checkbox"
+ },
+ "spark.executor.memory": {
+ "envName": "MEMORY_PER_EXECUTOR",
+ "propertyName": "spark.executor.memory",
+ "value": "EXECUTOR_MEMORY",
+ "description": "",
+ "type": "string"
+ }
},
"interpreterGroup": [
{
diff --git a/infrastructure-provisioning/src/general/templates/os/debian/jupyter-notebook.service b/infrastructure-provisioning/src/general/templates/os/debian/jupyter-notebook.service
index 6d2e36b..fe6305c 100644
--- a/infrastructure-provisioning/src/general/templates/os/debian/jupyter-notebook.service
+++ b/infrastructure-provisioning/src/general/templates/os/debian/jupyter-notebook.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Jupyter notebook
diff --git a/infrastructure-provisioning/src/general/templates/os/debian/spark-master.service b/infrastructure-provisioning/src/general/templates/os/debian/spark-master.service
index c0d07a3..b3c7ac2 100644
--- a/infrastructure-provisioning/src/general/templates/os/debian/spark-master.service
+++ b/infrastructure-provisioning/src/general/templates/os/debian/spark-master.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Apache Spark Master and Slave Servers
After=network.target
diff --git a/infrastructure-provisioning/src/general/templates/os/debian/spark-slave.service b/infrastructure-provisioning/src/general/templates/os/debian/spark-slave.service
index 3b9cd8b..f3c16f9 100644
--- a/infrastructure-provisioning/src/general/templates/os/debian/spark-slave.service
+++ b/infrastructure-provisioning/src/general/templates/os/debian/spark-slave.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Apache Spark Slave Servers
After=network.target
diff --git a/infrastructure-provisioning/src/general/templates/os/debian/ungit.service b/infrastructure-provisioning/src/general/templates/os/debian/ungit.service
index a2967a2..cd8a781 100644
--- a/infrastructure-provisioning/src/general/templates/os/debian/ungit.service
+++ b/infrastructure-provisioning/src/general/templates/os/debian/ungit.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=ungit
diff --git a/infrastructure-provisioning/src/general/templates/os/notebook_spark-defaults_local.conf b/infrastructure-provisioning/src/general/templates/os/notebook_spark-defaults_local.conf
index 11ea73c..f7cb340 100644
--- a/infrastructure-provisioning/src/general/templates/os/notebook_spark-defaults_local.conf
+++ b/infrastructure-provisioning/src/general/templates/os/notebook_spark-defaults_local.conf
@@ -1,2 +1,20 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
spark.driver.extraClassPath /opt/jars/*
spark.driver.extraLibraryPath /opt/jars
diff --git a/infrastructure-provisioning/src/general/templates/os/pyspark_dataengine-service_template.json b/infrastructure-provisioning/src/general/templates/os/pyspark_dataengine-service_template.json
index 6c9d85b..8c2a767 100644
--- a/infrastructure-provisioning/src/general/templates/os/pyspark_dataengine-service_template.json
+++ b/infrastructure-provisioning/src/general/templates/os/pyspark_dataengine-service_template.json
@@ -15,6 +15,6 @@
"SPARK_HOME": "SPARK_PATH",
"PYTHONPATH": "PY4J:SPARK_PATHpython/:",
"PYTHONSTARTUP": "SPARK_PATHpython/pyspark/shell.py",
- "PYSPARK_SUBMIT_ARGS": "--master yarn pyspark-shell"
+ "PYSPARK_SUBMIT_ARGS": "--deploy-mode client --master yarn pyspark-shell"
}
}
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/templates/os/r_dataengine-service_template.json b/infrastructure-provisioning/src/general/templates/os/r_dataengine-service_template.json
index 3cc93d1..c9cd356 100644
--- a/infrastructure-provisioning/src/general/templates/os/r_dataengine-service_template.json
+++ b/infrastructure-provisioning/src/general/templates/os/r_dataengine-service_template.json
@@ -5,7 +5,7 @@
"env": {
"HADOOP_CONF_DIR" : "/opt/DATAENGINE-SERVICE_VERSION/CLUSTER_NAME/conf/",
"YARN_CONF_DIR": "/opt/DATAENGINE-SERVICE_VERSION/CLUSTER_NAME/conf/",
- "SPARKR_SUBMIT_ARGS" : "--master YARN_CLI_TYPE sparkr-shell",
+ "SPARKR_SUBMIT_ARGS" : "--deploy-mode client --master YARN_CLI_TYPE sparkr-shell",
"SPARK_HOME": "SPARK_PATH"
}
}
\ No newline at end of file
diff --git a/infrastructure-provisioning/src/general/templates/os/redhat/jupyter-notebook.service b/infrastructure-provisioning/src/general/templates/os/redhat/jupyter-notebook.service
index e91517a..b83fdc1 100644
--- a/infrastructure-provisioning/src/general/templates/os/redhat/jupyter-notebook.service
+++ b/infrastructure-provisioning/src/general/templates/os/redhat/jupyter-notebook.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Jupyter notebook
diff --git a/infrastructure-provisioning/src/general/templates/os/redhat/spark-master.service b/infrastructure-provisioning/src/general/templates/os/redhat/spark-master.service
index c0d07a3..b3c7ac2 100644
--- a/infrastructure-provisioning/src/general/templates/os/redhat/spark-master.service
+++ b/infrastructure-provisioning/src/general/templates/os/redhat/spark-master.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Apache Spark Master and Slave Servers
After=network.target
diff --git a/infrastructure-provisioning/src/general/templates/os/redhat/spark-slave.service b/infrastructure-provisioning/src/general/templates/os/redhat/spark-slave.service
index 3b9cd8b..f3c16f9 100644
--- a/infrastructure-provisioning/src/general/templates/os/redhat/spark-slave.service
+++ b/infrastructure-provisioning/src/general/templates/os/redhat/spark-slave.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Apache Spark Slave Servers
After=network.target
diff --git a/infrastructure-provisioning/src/general/templates/os/redhat/ungit.service b/infrastructure-provisioning/src/general/templates/os/redhat/ungit.service
index 35f29d6..ca23d46 100644
--- a/infrastructure-provisioning/src/general/templates/os/redhat/ungit.service
+++ b/infrastructure-provisioning/src/general/templates/os/redhat/ungit.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=ungit
diff --git a/infrastructure-provisioning/src/general/templates/os/tensorboard.service b/infrastructure-provisioning/src/general/templates/os/tensorboard.service
index e494ae2..b4d69b9 100644
--- a/infrastructure-provisioning/src/general/templates/os/tensorboard.service
+++ b/infrastructure-provisioning/src/general/templates/os/tensorboard.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Tensorflow Tensorboard
diff --git a/infrastructure-provisioning/src/general/templates/os/toree_dataengine-service_template.json b/infrastructure-provisioning/src/general/templates/os/toree_dataengine-service_template.json
index 8d18488..c68e1d6 100644
--- a/infrastructure-provisioning/src/general/templates/os/toree_dataengine-service_template.json
+++ b/infrastructure-provisioning/src/general/templates/os/toree_dataengine-service_template.json
@@ -2,7 +2,7 @@
"language": "scala",
"display_name": "Apache Toree - Scala (Scala-SCALA_VERSION, SPARK_VERSION) [CLUSTER_NAME]",
"env": {
- "__TOREE_SPARK_OPTS__": "--master=yarn --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info",
+ "__TOREE_SPARK_OPTS__": "--deploy-mode client --master=yarn --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info",
"SPARK_HOME": "SPARK_PATH",
"__TOREE_OPTS__": "",
"DEFAULT_INTERPRETER": "Scala",
diff --git a/infrastructure-provisioning/src/general/templates/os/toree_dataengine-service_templatev2.json b/infrastructure-provisioning/src/general/templates/os/toree_dataengine-service_templatev2.json
index 9d65e1d..66012a4 100644
--- a/infrastructure-provisioning/src/general/templates/os/toree_dataengine-service_templatev2.json
+++ b/infrastructure-provisioning/src/general/templates/os/toree_dataengine-service_templatev2.json
@@ -2,7 +2,7 @@
"language": "scala",
"display_name": "Apache Toree - Scala (Scala-SCALA_VERSION, SPARK_VERSION) [CLUSTER_NAME]",
"env": {
- "__TOREE_SPARK_OPTS__": "--master=yarn --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info",
+ "__TOREE_SPARK_OPTS__": "--deploy-mode client --master=yarn --driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info",
"SPARK_HOME": "SPARK_PATH",
"__TOREE_OPTS__": "",
"DEFAULT_INTERPRETER": "Scala",
diff --git a/infrastructure-provisioning/src/ssn/scripts/configure_ui.py b/infrastructure-provisioning/src/ssn/scripts/configure_ui.py
index a26795a..0f86828 100644
--- a/infrastructure-provisioning/src/ssn/scripts/configure_ui.py
+++ b/infrastructure-provisioning/src/ssn/scripts/configure_ui.py
@@ -160,8 +160,12 @@
sudo('sed -i "s/LDAP_HOST/{0}/g" {1}/sources/services/security-service/security.yml'.format(
os.environ['ldap_hostname'], args.dlab_path))
- sudo('sed -i "s/LDAP_USER/{0}/g" {1}/sources/services/security-service/security.yml'.format('{0},{1}'.format(
- os.environ['ldap_service_username'], os.environ['ldap_dn']), args.dlab_path))
+ sudo('sed -i "s/LDAP_USER/{0}/g" {1}/sources/services/security-service/security.yml'.format(
+ os.environ['ldap_service_username'], args.dlab_path))
+ sudo('sed -i "s/LDAP_DN/{0}/g" {1}/sources/services/security-service/security.yml'.format(os.environ['ldap_dn'],
+ args.dlab_path))
+ sudo('sed -i "s/LDAP_OU/{0}/g" {1}/sources/services/security-service/security.yml'.format(os.environ['ldap_ou'],
+ args.dlab_path))
sudo("sed -i 's/LDAP_PASS/{0}/g' {1}/sources/services/security-service/security.yml".format(
os.environ['ldap_service_password'], args.dlab_path))
sudo('cp {0}/sources/services/security-service/security.yml {0}/webapp/security-service/conf/'.format(
diff --git a/infrastructure-provisioning/src/ssn/templates/gitlab.ini b/infrastructure-provisioning/src/ssn/templates/gitlab.ini
index 6da5cf0..4618c8a 100644
--- a/infrastructure-provisioning/src/ssn/templates/gitlab.ini
+++ b/infrastructure-provisioning/src/ssn/templates/gitlab.ini
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
#---- List of all GitLab parameters (some parameters already setuped) ----#
[conf]
diff --git a/infrastructure-provisioning/src/ssn/templates/gitlab.rb b/infrastructure-provisioning/src/ssn/templates/gitlab.rb
index 2b60ec3..b234f94 100644
--- a/infrastructure-provisioning/src/ssn/templates/gitlab.rb
+++ b/infrastructure-provisioning/src/ssn/templates/gitlab.rb
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
## GitLab configuration settings
##! This file is generated during initial installation and **is not** modified
##! during upgrades.
diff --git a/infrastructure-provisioning/src/ssn/templates/mongod.service_template b/infrastructure-provisioning/src/ssn/templates/mongod.service_template
index df056c2..e876852 100644
--- a/infrastructure-provisioning/src/ssn/templates/mongod.service_template
+++ b/infrastructure-provisioning/src/ssn/templates/mongod.service_template
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target
diff --git a/infrastructure-provisioning/src/ssn/templates/nginx_proxy.conf b/infrastructure-provisioning/src/ssn/templates/nginx_proxy.conf
index 8edc4f0..ace0705 100644
--- a/infrastructure-provisioning/src/ssn/templates/nginx_proxy.conf
+++ b/infrastructure-provisioning/src/ssn/templates/nginx_proxy.conf
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
server {
listen 80;
diff --git a/infrastructure-provisioning/src/ssn/templates/proxy_location_jenkins_template.conf b/infrastructure-provisioning/src/ssn/templates/proxy_location_jenkins_template.conf
index 42b737f..145b9e5 100644
--- a/infrastructure-provisioning/src/ssn/templates/proxy_location_jenkins_template.conf
+++ b/infrastructure-provisioning/src/ssn/templates/proxy_location_jenkins_template.conf
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
location /jenkins {
auth_basic "snowflake jenkins";
diff --git a/infrastructure-provisioning/src/ssn/templates/proxy_location_webapp_template.conf b/infrastructure-provisioning/src/ssn/templates/proxy_location_webapp_template.conf
index 36efad6..b768b8c 100644
--- a/infrastructure-provisioning/src/ssn/templates/proxy_location_webapp_template.conf
+++ b/infrastructure-provisioning/src/ssn/templates/proxy_location_webapp_template.conf
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
location / {
proxy_set_header Host $host;
diff --git a/infrastructure-provisioning/src/ssn/templates/squid.conf b/infrastructure-provisioning/src/ssn/templates/squid.conf
index dde69c4..a65cd19 100644
--- a/infrastructure-provisioning/src/ssn/templates/squid.conf
+++ b/infrastructure-provisioning/src/ssn/templates/squid.conf
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
#
# Recommended minimum configuration:
#
diff --git a/infrastructure-provisioning/src/ssn/templates/ssn_nginx.conf b/infrastructure-provisioning/src/ssn/templates/ssn_nginx.conf
index b9301e5..3bc1fab 100644
--- a/infrastructure-provisioning/src/ssn/templates/ssn_nginx.conf
+++ b/infrastructure-provisioning/src/ssn/templates/ssn_nginx.conf
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
diff --git a/infrastructure-provisioning/src/ssn/templates/supervisor_svc.conf b/infrastructure-provisioning/src/ssn/templates/supervisor_svc.conf
index c7fad5f..9b2510b 100644
--- a/infrastructure-provisioning/src/ssn/templates/supervisor_svc.conf
+++ b/infrastructure-provisioning/src/ssn/templates/supervisor_svc.conf
@@ -1,3 +1,21 @@
+; *****************************************************************************
+;
+; Copyright (c) 2016, EPAM SYSTEMS INC
+;
+; Licensed under the Apache License, Version 2.0 (the "License");
+; you may not use this file except in compliance with the License.
+; You may obtain a copy of the License at
+;
+; http://www.apache.org/licenses/LICENSE-2.0
+;
+; Unless required by applicable law or agreed to in writing, software
+; distributed under the License is distributed on an "AS IS" BASIS,
+; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+; See the License for the specific language governing permissions and
+; limitations under the License.
+;
+; ******************************************************************************
+
[supervisorctl]
[inet_http_server]
diff --git a/infrastructure-provisioning/src/zeppelin/scripts/configure_zeppelin_node.py b/infrastructure-provisioning/src/zeppelin/scripts/configure_zeppelin_node.py
index 636aaca..0e05991 100644
--- a/infrastructure-provisioning/src/zeppelin/scripts/configure_zeppelin_node.py
+++ b/infrastructure-provisioning/src/zeppelin/scripts/configure_zeppelin_node.py
@@ -118,6 +118,8 @@
put(templates_dir + 'interpreter_livy.json', '/tmp/interpreter.json')
sudo('sed -i "s|ENDPOINTURL|' + args.endpoint_url + '|g" /tmp/interpreter.json')
sudo('sed -i "s|OS_USER|' + args.os_user + '|g" /tmp/interpreter.json')
+ spark_memory = get_spark_memory()
+ sudo('sed -i "s|DRIVER_MEMORY|{}m|g" /tmp/interpreter.json'.format(spark_memory))
while not port_number_found:
port_free = sudo('nmap -p ' + str(default_port) + ' localhost | grep "closed" > /dev/null; echo $?')
port_free = port_free[:1]
@@ -145,6 +147,8 @@
put(templates_dir + 'interpreter_spark.json', '/tmp/interpreter.json')
sudo('sed -i "s|ENDPOINTURL|' + args.endpoint_url + '|g" /tmp/interpreter.json')
sudo('sed -i "s|OS_USER|' + args.os_user + '|g" /tmp/interpreter.json')
+ spark_memory = get_spark_memory()
+ sudo('sed -i "s|DRIVER_MEMORY|{}m|g" /tmp/interpreter.json'.format(spark_memory))
update_zeppelin_interpreters(args.multiple_clusters, r_enabled, 'local')
sudo('cp -f /tmp/interpreter.json /opt/zeppelin/conf/interpreter.json')
sudo('chown ' + args.os_user + ':' + args.os_user + ' -R /opt/zeppelin/')
diff --git a/infrastructure-provisioning/src/zeppelin/templates/livy-server-cluster.service b/infrastructure-provisioning/src/zeppelin/templates/livy-server-cluster.service
index 60eb124..fd9d758 100644
--- a/infrastructure-provisioning/src/zeppelin/templates/livy-server-cluster.service
+++ b/infrastructure-provisioning/src/zeppelin/templates/livy-server-cluster.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Livy server
diff --git a/infrastructure-provisioning/src/zeppelin/templates/livy-server.service b/infrastructure-provisioning/src/zeppelin/templates/livy-server.service
index 5227db9..650c217 100644
--- a/infrastructure-provisioning/src/zeppelin/templates/livy-server.service
+++ b/infrastructure-provisioning/src/zeppelin/templates/livy-server.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Livy server
diff --git a/infrastructure-provisioning/src/zeppelin/templates/zeppelin-notebook.service b/infrastructure-provisioning/src/zeppelin/templates/zeppelin-notebook.service
index 7a759b4..0298590 100644
--- a/infrastructure-provisioning/src/zeppelin/templates/zeppelin-notebook.service
+++ b/infrastructure-provisioning/src/zeppelin/templates/zeppelin-notebook.service
@@ -1,3 +1,21 @@
+# *****************************************************************************
+#
+# Copyright (c) 2016, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# ******************************************************************************
+
[Unit]
Description=Zeppelin notebook
diff --git a/integration-tests/examples/azure_templates/zeppelin/zeppelin-notebook.json b/integration-tests/examples/azure_templates/zeppelin/zeppelin-notebook.json
index e18ab02..2c7aff4 100644
--- a/integration-tests/examples/azure_templates/zeppelin/zeppelin-notebook.json
+++ b/integration-tests/examples/azure_templates/zeppelin/zeppelin-notebook.json
@@ -2,6 +2,6 @@
"image" : "docker.dlab-zeppelin",
"name" : "set the name",
"shape" : "Standard_F2s",
- "version" : "zeppelin-0.7.0",
- "template_name": "Apache Zeppelin 0.7.0"
+ "version" : "zeppelin-0.8.0",
+ "template_name": "Apache Zeppelin 0.8.0"
}
\ No newline at end of file
diff --git a/integration-tests/examples/copy_files.py b/integration-tests/examples/copy_files.py
index d841e85..403a802 100644
--- a/integration-tests/examples/copy_files.py
+++ b/integration-tests/examples/copy_files.py
@@ -86,7 +86,7 @@
def upload_gcp():
try:
for f in dataset_file:
- local('gsutil -m cp /tmp/{0} gs://{1}/{2}_dataset/'.format(f, args.storage, args.notebook))
+ local('sudo gsutil -m cp /tmp/{0} gs://{1}/{2}_dataset/'.format(f, args.storage, args.notebook))
except Exception as err:
print('Failed to upload test dataset to bucket', str(err))
sys.exit(1)
diff --git a/integration-tests/examples/ec2_templates/zeppelin/zeppelin-notebook.json b/integration-tests/examples/ec2_templates/zeppelin/zeppelin-notebook.json
index 79027c0..287ef45 100644
--- a/integration-tests/examples/ec2_templates/zeppelin/zeppelin-notebook.json
+++ b/integration-tests/examples/ec2_templates/zeppelin/zeppelin-notebook.json
@@ -2,6 +2,6 @@
"image" : "docker.dlab-zeppelin",
"name" : "set the name",
"shape" : "t2.medium",
- "version" : "zeppelin-0.7.0",
- "template_name": "Apache Zeppelin 0.7.0"
+ "version" : "zeppelin-0.8.0",
+ "template_name": "Apache Zeppelin 0.8.0"
}
\ No newline at end of file
diff --git a/integration-tests/examples/gcp_templates/jupyter/jupyter-notebook.json b/integration-tests/examples/gcp_templates/jupyter/jupyter-notebook.json
index bc55471..c109dc1 100644
--- a/integration-tests/examples/gcp_templates/jupyter/jupyter-notebook.json
+++ b/integration-tests/examples/gcp_templates/jupyter/jupyter-notebook.json
@@ -2,6 +2,6 @@
"image" : "docker.dlab-jupyter",
"name" : "set the name",
"shape" : "n1-standard-2",
- "version" : "jupyter_notebook-5.2.0",
- "template_name" : "Jupyter notebook 5.2.0"
+ "version" : "jupyter_notebook-5.7.4",
+ "template_name" : "Jupyter notebook 5.7.4"
}
\ No newline at end of file
diff --git a/integration-tests/examples/gcp_templates/rstudio/rstudio-notebook.json b/integration-tests/examples/gcp_templates/rstudio/rstudio-notebook.json
index 88cb77e..607992a 100644
--- a/integration-tests/examples/gcp_templates/rstudio/rstudio-notebook.json
+++ b/integration-tests/examples/gcp_templates/rstudio/rstudio-notebook.json
@@ -2,6 +2,6 @@
"image" : "docker.dlab-rstudio",
"name" : "set the name",
"shape" : "n1-standard-2",
- "version" : "RStudio-1.1.383",
- "template_name" : "RStudio 1.1.383"
+ "version" : "RStudio-1.1.463",
+ "template_name" : "RStudio 1.1.463"
}
\ No newline at end of file
diff --git a/integration-tests/examples/gcp_templates/zeppelin/zeppelin-notebook.json b/integration-tests/examples/gcp_templates/zeppelin/zeppelin-notebook.json
index 0d427d9..56b8790 100644
--- a/integration-tests/examples/gcp_templates/zeppelin/zeppelin-notebook.json
+++ b/integration-tests/examples/gcp_templates/zeppelin/zeppelin-notebook.json
@@ -2,6 +2,6 @@
"image" : "docker.dlab-zeppelin",
"name" : "set the name",
"shape" : "n1-standard-2",
- "version" : "zeppelin-0.7.2",
- "template_name": "Apache Zeppelin 0.7.2"
+ "version" : "zeppelin-0.8.0",
+ "template_name": "Apache Zeppelin 0.8.0"
}
\ No newline at end of file
diff --git a/integration-tests/examples/scenario_zeppelin/zeppelin_tests.py b/integration-tests/examples/scenario_zeppelin/zeppelin_tests.py
index c75d022..a7ddffe 100644
--- a/integration-tests/examples/scenario_zeppelin/zeppelin_tests.py
+++ b/integration-tests/examples/scenario_zeppelin/zeppelin_tests.py
@@ -66,10 +66,10 @@
return "OK"
def import_note(note_path, notebook_ip):
- files = {'file': (note_path, open(note_path, 'rb'), 'application/json', {'Expires': '0'})}
- response = requests.post('http://{0}:8080/api/notebook/import'.format(notebook_ip), files=files)
+ headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'Expires': '0'}
+ response = requests.post('http://{0}:8080/api/notebook/import'.format(notebook_ip), data=open(note_path, 'rb'), headers=headers)
status = json.loads(response.content)
- if status.get('status') == 'CREATED':
+ if status.get('status') == 'OK':
print('Imported notebook: {}'.format(note_path))
return status.get('body')
else:
diff --git a/integration-tests/examples/test_templates/deeplearning/conv.prototxt b/integration-tests/examples/test_templates/deeplearning/conv.prototxt
index 8671bb5..85a9cd0 100644
--- a/integration-tests/examples/test_templates/deeplearning/conv.prototxt
+++ b/integration-tests/examples/test_templates/deeplearning/conv.prototxt
@@ -1,3 +1,16 @@
+# Copyright (c) 2018, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
# Simple single-layer network to showcase editing model parameters.
name: "convolution"
layer {
diff --git a/integration-tests/examples/test_templates/jupyter/template_visualization_sparkr.ipynb b/integration-tests/examples/test_templates/jupyter/template_visualization_sparkr.ipynb
index 5afce54..10f09a7 100644
--- a/integration-tests/examples/test_templates/jupyter/template_visualization_sparkr.ipynb
+++ b/integration-tests/examples/test_templates/jupyter/template_visualization_sparkr.ipynb
@@ -86,7 +86,7 @@
"metadata": {},
"outputs": [],
"source": [
- "head(summary(flights))[c(\"summary\", \"ArrDelay\",\"CarrierDelay\",\"Distance\")]"
+ "head(summary(limit(flights,1000)))[c(\"summary\", \"ArrDelay\",\"CarrierDelay\",\"Distance\")]"
]
},
{
diff --git a/integration-tests/examples/test_templates/zeppelin/template_visualization_sparkr.json b/integration-tests/examples/test_templates/zeppelin/template_visualization_sparkr.json
index 863b7b5..62c1400 100644
--- a/integration-tests/examples/test_templates/zeppelin/template_visualization_sparkr.json
+++ b/integration-tests/examples/test_templates/zeppelin/template_visualization_sparkr.json
@@ -1 +1 @@
-{"paragraphs":[{"text":"%INTERPRETER_NAME\nsparkR.session()\n\nfull_path <- function(file_path) {\n working_storage <- \"WORKING_STORAGE\"\n output_directory <- \"zeppelin/r\"\n protocol_name <- 'PROTOCOL_NAME'\n sprintf('%s://%s/%s/%s', protocol_name, working_storage, output_directory, file_path)\n}","dateUpdated":"2018-01-04T09:24:59+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823463_-1635169484","id":"20170329-112414_1472595813","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:372"},{"text":"%INTERPRETER_NAME\ncarriers <- read.df(full_path(\"carriers\"), \"parquet\")\ncreateOrReplaceTempView(carriers, \"carriers\")\nprintSchema(carriers)\nhead(carriers, 20)","dateUpdated":"2018-01-04T09:24:07+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823469_-1639016973","id":"20170329-112449_1638412317","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:373"},{"text":"%INTERPRETER_NAME\nairports <- read.df(full_path(\"airports\"), \"parquet\")\ncreateOrReplaceTempView(airports, \"airports\")\nprintSchema(airports)\nhead(airports, 20)","dateUpdated":"2018-01-04T09:24:08+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823470_-1637862726","id":"20170329-112510_202152993","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:374"},{"text":"%INTERPRETER_NAME\nflights <- read.df(full_path(\"flights\"), \"parquet\")\ncreateOrReplaceTempView(flights, \"flights\")\nprintSchema(flights)\nhead(flights, 10)[c(\"ArrDelay\",\"CarrierDelay\",\"WeatherDelay\",\"Distance\")]","dateUpdated":"2018-01-04T09:24:10+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823471_-1638247475","id":"20170329-112523_1571758659","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:375"},{"text":"%INTERPRETER_NAME\nhead(summary(flights))[c(\"summary\", \"ArrDelay\",\"CarrierDelay\",\"Distance\")]","dateUpdated":"2018-01-04T09:24:10+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823472_-1627859255","id":"20170329-112535_1375397859","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:376"},{"text":"%INTERPRETER_NAME\nlibrary(ggplot2)\nlibrary(reshape2)\n\ndelay_sql <- sql(\"\nSELECT SUBSTR(c.description, 0, 15) as Carrier, WorkDayDelay, WeekendDelay \nFROM\n (SELECT CEIL( AVG(f.ArrDelay + f.DepDelay) ) as WorkDayDelay, f.UniqueCarrier\n FROM flights f\n WHERE f.DayOfWeek < 6\n GROUP BY f.UniqueCarrier \n ORDER BY WorkDayDelay desc \n LIMIT 10) t\n JOIN\n (SELECT CEIL( AVG(f.ArrDelay + f.DepDelay) ) as WeekendDelay, f.UniqueCarrier\n FROM flights f\n WHERE f.DayOfWeek > 5\n GROUP BY f.UniqueCarrier) t1\n ON t.UniqueCarrier = t1.UniqueCarrier\n JOIN carriers c \n ON t.UniqueCarrier = c.code \nORDER BY WeekendDelay DESC, WorkDayDelay DESC\n\")\n\ndelay <- collect(delay_sql)\ndelay_melt <- melt(delay[c('Carrier', 'WorkDayDelay', 'WeekendDelay')])\n\ncolor_range_days <- c(\"#2966FF\", \"#61F2FF\")\n\nggplot(data=delay_melt, aes(x=Carrier, y=value, fill=variable)) +\n geom_bar(stat=\"identity\", width=.7, position=\"dodge\") +\n stat_summary(fun.y=mean, geom = \"line\", mapping = aes(group = 1), color=\"red\") +\n stat_summary(fun.y=mean, geom = \"point\", mapping = aes(group = 1), color=\"red\") +\n theme(legend.position=\"right\", axis.text.x=element_text(angle=90)) +\n labs(x=\"Carrier\", y=\"Minutes\", fill=\"Day Type\") +\n coord_fixed(ratio = .2) +\n scale_fill_manual(values=color_range_days) +\n scale_y_continuous(breaks=seq(0, 30, 5))","dateUpdated":"2018-01-04T09:24:12+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823472_-1627859255","id":"20170329-112549_2110062261","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:377"},{"text":"%INTERPRETER_NAME\ntop_flights_sql <- sql(\"\nSELECT t.cnt as FlightsAmt, carriers.description as Carrier \nFROM (\n SELECT count(*) as cnt, flights.UniqueCarrier as carrier_code \n FROM flights \n GROUP BY flights.UniqueCarrier LIMIT 6) t \nLEFT JOIN carriers \n ON t.carrier_code = carriers.code\n\")\n\ntop_flights <- collect(top_flights_sql)\n\nggplot(transform(transform(top_flights, value=FlightsAmt/sum(FlightsAmt)), labPos=cumsum(FlightsAmt)-FlightsAmt/2), \n aes(x=\"\", y = FlightsAmt, fill = Carrier)) +\n geom_bar(width = 1, stat = \"identity\") +\n coord_polar(\"y\", start=0) +\n scale_fill_brewer(palette=\"Dark2\") +\n theme_bw() +\n theme(axis.text.x=element_blank() ,panel.grid.major=element_blank(),panel.grid.minor = element_blank(),panel.border = element_blank()) +\n geom_text(size=4, aes(y=labPos, label=scales::percent(value))) + \n geom_text(size=3, aes(x=1.8, y=labPos, label=top_flights$Carrier)) + \n theme(legend.position=\"none\")","dateUpdated":"2018-01-04T09:24:14+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823473_-1628244004","id":"20170329-112607_812774791","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:378"},{"text":"%INTERPRETER_NAME\ndistance_sql = sql(\"\nSELECT SUBSTR(c.description, 0, 15) as Carrier, COUNT(Distance) AS Distance \nFROM flights f \nJOIN carriers c \n ON f.UniqueCarrier = c.code \nGROUP BY c.description \nORDER BY distance DESC \nLIMIT 10\n\")\n\ndistance <- collect(distance_sql)\n\ndistance$Carrier <- factor(distance$Carrier, levels = distance$Carrier[order(-distance$Distance)])\n\ncolor_range <- c(\"#2966FF\", \"#2E73FF\",\"#3380FF\", \"#388CFF\", \"#3D99FF\", \"#42A6FF\", \"#47B2FF\", \"#4CBFFF\", \"#52CCFF\", \n \"#57D9FF\", \"#5CE6FF\", \"#61F2FF\", \"#66FFFF\")\n\nggplot(data=distance, aes(x=Carrier, y=Distance, fill=Carrier)) +\n geom_bar(stat=\"identity\", width=.7, position=\"dodge\") +\n theme(axis.text.x=element_text(angle=90)) +\n scale_fill_manual(values=color_range) +\n theme(legend.position=\"none\")","dateUpdated":"2018-01-04T09:24:16+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823474_-1627089757","id":"20170329-112623_1822577399","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:379"}],"name":"Flights_Visualization_SparkR","id":"2D3B7XFMR","angularObjects":{"2C6RJRBD2:shared_process":[],"2C6RJRBD1:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}
\ No newline at end of file
+{"paragraphs":[{"text":"%INTERPRETER_NAME\nsparkR.session()\n\nfull_path <- function(file_path) {\n working_storage <- \"WORKING_STORAGE\"\n output_directory <- \"zeppelin/r\"\n protocol_name <- 'PROTOCOL_NAME'\n sprintf('%s://%s/%s/%s', protocol_name, working_storage, output_directory, file_path)\n}","dateUpdated":"2018-01-04T09:24:59+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823463_-1635169484","id":"20170329-112414_1472595813","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:372"},{"text":"%INTERPRETER_NAME\ncarriers <- read.df(full_path(\"carriers\"), \"parquet\")\ncreateOrReplaceTempView(carriers, \"carriers\")\nprintSchema(carriers)\nhead(carriers, 20)","dateUpdated":"2018-01-04T09:24:07+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823469_-1639016973","id":"20170329-112449_1638412317","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:373"},{"text":"%INTERPRETER_NAME\nairports <- read.df(full_path(\"airports\"), \"parquet\")\ncreateOrReplaceTempView(airports, \"airports\")\nprintSchema(airports)\nhead(airports, 20)","dateUpdated":"2018-01-04T09:24:08+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823470_-1637862726","id":"20170329-112510_202152993","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:374"},{"text":"%INTERPRETER_NAME\nflights <- read.df(full_path(\"flights\"), \"parquet\")\ncreateOrReplaceTempView(flights, \"flights\")\nprintSchema(flights)\nhead(flights, 10)[c(\"ArrDelay\",\"CarrierDelay\",\"WeatherDelay\",\"Distance\")]","dateUpdated":"2018-01-04T09:24:10+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823471_-1638247475","id":"20170329-112523_1571758659","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:375"},{"text":"%INTERPRETER_NAME\nhead(summary(limit(flights,10)))[c(\"summary\", \"ArrDelay\",\"CarrierDelay\",\"Distance\")]","dateUpdated":"2018-01-04T09:24:10+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823472_-1627859255","id":"20170329-112535_1375397859","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:376"},{"text":"%INTERPRETER_NAME\nlibrary(ggplot2)\nlibrary(reshape2)\n\ndelay_sql <- sql(\"\nSELECT SUBSTR(c.description, 0, 15) as Carrier, WorkDayDelay, WeekendDelay \nFROM\n (SELECT CEIL( AVG(f.ArrDelay + f.DepDelay) ) as WorkDayDelay, f.UniqueCarrier\n FROM flights f\n WHERE f.DayOfWeek < 6\n GROUP BY f.UniqueCarrier \n ORDER BY WorkDayDelay desc \n LIMIT 10) t\n JOIN\n (SELECT CEIL( AVG(f.ArrDelay + f.DepDelay) ) as WeekendDelay, f.UniqueCarrier\n FROM flights f\n WHERE f.DayOfWeek > 5\n GROUP BY f.UniqueCarrier) t1\n ON t.UniqueCarrier = t1.UniqueCarrier\n JOIN carriers c \n ON t.UniqueCarrier = c.code \nORDER BY WeekendDelay DESC, WorkDayDelay DESC\n\")\n\ndelay <- collect(delay_sql)\ndelay_melt <- melt(delay[c('Carrier', 'WorkDayDelay', 'WeekendDelay')])\n\ncolor_range_days <- c(\"#2966FF\", \"#61F2FF\")\n\nggplot(data=delay_melt, aes(x=Carrier, y=value, fill=variable)) +\n geom_bar(stat=\"identity\", width=.7, position=\"dodge\") +\n stat_summary(fun.y=mean, geom = \"line\", mapping = aes(group = 1), color=\"red\") +\n stat_summary(fun.y=mean, geom = \"point\", mapping = aes(group = 1), color=\"red\") +\n theme(legend.position=\"right\", axis.text.x=element_text(angle=90)) +\n labs(x=\"Carrier\", y=\"Minutes\", fill=\"Day Type\") +\n coord_fixed(ratio = .2) +\n scale_fill_manual(values=color_range_days) +\n scale_y_continuous(breaks=seq(0, 30, 5))","dateUpdated":"2018-01-04T09:24:12+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823472_-1627859255","id":"20170329-112549_2110062261","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:377"},{"text":"%INTERPRETER_NAME\ntop_flights_sql <- sql(\"\nSELECT t.cnt as FlightsAmt, carriers.description as Carrier \nFROM (\n SELECT count(*) as cnt, flights.UniqueCarrier as carrier_code \n FROM flights \n GROUP BY flights.UniqueCarrier LIMIT 6) t \nLEFT JOIN carriers \n ON t.carrier_code = carriers.code\n\")\n\ntop_flights <- collect(top_flights_sql)\n\nggplot(transform(transform(top_flights, value=FlightsAmt/sum(FlightsAmt)), labPos=cumsum(FlightsAmt)-FlightsAmt/2), \n aes(x=\"\", y = FlightsAmt, fill = Carrier)) +\n geom_bar(width = 1, stat = \"identity\") +\n coord_polar(\"y\", start=0) +\n scale_fill_brewer(palette=\"Dark2\") +\n theme_bw() +\n theme(axis.text.x=element_blank() ,panel.grid.major=element_blank(),panel.grid.minor = element_blank(),panel.border = element_blank()) +\n geom_text(size=4, aes(y=labPos, label=scales::percent(value))) + \n geom_text(size=3, aes(x=1.8, y=labPos, label=top_flights$Carrier)) + \n theme(legend.position=\"none\")","dateUpdated":"2018-01-04T09:24:14+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823473_-1628244004","id":"20170329-112607_812774791","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:378"},{"text":"%INTERPRETER_NAME\ndistance_sql = sql(\"\nSELECT SUBSTR(c.description, 0, 15) as Carrier, COUNT(Distance) AS Distance \nFROM flights f \nJOIN carriers c \n ON f.UniqueCarrier = c.code \nGROUP BY c.description \nORDER BY distance DESC \nLIMIT 10\n\")\n\ndistance <- collect(distance_sql)\n\ndistance$Carrier <- factor(distance$Carrier, levels = distance$Carrier[order(-distance$Distance)])\n\ncolor_range <- c(\"#2966FF\", \"#2E73FF\",\"#3380FF\", \"#388CFF\", \"#3D99FF\", \"#42A6FF\", \"#47B2FF\", \"#4CBFFF\", \"#52CCFF\", \n \"#57D9FF\", \"#5CE6FF\", \"#61F2FF\", \"#66FFFF\")\n\nggplot(data=distance, aes(x=Carrier, y=Distance, fill=Carrier)) +\n geom_bar(stat=\"identity\", width=.7, position=\"dodge\") +\n theme(axis.text.x=element_text(angle=90)) +\n scale_fill_manual(values=color_range) +\n theme(legend.position=\"none\")","dateUpdated":"2018-01-04T09:24:16+0000","config":{"colWidth":12,"editorMode":"ace/mode/text","results":{},"enabled":true,"editorSetting":{"language":"text","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1515057823474_-1627089757","id":"20170329-112623_1822577399","dateCreated":"2018-01-04T09:23:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:379"}],"name":"Flights_Visualization_SparkR","id":"2D3B7XFMR","angularObjects":{"2C6RJRBD2:shared_process":[],"2C6RJRBD1:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index f2a1692..537de40 100644
--- a/pom.xml
+++ b/pom.xml
@@ -242,13 +242,25 @@
<exclude>.git/**</exclude>
<exclude>.idea/**</exclude>
<exclude>**/.gitignore</exclude>
+ <exclude>**/.idea/**</exclude>
<exclude>**/*.ini</exclude>
+ <exclude>**/*.ipynb</exclude>
+ <exclude>**/*.iml</exclude>
<exclude>**/*.json</exclude>
+ <exclude>**/*.r</exclude>
<exclude>**/__init__.py</exclude>
<exclude>**/*.conf</exclude>
<exclude>**/.gitkeep</exclude>
<exclude>**/*.lst</exclude>
<exclude>**/*template</exclude>
+ <exclude>**/*.pub</exclude>
+ <exclude>**/*.md</exclude>
+ <exclude>**/*.MD</exclude>
+ <exclude>**/*.txt</exclude>
+ <exclude>**/*.svg</exclude>
+ <exclude>**/*.MockMaker</exclude>
+ <exclude>**/*.BillingToolConfigurationFactory</exclude>
+ <exclude>**/assets/**</exclude>
</excludes>
</configuration>
<executions>
diff --git a/services/common/src/main/java/com/epam/dlab/exceptions/ResourceQuoteReachedException.java b/services/common/src/main/java/com/epam/dlab/exceptions/ResourceQuoteReachedException.java
index d7cafc1..805f2c9 100644
--- a/services/common/src/main/java/com/epam/dlab/exceptions/ResourceQuoteReachedException.java
+++ b/services/common/src/main/java/com/epam/dlab/exceptions/ResourceQuoteReachedException.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.exceptions;
public class ResourceQuoteReachedException extends DlabException {
diff --git a/services/dlab-model/src/main/java/com/epam/dlab/dto/aws/computational/ClusterConfig.java b/services/dlab-model/src/main/java/com/epam/dlab/dto/aws/computational/ClusterConfig.java
index 3077492..472ddd9 100644
--- a/services/dlab-model/src/main/java/com/epam/dlab/dto/aws/computational/ClusterConfig.java
+++ b/services/dlab-model/src/main/java/com/epam/dlab/dto/aws/computational/ClusterConfig.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.dto.aws.computational;
import com.fasterxml.jackson.annotation.JsonInclude;
diff --git a/services/dlab-model/src/main/java/com/epam/dlab/dto/reuploadkey/ReuploadKeyStatus.java b/services/dlab-model/src/main/java/com/epam/dlab/dto/reuploadkey/ReuploadKeyStatus.java
index b7293e6..805217f 100644
--- a/services/dlab-model/src/main/java/com/epam/dlab/dto/reuploadkey/ReuploadKeyStatus.java
+++ b/services/dlab-model/src/main/java/com/epam/dlab/dto/reuploadkey/ReuploadKeyStatus.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.dto.reuploadkey;
import java.util.Arrays;
diff --git a/services/dlab-model/src/main/java/com/epam/dlab/dto/reuploadkey/ReuploadKeyStatusDTO.java b/services/dlab-model/src/main/java/com/epam/dlab/dto/reuploadkey/ReuploadKeyStatusDTO.java
index f8276da..db3e724 100644
--- a/services/dlab-model/src/main/java/com/epam/dlab/dto/reuploadkey/ReuploadKeyStatusDTO.java
+++ b/services/dlab-model/src/main/java/com/epam/dlab/dto/reuploadkey/ReuploadKeyStatusDTO.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.dto.reuploadkey;
import com.epam.dlab.dto.StatusBaseDTO;
diff --git a/services/dlab-rest-client/src/main/java/com/epam/dlab/rest/client/RESTService.java b/services/dlab-rest-client/src/main/java/com/epam/dlab/rest/client/RESTService.java
index 4151cf2..e531b61 100644
--- a/services/dlab-rest-client/src/main/java/com/epam/dlab/rest/client/RESTService.java
+++ b/services/dlab-rest-client/src/main/java/com/epam/dlab/rest/client/RESTService.java
@@ -13,7 +13,6 @@
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-
****************************************************************************/
package com.epam.dlab.rest.client;
@@ -26,49 +25,51 @@
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.core.HttpHeaders;
import javax.ws.rs.core.MediaType;
+import java.net.URI;
import java.util.Collections;
import java.util.Map;
@Slf4j
public class RESTService {
- private Client client;
- private String url;
- private String userAgent;
+ private Client client;
+ private String url;
+ private String userAgent;
- public RESTService() {
- }
+ public RESTService() {
+ }
- RESTService(Client client, String url, String userAgent) {
- this.client = client;
- this.url = url;
- this.userAgent = userAgent;
- }
+ RESTService(Client client, String url, String userAgent) {
+ this.client = client;
+ this.url = url;
+ this.userAgent = userAgent;
+ }
- public <T> T get(String path, Class<T> clazz) {
- Invocation.Builder builder = getBuilder(path);
- log.debug("REST get {}", path);
- return builder.get(clazz);
- }
-
- public <T> T get(String path, Map<String, Object> queryParams, Class<T> clazz) {
- Invocation.Builder builder = getBuilder(path, null, queryParams);
+ public <T> T get(String path, Class<T> clazz) {
+ Invocation.Builder builder = getBuilder(path);
log.debug("REST get {}", path);
return builder.get(clazz);
}
- public <T> T post(String path, Object parameter, Class<T> clazz) {
- Invocation.Builder builder = getBuilder(path);
- log.debug("REST post {}", path);
- return builder.post(Entity.json(parameter), clazz);
- }
+ public <T> T get(URI path, Class<T> clazz) {
+ log.debug("REST get {}", path);
+ return client.target(URI.create(url + path.toString()))
+ .request()
+ .get(clazz);
+ }
- public <T> T get(String path, String accessToken, Class<T> clazz) {
+ public <T> T post(String path, Object parameter, Class<T> clazz) {
+ Invocation.Builder builder = getBuilder(path);
+ log.debug("REST post {}", path);
+ return builder.post(Entity.json(parameter), clazz);
+ }
+
+ public <T> T get(String path, String accessToken, Class<T> clazz) {
Invocation.Builder builder = getBuilder(path, accessToken, Collections.emptyMap());
- log.debug("REST get secured {} {}", path, accessToken);
- return builder.get(clazz);
- }
+ log.debug("REST get secured {} {}", path, accessToken);
+ return builder.get(clazz);
+ }
- public <T> T post(String path, String accessToken, Object parameter, Class<T> clazz) {
+ public <T> T post(String path, String accessToken, Object parameter, Class<T> clazz) {
Invocation.Builder builder = getBuilder(path, accessToken, Collections.emptyMap());
log.debug("REST post secured {} {}", path, accessToken);
return builder.post(Entity.json(parameter), clazz);
@@ -81,7 +82,7 @@
return builder.post(Entity.json(parameter), clazz);
}
- private Invocation.Builder getBuilder(String path) {
+ private Invocation.Builder getBuilder(String path) {
return getBuilder(path, null, Collections.emptyMap());
}
@@ -92,23 +93,23 @@
}
Invocation.Builder builder = webTarget
- .request(MediaType.APPLICATION_JSON)
- .accept(MediaType.APPLICATION_JSON);
+ .request(MediaType.APPLICATION_JSON)
+ .accept(MediaType.APPLICATION_JSON);
- if (token != null) {
- builder.header(HttpHeaders.AUTHORIZATION, "Bearer " + token);
- }
+ if (token != null) {
+ builder.header(HttpHeaders.AUTHORIZATION, "Bearer " + token);
+ }
- if (userAgent != null) {
- builder.header(HttpHeaders.USER_AGENT, userAgent);
- }
+ if (userAgent != null) {
+ builder.header(HttpHeaders.USER_AGENT, userAgent);
+ }
- return builder;
- }
+ return builder;
+ }
- private WebTarget getWebTarget(String path) {
- return client
- .target(url)
- .path(path);
- }
+ private WebTarget getWebTarget(String path) {
+ return client
+ .target(url)
+ .path(path);
+ }
}
diff --git a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/LongDeSerializer.java b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/LongDeSerializer.java
index 02c716f..7eca9c8 100644
--- a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/LongDeSerializer.java
+++ b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/LongDeSerializer.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.util.mongo;
import com.fasterxml.jackson.core.JsonParser;
diff --git a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/MongoStringDeserializer.java b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/MongoStringDeserializer.java
index fe4968b..182a4e1 100644
--- a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/MongoStringDeserializer.java
+++ b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/MongoStringDeserializer.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.util.mongo;
import com.fasterxml.jackson.databind.DeserializationContext;
diff --git a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/MongoStringSerializaer.java b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/MongoStringSerializaer.java
index 9a79320..7fb7794 100644
--- a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/MongoStringSerializaer.java
+++ b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/MongoStringSerializaer.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.util.mongo;
import com.fasterxml.jackson.core.JsonGenerator;
diff --git a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/modules/JavaPrimitiveModule.java b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/modules/JavaPrimitiveModule.java
index a2b1b48..c786731 100644
--- a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/modules/JavaPrimitiveModule.java
+++ b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/modules/JavaPrimitiveModule.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.util.mongo.modules;
import com.epam.dlab.util.mongo.LongDeSerializer;
diff --git a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/modules/MongoModule.java b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/modules/MongoModule.java
index 459925f..45574fe 100644
--- a/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/modules/MongoModule.java
+++ b/services/dlab-utils/src/main/java/com/epam/dlab/util/mongo/modules/MongoModule.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.util.mongo.modules;
import com.epam.dlab.util.mongo.MongoStringDeserializer;
diff --git a/services/dlab-webapp-common/src/main/java/com/epam/dlab/constants/ServiceConsts.java b/services/dlab-webapp-common/src/main/java/com/epam/dlab/constants/ServiceConsts.java
index 3b97949..83d6e3e 100644
--- a/services/dlab-webapp-common/src/main/java/com/epam/dlab/constants/ServiceConsts.java
+++ b/services/dlab-webapp-common/src/main/java/com/epam/dlab/constants/ServiceConsts.java
@@ -19,6 +19,7 @@
public final class ServiceConsts {
public static final String MONGO_NAME = "mongo";
public static final String PROVISIONING_SERVICE_NAME = "provisioningService";
+ public static final String MAVEN_SEARCH_API = "mavenSearchService";
public static final String SECURITY_SERVICE_NAME = "securityService";
public static final String SELF_SERVICE_NAME = "selfService";
public static final String PROVISIONING_USER_AGENT = "provisioning-service";
diff --git a/services/dlab-webapp-common/src/main/java/com/epam/dlab/rest/mappers/ResourceQuoteReachedExceptionMapper.java b/services/dlab-webapp-common/src/main/java/com/epam/dlab/rest/mappers/ResourceQuoteReachedExceptionMapper.java
index 6573bb6..e7c6723 100644
--- a/services/dlab-webapp-common/src/main/java/com/epam/dlab/rest/mappers/ResourceQuoteReachedExceptionMapper.java
+++ b/services/dlab-webapp-common/src/main/java/com/epam/dlab/rest/mappers/ResourceQuoteReachedExceptionMapper.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.rest.mappers;
import com.epam.dlab.exceptions.ResourceQuoteReachedException;
diff --git a/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/core/response/handlers/ReuploadKeyCallbackHandler.java b/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/core/response/handlers/ReuploadKeyCallbackHandler.java
index 2d5a88a..7d1f747 100644
--- a/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/core/response/handlers/ReuploadKeyCallbackHandler.java
+++ b/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/core/response/handlers/ReuploadKeyCallbackHandler.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.core.response.handlers;
import com.epam.dlab.auth.SystemUserInfoService;
diff --git a/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/resources/azure/ExploratoryResourceAzure.java b/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/resources/azure/ExploratoryResourceAzure.java
index faf09d2..ffebe4d 100644
--- a/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/resources/azure/ExploratoryResourceAzure.java
+++ b/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/resources/azure/ExploratoryResourceAzure.java
@@ -22,6 +22,7 @@
import com.epam.dlab.dto.azure.exploratory.ExploratoryActionStartAzure;
import com.epam.dlab.dto.azure.exploratory.ExploratoryActionStopAzure;
import com.epam.dlab.dto.azure.exploratory.ExploratoryCreateAzure;
+import com.epam.dlab.dto.exploratory.ExploratoryReconfigureSparkClusterActionDTO;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.google.inject.Inject;
import io.dropwizard.auth.Auth;
@@ -62,4 +63,10 @@
public String stop(@Auth UserInfo ui, ExploratoryActionStopAzure dto) throws JsonProcessingException {
return exploratoryService.action(ui.getName(), dto, DockerAction.STOP);
}
+
+ @Path("/reconfigure_spark")
+ @POST
+ public String reconfigureSpark(@Auth UserInfo ui, ExploratoryReconfigureSparkClusterActionDTO dto) throws JsonProcessingException {
+ return exploratoryService.action(ui.getName(), dto, DockerAction.RECONFIGURE_SPARK);
+ }
}
diff --git a/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/resources/gcp/ExploratoryResourceGcp.java b/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/resources/gcp/ExploratoryResourceGcp.java
index 4e9ef10..ba76bd7 100644
--- a/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/resources/gcp/ExploratoryResourceGcp.java
+++ b/services/provisioning-service/src/main/java/com/epam/dlab/backendapi/resources/gcp/ExploratoryResourceGcp.java
@@ -21,6 +21,7 @@
import com.epam.dlab.backendapi.resources.base.ExploratoryService;
import com.epam.dlab.dto.exploratory.ExploratoryActionDTO;
import com.epam.dlab.dto.exploratory.ExploratoryGitCredsUpdateDTO;
+import com.epam.dlab.dto.exploratory.ExploratoryReconfigureSparkClusterActionDTO;
import com.epam.dlab.dto.gcp.exploratory.ExploratoryCreateGcp;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.google.inject.Inject;
@@ -64,4 +65,10 @@
public String stop(@Auth UserInfo ui, ExploratoryActionDTO<?> dto) throws JsonProcessingException {
return exploratoryService.action(ui.getName(), dto, DockerAction.STOP);
}
+
+ @Path("/reconfigure_spark")
+ @POST
+ public String reconfigureSpark(@Auth UserInfo ui, ExploratoryReconfigureSparkClusterActionDTO dto) throws JsonProcessingException {
+ return exploratoryService.action(ui.getName(), dto, DockerAction.RECONFIGURE_SPARK);
+ }
}
diff --git a/services/security-azure/src/main/java/com/epam/dlab/auth/azure/service/AzureAuthorizationCodeServiceImpl.java b/services/security-azure/src/main/java/com/epam/dlab/auth/azure/service/AzureAuthorizationCodeServiceImpl.java
index 3fa1e69..14e2b43 100644
--- a/services/security-azure/src/main/java/com/epam/dlab/auth/azure/service/AzureAuthorizationCodeServiceImpl.java
+++ b/services/security-azure/src/main/java/com/epam/dlab/auth/azure/service/AzureAuthorizationCodeServiceImpl.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.auth.azure.service;
import com.epam.dlab.auth.UserInfo;
diff --git a/services/security-service/security.yml b/services/security-service/security.yml
index 3872183..c37eec4 100644
--- a/services/security-service/security.yml
+++ b/services/security-service/security.yml
@@ -24,10 +24,10 @@
ldapHost: LDAP_HOST
# ldapPort: 3890
ldapPort: 389
- name: LDAP_USER
+ name: LDAP_USER,LDAP_DN
credentials: LDAP_PASS
useLdapBindTemplate: true
-ldapBindTemplate: uid=%s,ou=People,dc=example,dc=com
+ldapBindTemplate: uid=%s,LDAP_OU,LDAP_DN
ldapBindAttribute: uid
ldapSearchAttribute: uid
ldapSearch:
@@ -41,7 +41,7 @@
- uid
- gidNumber
timeLimit: 0
- base: dc=example,dc=com
+ base: LDAP_DN
filter: "(&(objectClass=inetOrgPerson)(uid=%uid%))"
- name: userInfo
cache: true
@@ -51,7 +51,7 @@
- cn
- gidNumber
timeLimit: 0
- base: dc=example,dc=com
+ base: LDAP_DN
filter: "(&(objectClass=inetOrgPerson)(uid=%uid%))"
searchResultProcessor:
language: python
@@ -78,7 +78,7 @@
- gidNumber
- memberUid
timeLimit: 0
- base: dc=example,dc=com
+ base: LDAP_DN
filter: "(&(objectClass=posixGroup))"
searchResultProcessor:
language: javascript
diff --git a/services/self-service/pom.xml b/services/self-service/pom.xml
index 5dc53ec..66b7395 100644
--- a/services/self-service/pom.xml
+++ b/services/self-service/pom.xml
@@ -26,6 +26,13 @@
<relativePath>../../pom.xml</relativePath>
</parent>
+ <repositories>
+ <repository>
+ <id>michaelklishin</id>
+ <url>https://dl.bintray.com/michaelklishin/maven/</url>
+ </repository>
+ </repositories>
+
<artifactId>self-service</artifactId>
<properties>
@@ -37,6 +44,11 @@
<dependencies>
<dependency>
+ <groupId>com.novemberain</groupId>
+ <artifactId>quartz-mongodb</artifactId>
+ <version>2.1.0</version>
+ </dependency>
+ <dependency>
<groupId>com.epam.dlab</groupId>
<artifactId>common</artifactId>
</dependency>
@@ -69,7 +81,14 @@
<groupId>com.fiestacabin.dropwizard.quartz</groupId>
<artifactId>dropwizard-quartz</artifactId>
<version>${com.fiestacabin.dropwizard.quartz.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.quartz-scheduler</groupId>
+ <artifactId>quartz</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
+
<dependency>
<groupId>com.jcraft</groupId>
<artifactId>jsch</artifactId>
diff --git a/services/self-service/self-service.yml b/services/self-service/self-service.yml
index d401eef..6259c4e 100644
--- a/services/self-service/self-service.yml
+++ b/services/self-service/self-service.yml
@@ -101,6 +101,7 @@
level: INFO
loggers:
com.epam: TRACE
+ com.novemberain: ERROR
appenders:
<#if DEV_MODE == "true">
- type: console
@@ -122,35 +123,36 @@
license: Apache 2.0
licenseUrl: https://www.apache.org/licenses/LICENSE-2.0
+mavenSearchService:
+ protocol: http
+ host: search.maven.org
+ port: 80
+ jerseyClient:
+ timeout: 5s
+ connectionTimeout: 5s
+
dexIdentityProviderEnabled: false
schedulers:
inactivity:
enabled: false
- interval: 120
- timeUnit: MINUTES
+ cron: "0 0 0/2 ? * * *"
startComputationalScheduler:
enabled: true
- interval: 10
- timeUnit: SECONDS
+ cron: "*/20 * * ? * * *"
stopComputationalScheduler:
enabled: true
- interval: 10
- timeUnit: SECONDS
+ cron: "*/20 * * ? * * *"
startExploratoryScheduler:
enabled: true
- interval: 10
- timeUnit: SECONDS
+ cron: "*/20 * * ? * * *"
stopExploratoryScheduler:
enabled: true
- interval: 10
- timeUnit: SECONDS
+ cron: "*/20 * * ? * * *"
checkQuoteScheduler:
enabled: true
- interval: 1
- timeUnit: HOURS
+ cron: "0 0 * ? * * *"
checkUserQuoteScheduler:
enabled: true
- interval: 1
- timeUnit: HOURS
+ cron: "0 0 * ? * * *"
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/SelfServiceApplicationConfiguration.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/SelfServiceApplicationConfiguration.java
index 4d84eb2..22eeb0b 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/SelfServiceApplicationConfiguration.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/SelfServiceApplicationConfiguration.java
@@ -20,6 +20,8 @@
import com.epam.dlab.ServiceConfiguration;
import com.epam.dlab.backendapi.domain.SchedulerConfigurationData;
import com.epam.dlab.backendapi.validation.SelfServiceCloudConfigurationSequenceProvider;
+import com.epam.dlab.constants.ServiceConsts;
+import com.epam.dlab.rest.client.RESTServiceFactory;
import com.epam.dlab.validation.AwsValidation;
import com.epam.dlab.validation.AzureValidation;
import com.epam.dlab.validation.GcpValidation;
@@ -114,6 +116,11 @@
@NotNull
private JerseyClientConfiguration jerseyClient = new JerseyClientConfiguration();
+ @Valid
+ @NotNull
+ @JsonProperty(ServiceConsts.MAVEN_SEARCH_API)
+ private RESTServiceFactory mavenApiFactory;
+
@JsonProperty("jerseyClient")
public JerseyClientConfiguration getJerseyClientConfiguration() {
return jerseyClient;
@@ -230,6 +237,11 @@
return mongoMigrationEnabled;
}
+ @NotNull
+ public RESTServiceFactory getMavenApiFactory() {
+ return mavenApiFactory;
+ }
+
public boolean isDexIdentityProviderEnabled() {
return dexIdentityProviderEnabled;
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/BaseBillingDAO.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/BaseBillingDAO.java
index a681ac7..49e961f 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/BaseBillingDAO.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/BaseBillingDAO.java
@@ -63,6 +63,7 @@
private static final int ONE_HUNDRED = 100;
private static final String TOTAL_FIELD_NAME = "total";
private static final String COST_FIELD = "$cost";
+ private static final String SHARED_RESOURCE_NAME = "Shared resource";
@Inject
protected SettingsDAO settings;
@@ -120,13 +121,18 @@
public boolean isUserQuoteReached(String user) {
final Double userCost = getUserCost(user);
return userSettingsDAO.getAllowedBudget(user)
- .filter(allowedBudget -> userCost != 0D && allowedBudget <= userCost)
+ .filter(allowedBudget -> userCost.intValue() != 0 && allowedBudget <= userCost)
.isPresent();
}
+ protected String getUserOrDefault(String user) {
+ return StringUtils.isNotBlank(user) ? user : SHARED_RESOURCE_NAME;
+ }
+
private Integer toPercentage(Supplier<Optional<Integer>> allowedBudget, Double totalCost) {
return allowedBudget.get()
- .map(userBudget -> (totalCost.intValue() * ONE_HUNDRED) / userBudget)
+ .map(userBudget -> (totalCost * ONE_HUNDRED) / userBudget)
+ .map(Double::intValue)
.orElse(BigDecimal.ZERO.intValue());
}
@@ -197,8 +203,7 @@
return shapeNames == null || shapeNames.isEmpty() || Arrays.stream(shapes).anyMatch(shapeNames::contains);
}
- protected abstract void appendSsnAndEdgeNodeType(List<String> shapeNames, Map<String, ShapeInfo>
- shapes);
+ protected abstract void appendSsnAndEdgeNodeType(List<String> shapeNames, Map<String, ShapeInfo> shapes);
protected String generateShapeName(ShapeInfo shape) {
return Optional.ofNullable(shape).map(ShapeInfo::getName).orElse(StringUtils.EMPTY);
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/EnvDAO.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/EnvDAO.java
index 7883959..1c9f9fc 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/EnvDAO.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/EnvDAO.java
@@ -206,6 +206,12 @@
).collect(Collectors.toSet());
}
+ public Set<String> fetchUsersNotIn(Set<String> users) {
+ return stream(find(USER_EDGE, not(in(ID, users)),
+ fields(include(ID)))).map(d -> d.getString(ID))
+ .collect(Collectors.toSet());
+ }
+
public Set<String> fetchAllUsers() {
return stream(find(USER_EDGE)).map(d -> d.getString(ID))
.collect(Collectors.toSet());
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/KeyDAO.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/KeyDAO.java
index 994aaec..82f6513 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/KeyDAO.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/KeyDAO.java
@@ -163,6 +163,10 @@
return (d == null ? "" : d.getString(EDGE_STATUS));
}
+ public void removeEdge(String user) {
+ deleteOne(USER_EDGE, eq(ID, user));
+ }
+
public boolean edgeNodeExist(String user) {
return findOne(USER_EDGE, and(eq(ID, user), not(in(EDGE_STATUS, UserInstanceStatus.TERMINATING.toString(),
UserInstanceStatus.TERMINATED.toString()))))
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/UserGroupDao.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/UserGroupDao.java
index bf74984..a6c3cd8 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/UserGroupDao.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/UserGroupDao.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.dao;
import java.util.Set;
@@ -5,6 +22,8 @@
public interface UserGroupDao {
void addUsers(String group, Set<String> users);
+ void updateUsers(String group, Set<String> users);
+
void removeUser(String group, String user);
void removeGroup(String groupId);
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/UserGroupDaoImpl.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/UserGroupDaoImpl.java
index ef5d9ef..bbf87e8 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/UserGroupDaoImpl.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/UserGroupDaoImpl.java
@@ -1,6 +1,24 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.dao;
import com.google.inject.Singleton;
+import org.bson.Document;
import java.util.Set;
@@ -18,6 +36,11 @@
}
@Override
+ public void updateUsers(String group, Set<String> users) {
+ updateOne(USER_GROUPS, eq(ID, group), new Document(SET, new Document(USERS_FIELD, users)), true);
+ }
+
+ @Override
public void removeUser(String group, String user) {
updateOne(USER_GROUPS, eq(ID, group), pull(USERS_FIELD, user));
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/aws/AwsBillingDAO.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/aws/AwsBillingDAO.java
index f466603..85faa5c 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/aws/AwsBillingDAO.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/aws/AwsBillingDAO.java
@@ -150,11 +150,12 @@
costTotal += cost;
Document item = new Document()
- .append(FIELD_USER_ID, id.getString(USER))
+ .append(FIELD_USER_ID, getUserOrDefault(id.getString(USER)))
.append(FIELD_DLAB_ID, resourceId)
.append(DLAB_RESOURCE_TYPE, resourceTypeId)
.append(SHAPE, shapeName)
- .append(STATUS, status)
+ .append(STATUS,
+ Optional.ofNullable(status).map(UserInstanceStatus::toString).orElse(StringUtils.EMPTY))
.append(FIELD_PRODUCT, id.getString(FIELD_PRODUCT))
.append(FIELD_RESOURCE_TYPE, id.getString(FIELD_RESOURCE_TYPE))
.append(FIELD_COST, BillingCalculationUtils.formatDouble(cost))
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/azure/AzureBillingDAO.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/azure/AzureBillingDAO.java
index a7354cb..fdb2e5e 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/azure/AzureBillingDAO.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/dao/azure/AzureBillingDAO.java
@@ -111,10 +111,11 @@
costTotal += d.getDouble(MongoKeyWords.COST);
Document item = new Document()
- .append(MongoKeyWords.DLAB_USER, id.getString(USER))
+ .append(MongoKeyWords.DLAB_USER, getUserOrDefault(id.getString(USER)))
.append(MongoKeyWords.DLAB_ID, resourceId)
.append(SIZE, generateShapeName(shape))
- .append(STATUS, status)
+ .append(STATUS,
+ Optional.ofNullable(status).map(UserInstanceStatus::toString).orElse(StringUtils.EMPTY))
.append(MongoKeyWords.METER_CATEGORY, id.getString(MongoKeyWords.METER_CATEGORY))
.append(MongoKeyWords.RESOURCE_TYPE,
DlabResourceType.getResourceTypeName(id.getString(MongoKeyWords.RESOURCE_TYPE)))
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/domain/SchedulerConfigurationData.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/domain/SchedulerConfigurationData.java
index 55c82ce..cf5c042 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/domain/SchedulerConfigurationData.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/domain/SchedulerConfigurationData.java
@@ -23,13 +23,10 @@
import lombok.Data;
import javax.validation.constraints.NotNull;
-import java.util.concurrent.TimeUnit;
@Data
public class SchedulerConfigurationData {
private final boolean enabled;
@NotNull
- private final Integer interval;
- @NotNull
- private final TimeUnit timeUnit;
+ private final String cron;
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/AwsSelfServiceModule.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/AwsSelfServiceModule.java
index b2add35..71ef19a 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/AwsSelfServiceModule.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/AwsSelfServiceModule.java
@@ -18,6 +18,7 @@
import com.epam.dlab.auth.SecurityFactory;
import com.epam.dlab.backendapi.SelfServiceApplication;
+import com.epam.dlab.backendapi.SelfServiceApplicationConfiguration;
import com.epam.dlab.backendapi.annotation.BudgetLimited;
import com.epam.dlab.backendapi.auth.SelfServiceSecurityAuthenticator;
import com.epam.dlab.backendapi.dao.BillingDAO;
@@ -37,6 +38,7 @@
import com.epam.dlab.backendapi.service.aws.AwsInfrastructureInfoService;
import com.epam.dlab.backendapi.service.aws.AwsInfrastructureTemplateService;
import com.epam.dlab.cloud.CloudModule;
+import com.epam.dlab.mongo.MongoServiceFactory;
import com.fiestacabin.dropwizard.quartz.SchedulerConfiguration;
import com.google.inject.Injector;
import com.google.inject.Provides;
@@ -52,6 +54,10 @@
public class AwsSelfServiceModule extends CloudModule {
+ private static final String MONGO_URI_FORMAT = "mongodb://%s:%s@%s:%d/%s";
+ private static final String QUARTZ_MONGO_URI_PROPERTY = "org.quartz.jobStore.mongoUri";
+ private static final String QUARTZ_DB_NAME = "org.quartz.jobStore.dbName";
+
private final boolean useDex;
public AwsSelfServiceModule(boolean useDex) {
@@ -89,7 +95,13 @@
@Provides
@Singleton
- Scheduler provideScheduler() throws SchedulerException {
+ Scheduler provideScheduler(SelfServiceApplicationConfiguration configuration) throws SchedulerException {
+ final MongoServiceFactory mongoFactory = configuration.getMongoFactory();
+ final String database = mongoFactory.getDatabase();
+ final String mongoUri = String.format(MONGO_URI_FORMAT, mongoFactory.getUsername(), mongoFactory.getPassword(),
+ mongoFactory.getHost(), mongoFactory.getPort(), database);
+ System.setProperty(QUARTZ_MONGO_URI_PROPERTY, mongoUri);
+ System.setProperty(QUARTZ_DB_NAME, database);
return StdSchedulerFactory.getDefaultScheduler();
}
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/AzureSelfServiceModule.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/AzureSelfServiceModule.java
index b55bfaa..aa1187b 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/AzureSelfServiceModule.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/AzureSelfServiceModule.java
@@ -19,6 +19,7 @@
import com.epam.dlab.auth.SecurityFactory;
import com.epam.dlab.auth.rest.UserSessionDurationAuthorizer;
import com.epam.dlab.backendapi.SelfServiceApplication;
+import com.epam.dlab.backendapi.SelfServiceApplicationConfiguration;
import com.epam.dlab.backendapi.annotation.BudgetLimited;
import com.epam.dlab.backendapi.auth.SelfServiceSecurityAuthenticator;
import com.epam.dlab.backendapi.dao.BillingDAO;
@@ -40,6 +41,7 @@
import com.epam.dlab.backendapi.service.azure.AzureInfrastructureInfoService;
import com.epam.dlab.backendapi.service.azure.AzureInfrastructureTemplateService;
import com.epam.dlab.cloud.CloudModule;
+import com.epam.dlab.mongo.MongoServiceFactory;
import com.fiestacabin.dropwizard.quartz.SchedulerConfiguration;
import com.google.inject.Injector;
import com.google.inject.Provides;
@@ -57,6 +59,9 @@
@Slf4j
public class AzureSelfServiceModule extends CloudModule {
+ private static final String MONGO_URI_FORMAT = "mongodb://%s:%s@%s:%d/%s";
+ private static final String QUARTZ_MONGO_URI_PROPERTY = "org.quartz.jobStore.mongoUri";
+ private static final String QUARTZ_DB_NAME = "org.quartz.jobStore.dbName";
private boolean useLdap;
private long maxSessionDurabilityMilliseconds;
private final boolean useDex;
@@ -106,7 +111,13 @@
@Provides
@Singleton
- Scheduler provideScheduler() throws SchedulerException {
+ Scheduler provideScheduler(SelfServiceApplicationConfiguration configuration) throws SchedulerException {
+ final MongoServiceFactory mongoFactory = configuration.getMongoFactory();
+ final String database = mongoFactory.getDatabase();
+ final String mongoUri = String.format(MONGO_URI_FORMAT, mongoFactory.getUsername(), mongoFactory.getPassword(),
+ mongoFactory.getHost(), mongoFactory.getPort(), database);
+ System.setProperty(QUARTZ_MONGO_URI_PROPERTY, mongoUri);
+ System.setProperty(QUARTZ_DB_NAME, database);
return StdSchedulerFactory.getDefaultScheduler();
}
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/DevModule.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/DevModule.java
index a593571..6a45879 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/DevModule.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/DevModule.java
@@ -34,10 +34,8 @@
import com.epam.dlab.rest.dto.ErrorDTO;
import com.google.inject.name.Names;
import io.dropwizard.auth.Authorizer;
-import io.dropwizard.client.JerseyClientBuilder;
import io.dropwizard.setup.Environment;
-import javax.ws.rs.client.Client;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.Map;
@@ -89,8 +87,8 @@
bind(EdgeService.class).to(EdgeServiceImpl.class);
bind(ReuploadKeyService.class).to(ReuploadKeyServiceImpl.class);
bind(UserResourceService.class).to(UserResourceServiceImpl.class);
- bind(Client.class).toInstance(new JerseyClientBuilder(environment).using(configuration.getJerseyClientConfiguration())
- .build(""));
+ bind(RESTService.class).annotatedWith(Names.named(ServiceConsts.MAVEN_SEARCH_API))
+ .toInstance(configuration.getMavenApiFactory().build(environment, ServiceConsts.MAVEN_SEARCH_API));
bind(ExternalLibraryService.class).to(MavenCentralLibraryService.class);
bind(SystemInfoService.class).to(SystemInfoServiceImpl.class);
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/GcpSelfServiceModule.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/GcpSelfServiceModule.java
index 9710e64..1629bf0 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/GcpSelfServiceModule.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/GcpSelfServiceModule.java
@@ -34,6 +34,7 @@
import com.epam.dlab.backendapi.service.gcp.GcpInfrastructureInfoService;
import com.epam.dlab.backendapi.service.gcp.GcpInfrastructureTemplateService;
import com.epam.dlab.cloud.CloudModule;
+import com.epam.dlab.mongo.MongoServiceFactory;
import com.fiestacabin.dropwizard.quartz.SchedulerConfiguration;
import com.google.inject.Injector;
import com.google.inject.Provides;
@@ -46,6 +47,10 @@
public class GcpSelfServiceModule extends CloudModule {
+ private static final String MONGO_URI_FORMAT = "mongodb://%s:%s@%s:%d/%s";
+ private static final String QUARTZ_MONGO_URI_PROPERTY = "org.quartz.jobStore.mongoUri";
+ private static final String QUARTZ_DB_NAME = "org.quartz.jobStore.dbName";
+
private final boolean useDex;
public GcpSelfServiceModule(boolean useDex) {
@@ -82,7 +87,13 @@
@Provides
@Singleton
- Scheduler provideScheduler() throws SchedulerException {
+ Scheduler provideScheduler(SelfServiceApplicationConfiguration configuration) throws SchedulerException {
+ final MongoServiceFactory mongoFactory = configuration.getMongoFactory();
+ final String database = mongoFactory.getDatabase();
+ final String mongoUri = String.format(MONGO_URI_FORMAT, mongoFactory.getUsername(), mongoFactory.getPassword(),
+ mongoFactory.getHost(), mongoFactory.getPort(), database);
+ System.setProperty(QUARTZ_MONGO_URI_PROPERTY, mongoUri);
+ System.setProperty(QUARTZ_DB_NAME, database);
return StdSchedulerFactory.getDefaultScheduler();
}
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/ProductionModule.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/ProductionModule.java
index f38d7a0..7af1e08 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/ProductionModule.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/modules/ProductionModule.java
@@ -29,11 +29,8 @@
import com.epam.dlab.rest.client.RESTService;
import com.google.inject.name.Names;
import io.dropwizard.auth.Authorizer;
-import io.dropwizard.client.JerseyClientBuilder;
import io.dropwizard.setup.Environment;
-import javax.ws.rs.client.Client;
-
/**
* Production class for an application configuration of SelfService.
*/
@@ -75,8 +72,8 @@
bind(EdgeService.class).to(EdgeServiceImpl.class);
bind(ReuploadKeyService.class).to(ReuploadKeyServiceImpl.class);
bind(UserResourceService.class).to(UserResourceServiceImpl.class);
- bind(Client.class).toInstance(new JerseyClientBuilder(environment).using(configuration.getJerseyClientConfiguration())
- .build(""));
+ bind(RESTService.class).annotatedWith(Names.named(ServiceConsts.MAVEN_SEARCH_API))
+ .toInstance(configuration.getMavenApiFactory().build(environment, ServiceConsts.MAVEN_SEARCH_API));
bind(ExternalLibraryService.class).to(MavenCentralLibraryService.class);
bind(SystemInfoService.class).to(SystemInfoServiceImpl.class);
bind(UserGroupService.class).to(UserGroupServiceImpl.class);
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/EnvironmentResource.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/EnvironmentResource.java
index dd2348f..de8cca1 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/EnvironmentResource.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/EnvironmentResource.java
@@ -44,11 +44,11 @@
}
@GET
- @Path("user/active")
+ @Path("user")
@Produces(MediaType.APPLICATION_JSON)
- @ApiOperation("Fetches active users")
- @ApiResponses({@ApiResponse(code = 404, message = "Active users not found"),
- @ApiResponse(code = 200, message = "Active users were fetched successfully")})
+ @ApiOperation("Fetches environment users")
+ @ApiResponses({@ApiResponse(code = 404, message = "Users not found"),
+ @ApiResponse(code = 200, message = "Users were fetched successfully")})
public Response getUsersWithActiveEnv(@ApiParam(hidden = true) @Auth UserInfo userInfo) {
log.debug("User {} requested information about active environments", userInfo.getName());
return Response.ok(environmentService.getUsers()).build();
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/UserGroupResource.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/UserGroupResource.java
index 259260c..be65974 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/UserGroupResource.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/UserGroupResource.java
@@ -18,7 +18,7 @@
package com.epam.dlab.backendapi.resources;
import com.epam.dlab.auth.UserInfo;
-import com.epam.dlab.backendapi.resources.dto.CreateGroupDto;
+import com.epam.dlab.backendapi.resources.dto.GroupDTO;
import com.epam.dlab.backendapi.resources.dto.UpdateRoleGroupDto;
import com.epam.dlab.backendapi.resources.dto.UpdateUserGroupDto;
import com.epam.dlab.backendapi.resources.swagger.SwaggerSecurityInfo;
@@ -59,12 +59,25 @@
@ApiResponse(code = 404, message = "User role not found")
})
public Response createGroup(@ApiParam(hidden = true) @Auth UserInfo userInfo,
- @Valid @ApiParam CreateGroupDto dto) {
+ @Valid @ApiParam GroupDTO dto) {
log.debug("Creating new group {}", dto.getName());
userGroupService.createGroup(dto.getName(), dto.getRoleIds(), dto.getUsers());
return Response.ok().build();
}
+ @PUT
+ @ApiOperation("Updates user group")
+ @ApiResponses({
+ @ApiResponse(code = 200, message = "Group successfully updated"),
+ @ApiResponse(code = 404, message = "User role not found")
+ })
+ public Response updateGroup(@ApiParam(hidden = true) @Auth UserInfo userInfo,
+ @Valid @ApiParam GroupDTO dto) {
+ log.debug("Updating group {}", dto.getName());
+ userGroupService.updateGroup(dto.getName(), dto.getRoleIds(), dto.getUsers());
+ return Response.ok().build();
+ }
+
@GET
@ApiOperation("List groups with roles assigned to it")
@ApiResponses(@ApiResponse(code = 200, message = "Groups present in application"))
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/UserRoleResource.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/UserRoleResource.java
index 5cd25aa..aa1a365 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/UserRoleResource.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/UserRoleResource.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.resources;
import com.epam.dlab.auth.UserInfo;
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/azure/AzureOauthResource.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/azure/AzureOauthResource.java
index 9f4eafa..637188a 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/azure/AzureOauthResource.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/azure/AzureOauthResource.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.resources.azure;
import com.epam.dlab.constants.ServiceConsts;
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/callback/base/EdgeCallback.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/callback/base/EdgeCallback.java
index 36edafd..080dee1 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/callback/base/EdgeCallback.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/callback/base/EdgeCallback.java
@@ -54,6 +54,7 @@
if (UserInstanceStatus.of(status) == UserInstanceStatus.TERMINATED) {
log.debug("Removing key for user {}", user);
keyDAO.deleteKey(user);
+ keyDAO.removeEdge(user);
}
log.debug("Updating the status of EDGE node for user {} to {}", user, status);
keyDAO.updateEdgeStatus(user, status);
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/CreateGroupDto.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/CreateGroupDto.java
deleted file mode 100644
index d267a5c..0000000
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/CreateGroupDto.java
+++ /dev/null
@@ -1,18 +0,0 @@
-package com.epam.dlab.backendapi.resources.dto;
-
-import lombok.Getter;
-import lombok.Setter;
-import org.hibernate.validator.constraints.NotEmpty;
-
-import java.util.Collections;
-import java.util.Set;
-
-@Getter
-@Setter
-public class CreateGroupDto {
- @NotEmpty
- private String name;
- @NotEmpty
- private Set<String> roleIds;
- private Set<String> users = Collections.emptySet();
-}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/GroupDTO.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/GroupDTO.java
new file mode 100644
index 0000000..541821e
--- /dev/null
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/GroupDTO.java
@@ -0,0 +1,35 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+package com.epam.dlab.backendapi.resources.dto;
+
+import lombok.Getter;
+import lombok.Setter;
+import org.hibernate.validator.constraints.NotEmpty;
+
+import java.util.Collections;
+import java.util.Set;
+
+@Getter
+@Setter
+public class GroupDTO {
+ @NotEmpty
+ private String name;
+ @NotEmpty
+ private Set<String> roleIds;
+ private Set<String> users = Collections.emptySet();
+}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/UserDTO.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/UserDTO.java
index f6d3ff2..f8925f4 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/UserDTO.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/resources/dto/UserDTO.java
@@ -15,15 +15,22 @@
*/
package com.epam.dlab.backendapi.resources.dto;
+import lombok.AllArgsConstructor;
import lombok.Data;
import javax.validation.constraints.Min;
import javax.validation.constraints.NotNull;
@Data
+@AllArgsConstructor
public class UserDTO {
@NotNull
private final String name;
@Min(1)
private final Integer budget;
+ private Status status;
+
+ public enum Status {
+ ACTIVE, NOT_ACTIVE
+ }
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/roles/UserGroup.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/roles/UserGroup.java
index 765d826..e3b7fc6 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/roles/UserGroup.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/roles/UserGroup.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.roles;
import lombok.AllArgsConstructor;
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/schedulers/internal/ManagedScheduler.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/schedulers/internal/ManagedScheduler.java
index df6b6a6..d346b5d 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/schedulers/internal/ManagedScheduler.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/schedulers/internal/ManagedScheduler.java
@@ -33,10 +33,8 @@
import org.reflections.scanners.SubTypesScanner;
import java.util.Optional;
-import java.util.concurrent.TimeUnit;
import static org.quartz.JobBuilder.newJob;
-import static org.quartz.SimpleScheduleBuilder.simpleSchedule;
import static org.quartz.TriggerBuilder.newTrigger;
@Slf4j
@@ -62,6 +60,7 @@
.getSubTypesOf(Job.class)
.forEach(scheduledClass ->
Optional.ofNullable(scheduledClass.getAnnotation(Scheduled.class))
+ .filter(this::triggerNotExist)
.ifPresent(scheduleAnn -> schedule(scheduledClass, scheduleAnn)));
}
@@ -71,12 +70,10 @@
scheduler.shutdown();
}
- private Trigger getTrigger(SchedulerConfigurationData schedulerConfig) {
- return newTrigger().withSchedule(simpleSchedule()
- .withIntervalInMilliseconds(
- TimeUnit.MILLISECONDS.convert(schedulerConfig.getInterval(),
- schedulerConfig.getTimeUnit()))
- .repeatForever())
+ private Trigger getTrigger(SchedulerConfigurationData schedulerConfig, String schedulerName) {
+ return newTrigger()
+ .withIdentity(schedulerName)
+ .withSchedule(CronScheduleBuilder.cronSchedule(schedulerConfig.getCron()))
.startNow()
.build();
}
@@ -87,13 +84,13 @@
Optional.ofNullable(config.getSchedulers().get(schedulerName)).orElseThrow(() -> new IllegalStateException(
"There is no configuration provided for scheduler with name " + schedulerName));
if (schedulerConfig.isEnabled()) {
- scheduleJob(newJob(scheduledClass).build(), schedulerConfig);
+ scheduleJob(newJob(scheduledClass).build(), schedulerConfig, scheduleAnn.value());
}
}
- private void scheduleJob(JobDetail job, SchedulerConfigurationData schedulerConfig) {
+ private void scheduleJob(JobDetail job, SchedulerConfigurationData schedulerConfig, String schedulerName) {
try {
- final Trigger trigger = getTrigger(schedulerConfig);
+ final Trigger trigger = getTrigger(schedulerConfig, schedulerName);
scheduler.scheduleJob(job, trigger);
log.info("Scheduled job {} with trigger {}", job, trigger);
} catch (SchedulerException e) {
@@ -101,4 +98,13 @@
throw new DlabException("Can't schedule job due to: " + e.getMessage(), e);
}
}
+
+ private boolean triggerNotExist(Scheduled scheduled) {
+ try {
+ return !scheduler.checkExists(new TriggerKey(scheduled.value()));
+ } catch (SchedulerException e) {
+ log.error("Can not check if trigger exist due to: {}", e.getMessage());
+ throw new DlabException("Can not check if trigger exist due to: {}" + e.getMessage(), e);
+ }
+ }
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/EnvironmentService.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/EnvironmentService.java
index 41ecd0d..ed0bdee 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/EnvironmentService.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/EnvironmentService.java
@@ -24,7 +24,7 @@
public interface EnvironmentService {
- Set<String> getAllUsers();
+ Set<String> getUserNames();
List<UserDTO> getUsers();
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InactivityService.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InactivityService.java
index c0fa663..2bb626f 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InactivityService.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/InactivityService.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.service;
import com.epam.dlab.auth.UserInfo;
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserGroupService.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserGroupService.java
index a25ca14..1fe0b29 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserGroupService.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserGroupService.java
@@ -25,6 +25,7 @@
public interface UserGroupService {
void createGroup(String group, Set<String> roleIds, Set<String> users);
+ void updateGroup(String group, Set<String> roleIds, Set<String> users);
void addUsersToGroup(String group, Set<String> users);
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserRoleService.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserRoleService.java
index c28f674..fd10ee9 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserRoleService.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserRoleService.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.service;
import com.epam.dlab.backendapi.resources.dto.UserRoleDto;
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserRoleServiceImpl.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserRoleServiceImpl.java
index 3bcd6af..9b49599 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserRoleServiceImpl.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/UserRoleServiceImpl.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.service;
import com.epam.dlab.backendapi.dao.UserRoleDao;
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/EnvironmentServiceImpl.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/EnvironmentServiceImpl.java
index 9d150e4..0466e77 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/EnvironmentServiceImpl.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/EnvironmentServiceImpl.java
@@ -43,6 +43,8 @@
import java.util.Set;
import java.util.stream.Stream;
+import static com.epam.dlab.backendapi.resources.dto.UserDTO.Status.ACTIVE;
+import static com.epam.dlab.backendapi.resources.dto.UserDTO.Status.NOT_ACTIVE;
import static java.util.stream.Collectors.toList;
@Singleton
@@ -70,14 +72,22 @@
@Override
public List<UserDTO> getUsers() {
- return getAllUsers()
+ final Set<String> activeUsers = envDAO.fetchActiveEnvUsers();
+ log.trace("Active users: {}", activeUsers);
+ final Set<String> notActiveUsers = envDAO.fetchUsersNotIn(activeUsers);
+ log.trace("Not active users: {}", notActiveUsers);
+ final Stream<UserDTO> activeUsersStream = activeUsers
.stream()
- .map(u -> new UserDTO(u, settingsDAO.getAllowedBudget(u).orElse(null)))
+ .map(u -> toUserDTO(u, ACTIVE));
+ final Stream<UserDTO> notActiveUsersStream = notActiveUsers
+ .stream()
+ .map(u -> toUserDTO(u, NOT_ACTIVE));
+ return Stream.concat(activeUsersStream, notActiveUsersStream)
.collect(toList());
}
@Override
- public Set<String> getAllUsers() {
+ public Set<String> getUserNames() {
log.debug("Getting all users...");
return envDAO.fetchAllUsers();
}
@@ -86,14 +96,14 @@
public List<UserResourceInfo> getAllEnv() {
log.debug("Getting all user's environment...");
List<UserInstanceDTO> expList = exploratoryDAO.getInstances();
- return getAllUsers().stream().map(user -> getUserEnv(user, expList)).flatMap(Collection::stream)
+ return getUserNames().stream().map(user -> getUserEnv(user, expList)).flatMap(Collection::stream)
.collect(toList());
}
@Override
public void stopAll() {
log.debug("Stopping environment for all users...");
- getAllUsers().forEach(this::stopEnvironment);
+ getUserNames().forEach(this::stopEnvironment);
}
@Override
@@ -125,7 +135,7 @@
@Override
public void terminateAll() {
log.debug("Terminating environment for all users...");
- getAllUsers().forEach(this::terminateEnvironment);
+ getUserNames().forEach(this::terminateEnvironment);
}
@Override
@@ -149,6 +159,10 @@
terminateCluster(user, exploratoryName, computationalName);
}
+ private UserDTO toUserDTO(String u, UserDTO.Status status) {
+ return new UserDTO(u, settingsDAO.getAllowedBudget(u).orElse(null), status);
+ }
+
private void checkState(String user, String action) {
final List<UserInstanceDTO> userInstances = exploratoryDAO
.fetchUserExploratoriesWhereStatusIn(user,
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InactivityServiceImpl.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InactivityServiceImpl.java
index e126e05..032d08e 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InactivityServiceImpl.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/InactivityServiceImpl.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.service.impl;
import com.epam.dlab.auth.SystemUserInfoService;
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/MavenCentralLibraryService.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/MavenCentralLibraryService.java
index db026d6..e74cab2 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/MavenCentralLibraryService.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/MavenCentralLibraryService.java
@@ -21,30 +21,32 @@
package com.epam.dlab.backendapi.service.impl;
import com.epam.dlab.backendapi.domain.MavenSearchArtifactResponse;
-import com.epam.dlab.backendapi.service.ExternalLibraryService;
import com.epam.dlab.backendapi.resources.dto.LibraryDTO;
-import com.epam.dlab.exceptions.DlabException;
+import com.epam.dlab.backendapi.service.ExternalLibraryService;
+import com.epam.dlab.constants.ServiceConsts;
import com.epam.dlab.exceptions.ResourceNotFoundException;
+import com.epam.dlab.rest.client.RESTService;
import com.google.inject.Singleton;
+import com.google.inject.name.Named;
import lombok.extern.slf4j.Slf4j;
import javax.inject.Inject;
-import javax.ws.rs.client.Client;
-import javax.ws.rs.core.Response;
import java.net.URI;
+import static java.lang.String.format;
+import static java.lang.String.join;
+
@Singleton
@Slf4j
public class MavenCentralLibraryService implements ExternalLibraryService {
- private static final String MAVEN_SEARCH_API = "http://search.maven.org/solrsearch/select";
private static final String QUOTE_ENCODED = "%22";
- private static final String MAVEN_SEARCH_API_QUERY_FORMAT = "%s?q=%s&rows=20&wt=json&core=gav&p=jar";
+ private static final String SEARCH_API_QUERY_FORMAT = "/solrsearch/select?q=%s&rows=20&wt=json&core=gav&p=jar";
private static final String LIB_NOT_FOUND_MSG = "Artifact with id=%s, groupId=%s and version %s not found";
- private final Client restClient;
+ private final RESTService restClient;
@Inject
- public MavenCentralLibraryService(Client restClient) {
+ public MavenCentralLibraryService(@Named(ServiceConsts.MAVEN_SEARCH_API) RESTService restClient) {
this.restClient = restClient;
}
@@ -56,21 +58,13 @@
private LibraryDTO getMavenLibrary(String groupId, String artifactId, String version) {
final String query = and(artifactQuery(artifactId), groupQuery(groupId), versionQuery(version), jarPackage());
- final String mavenApiSearchUri = String.format(MAVEN_SEARCH_API_QUERY_FORMAT, MAVEN_SEARCH_API, query);
- log.trace("Calling maven api using the following uri: {}", mavenApiSearchUri);
- final Response apiResponse = restClient.target(URI.create(mavenApiSearchUri))
- .request()
- .get();
- if (apiResponse.getStatusInfo().getFamily() != Response.Status.Family.SUCCESSFUL) {
- log.error("Can not get artifact info from maven central due to: " + apiResponse.getStatusInfo().getReasonPhrase());
- throw new DlabException("Can not get artifact info from maven central due to: " + apiResponse.getStatusInfo().getReasonPhrase());
- }
- final MavenSearchArtifactResponse response = apiResponse.readEntity(MavenSearchArtifactResponse.class);
- return response.getArtifacts()
+ return restClient.get(URI.create(String.format(SEARCH_API_QUERY_FORMAT, query)),
+ MavenSearchArtifactResponse.class)
+ .getArtifacts()
.stream()
.findFirst()
- .map(artifact -> new LibraryDTO(String.join(":", groupId, artifactId), version))
- .orElseThrow(() -> new ResourceNotFoundException(String.format(LIB_NOT_FOUND_MSG, artifactId, groupId,
+ .map(artifact -> new LibraryDTO(join(":", groupId, artifactId), version))
+ .orElseThrow(() -> new ResourceNotFoundException(format(LIB_NOT_FOUND_MSG, artifactId, groupId,
version)));
}
@@ -91,7 +85,7 @@
}
private String and(String... strings) {
- return String.join("+AND+", strings);
+ return join("+AND+", strings);
}
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/UserGroupServiceImpl.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/UserGroupServiceImpl.java
index 7325db5..98e76b6 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/UserGroupServiceImpl.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/service/impl/UserGroupServiceImpl.java
@@ -50,6 +50,16 @@
}
@Override
+ public void updateGroup(String group, Set<String> roleIds, Set<String> users) {
+ log.debug("Updating users for group {}: {}", group, users);
+ userGroupDao.updateUsers(group, users);
+ log.debug("Removing group {} from existing roles", group);
+ userRoleDao.removeGroupWhenRoleNotIn(group, roleIds);
+ log.debug("Adding group {} to roles {}", group, roleIds);
+ userRoleDao.addGroupToRole(Collections.singleton(group), roleIds);
+ }
+
+ @Override
public void addUsersToGroup(String group, Set<String> users) {
userGroupDao.addUsers(group, users);
}
diff --git a/services/self-service/src/main/java/com/epam/dlab/backendapi/util/RequestBuilder.java b/services/self-service/src/main/java/com/epam/dlab/backendapi/util/RequestBuilder.java
index 726e1ab..b4afd19 100644
--- a/services/self-service/src/main/java/com/epam/dlab/backendapi/util/RequestBuilder.java
+++ b/services/self-service/src/main/java/com/epam/dlab/backendapi/util/RequestBuilder.java
@@ -417,8 +417,8 @@
computationalCreate = (T) newResourceSysBaseDTO(userInfo, SparkComputationalCreateAzure.class)
.withDataEngineInstanceCount(form.getDataEngineInstanceCount())
.withDataEngineMasterSize(form.getDataEngineInstanceShape())
- .withDataEngineSlaveSize(form.getDataEngineInstanceShape());
-
+ .withDataEngineSlaveSize(form.getDataEngineInstanceShape())
+ .withConfig(form.getConfig());
if (settingsDAO.isAzureDataLakeEnabled()) {
((SparkComputationalCreateAzure) computationalCreate)
.withAzureClientId(settingsDAO.getAzureDataLakeClientId())
@@ -433,7 +433,8 @@
computationalCreate = (T) newResourceSysBaseDTO(userInfo, SparkComputationalCreateGcp.class)
.withDataEngineInstanceCount(form.getDataEngineInstanceCount())
.withDataEngineMasterSize(form.getDataEngineInstanceShape())
- .withDataEngineSlaveSize(form.getDataEngineInstanceShape());
+ .withDataEngineSlaveSize(form.getDataEngineInstanceShape())
+ .withConfig(form.getConfig());
break;
default:
throw new IllegalArgumentException(UNSUPPORTED_CLOUD_PROVIDER_MESSAGE + cloudProvider());
diff --git a/services/self-service/src/main/resources/quartz.properties b/services/self-service/src/main/resources/quartz.properties
new file mode 100644
index 0000000..2fa3cc9
--- /dev/null
+++ b/services/self-service/src/main/resources/quartz.properties
@@ -0,0 +1,23 @@
+#
+# /*
+# * Copyright (c) 2018, EPAM SYSTEMS INC
+# *
+# * Licensed under the Apache License, Version 2.0 (the "License");
+# * you may not use this file except in compliance with the License.
+# * You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+#
+org.quartz.jobStore.class=com.novemberain.quartz.mongodb.MongoDBJobStore
+org.quartz.jobStore.collectionPrefix=scheduler
+org.quartz.jobStore.isClustered=true
+org.quartz.scheduler.instanceId=AUTO
+org.quartz.scheduler.instanceName=dlab
+org.quartz.threadPool.threadCount=1
\ No newline at end of file
diff --git a/services/self-service/src/main/resources/webapp/.editorconfig b/services/self-service/src/main/resources/webapp/.editorconfig
index 6e87a00..84b53d5 100644
--- a/services/self-service/src/main/resources/webapp/.editorconfig
+++ b/services/self-service/src/main/resources/webapp/.editorconfig
@@ -1,4 +1,18 @@
# Editor configuration, see http://editorconfig.org
+# Copyright (c) 2019, EPAM SYSTEMS INC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
root = true
[*]
diff --git a/services/self-service/src/main/resources/webapp/src/app/access-denied/access-denied.component.scss b/services/self-service/src/main/resources/webapp/src/app/access-denied/access-denied.component.scss
index 183a760..555623d 100644
--- a/services/self-service/src/main/resources/webapp/src/app/access-denied/access-denied.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/access-denied/access-denied.component.scss
@@ -1,3 +1,20 @@
+/***************************************************************************
+
+Copyright (c) 2019, EPAM SYSTEMS INC
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+****************************************************************************/
.no-access-page {
position: absolute;
top: 0;
@@ -36,4 +53,4 @@
}
}
}
-}
\ No newline at end of file
+}
diff --git a/services/self-service/src/main/resources/webapp/src/app/access-denied/access-denied.component.ts b/services/self-service/src/main/resources/webapp/src/app/access-denied/access-denied.component.ts
index d6e17bb..c41244c 100644
--- a/services/self-service/src/main/resources/webapp/src/app/access-denied/access-denied.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/access-denied/access-denied.component.ts
@@ -1,3 +1,18 @@
+/**
+* Copyright (c) 2018, EPAM SYSTEMS INC
+*
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
import { Component, OnInit } from '@angular/core';
@Component({
diff --git a/services/self-service/src/main/resources/webapp/src/app/core/services/applicationServiceFacade.service.ts b/services/self-service/src/main/resources/webapp/src/app/core/services/applicationServiceFacade.service.ts
index 77e733f..78a55a8 100644
--- a/services/self-service/src/main/resources/webapp/src/app/core/services/applicationServiceFacade.service.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/core/services/applicationServiceFacade.service.ts
@@ -386,6 +386,13 @@
this.getRequestOptions(true, true));
}
+ public BuildGetActiveSchcedulersData(param): Observable<Response> {
+ return this.buildRequest(RequestMethod.Get,
+ this.requestRegistry.Item(ApplicationServiceFacade.SCHEDULER) + param,
+ null,
+ this.getRequestOptions(true, true));
+ }
+
public buildGetActiveUsers(): Observable<Response> {
return this.buildRequest(RequestMethod.Get,
this.requestRegistry.Item(ApplicationServiceFacade.ACTIVE_LIST),
@@ -463,6 +470,13 @@
this.getRequestOptions(false, true));
}
+ public buildUpdateGroupData(data): Observable<Response> {
+ return this.buildRequest(RequestMethod.Put,
+ this.requestRegistry.Item(ApplicationServiceFacade.GROUPS),
+ data,
+ this.getRequestOptions(false, true));
+ }
+
public buildSetupRolesForGroup(data): Observable<Response> {
return this.buildRequest(RequestMethod.Put,
this.requestRegistry.Item(ApplicationServiceFacade.GROUP_ROLE),
@@ -526,7 +540,7 @@
this.requestRegistry.Add(ApplicationServiceFacade.LOGIN, '/api/user/login');
this.requestRegistry.Add(ApplicationServiceFacade.LOGOUT, '/api/user/logout');
this.requestRegistry.Add(ApplicationServiceFacade.AUTHORIZE, '/api/user/authorize');
- this.requestRegistry.Add(ApplicationServiceFacade.ACTIVE_LIST, '/api/environment/user/active');
+ this.requestRegistry.Add(ApplicationServiceFacade.ACTIVE_LIST, '/api/environment/user');
this.requestRegistry.Add(ApplicationServiceFacade.FULL_ACTIVE_LIST, '/api/environment/all');
this.requestRegistry.Add(ApplicationServiceFacade.ENV, '/api/environment');
diff --git a/services/self-service/src/main/resources/webapp/src/app/core/services/healthStatus.service.ts b/services/self-service/src/main/resources/webapp/src/app/core/services/healthStatus.service.ts
index a3e1341..869d933 100644
--- a/services/self-service/src/main/resources/webapp/src/app/core/services/healthStatus.service.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/core/services/healthStatus.service.ts
@@ -37,7 +37,7 @@
return this._statusData.asObservable();
}
- reloadInitialStatusData() {
+ public reloadInitialStatusData() {
this.getEnvironmentHealthStatus().subscribe(
(res: GeneralEnvironmentStatus) => {
this._statusData.next(res);
@@ -160,4 +160,8 @@
.map(response => response)
.catch(error => error);
}
+
+ public resetStatusValue() {
+ this._statusData.next(<GeneralEnvironmentStatus>{});
+ }
}
diff --git a/services/self-service/src/main/resources/webapp/src/app/core/services/librariesInstallation.service.ts b/services/self-service/src/main/resources/webapp/src/app/core/services/librariesInstallation.service.ts
index 3888796..d2d85a0 100644
--- a/services/self-service/src/main/resources/webapp/src/app/core/services/librariesInstallation.service.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/core/services/librariesInstallation.service.ts
@@ -47,7 +47,7 @@
}
public getAvailableDependencies(data): Observable<{}> {
- let body = `/maven?artifact=${data}`;
+ const body = `/maven?artifact=${data}`;
return this.applicationServiceFacade
.buildGetAvailableDependenciest(body)
diff --git a/services/self-service/src/main/resources/webapp/src/app/core/services/rolesManagement.service.ts b/services/self-service/src/main/resources/webapp/src/app/core/services/rolesManagement.service.ts
index 60e1c82..9abc454 100644
--- a/services/self-service/src/main/resources/webapp/src/app/core/services/rolesManagement.service.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/core/services/rolesManagement.service.ts
@@ -39,7 +39,7 @@
.map(response => response.json())
.catch(ErrorUtils.handleServiceError);
}
-
+
public setupNewGroup(data): Observable<{}> {
return this.applicationServiceFacade
.buildSetupNewGroup(data)
@@ -47,12 +47,20 @@
.catch(ErrorUtils.handleServiceError);
}
+ public updateGroup(data): Observable<{}> {
+ return this.applicationServiceFacade
+ .buildUpdateGroupData(data)
+ .map(response => response)
+ .catch(ErrorUtils.handleServiceError);
+ }
+
public setupRolesForGroup(data): Observable<{}> {
return this.applicationServiceFacade
.buildSetupRolesForGroup(data)
.map(response => response)
.catch(ErrorUtils.handleServiceError);
}
+
public setupUsersForGroup(data): Observable<{}> {
return this.applicationServiceFacade
.buildSetupUsersForGroup(data)
@@ -77,4 +85,4 @@
.map(response => response)
.catch(ErrorUtils.handleServiceError);
}
-}
\ No newline at end of file
+}
diff --git a/services/self-service/src/main/resources/webapp/src/app/core/services/scheduler.service.ts b/services/self-service/src/main/resources/webapp/src/app/core/services/scheduler.service.ts
index 3dd7076..b2fd1d1 100644
--- a/services/self-service/src/main/resources/webapp/src/app/core/services/scheduler.service.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/core/services/scheduler.service.ts
@@ -49,4 +49,12 @@
.map(response => response)
.catch(ErrorUtils.handleServiceError);
}
+
+ public getActiveSchcedulersData(offset: number): Observable<{}> {
+ const param = `/active?minuteOffset=${offset}`;
+ return this.applicationServiceFacade
+ .BuildGetActiveSchcedulersData(param)
+ .map(response => response.json())
+ .catch(ErrorUtils.handleServiceError);
+ }
}
diff --git a/services/self-service/src/main/resources/webapp/src/app/core/services/userAccessKey.service.ts b/services/self-service/src/main/resources/webapp/src/app/core/services/userAccessKey.service.ts
index 499688d..65158c2 100644
--- a/services/self-service/src/main/resources/webapp/src/app/core/services/userAccessKey.service.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/core/services/userAccessKey.service.ts
@@ -37,10 +37,8 @@
this.checkUserAccessKey().subscribe(
response => {
this._accessKeyEmitter.next(response);
- console.log('initial User Access Key Check OK');
}, error => {
this._accessKeyEmitter.next(error);
- console.error('initial User Access Key Check', 'Error retrieving access key');
});
}
@@ -87,4 +85,8 @@
.map(response => response)
.catch(ErrorUtils.handleServiceError);
}
+
+ public resetUserAccessKey() {
+ this._accessKeyEmitter.next(<any>{});
+ }
}
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status-grid/health-status-grid.component.html b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status-grid/health-status-grid.component.html
index a9674d6..a6a78fa 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status-grid/health-status-grid.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status-grid/health-status-grid.component.html
@@ -23,7 +23,7 @@
<th class="th_status">Status</th>
<th class="th_actions">Action</th>
</tr>
- <tr *ngFor="let env of environmentsHealthStatuses">
+ <tr *ngFor="let env of environmentsHealthStatuses" class="dashboard_table_body">
<td>{{env.type}}</td>
<td>{{env.resource_id}}</td>
<td class="status" ngClass="{{env.status || ''}}">{{env.status}}</td>
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status-grid/health-status-grid.component.scss b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status-grid/health-status-grid.component.scss
index 991b927..41e80cd 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status-grid/health-status-grid.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status-grid/health-status-grid.component.scss
@@ -14,4 +14,10 @@
See the License for the specific language governing permissions and
limitations under the License.
-****************************************************************************/
\ No newline at end of file
+****************************************************************************/
+
+.dashboard_table_body {
+ td:first-child {
+ cursor: default !important;
+ }
+}
\ No newline at end of file
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.component.html b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.component.html
index 653ab5b..a85b180 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.component.html
@@ -19,7 +19,7 @@
<div class="sub-nav base-retreat">
<div *ngIf="healthStatus?.admin" class="admin-group">
<button mat-raised-button class="butt ssn" (click)="openManageRolesDialog()">
- <i class="material-icons"></i>Roles
+ <i class="material-icons"></i>Manage roles
</button>
<button mat-raised-button class="butt ssn" (click)="openSsnMonitorDialog()">
<i class="material-icons"></i>SSN Monitor
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.component.ts b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.component.ts
index 1add5f4..eb69e01 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.component.ts
@@ -109,7 +109,7 @@
openSsnMonitorDialog() {
this.healthStatusService.getSsnMonitorData()
- .subscribe(data => this.ssnMonitorDialog.open({ isHeader: false, isFooter: false }, data));
+ .subscribe(data => this.ssnMonitorDialog.open({ isFooter: false }, data));
}
openManageRolesDialog() {
@@ -146,6 +146,7 @@
result.status === HTTP_STATUS_CODES.OK
&& res.status === HTTP_STATUS_CODES.NO_CONTENT
&& this.toastr.success('Budget limits updated!', 'Success!', { toastLife: 5000 });
+ this.buildGrid();
});
}, error => this.toastr.error(error.message, 'Oops!', { toastLife: 5000 }));
}
@@ -159,17 +160,10 @@
}, () => this.toastr.error('Group creation failed!', 'Oops!', { toastLife: 5000 }));
break;
case 'update':
- if ($event.type === 'roles') {
- this.rolesService.setupRolesForGroup($event.value).subscribe(res => {
- this.toastr.success('Roles list successfully updated!', 'Success!', { toastLife: 5000 });
- this.getGroupsData();
- }, () => this.toastr.error('Failed roles list updating!', 'Oops!', { toastLife: 5000 }));
- } else if ($event.type === 'users') {
- this.rolesService.setupUsersForGroup($event.value).subscribe(res => {
- this.toastr.success('Users list successfully updated!', 'Success!', { toastLife: 5000 });
- this.getGroupsData();
- }, () => this.toastr.error('Failed users list updating!', 'Oops!', { toastLife: 5000 }));
- }
+ this.rolesService.updateGroup($event.value).subscribe(res => {
+ this.toastr.success('Group data successfully updated!', 'Success!', { toastLife: 5000 });
+ this.getGroupsData();
+ }, () => this.toastr.error('Failed group data updating!', 'Oops!', { toastLife: 5000 }));
break;
case 'delete':
if ($event.type === 'users') {
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.module.ts b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.module.ts
index 737d102..f509b44 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.module.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/health-status.module.ts
@@ -36,12 +36,11 @@
import { HealthStatusGridComponent } from './health-status-grid/health-status-grid.component';
import { SsnMonitorComponent } from './ssn-monitor/ssn-monitor.component';
-import { ManageRolesGroupsComponent, ConfirmDeleteUserAccountDialog } from './manage-roles-groups/manage-roles-groups.component';
+import { ManageRolesGroupsComponent, ConfirmDeleteUserAccountDialogComponent } from './manage-roles-groups/manage-roles-groups.component';
export * from './environment-status.model';
@NgModule({
-
imports: [
CommonModule,
FormsModule,
@@ -62,12 +61,12 @@
BackupDilogComponent,
ManageEnvironmentComponent,
ConfirmActionDialogComponent,
- ConfirmDeleteUserAccountDialog,
+ ConfirmDeleteUserAccountDialogComponent,
SsnMonitorComponent,
ManageRolesGroupsComponent,
HealthStatusGridComponent
],
- entryComponents: [ConfirmActionDialogComponent, ConfirmDeleteUserAccountDialog],
+ entryComponents: [ConfirmActionDialogComponent, ConfirmDeleteUserAccountDialogComponent],
exports: [HealthStatusComponent, HealthStatusGridComponent, GroupNameValidationDirective]
})
export class HealthStatusModule {}
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.html b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.html
index 2b8b1ee..200dd61 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.html
@@ -27,34 +27,41 @@
<mat-list>
<mat-list-item class="list-header">
<div class="username">User</div>
- <div class="quotes">Limit</div>
+ <div class="quotes" *ngIf="DICTIONARY.cloud_provider !== 'gcp'">Limit</div>
<div class="action">Actions</div>
</mat-list-item>
<div class="scrolling-content" id="scrolling" formArrayName="users">
<mat-list-item *ngFor="let item of usersEnvironments.controls; let i=index" [formGroupName]="i" class="list-item">
<div class="username ellipsis">{{ manageUsersForm.controls['users'].controls[i].value['name'] }}</div>
- <div class="quotes">
+ <div class="quotes" *ngIf="DICTIONARY.cloud_provider !== 'gcp'">
<input type="number" min="0" placeholder="Enter limit, in USD" formControlName="budget">
<span class="danger_color" *ngIf="!manageUsersForm?.controls['users'].controls[i].controls['budget'].valid && !manageUsersForm?.controls['users'].controls[i].controls['budget'].hasError('overrun')">Only positive integers are allowed</span>
<span class="danger_color" *ngIf="manageUsersForm?.controls['users'].controls[i].controls['budget'].hasError('overrun')">Per-user quotes cannot be greater than total budget</span>
</div>
<div class="action">
- <span matTooltip="Stop" matTooltipPosition="above" (click)="applyAction('stop', item)"><i class="material-icons">pause_circle_outline</i></span>
+ <span *ngIf=" manageUsersForm.controls['users'].controls[i].value['status'] === 'ACTIVE'; else not_active" matTooltip="Stop" matTooltipPosition="above" (click)="applyAction('stop', item)"><i class="material-icons">pause_circle_outline</i></span>
+
+ <ng-template #not_active>
+ <span class="disabled" matTooltip="User's environment is not active" matTooltipPosition="above">
+ <i class="material-icons">pause_circle_outline</i>
+ </span>
+ </ng-template>
+
<span matTooltip="Terminate" matTooltipPosition="above" (click)="applyAction('terminate', item)"><i class="material-icons">phonelink_off</i></span>
- </div>
+ </div>
</mat-list-item>
</div>
- <div class="control-group total-budget">
+ <div class="control-group total-budget" *ngIf="DICTIONARY.cloud_provider !== 'gcp'">
<label class="label">Total budget</label>
<div class="control">
<input type="number" formControlName="total" placeholder="Enter total budget, in USD">
<span class="danger_color" *ngIf="manageUsersForm?.controls['total'].hasError('overrun')">Total budget cannot be lower than a sum of users quotes</span>
</div>
</div>
- <div class="text-center m-top-30">
+ <div class="text-center m-top-30" *ngIf="DICTIONARY.cloud_provider !== 'gcp'">
<button mat-raised-button type="button" (click)="bindDialog.close()" class="butt action">Cancel</button>
<button mat-raised-button type="submit" [disabled]="!manageUsersForm.valid"
- class="butt butt-success" [ngClass]="{'not-allowed': !manageUsersForm.valid}">Create</button>
+ class="butt butt-success" [ngClass]="{'not-allowed': !manageUsersForm.valid}">Apply</button>
</div>
</mat-list>
</form>
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.scss b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.scss
index d45055d..c10c2e8 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.scss
@@ -30,7 +30,7 @@
}
.scrolling-content {
max-height: 300px;
- min-height: 140px;
+ min-height: 85px;
overflow-y: auto;
}
.username {
@@ -39,9 +39,11 @@
.quotes {
width: 40%;
margin-right: 10px;
+ position: relative;
.danger_color {
position: absolute;
- padding-top: 2px;
+ left: 0;
+ top: 36px;
}
}
.action {
@@ -56,6 +58,14 @@
font-size: 20px;
}
}
+ .disabled {
+ cursor: not-allowed !important;
+ pointer-events: all;
+ opacity: .6;
+ &:hover {
+ color: #6b8299;
+ }
+ }
}
.total-budget {
border-top: 1px solid #edf1f5;
@@ -65,7 +75,8 @@
width: 56%;
.danger_color {
position: absolute;
- padding-top: 2px;
+ left: 0;
+ bottom: -20px;
}
}
.label {
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.ts b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.ts
index b294283..212a3b6 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-environment/manage-environment-dilog.component.ts
@@ -52,7 +52,7 @@
this.manageUsersForm.setControl('users',
this._fb.array((this.usersList || []).map((x: any) => this._fb.group({
- name: x.name, budget: [x.budget, [Validators.min(0), this.userValidityCheck.bind(this)]]
+ name: x.name, budget: [x.budget, [Validators.min(0), this.userValidityCheck.bind(this)]], status: x.status
}))));
this.manageUsersForm.controls['total'].setValue(settings.conf_max_budget || null);
this.bindDialog.open(param);
@@ -74,7 +74,7 @@
private initForm(): void {
this.manageUsersForm = this._fb.group({
total: [null, [Validators.min(0), this.totalValidityCheck.bind(this)]],
- users: this._fb.array([this._fb.group({ name: '', budget: null })])
+ users: this._fb.array([this._fb.group({ name: '', budget: null, status: ''})])
});
}
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/group-name-validarion.directive.ts b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/group-name-validarion.directive.ts
index 5e8b754..806d466 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/group-name-validarion.directive.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/group-name-validarion.directive.ts
@@ -25,7 +25,7 @@
providers: [{
multi: true,
provide: NG_VALIDATORS,
- useExisting: forwardRef(() => GroupNameValidationDirective)
+ useExisting: forwardRef(() => GroupNameValidationDirective)
}]
})
export class GroupNameValidationDirective implements Validator {
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.html b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.html
index 40fb57a..0803011 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.html
@@ -16,7 +16,7 @@
-->
-<modal-dialog #bindDialog modalClass="manage-roles-dialog modal-xl">
+<modal-dialog #bindDialog modalClass="manage-roles-dialog modal-xxl">
<modal-header>
<h4 class="modal-title">Manage roles</h4>
</modal-header>
@@ -28,7 +28,7 @@
<mat-horizontal-stepper #stepper *ngIf="stepperView" class="stepper ani">
<mat-step>
<ng-template matStepLabel>Groups</ng-template>
- <div class="inner-step">
+ <div class="inner-step mat-reset">
<input [validator]="groupValidarion()" type="text" placeholder="Enter group name" [(ngModel)]="setupGroup" #setupGroupName="ngModel">
<div class="danger_color" *ngIf="setupGroupName.errors?.patterns && setupGroupName.dirty">Group name can only contain letters, numbers, hyphens and '_'</div>
<div class="danger_color" *ngIf="setupGroupName.errors?.duplicate && setupGroupName.dirty">Group name already exists</div>
@@ -40,19 +40,27 @@
</mat-step>
<mat-step>
<ng-template matStepLabel>Roles</ng-template>
- <div class="inner-step">
+ <div class="inner-step mat-reset">
<div class="selector-wrapper">
<!-- <multi-select-dropdown (selectionChange)="onUpdate($event)" [type]="'role'" [items]="rolesList" [model]="setupRoles"></multi-select-dropdown> -->
<mat-form-field>
<mat-select multiple [compareWith]="compareObjects" name="roles" [(value)]="setupRoles" placeholder="Select roles">
+ <mat-option class="multiple-select" disabled>
+ <a class="select ani" (click)="selectAllOptions(setupRoles, rolesList)">
+ <i class="material-icons">playlist_add_check</i> All
+ </a>
+ <a class="deselect ani" (click)="selectAllOptions(setupRoles)">
+ <i class="material-icons">clear</i> None
+ </a>
+ </mat-option>
<mat-option *ngFor="let role of rolesList" [value]="role">
{{ role }}
</mat-option>
</mat-select>
+ <button class="caret">
+ <i class="material-icons">keyboard_arrow_down</i>
+ </button>
</mat-form-field>
- <button class="caret">
- <i class="material-icons">keyboard_arrow_down</i>
- </button>
</div>
</div>
<div class="text-center m-bott-10">
@@ -63,7 +71,7 @@
</mat-step>
<mat-step>
<ng-template matStepLabel>Users</ng-template>
- <div class="inner-step">
+ <div class="inner-step mat-reset">
<input type="text" placeholder="Enter user login" [(ngModel)]="setupUser">
</div>
<div class="text-center m-bott-10">
@@ -74,81 +82,73 @@
</div>
</mat-step>
</mat-horizontal-stepper>
+ <mat-divider></mat-divider>
<div *ngIf="groupsData.length" class="ani">
- <mat-list>
- <mat-list-item class="list-header">
- <div class="groups">Groups</div>
- <div class="roles">Role</div>
- <div class="users">Users</div>
- </mat-list-item>
- <mat-accordion class="scrolling-content" id="scrolling" [ngClass]="{'stepper-opened': stepperView }">
- <mat-expansion-panel *ngFor="let item of groupsData">
- <mat-expansion-panel-header>
- <div class="groups" matTooltip="{{item.group}}" matTooltipPosition="above" [matTooltipClass]="'break-tooltip'">
- <div class="ellipsis">{{ item.group }}</div>
- </div>
-
- <div class="roles">
- <div *ngIf="item.roles; then roles_list else no_roles"></div>
- <ng-template #roles_list>
- <div class="ellipsis" matTooltip="{{ item.selected_roles }}" matTooltipPosition="above" [matTooltipClass]="'break-tooltip'">
- <span *ngFor="let role of item.roles"><b> ▪ </b>{{ role.description }}</span>
- </div>
- </ng-template>
- <ng-template #no_roles><span class="no-details">no details</span></ng-template>
- </div>
-
- <div class="users ellipsis">
- <div *ngIf="item.users; then users_list else no_users"></div>
-
- <ng-template #users_list>
- <div class="ellipsis" matTooltip="{{item.users}}" matTooltipPosition="above" [matTooltipClass]="'break-tooltip'">
- <span *ngFor="let user of item.users"><b> ▪ </b>{{ user }}</span>
- </div>
- </ng-template>
- <ng-template #no_users><span class="no-details">no details</span></ng-template>
- </div>
- </mat-expansion-panel-header>
-
- <div class="expanded-panel">
- <div class="groups">
- <button mat-raised-button (click)="manageAction('delete', 'group', item)" class="butt">
- <i class="material-icons">clear</i>Delete group
+ <table class="dashboard_table">
+ <tr>
+ <th class="th_name groups">Name</th>
+ <th class="roles">Roles</th>
+ <th class="users">Users</th>
+ <th class="th_actions">Action</th>
+ </tr>
+ <tr *ngFor="let item of groupsData" class="dashboard_table_body filter-row">
+ <td>{{ item.group }}</td>
+ <td class="roles mat-reset">
+ <div class="selector-wrapper-edit">
+ <mat-form-field>
+ <mat-select multiple [compareWith]="compareObjects" name="selected_roles" [(value)]="item.selected_roles" placeholder="Select roles">
+ <mat-option class="multiple-select" disabled>
+ <a class="select ani" (click)="selectAllOptions(item, rolesList, 'selected_roles')">
+ <i class="material-icons">playlist_add_check</i> All
+ </a>
+ <a class="deselect ani" (click)="selectAllOptions(item, null, 'selected_roles')">
+ <i class="material-icons">clear</i> None
+ </a>
+ </mat-option>
+ <mat-option *ngFor="let role of rolesList" [value]="role">
+ {{ role }}
+ </mat-option>
+ </mat-select>
+ <button class="caret ani">
+ <i class="material-icons">keyboard_arrow_down</i>
</button>
- </div>
- <div class="roles">
- <div class="selector-wrapper-edit">
- <!-- <multi-select-dropdown (selectionChange)="onUpdate($event, item)" [type]="'update_roles'" [items]="rolesList" [model]="item.selected_roles"></multi-select-dropdown> -->
-
- <mat-form-field>
- <mat-select multiple [compareWith]="compareObjects" name="selected_roles" [(value)]="item.selected_roles" placeholder="Select roles">
- <mat-option *ngFor="let role of rolesList" [value]="role">
- {{ role }}
- </mat-option>
- </mat-select>
- </mat-form-field>
- <button class="caret ani" (click)="manageAction('update', 'roles', item, item.selected_roles)" [ngClass]="{'not-allowed': !item.selected_roles.length}">
- <i class="material-icons">done</i>
- </button>
- </div>
- </div>
- <div class="users">
- <div *ngFor="let user of item.users" class="users-list">
- <span class="ellipsis" matTooltip="{{ user }}" matTooltipPosition="above" [matTooltipClass]="'break-tooltip'">{{ user }}</span>
- <i class="material-icons" (click)="manageAction('delete', 'users', item, user)">highlight_off</i>
- </div>
- <div class="add-input-block">
- <input type="text" placeholder="Enter user login" [(ngModel)]="manageUser">
- <button class="caret" (click)="manageAction('update', 'users', item, manageUser)" [ngClass]="{'not-allowed': !manageUser}">
- <i class="material-icons">done</i>
- </button>
- </div>
- </div>
+ </mat-form-field>
</div>
- </mat-expansion-panel>
- </mat-accordion>
- </mat-list>
+ </td>
+ <td class="users-list ani">
+ <mat-form-field class="chip-list">
+ <input #user matInput placeholder="Enter user login" pattern="[@.-_0-9a-zA-Z]" (keydown.enter)="addUser(user.value, item); user.value = ''">
+ <button mat-icon-button matSuffix (click)="addUser(user.value, item); user.value = ''">
+ <mat-icon>person_add</mat-icon>
+ </button>
+ </mat-form-field>
+ <div class="list-selected list-container ani">
+ <mat-chip-list>
+ <mat-chip *ngFor="let user of item.users">
+ {{ user }}
+ <a class="material-icons" (click)="removeUser(item.users, user)">clear</a>
+ </mat-chip>
+ </mat-chip-list>
+ </div>
+ </td>
+ <td class="actions">
+ <button mat-icon-button class="reset ani" (click)="manageAction('delete', 'group', item)">
+ <i class="material-icons">close</i>
+ </button>
+
+ <button mat-icon-button class="apply ani" matTooltip="Group cannot be updated without any selected role"
+ matTooltipPosition="above"
+ [matTooltipDisabled]="item.selected_roles.length > 0">
+ <span class="mid" [ngClass]="{ 'not-allowed' : !item.selected_roles.length }"
+ (click)="manageAction('update', 'group', item)">
+ <i class="material-icons">done</i>
+ </span>
+ </button>
+ </td>
+ </tr>
+ </table>
</div>
</div>
</modal-content>
-</modal-dialog>
\ No newline at end of file
+</modal-dialog>
+
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.scss b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.scss
index 449c245..f9ba0eb 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.scss
@@ -16,152 +16,168 @@
****************************************************************************/
+::ng-deep .mat-option:first-child .mat-pseudo-checkbox { display: none; }
+
.caret {
width: 40px;
color: #4ab8dc;
border: none;
border-left: 1px solid #ececec;
background-color: #fff;
+ position: absolute;
+ right: 0;
+ top: 0px;
+ height: 36px;
cursor: pointer;
&.not-allowed {
background-color: #dcdcdc;
}
}
-.manage-roles-dialog {
- .content-box {
- height: 600px;
- padding: 20px 35px 35px;
- }
- .add-group {
- margin-left: 5px;
- }
- .no-details {
- color: #d8d8d8;
- }
- .stepper {
- height: 190px;
- margin-top: 10px;
- .inner-step {
- height: 70px;
- padding: 5px;
- display: flex;
- justify-content: center;
- flex-direction: column;
- text-align: center;
- input {
- width: 450px;
- align-self: center;
- }
- .caret {
- i {
- margin-top: 3px;
- }
- }
- }
- }
- .selector-wrapper {
+
+.content-box {
+ padding: 20px 30px 35px;
+ height: 85vh;
+ overflow-y: auto;
+}
+.no-details {
+ color: #d8d8d8;
+}
+.mat-divider {
+ margin: 10px 0;
+}
+.stepper {
+ height: 190px;
+ margin-top: 10px;
+ .inner-step {
+ height: 70px;
+ padding: 5px;
display: flex;
- align-self: center;
- width: 490px;
- height: 36px;
- padding-left: 10px;
- font-family: 'Open Sans', sans-serif;
- font-size: 15px;
- font-weight: 300;
- box-shadow: 0 3px 1px -2px rgba(0, 0, 0, 0.2), 0 2px 2px 0 rgba(0, 0, 0, 0.14), 0 1px 5px 0 rgba(0, 0, 0, 0.12);
- mat-form-field {
- width: 100%;
- .mat-form-field-wrapper {
- padding-bottom: 0;
- }
+ justify-content: center;
+ flex-direction: column;
+ text-align: center;
+ input {
+ width: 490px;
+ align-self: center;
}
- .dropdown-multiselect {
- width: 100% !important;
- > button {
- padding: 6px 22px;
- }
- }
- }
- .list-header {
- padding: 0 15px;
- }
- .scrolling-content {
- max-height: 450px;
- overflow-x: hidden;
- overflow-y: auto;
- display: block;
- padding: 15px 5px;
- &.stepper-opened {
- height: 250px;
- }
- }
- .roles {
- width: 45%;
- padding: 0 10px;
- .selector-wrapper-edit {
- display: flex;
- justify-content: space-between;
- height: 36px;
- padding-left: 10px;
- box-shadow: 0 3px 1px -2px rgba(0, 0, 0, 0.2), 0 2px 2px 0 rgba(0, 0, 0, 0.14), 0 1px 5px 0 rgba(0, 0, 0, 0.12);
- multi-select-dropdown {
- width: 100%;
- .dropdown-multiselect {
- > button {
- padding: 8px 22px 5px;
- }
- }
- }
- .caret {
- &:hover {
- box-shadow: 0 3px 1px -10px rgba(0, 0, 0, 0.2), 0 2px 1px 0 rgba(0, 0, 0, 0.17), 0 1px 5px 0 rgba(0, 0, 0, 0.12)
- }
- }
- }
- }
- .groups {
- width: 25%;
- }
- .users {
- width: 30%;
- padding: 0 10px;
- .users-list {
- display: flex;
- justify-content: space-between;
- padding: 5px 10px;
- font-family: 'Open Sans', sans-serif;
- font-size: 15px;
- font-weight: 300;
- color: #577289;
+ .caret {
i {
- color: #FF5722;
- font-size: 18px;
- cursor: pointer;
- }
- }
- }
- .expanded-panel {
- display: flex;
- align-items: flex-end;
- .add-input-block {
- display: flex;
- height: 36px;
- padding-right: 0;
- outline: none;
- box-shadow: 0 3px 1px -2px rgba(0, 0, 0, 0.2), 0 2px 2px 0 rgba(0, 0, 0, 0.14), 0 1px 5px 0 rgba(0, 0, 0, 0.12);
- input {
- height: 36px;
- padding: 0;
- padding-left: 10px;
- width: 170px;
- }
- .caret {
- width: 50px;
+ margin-top: 3px;
}
}
}
}
+.selector-wrapper {
+ display: flex;
+ align-self: center;
+ width: 490px;
+ height: 36px;
+ padding-left: 10px;
+ font-family: 'Open Sans', sans-serif;
+ font-size: 15px;
+ font-weight: 300;
+ box-shadow: 0 3px 1px -2px rgba(0, 0, 0, 0.2), 0 2px 2px 0 rgba(0, 0, 0, 0.14), 0 1px 5px 0 rgba(0, 0, 0, 0.12);
+ mat-form-field {
+ width: 100%;
+ .mat-form-field-wrapper {
+ padding-bottom: 0;
+ }
+ .mat-icon {
+ font-size: 20px;
+ }
+ }
+ .dropdown-multiselect {
+ width: 100% !important;
+ > button {
+ padding: 6px 22px;
+ }
+ }
+}
+.list-header {
+ padding: 0 15px;
+}
+.scrolling-content {
+ max-height: 450px;
+ overflow-x: hidden;
+ overflow-y: auto;
+ display: block;
+ padding: 15px 5px;
+ &.stepper-opened {
+ height: 250px;
+ }
+}
+.roles {
+ width: 30%;
+ padding: 0 10px;
+ .selector-wrapper-edit {
+ position: relative;
+ display: flex;
+ justify-content: space-between;
+ height: 36px;
+ padding-left: 10px;
+ background: #fff;
+ box-shadow: 0 3px 1px -2px rgba(0, 0, 0, 0.2), 0 2px 2px 0 rgba(0, 0, 0, 0.14), 0 1px 5px 0 rgba(0, 0, 0, 0.12);
+ multi-select-dropdown {
+ width: 100%;
+ .dropdown-multiselect {
+ > button {
+ padding: 8px 22px 5px;
+ }
+ }
+ }
+ .caret {
+ &:hover {
+ box-shadow: 0 3px 1px -10px rgba(0, 0, 0, 0.2), 0 2px 1px 0 rgba(0, 0, 0, 0.17), 0 1px 5px 0 rgba(0, 0, 0, 0.12)
+ }
+ }
+ }
+}
+.groups {
+ width: 20%;
+}
+.users {
+ width: 30%;
+ padding: 0 10px;
+}
+.users-list {
+ padding: 5px 10px;
+ font-family: 'Open Sans', sans-serif;
+ font-size: 15px;
+ font-weight: 300;
+ color: #577289;
+ position: relative;
+ i {
+ color: #FF5722;
+ font-size: 18px;
+ cursor: pointer;
+ }
+ .list-selected {
+ width: 100%;
+ margin-top: 50px;
+ height: inherit;
+ }
+}
+.expanded-panel {
+ display: flex;
+ align-items: flex-end;
+ .add-input-block {
+ display: flex;
+ height: 36px;
+ padding-right: 0;
+ outline: none;
+ box-shadow: 0 3px 1px -2px rgba(0, 0, 0, 0.2), 0 2px 2px 0 rgba(0, 0, 0, 0.14), 0 1px 5px 0 rgba(0, 0, 0, 0.12);
+ input {
+ height: 36px;
+ padding: 0;
+ padding-left: 10px;
+ width: 170px;
+ }
+ .caret {
+ width: 50px;
+ }
+ }
+}
+
mat-expansion-panel-header {
&.mat-expansion-panel-header {
@@ -181,4 +197,86 @@
font-size: 16px;
font-weight: 300;
}
-}
\ No newline at end of file
+}
+
+.dashboard_table {
+ .dashboard_table_body {
+ .actions {
+ padding: 10px 0;
+ }
+ &.filter-row {
+ .actions {
+ button {
+ background: none;
+ .mid {
+ vertical-align: sub;
+ }
+ }
+ }
+ }
+ }
+ .th_actions {
+ width: 10%;
+ }
+}
+
+.mat-chip:not(.mat-basic-chip) {
+ transition: box-shadow 280ms cubic-bezier(.4,0,.2,1);
+ padding: 7px 0 7px 10px;
+ border-radius: 24px;
+ cursor: default;
+ display: inline-block;
+ position: relative;
+ padding-right: 25px;
+ white-space: nowrap;
+ max-width: 100%;
+ overflow: hidden;
+ text-overflow: ellipsis;
+ margin: 2px !important;
+}
+mat-chip.mat-chip a {
+ position: absolute;
+ right: 15px;
+}
+mat-form-field.chip-list {
+ &.mat-input-container {
+ position: absolute;
+ top: 2px;
+ width: 94%;
+ font-size: 16px;
+ font-weight: 300;
+ }
+}
+
+.mat-raised-button {
+ &.butt {
+ &.butt-success {
+ margin-left: 0;
+ }
+ }
+}
+.multiple-select {
+ border-bottom: 1px solid #dedbdb;
+ padding: 0;
+ a {
+ display: inline-block;
+ width: 50%;
+ padding: 0 15px;
+ vertical-align: middle;
+ color: #575757;
+ cursor: pointer;
+ i {
+ vertical-align: sub;
+ font-size: 20px;
+ }
+ &:hover {
+ color: #4eaf3e;
+ background: #f9fafb;
+ }
+ &.deselect {
+ &:hover {
+ color: #f1696e;
+ }
+ }
+ }
+}
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.ts b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.ts
index 85634c5..5171eea 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/manage-roles-groups/manage-roles-groups.component.ts
@@ -16,22 +16,20 @@
****************************************************************************/
-import { Component, OnInit, ViewChild, Output, EventEmitter, ViewEncapsulation, Inject } from '@angular/core';
-import { ValidatorFn, FormControl, AbstractControl } from '@angular/forms';
+import { Component, OnInit, ViewChild, Output, EventEmitter, Inject } from '@angular/core';
+import { ValidatorFn, FormControl, NgModel } from '@angular/forms';
import { MatDialog, MatDialogRef, MAT_DIALOG_DATA } from '@angular/material';
import { DICTIONARY } from '../../../dictionary/global.dictionary';
@Component({
selector: 'dlab-manage-roles-groups',
templateUrl: './manage-roles-groups.component.html',
- styleUrls: ['./manage-roles-groups.component.scss'],
- encapsulation: ViewEncapsulation.None
+ styleUrls: ['../../resources/resources-grid/resources-grid.component.css', './manage-roles-groups.component.scss']
})
-export class ManageRolesGroupsComponent {
+export class ManageRolesGroupsComponent implements OnInit {
readonly DICTIONARY = DICTIONARY;
public groupsData: Array<any> = [];
-
public roles: Array<any> = [];
public rolesList: Array<string> = [];
public setupGroup: string = '';
@@ -63,39 +61,47 @@
public onUpdate($event) {
if ($event.type === 'role') {
- this.setupRoles = $event.model
+ this.setupRoles = $event.model;
} else {
this.updatedRoles = $event.model;
}
$event.$event.preventDefault();
}
+ public selectAllOptions(item, values, byKey?) {
+ byKey ? (item[byKey] = values ? values : []) : this.setupRoles = values ? values : [];
+ }
+
public manageAction(action: string, type: string, item?: any, value?) {
if (action === 'create') {
this.manageRolesGroupAction.emit(
{ action, type, value: {
name: this.setupGroup,
- users: this.setupUser ? this.setupUser.split(',').map(item => item.trim()) : [],
+ users: this.setupUser ? this.setupUser.split(',').map(elem => elem.trim()) : [],
roleIds: this.extractIds(this.roles, this.setupRoles)
}
});
this.stepperView = false;
} else if (action === 'delete') {
- let data = (type === 'users') ? {group: item.group, user: value} : {group: item.group, id: item};
- const dialogRef: MatDialogRef<ConfirmDeleteUserAccountDialog> = this.dialog.open(ConfirmDeleteUserAccountDialog, { data: data, width: '550px' });
+ const data = (type === 'users') ? {group: item.group, user: value} : {group: item.group, id: item};
+ const dialogRef: MatDialogRef<ConfirmDeleteUserAccountDialogComponent> = this.dialog.open(
+ ConfirmDeleteUserAccountDialogComponent,
+ { data: data, width: '550px' }
+ );
dialogRef.afterClosed().subscribe(result => {
if (result) {
- let emitValue = (type === 'users') ? {action, type, id: item.name, value: { user: value, group: item.group }} : {action, type, id: item.name, value: item.group} ;
+ const emitValue = (type === 'users')
+ ? {action, type, id: item.name, value: { user: value, group: item.group }}
+ : {action, type, id: item.name, value: item.group} ;
this.manageRolesGroupAction.emit(emitValue);
}
});
} else if (action === 'update') {
- let source = (type === 'roles')
- ? { group: item.group, roleIds: this.extractIds(this.roles, item.selected_roles) }
- : { group: item.group, users: value.split(',').map(item => item.trim())
- }
- this.manageRolesGroupAction.emit({action, type, value: source});
+ this.manageRolesGroupAction.emit({action, type, value: {
+ name: item.group,
+ roleIds: this.extractIds(this.roles, item.selected_roles),
+ users: item.users || [] }});
}
this.resetDialog();
}
@@ -117,8 +123,8 @@
public groupValidarion(): ValidatorFn {
- let duplicateList = this.groupsData.map(item => item.group)
- return <ValidatorFn>((control:FormControl) => {
+ const duplicateList = this.groupsData.map(item => item.group);
+ return <ValidatorFn>((control: FormControl) => {
if (control.value && duplicateList.includes(this.delimitersFiltering(control.value)))
return { duplicate: true };
@@ -129,7 +135,7 @@
});
}
- compareObjects(o1: any, o2: any): boolean {
+ public compareObjects(o1: any, o2: any): boolean {
return o1.toLowerCase() === o2.toLowerCase();
}
@@ -145,6 +151,16 @@
this.setupRoles = [];
this.updatedRoles = [];
}
+
+ public removeUser(list, item): void {
+ list.splice(list.indexOf(item), 1);
+ }
+
+ public addUser(value: string, item): void {
+ if (value && value.trim()) {
+ item.users instanceof Array ? item.users.push(value.trim()) : item.users = [value.trim()];
+ }
+ }
}
@@ -165,9 +181,9 @@
.content { color: #718ba6; padding: 20px 50px; font-size: 14px; font-weight: 400 }
`]
})
-export class ConfirmDeleteUserAccountDialog {
+export class ConfirmDeleteUserAccountDialogComponent {
constructor(
- public dialogRef: MatDialogRef<ConfirmDeleteUserAccountDialog>,
+ public dialogRef: MatDialogRef<ConfirmDeleteUserAccountDialogComponent>,
@Inject(MAT_DIALOG_DATA) public data: any
) { }
-}
\ No newline at end of file
+}
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.html b/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.html
index 8a0d57b..94f7564 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.html
@@ -104,6 +104,9 @@
</div>
</mat-tab>
</mat-tab-group>
+ <div class="text-center">
+ <button type="button" class="butt" mat-raised-button (click)="close()">Close</button>
+ </div>
</div>
<div class="info message" *ngIf="isEmpty(monitorData)">
No ssn monitor data available
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.scss b/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.scss
index 25523fd..a4d6f9d 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.scss
@@ -17,13 +17,22 @@
****************************************************************************/
.ssn-monitor-dialog {
+ .content-box {
+ padding-top: 10px !important;
+ }
.ssn-info {
min-height: 400px;
max-height: 500px;
.scrolling-content {
- max-height: 400px;
+ max-height: 310px;
overflow-y: auto;
}
+ .text-center {
+ position: absolute;
+ bottom: 20px;
+ left: 0;
+ right: 0;
+ }
}
.mat-list-item-content {
display: flex;
diff --git a/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.ts b/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.ts
index 560aed4..b7d5e0c 100644
--- a/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/health-status/ssn-monitor/ssn-monitor.component.ts
@@ -40,6 +40,9 @@
this.monitorData = data || {};
this.bindDialog.open(param);
}
+ public close(param, data): void {
+ this.bindDialog.close();
+ }
public isEmpty(obj) {
if (obj) return Object.keys(obj).length === 0;
diff --git a/services/self-service/src/main/resources/webapp/src/app/management/management-grid/management-grid.component.html b/services/self-service/src/main/resources/webapp/src/app/management/management-grid/management-grid.component.html
index b5443d8..e2406b7 100644
--- a/services/self-service/src/main/resources/webapp/src/app/management/management-grid/management-grid.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/management/management-grid/management-grid.component.html
@@ -26,7 +26,7 @@
<th class="th_actions">Action</th>
</tr>
- <tr *ngFor="let env of allEnvironmentData">
+ <tr *ngFor="let env of allEnvironmentData" class="dashboard_table_body">
<td>{{ env.user }}</td>
<td [ngClass]="{'capitalize': env.name === 'edge node'}">{{ env.name }}</td>
<td class="status" ngClass="{{env.status || ''}}">{{env.status}}</td>
diff --git a/services/self-service/src/main/resources/webapp/src/app/management/management-grid/management-grid.component.scss b/services/self-service/src/main/resources/webapp/src/app/management/management-grid/management-grid.component.scss
index 8fdb72f..badec12 100644
--- a/services/self-service/src/main/resources/webapp/src/app/management/management-grid/management-grid.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/management/management-grid/management-grid.component.scss
@@ -25,7 +25,12 @@
.th_status {
width: 12% !important;
}
+ .dashboard_table_body {
+ td:first-child {
+ cursor: default;
+ }
+ }
}
.source .resource-wrap .resource-name .detailed-link {
cursor: default !important;
-}
+}
\ No newline at end of file
diff --git a/services/self-service/src/main/resources/webapp/src/app/reporting/reporting-grid/reporting-grid.component.html b/services/self-service/src/main/resources/webapp/src/app/reporting/reporting-grid/reporting-grid.component.html
index f74f7c0..e54e7ad 100644
--- a/services/self-service/src/main/resources/webapp/src/app/reporting/reporting-grid/reporting-grid.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/reporting/reporting-grid/reporting-grid.component.html
@@ -63,7 +63,7 @@
</tr>
<ng-template [ngIf]="reportData">
- <tr *ngFor="let line of reportData">
+ <tr *ngFor="let line of reportData" class="dashboard_table_body">
<td class="user-field" *ngIf="full_report">{{ line.user }}</td>
<td class="env_name">{{ line[DICTIONARY.billing.dlabId] }}</td>
<td>{{ line[DICTIONARY.billing.resourceType] }}</td>
diff --git a/services/self-service/src/main/resources/webapp/src/app/reporting/reporting-grid/reporting-grid.component.scss b/services/self-service/src/main/resources/webapp/src/app/reporting/reporting-grid/reporting-grid.component.scss
index a6f1da3..2989412 100644
--- a/services/self-service/src/main/resources/webapp/src/app/reporting/reporting-grid/reporting-grid.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/reporting/reporting-grid/reporting-grid.component.scss
@@ -51,6 +51,18 @@
width: 8%;
}
}
+.dashboard_table_body {
+ td:first-child {
+ cursor: default;
+ }
+ .dropdown-multiselect {
+ button {
+ font-size: 14px;
+ height: 34px;
+ padding: 7px 20px;
+ }
+ }
+}
@media screen and (max-width: 1280px) {
.dashboard_table.reporting {
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/computational/cluster-details/cluster-details.component.html b/services/self-service/src/main/resources/webapp/src/app/resources/computational/cluster-details/cluster-details.component.html
index 38abbcf..e4cff8b 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/computational/cluster-details/cluster-details.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/computational/cluster-details/cluster-details.component.html
@@ -75,7 +75,7 @@
</p>
</div>
- <div class="checkbox-group" *ngIf="resource.image === 'docker.dlab-dataengine' && resource.status === 'running'">
+ <div class="checkbox-group" *ngIf="resource.image === 'docker.dlab-dataengine' && resource.status === 'running' && environment.image !== 'docker.dlab-zeppelin'">
<label>
<input #configurationNode type="checkbox" (change)="selectConfiguration()"/> Cluster configurations
</label>
@@ -88,6 +88,11 @@
</form>
</div>
</div>
+ <div *ngIf="environment.image === 'docker.dlab-zeppelin' && resource.status === 'running'">
+ <small>Spark default configuration for Apache Zeppelin can not be changed from DLab UI. Currently it can be done directly through Apache Zeppelin interpreter menu.
+ For more details please refer for Apache Zeppelin <a href="https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/overview.html" target="_blank">official documentation</a>.
+ </small>
+ </div>
<div class="text-center m-top-30" *ngIf="configuration?.nativeElement['checked']">
<button mat-raised-button type="button" (click)="bindDialog.close()" class="butt action">Cancel</button>
<button mat-raised-button type="submit" [disabled]="!configurationForm.valid"
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/computational/cluster-details/cluster-details.component.scss b/services/self-service/src/main/resources/webapp/src/app/resources/computational/cluster-details/cluster-details.component.scss
index 1d74914..5aa73e9 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/computational/cluster-details/cluster-details.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/computational/cluster-details/cluster-details.component.scss
@@ -36,7 +36,6 @@
opacity: 0;
visibility: hidden;
text-align: left;
- transition: all .35s linear .2s;
position: relative;
&.show {
height: 200px;
@@ -45,6 +44,9 @@
textarea {
height: 100%;
resize: none;
+ font-size: 14px;
+ line-height: 1.5;
+ font-family: Consolas, monospace;
}
span {
.danger_color {
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.html b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.html
index da1325b..1eb5bde 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.html
@@ -46,7 +46,7 @@
</div>
</div>
- <div class="control-group" *ngIf="model.selectedImage?.image">
+ <div class="control-group alias-name" *ngIf="model.selectedImage?.image">
<label class="label">Cluster alias</label>
<div class="control">
<input [class.danger_field]="computationalResourceExist || !resourceForm?.controls['cluster_alias_name'].valid
@@ -90,7 +90,7 @@
</div>
</div>
- <div class="checkbox-group" *ngIf="PROVIDER === 'gcp' && model.selectedImage?.image === 'docker.dlab-dataengine-service'">
+ <div class="checkbox-group m-top-30" *ngIf="PROVIDER === 'gcp' && model.selectedImage?.image === 'docker.dlab-dataengine-service'">
<label>
<input #preemptibleNode type="checkbox" (change)="selectPreemptibleNodes($event)" /> Preemptible nodes
</label>
@@ -110,7 +110,7 @@
</div>
</div>
- <div class="checkbox-group control-group" *ngIf="PROVIDER === 'aws'"
+ <div class="checkbox-group control-group m-top-15" *ngIf="PROVIDER === 'aws'"
[hidden]="!model.templates.length || !isAvailableSpots()">
<label class="spot-label label">
<input #spotInstancesCheck type="checkbox" (change)="selectSpotInstances($event)" />
@@ -127,10 +127,16 @@
</div>
- <div class="checkbox-group">
+ <div class="checkbox-group" [hidden]="PROVIDER === 'gcp' && model.selectedImage?.image === 'docker.dlab-dataengine-service'"
+ *ngIf="notebook_instance?.image !== 'docker.dlab-zeppelin'">
<label>
<input #configurationNode type="checkbox" (change)="selectConfiguration()"/> Cluster configurations
</label>
+ <div class="config-link" *ngIf="configuration?.nativeElement['checked']
+ && model.selectedImage?.image === 'docker.dlab-dataengine-service'
+ && DICTIONARY.cloud_provider === 'aws'">
+ To view example JSON of configurations refer for <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html" target="_blank">AWS official documentation</a>
+ </div>
</div>
<div class="checkbox-group">
<div class="config-details" [ngClass]="{ show: configuration?.nativeElement['checked'] }">
@@ -138,6 +144,11 @@
<span class="danger_color" *ngIf="!resourceForm?.controls.configuration_parameters.valid && resourceForm?.controls['configuration_parameters'].dirty">Configuration parameters is not in a valid format</span>
</div>
</div>
+ <div *ngIf="notebook_instance?.image === 'docker.dlab-zeppelin'">
+ <small>Spark default configuration for Apache Zeppelin can not be changed from DLab UI. Currently it can be done directly through Apache Zeppelin interpreter menu.
+ For more details please refer for Apache Zeppelin <a href="https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/overview.html" target="_blank">official documentation</a>.
+ </small>
+ </div>
<div class="text-center m-top-30">
<button mat-raised-button type="button" (click)="bindDialog.close()" class="butt action">Cancel</button>
<button mat-raised-button type="submit" [disabled]="!resourceForm?.valid"
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.scss b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.scss
index 964e908..e23df49 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.scss
@@ -34,10 +34,14 @@
&.danger_color {
position: absolute;
padding-left: 5px;
- bottom: 0;
right: 0;
}
}
+ .alias-name {
+ .danger-color {
+ right: auto;
+ }
+ }
}
&.compress {
flex-direction: column;
@@ -60,7 +64,6 @@
opacity: 0;
visibility: hidden;
text-align: left;
- transition: all .35s linear .2s;
&.show {
height: 60px;
@@ -68,6 +71,11 @@
opacity: 1;
}
}
+ .preemptible-details {
+ .control {
+ position: relative;
+ }
+ }
.spot-details {
&.show {
height: 50px;
@@ -84,7 +92,9 @@
background: #f8f8f8 !important;
height: 100%;
resize: none;
- font-size: 14px
+ font-size: 14px;
+ line-height: 1.5;
+ font-family: Consolas, monospace;
}
}
}
@@ -125,4 +135,9 @@
width: 80%;
}
}
+ .config-link {
+ font-size: 13px;
+ padding-top: 5px;
+ padding-bottom: 5px;
+ }
}
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.ts b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.ts
index 7b92568..ea9f678 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resource-create-dialog/computational-resource-create-dialog.component.ts
@@ -44,7 +44,6 @@
template_description: string;
shapes: any;
spotInstance: boolean = true;
- // clusterNamePattern: string = '[a-zA-Z0-9-_]+';
clusterNamePattern: string = '[-_a-zA-Z0-9]*[_-]*[a-zA-Z0-9]+';
nodeCountPattern: string = '^[1-9]\\d*$';
delimitersRegex = /[-_]?/g;
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resources-list/computational-resources-list.component.ts b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resources-list/computational-resources-list.component.ts
index a9ebce8..f683ebc 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resources-list/computational-resources-list.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resources-list/computational-resources-list.component.ts
@@ -16,7 +16,9 @@
****************************************************************************/
-import { Component, EventEmitter, Input, Output, ViewChild, Inject } from '@angular/core';
+import { Component, EventEmitter, Input, Output, ViewChild, Inject, ViewContainerRef } from '@angular/core';
+import { ToastsManager } from 'ng2-toastr';
+
import { UserResourceService } from '../../../core/services';
import { MatDialog, MatDialogRef, MAT_DIALOG_DATA } from '@angular/material';
@@ -39,23 +41,28 @@
constructor(
private userResourceService: UserResourceService,
- public dialog: MatDialog
- ) { }
+ public dialog: MatDialog,
+ public toastr: ToastsManager,
+ public vcr: ViewContainerRef
+ ) {
+ this.toastr.setRootViewContainerRef(vcr);
+ }
- toggleResourceAction(resource, action) {
+ toggleResourceAction(resource, action: string) {
if (action === 'stop' || action === 'terminate') {
- const dialogRef: MatDialogRef<ConfirmationDialog> = this.dialog.open(ConfirmationDialog, { data: {action, resource}, width: '550px' });
+ const dialogRef: MatDialogRef<ConfirmationDialogComponent> = this.dialog.open(ConfirmationDialogComponent,
+ { data: {action, resource}, width: '550px' });
dialogRef.afterClosed().subscribe(result => {
if (result && action === 'stop') {
this.userResourceService
.toggleStopStartAction(this.environment['name'], resource.computational_name, action)
- .subscribe(response => {
+ .subscribe(() => {
this.rebuildGrid();
});
} else if (result && action === 'terminate') {
this.userResourceService
.suspendComputationalResource(this.environment['name'], resource.computational_name)
- .subscribe(response => {
+ .subscribe(() => {
this.rebuildGrid();
});
}
@@ -63,9 +70,9 @@
} else if (action === 'start') {
this.userResourceService
.toggleStopStartAction(this.environment['name'], resource.computational_name, 'start')
- .subscribe(res => {
- this.rebuildGrid();
- });
+ .subscribe(
+ () => this.rebuildGrid(),
+ error => this.toastr.error(error.message || 'Computational resource starting failed!', 'Oops!', { toastLife: 5000 }));
}
}
@@ -103,9 +110,9 @@
.content { color: #718ba6; padding: 20px 50px; font-size: 14px; font-weight: 400 }
`]
})
-export class ConfirmationDialog {
+export class ConfirmationDialogComponent {
constructor(
- public dialogRef: MatDialogRef<ConfirmationDialog>,
+ public dialogRef: MatDialogRef<ConfirmationDialogComponent>,
@Inject(MAT_DIALOG_DATA) public data: any
) { }
-}
\ No newline at end of file
+}
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resources-list/index.ts b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resources-list/index.ts
index 896a2bd..f957c05 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resources-list/index.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/computational/computational-resources-list/index.ts
@@ -21,7 +21,7 @@
import { MaterialModule } from '../../../shared/material.module';
import { ModalModule } from '../../../shared';
-import { ComputationalResourcesListComponent, ConfirmationDialog } from './computational-resources-list.component';
+import { ComputationalResourcesListComponent, ConfirmationDialogComponent } from './computational-resources-list.component';
import { DetailComputationalResourcesModule } from '../cluster-details';
import { SchedulerModule } from '../../scheduler';
@@ -35,8 +35,8 @@
SchedulerModule,
MaterialModule
],
- declarations: [ComputationalResourcesListComponent, ConfirmationDialog],
- entryComponents: [ConfirmationDialog],
+ declarations: [ComputationalResourcesListComponent, ConfirmationDialogComponent],
+ entryComponents: [ConfirmationDialogComponent],
exports: [ComputationalResourcesListComponent]
})
export class ComputationalResourcesModule {}
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/ami-create-dialog/ami-create-dialog.component.ts b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/ami-create-dialog/ami-create-dialog.component.ts
index 35ad3ec..e7b1434 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/ami-create-dialog/ami-create-dialog.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/ami-create-dialog/ami-create-dialog.component.ts
@@ -29,7 +29,7 @@
templateUrl: './ami-create-dialog.component.html',
styleUrls: ['./ami-create-dialog.component.scss']
})
-export class AmiCreateDialogComponent {
+export class AmiCreateDialogComponent implements OnInit {
readonly DICTIONARY = DICTIONARY;
public notebook: any;
public createAMIForm: FormGroup;
@@ -75,7 +75,8 @@
this.buildGrid.emit();
}
},
- error => this.toastr.error(error.message || `${ DICTIONARY.image.toLocaleUpperCase() } creation failed!`, 'Oops!', { toastLife: 5000 }));
+ error => this.toastr.error(error.message
+ || `${ DICTIONARY.image.toLocaleUpperCase() } creation failed!`, 'Oops!', { toastLife: 5000 }));
}
private initFormModel(): void {
@@ -88,7 +89,7 @@
private providerMaxLength(control) {
if (DICTIONARY.cloud_provider !== 'aws')
- return control.value.length <=10 ? null : { valid: false };
+ return control.value.length <= 10 ? null : { valid: false };
}
private delimitersFiltering(resource): string {
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/detail-dialog/detail-dialog.component.html b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/detail-dialog/detail-dialog.component.html
index f171936..9d82fdd 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/detail-dialog/detail-dialog.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/detail-dialog/detail-dialog.component.html
@@ -32,6 +32,10 @@
</tr>
</table>
<div class="content-box">
+ <div class="detail-info" *ngIf="notebook.error_message">
+ <p class="failed">{{ notebook.error_message }}</p>
+ </div>
+
<div class="scroll-box" id="scrolling">
<div class="detail-info" *ngIf="!notebook.error_message">
<p>Edge Node IP Address {{notebook.ip}}</p>
@@ -76,7 +80,7 @@
</p>
</div>
- <div class="checkbox-group" *ngIf="notebook.status === 'running'">
+ <div class="checkbox-group" *ngIf="notebook.image !== 'docker.dlab-zeppelin'; else not_support" [hidden]="notebook.status !== 'running'">
<label>
<input #configurationNode type="checkbox" (change)="selectConfiguration()"/> Cluster configurations
</label>
@@ -89,6 +93,11 @@
</form>
</div>
</div>
+ <ng-template #not_support>
+ <small [hidden]="notebook.status !== 'running'">Spark default configuration for Apache Zeppelin can not be changed from DLab UI. Currently it can be done directly through Apache Zeppelin interpreter menu.
+ For more details please refer for Apache Zeppelin <a href="https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/overview.html" target="_blank">official documentation</a>.
+ </small>
+ </ng-template>
<div [scrollTo]="configuration?.nativeElement['checked']" class="text-center m-top-20 m-bott-10" *ngIf="configuration?.nativeElement['checked'] && notebook.status === 'running'">
<button mat-raised-button type="button" (click)="bindDialog.close()" class="butt action">Cancel</button>
<button mat-raised-button type="submit" [disabled]="!configurationForm.valid"
@@ -96,10 +105,6 @@
(click)="editClusterConfiguration(configurationForm.value)">Update</button>
</div>
</div>
-
- <div class="detail-info" *ngIf="notebook.error_message">
- <p class="failed">{{ notebook.error_message }}</p>
- </div>
</div>
</div>
</modal-content>
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/detail-dialog/detail-dialog.component.scss b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/detail-dialog/detail-dialog.component.scss
index f96958a..caafb1a 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/detail-dialog/detail-dialog.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/detail-dialog/detail-dialog.component.scss
@@ -50,12 +50,14 @@
}
.config-details {
text-align: left;
- transition: all .35s linear .2s;
position: relative;
- height: 320px;
+ height: 280px;
textarea {
height: 100%;
resize: none;
+ font-size: 14px;
+ line-height: 1.5;
+ font-family: Consolas, monospace;
}
span {
.danger_color {
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/create-environment.component.scss b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/create-environment.component.scss
index d2b860e..b765256 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/create-environment.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/create-environment.component.scss
@@ -22,18 +22,20 @@
opacity: 0;
visibility: hidden;
text-align: left;
- transition: all .35s linear .2s;
&.show {
- height: 250px;
+ height: 200px;
visibility: visible;
opacity: 1;
position: relative;
+ margin-top: 10px;
}
textarea {
height: 100%;
background: #f8f8f8 !important;
resize: none;
- font-size: 14px
+ font-size: 14px;
+ line-height: 1.5;
+ font-family: Consolas, monospace;
}
span {
&.danger_color {
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/exploratory-environment-create-dialog.component.html b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/exploratory-environment-create-dialog.component.html
index f3a14fa..63977d5 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/exploratory-environment-create-dialog.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/exploratory-environment-create-dialog.component.html
@@ -67,7 +67,7 @@
</div>
</div>
- <div class="checkbox-group">
+ <div class="checkbox-group" *ngIf="model.selectedItem.image !== 'docker.dlab-zeppelin'; else not_support">
<label>
<input #configurationNode type="checkbox" (change)="selectConfiguration()" /> Spark configurations
</label>
@@ -76,6 +76,11 @@
<span class="danger_color" *ngIf="!createExploratoryEnvironmentForm?.controls.configuration_parameters.valid && createExploratoryEnvironmentForm?.controls['configuration_parameters'].dirty">Configuration parameters is not in a valid format</span>
</div>
</div>
+ <ng-template #not_support>
+ <small>Spark default configuration for Apache Zeppelin can not be changed from DLab UI. Currently it can be done directly through Apache Zeppelin interpreter menu.
+ For more details please refer for Apache Zeppelin <a href="https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/overview.html" target="_blank">official documentation</a>.
+ </small>
+ </ng-template>
<div class="text-center m-top-30">
<button mat-raised-button type="button" class="butt action" (click)="close()">Cancel</button>
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/exploratory-environment-create-dialog.component.ts b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/exploratory-environment-create-dialog.component.ts
index b45afb5..6730211 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/exploratory-environment-create-dialog.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/exploratory-environment-create-dialog/exploratory-environment-create-dialog.component.ts
@@ -37,7 +37,6 @@
model: ExploratoryEnvironmentCreateModel;
templateDescription: string;
- // namePattern = '[a-zA-Z0-9-_]+';
namePattern = '[-_a-zA-Z0-9]*[_-]*[a-zA-Z0-9]+';
resourceGrid: any;
userImages: Array<any>;
@@ -194,6 +193,6 @@
this.initFormModel();
this.model.resetModel();
- this.configuration.nativeElement['checked'] = false;
+ if (this.configuration) this.configuration.nativeElement['checked'] = false;
}
}
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/index.ts b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/index.ts
index 3f0de89..54801bf 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/index.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/index.ts
@@ -25,7 +25,7 @@
import { FormControlsModule } from '../../../shared/form-controls';
import { KeysPipeModule, LibSortPipeModule, HighLightPipeModule } from '../../../core/pipes';
-import { InstallLibrariesComponent, ErrorMessageDialog } from './install-libraries.component';
+import { InstallLibrariesComponent, ErrorMessageDialogComponent } from './install-libraries.component';
export * from './install-libraries.component';
export * from './install-libraries.model';
@@ -43,8 +43,8 @@
MaterialModule,
BubbleModule
],
- declarations: [InstallLibrariesComponent, ErrorMessageDialog],
- entryComponents: [ErrorMessageDialog],
+ declarations: [InstallLibrariesComponent, ErrorMessageDialogComponent],
+ entryComponents: [ErrorMessageDialogComponent],
exports: [InstallLibrariesComponent]
})
export class InstallLibrariesModule {}
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.html b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.html
index b19cbe1..da51d7d 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.html
@@ -22,6 +22,7 @@
<b>{{ notebook?.name }}</b> Libraries</h4>
</modal-header>
<modal-content>
+ <div class="aligner">
<div class="info" *ngIf="notebook?.status !== 'running'">
<p class=" message">Cannot install libraries: Exploratory
<strong>{{ notebook?.name }}</strong> is not running</p>
@@ -32,8 +33,8 @@
<img src="assets/img/gif-spinner.gif" alt="loading">
</div>
</div>
- <div *ngIf="notebook?.status === 'running'" class="lib-view-wrap">
- <div class="search-box" *ngIf="!uploading">
+ <div *ngIf="notebook?.status === 'running' && !uploading" class="lib-view-wrap">
+ <div class="search-box">
<div class="search-form">
<div>
<div class="control-group constol-select">
@@ -148,9 +149,10 @@
</div>
</mat-list>
</div>
- <div class="text-center m-top-15 m-bott-10 actions" *ngIf="!uploading && notebook?.status === 'running'">
+ <div class="m-top-15 actions" *ngIf="!uploading && notebook?.status === 'running'">
<button mat-raised-button type="button" class="butt action" (click)="close()">Close</button>
<button mat-raised-button type="submit" class="butt butt-success action" (click)="model.confirmAction()" [disabled]="!model.selectedLibs.length || installingInProgress || !destination">Install</button>
</div>
+ </div>
</modal-content>
</modal-dialog>
\ No newline at end of file
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.scss b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.scss
index 649865a..9dc508c 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.scss
@@ -15,26 +15,52 @@
limitations under the License.
****************************************************************************/
+
+.aligner {
+ height: 100%;
+ display: flex;
+ flex-direction: column;
+ > div {
+ &.actions {
+ display: flex;
+ flex: initial;
+ justify-content: center;
+ flex-direction: initial;
+ margin-bottom: 25px;
+ }
+ }
+}
+
+.loading-block {
+ height: 45%;
+ display: flex;
+ justify-content: center;
+ .uploading {
+ flex-direction: column;
+ align-items: center;
+ align-self: center;
+ list-style: none;
+ display: flex;
+ p {
+ font-size: 16px;;
+ }
+ }
+}
+
+.info {
+ height: 40%;
+ display: flex;
+ flex-direction: row;
+ align-items: center;
+ justify-content: center;
+}
+
.install-libraries {
.dialog-header {
padding-left: 25px;
}
- .loading-block {
- height: 45%;
- display: flex;
- justify-content: center;
- .uploading {
- flex-direction: column;
- align-items: center;
- align-self: center;
- list-style: none;
- display: flex;
- p {
- font-size: 16px;;
- }
- }
- }
+
.info {
height: 40%;
display: flex;
@@ -44,7 +70,6 @@
}
.mat-list-item {
height: auto !important;
-
.mat-list-item-content {
height: auto !important;
min-height: 54px !important;
@@ -55,14 +80,14 @@
display: flex;
flex-direction: column;
}
-}
-.error {
- padding: 15px 20px;
-}
-.uploading {
- margin: 0 auto;
-}
+ .error {
+ padding: 15px 20px;
+ }
+ .uploading {
+ margin: 0 auto;
+ }
+}
.search-box {
display: flex;
@@ -105,7 +130,7 @@
.suggestions {
text-decoration: none;
list-style: none;
- .option {
+ mat-option {
font-size: 13px;
font-weight: 300;
font-family: 'Open Sans', sans-serif;
@@ -173,10 +198,12 @@
/* Tab info */
.libs-info {
- height: 45%;
padding: 0 20px;
+ display: flex;
+ flex: 1 1 auto;
+ min-height: 0px;
.mat-list {
- height: 100%;
+ width: 100%;
.scrollingList {
max-height: 80%;
height: 80%;
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.ts b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.ts
index 99e2ee1..d60c992 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/exploratory/install-libraries/install-libraries.component.ts
@@ -16,7 +16,15 @@
****************************************************************************/
-import { Component, OnInit, ViewChild, Output, EventEmitter, ViewEncapsulation, ChangeDetectorRef, Inject, ViewContainerRef } from '@angular/core';
+import { Component,
+ OnInit,
+ ViewChild,
+ Output,
+ EventEmitter,
+ ViewEncapsulation,
+ ChangeDetectorRef,
+ Inject,
+ ViewContainerRef } from '@angular/core';
import { MatDialog, MatDialogRef, MAT_DIALOG_DATA } from '@angular/material';
import { FormControl } from '@angular/forms';
import { ToastsManager } from 'ng2-toastr';
@@ -54,7 +62,14 @@
public isInSelectedList: boolean = false;
public installingInProgress: boolean = false;
public libSearch: FormControl = new FormControl();
- public groupsListMap = {'r_pkg': 'R packages', 'pip2': 'Python 2', 'pip3': 'Python 3', 'os_pkg': 'Apt/Yum', 'others': 'Others', 'java': 'Java'};
+ public groupsListMap = {
+ 'r_pkg': 'R packages',
+ 'pip2': 'Python 2',
+ 'pip3': 'Python 3',
+ 'os_pkg': 'Apt/Yum',
+ 'others': 'Others',
+ 'java': 'Java'
+ };
private readonly CHECK_GROUPS_TIMEOUT: number = 5000;
private clear: number;
@@ -69,7 +84,7 @@
constructor(
public dialog: MatDialog,
private librariesInstallationService: LibrariesInstallationService,
- private changeDetector : ChangeDetectorRef,
+ private changeDetector: ChangeDetectorRef,
public toastr: ToastsManager,
public vcr: ViewContainerRef
) {
@@ -98,20 +113,24 @@
this.libsUploadingStatus(response);
this.changeDetector.detectChanges();
- this.resource_select && this.resource_select.setDefaultOptions(this.getResourcesList(), this.destination.title, 'destination', 'title', 'array');
- this.group_select && this.group_select.setDefaultOptions(this.groupsList, 'Select group', 'group_lib', null, 'list', this.groupsListMap);
+ this.resource_select && this.resource_select.setDefaultOptions(
+ this.getResourcesList(),
+ this.destination.title, 'destination', 'title', 'array');
+
+ this.group_select && this.group_select.setDefaultOptions(
+ this.groupsList, 'Select group', 'group_lib', null, 'list', this.groupsListMap);
},
error => this.toastr.error(error.message || 'Groups list loading failed!', 'Oops!', { toastLife: 5000 }));
}
private getResourcesList() {
this.notebook.type = 'EXPLORATORY';
- this.notebook.title = `${ this.notebook.name } <em>notebook</em>`;
+ this.notebook.title = `${ this.notebook.name } <em class="capt">notebook</em>`;
return [this.notebook].concat(this.notebook.resources
.filter(item => item.status === 'running')
.map(item => {
item['name'] = item.computational_name;
- item['title'] = `${ item.computational_name } <em>cluster</em>`;
+ item['title'] = `${ item.computational_name } <em class="capt">cluster</em>`;
item['type'] = 'СOMPUTATIONAL';
return item;
}));
@@ -158,7 +177,7 @@
this.isInstalled = this.destination.libs.findIndex(libr => {
return select.group !== 'java'
? select.name === libr.name && select.group === libr.group && select.version === libr.version
- : select.name === libr.name && select.group === libr.group
+ : select.name === libr.name && select.group === libr.group;
}) >= 0;
return this.isInSelectedList || this.isInstalled;
@@ -207,7 +226,8 @@
}
public showErrorMessage(item): void {
- const dialogRef: MatDialogRef<ErrorMessageDialog> = this.dialog.open(ErrorMessageDialog, { data: item.error, width: '550px' });
+ const dialogRef: MatDialogRef<ErrorMessageDialogComponent> = this.dialog.open(
+ ErrorMessageDialogComponent, { data: item.error, width: '550px', panelClass: 'error-modalbox' });
}
public isInstallingInProgress(data): void {
@@ -226,7 +246,7 @@
public reinstallLibrary(item, lib) {
const retry = [{group: lib.group, name: lib.name, version: lib.version}];
- if (this.getResourcesList().find(el => el.name == item.resource).type === 'СOMPUTATIONAL') {
+ if (this.getResourcesList().find(el => el.name === item.resource).type === 'СOMPUTATIONAL') {
this.model.confirmAction(retry, item.resource);
} else {
this.model.confirmAction(retry);
@@ -267,7 +287,9 @@
.subscribe(
lib => this.filteredList = [lib],
error => {
- if (error.status === HTTP_STATUS_CODES.NOT_FOUND || error.status === HTTP_STATUS_CODES.BAD_REQUEST) {
+ if (error.status === HTTP_STATUS_CODES.NOT_FOUND
+ || error.status === HTTP_STATUS_CODES.BAD_REQUEST
+ || error.status === HTTP_STATUS_CODES.INTERNAL_SERVER_ERROR) {
this.validity_format = error.message;
this.filteredList = null;
}
@@ -278,8 +300,9 @@
}
}
- private selectorsReset():void {
- this.resource_select && this.resource_select.setDefaultOptions(this.getResourcesList(), 'Select resource', 'destination', 'title', 'array');
+ private selectorsReset(): void {
+ this.resource_select && this.resource_select.setDefaultOptions(this.getResourcesList(),
+ 'Select resource', 'destination', 'title', 'array');
this.group_select && this.group_select.setDefaultOptions([], '', 'group_lib', null, 'array');
}
@@ -305,12 +328,55 @@
@Component({
selector: 'error-message-dialog',
- template: `<div class="content">{{ data }}</div>`,
- styles: [`.content { color: #f1696e; padding: 20px 25px; font-size: 14px; font-weight: 400 }`]
+ template: `
+ <div class="dialog-header">
+ <h4 class="modal-title">Library installation error</h4>
+ <button type="button" class="close" (click)="dialogRef.close()">×</button>
+ </div>
+ <div class="content">{{ data }}</div>
+ <div class="text-center">
+ <button type="button" class="butt" mat-raised-button (click)="dialogRef.close()">Close</button>
+ </div>
+ `,
+ styles: [`
+ .content { color: #f1696e; padding: 20px 25px; font-size: 14px; font-weight: 400 }
+ .dialog-header {
+ position: relative;
+ top: 0;
+ padding-left: 30px;
+ background: #f6fafe;
+ height: 54px;
+ line-height: 54px;
+ }
+ .dialog-header h4 {
+ color: #455c74;
+ font-size: 18px;
+ font-weight: 600;
+ }
+ .close {
+ position: absolute;
+ top: 0;
+ right: 0;
+ height: 50px;
+ width: 50px;
+ font-size: 24px;
+ border: 0;
+ background: none;
+ color: #577289;
+ outline: none;
+ cursor: pointer;
+ transition: all .45s ease-in-out;
+ }
+ .close:hover {
+ color: #36afd5;
+ }
+ .text-center button {
+ margin-bottom: 25px;
+ }`]
})
-export class ErrorMessageDialog {
+export class ErrorMessageDialogComponent {
constructor(
- public dialogRef: MatDialogRef<ErrorMessageDialog>,
+ public dialogRef: MatDialogRef<ErrorMessageDialogComponent>,
@Inject(MAT_DIALOG_DATA) public data: any
) { }
}
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/resources-grid/resources-grid.component.ts b/services/self-service/src/main/resources/webapp/src/app/resources/resources-grid/resources-grid.component.ts
index 1f667a8..7ec23c8 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/resources-grid/resources-grid.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/resources-grid/resources-grid.component.ts
@@ -17,7 +17,9 @@
****************************************************************************/
/* tslint:disable:no-empty */
-import { Component, ViewChild, OnInit } from '@angular/core';
+import { Component, ViewChild, ViewContainerRef, OnInit } from '@angular/core';
+import { ToastsManager } from 'ng2-toastr';
+
import { UserResourceService } from '../../core/services';
import { ResourcesGridRowModel, FilterConfigurationModel, CreateResourceModel } from '.';
import { GeneralEnvironmentStatus } from '../../health-status/health-status.module';
@@ -69,8 +71,12 @@
];
constructor(
- private userResourceService: UserResourceService
- ) { }
+ private userResourceService: UserResourceService,
+ public toastr: ToastsManager,
+ public vcr: ViewContainerRef
+ ) {
+ this.toastr.setRootViewContainerRef(vcr);
+ }
ngOnInit(): void {
this.buildGrid();
@@ -286,7 +292,9 @@
} else if (action === 'run') {
this.userResourceService
.runExploratoryEnvironment({ notebook_instance_name: data.name })
- .subscribe(() => this.buildGrid());
+ .subscribe(
+ () => this.buildGrid(),
+ error => this.toastr.error(error.message || 'Exploratory starting failed!', 'Oops!', { toastLife: 5000 }));
} else if (action === 'stop') {
this.confirmationDialog.open({ isFooter: false }, data, ConfirmationDialogType.StopExploratory);
} else if (action === 'terminate') {
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/scheduler/scheduler.component.html b/services/self-service/src/main/resources/webapp/src/app/resources/scheduler/scheduler.component.html
index f61297d..9794115 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/scheduler/scheduler.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/scheduler/scheduler.component.html
@@ -32,14 +32,14 @@
<mat-slide-toggle labelPosition="after" [checked]="enableSchedule" (change)="toggleSchedule($event)">
<span *ngIf="toggleSchedule" class="hold-label">Enable scheduler</span>
</mat-slide-toggle>
- <div class="idle">
+ <div class="idle" *ngIf="idleImplemented">
<mat-slide-toggle labelPosition="before" [checked]="enableIdleTime" (change)="toggleIdleTimes($event)">
<span *ngIf="toggleSchedule" class="hold-label">Inactivity time</span>
</mat-slide-toggle>
</div>
</div>
- <div class="note m-bott-10">NOTE: In case of turning on inactivity time-check, your schedule configuration will be decommissioned.</div>
+ <div *ngIf="idleImplemented" lass="note m-bott-10">NOTE: In case of turning on inactivity time-check, your schedule configuration will be decommissioned.</div>
<div class="control-group idle-control" [ngClass]="{ show: enableIdleTime }">
<label class="label">Inactivity time</label>
diff --git a/services/self-service/src/main/resources/webapp/src/app/resources/scheduler/scheduler.component.ts b/services/self-service/src/main/resources/webapp/src/app/resources/scheduler/scheduler.component.ts
index 19bb8f7..70d7e75 100644
--- a/services/self-service/src/main/resources/webapp/src/app/resources/scheduler/scheduler.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/resources/scheduler/scheduler.component.ts
@@ -57,6 +57,9 @@
public inactivityLimits = { min: 10, max: 10080 };
+
+ public idleImplemented: boolean = false;
+
@ViewChild('bindDialog') bindDialog;
@ViewChild('resourceSelect') resource_select;
@Output() buildGrid: EventEmitter<{}> = new EventEmitter();
@@ -64,7 +67,7 @@
constructor(
private formBuilder: FormBuilder,
private schedulerService: SchedulerService,
- private changeDetector : ChangeDetectorRef,
+ private changeDetector: ChangeDetectorRef,
public toastr: ToastsManager,
public vcr: ViewContainerRef
) {
@@ -75,7 +78,7 @@
this.bindDialog.onClosing = () => {
this.resetDialog();
this.buildGrid.emit();
- }
+ };
}
public open(param, notebook, type, resource?): void {
@@ -90,7 +93,6 @@
response => {
if (response.status === HTTP_STATUS_CODES.OK) {
// this.toastr.success('Schedule data were successfully saved', 'Success!', { toastLife: 5000 });
- this.buildGrid.emit();
this.close();
}
},
@@ -160,7 +162,7 @@
this.schedulerForm.controls.inactivityTime.setValue(this.inactivityLimits.min);
}
}
-
+
public setInactivity(...params) {
this.model.setInactivityTime(params).subscribe(
() => {
@@ -172,7 +174,7 @@
public scheduleInstance_btnClick() {
if (!this.enableIdleTime) {
- let data = {
+ const data = {
startDate: this.schedulerForm.controls.startDate.value,
finishDate: this.schedulerForm.controls.finishDate.value
};
@@ -182,7 +184,7 @@
return false;
}
const selectedDays = Object.keys(this.selectedStartWeekDays);
- let parameters: any = {
+ const parameters: any = {
begin_date: data.startDate ? _moment(data.startDate).format(this.date_format) : null,
finish_date: data.finishDate ? _moment(data.finishDate).format(this.date_format) : null,
start_time: this.startTime ? this.convertTimeFormat(this.startTime) : null,
@@ -198,7 +200,7 @@
? this.model.confirmAction(this.notebook.name, parameters, this.destination.computational_name)
: this.model.confirmAction(this.notebook.name, parameters);
} else {
- let data = { check_inactivity_required: true, max_inactivity: this.schedulerForm.controls.inactivityTime.value };
+ const data = { check_inactivity_required: true, max_inactivity: this.schedulerForm.controls.inactivityTime.value };
(this.destination.type === 'СOMPUTATIONAL')
? this.setInactivity(this.notebook.name, data, this.destination.computational_name)
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/dropdown-list/dropdown-list.component.ts b/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/dropdown-list/dropdown-list.component.ts
index f4e57b7..d7f9361 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/dropdown-list/dropdown-list.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/dropdown-list/dropdown-list.component.ts
@@ -16,7 +16,7 @@
****************************************************************************/
-import { Output, Component, EventEmitter, ViewEncapsulation } from '@angular/core';
+import { Output, Component, EventEmitter } from '@angular/core';
export class DropdownListModel {
constructor(
@@ -29,8 +29,7 @@
@Component({
selector: 'dropdown-list',
templateUrl: 'dropdown-list.component.html',
- styleUrls: ['../dropdowns.component.scss'],
- encapsulation: ViewEncapsulation.None
+ styleUrls: ['../dropdowns.component.scss']
})
export class DropdownListComponent {
@@ -41,7 +40,7 @@
listStructure: string;
items: any;
map: any;
- empty: boolean = false;s
+ empty: boolean = false;
@Output() selectedItem: EventEmitter<{}> = new EventEmitter();
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/dropdowns.component.scss b/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/dropdowns.component.scss
index 3a1682a..db88ab6 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/dropdowns.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/dropdowns.component.scss
@@ -27,7 +27,8 @@
min-width: 100%;
background: #fff;
padding-left: 20px;
- font-size: 16px;
+ font-size: 14px;
+ // height: 34px;
text-align: left;
white-space: nowrap;
cursor: pointer;
@@ -42,6 +43,12 @@
line-height: 38px;
span {
color: #4a5c89;
+ em {
+ font-size: 13px;
+ color: #35afd5;
+ margin-right: 0px;
+ font-style: normal;
+ }
}
}
}
@@ -84,7 +91,7 @@
-moz-appearance: none;
border-left: 1px solid #ececec;
background: #fff;
- color: #36afd5;
+ color: #36afd5 !important;
}
.list-menu {
width: 100%;
@@ -102,6 +109,13 @@
text-transform: capitalize;
}
}
+ &.user {
+ .list-menu {
+ .list-item {
+ text-transform: lowercase;
+ }
+ }
+ }
}
.dropdown-list .list-menu a,
@@ -142,25 +156,17 @@
text-align: center;
}
-
-.dropdown-list .list-menu span,
-.dropdown-list .title span {
- display: flex;
- justify-content: space-between;
- align-items: center;
- font-weight: 300;
+.dropdown-list {
+ .list-menu, .title {
+ span {
+ display: flex;
+ justify-content: space-between;
+ align-items: center;
+ font-weight: 300;
+ }
+ }
}
-.dropdown-list .title em,
-.dropdown-list .list-menu li em {
- font-size: 13px;
- color: #35afd5;
- margin-right: 0px;
- font-style: normal;
-}
-.dropdown-list .title em {
- margin-right: 50px;
-}
.dropdown-list .list-menu li span.caption {
display: block;
padding: 10px 15px;
@@ -186,6 +192,12 @@
position: relative;
font-weight: 300;
cursor: pointer;
+ em {
+ font-size: 13px;
+ color: #35afd5;
+ margin-right: 0px;
+ font-style: normal;
+ }
}
.dropdown-list .list-menu a.empty {
height: 36px;
@@ -201,13 +213,13 @@
.dropdown-multiselect .list-menu .select_all:hover,
.dropdown-multiselect .list-menu .select_all:hover i {
- color: #4eaf3e;
+ color: #4eaf3e !important;
background: #f9fafb;
}
.dropdown-multiselect .list-menu .deselect_all:hover,
.dropdown-multiselect .list-menu .deselect_all:hover i {
- color: #f1696e;
+ color: #f1696e !important;
background: #f9fafb;
}
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/multi-select-dropdown/multi-select-dropdown.component.ts b/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/multi-select-dropdown/multi-select-dropdown.component.ts
index 5dbaa8d..f206c34 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/multi-select-dropdown/multi-select-dropdown.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/form-controls/multi-select-dropdown/multi-select-dropdown.component.ts
@@ -16,13 +16,12 @@
****************************************************************************/
-import { Input, Output, Component, EventEmitter, ViewEncapsulation } from '@angular/core';
+import { Input, Output, Component, EventEmitter } from '@angular/core';
@Component({
selector: 'multi-select-dropdown',
templateUrl: 'multi-select-dropdown.component.html',
- styleUrls: ['../dropdowns.component.scss'],
- encapsulation: ViewEncapsulation.None
+ styleUrls: ['../dropdowns.component.scss']
})
export class MultiSelectDropdownComponent {
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.html b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.html
index 1855e6d..6591db7 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.html
@@ -32,7 +32,7 @@
<p class="info text-center">{{ model.title }}</p>
<mat-list class="resources" [hidden]="model.notebook.type === 'Edge Node' || model.notebook.name === 'edge node'
- ||!model.notebook.resources || model.notebook.resources.length === 0 || (!isAliveResources && !confirmationType)">
+ || !model.notebook.resources || model.notebook.resources.length === 0 || (!isAliveResources && !confirmationType) || onlyKilled">
<mat-list-item class="list-header">
<div class="cluster">Cluster</div>
<div class="status">Further status</div>
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.scss b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.scss
index c22ecc4..a0b858c 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.scss
@@ -35,7 +35,7 @@
width: 30%;
}
.size {
- width: 15%;
+ width: 25%;
}
.scrolling-content {
max-height: 200px;
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.ts b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.ts
index 2cecc38..8940468 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/confirmation-dialog/confirmation-dialog.component.ts
@@ -36,6 +36,7 @@
readonly DICTIONARY = DICTIONARY;
model: ConfirmationDialogModel;
isAliveResources: boolean;
+ onlyKilled: boolean = false;
dataengines: Array<any> = [];
dataengineServices: Array<any> = [];
confirmationType: number = 0;
@@ -79,6 +80,7 @@
this.bindDialog.open(param);
if (!this.confirmationType) this.filterResourcesByType(notebook.resources);
this.isAliveResources = this.model.isAliveResources(notebook.resources);
+ this.onlyKilled = !notebook.resources.some(el => el.status !== 'terminated')
}
public close() {
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/key-upload-dialog/key-upload-dialog.component.html b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/key-upload-dialog/key-upload-dialog.component.html
index e46bc47..508c1fb 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/key-upload-dialog/key-upload-dialog.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/key-upload-dialog/key-upload-dialog.component.html
@@ -47,7 +47,7 @@
<button mat-raised-button type="button" class="butt butt-generate" (click)="generateUserAccessKey_btnClick($event)">
<i class="material-icons">move_to_inbox</i> Generate
</button>
- <div *ngIf="model.accessKeyLabel.length" class="m-bott-10 m-top-10" [class.danger_color]="!model.accessKeyFormValid">{{model.accessKeyLabel}}</div>
+ <div *ngIf="model.accessKeyLabel.length" class="m-bott-10 m-top-10 ellipsis" [class.danger_color]="!model.accessKeyFormValid">{{model.accessKeyLabel}}</div>
</div>
</div>
<div class="text-center m-top-30">
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/modal.component.html b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/modal.component.html
index 792bfba..3da9b12 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/modal.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/modal.component.html
@@ -23,9 +23,9 @@
<div [class]="'dialog-container ' + modalClass">
<div class="dialog-content" tabindex="0" *ngIf="isOpened">
<div [ngClass]="{ 'dialog-header': isHeader, 'hidden': !isHeader }">
- <button *ngIf="!hideCloseButton" type="button" class="close" data-dismiss="modal" [attr.aria-label]="cancelButtonLabel || 'Close'" (click)="close()">×</button>
<h4 class="modal-title" *ngIf="title">{{ title }}</h4>
<ng-content select="modal-header"></ng-content>
+ <button *ngIf="!hideCloseButton" type="button" class="close" data-dismiss="modal" [attr.aria-label]="cancelButtonLabel || 'Close'" (click)="close()">×</button>
</div>
<div class="dialog-body">
<ng-content select="modal-content"></ng-content>
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/modal.component.scss b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/modal.component.scss
index 6de9363..2809b30 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/modal.component.scss
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/modal.component.scss
@@ -179,17 +179,16 @@
outline: 0;
-webkit-box-shadow: 0 3px 9px rgba(0,0,0,.5);
box-shadow: 0 3px 9px rgba(0,0,0,.5);
+ .danger_color {
+ padding-left: 5px;
+ font-size: 12px;
+ display: inline-block;
+ font-family: 'Open Sans', sans-serif;
+ font-weight: 300;
+ border-color: #d9534f !important;
+ }
}
-
-.dialog-content .danger_color {
- padding-top: 5px;
- padding-left: 5px;
- font-size: 12px;
- display: inline-block;
-}
-.dialog-content .danger_field {
- border-color: #d9534f !important;
-}
+
.modal-footer .btn {
margin-top: 0;
}
@@ -206,8 +205,10 @@
height: 54px;
line-height: 54px;
.modal-title {
+ width: 90%;
margin: 0;
overflow: hidden;
+ white-space: nowrap;
text-overflow: ellipsis;
font-weight: 600;
color: #455c74;
@@ -327,7 +328,7 @@
}
}
.content-box {
- padding: 25px 35px 35px;
+ padding: 25px 30px 35px;
}
.detail-header {
border-bottom: 0;
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/notification-dialog/index.ts b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/notification-dialog/index.ts
index 2a8d485..d419ecb 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/notification-dialog/index.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/notification-dialog/index.ts
@@ -31,4 +31,4 @@
entryComponents: [NotificationDialogComponent],
exports: [NotificationDialogComponent]
})
-export class NotificationDialogModule {}
\ No newline at end of file
+export class NotificationDialogModule {}
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/notification-dialog/notification-dialog.component.ts b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/notification-dialog/notification-dialog.component.ts
index 3f5abdb..cfd3e47 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/notification-dialog/notification-dialog.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/modal-dialog/notification-dialog/notification-dialog.component.ts
@@ -22,15 +22,40 @@
@Component({
selector: 'notification-dialog',
template: `
- <div mat-dialog-content class="content">
- {{ data }}
- </div>
- <div class="text-center">
- <button type="button" class="butt" mat-raised-button (click)="dialogRef.close()">Close</button>
+ <header>
+ <h4><i class="material-icons">priority_high</i>Warning</h4>
+ <a class="ani" (click)="dialogRef.close()"><i class="material-icons">close</i></a>
+ </header>
+ <div mat-dialog-content class="content info message">
+ <div *ngIf="data.type === 'list'; else info">
+ <div *ngIf="data.template.notebook.length > 0">
+ Following notebook server<span *ngIf="data.template.notebook.length>1">s </span>
+ <span *ngFor="let item of data.template.notebook">
+ <b>{{ item.exploratory_name }}</b>
+ <span *ngIf="data.template.notebook.length > 1">, </span>
+ </span> will be stopped and all computational resources will be stopped/terminated
+ </div>
+
+ <div *ngIf="data.template.cluster.length > 0">
+ <p *ngFor="let item of data.template.cluster">
+ Computational resource<span *ngIf="data.template.cluster.length > 1">s </span>
+ <b>{{ item.computational_name }}</b> on <b>{{ item.exploratory_name }}</b>
+ will be stopped
+ </p>
+ </div>
+ <strong>by a schedule in 15 minutes.</strong>
+ </div>
+ <ng-template #info><span [innerHTML]="data.template"></span></ng-template>
</div>
`,
styles: [`
.content { color: #718ba6; padding: 20px 50px; font-size: 14px; font-weight: 400 }
+ .info { color: #35afd5; }
+ header { display: flex; justify-content: space-between; color: #607D8B; }
+ header h4 i { vertical-align: bottom; }
+ header a i { font-size: 20px; }
+ header a:hover i { color: #35afd5; cursor: pointer; }
+ .plur { font-style: normal; }
`]
})
export class NotificationDialogComponent {
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/navbar/navbar.component.html b/services/self-service/src/main/resources/webapp/src/app/shared/navbar/navbar.component.html
index 1e74d6e..b7b833c 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/navbar/navbar.component.html
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/navbar/navbar.component.html
@@ -50,4 +50,4 @@
(generateUserKey)="generateUserKey($event)"
(checkInfrastructureCreationProgress)="checkCreationProgress($event)"
[primaryUploading]="true"></key-upload-dialog>
-<progress-dialog #preloaderModal></progress-dialog>
\ No newline at end of file
+<progress-dialog #preloaderModal></progress-dialog>
diff --git a/services/self-service/src/main/resources/webapp/src/app/shared/navbar/navbar.component.ts b/services/self-service/src/main/resources/webapp/src/app/shared/navbar/navbar.component.ts
index e5ac927..b303345 100644
--- a/services/self-service/src/main/resources/webapp/src/app/shared/navbar/navbar.component.ts
+++ b/services/self-service/src/main/resources/webapp/src/app/shared/navbar/navbar.component.ts
@@ -21,7 +21,16 @@
import { Subscription } from 'rxjs/Subscription';
import { ToastsManager } from 'ng2-toastr';
-import { ApplicationSecurityService, HealthStatusService, AppRoutingService, UserAccessKeyService } from '../../core/services';
+import { TimerObservable } from 'rxjs/observable/TimerObservable';
+import { timer } from 'rxjs/observable/timer';
+import { interval } from 'rxjs/observable/interval';
+import 'rxjs/add/operator/takeWhile';
+
+import { ApplicationSecurityService,
+ HealthStatusService,
+ AppRoutingService,
+ UserAccessKeyService,
+ SchedulerService } from '../../core/services';
import { GeneralEnvironmentStatus } from '../../health-status/environment-status.model';
import { DICTIONARY } from '../../../dictionary/global.dictionary';
import { HTTP_STATUS_CODES, FileUtils } from '../../core/util';
@@ -35,7 +44,11 @@
})
export class NavbarComponent implements OnInit, OnDestroy {
readonly PROVIDER = DICTIONARY.cloud_provider;
- private readonly CHECK_ACCESS_KEY_TIMEOUT: number = 20000;
+
+ private alive: boolean = false;
+ private readonly CHECK_ACCESS_KEY_TIMEOUT: number = 30000;
+ private readonly CHECK_ACTIVE_SCHEDULE_TIMEOUT: number = 55000;
+ private readonly CHECK_ACTIVE_SCHEDULE_PERIOD: number = 15;
currentUserName: string;
quotesLimit: number;
@@ -48,38 +61,44 @@
@ViewChild('preloaderModal') preloaderDialog;
constructor(
+ public toastr: ToastsManager,
+ public vcr: ViewContainerRef,
private applicationSecurityService: ApplicationSecurityService,
private appRoutingService: AppRoutingService,
private healthStatusService: HealthStatusService,
private userAccessKeyService: UserAccessKeyService,
- private dialog: MatDialog,
- public toastr: ToastsManager,
- public vcr: ViewContainerRef
+ private schedulerService: SchedulerService,
+ private dialog: MatDialog
) {
this.toastr.setRootViewContainerRef(vcr);
}
ngOnInit() {
this.applicationSecurityService.loggedInStatus.subscribe(response => {
+ this.subscriptions.unsubscribe();
+ this.subscriptions.closed = false;
+ this.alive = false;
+
this.isLoggedIn = response;
if (this.isLoggedIn) {
this.subscriptions.add(this.healthStatusService.statusData.subscribe(result => {
this.healthStatus = result;
- this.checkQuoteUsed(this.healthStatus);
+ result.status && this.checkQuoteUsed(this.healthStatus);
}));
this.subscriptions.add(this.userAccessKeyService.accessKeyEmitter.subscribe(result => {
result && this.processAccessKeyStatus(result.status);
}));
+ this.subscriptions.add(timer(0, this.CHECK_ACTIVE_SCHEDULE_TIMEOUT).subscribe(() => this.refreshSchedulerData()));
this.currentUserName = this.getUserName();
}
});
-
this.quotesLimit = 70;
}
ngOnDestroy(): void {
this.subscriptions.unsubscribe();
+ this.alive = false;
}
getUserName(): string {
@@ -87,14 +106,19 @@
}
logout_btnClick(): void {
+ this.healthStatusService.resetStatusValue();
+ this.userAccessKeyService.resetUserAccessKey();
this.applicationSecurityService.logout().subscribe(
- () => this.appRoutingService.redirectToLoginPage(),
+ () => {
+ this.appRoutingService.redirectToLoginPage();
+ this.subscriptions.unsubscribe();
+ },
error => console.error(error));
}
- public emitQuotes(): void {
+ public emitQuotes(alert, user_quota?, total_quota?): void {
const dialogRef: MatDialogRef<NotificationDialogComponent> = this.dialog.open(NotificationDialogComponent, {
- data: `NOTE: Currently used billing quote is ${ this.healthStatus.billingQuoteUsed }%`,
+ data: { template: this.selectQuotesAlert(alert, user_quota, total_quota), type: 'message' },
width: '550px'
});
dialogRef.afterClosed().subscribe(() => {
@@ -115,22 +139,81 @@
}
private checkQuoteUsed(params): void {
- if (params.billingQuoteUsed >= this.quotesLimit && !this.applicationSecurityService.getBillingQuoteUsed()) {
+ if (!this.applicationSecurityService.getBillingQuoteUsed() && params) {
+ let checkQuotaAlert = '';
+
+ if (params.billingUserQuoteUsed >= this.quotesLimit && params.billingUserQuoteUsed < 100) checkQuotaAlert = 'user_quota';
+ if (params.billingQuoteUsed >= this.quotesLimit && params.billingQuoteUsed < 100) checkQuotaAlert = 'total_quota';
+ if (Number(params.billingUserQuoteUsed) >= 100) checkQuotaAlert = 'user_exceed';
+ if (Number(params.billingQuoteUsed) >= 100) checkQuotaAlert = 'total_exceed';
+
if (this.dialog.openDialogs.length > 0 || this.dialog.openDialogs.length > 0) return;
- this.emitQuotes();
+ checkQuotaAlert && this.emitQuotes(checkQuotaAlert, params.billingUserQuoteUsed, params.billingQuoteUsed);
}
}
private processAccessKeyStatus(status: number): void {
if (status === HTTP_STATUS_CODES.NOT_FOUND) {
this.keyUploadDialog.open({ isFooter: false });
+ this.alive = false;
} else if (status === HTTP_STATUS_CODES.ACCEPTED) {
!this.preloaderDialog.bindDialog.isOpened && this.preloaderDialog.open({ isHeader: false, isFooter: false });
- setTimeout(() => this.userAccessKeyService.initialUserAccessKeyCheck(), this.CHECK_ACCESS_KEY_TIMEOUT);
+
+ if (!this.alive) {
+ this.alive = true;
+ this.subscriptions.add(interval(this.CHECK_ACCESS_KEY_TIMEOUT)
+ .takeWhile(() => this.alive)
+ .subscribe(() => this.userAccessKeyService.initialUserAccessKeyCheck()));
+ }
+
} else if (status === HTTP_STATUS_CODES.OK) {
+ this.alive = false;
this.userAccessKeyService.emitActionOnKeyUploadComplete();
this.preloaderDialog.close();
this.keyUploadDialog.close();
}
}
+
+ private refreshSchedulerData(): void {
+ this.schedulerService.getActiveSchcedulersData(this.CHECK_ACTIVE_SCHEDULE_PERIOD).subscribe((list: Array<any>) => {
+ if (list.length) {
+ if (this.dialog.openDialogs.length > 0 || this.dialog.openDialogs.length > 0) return;
+ const filteredData = this.groupSchedulerData(list);
+ const dialogRef: MatDialogRef<NotificationDialogComponent> = this.dialog.open(NotificationDialogComponent, {
+ data: { template: filteredData, type: 'list' },
+ width: '550px'
+ });
+ }
+ });
+ }
+
+ private groupSchedulerData(sheduler_data) {
+ const memo = { notebook: [], cluster: [] };
+ sheduler_data.map(item => !item.computational_name ? memo.notebook.push(item) : memo.cluster.push(item));
+ memo.cluster = memo.cluster.filter(el => !memo.notebook.some(elm => el.exploratory_name === elm.exploratory_name));
+ return memo;
+ }
+
+ private selectQuotesAlert(type: string, user_quota?: number, total_quota?: number): string {
+ const alerts = {
+ user_exceed: `Dear <b>${ this.currentUserName }</b>,<br />
+ DLab cloud infrastructure usage quota associated with your user has been exceeded.
+ All your analytical environment will be stopped. To proceed working with environment,
+ request increase of user quota from DLab administrator.`,
+ total_exceed: `Dear <b>${ this.currentUserName }</b>,<br />
+ DLab cloud infrastructure usage quota has been exceeded.
+ All your analytical environment will be stopped. To proceed working with environment,
+ request increase application quota from DLab administrator.`,
+ user_quota: `Dear <b>${ this.currentUserName }</b>,<br />
+ Cloud infrastructure usage quota associated with your user has been used for <b>${user_quota}%</b>.
+ Once quota is depleted all your analytical environment will be stopped.
+ To proceed working with environment you'll have to request increase of user quota from DLab administrator.`,
+ total_quota: `Dear <b>${ this.currentUserName }</b>,<br />
+ DLab cloud infrastructure usage quota has been used for <b>${total_quota}%</b>.
+ Once quota is depleted all your analytical environment will be stopped.
+ To proceed working with environment you'll have to request increase of user quota from DLab administrator. `
+ };
+
+ return alerts[type];
+ }
}
diff --git a/services/self-service/src/main/resources/webapp/src/assets/styles/_general.scss b/services/self-service/src/main/resources/webapp/src/assets/styles/_general.scss
index 7bab6b0..42225a4 100644
--- a/services/self-service/src/main/resources/webapp/src/assets/styles/_general.scss
+++ b/services/self-service/src/main/resources/webapp/src/assets/styles/_general.scss
@@ -120,4 +120,20 @@
color: #577289 !important;
font-size: 15px !important;
font-weight: 600 !important;
+}
+
+.dropdown-list {
+ .title {
+ span {
+ .capt {
+ margin-right: 50px;
+ }
+ }
+ }
+ .capt {
+ font-size: 13px;
+ color: #35afd5;
+ margin-right: 0px;
+ font-style: normal;
+ }
}
\ No newline at end of file
diff --git a/services/self-service/src/main/resources/webapp/src/assets/styles/_reset.scss b/services/self-service/src/main/resources/webapp/src/assets/styles/_reset.scss
index 412e82a..91f1670 100644
--- a/services/self-service/src/main/resources/webapp/src/assets/styles/_reset.scss
+++ b/services/self-service/src/main/resources/webapp/src/assets/styles/_reset.scss
@@ -78,4 +78,11 @@
[type="radio"] {
box-sizing: border-box;
padding: 0;
+}
+
+input {
+ &::-ms-clear,
+ &::-ms-reveal {
+ display: none;
+ }
}
\ No newline at end of file
diff --git a/services/self-service/src/main/resources/webapp/src/assets/styles/_theme.scss b/services/self-service/src/main/resources/webapp/src/assets/styles/_theme.scss
index 2bf5804..d590c30 100644
--- a/services/self-service/src/main/resources/webapp/src/assets/styles/_theme.scss
+++ b/services/self-service/src/main/resources/webapp/src/assets/styles/_theme.scss
@@ -142,53 +142,82 @@
word-break: break-all;
}
.manage-roles-dialog {
- .mat-form-field-type-mat-select:not(.mat-form-field-disabled) {
- .mat-form-field-flex {
- line-height: 20px;
+ .mat-form-field {
+ width: 100%;
+ }
+ .inner-step {
+ .mat-select-value {
+ width: 96%;
}
}
- .mat-input-wrapper {
- &.mat-form-field-wrapper {
- overflow: hidden;
- .mat-input-infix.mat-form-field-infix {
- width: 275px;
- border: 0;
- font-family: 'Open Sans', sans-serif;
- font-size: 16px;
- font-weight: 300;
- .mat-select-value-text {
- span {
- padding: 0 10px;
- color: #607D8B;
- }
- }
- }
- .mat-form-field-underline {
- display: none;
- }
- .mat-select-arrow {
- color: transparent !important;
- }
- .mat-form-field-label {
- padding-top: 0;
- }
- .mat-select{
- &:focus {
- .mat-input-underline {
- .mat-input-ripple {
- background-color: transparent !important;
- }
- }
- &:not(.mat-select-disabled) {
- .mat-select-arrow {
- color: transparent;
- }
- }
- }
+ .mat-reset {
+ .mat-form-field-type-mat-select:not(.mat-form-field-disabled) {
+ .mat-form-field-flex {
+ line-height: 20px;
}
}
- .mat-input-wrapper.mat-form-field-wrapper .mat-form-field-underline {
- display: none;
+ .mat-input-wrapper {
+ &.mat-form-field-wrapper {
+ overflow: hidden;
+ .mat-input-infix.mat-form-field-infix {
+ width: 275px;
+ border: 0;
+ font-family: 'Open Sans', sans-serif;
+ font-size: 16px;
+ font-weight: 300;
+ .mat-select-value-text {
+ span {
+ // padding: 0 10px;
+ color: #607D8B;
+ }
+ }
+ }
+ .mat-form-field-underline {
+ display: none;
+ }
+ .mat-select-arrow {
+ color: transparent !important;
+ }
+ .mat-form-field-label {
+ padding-top: 0;
+ }
+ .mat-select{
+ &:focus {
+ .mat-input-underline {
+ .mat-input-ripple {
+ background-color: transparent !important;
+ }
+ }
+ &:not(.mat-select-disabled) {
+ .mat-select-arrow {
+ color: transparent;
+ }
+ }
+ }
+ }
+ }
+ .mat-input-wrapper.mat-form-field-wrapper .mat-form-field-underline {
+ display: none;
+ }
+ }
+ }
+ .mat-form-field-type-mat-chip-list {
+ width: 100%;
+ }
+ .mat-step-header {
+ .mat-step-icon {
+ background-color: #36afd5;
+ }
+ }
+ .mat-chip {
+ background-color: rgba(47, 174, 215, 0.2) !important;
+ color: #687e96;
+ }
+ .roles {
+ .selector-wrapper-edit {
+ .mat-select-value {
+ width: 93%;
+ }
}
}
}
@@ -208,4 +237,15 @@
.mat-pseudo-checkbox-checked {
background: #00BCD4;
}
+}
+
+.mat-icon-button .mat-icon, .mat-icon-button i {
+ line-height: 1 !important;
+ font-size: 18px;
+}
+
+.error-modalbox {
+ .mat-dialog-container {
+ padding: 0;
+ }
}
\ No newline at end of file
diff --git a/services/self-service/src/main/resources/webapp/src/environments/environment.prod.ts b/services/self-service/src/main/resources/webapp/src/environments/environment.prod.ts
index 3612073..3c8c4fd 100644
--- a/services/self-service/src/main/resources/webapp/src/environments/environment.prod.ts
+++ b/services/self-service/src/main/resources/webapp/src/environments/environment.prod.ts
@@ -1,3 +1,18 @@
+/**
+* Copyright (c) 2018, EPAM SYSTEMS INC
+*
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
export const environment = {
production: true
};
diff --git a/services/self-service/src/main/resources/webapp/src/environments/environment.ts b/services/self-service/src/main/resources/webapp/src/environments/environment.ts
index b7f639a..5262678 100644
--- a/services/self-service/src/main/resources/webapp/src/environments/environment.ts
+++ b/services/self-service/src/main/resources/webapp/src/environments/environment.ts
@@ -1,3 +1,18 @@
+/**
+* Copyright (c) 2018, EPAM SYSTEMS INC
+*
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
// The file contents for the current environment will overwrite these during build.
// The build system defaults to the dev environment which uses `environment.ts`, but if you do
// `ng build --env=prod` then `environment.prod.ts` will be used instead.
diff --git a/services/self-service/src/main/resources/webapp/src/polyfills.ts b/services/self-service/src/main/resources/webapp/src/polyfills.ts
index 2a02751..e47be1f 100644
--- a/services/self-service/src/main/resources/webapp/src/polyfills.ts
+++ b/services/self-service/src/main/resources/webapp/src/polyfills.ts
@@ -1,4 +1,20 @@
/**
+* Copyright (c) 2018, EPAM SYSTEMS INC
+*
+* Licensed under the Apache License, Version 2.0 (the "License");
+* you may not use this file except in compliance with the License.
+* You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+ /**
* This file includes polyfills needed by Angular and is loaded before the app.
* You can add your own extra polyfills to this file.
*
diff --git a/services/self-service/src/main/resources/webapp/src/styles.scss b/services/self-service/src/main/resources/webapp/src/styles.scss
index 5196149..972b8a6 100644
--- a/services/self-service/src/main/resources/webapp/src/styles.scss
+++ b/services/self-service/src/main/resources/webapp/src/styles.scss
@@ -50,13 +50,13 @@
display: none !important;
}
-.install-libraries mat-chip.mat-chip {
+mat-chip.mat-chip {
outline: none;
color: #758ea8;
font-size: 14px;
padding: 8px 18px;
}
-.install-libraries mat-chip.mat-chip a {
+mat-chip.mat-chip a {
display: inline-block;
width: 12px;
cursor: pointer;
@@ -66,7 +66,7 @@
vertical-align: middle;
}
-.install-libraries mat-chip.mat-chip strong {
+mat-chip.mat-chip strong {
font-size: 10px;
font-weight: 300;
}
diff --git a/services/self-service/src/main/resources/webapp/src/typings.d.ts b/services/self-service/src/main/resources/webapp/src/typings.d.ts
deleted file mode 100644
index edc2f1e..0000000
--- a/services/self-service/src/main/resources/webapp/src/typings.d.ts
+++ /dev/null
@@ -1,4 +0,0 @@
-/* SystemJS module definition */
-declare var module: {
- id: string;
-};
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/EnvironmentResourceTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/EnvironmentResourceTest.java
index 7a7c5bc..cb36841 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/EnvironmentResourceTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/EnvironmentResourceTest.java
@@ -54,15 +54,15 @@
@Test
public void getUsersWithActiveEnv() {
when(environmentService.getUsers()).thenReturn(Collections.singletonList(new UserDTO("activeUser",
- null)));
+ null, UserDTO.Status.ACTIVE)));
final Response response = resources.getJerseyTest()
- .target("/environment/user/active")
+ .target("/environment/user")
.request()
.header("Authorization", "Bearer " + TOKEN)
.get();
assertEquals(HttpStatus.SC_OK, response.getStatus());
- assertEquals(Collections.singletonList(new UserDTO("activeUser", null)),
+ assertEquals(Collections.singletonList(new UserDTO("activeUser", null, UserDTO.Status.ACTIVE)),
response.readEntity(new GenericType<List<UserDTO>>() {
}));
assertEquals(MediaType.APPLICATION_JSON, response.getHeaderString(HttpHeaders.CONTENT_TYPE));
@@ -75,9 +75,9 @@
public void getUsersWithActiveEnvWithFailedAuth() throws AuthenticationException {
authFailSetup();
when(environmentService.getUsers()).thenReturn(Collections.singletonList(new UserDTO("activeUser",
- null)));
+ null, UserDTO.Status.ACTIVE)));
final Response response = resources.getJerseyTest()
- .target("/environment/user/active")
+ .target("/environment/user")
.request()
.header("Authorization", "Bearer " + TOKEN)
.get();
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserGroupResourceTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserGroupResourceTest.java
index 58a1305..b50e0cb 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserGroupResourceTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserGroupResourceTest.java
@@ -20,7 +20,7 @@
package com.epam.dlab.backendapi.resources;
-import com.epam.dlab.backendapi.resources.dto.CreateGroupDto;
+import com.epam.dlab.backendapi.resources.dto.GroupDTO;
import com.epam.dlab.backendapi.resources.dto.UpdateRoleGroupDto;
import com.epam.dlab.backendapi.resources.dto.UpdateUserGroupDto;
import com.epam.dlab.backendapi.resources.dto.UserGroupDto;
@@ -52,7 +52,7 @@
private static final String USER = "user";
private static final String ROLE_ID = "id";
private static final String GROUP = "group";
- private UserGroupService rolesService = mock(UserGroupService.class);
+ private UserGroupService userGroupService = mock(UserGroupService.class);
@Before
public void setup() throws AuthenticationException {
@@ -61,7 +61,7 @@
@Rule
public final ResourceTestRule resources =
- getResourceTestRuleInstance(new UserGroupResource(rolesService));
+ getResourceTestRuleInstance(new UserGroupResource(userGroupService));
@Test
public void createGroup() {
@@ -74,8 +74,8 @@
assertEquals(HttpStatus.SC_OK, response.getStatus());
- verify(rolesService).createGroup(GROUP, Collections.singleton(ROLE_ID), Collections.singleton(USER));
- verifyNoMoreInteractions(rolesService);
+ verify(userGroupService).createGroup(GROUP, Collections.singleton(ROLE_ID), Collections.singleton(USER));
+ verifyNoMoreInteractions(userGroupService);
}
@Test
@@ -89,7 +89,7 @@
assertEquals(HttpStatus.SC_UNPROCESSABLE_ENTITY, response.getStatus());
- verifyZeroInteractions(rolesService);
+ verifyZeroInteractions(userGroupService);
}
@Test
@@ -103,12 +103,27 @@
assertEquals(HttpStatus.SC_UNPROCESSABLE_ENTITY, response.getStatus());
- verifyZeroInteractions(rolesService);
+ verifyZeroInteractions(userGroupService);
+ }
+
+ @Test
+ public void updateGroup() {
+
+ final Response response = resources.getJerseyTest()
+ .target("/group")
+ .request()
+ .header("Authorization", "Bearer " + TOKEN)
+ .put(Entity.json(getCreateGroupDto(GROUP, Collections.singleton(ROLE_ID))));
+
+ assertEquals(HttpStatus.SC_OK, response.getStatus());
+
+ verify(userGroupService).updateGroup(GROUP, Collections.singleton(ROLE_ID), Collections.singleton(USER));
+ verifyNoMoreInteractions(userGroupService);
}
@Test
public void getGroups() {
- when(rolesService.getAggregatedRolesByGroup()).thenReturn(Collections.singletonList(getUserGroup()));
+ when(userGroupService.getAggregatedRolesByGroup()).thenReturn(Collections.singletonList(getUserGroup()));
final Response response = resources.getJerseyTest()
.target("/group")
@@ -124,8 +139,8 @@
assertTrue(actualRoles.get(0).getRoles().isEmpty());
assertEquals(MediaType.APPLICATION_JSON, response.getHeaderString(HttpHeaders.CONTENT_TYPE));
- verify(rolesService).getAggregatedRolesByGroup();
- verifyNoMoreInteractions(rolesService);
+ verify(userGroupService).getAggregatedRolesByGroup();
+ verifyNoMoreInteractions(userGroupService);
}
@Test
@@ -139,8 +154,8 @@
assertEquals(HttpStatus.SC_OK, response.getStatus());
- verify(rolesService).updateRolesForGroup(GROUP, singleton(ROLE_ID));
- verifyNoMoreInteractions(rolesService);
+ verify(userGroupService).updateRolesForGroup(GROUP, singleton(ROLE_ID));
+ verifyNoMoreInteractions(userGroupService);
}
@Test
@@ -154,7 +169,7 @@
assertEquals(HttpStatus.SC_UNPROCESSABLE_ENTITY, response.getStatus());
- verifyZeroInteractions(rolesService);
+ verifyZeroInteractions(userGroupService);
}
@Test
@@ -170,8 +185,8 @@
assertEquals(HttpStatus.SC_OK, response.getStatus());
- verify(rolesService).removeGroupFromRole(singleton(GROUP), singleton(ROLE_ID));
- verifyNoMoreInteractions(rolesService);
+ verify(userGroupService).removeGroupFromRole(singleton(GROUP), singleton(ROLE_ID));
+ verifyNoMoreInteractions(userGroupService);
}
@Test
@@ -185,8 +200,8 @@
assertEquals(HttpStatus.SC_OK, response.getStatus());
- verify(rolesService).removeGroup(GROUP);
- verifyNoMoreInteractions(rolesService);
+ verify(userGroupService).removeGroup(GROUP);
+ verifyNoMoreInteractions(userGroupService);
}
@Test
@@ -200,7 +215,7 @@
assertEquals(HttpStatus.SC_BAD_REQUEST, response.getStatus());
- verifyZeroInteractions(rolesService);
+ verifyZeroInteractions(userGroupService);
}
@Test
@@ -213,8 +228,8 @@
assertEquals(HttpStatus.SC_OK, response.getStatus());
- verify(rolesService).addUsersToGroup(GROUP, singleton(USER));
- verifyNoMoreInteractions(rolesService);
+ verify(userGroupService).addUsersToGroup(GROUP, singleton(USER));
+ verifyNoMoreInteractions(userGroupService);
}
@Test
@@ -227,7 +242,7 @@
assertEquals(HttpStatus.SC_UNPROCESSABLE_ENTITY, response.getStatus());
- verifyZeroInteractions(rolesService);
+ verifyZeroInteractions(userGroupService);
}
@Test
@@ -243,8 +258,8 @@
assertEquals(HttpStatus.SC_OK, response.getStatus());
- verify(rolesService).removeUserFromGroup(GROUP, USER);
- verifyNoMoreInteractions(rolesService);
+ verify(userGroupService).removeUserFromGroup(GROUP, USER);
+ verifyNoMoreInteractions(userGroupService);
}
@Test
@@ -258,15 +273,15 @@
assertEquals(HttpStatus.SC_BAD_REQUEST, response.getStatus());
- verifyZeroInteractions(rolesService);
+ verifyZeroInteractions(userGroupService);
}
private UserGroupDto getUserGroup() {
return new UserGroupDto(GROUP, Collections.emptyList(), Collections.emptySet());
}
- private CreateGroupDto getCreateGroupDto(String group, Set<String> roleIds) {
- final CreateGroupDto dto = new CreateGroupDto();
+ private GroupDTO getCreateGroupDto(String group, Set<String> roleIds) {
+ final GroupDTO dto = new GroupDTO();
dto.setName(group);
dto.setRoleIds(roleIds);
dto.setUsers(Collections.singleton(USER));
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserRoleResourceTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserRoleResourceTest.java
index 48c2855..6cee51c 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserRoleResourceTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserRoleResourceTest.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.resources;
import com.epam.dlab.backendapi.resources.dto.UserRoleDto;
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserSettingsResourceTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserSettingsResourceTest.java
index a87f879..3c8481b 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserSettingsResourceTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/resources/UserSettingsResourceTest.java
@@ -145,12 +145,12 @@
.target("/user/settings/budget")
.request()
.header("Authorization", "Bearer " + TOKEN)
- .put(Entity.json(singletonList(new UserDTO(USER, 10))));
+ .put(Entity.json(singletonList(new UserDTO(USER, 10, UserDTO.Status.ACTIVE))));
assertEquals(HttpStatus.SC_OK, response.getStatus());
assertNull(response.getHeaderString(HttpHeaders.CONTENT_TYPE));
- verify(userSettingService).updateUsersBudget(singletonList(new UserDTO(USER, 10)));
+ verify(userSettingService).updateUsersBudget(singletonList(new UserDTO(USER, 10, UserDTO.Status.ACTIVE)));
verifyNoMoreInteractions(userSettingService);
}
}
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/schedulers/CheckUserQuoteSchedulerTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/schedulers/CheckUserQuoteSchedulerTest.java
index 1158871..a6c96c6 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/schedulers/CheckUserQuoteSchedulerTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/schedulers/CheckUserQuoteSchedulerTest.java
@@ -49,7 +49,7 @@
@Test
public void testWhenUserQuoteReached() {
when(billingDAO.isUserQuoteReached(anyString())).thenReturn(true);
- when(environmentService.getUsers()).thenReturn(Collections.singletonList(new UserDTO(USER, 1)));
+ when(environmentService.getUsers()).thenReturn(Collections.singletonList(new UserDTO(USER, 1, UserDTO.Status.ACTIVE)));
checkUserQuoteScheduler.execute(jobExecutionContext);
@@ -63,7 +63,7 @@
@Test
public void testWhenUserQuoteNotReached() {
when(billingDAO.isUserQuoteReached(anyString())).thenReturn(false);
- when(environmentService.getUsers()).thenReturn(Collections.singletonList(new UserDTO(USER, 1)));
+ when(environmentService.getUsers()).thenReturn(Collections.singletonList(new UserDTO(USER, 1, UserDTO.Status.ACTIVE)));
checkUserQuoteScheduler.execute(jobExecutionContext);
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/UserRoleServiceImplTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/UserRoleServiceImplTest.java
index 50749b4..7db6ebb 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/UserRoleServiceImplTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/UserRoleServiceImplTest.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.service;
import com.epam.dlab.backendapi.dao.UserRoleDao;
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/UserSettingServiceImplTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/UserSettingServiceImplTest.java
index 4ca56f6..e337c8b 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/UserSettingServiceImplTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/UserSettingServiceImplTest.java
@@ -74,7 +74,7 @@
}
private UserDTO getBudgetDTO() {
- return new UserDTO(USER, BUDGET);
+ return new UserDTO(USER, BUDGET, UserDTO.Status.ACTIVE);
}
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/EnvironmentServiceImplTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/EnvironmentServiceImplTest.java
index 5c79ec2..442dcc5 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/EnvironmentServiceImplTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/EnvironmentServiceImplTest.java
@@ -83,21 +83,25 @@
@Test
public void getActiveUsers() {
- doReturn(Collections.singleton(USER)).when(envDAO).fetchAllUsers();
+ doReturn(Collections.singleton(USER)).when(envDAO).fetchActiveEnvUsers();
+ doReturn(Collections.singleton(USER + "2")).when(envDAO).fetchUsersNotIn(anySet());
when(userSettingsDAO.getAllowedBudget(anyString())).thenReturn(Optional.empty());
final List<UserDTO> activeUsers = environmentService.getUsers();
- assertEquals(1, activeUsers.size());
+ assertEquals(2, activeUsers.size());
assertEquals(USER, activeUsers.get(0).getName());
+ assertEquals(USER + "2", activeUsers.get(1).getName());
verify(userSettingsDAO).getAllowedBudget(USER);
- verify(envDAO).fetchAllUsers();
+ verify(userSettingsDAO).getAllowedBudget(USER + "2");
+ verify(envDAO).fetchActiveEnvUsers();
+ verify(envDAO).fetchUsersNotIn(Collections.singleton(USER));
verifyNoMoreInteractions(envDAO);
}
@Test
public void getActiveUsersWithException() {
- doThrow(new DlabException("Users not found")).when(envDAO).fetchAllUsers();
+ doThrow(new DlabException("Users not found")).when(envDAO).fetchActiveEnvUsers();
expectedException.expect(DlabException.class);
expectedException.expectMessage("Users not found");
@@ -108,7 +112,7 @@
@Test
public void getAllUsers() {
doReturn(Collections.singleton(USER)).when(envDAO).fetchAllUsers();
- final Set<String> users = environmentService.getAllUsers();
+ final Set<String> users = environmentService.getUserNames();
assertEquals(1, users.size());
assertTrue(users.contains(USER));
@@ -124,7 +128,7 @@
expectedException.expect(DlabException.class);
expectedException.expectMessage("Users not found");
- environmentService.getAllUsers();
+ environmentService.getUserNames();
}
@Test
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/InactivityServiceImplTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/InactivityServiceImplTest.java
index 136bfb5..0d4dee9 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/InactivityServiceImplTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/InactivityServiceImplTest.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.service.impl;
import com.epam.dlab.auth.SystemUserInfoService;
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/MavenCentralLibraryServiceTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/MavenCentralLibraryServiceTest.java
index 1426030..b6df094 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/MavenCentralLibraryServiceTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/MavenCentralLibraryServiceTest.java
@@ -23,6 +23,7 @@
import com.epam.dlab.backendapi.domain.MavenSearchArtifactResponse;
import com.epam.dlab.backendapi.resources.dto.LibraryDTO;
import com.epam.dlab.exceptions.DlabException;
+import com.epam.dlab.rest.client.RESTService;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -31,10 +32,6 @@
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;
-import javax.ws.rs.client.Client;
-import javax.ws.rs.client.Invocation;
-import javax.ws.rs.client.WebTarget;
-import javax.ws.rs.core.Response;
import java.net.URI;
import java.util.Collections;
@@ -48,7 +45,7 @@
public class MavenCentralLibraryServiceTest {
@Mock
- private Client client;
+ private RESTService client;
@InjectMocks
private MavenCentralLibraryService mavenCentralLibraryService;
@Rule
@@ -56,16 +53,7 @@
@Test
public void getLibraryId() {
- final WebTarget webTarget = mock(WebTarget.class);
- final Invocation.Builder requestBuilder = mock(Invocation.Builder.class);
- final Response response = mock(Response.class);
- final Response.StatusType statusType = mock(Response.StatusType.class);
- when(client.target(any(URI.class))).thenReturn(webTarget);
- when(webTarget.request()).thenReturn(requestBuilder);
- when(requestBuilder.get()).thenReturn(response);
- when(response.getStatusInfo()).thenReturn(statusType);
- when(statusType.getFamily()).thenReturn(Response.Status.Family.SUCCESSFUL);
- when(response.readEntity(MavenSearchArtifactResponse.class)).thenReturn(getMavenResponse());
+ when(client.get(any(URI.class), any())).thenReturn(getMavenResponse());
final LibraryDTO libDTO = mavenCentralLibraryService.getLibrary("groupId", "artifactId", "version");
@@ -73,25 +61,15 @@
assertEquals("groupId:artifactId", libDTO.getName());
assertEquals("version", libDTO.getVersion());
- verify(client).target(refEq(URI.create("http://search.maven.org/solrsearch/select?q=a:%22artifactId%22+AND+g" +
- ":%22groupId%22+AND+v:%22version%22+AND+p:%22jar%22&rows=20&wt=json&core=gav&p=jar")));
- verify(webTarget).request();
- verify(response).readEntity(MavenSearchArtifactResponse.class);
+ verify(client).get(refEq(URI.create("/solrsearch/select?q=a:%22artifactId%22+AND+g" +
+ ":%22groupId%22+AND+v:%22version%22+AND+p:%22jar%22&rows=20&wt=json&core=gav&p=jar")),
+ eq(MavenSearchArtifactResponse.class));
}
@Test
public void getLibraryIdWithException() {
- final WebTarget webTarget = mock(WebTarget.class);
- final Invocation.Builder requestBuilder = mock(Invocation.Builder.class);
- final Response response = mock(Response.class);
- final Response.StatusType statusType = mock(Response.StatusType.class);
- when(client.target(any(URI.class))).thenReturn(webTarget);
- when(webTarget.request()).thenReturn(requestBuilder);
- when(requestBuilder.get()).thenReturn(response);
- when(response.getStatusInfo()).thenReturn(statusType);
- when(statusType.getStatusCode()).thenReturn(500);
- when(statusType.getReasonPhrase()).thenReturn("Exception");
- when(response.readEntity(MavenSearchArtifactResponse.class)).thenReturn(getMavenResponse());
+ when(client.get(any(URI.class), any())).thenThrow(new DlabException("Can not get artifact info from maven " +
+ "central due to: Exception"));
expectedException.expect(DlabException.class);
expectedException.expectMessage("Can not get artifact info from maven central due to: Exception");
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/ReuploadKeyServiceImplTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/ReuploadKeyServiceImplTest.java
index 6ce9e37..cbdc67e 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/ReuploadKeyServiceImplTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/ReuploadKeyServiceImplTest.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.service.impl;
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/UserGroupServiceImplTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/UserGroupServiceImplTest.java
index b85b228..6298d7d 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/UserGroupServiceImplTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/UserGroupServiceImplTest.java
@@ -51,7 +51,7 @@
@Mock
private UserGroupDao userGroupDao;
@InjectMocks
- private UserGroupServiceImpl userRolesService;
+ private UserGroupServiceImpl userGroupService;
@Rule
public ExpectedException expectedException = ExpectedException.none();
@@ -60,7 +60,7 @@
public void createGroup() {
when(userRoleDao.addGroupToRole(anySet(), anySet())).thenReturn(true);
- userRolesService.createGroup(GROUP, Collections.singleton(ROLE_ID), Collections.singleton(USER));
+ userGroupService.createGroup(GROUP, Collections.singleton(ROLE_ID), Collections.singleton(USER));
verify(userRoleDao).addGroupToRole(Collections.singleton(GROUP), Collections.singleton(ROLE_ID));
verify(userGroupDao).addUsers(GROUP, Collections.singleton(USER));
@@ -70,7 +70,7 @@
public void createGroupWithNoUsers() {
when(userRoleDao.addGroupToRole(anySet(), anySet())).thenReturn(true);
- userRolesService.createGroup(GROUP, Collections.singleton(ROLE_ID), Collections.emptySet());
+ userGroupService.createGroup(GROUP, Collections.singleton(ROLE_ID), Collections.emptySet());
verify(userRoleDao).addGroupToRole(Collections.singleton(GROUP), Collections.singleton(ROLE_ID));
verify(userGroupDao, never()).addUsers(anyString(), anySet());
@@ -81,14 +81,14 @@
when(userRoleDao.addGroupToRole(anySet(), anySet())).thenReturn(false);
expectedException.expect(ResourceNotFoundException.class);
- userRolesService.createGroup(GROUP, Collections.singleton(ROLE_ID), Collections.singleton(USER));
+ userGroupService.createGroup(GROUP, Collections.singleton(ROLE_ID), Collections.singleton(USER));
}
@Test
public void getAggregatedRoles() {
when(userRoleDao.aggregateRolesByGroup()).thenReturn(Collections.singletonList(getUserGroup()));
- final List<UserGroupDto> aggregatedRolesByGroup = userRolesService.getAggregatedRolesByGroup();
+ final List<UserGroupDto> aggregatedRolesByGroup = userGroupService.getAggregatedRolesByGroup();
assertEquals(1, aggregatedRolesByGroup.size());
assertEquals(GROUP, aggregatedRolesByGroup.get(0).getGroup());
@@ -100,7 +100,7 @@
@Test
public void addUserToGroup() {
- userRolesService.addUsersToGroup(GROUP, Collections.singleton(USER));
+ userGroupService.addUsersToGroup(GROUP, Collections.singleton(USER));
verify(userGroupDao).addUsers(eq(GROUP), refEq(Collections.singleton(USER)));
verifyNoMoreInteractions(userRoleDao, userGroupDao);
@@ -110,7 +110,7 @@
public void addRolesToGroup() {
when(userRoleDao.addGroupToRole(anySetOf(String.class), anySetOf(String.class))).thenReturn(true);
- userRolesService.updateRolesForGroup(GROUP, Collections.singleton(ROLE_ID));
+ userGroupService.updateRolesForGroup(GROUP, Collections.singleton(ROLE_ID));
verify(userRoleDao).addGroupToRole(refEq(Collections.singleton(GROUP)), refEq(Collections.singleton(ROLE_ID)));
verify(userRoleDao).removeGroupWhenRoleNotIn(GROUP, Collections.singleton(ROLE_ID));
@@ -120,7 +120,7 @@
@Test
public void removeUserFromGroup() {
- userRolesService.removeUserFromGroup(GROUP, USER);
+ userGroupService.removeUserFromGroup(GROUP, USER);
verify(userGroupDao).removeUser(GROUP, USER);
verifyNoMoreInteractions(userGroupDao);
@@ -131,7 +131,7 @@
when(userRoleDao.removeGroupFromRole(anySetOf(String.class), anySetOf(String.class))).thenReturn(true);
- userRolesService.removeGroupFromRole(Collections.singleton(GROUP), Collections.singleton(ROLE_ID));
+ userGroupService.removeGroupFromRole(Collections.singleton(GROUP), Collections.singleton(ROLE_ID));
verify(userRoleDao).removeGroupFromRole(refEq(Collections.singleton(GROUP)),
refEq(Collections.singleton(ROLE_ID)));
@@ -145,7 +145,7 @@
expectedException.expectMessage("Any of role : [" + ROLE_ID + "] were not found");
expectedException.expect(ResourceNotFoundException.class);
- userRolesService.removeGroupFromRole(Collections.singleton(GROUP), Collections.singleton(ROLE_ID));
+ userGroupService.removeGroupFromRole(Collections.singleton(GROUP), Collections.singleton(ROLE_ID));
}
@Test
@@ -154,7 +154,7 @@
when(userRoleDao.removeGroup(anyString())).thenReturn(true);
doNothing().when(userGroupDao).removeGroup(anyString());
- userRolesService.removeGroup(GROUP);
+ userGroupService.removeGroup(GROUP);
verify(userRoleDao).removeGroup(GROUP);
verify(userGroupDao).removeGroup(GROUP);
@@ -167,7 +167,7 @@
when(userRoleDao.removeGroup(anyString())).thenReturn(false);
doNothing().when(userGroupDao).removeGroup(anyString());
- userRolesService.removeGroup(GROUP);
+ userGroupService.removeGroup(GROUP);
verify(userRoleDao).removeGroup(GROUP);
verify(userGroupDao, never()).removeGroup(GROUP);
@@ -181,7 +181,17 @@
expectedException.expectMessage("Exception");
expectedException.expect(DlabException.class);
- userRolesService.removeGroup(GROUP);
+ userGroupService.removeGroup(GROUP);
+ }
+
+ @Test
+ public void updateGroup() {
+ userGroupService.updateGroup(GROUP, Collections.singleton(ROLE_ID), Collections.singleton(USER));
+
+ verify(userGroupDao).updateUsers(GROUP, Collections.singleton(USER));
+ verify(userRoleDao).removeGroupWhenRoleNotIn(GROUP, Collections.singleton(ROLE_ID));
+ verify(userRoleDao).addGroupToRole(Collections.singleton(GROUP), Collections.singleton(ROLE_ID));
+ verifyNoMoreInteractions(userRoleDao, userGroupDao);
}
private UserGroupDto getUserGroup() {
diff --git a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/UserResourceServiceImplTest.java b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/UserResourceServiceImplTest.java
index 47a000d..16c6793 100644
--- a/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/UserResourceServiceImplTest.java
+++ b/services/self-service/src/test/java/com/epam/dlab/backendapi/service/impl/UserResourceServiceImplTest.java
@@ -1,3 +1,20 @@
+/*
+ *
+ * * Copyright (c) 2018, EPAM SYSTEMS INC
+ * *
+ * * Licensed under the Apache License, Version 2.0 (the "License");
+ * * you may not use this file except in compliance with the License.
+ * * You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
package com.epam.dlab.backendapi.service.impl;
