SUBMARINE-938. [User] Modify user documentation
### What is this PR for?
1. Delete unimportant information and add critical ones.
2. Make the style consistent in the subpages.
### What type of PR is it?
[Documentation]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-938
### How should this be tested?
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
Author: ByronHsu <byronhsu1230@gmail.com>
Signed-off-by: Kevin <pingsutw@apache.org>
Closes #678 from ByronHsu/modify-user-page and squashes the following commits:
dfe6f724 [ByronHsu] fix typo
c3ee1aa5 [ByronHsu] revert yarn
01c1269d [ByronHsu] revise user doc
diff --git a/dev-support/examples/quickstart/post.sh b/dev-support/examples/quickstart/post.sh
new file mode 100644
index 0000000..39336bc
--- /dev/null
+++ b/dev-support/examples/quickstart/post.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+curl -X POST -H "Content-Type: application/json" -d '
+{
+ "meta": {
+ "name": "quickstart",
+ "namespace": "default",
+ "framework": "TensorFlow",
+ "cmd": "python /opt/train.py",
+ "envVars": {
+ "ENV_1": "ENV1"
+ }
+ },
+ "environment": {
+ "image": "quickstart:0.6.0-SNAPSHOT"
+ },
+ "spec": {
+ "Worker": {
+ "replicas": 3,
+ "resources": "cpu=1,memory=1024M"
+ }
+ }
+}
+' http://127.0.0.1:32080/api/v1/experiment
\ No newline at end of file
diff --git a/dev-support/examples/tracking/tracking.py b/dev-support/examples/tracking/tracking.py
index 641c25e..61c9ad8 100644
--- a/dev-support/examples/tracking/tracking.py
+++ b/dev-support/examples/tracking/tracking.py
@@ -20,10 +20,10 @@
import time
if __name__ == "__main__":
- periscope = ModelsClient()
- with periscope.start() as run:
- periscope.log_param("learning_rate", random.random())
+ modelClient = ModelsClient()
+ with modelClient.start() as run:
+ modelClient.log_param("learning_rate", random.random())
for i in range(100):
time.sleep(1)
- periscope.log_metric("mse", random.random() * 100, i)
- periscope.log_metric("acc", random.random(), i)
\ No newline at end of file
+ modelClient.log_metric("mse", random.random() * 100, i)
+ modelClient.log_metric("acc", random.random(), i)
\ No newline at end of file
diff --git a/website/docs/userDocs/api/environment.md b/website/docs/userDocs/api/environment.md
index 8559aed..851024b 100644
--- a/website/docs/userDocs/api/environment.md
+++ b/website/docs/userDocs/api/environment.md
@@ -45,9 +45,9 @@
| Field Name | Type | Description |
| ----------------- | -------------- | ---------------------------------- |
| name | String | Kernel name. |
-| channels | List\<String\> | Names of the channels. |
-| condaDependencies | List\<String\> | List of kernel conda dependencies. |
-| pipDependencies | List\<String\> | List of kernel pip dependencies. |
+| channels | List<String\> | Names of the channels. |
+| condaDependencies | List<String\> | List of kernel conda dependencies. |
+| pipDependencies | List<String\> | List of kernel pip dependencies. |
### Code Example
diff --git a/website/docs/userDocs/api/experiment-template.md b/website/docs/userDocs/api/experiment-template.md
index 274748d..76dbed5 100644
--- a/website/docs/userDocs/api/experiment-template.md
+++ b/website/docs/userDocs/api/experiment-template.md
@@ -34,7 +34,7 @@
| name | String | body | Experiment template name. This is required. |
| author | String | body | Author name. |
| description | String | body | Description of the experiment template. |
-| parameters | List\<ExperimentTemplateParamSpec\> | body | Parameters of the experiment template. |
+| parameters | List<ExperimentTemplateParamSpec\> | body | Parameters of the experiment template. |
| experimentSpec | ExperimentSpec | body | Spec of the experiment template. |
#### **ExperimentTemplateParamSpec**
@@ -52,7 +52,7 @@
| ----------- | --------------------------------- | --------------------------------------- |
| meta | ExperimentMeta | Meta data of the experiment template. |
| environment | EnvironmentSpec | Environment of the experiment template. |
-| spec | Map\<String, ExperimentTaskSpec\> | Spec of pods. |
+| spec | Map<String, ExperimentTaskSpec\> | Spec of pods. |
| code | CodeSpec | Experiment codespec. |
#### **ExperimentMeta**
@@ -62,7 +62,7 @@
| namespace | String | Experiment namespace. |
| framework | String | Experiment framework. |
| cmd | String | Command. |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String\> | Environmental variables. |
#### **EnvironmentSpec**
@@ -77,7 +77,7 @@
| name | String | Task name. |
| image | String | Image name. |
| cmd | String | Command. |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String\> | Environmental variables. |
#### **CodeSpec**
@@ -413,7 +413,7 @@
| name | String | path and body | Experiment template name. This is required. |
| author | String | body | Author name. |
| description | String | body | Description of the experiment template. |
-| parameters | List\<ExperimentTemplateParamSpec\> | body | Parameters of the experiment template. |
+| parameters | List<ExperimentTemplateParamSpec\> | body | Parameters of the experiment template. |
| experimentSpec | ExperimentSpec | body | Spec of the experiment template. |
### Code Example
diff --git a/website/docs/userDocs/api/experiment.md b/website/docs/userDocs/api/experiment.md
index e2e534a..87b8019 100644
--- a/website/docs/userDocs/api/experiment.md
+++ b/website/docs/userDocs/api/experiment.md
@@ -37,7 +37,7 @@
| ----------- | --------------------------------- | --------------------------------------- |
| meta | ExperimentMeta | Meta data of the experiment template. |
| environment | EnvironmentSpec | Environment of the experiment template. |
-| spec | Map\<String, ExperimentTaskSpec\> | Spec of pods. |
+| spec | Map<String, ExperimentTaskSpec> | Spec of pods. |
| code | CodeSpec | Experiment codespec. |
#### **ExperimentMeta**
@@ -48,10 +48,14 @@
| namespace | String | Experiment namespace. |
| framework | String | Experiemnt framework. |
| cmd | String | Command. |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String> | Environmental variables. |
#### **EnvironmentSpec**
+There are two types of environment: Anonymous and Predefined.
+- Anonymous environment: only specify `dockerImage` in environment spec. The container will be built on the docker image.
+- Embedded environment: specify `name` in environment spec. The container will be built on the existing environment (including dockerImage and kernalSpec).
+
See more details in [environment api](https://submarine.apache.org/docs/userDocs/api/environment).
#### **ExperimentTaskSpec**
@@ -63,14 +67,16 @@
| name | String | Task name. |
| image | String | Image name. |
| cmd | String | Command. |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String> | Environmental variables. |
#### **CodeSpec**
-| Field Name | Type | Description |
-| ---------- | ------ | ----------------------- |
-| syncMode | String | sync mode of code spec. |
-| url | String | url of code spec. |
+Currently only support pulling from github. HDFS, NFS and s3 are in development
+
+| Field Name | Type | Description |
+| ---------- | ------------------------------| ----------------------- |
+| syncMode | String \(git\|hdfs\|nfs\|s3\) | sync mode of code spec. |
+| url | String | url of code spec. |
### Code Example
@@ -479,7 +485,7 @@
| id | String | path | Experiment id. |
| meta | ExperimentMeta | body | Meta data of the experiment template. |
| environment | EnvironmentSpec | body | Environment of the experiment template. |
-| spec | Map\<String, ExperimentTaskSpec\> | body | Spec of pods. |
+| spec | Map<String, ExperimentTaskSpec> | body | Spec of pods. |
| code | CodeSpec | body | TODO |
### Code Example
diff --git a/website/docs/userDocs/api/notebook.md b/website/docs/userDocs/api/notebook.md
index f3e2ee6..1c9eb43 100644
--- a/website/docs/userDocs/api/notebook.md
+++ b/website/docs/userDocs/api/notebook.md
@@ -55,7 +55,7 @@
| Field Name | Type | Description |
| ---------- | --------------------- | ------------------------ |
-| envVars | Map\<String, String\> | Environmental variables. |
+| envVars | Map<String, String\> | Environmental variables. |
| resources | String | Resourecs of the pod. |
### Code Example
diff --git a/website/docs/userDocs/k8s/how-to-use-tensorboard.md b/website/docs/userDocs/k8s/how-to-use-tensorboard.md
deleted file mode 100644
index d5ef5cf..0000000
--- a/website/docs/userDocs/k8s/how-to-use-tensorboard.md
+++ /dev/null
@@ -1,64 +0,0 @@
----
-title: How to use tensorboard in Submarine
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-## Write to LogDirs by the environment variable
-
-### Environment variable
-
-- `SUBMARINE_TENSORBOARD_LOG_DIR`: Exist in every experiment container. You just need to direct your logs to `$(SUBMARINE_TENSORBOARD_LOG_DIR)` (**NOTICE: it is `()` not `{}`**), and you can inspect the process on the tensorboard webpage.
-
-### Example
-
-```
-{
- "meta": {
- "name": "tensorflow-tensorboard-dist-mnist",
- "namespace": "default",
- "framework": "TensorFlow",
- "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=$(SUBMARINE_TENSORBOARD_LOG_DIR) --learning_rate=0.01 --batch_size=20",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:tf-mnist-with-summaries-1.0"
- },
- "spec": {
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=512M"
- }
- }
-}
-
-```
-
-## Connect to the tensorboard webpaage
-
-1. Open the experiment page in the workbench, and Click the `TensorBoard` button.
-
-
-
-2. Inspect the process on tensorboard page.
-
-
diff --git a/website/docs/userDocs/k8s/run-experiment-template-rest.md b/website/docs/userDocs/k8s/run-experiment-template-rest.md
deleted file mode 100644
index d379aaa..0000000
--- a/website/docs/userDocs/k8s/run-experiment-template-rest.md
+++ /dev/null
@@ -1,58 +0,0 @@
----
-title: Run Experiment Template Guide (REST)
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-## Experiment Template Spec
-
-The experiment is represented in [JSON](https://www.json.org) or [YAML](https://yaml.org) format.
-
-### Use existing experiment template to create a experiment
-
-`POST /api/v1/experiment/{template-name}`
-
-**Example Request:**
-
-```sh
-curl -X POST -H "Content-Type: application/json" -d '
-{
- "params": {
- "learning_rate":"0.01",
- "batch_size":"150",
- "experiment_name":"newexperiment1"
- }
-}
-' http://127.0.0.1:32080/api/v1/experiment/tf-mnist
-```
-
-**Example Request:**
-
-```sh
-curl -X POST -H "Content-Type: application/json" -d '
-{
- "params": {
- "experiment_name":"new-pytorch-mnist"
- }
-}
-' http://127.0.0.1:32080/api/v1/experiment/pytorch-mnist
-```
-
-Register experiment template and more info see [Experiment Template API Reference](api/experiment-template.md).
diff --git a/website/docs/userDocs/k8s/run-pytorch-experiment-rest.md b/website/docs/userDocs/k8s/run-pytorch-experiment-rest.md
deleted file mode 100644
index f3c0053..0000000
--- a/website/docs/userDocs/k8s/run-pytorch-experiment-rest.md
+++ /dev/null
@@ -1,148 +0,0 @@
----
-title: Run PyTorch Experiment Guide (REST)
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-## Experiment Spec
-The experiment is represented in [JSON](https://www.json.org) or [YAML](https://yaml.org) format.
-
-**YAML Format:**
-```yaml
-meta:
- name: pytorch-mnist-yaml
- namespace: default
- framework: PyTorch
- cmd: python /var/mnist.py --backend gloo
- envVars:
- ENV_1: ENV1
-environment:
- image: apache/submarine:pytorch-dist-mnist-1.0
-spec:
- Master:
- replicas: 1
- resources: cpu=1,memory=1024M
- Worker:
- replicas: 1
- resources: cpu=1,memory=1024M
-```
-
-**JSON Format:**
-```json
-{
- "meta": {
- "name": "pytorch-mnist-json",
- "namespace": "default",
- "framework": "PyTorch",
- "cmd": "python /var/mnist.py --backend gloo",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:pytorch-dist-mnist-1.0"
- },
- "spec": {
- "Master": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
-}
-
-```
-
-## Create Experiment by REST API
-`POST /api/v1/experiment`
-
-**Example Request**
-```sh
-curl -X POST -H "Content-Type: application/json" -d '
-{
- "meta": {
- "name": "pytorch-mnist-json",
- "namespace": "default",
- "framework": "PyTorch",
- "cmd": "python /var/mnist.py --backend gloo",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:pytorch-dist-mnist-1.0"
- },
- "spec": {
- "Master": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
-}
-' http://127.0.0.1:32080/api/v1/experiment
-```
-
-**Example Response:**
-```json
-{
- "status": "OK",
- "code": 200,
- "result": {
- "experimentId": "experiment_1592057447228_0002",
- "name": "mnist",
- "uid": "38e39dcd-77d4-11ea-8dbb-0242ac110003",
- "status": "Accepted",
- "acceptedTime": "2020-06-13T22:19:29.000+08:00",
- "spec": {
- "meta": {
- "name": "pytorch-mnist-json",
- "namespace": "default",
- "framework": "PyTorch",
- "cmd": "python /var/mnist.py --backend gloo",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:pytorch-dist-mnist-1.0"
- },
- "spec": {
- "Master": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
- }
- }
-}
-```
-
-More info see [Experiment API Reference](api/experiment.md).
diff --git a/website/docs/userDocs/k8s/run-tensorflow-experiment-rest.md b/website/docs/userDocs/k8s/run-tensorflow-experiment-rest.md
deleted file mode 100644
index ae1c304..0000000
--- a/website/docs/userDocs/k8s/run-tensorflow-experiment-rest.md
+++ /dev/null
@@ -1,147 +0,0 @@
----
-title: Run TensorFlow Experiment Guide (REST)
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-## Experiment Spec
-The experiment is represented in [JSON](https://www.json.org) or [YAML](https://yaml.org) format.
-
-**YAML Format:**
-```yaml
-meta:
- name: "tf-mnist-yaml"
- namespace: "default"
- framework: "TensorFlow"
- cmd: "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150"
- envVars:
- ENV_1: "ENV1"
-environment:
- image: "apache/submarine:tf-mnist-with-summaries-1.0"
-spec:
- Ps:
- replicas: 1
- resources: "cpu=1,memory=1024M"
- Worker:
- replicas: 1
- resources: "cpu=1,memory=1024M"
-```
-
-**JSON Format:**
-```json
-{
- "meta": {
- "name": "tf-mnist-json",
- "namespace": "default",
- "framework": "TensorFlow",
- "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:tf-mnist-with-summaries-1.0"
- },
- "spec": {
- "Ps": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
-}
-```
-
-## Create Experiment by REST API
-`POST /api/v1/experiment`
-
-**Example Request**
-```sh
-curl -X POST -H "Content-Type: application/json" -d '
-{
- "meta": {
- "name": "tf-mnist-json",
- "namespace": "default",
- "framework": "TensorFlow",
- "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:tf-mnist-with-summaries-1.0"
- },
- "spec": {
- "Ps": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
-}
-' http://127.0.0.1:32080/api/v1/experiment
-```
-
-**Example Response:**
-```json
-{
- "status": "OK",
- "code": 200,
- "result": {
- "experimentId": "experiment_1592057447228_0001",
- "name": "tf-mnist-json",
- "uid": "28e39dcd-77d4-11ea-8dbb-0242ac110003",
- "status": "Accepted",
- "acceptedTime": "2020-06-13T22:59:29.000+08:00",
- "spec": {
- "meta": {
- "name": "tf-mnist-json",
- "namespace": "default",
- "framework": "TensorFlow",
- "cmd": "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150",
- "envVars": {
- "ENV_1": "ENV1"
- }
- },
- "environment": {
- "image": "apache/submarine:tf-mnist-with-summaries-1.0"
- },
- "spec": {
- "Ps": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- },
- "Worker": {
- "replicas": 1,
- "resources": "cpu=1,memory=1024M"
- }
- }
- }
- }
-}
-```
-
-More info see [Experiment API Reference](api/experiment.md).
diff --git a/website/docs/userDocs/others/mlflow.md b/website/docs/userDocs/others/mlflow.md
index 6d1043c..0341b4f 100644
--- a/website/docs/userDocs/others/mlflow.md
+++ b/website/docs/userDocs/others/mlflow.md
@@ -29,11 +29,25 @@
UI.
### Example
-1. `cd ./submarine/dev-support/examples/tracking`
-2. `eval $(minikube -p minikube docker-env)`
-3. `sh build.sh`(it will take few minutes)
-4. `sh post.sh`
-5. In the MLflow UI page, you can see the log_param and the log_metric
+1. Run the following code in the cluster
+
+```python
+from submarine import ModelsClient
+import random
+import time
+
+if __name__ == "__main__":
+ modelClient = ModelsClient()
+ with modelClient.start() as run:
+ modelClient.log_param("learning_rate", random.random())
+ for i in range(100):
+ time.sleep(1)
+ modelClient.log_metric("mse", random.random() * 100, i)
+ modelClient.log_metric("acc", random.random(), i)
+```
+
+2. In the MLflow UI page, you can see the log_param and the log_metric
result. You can also compare the training between different workers.
+
diff --git a/website/docs/userDocs/submarine-sdk/experiment-client.md b/website/docs/userDocs/submarine-sdk/experiment-client.md
index e238595..096a0a9 100644
--- a/website/docs/userDocs/submarine-sdk/experiment-client.md
+++ b/website/docs/userDocs/submarine-sdk/experiment-client.md
@@ -30,8 +30,8 @@
Create an experiment.
> **Parameters**
- **experiment_spec**: Submarine experiment spec. More detailed information can be found at [Experiment API](https://submarine.apache.org/docs/userDocs/api/experiment).
-> Returns
- - The detailed info about the submarine experiment.
+
+> **Returns**: The detailed info about the submarine experiment.
Example
@@ -64,63 +64,14 @@
})
```
-Output
-
-```python
-{
- 'experimentId': 'experiment_1626160071451_0008',
- 'name': 'tf-mnist-json', 'uid': '3513233e-33f2-4399-8fba-2a44ca2af730',
- 'status': 'Accepted',
- 'acceptedTime': '2021-07-13T21:29:33.000+08:00',
- 'createdTime': None,
- 'runningTime': None,
- 'finishedTime': None,
- 'spec': {
- 'meta': {
- 'name': 'tf-mnist-json',
- 'namespace': 'default',
- 'framework': 'TensorFlow',
- 'cmd': 'python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log --learning_rate=0.01 --batch_size=150',
- 'envVars': {'ENV_1': 'ENV1'}
- },
- 'environment': {
- 'name': None,
- 'dockerImage': None,
- 'kernelSpec': None,
- 'description': None,
- 'image': 'apache/submarine:tf-mnist-with-summaries-1.0'
- },
- 'spec': {
- 'Ps': {
- 'replicas': 1,
- 'resources': 'cpu=1,memory=1024M',
- 'name': None,
- 'image': None,
- 'cmd': None,
- 'envVars': None,
- 'resourceMap': {'memory': '1024M', 'cpu': '1'}
- },
- 'Worker': {
- 'replicas': 1,
- 'resources': 'cpu=1,memory=1024M',
- 'name': None,
- 'image': None,
- 'cmd': None,
- 'envVars': None,
- 'resourceMap': {'memory': '1024M', 'cpu': '1'}
- }
- },
- 'code': None
- }
-}
-```
-
### `patch_experiment(id: str, experiment_spec: json) -> dict`
Patch an experiment.
> **Parameters**
- **id**: Submarine experiment id.
- **experiment_spec**: Submarine experiment spec. More detailed information can be found at [Experiment API](https://submarine.apache.org/docs/userDocs/api/experiment).
+
+
> **Returns**
- The detailed info about the submarine experiment.
@@ -203,26 +154,21 @@
- **id**: Submarine experiment id.
- **onlyMaster**: By default include pod log of "master" which might be Tensorflow PS/Chief or PyTorch master.
+> **Return**
+ - The info of pod logs
+
Example
```python
client.get_log("experiment_1626160071451_0009")
```
-Output
-
-```
-The logs of Pod tf-mnist-json-2-ps-0:
-
-The logs of Pod tf-mnist-json-2-worker-0:
-
-```
-
### `list_log(status: str) -> list[dict]`
List experiment log.
> **Parameters**
- **status**: Accepted, Created, Running, Succeeded, Deleted.
+
> **Returns**
- List of submarine experiment logs.
@@ -232,22 +178,13 @@
logs = client.list_log("Succeeded")
```
-Output
-
-```python
-[{'experimentId': 'experiment_1626160071451_0009',
- 'logContent':
- [{'podName': 'tf-mnist-json-2-ps-0', 'podLog': []},
- {'podName': 'tf-mnist-json-2-worker-0', 'podLog': []}]
-}]
-```
-
### `wait_for_finish(id: str, polling_interval: Optional[int]=10) -> dict`
Waits until the experiment is finished or failed.
> **Parameters**
- **id**: Submarine experiment id.
- **polling_interval**: How many seconds between two polls for the status of the experiment.
+
> **Returns**
- Submarine experiment logs.
diff --git a/website/docs/userDocs/submarine-sdk/model-client.md b/website/docs/userDocs/submarine-sdk/model-client.md
index 150d3a1..9e928d2 100644
--- a/website/docs/userDocs/submarine-sdk/model-client.md
+++ b/website/docs/userDocs/submarine-sdk/model-client.md
@@ -21,122 +21,123 @@
under the License.
-->
-The submarine ModelsClient provides a high-level API for managing and serving your models.
+## class ModelClient()
-## ModelsClient.start()->[Active Run](https://mlflow.org/docs/latest/_modules/mlflow/tracking/fluent.html#ActiveRun)
+The submarine ModelsClient provides a high-level API for logging metrics / parameters and managing models.
-1. Start a new Mlflow run
-2. Direct the logging of the artifacts and metadata to the Run named "worker_i" under Experiment "job_id"
-3. If in distributed training, worker and job id would be parsed from environment variable
-4. If in local traning, worker and job id will be generated.
- :return: Active Run
+### `ModelsClient(tracking_uri=None, registry_uri=None)->ModelsClient`
-## ModelsClient.log_param(key, value)->None
+Initialize a `ModelsClient` instance.
+
+> **Parameters**
+ - **tracking_uri**: If run in Submarine, you do not need to specify it. Otherwise, specify the external tracking_uri.
+ - **registry_uri**: If run in Submarine, you do not need to specify it. Otherwise, specify the external registry_uri.
+
+> **Returns**
+ - ModelsClient instance
+
+Example
+
+```python
+from submarine import ModelsClient
+
+modelClient = ModelsClient(tracking_uri="0.0.0.0:4000", tracking_uri="0.0.0.0:5000")
+```
+### `ModelsClient.start()->[Active Run]`
+
+For details of [Active Run](https://mlflow.org/docs/latest/_modules/mlflow/tracking/fluent.html#ActiveRun)
+
+Start a new Mlflow run, and direct the logging of the artifacts and metadata to the Run named "worker_i" under Experiment "job_id". If in distributed training, worker and job id would be parsed from environment variable. If in local traning, worker and job id will be generated.
+
+> **Returns**
+ - Active Run
+
+### `ModelsClient.log_param(key, value)->None`
Log parameter under the current run.
-- ### Parameters
- - **key** – Parameter name (string)
- - **value** – Parameter value (string, but will be string-ified if not)
-- ### example
+> **Parameters**
+ - **key** – Parameter name
+ - **value** – Parameter value
-```
+Example
+
+```python
from submarine import ModelsClient
-periscope = ModelsClient()
-with periscope.start() as run:
- periscope.log_param("learning_rate", 0.01)
+modelClient = ModelsClient()
+with modelClient.start() as run:
+ modelClient.log_param("learning_rate", 0.01)
```
-## ModelsClient.log_params(params)->None
+### `ModelsClient.log_params(params)->None`
Log a batch of params for the current run.
-- ### Parameters
+> **Parameters**
+ - **params** – Dictionary of param_name: String -> value
- - **params** – Dictionary of param_name: String -> value: (String, but will be string-ified if not)
+Example
-- ### example
-
-```
+```python
from submarine import ModelsClient
params = {"learning_rate": 0.01, "n_estimators": 10}
-periscope = ModelsClient()
-with periscope.start() as run:
- periscope.log_params(params)
+modelClient = ModelsClient()
+with modelClient.start() as run:
+ modelClient.log_params(params)
```
-## ModelsClient.log_metric(self, key, value, step=None)->None
+### `ModelsClient.log_metric(self, key, value, step=None)->None`
Log a metric under the current run.
-- ### Parameters
-
+> **Parameters**
- **key** – Metric name (string).
- - **value** – Metric value (float). Note that some special values such as +/- Infinity may be replaced by other values depending on the store. For example, the SQLAlchemy store replaces +/- Infinity with max / min float values.
+ - **value** – Metric value (float).
- **step** – Metric step (int). Defaults to zero if unspecified.
-- ### example
+Example
-```
+```python
from submarine import ModelsClient
-periscope = ModelsClient()
-with periscope.start() as run:
- periscope.log_metric("mse", 2500.00)
+modelClient = ModelsClient()
+with modelClient.start() as run:
+ modelClient.log_metric("mse", 2500.00)
```
-## ModelsClient.log_metrics(self, metrics, step=None)->None
+### `ModelsClient.log_metrics(self, metrics, step=None)->None`
Log multiple metrics for the current run.
-- ### Parameters
-
- - **metrics** – Dictionary of metric_name: String -> value: Float. Note that some special values such as +/- Infinity may be replaced by other values depending on the store. For example, sql based store may replace +/- Infinity with max / min float values.
+> **Parameters**
+ - **metrics** – Dictionary of metric_name: String -> value: Float.
- **step** – A single integer step at which to log the specified Metrics. If unspecified, each metric is logged at step zero.
-- ### example
+Example
-```
+```python
from submarine import ModelsClient
metrics = {"mse": 2500.00, "rmse": 50.00}
-periscope = ModelsClient()
-with periscope.start() as run:
- periscope.log_metrics(metrics)
+modelClient = ModelsClient()
+with modelClient.start() as run:
+ modelClient.log_metrics(metrics)
```
-## ModelsClient.load_model(self, name, version)->[ mlflow.pyfunc.PyFuncModel](https://mlflow.org/docs/latest/_modules/mlflow/pyfunc.html#PyFuncModel)
+### `(Beta) ModelsClient.save_model(self, model_type, model, artifact_path, registered_model_name=None)`
-Load a model stored in models Python function format with specific name and version.
+Save model to model registry.
+### `(Beta) ModelsClient.load_model(self, name, version)->mlflow.pyfunc.PyFuncModel`
-- ### Parameters
- - **name** – Name of the containing registered model.(string).
- - **version** – Version number of the model version.(string).
+Load a model from model registry.
+### `(Beta) ModelsClient.update_model(self, name, new_name)->None`
-## ModelsClient.update_model(self, name, new_name)->None
+Update a model by new name.
-Update registered model name.
+### `(Beta) ModelsClient.delete_model(self, name, version)->None`
-- ### Parameters
- - **name** – Name of the registered model to update(string).
- - **new name** – New proposed name for the registered model(string).
-
-## ModelsClient.delete_model(self, name, version)->None
-
-Delete model version in backend.
-
-- ### Parameters
- - **name** – Name of the containing registered model.(string).
- - **version** – Version number of the model version.(string).
-
-## ModelsClient.save_model(self, model_type, model, artifact_path, registered_model_name=None)
-
-Beta: Save model to server for managment and servering.
-
-- ### Parameters
- - **name** – Name of the containing registered model.(string).
- - **version** – Version number of the model version.(string).
+Delete a model in model registry.
diff --git a/website/docs/userDocs/submarine-sdk/pysubmarine/tracking.md b/website/docs/userDocs/submarine-sdk/pysubmarine/tracking.md
deleted file mode 100644
index 7ab6e01..0000000
--- a/website/docs/userDocs/submarine-sdk/pysubmarine/tracking.md
+++ /dev/null
@@ -1,69 +0,0 @@
----
-title: PySubmarine Tracking
----
-<!---
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License. See accompanying LICENSE file.
--->
-
-It helps developers use submarine's internal data caching,
-data exchange, and task tracking capabilities to more efficiently improve the
-development and execution of machine learning productivity
-- Allow data scientist to track distributed ML experiemnt
-- Support store ML parameters and metrics in Submarine-server
-- Support hdfs, S3 and mysql (Currently we only support mysql)
-
-## Quickstart
-1. [Start mini-submarine](https://github.com/apache/submarine/tree/master/dev-support/mini-submarine#run-mini-submarine-image)
-
-2. [Start Mysql server in mini-submarine](https://github.com/apache/submarine/tree/master/dev-support/mini-submarine#run-workbench-server)
-
-3. Uncomment the log_param and log_metric in
-[mnist_distributed.py](https://github.com/apache/submarine/blob/master/dev-support/mini-submarine/submarine/mnist_distributed.py)
-
-4. Start Submarine experiment (e.g., run_submarine_mnist_tony.sh)
-
-## Functions
-### `submarine.get_tracking_uri()`
-
-return the tracking URI.
-
-### `submarine.set_tracking_uri(URI)`
-
-set the tracking URI. You can also set the
-SUBMARINE_TRACKING_URI environment variable to have Submarine find a URI from
-there. The URI should be database connection string.
-
-**Parameters**
-
-- URI - Submarine record data to Mysql server. The database URL
-is expected in the format ``<dialect>+<driver>://<username>:<password>@<host>:<port>/<database>``.
-By default it's `mysql+pymysql://submarine:password@localhost:3306/submarine`.
-More detail : [SQLAlchemy docs](https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls)
-
-### `submarine.log_metric(key, value, step=0)`
-
-logs a single key-value metric. The value must always be a number.
-
-**Parameters**
-- key - Metric name (string).
-- value - Metric value (float).
-- step - A single integer step at which to log the specified Metrics,
-by default it's 0.
-
-### `submarine.log_param(key, value)`
-
-logs a single key-value parameter. The key and value are both strings.
-
-**Parameters**
-- key - Parameter name (string).
-- value - Parameter value (string).
diff --git a/website/docs/userDocs/submarine-sdk/tracking.md b/website/docs/userDocs/submarine-sdk/tracking.md
index 887683d..afacf39 100644
--- a/website/docs/userDocs/submarine-sdk/tracking.md
+++ b/website/docs/userDocs/submarine-sdk/tracking.md
@@ -36,7 +36,6 @@
Get the tracking URI. If none has been specified, check the environmental variables. If uri is still none, return the default submarine jdbc url.
> **Returns**
-
- The tracking URI.
### `submarine.set_tracking_uri(uri: str) -> None`
@@ -44,7 +43,6 @@
set the tracking URI. You can also set the SUBMARINE_TRACKING_URI environment variable to have Submarine find a URI from there. The URI should be database connection string.
> **Parameters**
-
- **uri** \- Submarine record data to Mysql server. The database URL is expected in the format ``<dialect>+<driver>://<username>:<password>@<host>:<port>/<database>``.
By default it's `mysql+pymysql://submarine:password@localhost:3306/submarine`.
More detail : [SQLAlchemy docs](https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls)
@@ -54,7 +52,6 @@
log a single key-value parameter. The key and value are both strings.
> **Parameters**
-
- **key** - Parameter name.
- **value** - Parameter value.
@@ -63,7 +60,6 @@
log a single key-value metric. The value must always be a number.
> **Parameters**
-
- **key** - Metric name.
- **value** - Metric value.
- **step** - A single integer step at which to log the specified Metrics, by default it's 0.
