Google Git
Sign in
apache / submarine / 747b8cd81f04f949750db1830c78cafc120c7438
commit747b8cd81f04f949750db1830c78cafc120c7438[log] [tgz]
authorKevin Su <pingsutw@gmail.com>Tue Oct 13 14:35:56 2020 +0800
committerpingsutw <pingsutw@apache.org>Tue Oct 13 06:40:08 2020 +0000
treefc59d8deca7a099f2a5dc59323f47603b9a42c37
parent17579750e490f9a8eef8bb1cce3e2f86ca7cca41 [diff]
SUBMARINE-660. Remove submarine-workbench/workbench-web/node/node

### What is this PR for?
SUBMARINE-656 accidentally added submarine-workbench/workbench-web/node/node into the repo, thus we need to remove it.

### What type of PR is it?
[Bug Fix]

### Todos
* [ ] - Task

### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-660

### How should this be tested?
* First time? Setup Travis CI as described on https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Kevin Su <pingsutw@gmail.com>

Closes #435 from pingsutw/SUBMARINE-660 and squashes the following commits:

c27a0d8 [Kevin Su] SUBMARINE-660. Remove submarine-workbench/workbench-web/node/node
1 file changed
tree: fc59d8deca7a099f2a5dc59323f47603b9a42c37
  1. .github/
  2. bin/
  3. conf/
  4. dev-support/
  5. docs/
  6. helm-charts/
  7. licenses-binary/
  8. submarine-all/
  9. submarine-client/
  10. submarine-cloud/
  11. submarine-commons/
  12. submarine-dist/
  13. submarine-sdk/
  14. submarine-security/
  15. submarine-server/
  16. submarine-test/
  17. submarine-workbench/
  18. .asf.yaml
  19. .editorconfig
  20. .gitignore
  21. .travis.yml
  22. LICENSE
  23. LICENSE-binary
  24. NOTICE
  25. NOTICE-binary
  26. pom.xml
  27. README.md
README.md

color_logo_with_text

Build Status License HitCount PyPI version

What is Apache Submarine?

Apache Submarine (Submarine for short) is an End-to-End Machine Learning PLATFORM to allow data scientists to create end-to-end machine learning workflows. To elaborate, on Submarine, data scientists can finish each stage in the ML model lifecycle, including data exploration, data pipeline creation, model training, serving, and monitoring.

Why Submarine?

Some open-source and commercial projects are trying to build an end-to-end ML platform. What's the vision of Submarine?

Problems

  1. Many platforms lack easy-to-use user interfaces (API, SDK, and IDE, etc.)
  2. In the same company, data scientists in different teams usually spend much time on developments of existing feature sets and models.
  3. Data scientists put emphasis on domain-specific tasks (e.g. Click-Through-Rate), but they need to implement their models from scratch with SDKs provided by existing platforms.
  4. Many platforms lack a unified workbench to manage each component in the ML lifecycle.

Theodore Levitt once said:

“People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.”

Goals of Submarine

Model Training (Experiment)

  • Run/Track distributed training experiment on prem or cloud via easy-to-use UI/API/SDK.
  • Easy for data scientists to manage versions of experiment and dependencies of environment
  • Support popular machine learning frameworks, including TensorFlow, PyTorch, Horovod, and MXNet
  • Provide pre-defined template for data scientists to implement domain-specific tasks easily (e.g. using DeepFM template to build a CTR prediction model)
  • Support many compute resources (e.g. CPU and GPU, etc.)
  • Support Kubernetes and YARN
  • Pipeline is also on the backlog, we will look into pipeline for training in the future.

Notebook Service

  • Submarine aims to provide a notebook service (e.g. Jupyter notebook) which allows users to manage notebook instances running on the cluster.

Model Management (Serving/versioning/monitoring, etc.)

  • Model management for model-serving/versioning/monitoring is on the roadmap.

Easy-to-use User Interface

As mentioned above, Submarine attempts to provide Data-Scientist-friendly UI to make data scientists have a good user experience. Here're some examples.

Example: Submit a distributed Tensorflow experiment via Submarine Python SDK

Run a Tensorflow Mnist experiment


# New a submarine client of the submarine server
submarine_client = submarine.ExperimentClient(host='http://localhost:8080')

# The experiment's environment, could be Docker image or Conda environment based
environment = EnvironmentSpec(image='apache/submarine:tf-dist-mnist-test-1.0')

# Specify the experiment's name, framework it's using, namespace it will run in,
# the entry point. It can also accept environment variables. etc.
# For PyTorch job, the framework should be 'Pytorch'.
experiment_meta = ExperimentMeta(name='mnist-dist',
                                 namespace='default',
                                 framework='Tensorflow',
                                 cmd='python /var/tf_dist_mnist/dist_mnist.py --train_steps=100')
# 1 PS task of 2 cpu, 1GB
ps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',
                             replicas=1)
# 1 Worker task
worker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M',
                                 replicas=1)

# Wrap up the meta, environment and task specs into an experiment.
# For PyTorch job, the specs would be "Master" and "Worker".
experiment_spec = ExperimentSpec(meta=experiment_meta,
                                 environment=environment,
                                 spec={'Ps':ps_spec, 'Worker': worker_spec})

# Submit the experiment to submarine server
experiment = submarine_client.create_experiment(experiment_spec=experiment_spec)

# Get the experiment ID
id = experiment['experimentId']

Query a specific experiment

submarine_client.get_experiment(id)

Wait for finish

submarine_client.wait_for_finish(id)

Get the experiment's log

submarine_client.get_log(id)

Get all running experiment

submarine_client.list_experiments(status='running')

For a quick-start, see Submarine On K8s

Example: Submit a pre-defined experiment template job

Example: Submit an experiment via Submarine UI

(Available on 0.6.0, see Roadmap)

Architecture, Design and requirements

If you want to know more about Submarine's architecture, components, requirements and design doc, they can be found on Architecture-and-requirement

Detailed design documentation, implementation notes can be found at: Implementation notes

Apache Submarine Community

Read the Apache Submarine Community Guide

How to contribute Contributing Guide

Issue Tracking: https://issues.apache.org/jira/projects/SUBMARINE

User Document

See User Guide Home Page

Developer Document

See Developer Guide Home Page

Roadmap

What to know more about what's coming for Submarine? Please check the roadmap out: https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap

License

The Apache Submarine project is licensed under the Apache 2.0 License. See the LICENSE file for details.