Google Git
Sign in
apache / submarine / c638401cbbe07a19c2b585e706fa8916acaaf2b3
commitc638401cbbe07a19c2b585e706fa8916acaaf2b3[log] [tgz]
authorKent Yao <yaooqinn@hotmail.com>Sun May 03 22:49:17 2020 +0800
committerLiu Xun <liuxun@apache.org>Mon May 04 13:38:34 2020 +0000
treed9ee47a97f63ea7162c251ac7188c0c086055071
parent4db85d849203d8c548e4bbc54dadd9eecf7cb224 [diff]
SUBMARINE-488. DCL Framework: SHOW ROLES syntax

### What is this PR for?

Part of DCL framework, this PR adds the show roles syntax support

### What type of PR is it?
Feature

### Todos
* [ ] - Task

### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-488

### How should this be tested?

add new unit tests

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No

Author: Kent Yao <yaooqinn@hotmail.com>

Closes #274 from yaooqinn/SUBMARINE-488 and squashes the following commits:

0edd58e [Kent Yao] SUBMARINE-488. DCL Framework: SHOW ROLES syntax
5 files changed
tree: d9ee47a97f63ea7162c251ac7188c0c086055071
  1. .github/
  2. bin/
  3. conf/
  4. dev-support/
  5. docs/
  6. licenses-binary/
  7. submarine-all/
  8. submarine-client/
  9. submarine-cloud/
  10. submarine-commons/
  11. submarine-dist/
  12. submarine-sdk/
  13. submarine-security/
  14. submarine-server/
  15. submarine-test/
  16. submarine-workbench/
  17. .asf.yaml
  18. .editorconfig
  19. .gitignore
  20. .travis.yml
  21. LICENSE
  22. LICENSE-binary
  23. NOTICE
  24. NOTICE-binary
  25. pom.xml
  26. README.md
README.md

color_logo_with_text

Build Status License HitCount

What is Apache Submarine?

Apache Submarine is a unified AI platform which allows engineers and data scientists to run Machine Learning and Deep Learning workload in distributed cluster.

Goals of Submarine:

  • It allows jobs easy access data/models in HDFS and other storages.
  • Can launch services to serve TensorFlow/PyTorch/MXNet models.
  • Support run distributed TensorFlow jobs with simple configs.
  • Support run user-specified Docker images.
  • Support specify GPU and other resources.
  • Support launch TensorBoard for training jobs if user specified.
  • Support customized DNS name for roles (like TensorBoard.$user.$domain:6006)

Architecture

image-20190811191220934

Components

Submarine Workbench

Submarine Workbench is a WEB system. Algorithm engineers can perform complete lifecycle management of machine learning jobs in the Workbench.

  • Projects

    Manage machine learning jobs through project.

  • Data

    Data processing, data conversion, feature engineering, etc. in the workbench.

  • Job

    Data processing, algorithm development, and model training in machine learning jobs as a job run.

  • Model

    Algorithm selection, parameter adjustment, model training, model release, model Serving.

  • Workflow

    Automate the complete life cycle of machine learning operations by scheduling workflows for data processing, model training, and model publishing.

  • Team

    Support team development, code sharing, comments, code and model version management.

Submarine Core

The submarine core is the execution engine of the system and has the following features：

  • ML Engine

    Support for multiple machine learning framework access, such as tensorflow, pytorch, mxnet.

  • Data Engine

    Docking the externally deployed Spark calculation engine for data processing.

  • SDK

    Support Python, Scala, R language for algorithm development, The SDK is provided to help developers use submarine's internal data caching, data exchange, and task tracking to more efficiently improve the development and execution of machine learning tasks.

  • Submitter

    Compatible with the underlying hybrid scheduling system of yarn and k8s for unified task scheduling and resource management, so that users are not aware.

  • Hybrid Scheduler
    • YARN
    • Kubernetes

Quick start

Run mini-submarine in one step

You can use mini-submarine for a quick experience submairne.

This is a docker image built for submarine development and quick start test.

Installation and deployment

Read the Quick Start Guide

Apache Submarine Community

Read the Apache Submarine Community Guide

How to contribute Contributing Guide

License

The Apache Submarine project is licensed under the Apache 2.0 License. See the LICENSE file for details.