tag | b414e3ccd07780aa720bc4599d87bdc0e46be33b | |
---|---|---|
tagger | Kevin <pingsutw@apache.org> | Wed Dec 16 19:48:39 2020 -0800 |
object | 3041ef26ad04c0bddd2257a28694aa4e2b4cc837 |
Submarine 0.5.0 release
commit | 3041ef26ad04c0bddd2257a28694aa4e2b4cc837 | [log] [tgz] |
---|---|---|
author | atovk <neatovk@gmail.com> | Tue Dec 01 18:05:44 2020 +0800 |
committer | Liu Xun <liuxun@apache.org> | Tue Dec 08 05:13:56 2020 +0000 |
tree | db9c04cfa1e2665d818f32d67281f9f294cdeb68 | |
parent | 6ec50ec4b96b42338e3cefadda58129dac0919e8 [diff] |
SUBMARINE-679. Spark Security Plugin Support Spark-3.0.1 ### What is this PR for? Submarine Spark Security Plugin Support Spark-3.x ### What type of PR is it? [Improvement | Feature] ### What is the Jira issue? [SUBMARINE-679](https://issues.apache.org/jira/browse/SUBMARINE-679) ### Questions: * Does this need documentation? YES Author: atovk <neatovk@gmail.com> Closes #463 from atovk/SUBMARINE-679 and squashes the following commits: 56742a9 [atovk] fit scala-style check 90788fa [atovk] SubmarineSqlParser class Compatible for spark2.x and spark3.x b1309f9 [atovk] clean Dependency for test case in profiles hadoop-2.9 (hadoop3.x untested) 27eb9c2 [atovk] add doc available profiles spark-3.0 ba296c6 [atovk] fit test case in spark2.x and spark3.x eee31b5 [atovk] fix ranger in spark3.x subquery Permission filter e002c3d [atovk] fit spark 3.0 Dependency version 138f729 [atovk] add Test submarine spark security with spark 3.0 276af6c [atovk] SUBMARINE-679. Spark Security Plugin Support Spark-3.0.1
Apache Submarine (Submarine for short) is an End-to-End Machine Learning PLATFORM to allow data scientists to create end-to-end machine learning workflows. To elaborate, on Submarine, data scientists can finish each stage in the ML model lifecycle, including data exploration, data pipeline creation, model training, serving, and monitoring.
Some open-source and commercial projects are trying to build an end-to-end ML platform. What's the vision of Submarine?
Theodore Levitt once said:
“People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.”
experiment
on prem or cloud via easy-to-use UI/API/SDK.experiment
and dependencies of environment
As mentioned above, Submarine attempts to provide Data-Scientist-friendly UI to make data scientists have a good user experience. Here're some examples.
# New a submarine client of the submarine server submarine_client = submarine.ExperimentClient(host='http://localhost:8080') # The experiment's environment, could be Docker image or Conda environment based environment = EnvironmentSpec(image='apache/submarine:tf-dist-mnist-test-1.0') # Specify the experiment's name, framework it's using, namespace it will run in, # the entry point. It can also accept environment variables. etc. # For PyTorch job, the framework should be 'Pytorch'. experiment_meta = ExperimentMeta(name='mnist-dist', namespace='default', framework='Tensorflow', cmd='python /var/tf_dist_mnist/dist_mnist.py --train_steps=100') # 1 PS task of 2 cpu, 1GB ps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M', replicas=1) # 1 Worker task worker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M', replicas=1) # Wrap up the meta, environment and task specs into an experiment. # For PyTorch job, the specs would be "Master" and "Worker". experiment_spec = ExperimentSpec(meta=experiment_meta, environment=environment, spec={'Ps':ps_spec, 'Worker': worker_spec}) # Submit the experiment to submarine server experiment = submarine_client.create_experiment(experiment_spec=experiment_spec) # Get the experiment ID id = experiment['experimentId']
submarine_client.get_experiment(id)
submarine_client.wait_for_finish(id)
submarine_client.get_log(id)
submarine_client.list_experiments(status='running')
For a quick-start, see Submarine On K8s
(Available on 0.6.0, see Roadmap)
If you want to know more about Submarine's architecture, components, requirements and design doc, they can be found on Architecture-and-requirement
Detailed design documentation, implementation notes can be found at: Implementation notes
Read the Apache Submarine Community Guide
How to contribute Contributing Guide
Issue Tracking: https://issues.apache.org/jira/projects/SUBMARINE
What to know more about what's coming for Submarine? Please check the roadmap out: https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap
The Apache Submarine project is licensed under the Apache 2.0 License. See the LICENSE file for details.