tag | a456dd1d12b3acaeccb8755b26a6cde1927a106b | |
---|---|---|
tagger | Wanqiang Ji <jiwq@apache.org> | Fri Jun 26 19:18:28 2020 +0800 |
object | 1ae00ff6eb328040fbf8e5ccb2ff49de1efd7714 |
Release candidate - 0.4.0-RC0
commit | 1ae00ff6eb328040fbf8e5ccb2ff49de1efd7714 | [log] [tgz] |
---|---|---|
author | Wanqiang Ji <jiwq@apache.org> | Fri Jun 26 19:12:54 2020 +0800 |
committer | Wanqiang Ji <jiwq@apache.org> | Fri Jun 26 19:12:54 2020 +0800 |
tree | e0ceb0752d2524d33590b117fa93bd18fea6c99d | |
parent | f0b3110bebfaef1326b3000a8a72f5d62d6ed11a [diff] |
Preparing for 0.4.0 release
Apache Submarine (Submarine for short) is the ONE PLATFORM
to allow Data Scientists to create end-to-end machine learning workflow. ONE PLATFORM
means it supports Data Scientists to finish their jobs on the same platform without frequently switching their toolsets. From dataset exploring data pipeline creation, model training (experiments), and push model to production (model serving and monitoring). All these steps can be completed within the ONE PLATFORM
.
There‘re already a lot of open-source and comericial projects are trying to create a end-to-end machine-learning/deep-learning platform, what’s the vision of Submarine?
Theodore Levitt once said:
“People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.”
Like mentioned above, Submarine is targeted to bring Data-Scientist-friendly user-interfaces to make their life easier. Here're some examples of Submarine user-interfaces.
# New a submarine client of the submarine server submarine_client = submarine.ExperimentClient(host='http://localhost:8080') # The experiment's environment, could be Docker image or Conda environment based environment = Environment(image='gcr.io/kubeflow-ci/tf-dist-mnist-test:1.0') # Specify the experiment's name, framework it's using, namespace it will run in, # the entry point. It can also accept environment variables. etc. # For PyTorch job, the framework should be 'Pytorch'. experiment_meta = ExperimentMeta(name='mnist-dist', namespace='default', framework='Tensorflow', cmd='python /var/tf_dist_mnist/dist_mnist.py --train_steps=100') # 1 PS task of 2 cpu, 1GB ps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M', replicas=1) # 1 Worker task worker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M', replicas=1) # Wrap up the meta, environment and task specs into an experiment. # For PyTorch job, the specs would be "Master" and "Worker". experiment_spec = ExperimentSpec(meta=experiment_meta, environment=environment, spec={'Ps':ps_spec, 'Worker': worker_spec}) # Submit the experiment to submarine server experiment = submarine_client.create_experiment(experiment_spec=experiment_spec) # Get the experiment ID id = experiment['experimentId']
submarine_client.get_experiment(id)
submarine_client.wait_for_finish(id)
submarine_client.get_log(id)
submarine_client.list_experiments(status='running')
For a quick-start, see Submarine On K8s
(Available on 0.6.0, see Roadmap)
If you want to knwow more about Submarine's architecture, components, requirements and design doc, they can be found on Architecture-and-requirement
Detailed design documentation, implementation notes can be found at: Implementation notes
Read the Apache Submarine Community Guide
How to contribute Contributing Guide
Issue Tracking: https://issues.apache.org/jira/projects/SUBMARINE
See Developper Guide Home Page
What to know more about what's coming for Submarine? Please check the roadmap out: https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap
The Apache Submarine project is licensed under the Apache 2.0 License. See the LICENSE file for details.