commit | 21ba037555242e5b9633fee2feb53a2d9e2e31dc | [log] [tgz] |
---|---|---|
author | cdmikechen <cdmikechen@apache.org> | Sat Jun 10 20:04:56 2023 +0800 |
committer | cdmikechen <cdmikechen@apache.org> | Wed Jun 21 20:05:40 2023 +0800 |
tree | 7cdcfe0381c9d71b5ca36d59586f93b94d5cda15 | |
parent | 93d6ee5311d77a2e69537e7d067c624a55da1ff9 [diff] |
SUBMARINE-1383. Update submarine-sdk generated client api ### What is this PR for? With the upgrade, we have fixed fixed an issue in server openapi endpoint caused by missing dependency in the meantime. At the same time, in order to ensure the validity of the openapi-generator-cli generated codes, we continue to use the old code generation method for the time being. Also, given the range of codes involved, we haven't upgraded the openapi-generator-cli version yet. After local testing, there are significant changes with each version of 5.0 or 6.0. We'll stick with 4.3.1 for the time being until we know that major changes are needed. ### What type of PR is it? Bug Fix ### Todos * [x] - Update sdk api by `gen-sdk.sh` (keep openapi-generator-cli method with keeping python2 and python3). * [x] - Fix submarine server openapi endpoint error. * [x] - Add a test case for openapi endpoint. ### What is the Jira issue? https://issues.apache.org/jira/browse/SUBMARINE-1383 ### How should this be tested? Git CI ### Screenshots (if appropriate) ### Questions: * Do the license files need updating? No * Are there breaking changes for older versions? No * Does this need new documentation? Yes Author: cdmikechen <cdmikechen@apache.org> Signed-off-by: cdmikechen <cdmikechen@apache.org> Closes #1069 from cdmikechen/SUBMARINE-1383-keep-4.3.1 and squashes the following commits: 4f5e6720 [cdmikechen] add a test case for openapi endpoint e7a4ed41 [cdmikechen] add some comments d18fae4a [cdmikechen] use sed to replace manual a7aec2a5 [cdmikechen] remove classgraph exclusion c8899092 [cdmikechen] api re-generator
Apache Submarine (Submarine for short) is an End-to-End Machine Learning Platform to allow data scientists to create end-to-end machine learning workflows. On Submarine, data scientists can finish each stage in the ML model lifecycle, including data exploration, data pipeline creation, model training, serving, and monitoring.
Some open-source and commercial projects are trying to build an end-to-end ML platform. What's the vision of Submarine?
Theodore Levitt once said:
“People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.”
experiment
on prem or cloud via easy-to-use UI/API/SDK.experiment
and dependencies of environment
.As mentioned above, Submarine attempts to provide Data-Scientist-friendly UI to make data scientists have a good user experience. Here're some examples.
# New a submarine client of the submarine server submarine_client = submarine.ExperimentClient(host='http://localhost:8080') # The experiment's environment, could be Docker image or Conda environment based environment = EnvironmentSpec(image='apache/submarine:tf-dist-mnist-test-1.0') # Specify the experiment's name, framework it's using, namespace it will run in, # the entry point. It can also accept environment variables. etc. # For PyTorch job, the framework should be 'Pytorch'. experiment_meta = ExperimentMeta(name='mnist-dist', namespace='default', framework='Tensorflow', cmd='python /var/tf_dist_mnist/dist_mnist.py --train_steps=100') # 1 PS task of 2 cpu, 1GB ps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M', replicas=1) # 1 Worker task worker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M', replicas=1) # Wrap up the meta, environment and task specs into an experiment. # For PyTorch job, the specs would be "Master" and "Worker". experiment_spec = ExperimentSpec(meta=experiment_meta, environment=environment, spec={'Ps':ps_spec, 'Worker': worker_spec}) # Submit the experiment to submarine server experiment = submarine_client.create_experiment(experiment_spec=experiment_spec) # Get the experiment ID id = experiment['experimentId']
submarine_client.get_experiment(id)
submarine_client.wait_for_finish(id)
submarine_client.get_log(id)
submarine_client.list_experiments(status='running')
For a quick-start, see Submarine On K8s
(Available on 0.5.0, see Roadmap)
If you want to know more about Submarine's architecture, components, requirements and design doc, they can be found on Architecture-and-requirement
Detailed design documentation, implementation notes can be found at: Implementation notes
Read the Apache Submarine Community Guide
How to contribute Contributing Guide
Login Submarine slack channel: https://join.slack.com/t/asf-submarine/shared_invite
Issue Tracking: https://issues.apache.org/jira/projects/SUBMARINE
What to know more about what's coming for Submarine? Please check the roadmap out: https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap
From here, you can know the changelog and the issue tracker of different version of Apache Submarine.
Apache submarine: a unified machine learning platform made simple at EuroMLSys '22
The Apache Submarine project is licensed under the Apache 2.0 License. See the LICENSE file for details.