| commit | 793c8a26699ca9187c6960842424028a96a04d48 | [log] [tgz] |
|---|---|---|
| author | FatalLin <fatallin@gmail.com> | Tue Jan 17 16:28:27 2023 +0800 |
| committer | Thinking <744417972@qq.com> | Thu Jan 19 20:17:38 2023 +0800 |
| tree | 1f1bebc4401a5770df3f33a00d72c3d7f82c5a6e | |
| parent | 4cd2af10499ac6dc4f82bda179d9f414a522abef [diff] |
SUBMARINE-1365. Fix failure python unit test environment ### What is this PR for? https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929 we got following error in python unit test. `[24|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:25]Collecting wheel [25|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:26] Downloading wheel-0.38.4-py3-none-any.whl (36 kB) [26|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:27]Installing collected packages: wheel, setuptools [27|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:28] Attempting uninstall: setuptools [28|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:29] Found existing installation: setuptools 65.5.0 [29|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:30] Uninstalling setuptools-65.5.0: [30|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:31] Successfully uninstalled setuptools-65.5.0 [31|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:32]Successfully installed setuptools-66.0.0 wheel-0.38.4 [32|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:33]Obtaining file:///home/runner/work/submarine/submarine/submarine-sdk/pysubmarine [33|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:34] Preparing metadata (setup.py): started [34|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:35] Preparing metadata (setup.py): finished with status 'error' [35|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:36] error: subprocess-exited-with-error [36|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:37] [37|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:38] × python setup.py egg_info did not run successfully. [38|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:39] │ exit code: 1 [39|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:40] ╰─> [27 lines of output] [40|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:41] /opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/dist.py:543: UserWarning: The version specified ('0.8.0-SNAPSHOT') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details. [41|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:42] warnings.warn( [42|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:43] running egg_info [43|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:44] Traceback (most recent call last): [44|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:45] File "<string>", line 2, in <module> [45|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:46] File "<pip-setuptools-caller>", line 34, in <module> [46|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:47] File "/home/runner/work/submarine/submarine/submarine-sdk/pysubmarine/setup.py", line 21, in <module> [47|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:48] setup( [48|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:49] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup [49|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:50] return distutils.core.setup(**attrs) [50|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:51] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup [51|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:52] return run_commands(dist) [52|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:53] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands [53|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:54] dist.run_commands() [54|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:55] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands [55|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:56] self.run_command(cmd) [56|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:57] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command [57|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:58] super().run_command(command) [58|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:59] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command [59|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:60] cmd_obj.ensure_finalized() [60|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:61] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized [61|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:62] self.finalize_options() [62|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:63] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 219, in finalize_options [63|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:64] parsed_version = parse_version(self.egg_version) [64|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:65] File "/opt/hostedtoolcache/Python/3.10.9/x64/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__ [65|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:66] raise InvalidVersion(f"Invalid version: '\{version}'") [66|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:67] pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '0.8.0-SNAPSHOT' [67|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:68] [end of output] [68|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:69] [69|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:70] note: This error originates from a subprocess, and is likely not a problem with pip. [70|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:71]error: metadata-generation-failed [71|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:72] [72|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:73]× Encountered error while generating package metadata. [73|https://github.com/apache/submarine/actions/runs/3929247568/jobs/6717831929#step:15:74]╰─> See above for output.` I found a similar issue and it looks like there's some trouble on setuptools. https://github.com/pypa/setuptools/issues/3772 I'd like to take their adoption to fix the setuptools to 65.7.0 as a workaround here. ### What type of PR is it? Hot fix ### Todos N/A ### What is the Jira issue? https://issues.apache.org/jira/browse/SUBMARINE-1365 ### How should this be tested? should run python unit test as expected. ### Screenshots (if appropriate) ### Questions: * Do the license files need updating? No * Are there breaking changes for older versions? No * Does this need new documentation? No Author: FatalLin <fatallin@gmail.com> Signed-off-by: Thinking <744417972@qq.com> Closes #1040 from FatalLin/SUBMARINE-1365 and squashes the following commits: e2fe9b5d [FatalLin] SUBMARINE-1365. Fix faulure python unit test environment
Apache Submarine (Submarine for short) is an End-to-End Machine Learning Platform to allow data scientists to create end-to-end machine learning workflows. On Submarine, data scientists can finish each stage in the ML model lifecycle, including data exploration, data pipeline creation, model training, serving, and monitoring.
Some open-source and commercial projects are trying to build an end-to-end ML platform. What's the vision of Submarine?
Theodore Levitt once said:
“People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.”
experiment on prem or cloud via easy-to-use UI/API/SDK.experiment and dependencies of environment.As mentioned above, Submarine attempts to provide Data-Scientist-friendly UI to make data scientists have a good user experience. Here're some examples.
# New a submarine client of the submarine server submarine_client = submarine.ExperimentClient(host='http://localhost:8080') # The experiment's environment, could be Docker image or Conda environment based environment = EnvironmentSpec(image='apache/submarine:tf-dist-mnist-test-1.0') # Specify the experiment's name, framework it's using, namespace it will run in, # the entry point. It can also accept environment variables. etc. # For PyTorch job, the framework should be 'Pytorch'. experiment_meta = ExperimentMeta(name='mnist-dist', namespace='default', framework='Tensorflow', cmd='python /var/tf_dist_mnist/dist_mnist.py --train_steps=100') # 1 PS task of 2 cpu, 1GB ps_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M', replicas=1) # 1 Worker task worker_spec = ExperimentTaskSpec(resources='cpu=2,memory=1024M', replicas=1) # Wrap up the meta, environment and task specs into an experiment. # For PyTorch job, the specs would be "Master" and "Worker". experiment_spec = ExperimentSpec(meta=experiment_meta, environment=environment, spec={'Ps':ps_spec, 'Worker': worker_spec}) # Submit the experiment to submarine server experiment = submarine_client.create_experiment(experiment_spec=experiment_spec) # Get the experiment ID id = experiment['experimentId']
submarine_client.get_experiment(id)
submarine_client.wait_for_finish(id)
submarine_client.get_log(id)
submarine_client.list_experiments(status='running')
For a quick-start, see Submarine On K8s
(Available on 0.5.0, see Roadmap)
If you want to know more about Submarine's architecture, components, requirements and design doc, they can be found on Architecture-and-requirement
Detailed design documentation, implementation notes can be found at: Implementation notes
Read the Apache Submarine Community Guide
How to contribute Contributing Guide
Login Submarine slack channel: https://join.slack.com/t/asf-submarine/shared_invite
Issue Tracking: https://issues.apache.org/jira/projects/SUBMARINE
What to know more about what's coming for Submarine? Please check the roadmap out: https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap
From here, you can know the changelog and the issue tracker of different version of Apache Submarine.
Apache submarine: a unified machine learning platform made simple at EuroMLSys '22
The Apache Submarine project is licensed under the Apache 2.0 License. See the LICENSE file for details.