[migrate] Add tag 3.0.1 source code (#5)
Add version 3.0.1 code from https://github.com/apache/dolphinscheduler/tree/3.0.1/dolphinscheduler-python/pydolphinscheduler
diff --git a/DEVELOP.md b/DEVELOP.md
index bdd0416..9972094 100644
--- a/DEVELOP.md
+++ b/DEVELOP.md
@@ -34,7 +34,7 @@
```shell
cd dolphinscheduler/dolphinscheduler-python/pydolphinscheduler
-python -m pip install .[dev]
+python -m pip install -e '.[dev]'
```
Next, we have to open pydolphinscheduler project in you editor. We recommend you use [pycharm][pycharm]
@@ -146,10 +146,24 @@
#### Testing
-## Build Docs
+## Build Document
We use [sphinx][sphinx] to build docs. Dolphinscheduler Python API CI would automatically build docs when you submit pull request in
-GitHub. You may locally ensure docs could be built suceessfully in case the failure blocks CI.
+GitHub. You may locally ensure docs could be built successfully in case the failure blocks CI, you can build by tox or manual.
+
+### Build Document Automatically with tox
+
+We integrated document build process into tox, you can build the latest document and all document(including history documents) via
+single command
+
+```shell
+# Build the latest document in dev branch
+tox -e doc-build
+# Build all documents, which including the latest and all history documents
+tox -e doc-build-multi
+```
+
+### Build Document Manually
To build docs locally, install sphinx and related python modules first via:
@@ -157,13 +171,16 @@
python -m pip install '.[doc]'
```
-Then
+Then go to document directory and execute the build command
```shell
cd pydolphinscheduler/docs/
make clean && make html
```
+> NOTE: We support build multiple versions of documents with [sphinx-multiversion](https://holzhaus.github.io/sphinx-multiversion/master/index.html),
+> you can build with command `git fetch --tags && make clean && make multiversion`
+
## Testing
pydolphinscheduler using [pytest][pytest] to test our codebase. GitHub Action will run our test when you create
diff --git a/README.md b/README.md
index 316d604..71119bd 100644
--- a/README.md
+++ b/README.md
@@ -45,12 +45,15 @@
# 0.1.0
```
+> NOTE: package apache-dolphinscheduler not work on above Python version 3.10(including itself) in Window operating system
+> due to dependence [py4j](https://pypi.org/project/py4j/) not work on those environments.
+
Here we show you how to install and run a simple example of pydolphinscheduler
### Start Server And Run Example
Before you run an example, you have to start backend server. You could follow
-[development setup](https://dolphinscheduler.apache.org/en-us/development/development-environment-setup.html)
+[development setup](../../docs/docs/en/contribute/development-environment-setup.md)
section "DolphinScheduler Standalone Quick Start" to set up developer environment. You have to start backend
and frontend server in this step, which mean that you could view DolphinScheduler UI in your browser with URL
http://localhost:12345/dolphinscheduler
@@ -84,4 +87,4 @@
## What's more
-For more detail information, please go to see **PyDolphinScheduler** [document](https://dolphinscheduler.apache.org/python/index.html)
+For more detail information, please go to see **PyDolphinScheduler** latest(unreleased) [document](https://dolphinscheduler.apache.org/python/dev/index.html)
diff --git a/RELEASE.md b/RELEASE.md
index 6c2b46e..c90107a 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -25,7 +25,7 @@
## To ASF Distribution Directory
You could release to [ASF Distribution Directory](https://downloads.apache.org/dolphinscheduler/) according to
-[release guide](https://dolphinscheduler.apache.org/en-us/community/release-prepare.html) in DolphinScheduler
+[release guide](../../docs/docs/en/contribute/release/release-prepare.md) in DolphinScheduler
website.
## To PyPi
diff --git a/docs/Makefile b/docs/Makefile
index 985198a..ff2c4eb 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -24,6 +24,7 @@
# Add opts `turn warnings into errors` strict sphinx-build behavior
SPHINXOPTS ?= -W
SPHINXBUILD ?= sphinx-build
+SPHINXMULTIVERSION ?= sphinx-multiversion
SOURCEDIR = source
BUILDDIR = build
@@ -37,3 +38,7 @@
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+# Create multiple version of docs
+multiversion:
+ @$(SPHINXMULTIVERSION) "$(SOURCEDIR)" "$(BUILDDIR)/html"
diff --git a/docs/source/_templates/versioning.html b/docs/source/_templates/versioning.html
new file mode 100644
index 0000000..47136c4
--- /dev/null
+++ b/docs/source/_templates/versioning.html
@@ -0,0 +1,27 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% if versions %}
+<h3>{{ _('Versions') }}</h3>
+<ul>
+ {%- for item in versions %}
+ <li><a href="{{ item.url }}">{{ item.name }}</a></li>
+ {%- endfor %}
+</ul>
+{% endif %}
diff --git a/docs/source/_templates/versions.html b/docs/source/_templates/versions.html
new file mode 100644
index 0000000..51b7271
--- /dev/null
+++ b/docs/source/_templates/versions.html
@@ -0,0 +1,46 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{%- if current_version %}
+<div class="rst-versions" data-toggle="rst-versions" role="note" aria-label="versions">
+ <span class="rst-current-version" data-toggle="rst-current-version">
+ <span class="fa fa-book"> Other Versions</span>
+ v: {{ current_version.name }}
+ <span class="fa fa-caret-down"></span>
+ </span>
+ <div class="rst-other-versions">
+ {%- if versions.tags %}
+ <dl>
+ <dt>Tags</dt>
+ {%- for item in versions.tags %}
+ <dd><a href="{{ item.url }}">{{ item.name }}</a></dd>
+ {%- endfor %}
+ </dl>
+ {%- endif %}
+ {%- if versions.branches %}
+ <dl>
+ <dt>Branches</dt>
+ {%- for item in versions.branches %}
+ <dd><a href="{{ item.url }}">{{ item.name }}</a></dd>
+ {%- endfor %}
+ </dl>
+ {%- endif %}
+ </div>
+</div>
+{%- endif %}
diff --git a/docs/source/api.rst b/docs/source/api.rst
index 8e55ea5..b170b6f 100644
--- a/docs/source/api.rst
+++ b/docs/source/api.rst
@@ -24,10 +24,10 @@
.. automodule:: pydolphinscheduler.core
:inherited-members:
-Sides
------
+Models
+------
-.. automodule:: pydolphinscheduler.side
+.. automodule:: pydolphinscheduler.models
:inherited-members:
Tasks
diff --git a/docs/source/conf.py b/docs/source/conf.py
index b162e0c..23fc117 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -26,17 +26,31 @@
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+
+import os
+import sys
+from pathlib import Path
+
+# For sphinx-multiversion, we need to build API docs of the corresponding package version, related issue:
+# https://github.com/Holzhaus/sphinx-multiversion/issues/42
+pkg_src_dir = (
+ Path(os.environ.get("SPHINX_MULTIVERSION_SOURCEDIR", default="."))
+ .joinpath("../../src")
+ .resolve()
+)
+sys.path.insert(0, str(pkg_src_dir))
+# Debug to uncomment this to see the source path
+# print("=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=")
+# print(pkg_src_dir)
+# [print(p) for p in sys.path]
+# print("=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=")
# -- Project information -----------------------------------------------------
project = "pydolphinscheduler"
copyright = "2022, apache"
-author = "apache"
+author = "apache dolphinscheduler contributors"
# The full version, including alpha/beta/rc tags
release = "0.0.1"
@@ -60,11 +74,25 @@
# Add inline tabbed content
"sphinx_inline_tabs",
"sphinx_copybutton",
+ "sphinx_multiversion",
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
+# sphinx_multiversion configuration
+html_sidebars = {
+ "**": [
+ "versioning.html",
+ ],
+}
+# Match all exists tag for pydolphinscheduler expect version 2.0.4(not release apache dolphinscheduler)
+smv_tag_whitelist = r"^(?!2.0.4)\d+\.\d+\.\d+$"
+smv_branch_whitelist = "dev"
+smv_remote_whitelist = r"^(origin|upstream)$"
+smv_released_pattern = "^refs/tags/.*$"
+smv_outputdir_format = "versions/{ref.name}"
+
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
diff --git a/docs/source/config.rst b/docs/source/config.rst
index feb147a..29a143d 100644
--- a/docs/source/config.rst
+++ b/docs/source/config.rst
@@ -200,7 +200,7 @@
Here are all our configurations for pydolphinscheduler.
-.. literalinclude:: ../../src/pydolphinscheduler/core/default_config.yaml
+.. literalinclude:: ../../src/pydolphinscheduler/default_config.yaml
:language: yaml
:lines: 18-
diff --git a/setup.py b/setup.py
index b1e3742..5d1903c 100644
--- a/setup.py
+++ b/setup.py
@@ -32,7 +32,7 @@
logger = logging.getLogger(__name__)
-version = "3.0.0"
+version = "3.0.1"
# Start package required
prod = [
@@ -53,6 +53,10 @@
"sphinx-click>=3.0",
"sphinx-inline-tabs",
"sphinx-copybutton>=0.4.0",
+ # Unreleased package have a feature we want(use correct version package for API ref), so we install from
+ # GitHub directly, see also:
+ # https://github.com/Holzhaus/sphinx-multiversion/issues/42#issuecomment-1210539786
+ "sphinx-multiversion @ git+https://github.com/Holzhaus/sphinx-multiversion#egg=sphinx-multiversion",
]
test = [
@@ -135,7 +139,7 @@
],
project_urls={
"Homepage": "https://dolphinscheduler.apache.org",
- "Documentation": "https://dolphinscheduler.apache.org/python/index.html",
+ "Documentation": "https://dolphinscheduler.apache.org/python/dev/index.html",
"Source": "https://github.com/apache/dolphinscheduler/tree/dev/dolphinscheduler-python/"
"pydolphinscheduler",
"Issue Tracker": "https://github.com/apache/dolphinscheduler/issues?"
@@ -147,12 +151,12 @@
package_dir={"": "src"},
include_package_data=True,
package_data={
- "pydolphinscheduler": ["core/default_config.yaml"],
+ "pydolphinscheduler": ["default_config.yaml"],
},
platforms=["any"],
classifiers=[
# complete classifier list: http://pypi.python.org/pypi?%3Aaction=list_classifiers
- "Development Status :: 3 - Alpha",
+ "Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
@@ -165,6 +169,8 @@
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
"Topic :: Software Development :: User Interfaces",
diff --git a/src/pydolphinscheduler/cli/commands.py b/src/pydolphinscheduler/cli/commands.py
index e2ca86b..d78e503 100644
--- a/src/pydolphinscheduler/cli/commands.py
+++ b/src/pydolphinscheduler/cli/commands.py
@@ -20,8 +20,8 @@
import click
from click import echo
-from pydolphinscheduler import __version__
-from pydolphinscheduler.core.configuration import (
+import pydolphinscheduler
+from pydolphinscheduler.configuration import (
get_single_config,
init_config_file,
set_single_config,
@@ -48,9 +48,9 @@
"""Show current version of pydolphinscheduler."""
if part:
idx = version_option_val.index(part)
- echo(f"{__version__.split('.')[idx]}")
+ echo(f"{pydolphinscheduler.__version__.split('.')[idx]}")
else:
- echo(f"{__version__}")
+ echo(f"{pydolphinscheduler.__version__}")
@cli.command()
diff --git a/src/pydolphinscheduler/configuration.py b/src/pydolphinscheduler/configuration.py
new file mode 100644
index 0000000..860f986
--- /dev/null
+++ b/src/pydolphinscheduler/configuration.py
@@ -0,0 +1,193 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Configuration module for pydolphinscheduler."""
+import os
+from pathlib import Path
+from typing import Any
+
+from pydolphinscheduler.exceptions import PyDSConfException
+from pydolphinscheduler.utils import file
+from pydolphinscheduler.utils.yaml_parser import YamlParser
+
+BUILD_IN_CONFIG_PATH = Path(__file__).resolve().parent.joinpath("default_config.yaml")
+
+
+def config_path() -> Path:
+ """Get the path of pydolphinscheduler configuration file."""
+ pyds_home = os.environ.get("PYDS_HOME", "~/pydolphinscheduler")
+ config_file_path = Path(pyds_home).joinpath("config.yaml").expanduser()
+ return config_file_path
+
+
+def get_configs() -> YamlParser:
+ """Get all configuration settings from configuration file.
+
+ Will use custom configuration file first if it exists, otherwise default configuration file in
+ default path.
+ """
+ path = str(config_path()) if config_path().exists() else BUILD_IN_CONFIG_PATH
+ with open(path, mode="r") as f:
+ return YamlParser(f.read())
+
+
+def init_config_file() -> None:
+ """Initialize configuration file by default configs."""
+ if config_path().exists():
+ raise PyDSConfException(
+ "Initialize configuration false to avoid overwrite configure by accident, file already exists "
+ "in %s, if you wan to overwrite the exists configure please remove the exists file manually.",
+ str(config_path()),
+ )
+ file.write(content=str(get_configs()), to_path=str(config_path()))
+
+
+def get_single_config(key: str) -> Any:
+ """Get single config to configuration file.
+
+ Support get from nested keys by delimiter ``.``.
+
+ For example, yaml config as below:
+
+ .. code-block:: yaml
+
+ one:
+ two1:
+ three: value1
+ two2: value2
+
+ you could get ``value1`` and ``value2`` by nested path
+
+ .. code-block:: python
+
+ value1 = get_single_config("one.two1.three")
+ value2 = get_single_config("one.two2")
+
+ :param key: The config key want to get it value.
+ """
+ config = get_configs()
+ if key not in config:
+ raise PyDSConfException(
+ "Configuration path %s do not exists. Can not get configuration.", key
+ )
+ return config[key]
+
+
+def set_single_config(key: str, value: Any) -> None:
+ """Change single config to configuration file.
+
+ For example, yaml config as below:
+
+ .. code-block:: yaml
+
+ one:
+ two1:
+ three: value1
+ two2: value2
+
+ you could change ``value1`` to ``value3``, also change ``value2`` to ``value4`` by nested path assigned
+
+ .. code-block:: python
+
+ set_single_config["one.two1.three"] = "value3"
+ set_single_config["one.two2"] = "value4"
+
+ :param key: The config key want change.
+ :param value: The new value want to set.
+ """
+ config = get_configs()
+ if key not in config:
+ raise PyDSConfException(
+ "Configuration path %s do not exists. Can not set configuration.", key
+ )
+ config[key] = value
+ file.write(content=str(config), to_path=str(config_path()), overwrite=True)
+
+
+def get_int(val: Any) -> int:
+ """Covert value to int."""
+ return int(val)
+
+
+def get_bool(val: Any) -> bool:
+ """Covert value to boolean."""
+ if isinstance(val, str):
+ return val.lower() in {"true", "t"}
+ elif isinstance(val, int):
+ return val == 1
+ else:
+ return bool(val)
+
+
+# Start Common Configuration Settings
+
+# Add configs as module variables to avoid read configuration multiple times when
+# Get common configuration setting
+# Set or get multiple configs in single time
+configs: YamlParser = get_configs()
+
+# Java Gateway Settings
+JAVA_GATEWAY_ADDRESS = os.environ.get(
+ "PYDS_JAVA_GATEWAY_ADDRESS", configs.get("java_gateway.address")
+)
+JAVA_GATEWAY_PORT = get_int(
+ os.environ.get("PYDS_JAVA_GATEWAY_PORT", configs.get("java_gateway.port"))
+)
+JAVA_GATEWAY_AUTO_CONVERT = get_bool(
+ os.environ.get(
+ "PYDS_JAVA_GATEWAY_AUTO_CONVERT", configs.get("java_gateway.auto_convert")
+ )
+)
+
+# User Settings
+USER_NAME = os.environ.get("PYDS_USER_NAME", configs.get("default.user.name"))
+USER_PASSWORD = os.environ.get(
+ "PYDS_USER_PASSWORD", configs.get("default.user.password")
+)
+USER_EMAIL = os.environ.get("PYDS_USER_EMAIL", configs.get("default.user.email"))
+USER_PHONE = str(os.environ.get("PYDS_USER_PHONE", configs.get("default.user.phone")))
+USER_STATE = get_int(
+ os.environ.get("PYDS_USER_STATE", configs.get("default.user.state"))
+)
+
+# Workflow Settings
+WORKFLOW_PROJECT = os.environ.get(
+ "PYDS_WORKFLOW_PROJECT", configs.get("default.workflow.project")
+)
+WORKFLOW_TENANT = os.environ.get(
+ "PYDS_WORKFLOW_TENANT", configs.get("default.workflow.tenant")
+)
+WORKFLOW_USER = os.environ.get(
+ "PYDS_WORKFLOW_USER", configs.get("default.workflow.user")
+)
+WORKFLOW_QUEUE = os.environ.get(
+ "PYDS_WORKFLOW_QUEUE", configs.get("default.workflow.queue")
+)
+WORKFLOW_RELEASE_STATE = os.environ.get(
+ "PYDS_WORKFLOW_RELEASE_STATE", configs.get("default.workflow.release_state")
+)
+WORKFLOW_WORKER_GROUP = os.environ.get(
+ "PYDS_WORKFLOW_WORKER_GROUP", configs.get("default.workflow.worker_group")
+)
+WORKFLOW_TIME_ZONE = os.environ.get(
+ "PYDS_WORKFLOW_TIME_ZONE", configs.get("default.workflow.time_zone")
+)
+WORKFLOW_WARNING_TYPE = os.environ.get(
+ "PYDS_WORKFLOW_WARNING_TYPE", configs.get("default.workflow.warning_type")
+)
+
+# End Common Configuration Setting
diff --git a/src/pydolphinscheduler/core/__init__.py b/src/pydolphinscheduler/core/__init__.py
index 7497d1f..b997c3e 100644
--- a/src/pydolphinscheduler/core/__init__.py
+++ b/src/pydolphinscheduler/core/__init__.py
@@ -23,8 +23,8 @@
from pydolphinscheduler.core.task import Task
__all__ = [
+ "Database",
"Engine",
"ProcessDefinition",
"Task",
- "Database",
]
diff --git a/src/pydolphinscheduler/core/process_definition.py b/src/pydolphinscheduler/core/process_definition.py
index 63e0808..69dcbc1 100644
--- a/src/pydolphinscheduler/core/process_definition.py
+++ b/src/pydolphinscheduler/core/process_definition.py
@@ -21,12 +21,11 @@
from datetime import datetime
from typing import Any, Dict, List, Optional, Set
+from pydolphinscheduler import configuration
from pydolphinscheduler.constants import TaskType
-from pydolphinscheduler.core import configuration
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.exceptions import PyDSParamException, PyDSTaskNoFoundException
from pydolphinscheduler.java_gateway import launch_gateway
-from pydolphinscheduler.side import Project, Tenant, User
+from pydolphinscheduler.models import Base, Project, Tenant, User
from pydolphinscheduler.utils.date import MAX_DATETIME, conv_from_str, conv_to_schedule
@@ -170,7 +169,7 @@
def user(self) -> User:
"""Get user object.
- For now we just get from python side but not from java gateway side, so it may not correct.
+ For now we just get from python models but not from java gateway models, so it may not correct.
"""
return User(name=self._user, tenant=self._tenant)
@@ -358,10 +357,10 @@
self.start()
def _ensure_side_model_exists(self):
- """Ensure process definition side model exists.
+ """Ensure process definition models model exists.
- For now, side object including :class:`pydolphinscheduler.side.project.Project`,
- :class:`pydolphinscheduler.side.tenant.Tenant`, :class:`pydolphinscheduler.side.user.User`.
+ For now, models object including :class:`pydolphinscheduler.models.project.Project`,
+ :class:`pydolphinscheduler.models.tenant.Tenant`, :class:`pydolphinscheduler.models.user.User`.
If these model not exists, would create default value in
:class:`pydolphinscheduler.constants.ProcessDefinitionDefault`.
"""
diff --git a/src/pydolphinscheduler/core/resource.py b/src/pydolphinscheduler/core/resource.py
index bd4ffd4..a3aab81 100644
--- a/src/pydolphinscheduler/core/resource.py
+++ b/src/pydolphinscheduler/core/resource.py
@@ -19,7 +19,7 @@
from typing import Optional
-from pydolphinscheduler.core.base import Base
+from pydolphinscheduler.models import Base
class Resource(Base):
diff --git a/src/pydolphinscheduler/core/task.py b/src/pydolphinscheduler/core/task.py
index 90c0e89..4d4e67e 100644
--- a/src/pydolphinscheduler/core/task.py
+++ b/src/pydolphinscheduler/core/task.py
@@ -20,6 +20,7 @@
from logging import getLogger
from typing import Dict, List, Optional, Sequence, Set, Tuple, Union
+from pydolphinscheduler import configuration
from pydolphinscheduler.constants import (
Delimiter,
ResourceKey,
@@ -27,13 +28,12 @@
TaskPriority,
TaskTimeoutFlag,
)
-from pydolphinscheduler.core import configuration
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.core.process_definition import (
ProcessDefinition,
ProcessDefinitionContext,
)
from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models import Base
logger = getLogger(__name__)
diff --git a/src/pydolphinscheduler/default_config.yaml b/src/pydolphinscheduler/default_config.yaml
new file mode 100644
index 0000000..98d7b99
--- /dev/null
+++ b/src/pydolphinscheduler/default_config.yaml
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Setting about Java gateway server
+java_gateway:
+ # The address of Python gateway server start. Set its value to `0.0.0.0` if your Python API run in different
+ # between Python gateway server. It could be be specific to other address like `127.0.0.1` or `localhost`
+ address: 127.0.0.1
+
+ # The port of Python gateway server start. Define which port you could connect to Python gateway server from
+ # Python API models.
+ port: 25333
+
+ # Whether automatically convert Python objects to Java Objects. Default value is ``True``. There is some
+ # performance lost when set to ``True`` but for now pydolphinscheduler do not handle the convert issue between
+ # java and Python, mark it as TODO item in the future.
+ auto_convert: true
+
+# Setting about dolphinscheduler default value, will use the value set below if property do not set, which
+# including ``user``, ``workflow``
+default:
+ # Default value for dolphinscheduler's user object
+ user:
+ name: userPythonGateway
+ password: userPythonGateway
+ email: userPythonGateway@dolphinscheduler.com
+ tenant: tenant_pydolphin
+ phone: 11111111111
+ state: 1
+ # Default value for dolphinscheduler's workflow object
+ workflow:
+ project: project-pydolphin
+ tenant: tenant_pydolphin
+ user: userPythonGateway
+ queue: queuePythonGateway
+ worker_group: default
+ # Release state of workflow, default value is ``online`` which mean setting workflow online when it submits
+ # to Java gateway, if you want to set workflow offline set its value to ``offline``
+ release_state: online
+ time_zone: Asia/Shanghai
+ # Warning type of the workflow, default value is ``NONE`` mean do not warn user in any cases of workflow state,
+ # change to ``FAILURE`` if you want to warn users when workflow failed. All available enum value are
+ # ``NONE``, ``SUCCESS``, ``FAILURE``, ``ALL``
+ warning_type: NONE
diff --git a/src/pydolphinscheduler/examples/task_dependent_example.py b/src/pydolphinscheduler/examples/task_dependent_example.py
index 88d6ea2..db53bcc 100644
--- a/src/pydolphinscheduler/examples/task_dependent_example.py
+++ b/src/pydolphinscheduler/examples/task_dependent_example.py
@@ -35,7 +35,7 @@
task_dependent(this task dependent on task_dependent_external.task_1 and task_dependent_external.task_2).
"""
-from pydolphinscheduler.core import configuration
+from pydolphinscheduler import configuration
from pydolphinscheduler.core.process_definition import ProcessDefinition
from pydolphinscheduler.tasks.dependent import And, Dependent, DependentItem, Or
from pydolphinscheduler.tasks.shell import Shell
diff --git a/src/pydolphinscheduler/java_gateway.py b/src/pydolphinscheduler/java_gateway.py
index 8560638..7b85902 100644
--- a/src/pydolphinscheduler/java_gateway.py
+++ b/src/pydolphinscheduler/java_gateway.py
@@ -22,8 +22,8 @@
from py4j.java_collections import JavaMap
from py4j.java_gateway import GatewayParameters, JavaGateway
+from pydolphinscheduler import configuration
from pydolphinscheduler.constants import JavaGatewayDefault
-from pydolphinscheduler.core import configuration
from pydolphinscheduler.exceptions import PyDSJavaGatewayException
diff --git a/src/pydolphinscheduler/models/__init__.py b/src/pydolphinscheduler/models/__init__.py
new file mode 100644
index 0000000..b289954
--- /dev/null
+++ b/src/pydolphinscheduler/models/__init__.py
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Init Models package, keeping object related to DolphinScheduler covert from Java Gateway Service."""
+
+from pydolphinscheduler.models.base import Base
+from pydolphinscheduler.models.base_side import BaseSide
+from pydolphinscheduler.models.project import Project
+from pydolphinscheduler.models.queue import Queue
+from pydolphinscheduler.models.tenant import Tenant
+from pydolphinscheduler.models.user import User
+from pydolphinscheduler.models.worker_group import WorkerGroup
+
+__all__ = [
+ "Base",
+ "BaseSide",
+ "Project",
+ "Tenant",
+ "User",
+ "Queue",
+ "WorkerGroup",
+]
diff --git a/src/pydolphinscheduler/models/base.py b/src/pydolphinscheduler/models/base.py
new file mode 100644
index 0000000..2647714
--- /dev/null
+++ b/src/pydolphinscheduler/models/base.py
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DolphinScheduler Base object."""
+
+from typing import Dict, Optional
+
+# from pydolphinscheduler.models.user import User
+from pydolphinscheduler.utils.string import attr2camel
+
+
+class Base:
+ """DolphinScheduler Base object."""
+
+ # Object key attribute, to test whether object equals and so on.
+ _KEY_ATTR: set = {"name", "description"}
+
+ # Object defines attribute, use when needs to communicate with Java gateway server.
+ _DEFINE_ATTR: set = set()
+
+ # Object default attribute, will add those attribute to `_DEFINE_ATTR` when init assign missing.
+ _DEFAULT_ATTR: Dict = {}
+
+ def __init__(self, name: str, description: Optional[str] = None):
+ self.name = name
+ self.description = description
+
+ def __repr__(self) -> str:
+ return f'<{type(self).__name__}: name="{self.name}">'
+
+ def __eq__(self, other):
+ return type(self) == type(other) and all(
+ getattr(self, a, None) == getattr(other, a, None) for a in self._KEY_ATTR
+ )
+
+ def get_define_custom(
+ self, camel_attr: bool = True, custom_attr: set = None
+ ) -> Dict:
+ """Get object definition attribute by given attr set."""
+ content = {}
+ for attr in custom_attr:
+ val = getattr(self, attr, None)
+ if camel_attr:
+ content[attr2camel(attr)] = val
+ else:
+ content[attr] = val
+ return content
+
+ def get_define(self, camel_attr: bool = True) -> Dict:
+ """Get object definition attribute communicate to Java gateway server.
+
+ use attribute `self._DEFINE_ATTR` to determine which attributes should including when
+ object tries to communicate with Java gateway server.
+ """
+ content = self.get_define_custom(camel_attr, self._DEFINE_ATTR)
+ update_default = {
+ k: self._DEFAULT_ATTR.get(k) for k in self._DEFAULT_ATTR if k not in content
+ }
+ content.update(update_default)
+ return content
diff --git a/src/pydolphinscheduler/models/base_side.py b/src/pydolphinscheduler/models/base_side.py
new file mode 100644
index 0000000..67ac88d
--- /dev/null
+++ b/src/pydolphinscheduler/models/base_side.py
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Module for models object."""
+
+from typing import Optional
+
+from pydolphinscheduler import configuration
+from pydolphinscheduler.models import Base
+
+
+class BaseSide(Base):
+ """Base class for models object, it declare base behavior for them."""
+
+ def __init__(self, name: str, description: Optional[str] = None):
+ super().__init__(name, description)
+
+ @classmethod
+ def create_if_not_exists(
+ cls,
+ # TODO comment for avoiding cycle import
+ # user: Optional[User] = ProcessDefinitionDefault.USER
+ user=configuration.WORKFLOW_USER,
+ ):
+ """Create Base if not exists."""
+ raise NotImplementedError
diff --git a/src/pydolphinscheduler/models/project.py b/src/pydolphinscheduler/models/project.py
new file mode 100644
index 0000000..ad72211
--- /dev/null
+++ b/src/pydolphinscheduler/models/project.py
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DolphinScheduler Project object."""
+
+from typing import Optional
+
+from pydolphinscheduler import configuration
+from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models import BaseSide
+
+
+class Project(BaseSide):
+ """DolphinScheduler Project object."""
+
+ def __init__(
+ self,
+ name: str = configuration.WORKFLOW_PROJECT,
+ description: Optional[str] = None,
+ ):
+ super().__init__(name, description)
+
+ def create_if_not_exists(self, user=configuration.USER_NAME) -> None:
+ """Create Project if not exists."""
+ gateway = launch_gateway()
+ gateway.entry_point.createOrGrantProject(user, self.name, self.description)
+ # TODO recover result checker
+ # gateway_result_checker(result, None)
diff --git a/src/pydolphinscheduler/models/queue.py b/src/pydolphinscheduler/models/queue.py
new file mode 100644
index 0000000..3f8f81d
--- /dev/null
+++ b/src/pydolphinscheduler/models/queue.py
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DolphinScheduler User object."""
+
+from typing import Optional
+
+from pydolphinscheduler import configuration
+from pydolphinscheduler.java_gateway import gateway_result_checker, launch_gateway
+from pydolphinscheduler.models import BaseSide
+
+
+class Queue(BaseSide):
+ """DolphinScheduler Queue object."""
+
+ def __init__(
+ self,
+ name: str = configuration.WORKFLOW_QUEUE,
+ description: Optional[str] = "",
+ ):
+ super().__init__(name, description)
+
+ def create_if_not_exists(self, user=configuration.USER_NAME) -> None:
+ """Create Queue if not exists."""
+ gateway = launch_gateway()
+ # Here we set Queue.name and Queue.queueName same as self.name
+ result = gateway.entry_point.createProject(user, self.name, self.name)
+ gateway_result_checker(result, None)
diff --git a/src/pydolphinscheduler/models/tenant.py b/src/pydolphinscheduler/models/tenant.py
new file mode 100644
index 0000000..148a8f6
--- /dev/null
+++ b/src/pydolphinscheduler/models/tenant.py
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DolphinScheduler Tenant object."""
+
+from typing import Optional
+
+from pydolphinscheduler import configuration
+from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models import BaseSide
+
+
+class Tenant(BaseSide):
+ """DolphinScheduler Tenant object."""
+
+ def __init__(
+ self,
+ name: str = configuration.WORKFLOW_TENANT,
+ queue: str = configuration.WORKFLOW_QUEUE,
+ description: Optional[str] = None,
+ ):
+ super().__init__(name, description)
+ self.queue = queue
+
+ def create_if_not_exists(
+ self, queue_name: str, user=configuration.USER_NAME
+ ) -> None:
+ """Create Tenant if not exists."""
+ gateway = launch_gateway()
+ gateway.entry_point.createTenant(self.name, self.description, queue_name)
+ # gateway_result_checker(result, None)
diff --git a/src/pydolphinscheduler/models/user.py b/src/pydolphinscheduler/models/user.py
new file mode 100644
index 0000000..de2d8b1
--- /dev/null
+++ b/src/pydolphinscheduler/models/user.py
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DolphinScheduler User object."""
+
+from typing import Optional
+
+from pydolphinscheduler import configuration
+from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models import BaseSide, Tenant
+
+
+class User(BaseSide):
+ """DolphinScheduler User object."""
+
+ _KEY_ATTR = {
+ "name",
+ "password",
+ "email",
+ "phone",
+ "tenant",
+ "queue",
+ "status",
+ }
+
+ def __init__(
+ self,
+ name: str,
+ password: Optional[str] = configuration.USER_PASSWORD,
+ email: Optional[str] = configuration.USER_EMAIL,
+ phone: Optional[str] = configuration.USER_PHONE,
+ tenant: Optional[str] = configuration.WORKFLOW_TENANT,
+ queue: Optional[str] = configuration.WORKFLOW_QUEUE,
+ status: Optional[int] = configuration.USER_STATE,
+ ):
+ super().__init__(name)
+ self.password = password
+ self.email = email
+ self.phone = phone
+ self.tenant = tenant
+ self.queue = queue
+ self.status = status
+
+ def create_tenant_if_not_exists(self) -> None:
+ """Create tenant object."""
+ tenant = Tenant(name=self.tenant, queue=self.queue)
+ tenant.create_if_not_exists(self.queue)
+
+ def create_if_not_exists(self, **kwargs):
+ """Create User if not exists."""
+ # Should make sure queue already exists.
+ self.create_tenant_if_not_exists()
+ gateway = launch_gateway()
+ gateway.entry_point.createUser(
+ self.name,
+ self.password,
+ self.email,
+ self.phone,
+ self.tenant,
+ self.queue,
+ self.status,
+ )
+ # TODO recover result checker
+ # gateway_result_checker(result, None)
diff --git a/src/pydolphinscheduler/models/worker_group.py b/src/pydolphinscheduler/models/worker_group.py
new file mode 100644
index 0000000..bc55eaf
--- /dev/null
+++ b/src/pydolphinscheduler/models/worker_group.py
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DolphinScheduler Worker Group object."""
+
+from typing import Optional
+
+from pydolphinscheduler.models import BaseSide
+
+
+class WorkerGroup(BaseSide):
+ """DolphinScheduler Worker Group object."""
+
+ def __init__(self, name: str, address: str, description: Optional[str] = None):
+ super().__init__(name, description)
+ self.address = address
diff --git a/src/pydolphinscheduler/tasks/condition.py b/src/pydolphinscheduler/tasks/condition.py
index 50aac25..cb139f1 100644
--- a/src/pydolphinscheduler/tasks/condition.py
+++ b/src/pydolphinscheduler/tasks/condition.py
@@ -20,9 +20,9 @@
from typing import Dict, List
from pydolphinscheduler.constants import TaskType
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.core.task import Task
from pydolphinscheduler.exceptions import PyDSParamException
+from pydolphinscheduler.models.base import Base
class Status(Base):
diff --git a/src/pydolphinscheduler/tasks/dependent.py b/src/pydolphinscheduler/tasks/dependent.py
index 7cff24c..cc6d25b 100644
--- a/src/pydolphinscheduler/tasks/dependent.py
+++ b/src/pydolphinscheduler/tasks/dependent.py
@@ -20,10 +20,10 @@
from typing import Dict, Optional, Tuple
from pydolphinscheduler.constants import TaskType
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.core.task import Task
from pydolphinscheduler.exceptions import PyDSJavaGatewayException, PyDSParamException
from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models.base import Base
DEPENDENT_ALL_TASK_IN_WORKFLOW = "0"
@@ -31,8 +31,8 @@
class DependentDate(str):
"""Constant of Dependent date value.
- These values set according to Java server side, if you want to add and change it,
- please change Java server side first.
+ These values set according to Java server models, if you want to add and change it,
+ please change Java server models first.
"""
# TODO Maybe we should add parent level to DependentDate for easy to use, such as
diff --git a/src/pydolphinscheduler/tasks/sql.py b/src/pydolphinscheduler/tasks/sql.py
index a125982..716a024 100644
--- a/src/pydolphinscheduler/tasks/sql.py
+++ b/src/pydolphinscheduler/tasks/sql.py
@@ -98,7 +98,7 @@
return self.param_sql_type
pattern_select_str = (
"^(?!(.* |)insert |(.* |)delete |(.* |)drop "
- "|(.* |)update |(.* |)alter |(.* |)create ).*"
+ "|(.* |)update |(.* |)truncate |(.* |)alter |(.* |)create ).*"
)
pattern_select = re.compile(pattern_select_str, re.IGNORECASE)
if pattern_select.match(self.sql) is None:
diff --git a/src/pydolphinscheduler/tasks/switch.py b/src/pydolphinscheduler/tasks/switch.py
index 0c9a2b8..35eece8 100644
--- a/src/pydolphinscheduler/tasks/switch.py
+++ b/src/pydolphinscheduler/tasks/switch.py
@@ -20,9 +20,9 @@
from typing import Dict, Optional
from pydolphinscheduler.constants import TaskType
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.core.task import Task
from pydolphinscheduler.exceptions import PyDSParamException
+from pydolphinscheduler.models.base import Base
class SwitchBranch(Base):
@@ -99,23 +99,22 @@
result = []
num_branch_default = 0
for condition in self.args:
- if isinstance(condition, SwitchBranch):
- if num_branch_default < 1:
- if isinstance(condition, Default):
- self._DEFINE_ATTR.add("next_node")
- setattr(self, "next_node", condition.next_node)
- num_branch_default += 1
- elif isinstance(condition, Branch):
- result.append(condition.get_define())
- else:
- raise PyDSParamException(
- "Task Switch's parameter only support exactly one default branch."
- )
- else:
+ if not isinstance(condition, SwitchBranch):
raise PyDSParamException(
"Task Switch's parameter only support SwitchBranch but got %s.",
type(condition),
)
+ # Default number branch checker
+ if num_branch_default >= 1 and isinstance(condition, Default):
+ raise PyDSParamException(
+ "Task Switch's parameter only support exactly one default branch."
+ )
+ if isinstance(condition, Default):
+ self._DEFINE_ATTR.add("next_node")
+ setattr(self, "next_node", condition.next_node)
+ num_branch_default += 1
+ elif isinstance(condition, Branch):
+ result.append(condition.get_define())
# Handle switch default branch, default value is `""` if not provide.
if num_branch_default == 0:
self._DEFINE_ATTR.add("next_node")
diff --git a/tests/cli/test_config.py b/tests/cli/test_config.py
index d913277..516ad75 100644
--- a/tests/cli/test_config.py
+++ b/tests/cli/test_config.py
@@ -23,7 +23,7 @@
import pytest
from pydolphinscheduler.cli.commands import cli
-from pydolphinscheduler.core.configuration import BUILD_IN_CONFIG_PATH, config_path
+from pydolphinscheduler.configuration import BUILD_IN_CONFIG_PATH, config_path
from tests.testing.cli import CliTestWrapper
from tests.testing.constants import DEV_MODE, ENV_PYDS_HOME
from tests.testing.file import get_file_content
diff --git a/tests/cli/test_version.py b/tests/cli/test_version.py
index f0dcb0e..b61d26d 100644
--- a/tests/cli/test_version.py
+++ b/tests/cli/test_version.py
@@ -17,9 +17,11 @@
"""Test command line interface subcommand `version`."""
+from unittest.mock import patch
+
import pytest
-from pydolphinscheduler import __version__
+import pydolphinscheduler
from pydolphinscheduler.cli.commands import cli
from tests.testing.cli import CliTestWrapper
@@ -27,21 +29,27 @@
def test_version():
"""Test whether subcommand `version` correct."""
cli_test = CliTestWrapper(cli, ["version"])
- cli_test.assert_success(output=f"{__version__}")
+ cli_test.assert_success(output=f"{pydolphinscheduler.__version__}")
@pytest.mark.parametrize(
- "part, idx",
+ "version, part, idx",
[
- ("major", 0),
- ("minor", 1),
- ("micro", 2),
+ ("1.2.3", "major", 0),
+ ("0.1.3", "minor", 1),
+ ("3.1.0", "micro", 2),
+ ("1.2.3-beta-1", "micro", 2),
+ ("1.2.3-alpha", "micro", 2),
+ ("1.2.3a2", "micro", 2),
+ ("1.2.3b1", "micro", 2),
],
)
-def test_version_part(part: str, idx: int):
+@patch("pydolphinscheduler.__version__")
+def test_version_part(mock_version, version: str, part: str, idx: int):
"""Test subcommand `version` option `--part`."""
+ mock_version.return_value = version
cli_test = CliTestWrapper(cli, ["version", "--part", part])
- cli_test.assert_success(output=f"{__version__.split('.')[idx]}")
+ cli_test.assert_success(output=f"{pydolphinscheduler.__version__.split('.')[idx]}")
@pytest.mark.parametrize(
diff --git a/tests/core/test_configuration.py b/tests/core/test_configuration.py
index c7e217a..b9dc8cb 100644
--- a/tests/core/test_configuration.py
+++ b/tests/core/test_configuration.py
@@ -24,8 +24,8 @@
import pytest
-from pydolphinscheduler.core import configuration
-from pydolphinscheduler.core.configuration import (
+from pydolphinscheduler import configuration
+from pydolphinscheduler.configuration import (
BUILD_IN_CONFIG_PATH,
config_path,
get_single_config,
diff --git a/tests/core/test_process_definition.py b/tests/core/test_process_definition.py
index 5cb6dab..30445bf 100644
--- a/tests/core/test_process_definition.py
+++ b/tests/core/test_process_definition.py
@@ -24,11 +24,11 @@
import pytest
from freezegun import freeze_time
-from pydolphinscheduler.core import configuration
+from pydolphinscheduler import configuration
from pydolphinscheduler.core.process_definition import ProcessDefinition
from pydolphinscheduler.core.resource import Resource
from pydolphinscheduler.exceptions import PyDSParamException
-from pydolphinscheduler.side import Project, Tenant, User
+from pydolphinscheduler.models import Project, Tenant, User
from pydolphinscheduler.tasks.switch import Branch, Default, Switch, SwitchCondition
from pydolphinscheduler.utils.date import conv_to_schedule
from tests.testing.task import Task
diff --git a/tests/tasks/test_sql.py b/tests/tasks/test_sql.py
index ee0acc4..50ccd94 100644
--- a/tests/tasks/test_sql.py
+++ b/tests/tasks/test_sql.py
@@ -54,6 +54,7 @@
("delete from table_name where id < 10", None, SqlType.NOT_SELECT),
("alter table table_name add column col1 int", None, SqlType.NOT_SELECT),
("create table table_name2 (col1 int)", None, SqlType.NOT_SELECT),
+ ("truncate table table_name", None, SqlType.NOT_SELECT),
("create table table_name2 (col1 int)", SqlType.SELECT, SqlType.SELECT),
("select 1", SqlType.NOT_SELECT, SqlType.NOT_SELECT),
("create table table_name2 (col1 int)", SqlType.NOT_SELECT, SqlType.NOT_SELECT),
diff --git a/tests/testing/path.py b/tests/testing/path.py
index d1e520b..68d93c4 100644
--- a/tests/testing/path.py
+++ b/tests/testing/path.py
@@ -26,7 +26,7 @@
path_example = project_root.joinpath("src", "pydolphinscheduler", "examples")
path_doc_tasks = project_root.joinpath("docs", "source", "tasks")
path_default_config_yaml = project_root.joinpath(
- "src", "pydolphinscheduler", "core", "default_config.yaml"
+ "src", "pydolphinscheduler", "default_config.yaml"
)
diff --git a/tox.ini b/tox.ini
index 8e9280f..d90e8a3 100644
--- a/tox.ini
+++ b/tox.ini
@@ -16,10 +16,12 @@
# under the License.
[tox]
-envlist = local-ci, auto-lint, lint, doc-build-test, code-test, integrate-test, py{36,37,38,39}
+envlist = local-ci, auto-lint, lint, doc-build, doc-build-multi, code-test, integrate-test, local-integrate-test, py{36,37,38,39,310,311}
[testenv]
-whitelist_externals = make
+allowlist_externals =
+ make
+ git
[testenv:auto-lint]
extras = style
@@ -42,12 +44,20 @@
commands =
python -m pytest --cov=pydolphinscheduler --cov-config={toxinidir}/.coveragerc tests/
-[testenv:doc-build-test]
+[testenv:doc-build]
extras = doc
commands =
make -C {toxinidir}/docs clean
make -C {toxinidir}/docs html
+[testenv:doc-build-multi]
+extras = doc
+commands =
+ # Get all tags for `multiversion` subcommand
+ git fetch --tags
+ make -C {toxinidir}/docs clean
+ make -C {toxinidir}/docs multiversion
+
[testenv:integrate-test]
extras = test
commands =
@@ -58,4 +68,4 @@
commands =
{[testenv:lint]commands}
{[testenv:code-test]commands}
- {[testenv:doc-build-test]commands}
+ {[testenv:doc-build]commands}
