| commit | 34492851f5c85ba2325ec40672f5b647e681eb02 | [log] [tgz] |
|---|---|---|
| author | Sebastian Baunsgaard <baunsgaard@apache.org> | Fri Apr 05 17:24:25 2024 +0200 |
| committer | Sebastian Baunsgaard <baunsgaard@apache.org> | Mon Apr 08 18:18:15 2024 +0200 |
| tree | 226ee398b95b3961c9340d12d7cc28b7c99fd8fc | |
| parent | 91834886fe234d86cd6d3b176e8b51fda8900ff7 [diff] |
[SYSTEMDS-3572] Thread pool ParFor name threads
This commit properly reuse common pools for individual
ParFor threads.
The update improves the performance of the following script from
9.6 to 7.5 sec on my laptop.
```
X = rand(rows = 100, cols = 100, min = 0, max = 100)
Z = rand(rows = 1000, cols = 100, min = 0, max = 100)
parfor( i in 1: 4){
for( i in 1 : 1000){
Y = Z %*% X
}
print(sum(Y))
}
```
Additionally this commit goes through all instances where
we allocate a thread pool and cleans up the instantiation and
shutdown of the threads in finally blocks, to make our system
more robust to threading errors.
Future work is to optionally replace many of these instances
with cleaner calls to the CommonThreadPool.invokeAndShutdown(pool, tasks).
Or an alternative API. This change would remove the need for try catch
blocks many places where we do parallelization.
Further more we can consider changing the way we propagate parallelism to
be contained in the threads allocated in ParFor, this would make integration
easier for instructions since we no longer would have to include it in The
lop strings.
Overview: SystemDS is an open source ML system for the end-to-end data science lifecycle from data integration, cleaning, and feature engineering, over efficient, local and distributed ML model training, to deployment and serving. To this end, we aim to provide a stack of declarative languages with R-like syntax for (1) the different tasks of the data-science lifecycle, and (2) users with different expertise. These high-level scripts are compiled into hybrid execution plans of local, in-memory CPU and GPU operations, as well as distributed operations on Apache Spark. In contrast to existing systems - that either provide homogeneous tensors or 2D Datasets - and in order to serve the entire data science lifecycle, the underlying data model are DataTensors, i.e., tensors (multi-dimensional arrays) whose first dimension may have a heterogeneous and nested schema.
| Resource | Links |
|---|---|
| Quick Start | Install, Quick Start and Hello World |
| Documentation: | SystemDS Documentation |
| Python Documentation | Python SystemDS Documentation |
| Issue Tracker | Jira Dashboard |
Status and Build: SystemDS is renamed from SystemML which is an Apache Top Level Project. To build from source visit SystemDS Install from source
