| commit | e20a24a6aca44968fe6a597be31e08a552d38dec | [log] [tgz] |
|---|---|---|
| author | Anup Ghatage <ghatageanup@gmail.com> | Tue Apr 21 13:27:30 2020 -0700 |
| committer | GitHub <noreply@github.com> | Tue Apr 21 13:27:30 2020 -0700 |
| tree | f3e87efd2059f392628da7f1c328a2d20e2de68f | |
| parent | 76286bc5505e4ba1cfeb47ce30bbe627d53f4869 [diff] |
Add forkCount option to parallelize build and tests ### Motivation Current code forks a new JVM per module. (bookkeeper-server, bookkeeper-proto etc) This means one fork per module for build and testing, no parallelism within the module where majority of the time goes. We need parallelism within a module during the test execution so we can have the builds complete faster and have the artifacts shipped out quicker. ### Changes We use the maven surefire plugin but don't define the `forkCount` and hence set it to default of 1. This means it executes each module with one thread. This change sets `forkCount` to 5, enabling parallelism in testing and drastically reducing total turnaround time. (by about 2/3rds!) *Total build+test time without this change* ``` [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:06 h [INFO] Finished at: 2020-03-05T02:01:29-08:00 [INFO] ------------------------------------------------------------------------ ``` *Total build+test time with this change* ``` [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 18:23 min [INFO] Finished at: 2020-03-05T02:38:22-08:00 [INFO] ------------------------------------------------------------------------ ``` ### Things to watch Added parallelism may cause some flappers but with much trial and error I have come to the number `5`. The flappers are usually only from conflict in obtaining the same port number. If needed, we can increase the retryCount, but as of now I consistently don't see any flappers at a `forkCount` of 5 Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Sijie Guo <None> This closes #2278 from Ghatage/parallelizeBuildAndTests
Apache BookKeeper is a scalable, fault tolerant and low latency storage service optimized for append-only workloads.
It is suitable for being used in following scenarios:
You can also read Turning Ledgers into Logs to learn how to turn ledgers into continuous log streams. If you are looking for a high level log stream API, you can checkout DistributedLog.
For filing bugs, suggesting improvements, or requesting new features, help us out by opening a Github issue or opening an Apache jira.
Subscribe or mail the user@bookkeeper.apache.org list - Ask questions, find answers, and also help other users.
Subscribe or mail the dev@bookkeeper.apache.org list - Join development discussions, propose new ideas and connect with contributors.
Join us on Slack - This is the most immediate way to connect with Apache BookKeeper committers and contributors.
We feel that a welcoming open community is important and welcome contributions.
See Developer Setup to get your local environment setup.
Take a look at our open issues: JIRA Issues Github Issues.
Review our coding style and follow our pull requests to learn about our conventions.
Make your changes according to our contribution guide.