| <!DOCTYPE html> |
| <!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]--> |
| <!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]--> |
| <!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]--><head> |
| <meta charset='utf-8'/><meta http-equiv='X-UA-Compatible' content='IE=edge'/><meta name='viewport' content='width=device-width, initial-scale=1'/><meta name='keywords' content='groovy, java, spock, testing, jqwik, pitest, junit, jacoco'/><meta name='description' content='This post looks at testing Java using Groovy, Spock, JUnit5, Jacoco, Jqwik and Pitest'/><title>The Apache Groovy programming language - Blogs - Testing your Java with Groovy, Spock, JUnit5, Jacoco, Jqwik and Pitest</title><link href='../img/favicon.ico' type='image/x-ico' rel='icon'/><link rel='stylesheet' type='text/css' href='../css/bootstrap.css'/><link rel='stylesheet' type='text/css' href='../css/font-awesome.min.css'/><link rel='stylesheet' type='text/css' href='../css/style.css'/><link rel='stylesheet' type='text/css' href='https://cdnjs.cloudflare.com/ajax/libs/prettify/r298/prettify.min.css'/> |
| </head><body> |
| <div id='fork-me'> |
| <a href='https://github.com/apache/groovy'> |
| <img style='position: fixed; top: 20px; right: -58px; border: 0; z-index: 100; transform: rotate(45deg);' src='/img/horizontal-github-ribbon.png'/> |
| </a> |
| </div><div id='st-container' class='st-container st-effect-9'> |
| <nav class='st-menu st-effect-9' id='menu-12'> |
| <h2 class='icon icon-lab'>Socialize</h2><ul> |
| <li> |
| <a href='https://groovy-lang.org/mailing-lists.html' class='icon'><span class='fa fa-envelope'></span> Discuss on the mailing-list</a> |
| </li><li> |
| <a href='https://twitter.com/ApacheGroovy' class='icon'><span class='fa fa-twitter'></span> Groovy on Twitter</a> |
| </li><li> |
| <a href='https://groovy-lang.org/events.html' class='icon'><span class='fa fa-calendar'></span> Events and conferences</a> |
| </li><li> |
| <a href='https://github.com/apache/groovy' class='icon'><span class='fa fa-github'></span> Source code on GitHub</a> |
| </li><li> |
| <a href='https://groovy-lang.org/reporting-issues.html' class='icon'><span class='fa fa-bug'></span> Report issues in Jira</a> |
| </li><li> |
| <a href='http://stackoverflow.com/questions/tagged/groovy' class='icon'><span class='fa fa-stack-overflow'></span> Stack Overflow questions</a> |
| </li><li> |
| <a href='http://groovycommunity.com/' class='icon'><span class='fa fa-slack'></span> Slack Community</a> |
| </li> |
| </ul> |
| </nav><div class='st-pusher'> |
| <div class='st-content'> |
| <div class='st-content-inner'> |
| <!--[if lt IE 7]> |
| <p class="browsehappy">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your experience.</p> |
| <![endif]--><div><div class='navbar navbar-default navbar-static-top' role='navigation'> |
| <div class='container'> |
| <div class='navbar-header'> |
| <button type='button' class='navbar-toggle' data-toggle='collapse' data-target='.navbar-collapse'> |
| <span class='sr-only'></span><span class='icon-bar'></span><span class='icon-bar'></span><span class='icon-bar'></span> |
| </button><a class='navbar-brand' href='../index.html'> |
| <i class='fa fa-star'></i> Apache Groovy |
| </a> |
| </div><div class='navbar-collapse collapse'> |
| <ul class='nav navbar-nav navbar-right'> |
| <li class=''><a href='https://groovy-lang.org/learn.html'>Learn</a></li><li class=''><a href='https://groovy-lang.org/documentation.html'>Documentation</a></li><li class=''><a href='/download.html'>Download</a></li><li class=''><a href='https://groovy-lang.org/support.html'>Support</a></li><li class=''><a href='/'>Contribute</a></li><li class=''><a href='https://groovy-lang.org/ecosystem.html'>Ecosystem</a></li><li class=''><a href='/blog'>Blog posts</a></li><li class=''><a href='https://groovy.apache.org/events.html'></a></li><li> |
| <a data-effect='st-effect-9' class='st-trigger' href='#'>Socialize</a> |
| </li><li class=''> |
| <a href='../search.html'> |
| <i class='fa fa-search'></i> |
| </a> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div><div id='content' class='page-1'><div class='row'><div class='row-fluid'><div class='col-lg-3'><ul class='nav-sidebar'><li><a href='./'>Blog index</a></li><li class='active'><a href='#doc'>Testing your Java with Groovy, Spock, JUnit5, Jacoco, Jqwik and Pitest</a></li><li><a href='#_the_system_under_test' class='anchor-link'>The system under test</a></li><li><a href='#_testing_with_spock' class='anchor-link'>Testing with Spock</a></li><li><a href='#_mutation_testing_with_pitest' class='anchor-link'>Mutation testing with Pitest</a></li><li><a href='#_using_property_based_testing' class='anchor-link'>Using Property-based Testing</a></li><li><a href='#_more_information' class='anchor-link'>More information</a></li><li><a href='#_conclusion' class='anchor-link'>Conclusion</a></li></ul><br/><ul class='nav-sidebar'><li style='padding: 0.35em 0.625em; background-color: #eee'><span>Related posts</span></li><li><a href='./testing_permutations_combinations'>Groovy Testing with Combinations and Permutations</a></li><li><a href='./groovy-record-performance'>Groovy Record Performance</a></li><li><a href='./fun-with-obfuscated-groovy'>Fun with obfuscated Groovy</a></li></ul></div><div class='col-lg-8 col-lg-pull-0'><a name='doc'></a><h1>Testing your Java with Groovy, Spock, JUnit5, Jacoco, Jqwik and Pitest</h1><p><span>Author: <i>Paul King</i></span><br/><span>Published: 2022-07-15 08:26AM</span></p><hr/><div id="preamble"> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p><span class="image right"><img src="img/spock_logo.png" alt="spock logo" width="100"></span> |
| This blog post covers a common scenario seen in the Groovy community which is |
| projects which use Java for their production code and Groovy for their tests. |
| This can be a low risk way for Java shops to try out and become more familiar |
| with Groovy. We’ll write our initial tests using the |
| <a href="https://spockframework.org/">Spock testing framework</a>, and we’ll use |
| <a href="https://junit.org/junit5/">JUnit5</a> later with our jqwik tests. |
| You can usually use your favorite Java testing libraries if you switch to Groovy.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_the_system_under_test">The system under test</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>For illustrative purposes, we will test a Java mathematics utility function |
| <code>sumBiggestPair</code>. Given three numbers, it finds the two biggest and then adds them up. |
| An initial stab at the code for this might look something like this:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="java">public class MathUtil { // Java |
| |
| public static int sumBiggestPair(int a, int b, int c) { |
| int op1 = a; |
| int op2 = b; |
| if (c > a) { |
| op1 = c; |
| } else if (c > b) { |
| op2 = c; |
| } |
| return op1 + op2; |
| } |
| |
| private MathUtil(){} |
| }</code></pre> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_testing_with_spock">Testing with Spock</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>An initial test could look like this:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">class MathUtilSpec extends Specification { |
| def "sum of two biggest numbers"() { |
| expect: |
| MathUtil.sumBiggestPair(2, 5, 3) == 8 |
| } |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>When we run this test, all tests pass:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/MathUtilSpecResult.png" alt="MathUtilSpec test result"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>But if we look at the coverage report, generated with |
| <a href="https://github.com/jacoco/jacoco">Jacoco</a>, we see that our test |
| hasn’t covered all lines of code:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/MathUtilJacocoReport.png" alt="MathUtilSpec coverage report"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>We’ll swap to use Spock’s data-driven feature and include an additional testcase:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">def "sum of two biggest numbers"(int a, int b, int c, int d) { |
| expect: |
| MathUtil.sumBiggestPair(a, b, c) == d |
| |
| where: |
| a | b | c | d |
| 2 | 5 | 3 | 8 |
| <strong>5 | 2 | 3 | 8</strong> |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>We can check our coverage again:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/MathUtilJacocoReport2.png" alt="MathUtilSpec coverage report"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>That is a little better. We now have 100% line coverage but not 100% branch coverage. |
| Let’s add one more testcase:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">def "sum of two biggest numbers"(int a, int b, int c, int d) { |
| expect: |
| MathUtil.sumBiggestPair(a, b, c) == d |
| |
| where: |
| a | b | c | d |
| 2 | 5 | 3 | 8 |
| 5 | 2 | 3 | 8 |
| <strong>5 | 4 | 1 | 9</strong> |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>And now we can see that we have reached 100% line coverage and 100% branch coverage:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/MathUtilJacocoReport3.png" alt="MathUtilSpec coverage report"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>At this point, we might be very confident in our code and ready to ship it to production. |
| Before we do, we’ll add one more testcase:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">def "sum of two biggest numbers"(int a, int b, int c, int d) { |
| expect: |
| MathUtil.sumBiggestPair(a, b, c) == d |
| |
| where: |
| a | b | c | d |
| 2 | 5 | 3 | 8 |
| 5 | 2 | 3 | 8 |
| 5 | 4 | 1 | 9 |
| <strong>3 | 2 | 6 | 9</strong> |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>When we re-run our tests, we discover that the last testcase failed!:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/MathUtilSpecResult2.png" alt="MathUtilSpec test result"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>And examining the testcase, we can indeed see that there is a flaw in our algorithm. |
| Basically, having the <code>else</code> logic doesn’t cater for when <code>c</code> is |
| greater than both <code>a</code> and <code>b</code>!</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/MathUtilSpecResultFailedAssertion.png" alt="MathUtilSpec failed assertion"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>We succumbed to faulty expectations of what 100% coverage would give us.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/ImperfectPuzzle.jpg" alt="Imperfect puzzle" width="250"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The good news is that we can fix this. Here is an updated algorithm:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="java">public static int sumBiggestPair(int a, int b, int c) { // Java |
| int op1 = a; |
| int op2 = b; |
| if (c > Math.min(a, b)) { |
| op1 = c; |
| op2 = Math.max(a, b); |
| } |
| return op1 + op2; |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>With this new algorithm, all 4 testcases now pass, |
| and we again have 100% line and branch coverage.</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre>> Task :SumBiggestPairPitest:test |
| <strong class="lime">✔</strong> Test sum of two biggest numbers [Tests: 4/<strong class="lime">4</strong>/<strong class="red">0</strong>/<strong class="gold">0</strong>] [Time: 0.317 s] |
| <strong class="lime">✔</strong> Test util.MathUtilSpec [Tests: 4/<strong class="lime">4</strong>/<strong class="red">0</strong>/<strong class="gold">0</strong>] [Time: 0.320 s] |
| <strong class="lime">✔</strong> Test Gradle Test Run :SumBiggestPairPitest:test [Tests: 4/<strong class="lime">4</strong>/<strong class="red">0</strong>/<strong class="gold">0</strong>]</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>But haven’t we been here before? How can we be sure there isn’t some additional test |
| cases that might reveal another flaw in our algorithm? We could keep writing lots more |
| testcases, but we’ll look at two other techniques that can help.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_mutation_testing_with_pitest">Mutation testing with Pitest</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>An interesting but not widely used technique is mutation testing. It probably deserves |
| to be more widely used. It can test the quality of a testsuite but has the drawback of |
| sometimes being quite resource intensive. It modifies (mutates) production code and |
| re-runs your testsuite. If your test suite still passes with modified code, it possibly |
| indicates that your testsuite is lacking sufficient coverage. Earlier, we had an algorithm |
| with a flaw and our testsuite didn’t initially pick it up. You can think of mutation |
| testing as adding a deliberate flaw and seeing whether your testsuite is good enough |
| to detect that flaw.</p> |
| </div> |
| <div class="paragraph"> |
| <p>If you’re a fan of test-driven development (TDD), it espouses a rule that not a single |
| line of production code should be added unless a failing test forces that line to be |
| added. A corollary is that if you change a single line of production code in any |
| meaningful way, that some test should fail.</p> |
| </div> |
| <div class="paragraph"> |
| <p>So, let’s have a look at what mutation testing says about our initial flawed algorithm. |
| We’ll use Pitest (also known as PIT). We’ll go back to our initial algorithm and the point |
| where we erroneously thought we had 100% coverage. When we run Pitest, we get the |
| following result:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/PitestCoverageReport.png" alt="Pitest coverage report summary"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>And looking at the code we see:</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/PitestMathUtilCoverage.png" alt="Pitest coverage report"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>With output including some statistics:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre>====================================================================== |
| - Statistics |
| ====================================================================== |
| >> Line Coverage: 7/8 (88%) |
| >> Generated 6 mutations Killed 4 (67%) |
| >> Mutations with no coverage 0. Test strength 67% |
| >> Ran 26 tests (4.33 tests per mutation)</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>What is this telling us? Pitest mutated our code in ways that you might expect to break |
| it but our testsuite passed (survived) in a couple of instances. That means one of two |
| things. Either, there are multiple valid implementations of our algorithm and Pitest |
| found one of those equivalent solutions, or our testsuite is lacking some key testcases. |
| In our case, we know that the testsuite was insufficient.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Let’s run it again but this time with all of our tests and the corrected algorithm.</p> |
| </div> |
| <div class="paragraph"> |
| <p><span class="image"><img src="img/PitestCoverage2.png" alt="Pitest coverage report"></span></p> |
| </div> |
| <div class="paragraph"> |
| <p>The output when running the test has also changed slightly:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre>====================================================================== |
| - Statistics |
| ====================================================================== |
| >> Line Coverage: 6/7 (86%) |
| >> Generated 4 mutations Killed 3 (75%) |
| >> Mutations with no coverage 0. Test strength 75% |
| >> Ran 25 tests (6.25 tests per mutation)</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Our warnings from Pitest have reduced but not gone completely away and our test strength |
| has gone up but is still not 100%. It does mean that we are in better shape than before. |
| But should we be concerned?</p> |
| </div> |
| <div class="paragraph"> |
| <p>It turns out in this case, we don’t need to worry (too much). As an example, an equally |
| valid algorithm for our function under test would be to replace the conditional with |
| <code>c >= Math.min(a, b)</code>. Note the greater-than-equals operator rather than just greater-than. For this algorithm, a different path would be taken for the case when <code>c</code> equals <code>a</code> or <code>b</code>, but the end result would be the same. So, that would be an inconsequential or equivalent mutation. In such a case, there may be no additional testcase that we can write to keep Pitest happy. We have to be aware of this possible outcome when using this technique.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Finally, let’s look at our build file that ran Spock, Jacoco and Pitest:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">plugins { |
| id 'info.solidsoft.pitest' version '1.7.4' |
| } |
| apply plugin: 'groovy' |
| |
| repositories { |
| mavenCentral() |
| } |
| |
| dependencies { |
| implementation "org.apache.groovy:groovy-test-junit5:4.0.3" |
| testImplementation("org.spockframework:spock-core:2.2-M3-groovy-4.0") { |
| transitive = false |
| } |
| } |
| |
| pitest { |
| junit5PluginVersion = '1.0.0' |
| pitestVersion = '1.9.2' |
| timestampedReports = false |
| targetClasses = ['util.*'] |
| } |
| |
| tasks.named('test') { |
| useJUnitPlatform() |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The astute reader might note some subtle hints which show that the latest Spock versions |
| run on top of the JUnit 5 platform.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_using_property_based_testing">Using Property-based Testing</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>Property-based testing is another technology which probably deserves much more attention. |
| Here we’ll use <a href="https://jqwik.net/">jqwik</a> which runs on top of JUnit5, |
| but you might also like to consider |
| <a href="https://github.com/Bijnagte/spock-genesis">Genesis</a> |
| which provides random generators and especially targets Spock.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Earlier, we looked at writing <em>more</em> tests to make our coverage stronger. Property-based |
| testing can often lead to writing <em>less</em> tests. Instead, we generate many random tests |
| automatically and see whether certain properties hold.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Previously, we fed in the inputs and the expected output. For property-based testing, |
| the inputs are typically randomly-generated values, we don’t know the output. |
| So, instead of testing directly against some known output, we’ll just check various |
| properties of the answer.</p> |
| </div> |
| <div class="paragraph"> |
| <p>As an example, here is a test we could use:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">@Property |
| void "result should be bigger than any individual and smaller than sum of all"( |
| @ForAll @IntRange(min = 0, max = 1000) Integer a, |
| @ForAll @IntRange(min = 0, max = 1000) Integer b, |
| @ForAll @IntRange(min = 0, max = 1000) Integer c) { |
| def result = sumBiggestPair(a, b, c) |
| assert [a, b, c].every { individual -> result >= individual } |
| assert result <= a + b + c |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>The <code>@ForAll</code> annotations indicate places where jqwik will insert random values. |
| The <code>@IntRange</code> annotation indicates that we want the random values to be contained |
| between 0 and 1000.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Here we are checking that (at least for small positive numbers) adding the two biggest |
| numbers should be greater than or equal to any individual number and should be less than |
| or equal to adding all three of the numbers. These are necessary but insufficient |
| properties to ensure our system works.</p> |
| </div> |
| <div class="paragraph"> |
| <p>When we run this we see the following output in the logs:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre> |--------------------jqwik-------------------- |
| tries = 1000 | # of calls to property |
| checks = 1000 | # of not rejected calls |
| generation = RANDOMIZED | parameters are randomly generated |
| after-failure = PREVIOUS_SEED | use the previous seed |
| when-fixed-seed = ALLOW | fixing the random seed is allowed |
| edge-cases#mode = MIXIN | edge cases are mixed in |
| edge-cases#total = 125 | # of all combined edge cases |
| edge-cases#tried = 117 | # of edge cases tried in current run |
| seed = -311315135281003183 | random seed to reproduce generated values</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>So, we wrote 1 test and 1000 testcases were executed. The number of tests run is |
| configurable. We won’t go into the details here. This looks great at first glance. |
| It turns out however, that this particular property is not very discriminating in |
| terms of the bugs it can find. This test passes for both our original flawed algorithm |
| as well as the fixed one. Let’s try a different property:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">@Property |
| void "sum of any pair should not be greater than result"( |
| @ForAll @IntRange(min = 0, max = 1000) Integer a, |
| @ForAll @IntRange(min = 0, max = 1000) Integer b, |
| @ForAll @IntRange(min = 0, max = 1000) Integer c) { |
| def result = sumBiggestPair(a, b, c) |
| assert [a + b, b + c, c + a].every { sumOfPair -> result >= sumOfPair } |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>If we calculate the biggest pair, then surely it must be greater than or equal to any |
| arbitrary pair. Trying this on our flawed algorithm gives:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre>org.codehaus.groovy.runtime.powerassert.PowerAssertionError: |
| assert [a + b, b + c, c + a].every { sumOfPair -> result >= sumOfPair } |
| | | | | | | | | | | |
| 1 1 0 0 2 2 2 3 1 false |
| |--------------------jqwik-------------------- |
| tries = 12 | # of calls to property |
| checks = 12 | # of not rejected calls |
| generation = RANDOMIZED | parameters are randomly generated |
| after-failure = PREVIOUS_SEED | use the previous seed |
| when-fixed-seed = ALLOW | fixing the random seed is allowed |
| edge-cases#mode = MIXIN | edge cases are mixed in |
| edge-cases#total = 125 | # of all combined edge cases |
| edge-cases#tried = 2 | # of edge cases tried in current run |
| seed = 4830696361996686755 | random seed to reproduce generated values |
| |
| Shrunk Sample (6 steps) |
| ----------------------- |
| arg0: 1 |
| arg1: 0 |
| arg2: 2 |
| |
| Original Sample |
| --------------- |
| arg0: 247 |
| arg1: 32 |
| arg2: 267 |
| |
| Original Error |
| -------------- |
| org.codehaus.groovy.runtime.powerassert.PowerAssertionError: |
| assert [a + b, b + c, c + a].every { sumOfPair -> result >= sumOfPair } |
| | | | | | | | | | | |
| | | 32 32| 267| | | false |
| | 279 299 | | 247 |
| 247 | 514 |
| 267</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Not only did it find a case which highlighted the flaw, but it shrunk it down to a very |
| simple example. On our fixed algorithm, the 1000 tests pass!</p> |
| </div> |
| <div class="paragraph"> |
| <p>The previous property can be refactored a little to not only calculate all three pairs |
| but then find the maximum of those. This simplifies the condition somewhat:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">@Property |
| void "result should be the same as alternative oracle implementation"( |
| @ForAll @IntRange(min = 0, max = 1000) Integer a, |
| @ForAll @IntRange(min = 0, max = 1000) Integer b, |
| @ForAll @IntRange(min = 0, max = 1000) Integer c) { |
| assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>This approach, where an alternative implementation is used, is known as a test oracle. |
| The alternative implementation might be less efficient, so not ideal for production code, |
| but fine for testing. When revamping or replacing some software, the oracle might be the |
| existing system. When run on our fixed algorithm, we again have 1000 testcases passing.</p> |
| </div> |
| <div class="paragraph"> |
| <p>Let’s go one step further and remove our <code>@IntRange</code> boundaries on the Integers:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">@Property |
| void "result should be the same as alternative oracle implementation"(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { |
| assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>When we run the test now, we might be surprised:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre> org.codehaus.groovy.runtime.powerassert.PowerAssertionError: |
| assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() |
| | | | | | ||| ||| ||| | |
| -2147483648 0 1 | | 0|1 0|| 1|| 2147483647 |
| | | 1 || |2147483647 |
| | false || -2147483648 |
| 2147483647 |2147483647 |
| 2147483647 |
| Shrunk Sample (13 steps) |
| ------------------------ |
| arg0: 0 |
| arg1: 1 |
| arg2: 2147483647</pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>It fails! Is this another bug in our algorithm? Possibly? But it could equally be |
| a bug in our property test. Further investigation is warranted.</p> |
| </div> |
| <div class="paragraph"> |
| <p>It turns out that our algorithm suffers from Integer overflow when trying to add <code>1</code> to |
| <code>Integer.MAX_VALUE</code>. Our test partially suffers from the same problem but when we call |
| <code>max()</code>, the negative value will be discarded. There is no always correct answer as to |
| what should happen in this scenario. We go back to the customer and check the real |
| requirement. In this case, let’s assume the customer was happy for the overflow to |
| occur - since that is what would happen if performing the operation long-hand in Java. |
| With that knowledge we should fix our test to at least pass correctly when overflow occurs.</p> |
| </div> |
| <div class="paragraph"> |
| <p>We have a number of options to fix this. We already saw previously we can use <code>@IntRange</code>. |
| This is one way to "avoid" the problem and we have a few similar approaches which do the |
| same. We could use a more confined data type, e.g. <code>Short</code>:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">@Property |
| void checkShort(@ForAll Short a, @ForAll Short b, @ForAll Short c) { |
| assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Or we could use a customised provider method:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">@Property |
| void checkIntegerConstrainedProvider(@ForAll('halfMax') Integer a, |
| @ForAll('halfMax') Integer b, |
| @ForAll('halfMax') Integer c) { |
| assert sumBiggestPair(a, b, c) == [a+b, a+c, b+c].max() |
| } |
| |
| @Provide |
| Arbitrary<Integer> halfMax() { |
| int halfMax = Integer.MAX_VALUE >> 1 |
| return Arbitraries.integers().between(-halfMax, halfMax) |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>But rather than avoiding the problem, we could change our test so that it allowed for |
| the possibility of overflow within <code>sumBiggestPair</code> but didn’t compound the problem with |
| its own overflow. E.g. we could use Long’s to do our calculations within our test:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">@Property |
| void checkIntegerWithLongCalculations(@ForAll Integer a, @ForAll Integer b, @ForAll Integer c) { |
| def (al, bl, cl) = [a, b, c]*.toLong() |
| assert sumBiggestPair(a, b, c) == [al+bl, al+cl, bl+cl].max().toInteger() |
| }</code></pre> |
| </div> |
| </div> |
| <div class="paragraph"> |
| <p>Finally, let’s again look at our Gradle build file:</p> |
| </div> |
| <div class="listingblock"> |
| <div class="content"> |
| <pre class="prettyprint highlight"><code data-lang="groovy">apply plugin: 'groovy' |
| |
| repositories { |
| mavenCentral() |
| } |
| |
| dependencies { |
| testImplementation project(':SumBiggestPair') |
| testImplementation "org.apache.groovy:groovy-test-junit5:4.0.3" |
| testImplementation "net.jqwik:jqwik:1.6.5" |
| } |
| |
| test { |
| useJUnitPlatform { |
| includeEngines 'jqwik' |
| } |
| }</code></pre> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_more_information">More information</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>The examples in this blog post are excerpts from the following repo:<br> |
| <a href="https://github.com/paulk-asert/property-based-testing" class="bare">https://github.com/paulk-asert/property-based-testing</a></p> |
| </div> |
| <div class="paragraph"> |
| <p>Library versions used:<br> |
| Gradle 7.5, Groovy 4.0.3, jqwik 1.6.5, pitest 1.9.2, Spock 2.2-M3-groovy-4.0, Jacoco 0.8.8.<br> |
| Tested with JDK 8, 11, 17, 18.</p> |
| </div> |
| <div class="paragraph"> |
| <p>There are many sites with valuable information about the technologies covered here. There are also some great books. Books on Spock include <a href="https://www.oreilly.com/library/view/spock-up-and/9781491923283/">Spock: Up and Running</a>, <a href="https://www.manning.com/books/java-testing-with-spock">Java Testing with Spock</a>, and |
| <a href="https://leanpub.com/spockframeworknotebook">Spocklight Notebook</a>. |
| Books on Groovy include: |
| <a href="https://www.manning.com/books/groovy-in-action-second-edition">Groovy in Action</a> |
| and <a href="https://link.springer.com/book/10.1007/978-1-4842-5058-7">Learning Groovy 3</a>. |
| If you want general information about using Java and Groovy together, consider |
| <a href="https://www.manning.com/books/making-java-groovy">Making Java Groovy</a>. |
| And there’s a section on mutation testing in <a href="http://kaczanowscy.pl/books/practical_unit_testing_junit_testng_mockito.html">Practical Unit Testing With Testng And Mockito</a>. The most recent book for property testing is for the <a href="https://pragprog.com/titles/fhproper/property-based-testing-with-proper-erlang-and-elixir/">Erlang and Elixir languages</a>.</p> |
| </div> |
| </div> |
| </div> |
| <div class="sect1"> |
| <h2 id="_conclusion">Conclusion</h2> |
| <div class="sectionbody"> |
| <div class="paragraph"> |
| <p>We have looked at testing Java code using Groovy and Spock with some additional |
| tools like Jacoco, jqwik and Pitest. Generally using Groovy to test Java is a |
| straight-forward experience. Groovy also lends itself to writing testing DSLs |
| which allow non-hard-core programmers to write very simple looking tests; |
| but that’s a topic for another blog!</p> |
| </div> |
| </div> |
| </div></div></div></div></div><footer id='footer'> |
| <div class='row'> |
| <div class='colset-3-footer'> |
| <div class='col-1'> |
| <h1>Groovy</h1><ul> |
| <li><a href='https://groovy-lang.org/learn.html'>Learn</a></li><li><a href='https://groovy-lang.org/documentation.html'>Documentation</a></li><li><a href='/download.html'>Download</a></li><li><a href='https://groovy-lang.org/support.html'>Support</a></li><li><a href='/'>Contribute</a></li><li><a href='https://groovy-lang.org/ecosystem.html'>Ecosystem</a></li><li><a href='/blog'>Blog posts</a></li><li><a href='https://groovy.apache.org/events.html'></a></li> |
| </ul> |
| </div><div class='col-2'> |
| <h1>About</h1><ul> |
| <li><a href='https://github.com/apache/groovy'>Source code</a></li><li><a href='https://groovy-lang.org/security.html'>Security</a></li><li><a href='https://groovy-lang.org/learn.html#books'>Books</a></li><li><a href='https://groovy-lang.org/thanks.html'>Thanks</a></li><li><a href='http://www.apache.org/foundation/sponsorship.html'>Sponsorship</a></li><li><a href='https://groovy-lang.org/faq.html'>FAQ</a></li><li><a href='https://groovy-lang.org/search.html'>Search</a></li> |
| </ul> |
| </div><div class='col-3'> |
| <h1>Socialize</h1><ul> |
| <li><a href='https://groovy-lang.org/mailing-lists.html'>Discuss on the mailing-list</a></li><li><a href='https://twitter.com/ApacheGroovy'>Groovy on Twitter</a></li><li><a href='https://groovy-lang.org/events.html'>Events and conferences</a></li><li><a href='https://github.com/apache/groovy'>Source code on GitHub</a></li><li><a href='https://groovy-lang.org/reporting-issues.html'>Report issues in Jira</a></li><li><a href='http://stackoverflow.com/questions/tagged/groovy'>Stack Overflow questions</a></li><li><a href='http://groovycommunity.com/'>Slack Community</a></li> |
| </ul> |
| </div><div class='col-right'> |
| <p> |
| The Groovy programming language is supported by the <a href='http://www.apache.org'>Apache Software Foundation</a> and the Groovy community. |
| </p><div text-align='right'> |
| <img src='../img/asf_logo.png' title='The Apache Software Foundation' alt='The Apache Software Foundation' style='width:60%'/> |
| </div><p>Apache® and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.</p> |
| </div> |
| </div><div class='clearfix'>© 2003-2023 the Apache Groovy project — Groovy is Open Source: <a href='http://www.apache.org/licenses/LICENSE-2.0.html' alt='Apache 2 License'>license</a>, <a href='https://privacy.apache.org/policies/privacy-policy-public.html'>privacy policy</a>.</div> |
| </div> |
| </footer></div> |
| </div> |
| </div> |
| </div> |
| </div><script src='../js/vendor/jquery-1.10.2.min.js' defer></script><script src='../js/vendor/classie.js' defer></script><script src='../js/vendor/bootstrap.js' defer></script><script src='../js/vendor/sidebarEffects.js' defer></script><script src='../js/vendor/modernizr-2.6.2.min.js' defer></script><script src='../js/plugins.js' defer></script><script src='https://cdnjs.cloudflare.com/ajax/libs/prettify/r298/prettify.min.js'></script><script>document.addEventListener('DOMContentLoaded',prettyPrint)</script><script> |
| (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
| (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), |
| m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
| })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); |
| |
| ga('create', 'UA-257558-10', 'auto'); |
| ga('send', 'pageview'); |
| </script> |
| </body></html> |
