| --- |
| active_crumb: Test Framework |
| layout: documentation |
| id: test_framework |
| fa_icon: fa-tools |
| --- |
| |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| |
| <div class="col-md-8 second-column" xmlns="http://www.w3.org/1999/html"> |
| <section id="overview"> |
| <h2 class="section-title">Overview <a href="#"><i class="top-link fas fa-fw fa-angle-double-up"></i></a></h2> |
| <p> |
| NLPCraft comes built-in with a simple-to-use testing framework that can be used with any |
| popular unit testing JVM frameworks such as <a href="https://testng.org">TestNG</a> or |
| <a href="https://junit.org">JUnit</a>. Test framework is essentially a simplified implementation |
| of the REST client tailor-made for model unit testing. |
| </p> |
| <p> |
| Test framework consists of the following types in <code>org.apache.nlpcraft.model.tools.test</code> package: |
| </p> |
| <ul> |
| <li> |
| Auto-testing: |
| <ul> |
| <li> |
| <a |
| target="javadoc" |
| href="/apis/latest/org/apache/nlpcraft/model/tools/test/NCTestAutoModelValidator.html"> |
| NCTestAutoModelValidator |
| </a> - auto-runner for model tests based on <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/NCIntentSample.html">@NCIntentSample</a> |
| and <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/NCIntentSampleRef.html">@NCIntentSampleRef</a> annotations. |
| </li> |
| </ul> |
| </li> |
| <li> |
| Using test client directly: |
| <ul> |
| <li> |
| <a |
| target="javadoc" |
| href="/apis/latest/org/apache/nlpcraft/model/tools/test/NCTestClient.html"> |
| NCTestClient |
| </a> - main model test client interface. |
| </li> |
| <li> |
| <a |
| target="javadoc" |
| href="/apis/latest/org/apache/nlpcraft/model/tools/test/NCTestClientBuilder.html"> |
| NCTestClientBuilder |
| </a> - builder for <code>NCTestClient</code> instances. |
| </li> |
| <li> |
| <a |
| target="javadoc" |
| href="/apis/latest/org/apache/nlpcraft/model/tools/test/NCTestResult.html"> |
| NCTestResult |
| </a> - result holder of the test sentence processing. |
| </li> |
| <li> |
| <a |
| target="javadoc" |
| href="/apis/latest/org/apache/nlpcraft/model/tools/test/NCTestClientException.html"> |
| NCTestClientException |
| </a> - exception used by the test framework. |
| </li> |
| |
| </ul> |
| </ul> |
| </section> |
| <section id="usage"> |
| <h2 class="section-title">Test client <a href="#"><i class="top-link fas fa-fw fa-angle-double-up"></i></a></h2> |
| <p> |
| Here's a code for unit test using test client for the model from <a href="/examples/alarm_clock.html">Alarm Clock</a> |
| example illustrating the usage of test framework together with JUnit 5: |
| </p> |
| <pre class="brush: java, highlight: [8, 10, 16, 24, 25, 26]"> |
| public class AlarmTest { |
| private NCTestClient cli; |
| |
| @BeforeEach |
| void setUp() throws NCException, IOException { |
| NCEmbeddedProbe.start(AlarmModel.class); |
| |
| cli = new NCTestClientBuilder().newBuilder().build(); |
| |
| cli.open("nlpcraft.alarm.ex"); |
| } |
| |
| @AfterEach |
| void tearDown() throws NCException, IOException { |
| if (cli != null) |
| cli.close(); |
| |
| NCEmbeddedProbe.stop(); |
| } |
| |
| @Test |
| public void test() throws NCException, IOException { |
| // Should be passed. |
| assertTrue(cli.ask("Ping me in 3 minutes").isOk()); |
| assertTrue(cli.ask("Buzz me in an hour and 15mins").isOk()); |
| assertTrue(cli.ask("Set my alarm for 30s").isOk()); |
| } |
| } |
| </pre> |
| <p> |
| <b>NOTES:</b> |
| </p> |
| <ul> |
| <li> |
| Before each unit test execution on line 8 we create new instance of <code>NCTestClient</code> using |
| client builder with default configuration. On line 10 we open test client connection for model |
| with ID <code>nlpcraft.alarm.ex</code>. |
| </li> |
| <li> |
| After each test execution we close test client connection on line 16. |
| </li> |
| <li> |
| On lines 24-26 we submit ("ask") test sentences to our model and check for succesfull processing. |
| </li> |
| </ul> |
| <div class="bq info"> |
| <p> |
| <b>Embedded Probe</b> |
| </p> |
| <p> |
| Note that this example (lines 6 and 18), as all other examples shipped with NLPCraft, uses |
| <a href="embedded_probe.html">Embedded Probe</a>. |
| </p> |
| <p> |
| Typically, data probes are launched in their own independent JVMs. However, in some |
| cases - e.g. during unit testing - it is more convenient for model implementation or preferable |
| for performance reasons to host a data model (and hence the data probe) in the same JVM. |
| </p> |
| </div> |
| </section> |
| <section id="autotest"> |
| <h2 class="section-title">Auto Model Validation <a href="#"><i class="top-link fas fa-fw fa-angle-double-up"></i></a></h2> |
| <p> |
| The same model from <a href="/examples/alarm_clock.html">Alarm Clock</a> example can be auto validated using |
| <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/tools/test/NCTestAutoModelValidator.html">NCTestAutoModelValidator</a> |
| class and user defined <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/NCIntentSample.html">@NCIntentSample</a> and |
| <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/NCIntentSampleRef.html">@NCIntentSampleRef</a> annotations from the model's callback method: |
| </p> |
| <pre class="brush: java, highlight: [4, 5, 6]"> |
| public class AlarmModel extends NCModelFileAdapter { |
| @NCIntentRef("alarm") |
| @NCIntentSample({ |
| "Ping me in 3 minutes", |
| "Buzz me in an hour and 15mins", |
| "Set my alarm for 30s" |
| }) |
| NCResult onMatch( |
| NCIntentMatch ctx, |
| @NCIntentTerm("nums") List<NCToken> numToks |
| ) { |
| // ... |
| } |
| } |
| </pre> |
| <p> |
| Auto model validator takes one or more model IDs (or class names) and performs validation. Validation consists |
| of starting an <a href="/tools/embedded_probe.html">embedded probe</a> with a given model, scanning for |
| <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/NCIntentSample.html">@NCIntentSample</a> and |
| <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/NCIntentSampleRef.html">@NCIntentSampleRef</a> annotations and their corresponding callback methods, submitting each |
| sample input sentences from these annotations and checking that resulting intent matches the intent the sample was attached to. |
| </p> |
| <h2 class="section-sub-title">Running <a href="#"><i class="top-link fas fa-fw fa-angle-double-up"></i></a></h2> |
| <p> |
| Auto mode validator can be executed in two different ways: |
| </p> |
| <nav> |
| <div class="nav nav-tabs" role="tablist"> |
| <a class="nav-item nav-link active" data-toggle="tab" href="#nav-script" role="tab">NLPCraft CLI</a> |
| <a class="nav-item nav-link" data-toggle="tab" href="#nav-class" role="tab">Java Class <img src="/images/java2-h20.png" alt=""></a> |
| </div> |
| </nav> |
| <div class="tab-content"> |
| <div class="tab-pane fade show active" id="nav-script" role="tabpanel"> |
| <pre class="brush: bash"> |
| $ bin/nlpcraft.sh help --cmd=test-model |
| $ bin/nlpcraft.sh test-model --cp=/path/to/my/model/classes |
| </pre> |
| <p> |
| <b>NOTES:</b> |
| </p> |
| <ul> |
| <li> |
| <a href="/tools/script.html">NLPCraft CLI</a> is available as <code>nlpcraft.sh</code> for |
| <i class="fab fa-fw fa-linux"></i> and <code>nlpcraft.cmd</code> |
| for <i class="fab fa-fw fa-windows"></i>. |
| </li> |
| <li> |
| Run <code class="script">bin/nlpcraft.sh help --cmd=test-model</code> to get a full help on this command. |
| </li> |
| <li> |
| In REPL mode you can use <code>retest-model</code> command to re-run the last mode test with the |
| same parameters avoiding the need to retype all the parameters again. |
| </li> |
| </ul> |
| </div> |
| <div class="tab-pane fade show" id="nav-class" role="tabpanel"> |
| <pre class="brush: bash"> |
| java -cp apache-nlpcraft-incubating-{{site.latest_version}}-all-deps.jar:/path/to/my/model/classes -DNLPCRAFT_TEST_MODELS=org.apache.nlpcraft.examples.alarm.AlarmModel org.apache.nlpcraft.model.tools.test.NCTestAutoModelValidator |
| </pre> |
| <p> |
| <b>NOTES:</b> |
| </p> |
| <ul> |
| <li> |
| Make sure to add necessary classpath components for the model(s) you want to auto-validate. |
| </li> |
| <li> |
| Specify the following three system property (all optional): |
| <table class="gradient-table"> |
| <thead> |
| <tr> |
| <th>System Property</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><code>NLPCRAFT_TEST_MODELS</code></td> |
| <td> |
| <p> |
| Optional system property that should contain comma separate |
| list of the data model classes to test. If not provided, the models from the probe |
| configuration will be used. |
| </p> |
| <p> |
| <b>Example:</b><br/> |
| <code>-DNLPCRAFT_TEST_MODELS=org.apache.nlpcraft.examples.alarm.AlarmModel</code> |
| </p> |
| </td> |
| </tr> |
| <tr> |
| <td><code>NLPCRAFT_PROBE_CONFIG</code></td> |
| <td> |
| <p> |
| Optional system property that should contain the path to probe configuration file. |
| If not provided, the default probe configuration file will be used. |
| </p> |
| <p> |
| <b>Example:</b><br/> |
| <code>-NLPCRAFT_PROBE_CONFIG=/opt/my_probe/probe.conf</code> |
| </p> |
| </td> |
| </tr> |
| <tr> |
| <td><code>NLPCRAFT_TEST_INTENT_IDS</code></td> |
| <td> |
| <p> |
| Optional system property that should contain the comma separated list of |
| intent IDs to test. If not provided, all detected intents will be tested. |
| </p> |
| <p> |
| <b>Example:</b><br/> |
| <code>-NLPCRAFT_TEST_INTENT_IDS=int1,int2</code> |
| </p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </li> |
| <li> |
| You can also use <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/tools/test/NCTestAutoModelValidator.html">NCTestAutoModelValidator</a> class |
| directly to call it programmatically from the code or from IDE. |
| </li> |
| </ul> |
| </div> |
| </div> |
| <p> |
| In the log output you should see the following validation results: |
| </p> |
| <figure> |
| <img class="img-fluid-no-border" src="/images/auto_validation.png" alt=""> |
| <figcaption><b>Fig 1.</b> Model auto-validation result.</figcaption> |
| </figure> |
| <p> |
| See <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/tools/test/NCTestAutoModelValidator.html">NCTestAutoModelValidator</a> |
| class for more details. |
| </p> |
| </section> |
| <section id="examples"> |
| <h2 class="section-title">Examples <a href="#"><i class="top-link fas fa-fw fa-angle-double-up"></i></a></h2> |
| <p> |
| All <a target="github" href="https://github.com/apache/incubator-nlpcraft/tree/master/nlpcraft-examples">examples</a> |
| shipped with NLPCraft come with instructions on how to auto-validate their models. |
| </p> |
| </section> |
| </div> |
| <div class="col-md-2 third-column"> |
| <ul class="side-nav"> |
| <li class="side-nav-title">On This Page</li> |
| <li><a href="#overview">Overview</a></li> |
| <li><a href="#usage">Test client</a></li> |
| <li><a href="#autotest">Auto model validation</a></li> |
| <li><a href="#examples">Examples</a></li> |
| {% include quick-links.html %} |
| </ul> |
| </div> |
| |
| |
| |
| |
| |
| |
