| <!-- |
| |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| |
| --> |
| <html> |
| <body> |
| <h2>Unit Testing</h2> |
| <p> |
| GSF makes unit testing your specific features easy. |
| By extending the <code>GsfTestBase</code> unit test class |
| (which in turn extends <code>NbTestCase</code> which in turn |
| extends JUnit's <code>TestCase</code>), you inherit a number |
| of additional unit testing methods. |
| </p> |
| <p> |
| Unfortunately, you have to write a few configuration steps |
| for your language, once - this is described at the end of |
| this document in the <a href="#configuration">Configuration</a> section. |
| </p> |
| <h3>Unit Test Workflow</h3> |
| <p> |
| All GSF unit tests are based on "golden files". |
| <ul> |
| <li>You have a source file in your language.</li> |
| <li>The source file is parsed and then your language feature |
| is run on the given parse result</li> |
| <li> The result from your feature is pretty printed</li> |
| <li> The pretty-printed output is compared to a previously |
| saved golden file result. If there is a difference, the test fails. |
| </li> |
| <li>Crucially, the FIRST time you run the test, there is no golden file, |
| so one is created instead! This makes it trivial to create new |
| tests. Just add more test files, manually check that your feature |
| is producing the correct result for these test files, and then |
| run them as unit tests. You now have golden files which ensure |
| that for these testcases you get the same feature output every time. |
| </li> |
| </ul> |
| </p> |
| <h3>Example: Semantic Highlighting</h3> |
| <p> |
| To illustrate how this works, I'll use an example. To create |
| a semantic highlighting unit test, create a new test method like this |
| in your SemanticHighlighterTest: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| public void testSemantic1() throws Exception { |
| checkSemantic("testfiles/test1.js"); |
| } |
| </pre> |
| Here, the <code>checkSemantic</code> method is inherited for you from |
| <code>GsfTestBase</code>, and handles testing semantic highlighting code |
| by parsing your testfile (which is specified by a path relative to your |
| <code>test/unit/data/</code> directory in your plugin), then hands it |
| to your semantic highlighter, and then prints and diffs the output. |
| The <code>checkSemantic</code> method will pretty print your source |
| file by inserting markers for the logical attributes you've added superimposed |
| on top of the code. |
| For example, here is <code>test1.js</code>, a simple JavaScript file: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| Draggable.prototype = { |
| scroll: function() { |
| var wind, window2, y = 1, win3 = 50; |
| var windo |
| if(this.scroll == window) { |
| } |
| } |
| } |
| </pre> |
| When I run the above test the first time, it fails, along with this error |
| message: "<code>Please re-run the test</code>". |
| If you now run the test again, it will pass, and will continue to pass |
| until you the output of your semantic highlighter changes for this input. |
| If we look in the unit test directory, there is a new file right next |
| to the testfile with the extra extension <code>.semantic</code>: |
| <code>test1.js.semantic</code>. |
| Here's how that file looks: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| |>CLASS:Draggable<|.prototype = { |
| |>METHOD:scroll<|: function() { |
| var |>UNUSED:wind<|, |>UNUSED:window2<|, |>UNUSED:y<| = 1, |>UNUSED:win3<| = 50; |
| var |>UNUSED:windo<| |
| if(this.scroll == |>GLOBAL:window<|) { |
| } |
| } |
| } |
| </pre> |
| |
| Notice how this is identical to the input file, but with |>MARKERS:<| inserted around |
| sections of code that indicate what kind of highlighting attribute is associated with |
| that span. It should be trivial to look at this golden file and convince yourself |
| that the semantic highlighting worked correctly here. |
| </p> |
| <p> |
| Repeat this process for all the different scenarios you want to test your semantic |
| highlighter for. |
| </p> |
| <p> |
| Occasionally, you get a unit test failure. Perhaps you've accidentally |
| broken your code - or perhaps you've actually fixed a bug, which changes the |
| output of your semantic highlighter in some way, so your golden file diff |
| fails and the test files. There is a textual diff printed to the output when |
| you run the test in the IDE, but personally I find it hard to read. |
| Your tests, and your golden files, should all be checked in with your code, |
| and therefore version controlled. So what I do in this scenario is just |
| remove all the golden files (<code>rm *.semantic</code> for example), |
| and run the tests again. I then use the builtin IDE diff tool to diff |
| all the golden files and see what the changes are. Sometimes, the changes |
| are an improvement, and I want the goldenfile updated (which it now has been!) |
| to reflect the new behavior. And in some cases, the change is a bad one, |
| and I have a regression on my hands, so I revert the file change, |
| and start debugging to track down the problem. |
| Here's an example of diffing golden files, where in this case there is |
| definitely a problem - the highlighter is no longer identifying regular |
| expressions in JavaScript (which has to be done at the parse level rather |
| than at the lexical level in JavaScript): |
| <br/> |
| <img src="test-diff.png" /> |
| <br/> |
| <br/> |
| </p> |
| <h3>Caret-based Tests</h3> |
| <p> |
| Many GSF features are associated with a caret position. For example, |
| code completion applies at a particular offset, and ditto for |
| mark occurrences. |
| </p> |
| <p> |
| The convention in GSF is to tell unit tests about a caret position |
| by passing in a substring of the testfile, with an extra <code>^</code> |
| (arrow up) indicating the caret position. For example, in the |
| above semantic highlighting code fragment, if I wanted to identify |
| a caret position between the "f" and the "u" of the function keyword |
| on the second line, I could pass in this string: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| "scroll: f<b>^</b>unction" |
| </pre> |
| Note that this isn't a complete line of the source - it can be just a |
| portion of a line. The key part is that it tells GSF where the caret |
| offset should be in the file, without having to tell it something |
| awkward like an integer offset. Unit tests will assert if you're |
| passing in a caret offset and it cannot find the corresponding position |
| in the source file. |
| </p> |
| <p> |
| Here's an example of a unit test which uses this: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| checkOccurrences("testfiles/semantic4.js", "function myfunc(param1, pa^ram2) {", true); |
| </pre> |
| This is the test for the Mark Occurrences feature. We're telling it |
| which source file we want to test, <b>and</b> where the caret is. <b>Your</b> |
| occurrences marker will be asked to compute the occurrences corresponding |
| to the given offset. If we run this test, it will compute all the occurrences |
| and pretty print them by listing ONLY the lines in the file that contain |
| an occurrence, and showing that occurrence in context (e.g. the source |
| code with the region containing the marker). Here's the occurrences output |
| for another test file: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| Spry.Effect.Registry.prototype.getRegisteredEffect = function(|>MARK_OCCURRENCES:elem^ent<|, effect) |
| var eleIdx = this.getIndexOfElement(|>MARK_OCCURRENCES:element<|); |
| var addedElement = new Spry.Effect.AnimatedElement(|>MARK_OCCURRENCES:element<|); |
| </pre> |
| |
| </p> |
| <h3>Code Completion Tests</h3> |
| <p> |
| Here's an example of a code completion test. Again, we point to some test |
| file, as well as a caret position - in this case, it's at the beginning |
| of the symbol "escape", so there's no prefix: |
| |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| public void testExpression1() throws Exception { |
| checkCompletion("testfiles/completion/lib/expressions.js", "^escape", false); |
| } |
| </pre> |
| This will generate (and then diff as usual) the following code completion output: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| Results for escape with queryType=COMPLETION and nameKind=PREFIX |
| METHOD exec(str) : void RegExp |
| METHOD test(str) : Boolean RegExp |
| METHOD toSource() : void RegExp |
| METHOD toString() : String RegExp |
| METHOD watch(prop, handler) : void RegExp |
| PROPERTY constructor RegExp |
| PROPERTY global : Boolean RegExp |
| PROPERTY ignoreCase : Boolean RegExp |
| PROPERTY lastIndex : Number RegExp |
| PROPERTY multiline RegExp |
| PROPERTY prototype RegExp |
| PROPERTY source : String RegExp |
| ------------------------------------ |
| METHOD eval(string) : void Object |
| METHOD hasOwnProperty(prop) : Boolean Object |
| METHOD isPrototypeOf(object) : Boolea Object |
| METHOD propertyIsEnumerable(prop) : B Object |
| METHOD toLocaleString() : String Object |
| </pre> |
| The divider line here is showing the difference between the smart |
| items and the non-smart items (just like the dividing line you see |
| in the IDE). |
| </p> |
| <p> |
| As you can see, writing code completion tests is pretty trivial. |
| </p> |
| <h3>Other Features</h3> |
| <p> |
| There are many other feature tests like this, and all are very |
| similar to the above: a single line call with test data which |
| gets pretty printed and diffed. For example, |
| the <b>insertChar</b> test (and removeChar and insertBreak etc) let |
| you test what happens when you type keys in the editor. You provide |
| the document contents and caret position before, the character to be |
| typed, and the contents and caret position after the insert, like this: |
| <pre> |
| insertChar("x = ^", '\'', "x = '^'"); |
| </pre> |
| This test ensures that bracket matching works correctly for the apostrophy |
| for example. You should obviously write lots of other tests for this to |
| make sure that bracket matching doesn't happen inside strings and |
| comments, when there is already a bracket there, etc: |
| <pre> |
| insertChar("x = '^'", '\'', "x = ''^"); |
| insertChar("// Hello^", '\'', "// Hello'^"); |
| <i>etc.</i> |
| </pre> |
| There are unit tests for all kinds of features. Formatting. Keystroke |
| handling. Quickfixes. Instant Rename. Indexing. Code Folding. Navigation. |
| </p> |
| <h3>AST Offset Tests</h3> |
| <p> |
| Unlike the feature tests, the AST offset test support involves a bit |
| more work. That's because GSF does not have any clue about |
| <b>your</b> AST. GSF doesn't even know about it; you typically stash |
| it in your own subclass of the <code>ParserResult</code> class. |
| However, GSF makes it easier to make sure that your AST offsets |
| are correct, which is typically vital for correct IDE operation. |
| To test your own ASTs, you need to override a couple of protected |
| methods in the GsfTestBase: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| protected String <b>describeNode</b>(CompilationInfo info, Object node, boolean includePath); |
| protected void <b>initializeNodes</b>(CompilationInfo info, List validNodes, |
| Map<Object,OffsetRange> positions, List invalidNodes); |
| </pre> |
| </p> |
| <p> |
| The first method just lets you create a description of the node |
| (which will show up in the AST golden file). Typically there will be some |
| AST specific way to identify node types - for example, STRING, EXPRESSION, |
| FUNCTION, etc. |
| </p> |
| <p> |
| In the second method, you need to go and pull ALL nodes out of the |
| AST for your parser result, and put all of them into a map, where |
| each node maps to a given OffsetRange (the start/end offsets) of your |
| nodes that you are trying to test. You can ignore the other offsets |
| for now. Once you do that, you can write tests like this: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| public void testOffsets1() throws Exception { |
| checkOffsets("testfiles/test1.js"); |
| } |
| </pre> |
| This is the <code>test1.js</code> file we saw for semantic highlighting above. |
| Here the golden file will be named <code>test1.js.offsets</code> (as opposed |
| to the <code>.semantic</code> golden file we get from the semantic test). |
| The golden file will use XML-like syntax to show the node descriptions around |
| the begin/end offsets for your nested AST hierarchy that corresponds to |
| your source: |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| <SCRIPT><EXPR_RESULT><SETPROP><NAME><b>Draggable</b></NAME><b>.</b><STRING><b>prototype</b></STRING><b> = </b><OBJECTLIT><b>{</b> |
| <OBJLITNAME><b>scroll</b></OBJLITNAME><b>: </b><FUNCTION><b>function() </b><BLOCK><b>{</b> |
| <VAR>var <NAME>wind</NAME>, <NAME>window2</NAME>, <NAME>y = <NUMBER>1</NUMBER></NAME>, <NAME>win3 = <NUMBER>50</NUMBER></NAME></VAR>; |
| <VAR>var <NAME>windo</NAME></VAR> |
| <BLOCK>if(<IFNE><EQ><GETPROP><THIS>this</THIS>.<STRING>scroll</STRING></GETPROP> == <NAME>window</NAME></EQ></IFNE>) <BLOCK>{ |
| }<TARGET></TARGET></BLOCK></BLOCK> <RETURN></RETURN></BLOCK>}</FUNCTION> |
| }</OBJECTLIT></SETPROP></EXPR_RESULT></SCRIPT> |
| </pre> |
| I've bolded the beginning of the embedded code here so you can see the code |
| through the AST node markup. |
| </p> |
| <p> |
| I have made a lot of diffs to the Rhino parser for the JavaScript support to add in |
| AST offset support (Rhino doesn't record source offsets, we've added that for NetBeans). |
| Having tests like these makes it a lot easier to feel confident when I make any changes to the |
| parser. As shown earlier, if there are diffs, I delete the offset files, run the tests, and |
| then diff the golden files through the IDE. |
| </p> |
| |
| <a name="Configuration"/> |
| <h3>Configuration</h3> |
| <p> |
| Typically, you'll want to create your own <code>TestBase</code> class, |
| which extends <code>GsfTestBase</code>, and then make all your unit tests |
| extend your own <code>TestBase</code>. In your superclass, you need to |
| do a few configuration steps. |
| </p> |
| |
| <p> |
| <blockquote style="background-color: #ffdddd; color: black; padding: 20px; border: solid 1px black"> |
| <b>These steps should not be necessary</b>. Unfortunately, I haven't |
| been able to get the GSF services working without them. |
| What I'm observing is that from a unit test, services that are |
| registered via the layer file systems don't work. I've been playing |
| around with module dependencies, Mock services etc. but I haven't |
| gotten it to work. So, for example, for some reason from a unit test, |
| the GSF Data Loader isn't used, and attempting to open documents give |
| me simple PlainDocuments instead of BaseDocuments. |
| I would <b>really</b> appreciate some help from people who know more |
| about unit testing and netbeans service loading to help me track |
| this down so that these manual steps aren't necessary. |
| </blockquote> |
| </p> |
| <p> |
| Here's the code you have to add to your own test class such that all |
| your unit tests inherit it. |
| <pre style="background: #ffffcc; color: black; border: solid 1px black; padding: 5px"> |
| TODO - document this. For now, anyone curious, go look at |
| <code>javascript.editing/test/unit/src/org/netbeans/modules/javascript/editing/JsTestBase.java</code> |
| </pre> |
| </p> |
| <br/> |
| <span style="color: #cccccc">Tor Norbye <tor@netbeans.org></span> |
| </body> |
| </html> |
