| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to you under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> |
| <html> |
| <head> |
| <meta http-equiv="CONTENT-TYPE" |
| content="text/html; charset=windows-1252"> |
| <title>readme.htm</title> |
| <meta name="GENERATOR" content="OpenOffice.org 1.1.4 (Win32)"> |
| <meta name="CREATED" content="20051031;9095275"> |
| <meta name="CHANGED" content="20051031;9274086"> |
| </head> |
| <body dir="ltr" lang="en-US"> |
| <h1><a name="mozTocId934928"></a>Derby Functional Tests</h1> |
| <h2><a name="mozTocId504000"></a>Package: org.apache.derbyTesting<!--mozToc h1 1 h2 2 h3 3 h4 4 h5 5 h6 6--></h2> |
| <p><font size="2"><br> |
| </font></p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#migrate">0. Migration to JUnit</a> |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#intro">1. Introduction</a> |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#quickstart">2. Quickstart</a> |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a |
| href="#2.1_running_with_derby_classes_">2.1 running tests</a></p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a |
| href="#building_derbyTesting__running_with">2.2 building derbyTesting |
| package</a></p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#run">3. More details on |
| running the derby functional tests</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#run1">3.1 Running 1 test</a> |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#run2">3.2 Running suites |
| of tests</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#overview">4. Harness |
| internals for developers</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#ov1">4.1 Test types</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#ov2">4.2 Supporting files |
| for tests</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#ov3">4.3 |
| <testname>_app.properties</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#ov4">4.4 |
| <testname>_derby.properties</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#ov5">4.5 tmp files, out |
| files, master files, and canons</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#ov6">4.6 Masking and |
| comparing</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#Adding_a_new_test">4.7 |
| Adding a new test</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a |
| href="#4.8_Suites_and_Adding_a_new_suite">4.8 Suites and adding a new |
| suite</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#ov9">4.9 Running with a |
| new jvm</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#skipping">4.10 Skipping a |
| test</a> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><a href="#frameworks">4.11 Frameworks</a> |
| </p> |
| </li> |
| <li> |
| <p><a href="#props">4.12 Some test harness properties</a> </p> |
| </li> |
| <li> |
| <p><a href="#security">4.13 SecurityManager testing by default</a></p> |
| </li> |
| <li><a href="#hostName">4.14 Testing with Network Server on a remote |
| host</a></li> |
| <li> |
| <p><a href="#4.15_Encoding_issues">4.15 Encoding Issues</a></p> |
| </li> |
| <li> |
| <p><a |
| href="./org/apache/derbyTesting/functionTests/tests/junitTests/compatibility/README.html">5. |
| Compatibility Tests</a></p> |
| </li> |
| <li> |
| <p><a |
| href="./org/apache/derbyTesting/functionTests/tests/jdbc4/README.html">6. |
| JDBC4 Tests</a></p> |
| </li> |
| </ul> |
| <p><br> |
| <br> |
| </p> |
| <h2><a name="migrate">0. Migration to JUnit</h2> |
| In the trunk codeline (10.3 and later) Derby is migrating testing using JUnit. |
| All new tests are being written using JUnit and a number of the older |
| harness based tests have been converted. |
| <P> |
| Since the conversion is ongoing the current JUnit state is not represented |
| in this file, sections 1 onwards apply to running tests using the |
| old harness which still applies to a significant number of tests. |
| Currently to run the complete set of tests two runs are required, |
| run the harness based tests decribed in this document and then |
| running all the JUnit bases tests. |
| <P> |
| How to run the JUnit based tests is currently being maintained |
| on Derby's wiki at: |
| <BR> |
| <a href="http://wiki.apache.org/db-derby/DerbyJUnitTesting">http://wiki.apache.org/db-derby/DerbyJUnitTesting</a> |
| <BR> |
| The wiki also includes information about writing new JUnit tests |
| and conversion of existing tests. |
| <P> |
| Please also consult <a href="http://wiki.apache.org/db-derby/DerbyTesting">http://wiki.apache.org/db-derby/DerbyTesting</a> for more information on Derby testing. |
| <h2><a name="intro"></a>1. Introduction</h2> |
| <p>This document describes functionality of the derby functional |
| testing package org.apache.derbyTesting. This package is based on the |
| functional tests in use at IBM for testing the Cloudscape product |
| before its contribution to ASF.</p> |
| <p>In the following, instructions are geared towards a unix |
| environment. For other environments, some details may need to be |
| adjusted. For instance, the document may refer to $ANT_HOME, for DOS, |
| this would be %ANT_HOME%.</p> |
| <p>In the following the top directory under which the subversion tree |
| is placed is referred to as ${derby.source} - see also the derby |
| <a |
| href="http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.html">BUILDING.html</a>.</p> |
| <p>The version of the classes and supporting files of the |
| derbyTesting package have to match the version of the classes of the |
| derby package. Thus you either need to build all jars yourself, or |
| get all jar files from the Derby site at the same time when |
| available. </p> |
| <h2><a name="mozTocId191589"></a><a name="quickstart"></a>2. |
| QuickStart</h2> |
| <h3><a name="2.1_running_with_derby_classes_"></a>2.1 running tests</h3> |
| <p>The derbyTesting package enables you to run 1 test or a suite of |
| tests. Before you can run, you need to setup your environment:</p> |
| <ul> |
| <li> |
| <p>Obtain a jdk or jre (based on jdk 1.4 specification or higher). Add the |
| bin directory to your $PATH. Currently supported with regards to the tests are:</p> |
| </li> |
| </ul> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p> |
| <font size="2">jdk141 - Sun HotSpot jdk1.4.1</font><br> |
| <font size="2">jdk142 - Sun HotSpot jdk1.4.2</font><br> |
| <font size="2">jdk15 - Sun HotSpot jdk1.5</font><br> |
| <font size="2">jdk16 - Sun HotSpot jdk1.6</font><br> |
| <font size="2">jdk17 - Sun HotSpot jdk1.7</font><br> |
| <font size="2">ibm141 - IBM Classic jdk1.4.1</font><br> |
| <font size="2">ibm142 - IBM Classic jdk1.4.2</font><br> |
| <font size="2">ibm15 - IBM Classic jdk1.5</font><br> |
| <font size="2">ibm16 - IBM J9 VM jdk1.6</font><br> |
| <font size="2">ibm17 - IBM J9 VM jdk1.7</font><br> |
| <font size="2">j9_foundation11 - IBM weme jvm (available |
| with IBM Websphere EveryPlace Micro Edition, 6.2), version 2.4, j2ME 1.1 </font> </p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">cd into a directory that does not |
| have any colons or spaces in it.</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">set $CLASSPATH to include the |
| following jars: </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;"><font size="2">derbyTesting.jar<br> |
| </font> <font size="2">test files and |
| classes</font></p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><font size="2">derbyrun.jar<br> |
| </font> <font size="2">executable jar file |
| for tools like ij and dblook (this jar file will automatically pull |
| other required jar files - derby.jar, derbynet.jar, derbyclient.jar |
| and derbyLocale_*.jar - into your classpath) </font></p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"><font size="2"> db2jcc.jar and |
| db2jcc_license_c.jar <br> |
| <font size="2">IBM Universal JDBC Driver classes. (See IBM <a |
| href="http://www-106.ibm.com/developerworks/db2/downloads/jcc/">developerworks</a> |
| for download)</font> <br> |
| <b>These jars are optional. The tests using the IBM Universal |
| JDBC driver are not run if these jar files are not present in your |
| classpath</b> </font> </p> |
| </li> |
| <li> |
| <p><font size="2">junit.jar <br> |
| assertion-based test machinery.</font></p> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| <p>For example:</p> |
| <dl> |
| <dd> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="2">(note that $jardir is only a convenience |
| variable that points to the directory containing the Derby jar files, |
| and $tstjardir ia a convenience variable that points to the directory |
| containing the jar file for JUnit):<br> |
| set jardir=/local/derbyjar<br> |
| set tstjardir=/local/testingjars<br> |
| set |
| CLASSPATH="$jardir/derbyrun.jar:$jardir/derbyTesting.jar:$tstjardir/junit.jar"</font><br> |
| <font size="2">set PATH=/local/jdk141/bin:$PATH</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd> |
| </dl> |
| <p>To run 1 test: </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p>syntax:<br> |
| <font size="2">java -D<testproperty> |
| org.apache.derbyTesting.functionTests.harness.RunTest |
| <testdir>/<testname></font><br> |
| <font size="2">where </font> </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;"> <font size="2"><testproperty> |
| are test specific properties, such as 'framework' for the RunTest |
| class. </font> </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"> <font size="2"><testdir> |
| is one of the directories under functionTests/tests where the actual |
| test is located</font> </p> |
| </li> |
| <li> |
| <p> <font size="2"><testname> is the |
| actual name of the test</font> </p> |
| </li> |
| </ul> |
| <p><font size="2">examples:<br> |
| to run the test supersimple against the embedded driver:<br> |
| </font> <font size="2">java |
| org.apache.derbyTesting.functionTests.harness.RunTest |
| lang/supersimple.sql<br> |
| <br> |
| To run a test with network server, using the derbyclient driver, add |
| -Dframework=DerbyNetClient to the run. The test harness will to start |
| network server at port 1527 or connect to a running one, run the test, |
| and stop network server thereafter.<br> |
| for example:<br> |
| java -Dframework=DerbyNetClient |
| org.apache.derbyTesting.functionTests.harness.RunTest |
| lang/supersimple.sql</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>A successful run will have a .pass file, and the output to the |
| console will show no difference between expected and actual test |
| result. A failed test run will have at least a .fail file and the |
| output to the console will show the difference between expected and |
| actual result.</p> |
| <p>To run a suite: </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p>syntax:<br> |
| <font size="2">java -D<testproperty> |
| org.apache.derbyTesting.functionTests.harness.RunSuite |
| <testsuite></font><br> |
| <font size="2">where </font> </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;"> <font size="2"><testproperty> |
| are test specific properties, such as 'verbose' for the RunSuite class. |
| </font> </p> |
| </li> |
| <li> |
| <p> <font size="2"><testsuite> is one of |
| the suites under org/apache/derbyTesting/functionTests/suites</font> </p> |
| </li> |
| </ul> |
| <p><font size="2">for example for running the suite |
| derbylang:<br> |
| </font> <font size="2">java -Dverbose=true |
| org.apache.derbyTesting.functionTests.harness.RunSuite derbylang</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>Each suite run should be started in a clean directory. The test |
| output directory will not be emptied out before testing is begun, |
| although individual test files and result files will be cleaned out |
| and overwritten. </p> |
| <p>The suites provided are: </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">derbylang: </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">basic functionality of |
| language implementation in derby. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">Mostly .sql type tests. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">tested on a variety of hardware |
| takes from 1.15m to 2.00 hours</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">derbynetclientmats </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">basic network server tests using |
| the derby client</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">variety of tests, including some |
| from derbylang suite </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">tested on a variety of hardware |
| takes from 15 to 30 minutes </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">derbynetmats </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">basic network server tests using |
| the IBM Universal JDBC driver</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">variety of tests, including some |
| from derbylang suite </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">tested on a variety of hardware |
| takes from 15 to 30 minutes </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">propertyinfo </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">runs test to get property |
| information</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">storeall </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests for storage area </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">includes: </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">storemats: most basic quick |
| verification tests.</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">storemore: more extensive |
| storage tests </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">storetests: set of store |
| tests grouped together because they do not each need to create a new |
| database </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">tested on a variety of hardware |
| takes from 25 to 50 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">xa </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests the xa implementation. |
| There is both a storage and language element to these tests </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">tested on a variety of hardware |
| takes from 2 to 4 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">storeunit </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests store-related unit tests. |
| Runs from 8 to 15 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">unit </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests 4 general functionality |
| unit tests. runs from 5 to 10 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">jdbcapi </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests implementation of jdbc api |
| such as Connection class implementation, Metadata etc. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">takes from 20 to 40 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">jdbc20</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests implementation of features |
| from the jdbc 20 specification </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">takes 2 to 5 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">jdbc4 </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests implementation of jdbc 4.0 |
| </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">jdk14 </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests implementation of features |
| from the jdk14 specification </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">takes 2 to 5 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">demo, simpledemo</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests the SimpleApp example </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">simpledemo runs SimpleApp itself |
| - and thus has a different default resource package name (namely, no |
| package) than all the other tests. Hence it needed its own |
| suite.properties file. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">takes 30 to 1 minute </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">nist </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">test obtained from the NIST SQL |
| suite v 6.0 </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">takes 5 to 10 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">encryptionAll </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">takes 30 to 55 minutes </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">runs a few encryption tests plus |
| the following encryption tests suites</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">encryption </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">runs the storemats, |
| sysinfo and multi suites in encryption scheme DESede</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">takes 25 to 40 minutes </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">encryptionAES - tests AES |
| encryption scheme</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">encryptionBlowfish - tests |
| Blowfish encryption scheme</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">encryptionCFB - tests CFB |
| encryption scheme</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">encryptionDES - tests DES |
| encryption scheme</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">encryptionECB - tests ECB |
| encryption scheme</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">encryptionOFB - tests OFB |
| encryption scheme </p> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">multi </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">runs a simple test case with 10 |
| threads </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">runs for 10 minutes, then shuts |
| down all threads</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">derbytools</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests for dblook, ij, and |
| import/export utilities </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">takes 5 to 10 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">i18nTest </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">tests that characters outside |
| simple ascii scope do not result in errors. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">takes 5 to 10 minutes</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p>encodingTests</p> |
| </li> |
| <ul> |
| <li> |
| <p>runs tests with derbyTesting=UTF-16. </p> |
| </li> |
| <li> |
| <p>takes 2 to 5 minutes</p> |
| </li> |
| </ul> |
| <li> |
| <p style="margin-bottom: 0in;">derbyall </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">contains all suites typically |
| run by all developers</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">tested on a variety of hardware |
| takes from 3.00 - 6.00 hours </p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">largeData</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">Contains tests that deal with |
| large amounts of data and thus require more machine resources. |
| This suite is NOT run as part of 'derbyall' because the tests it |
| contains require either 1) more machine resources than what the typical |
| Derby developer might have, and/or 2) a significant amount of time to |
| run, and thus shouldn't be run every night.</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">As tests are added to this |
| quite, it could require more and more time to run (several minutes to |
| several hours to several days), which is why it is NOT included as part |
| of the derbyall suite. Currently the largedata/LobLimits.java test |
| takes about 17Gb of disk space. On a linux machine with 2.8Ghz Intel |
| Xeon CPU, 4Gb RAM , Linux machine and IBM 1.4.2 JVM with default memory |
| heap size, the test ran for about 4.5 hrs. If the test is successful, |
| it will cleanup the database and other files.</p> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p><a href="#Note2:"><font size="2">See Note2</font></a></p> |
| </li> |
| </ul> |
| <p>A successful run with all tests passing will have no *.fail files |
| created, the <testsuite>_fail.txt file will be empty, and the |
| <testsuite>_report.txt file will show no failures in the |
| Summary results section. </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="2">-------snippet from derbylang_report.txt -----<br> |
| -----------------------------------------------------------<br> |
| Summary results:<br> |
| <br> |
| Test Run Started: 2004-11-10 11:27:55.0<br> |
| Test Run Duration: 00:04:09<br> |
| <br> |
| 129 Tests Run<br> |
| 100% Pass (129 tests passed)<br> |
| 0% Fail (0 tests failed)<br> |
| 0 Suites skipped</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p><br> |
| <br> |
| </p> |
| <h3><a name="building_derbyTesting__running_with"></a>2.2 building |
| derbyTesting package</h3> |
| <p>To build the derbyTesting package:</p> |
| <ul> |
| <li> |
| <p>follow all the steps in the derby <a |
| href="http://svn.apache.org/repos/asf/db/derby/code/trunk/BUILDING.html">BUILDING.html</a>. |
| </p> |
| </li> |
| </ul> |
| <p>This is some typical output for the ant build process.</p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="2">> cd /local/derby/java/testing<br> |
| > ant.ksh<br> |
| Searching for build.xml ...<br> |
| Buildfile: /local/derby/java/testing/build.xml<br> |
| <br> |
| compile:<br> |
| [javac] Compiling 30 source files to |
| /local/derby/classes<br> |
| ...<br> |
| [copy] Copying 1 file to |
| /local/derby/classes/org/apache/derbyTesting/funct<br> |
| ionTests<br> |
| <br> |
| BUILD SUCCESSFUL<br> |
| Total time: 10 minutes 3 seconds</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>Building using the ant all target places all files, that is, |
| classes, but also supporting files such as expected output (*.out), |
| sql test files (*.sql), properties files and any data files used in |
| individual tests into the classes directory so they can all be found |
| using the CLASSPATH. </p> |
| <p>Once you have built the derbyTesting package, you can make a |
| derbyTesting.jar using the jar build target at the |
| ${derby.source}level. </p> |
| <p>This will look something like: </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="2">c:> ant derbytestingjar<br> |
| Searching for build.xml ...<br> |
| Buildfile: C:\derby\build.xml<br> |
| <br> |
| initjars:<br> |
| [mkdir] Created dir: C:\derby\jars\<br> |
| [mkdir] Created dir: C:\derby\jars\lists<br> |
| [echo] Revision number set to exported<br> |
| [echo] .<br> |
| <br> |
| derbytestingjar:<br> |
| [echo] Beginning derbytesting.jar build<br> |
| .....<br> |
| BUILD SUCCESSFULL</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p><br> |
| <br> |
| </p> |
| <h2><a name="mozTocId582299"></a><a name="run"></a>3. More details on |
| running the derby functional tests</h2> |
| <p>The functional tests are run using a class called 'RunTest'. This |
| class calls a number of other classes. A group of tests, called a |
| 'suite' is executed using a class called 'RunSuite'.</p> |
| <h3><a name="mozTocId595945"></a><a name="run1"></a>3.1 Running 1 |
| test</h3> |
| <p>See section 2.1 for the basic steps to run 1 test. </p> |
| <p>To pass on system level properties to the test harness, use the |
| test harness property -DtestSpecialProps. For example, to ensure |
| extra information is appended to the log: </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="2">java -Dframework=DerbyNetClient |
| -DtestSpecialProps=derby.infolog.append=true |
| org.apache.derbyTesting.functionTests.harness.RunTest |
| lang/supersimple.sql</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p><br> |
| Tests will be executed in the current directory. When running |
| a test using the network server and derby client, i.e. |
| -Dframework=DerbyNetClient, the test will run in a subdirectory |
| (automatically created) 'DerbyNetClient'. </p> |
| <p>The test will normally create the following:</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">a database. The default name is |
| 'wombat'. However, the name may be different depending on certain |
| properties passed in to the test harness. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">a .out file: the final result file </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">a .tmp file; the initial result |
| file, before any modification to prevent irrelevant differences has |
| been applied (before 'masking'). </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">a .diff file; the differences |
| between the .out and the master file with expected output it is |
| compared to. </p> |
| </li> |
| <li> |
| <p>a .pass or .fail file. This file lists the test if it passes |
| under .pass, and under .fail if the output in .out is different from |
| the expected output in the master.</p> |
| </li> |
| <li> |
| <p>.tmpmstr file. This file is a copy of the master file created in |
| local encoding, and in the case of networkserver, massaged to eliminate |
| irrelevant differences. This is the file the .out file is compared with.</p> |
| </li> |
| </ul> |
| <p>possibly created:</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">additional files used in a specific |
| test may get copied over to the test directory. These normally do not |
| get cleaned up. </p> |
| </li> |
| <li> |
| <p>.err and .out files in network server database files for any |
| additional error output.</p> |
| </li> |
| <li> |
| <p>one or more database directories</p> |
| </li> |
| <li> |
| <p>.utf8out file - see section on <a href="#4.15_Encoding_issues">Encoding</a> |
| Issues.<br> |
| </p> |
| </li> |
| </ul> |
| <p>When the test is successful, cleanup will occur unless the test |
| harness property -Dkeepfiles=true is used. Cleanup will attempt to |
| cleanup all files except for .pass. <font size="2"><br> |
| <a href="#Note2:">See |
| Note2.</a></font> </p> |
| <p>A successful run (this example is from a dos environment) would |
| look for instance like: </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="2">c:> |
| derbyTesting.functionTests.harness.RunTest lang/supersimple.sql<br> |
| C:\derby\run2<br> |
| supersimple<br> |
| -- listing properties --<br> |
| derby.locks.deadlockTimeout=3<br> |
| derby.locks.waitTimeout=3<br> |
| *** Start: supersimple jdk1.4.2_03 2004-11-10 16:51:02 ***<br> |
| The test should be running...<br> |
| MasterFileName = master/supersimple.out<br> |
| *** End: supersimple jdk1.4.2_03 2004-11-10 16:51:25 ***</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p><br> |
| <br> |
| </p> |
| <p>A Test Failure shows the diff, creates a .fail file, does not |
| create a .pass file, and does not cleanup any files upon completion. |
| The output might look like this:</p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p> <font size="2">c:> |
| derbyTesting.functionTests.harness.RunTest lang/supersimple.sql<br> |
| C:\derby\run2<br> |
| supersimple<br> |
| -- listing properties --<br> |
| derby.locks.deadlockTimeout=3<br> |
| derby.locks.waitTimeout=3<br> |
| *** Start: supersimple jdk1.4.2_03 2004-11-10 16:54:39 ***<br> |
| The test should be running...<br> |
| MasterFileName = master/supersimple.out<br> |
| 10 del<br> |
| < 10<br> |
| 10a10<br> |
| > 1<br> |
| Test Failed.<br> |
| *** End: supersimple jdk1.4.2_03 2004-11-10 16:55:02 ***</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p><br> |
| <br> |
| </p> |
| <h3><a name="mozTocId368566"></a><a name="run2"></a>3.2 Running a |
| suite of tests</h3> |
| <p>See section 2.1 for a basic explanation on how to run a suite of |
| tests.</p> |
| <p>Tests will be run in a subdirectory with the name of the test |
| suite under the current directory. Eg. for derbylang suite, a |
| directory derbylang will be created. While the tests are run, |
| information about the run is inserted into a <testsuite>.sum |
| file. When all tests have completed summary files are created |
| <testsuite>_pass.txt, _fail.txt, and _diff.txt files are |
| created as well as a <testsuite>_report.txt with additional |
| details. Some of the information is duplicate. Also, a .skip file |
| will be created holding a list of the tests that were skipped (for |
| more details on this, see the section on <a href="#skipping">skipping |
| tests</a>). </p> |
| <p>RunSuite does not empty the top level directory before running. |
| Thus, if another suite was run in the same directory at an earlier |
| time, the resulting summary files might contain results for more than |
| the current run. Therefore it is important to run each suite in a |
| clean directory. </p> |
| <p>Sample output from RunSuite:</p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="1">c:> $ java |
| org.apache.derbyTesting.functionTests.harness.RunSuite derbylang<br> |
| Top suite: derbylang<br> |
| Suite to run: derbylang:derbylang<br> |
| Now do RunList<br> |
| Now run the suite's tests<br> |
| Run the tests...<br> |
| Execute command: java -DjavaCmd=java |
| -Doutputdir=C:\derbyt1\derbylang\derbylang |
| -Dtopsuitedir=C:\derbyt1\derbylang -Dtopreportdir=C:\derbyt1\derbylang |
| -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang |
| -Dtopsuitename=derbylang |
| org.apache.derbyTesting.functionTests.harness.RunTest |
| lang/arithmetic.sql<br> |
| ...(.more tests)....<br> |
| Generated report: derbylang_report.txt</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>This output does not show whether the tests passed or failed. The |
| Summary section in <testsuite>_report.txt shows the statistics |
| of the passed vs. failed tests, the summary <testsuite>_*.txt |
| files list the tests that passed and failed. </p> |
| <p><br> |
| <br> |
| </p> |
| <h2><a name="mozTocId635355"></a><a name="overview"></a>4. Harness |
| internals for developers</h2> |
| <p>The following is intended for people who have the subversion tree |
| available and want to add or modify tests. </p> |
| <p>The test harness executing one test basically does the following |
| in sequence: </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">identify test to run </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">identify properties to run with </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">copy needed support files </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">find the expected output </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">if network server, start network |
| server </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">run the test, creating the database </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">if network server, shutdown the |
| server </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">modify the output based on Sed class |
| and _sed.properties file for the test </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">compare expected output with actual |
| output </p> |
| </li> |
| <li> |
| <p>if pass, cleanup. </p> |
| </li> |
| </ul> |
| <p><br> |
| <br> |
| </p> |
| <h3><a name="mozTocId344499"></a><a name="ov1"></a>4.1 Test types</h3> |
| <p>The test harness recognizes, or will recognize tests with the |
| following extensions:</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;"> .java tests |
| that run in a separate jvm. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"> .sql tests |
| that run using ij </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"> .sql2 |
| related to .sql </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;"> .multi |
| multi threaded tests. There is currently only 1 test being run. The |
| multi test functions a little differently from .java and .sql* tests in |
| that RunTest starts a separate harness class called MultiTest to |
| control the details of the run. Also, the actual test files live under |
| org/apache/derbyTesting/functionTests/multi/stress, rather than |
| org/apache/derbyTesting/functionTests/tests. </p> |
| </li> |
| <li> |
| <p> .unit unit tests. The unit tests |
| actually refer to <testname>_derby.properties files under |
| org/apache/derbyTesting/functionTests/tests/unit that activate the |
| actual unit test harness and tests under |
| org/apache/derbyTesting/unitTests. These tests test more underlying |
| functionality than the (rest of the) functionTests, which are more |
| geared toward how end-users might use functionality.</p> |
| </li> |
| <li> |
| <p>.junit junit tests. These require |
| junit.jar to run and are actually run with junit.textui.TestRunner. The |
| actual tests are .java files.<br> |
| </p> |
| </li> |
| </ul> |
| <h3><a name="mozTocId809770"></a><a name="ov2"></a>4.2 Supporting |
| files for tests</h3> |
| <p>Various additional files may be used by a test, for instance, to |
| create large data values, to test using of jar files and the like. |
| Any files that need to be accessed by a particular test that are not |
| accessed from the classpath need to be listed under supportfiles= in |
| the <testname>_app.properties file.<br> |
| Tests can refer to |
| classes without being in the classpath, and sql tests can use the ij |
| command 'run resource ' to execute additional .sql files without |
| changes to the _app.properties files. </p> |
| <p>For example, in the file |
| (org/apache/derbyTesting/functionTests/tests/)tools/dblook_test_app.properties:<br> |
| |
| <font size="2">supportfiles=tools/dblook_makeDB.sql,tools/dblook_test.jar<br> |
| </font></p> |
| <p>To support running on non-ISO-8859 systems, the harness copies files |
| ending in sql, .view, .multi, .properties, .txt and .policy into local |
| encoding; all other files are copied into UTF-8 encoding.</p> |
| <h3><a name="mozTocId427577"></a><a name="ov3"></a>4.3 |
| <testname>_app.properties</h3> |
| <p>Every test directory has a default_app.properties. This file is |
| for system level properties generic to all the tests in that test |
| directory. </p> |
| <p>If a test requires different system level properties, a test |
| specific properties file can be created to overwrite the defaults. |
| The test specific properties file needs to have a name starting with |
| the test file name, followed with _app.properties</p> |
| <p>For example, for the test tools/dblook_test.java, there is a |
| properties file called tools/dblook_test_app.properties</p> |
| <h3><a name="mozTocId715566"></a><a name="ov4"></a>4.4 |
| <testname>_derby.properties</h3> |
| <p>Every test directory has a default_derby.properties. This file is |
| for derby specific properties common to all the tests in that test |
| directory.<br> |
| If a test requires different derby properties, a test |
| specific properties file can be created to overwrite the defaults. |
| The test specific properties file needs to have a name starting with |
| the test file name, followed with _derby.properties</p> |
| <h3><a name="mozTocId874096"></a><a name="ov5"></a>4.5 tmp files, out |
| files, master files, and canons</h3> |
| <p>The test's output will be put into a file testname.tmp. Then the |
| output is modified if masking is required and the result is put into |
| a .out file.<br> |
| The expected output is found by examining the |
| following directories, based on certain input</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">functionTests/master/framework/jcc_version/jvmcode |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">functionTests/master/framework/jcc_version/earlier_jvmcode |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">functionTests/master/framework/jcc_version |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">functionTests/master/framework/jvmcode |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">functionTests/master/framework/earlier_jvmcode |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">functionTests/master/jvmcode </p> |
| </li> |
| <li> |
| <p>functionTests/master </p> |
| </li> |
| </ul> |
| <p>For example, if we are running a test and the flag |
| -Dframework=DerbyNet is used, to use network server and the IBM |
| Universal JDBC Driver ('jcc'), and the jvm we are using is Sun's jdk |
| 142, and the jcc version is 2.4 (not available at time of writing) |
| then the search for the master to compare with starts in the |
| functionTests/derbynet/jcc2.4/jdk14 directory. If a .out file with |
| the same name as the test is found in that directory, that master is |
| taken. If there is no such file in that directory, search continues |
| in the directory functionTests/derbynet/jcc2.4/jdk14 if it exists.</p> |
| <p>If there is no file there, nor for any other jcc directory, it |
| will continue to derbynet/jdk14, and the search is continued for |
| earlier jvm versions.<br> |
| If we are not running network server, the |
| DerbyNet and jcc_version directories are not traversed.</p> |
| <p>The version details do not go into the subversion level, i.e. |
| running with jdk141 or jdk142 is expected to have the same behavior. </p> |
| <p>This functionality supports dealing with minor differences in |
| behavior caused by minor differences in behavior in the underlying |
| jvms, jcc versions, differences between results returned through |
| network server vs. embedded and minor differences between a debug and |
| non debug (jar) build. </p> |
| <p>However, having a large number of these files means a maintenance |
| problem. Every time test output changes due to modifications to derby |
| or to the test, all output files in all directories need to be |
| updated accordingly. If at all possible, irrelevant differences |
| should be masked out, or the test should be written so that the |
| output does not reflect such items. </p> |
| <p>Suggestions to minimize canons: </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">create test specific masking </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">ensure test data has a specific |
| correct returned order; or an order by should be added to a query </p> |
| </li> |
| <li> |
| <p>when writing java tests, ensure only pertinent output is |
| reflected. </p> |
| </li> |
| </ul> |
| <p><br> |
| <br> |
| </p> |
| <h3><a name="mozTocId68107"></a><a name="ov6"></a>4.6 Masking and |
| comparing</h3> |
| <p>Tests often fail because of unimportant differences, such as |
| process ids, statement ids, timestamps. The derby functional test |
| harness provides for masking of these differences at 2 levels:</p> |
| <ol> |
| <li> |
| <p style="margin-bottom: 0in;">overall level. Masking required in |
| all, or many tests can be achieved using the class Sed in the test |
| harness directory. This class can either delete a reference present in |
| the .tmp file from the .out file, or replace it with a generic string. </p> |
| </li> |
| <li> |
| <p>test specific level. To make masking for only one test, a |
| (testname)_sed.properties file can be created which allows to either |
| remove a string from the output or to replace it. </p> |
| </li> |
| </ol> |
| <p>The diff is executed between the final resulting output and the |
| master file found copied into local encoding as .tmpmstr.</p> |
| <h3><a name="Adding_a_new_test"></a>4.7 Adding a new test</h3> |
| <p>To add a new test: </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">create the test file (e.g. |
| newfunctest.java or newfunctest.sql) in the appropriate tests |
| subdirectory</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">list any files needed that are not |
| .sql or .java files in a supportfiles entry in a test specific |
| _app.properties file. e.g. newfunctest_app.properties: |
| supportfiles=xyz.jar</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">list any specific derby properties |
| in a test specific _derby.properties file. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">add the properties files to the |
| copyfiles.ant file in the test specific directory </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">run the test. The first time around, |
| the test will fail because no master file will be found. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">if the output is correct, copy it to |
| the master directory. If you are running the test on a non-ISO8859 |
| encoded system, run the test with -DgenerateUTF8Out=true. This will |
| create a .utf8out file in fixed UTF-8 encoding - use that file to copy |
| into svn.<br> |
| </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">Build. This will copy the newly |
| created master into a matching directory structure in the classes |
| directory and, when building jars, the master files will be included in |
| the |
| derbyTesting.jar, so it can be found by the harness at run time. Note |
| that there is no copyfiles.ant file needed |
| for the master directory, all .out files are automatically copied to |
| the classes directory on build. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">run the test again. Investigate if |
| any differences need to be masked out using a test specific |
| sed.properties file (e.g. newfunctest_sed.properties). If so, ensure |
| this is added to copyfiles.ant.</p> |
| </li> |
| </ul> |
| <ul> |
| <li> |
| <p>ensure the test cleans up any testobjects created. This is |
| important for running the tests in suites with a remote server, or with |
| useprocess=false, because |
| in those cases, tests run against databases in the same |
| directory. Even if you do not anticipate your test to run in |
| those configurations, it makes good practice. You may also want to |
| protect your test from other tests leaving things behind by removing |
| them before running also. The class |
| org.apache.derbyTesting.functionTests.util.TestUtil has a cleanUpTest |
| method that may be useful.</p> |
| </li> |
| <li> |
| <p>ensure that connections can be made with all supported jvms. |
| This |
| includes the J2ME/CDC/JSR169 configuration, which only has |
| java.sql.Datasource available, no java.sql.DriverManager. One can use |
| the method ij.startJBMS() to get an appropriate connection, or |
| TestUtil.getConnection() if one needs to specify connection properties |
| different from what is defined in the <test>_app.properties file. |
| For instance, to shutdown a database one can use:</p> |
| </li> |
| </ul> |
| <div style="margin-left: 80px;"> |
| <p>TestUtil.getConnection("wombat","shutdown=true");</p> |
| </div> |
| <ul> |
| <ul> |
| </ul> |
| <li> |
| <p>ensure that the test does not cause any problems on non-ISO-8859 |
| systems. See section <a href="#4.15_Encoding_issues">4.15.</a></p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">ensure the test does not have hard |
| coded 'localhost' for network |
| connections. Instead, use the method TestUtil.getHostName() to find if |
| a hostName was specified for remote server testing.</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">add the test to a specific |
| suites/*.xml file, maintaining proper xml syntax. </p> |
| </li> |
| <li> |
| <p>run the suite, and correct any problems found.</p> |
| </li> |
| <li>make sure you svn add all new files, and that the svn:eol-style |
| is native for all text files, and not for any binary files.<br> |
| </li> |
| </ul> |
| <p><br> |
| <br> |
| </p> |
| <h3><a name="4.8_Suites_and_Adding_a_new_suite"></a>4.8 Suites and |
| Adding a new suite</h3> |
| <p>A suite constitutes of a <suitename>.properties file and/or |
| a <suitename>.runall file in the |
| org/apache/derbyTesting/functionTests/suites directory. The |
| .properties files hold references to other .properties files, or |
| .runall files, the .runall files are the actual lists of tests. </p> |
| <p>The lowest level suite always needs to have a .runall file. </p> |
| <p>For example, the derbyall suite is only a derbyall.properties file |
| that refers to other suites in the 'suites' property: </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="2">-----------------------derbyall.properties---------------<br> |
| suites=derbylang derbynetclientmats derbynetmats storeall xa derbytools<br> |
| derby.debug.true=enableBtreeConsistencyCheck<br> |
| derby.stream.error.logSeverityLevel=0</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>The derbylang suite is only a derbylang.runall, which lists the |
| tests. The derbynetclientmats suite has both a .runall and a |
| .properties file, so some additional properties can be specified that |
| are true for all tests in that suite. </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p><font size="2">------------------derbynetclientmats.properties-----------------<br> |
| framework=DerbyNetClient<br> |
| suites=derbynetclientmats derbynetmats<br> |
| jdk12test=true<br> |
| runwithj9=false<br> |
| timeout=60</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>To add a suite, you need to create at least a <suite>.runall |
| file, which lists the actual tests, or a properties file that refers |
| to other suites that do have a .runall file. The suite should be |
| added into the directory |
| ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites.</p> |
| <h3><a name="4.9_Running_with_a_new_jvm_"></a><a name="ov9"></a>4.9 |
| Running with a new jvm</h3> |
| <p>Currently, the supported jvms are: </p> |
| <table border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td> |
| <p> |
| <font size="2">jdk141 - Sun HotSpot jdk1.4.1 - |
| class jdk14</font><br> |
| <font size="2">jdk142 - Sun HotSpot jdk1.4.2 - |
| class jdk14</font><br> |
| <font size="2">jdk15 - Sun HotSpot jdk1.5 - class |
| jdk15</font><br> |
| <font size="2">jdk16 - Sun HotSpot jdk1.6 - class |
| jdk16</font><br> |
| <font size="2">jdk17 - Sun HotSpot jdk1.7 - class |
| jdk17</font><br> |
| <font size="2">ibm141 - IBM Classic jdk1.4.1 - |
| class ibm14</font><br> |
| <font size="2">ibm142 - IBM Classic jdk1.4.1 - |
| class ibm14</font><br> |
| <font size="2">ibm15 - IBM Classic jdk1.5 - |
| class ibm15</font><br> |
| <font size="2">j9_foundation11 - WEME jvm (available |
| with IBM Websphere Everyplace Micro Edition, 6.1.1), version 2.3, j2ME 1.1 - |
| class j9_foundation11</font></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>The classes above are subclasses of |
| org.apache.derbyTesting.functionTests.harness.jvm. The name at the |
| front is just a convention.</p> |
| <p>To run a test with a jvm that does not have a matching class under |
| org.apache.derbyTesting.functionTests.harness, do the following:</p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">just run the tests as if there is a |
| jvm class. The harness will default to using the jdk16 class. Unlikely, |
| but possibly there are no differences</p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">if there are failures showing that |
| cannot be explained any other way but genuine, acceptable jvm |
| differences, do the following: </p> |
| <ul> |
| <li> |
| <p style="margin-bottom: 0in;">create a subclass of |
| org.apache.derbyTesting.functionTests.harness.jvm. In this class, |
| specify any jvm specific property settings required </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">compile the new jvm class and |
| run the tests </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">create a new canon directory for |
| any additional canons that need to be created. </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">in rare occasions, other harness |
| changes may be required </p> |
| </li> |
| <li> |
| <p style="margin-bottom: 0in;">for any tests that should not |
| run with this environment, add a line in the testname_app.properties |
| file indicating this. For instance to add a line for a jvm called |
| jdk29, it would be like this: runwithjdk29=false. Note that the |
| versioning does not currently extend past 2 digits. For j9 jvms, |
| versioning does not apply currently. For all j9 versions, use |
| runwithj9=false. For j9_foundation*, use runwithfoundation=false. </p> |
| </li> |
| <li> |
| <p>Add code in RunTest.java to switch to the new jvm based on |
| values for system and vendor properties </p> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| <p><br> |
| <br> |
| </p> |
| <h3><a name="skipping"></a>4.10 Skipping a test</h3> |
| <p>There are 2 skipping mechanisms in place for different kinds of |
| skipping of a test.</p> |
| <p>Some tests are written to test specific functionality only |
| available with for instance certain jvms, or, with network server, |
| certain versions of the IBM Universal Driver. To control this, |
| properties can be set for each test, for instance, if a test should |
| not be run when using an ibm jvm, set runwithibmjvm=false. If a test |
| should be run with Sun Hotspot jvm version 14, then set |
| runwithjdk14=true.<br> |
| The skip setting does not go into the |
| subversion level, i.e. setting runwithjdk141=false has no effect, and |
| setting runwithjdk14 affects runs with jdk141 as well as |
| jdk142.<br> |
| Other skip reasons are encryption protocols specific to a |
| certain jvm. </p> |
| <p>The property for skipping a test based on the version of the IBM |
| Universal Driver is "excludeJCC". The keywords |
| "<b>at-or-before</b>" and "<b>at-or-after</b>" |
| can be used to specify which range of JCC versions should be |
| excluded. If neither of these keywords is provided, the default |
| is "<b>at-or-before</b>". For example:</p> |
| <p>To skip a test when running with any version of the IBM Universal |
| Driver that is 2.4 or earlier:<br> |
| excludeJCC=at-or-before:2.4<br> |
| <br> |
| To |
| skip a test when running with any version of the IBM Universal Driver |
| that is 2.0 or later:<br> |
| excludeJCC=at-or-after:2.0</p> |
| <p>You can also specify an (optional) jvm clause to further tune the |
| exclusion criteria. This clause starts with the "<b>,when</b>" |
| tag and is followed by a three-part jvm version. In this case, |
| a test will only be skipped if BOTH the JCC clause AND the jvm clause |
| are true. For example:</p> |
| <p>To skip a test when running with any version of the IBM Universal |
| Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or |
| earlier:<br> |
| excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1</p> |
| <p>To skip a test when running with any version of the IBM Universal |
| Driver that is 2.0 or earlier, but ONLY if the jvm is 1.5 or |
| later:<br> |
| excludeJCC=at-or-before:2.0,when-at-or-after:jdk1.5.1</p> |
| <p>Another skipping mechanism works on entire 'frameworks'. Currently |
| there are only 3 supported in the harness, embedded, network server |
| with the derbyclient driver ('DerbyNetClient') and network server |
| with the IBM Universal JDBC Driver ('DerbyNet'). In the suites |
| directory there are .exclude files for each of the frameworks - if |
| for some reason an exclude file were not there, you would see a |
| warning message show up for every test run. In this |
| 'framework'.exclude file tests can be placed that for some reason |
| need to be excluded from running with that framework. This mechanism |
| enables adding of suites to a framework run even if a few of the |
| tests are not appropriate for that particular framework.</p> |
| <p>Note that at this time, only skipped suites show up in the .skip |
| result file. This still needs to be corrected.</p> |
| <p><br> |
| <br> |
| </p> |
| <h3><a name="frameworks"></a>4.11 Frameworks</h3> |
| <p>Currently, there are two frameworks used for network |
| server, DerbyNetClient, which uses the derby Client driver, and |
| DerbyNet, which uses the IBM Universal JDBC Driver. <br> |
| Setting the |
| framework property will invoke the test harness class NetServer which |
| has the actual configuration (driver name, portnumber etc.) used for |
| the individual frameworks. <a href="#Note3:"><font size="2">See |
| Note3.</font></a><br> |
| Setting this framework also causes the search |
| for expected output to include appropriate DerbyNetClient or |
| DerbyNet and jcc version specific subdirectories under the master |
| directory.</p> |
| <p><br> |
| <br> |
| </p> |
| <h3><a name="props"></a>4.12 Some test harness properties</h3> |
| <p>For a complete set, refer to comments in RunTest.java, but here |
| are some valuable test properties which can be passed to the RunTest |
| class: </p> |
| <table style="width: 791px; height: 632px;" border="1" cellpadding="2" |
| cellspacing="3"> |
| <col width="994"> <tbody> |
| <tr> |
| <td width="994"> |
| <ul> |
| <li>runwith<jvm> - See above section <a href="#skipping">4.10</a><br> |
| <br> |
| </li> |
| <li>framework - specifies which framework to run with. For |
| example:<br> |
| java -Dframework=DerbyNetClient |
| org.apache.derbyTesting.functionTests.harness.RunTest <br> |
| <br> |
| lang/supersimple.sql<br> |
| </li> |
| <li>verbose - Shows more detailed output if set to "true". For |
| example:<br> |
| <small>java -Dverbose=true |
| org.apache.derbyTesting.functionTests.harness.RunTest lang/arithmetic.sql</small><br> |
| <br> |
| </li> |
| <li>keepfiles - Indicates to not clean up any of the files if |
| the test |
| passed if set to "true".<br> |
| <br> |
| </li> |
| <li>testSpecialProps - sets additional properties. Several can |
| be set using |
| '^' as separator: <br> |
| -DtestSpecialProps=<prop-1>=<value-1>^ |
| ... ^<prop-n>=<value-n>. For example:<br> |
| <small>java -DTestSpecialProps=derby.infolog.append=true |
| org.apache.derbyTesting.functionTests.harness.RunTest lang/arithmetic.sql</small> |
| <br> |
| <br> |
| </li> |
| <li>jvmflags - sets specific jvm properties for each jvm instantiated by the |
| test harness, for instance for setting initial and maximum heap size (note that setting |
| only initial size or only max size may cause incompatibilities with heap settings picked |
| up from harness property files or from the command line), or other properties |
| that need to be passed on to the jvm, separated by '^'. This property |
| can be set on the commandline for either RunTest or RunSuite, or in one |
| of the properties files used by the tests. The property jvmflags |
| when passed on at the command line supercedes the properties set in a suite's propertyfile, |
| and those supercede jvmflags properties set in a subsuite's or test's propertyfile, |
| because the 'highest' level properties are passed to the jvm last. Setting this property |
| for a test that runs with useprocess=false cannot have any effect. |
| Example: <br> |
| <small>java -Djvmflags=-Xms32M^-Xmx128M |
| org.apache.derbyTesting.functionTests.harness.RunTest lang/concateTests.java</small><br> |
| <br> |
| </li> |
| <li>excludeJCC - |
| See above section <a href="#skipping">4.10</a><br> |
| <br> |
| </li> |
| <li>useprocess - (default=true) Controls whether RunTest runs |
| the test in a separate VM or in a thread in harness VM. Also does not |
| create |
| subdirectories for each test and thus will |
| attempt to reuse databases with the same name. It is |
| potentially useful for debugging tests. Unit tests are not (yet) |
| runnable with |
| "useprocess=false". <br> |
| <br> |
| </li> |
| <li>startServer - allows for Network Server tests to start and |
| shutdown |
| Network Server from the test itself. Default is true - i.e. the |
| test harness will start Network Server.<br> |
| <br> |
| </li> |
| <li>noSecurityManager – disable the client JVM from installing |
| a |
| SecurityManager. See section <a href="#security">4.13</a></li> |
| <li> |
| <p>derbyTesting.replacePolicyFile - replace or append the |
| contents of the default policy file derby_tests.policy. default is |
| false, i.e. append. See section <a href="README.htm#security">4.13</a> |
| </p> |
| </li> |
| <li> |
| <p>hostName - allows for running Network Server on a remote |
| host. See |
| section <a href="#hostName">4.14</a>.</p> |
| </li> |
| <li>derbyTesting.encoding - allows for running the harness with |
| a different |
| encoding. Only supported with jdk15. Example: <br> |
| <br> |
| <small> java |
| -DderbyTesting.encoding=UTF-16 |
| org.apache.derbyTesting.functionTests.harness.RunTest |
| jdbcapi/resultset.java</small><br> |
| <br> |
| </li> |
| <li>generateUTF8Out - copies the test .out into UTF-8 encoding. |
| See section <a href="#4.15_Encoding_issues">4.15</a>.<br> |
| <br> |
| </li> |
| <li>derbyTesting.jar.path - This property is required to run |
| upgrade tests. Set this property to point to the location of jar files |
| from a previous release from which we need to test upgrade. This |
| property needs to be set only if the source files will not be available |
| when running tests. In this case, the jars can be copied to any |
| location with the following relative path |
| ${jar_location}/{majorversion.minorversion}. The property should be set |
| as follows:<br> |
| <br> |
| derbyTesting.jar.path=${jar_location}<br> |
| <br> |
| If the test is run on the machine where the source files from svn have |
| been built, then it is not required to set this property. The jars |
| checked into svn will be used. If the tests are being run in a location |
| where the jars from the previous release are available in a different |
| location than where the source files for the upgrade tests have been |
| built, the following property needs to be set on the command line for |
| RunSuite or RunTest where the tests are being run:<br> |
| <br> |
| -Djvmflags=-DderbyTesting.jar.path={path_to_derby_jars}<br> |
| <br> |
| where {path_to_derby_jars} points to a directory where the jars for the |
| previous Derby release being tested for upgrade reside in a |
| subdirectory named according to the major.minor version. For example:<br> |
| <br> |
| If the Derby 10.1 jars for upgrade testing reside in |
| /opt/testing/derby/10.1 then the command line needed for the upgrade |
| tests to complete would include the following:<br> |
| <br> |
| <small>java |
| -Djvmflags=-DderbyTesting.jar.path=/opt/testing/derby |
| org.apache.derbyTesting.functionTests.harness.RunSuite upgrade</small><br> |
| <br> |
| Please note that if you change the value of derbyTesting.jar.path you |
| will need to remove the file <small>org/apache/derbyTesting/functionTests/tests/upgradeTests/Upgrade_10_1_10_2.properties</small> |
| from your output directory and run 'ant all' for the updated value of |
| the property to be written into that generated file.<br> |
| </li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <h3><a name="security"></a><br> |
| 4.13 SecurityManager testing by |
| default</h3> |
| <p>By default the tests install the standard Java SecurityManager |
| using the system property java.security.manager and use a policy file |
| derby_tests.policy. The default file is sourced |
| at<br> |
| ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/util/derby_tests.policy</p> |
| <p>During a test run this policy file is copied into ${user.dir} and |
| used from there.<br> |
| Optionally a test-specific or suite-specific policy file may be |
| appended or used instead. The name of the policy file remains |
| derby_tests.policy. <br> |
| </p> |
| <p>A test_specific policy file has the same name as the test, has |
| extension .policy, and is located in the same location as the test. For |
| instance, |
| ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/tests/lang/errorStream.policy |
| is the test-specific policy file for the test errorStream.java. A |
| suite-specific policy file has the same name as the suite, has |
| extension .policy, and is located in the |
| ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites |
| directory.<br> |
| </p> |
| <p>By default, a test-specific and/or suite-specific policy file is |
| appended to the default policy file (derby_tests.policy). However, if a |
| test's _app.properties or suite's .properties file contains the |
| property derbyTesting.replacePolicyFile=true, then the file |
| derby_tests.policy will be overwritten with the contents of the test- |
| or suite-specific policy file.<br> |
| </p> |
| <p>There are two environments for the installation of the |
| SecurityManager.</p> |
| <ul> |
| <li> |
| <p>Server JVM for the network client tests. Always uses a |
| SecurityManager.</p> |
| </li> |
| <li> |
| <p>Client side JVM, this is the JVM executing the JDBC calls |
| against any Derby driver. |
| Installs a SecurityManager unless:</p> |
| </li> |
| <ul> |
| <li> |
| <p><font face="Courier New"><font size="2"><font color="#000000">noSecurityManager=</font><font |
| color="#2a00ff">true </font><font size="3"><font |
| face="Times New Roman, serif"><font color="#000000">in the test's |
| _app.properties file.Used to disable individual tests that cannot run |
| under the security manager, or if the test has a functional requirement |
| not to run with the SecurityManager. Ideally, few tests will have this |
| property set. Any disabling of the SecurityManager for a test requires |
| a comment in the test's _app.properties file indicating why the test |
| cannot run under the SecurityManager. </font></font></font></font></font></p> |
| </li> |
| <li> |
| <p><font color="#000000"><font face="Times New Roman, serif"><font |
| size="3">Client JDBC driver is DB2's Universal driver for JDBC. |
| Currently does not install a SecurityManager, no technical reasons, |
| just has not been done.</font></font></font></p> |
| </li> |
| </ul> |
| </ul> |
| <h3><a name="hostName"></a><br> |
| 4.14 Testing with Network Server on a remote host</h3> |
| To enable testing against a Network |
| Server running on a remote |
| machine - which is for instance needed to test on IPV6 machines - one |
| can use the property hostName. <br> |
| By default the network server is assumed not to be running, and the |
| test harness will start Network Server on the local machine for each |
| test - unless the test |
| has been qualified explicitly with the property startServer to |
| false. In that case, the test itself is assumed to be starting Network |
| Server. |
| (This can be set either in a test's property file or with the -D on the |
| commandline). Thus, by default Network Server is started and shutdown |
| in the subdirectory for the framework created for each test, and new |
| databases are created in each test directory (e.g. |
| ./getCurConnJdbc20/wombat). The test policy file also gets copied to |
| the appropriate location. <br> |
| When one wants to run against a Network Server running on a remote |
| machine, the test harness cannot start or stop it because of security |
| reasons. Thus, the setting of the property hostName automatically sets |
| startServer to false. The hostName is automatically passed on to the |
| actual test run as the property ${derbyTesting.serverhost} which is |
| used in the derby_tests.policy file on the client machine.<br> |
| The hostName property is not a pre-set value in |
| any |
| suite, some manual steps are required to run tests this way:<br> |
| <ol> |
| <li>Extract and/or copy the |
| (java/testing/org/apache/derbyTesting/functionTests/util)derby_tests.policy |
| from the |
| derbyTesting.jar or from the svn source to a location on |
| the host on which you want to run the NetworkServer</li> |
| <li>Set up the classpath for Network Server to include derbynet.jar |
| and derbyTesting.jar (or equivalent classes). You need to have |
| derby.jar available also, although it does not need to be in the |
| classpath.<br> |
| </li> |
| <li>Start Network Server like so when using compiled classes for it:</li> |
| </ol> |
| <table |
| style="width: 751px; height: 72px; text-align: left; margin-right: auto; margin-left: 40px;" |
| border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td style="vertical-align: top;">java -Djava.security.manager |
| -Djava.security.policy=copied_derby_tests.policy |
| -DderbyTesting.codeclasses=file:/derby/classes/ |
| -DderbyTesting.codedir=/derby/classes |
| -DderbyTesting.serverhost=localhost |
| -DderbyTesting.clienthost=clientIPaddress |
| -DderbyTesting.codejar=file://unused/ |
| org.apache.derby.drda.NetworkServerControl -h 0.0.0.0 start<br> |
| <small>where /derbyt/classes is the directory where the |
| classes can be found, </small><small>and clientIPaddress is your |
| client machine's IP |
| address or name. In this example, derby.system.home is in the directory |
| where Network Server is started, and the derby_tests.policy file is in |
| that directory too.</small><br> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <div style="text-align: left;"> |
| or like so when using jars:<br> |
| <table |
| style="width: 751px; height: 72px; text-align: left; margin-right: auto; margin-left: 40px;" |
| border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td style="vertical-align: top;">java |
| -Dderby.system.home=/home/derbytst/ns -Djava.security.manager |
| -Djava.security.policy=/home/derbytst/ns/copied_derby_tests.policy |
| -DderbyTesting.clienthost=clientIPaddress |
| -DderbyTesting.codejar=file:/derby/tstbld/ |
| -DderbyTesting.codedir=/derby/tstbld |
| -DderbyTesting.serverhost=localhost |
| -DderbyTesting.codeclasses=file://unused/ |
| org.apache.derby.drda.NetworkServerControl -h 0.0.0.0 start<br> |
| <small>where /derbyt/tstbld is the directory where |
| derbynet.jar is found and derby.system.home points to the directory |
| where your |
| derby.properties are and where you want to |
| have derby.log and any databases go, </small><small>and |
| clientIPaddress is your client machine's IP address or name</small><small>. |
| </small><br> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <ol start="4"> |
| <li>On the client machine, start the tests as usual, but specifying |
| hostName, like so:</li> |
| </ol> |
| </div> |
| <table |
| style="text-align: left; margin-right: auto; margin-left: 40px; height: 31px; width: 786px;" |
| border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td style="vertical-align: top;">java |
| -DhostName=favorite.remotehost.com |
| org.apache.derbyTesting.functionTests.harness.RunSuite |
| derbynetclientmats<br> |
| <small>where favorite.remotehost.com is the host name or ip |
| address.</small><br> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <br> |
| RunTest can pick up the property in a similar way. You can set |
| -Dframework as usual. You can also run derbynetmats with jcc using |
| -DhostName flag; in that case you do not need to worry about running |
| with a policy |
| file - at the moment, tests run with DerbyNet framework do not run with |
| SecurityManager. Tests in |
| the derbynet(client)mats suite have been modified to run against a |
| remote server as much as possible. <br> |
| <br> |
| When the test harness encounters the -DhostName, and the value is not |
| null and not localhost, it switches to startServer=false. Also, against |
| a remote host, only the ping operations of the NetworkServerControl is |
| supported. Finally, because on the remote server no new directories can |
| be created by the test harness because of security, databases will be |
| reused. This |
| has a number of implications:<br> |
| <ol> |
| <li>the test harness, or individual tests cannot start and stop |
| Network Server. <br> |
| </li> |
| <li>the tests cannot check derby.log, nor data files created on the |
| server by import/export or dblook utilities.<br> |
| </li> |
| <li>as the database location is the same for each test, tests |
| run this way need to be diligent |
| about cleaning up after or will cause failures in later tests. The |
| .java |
| tests can use the TestUtil.cleanUp method to drop any database object |
| without any output being reflected. Test writers need to ensure that |
| the objects are in fact getting dropped or the test needs to be |
| excluded from running remotely. To facilitate cleanup, a method was |
| added to org.apache.derbyTesting.functionTests.util.TestUtil, |
| cleanUpTest(Statement, testObjects[]). This method drops anything |
| passed in the array of strings and disregards any SQLExceptions. The |
| benefit of this is that a possible failure of drop does not stop the |
| test, and does not prevent other objects passed in the string to be |
| dropped. An example of usage of this method:<br> |
| <table style="width: 100%; text-align: left;" border="1" |
| cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td style="vertical-align: top;"> static |
| void teardown() throws SQLException {<br> |
| Statement stmt = |
| conn.createStatement();<br> |
| String[] testObjects = { |
| "table t1", "table t2", "procedure p1", "schema s restrict"};<br> |
| TestUtil.cleanUpTest(stmt, |
| testObjects);<br> |
| conn.commit();<br> |
| stmt.close();<br> |
| } <br> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| Note that if something prohibits an object from being dropped, e.g. if |
| there still are locks on it, this will not be noticed with this method |
| - the test writer should still verify that the test objects get cleaned |
| up appropriately.<br> |
| </li> |
| <li>tests cannot use a mix of Network Server and local connections to |
| the same database.</li> |
| <li>tests should not hardcode the string 'localhost' or they cannot |
| be run in this |
| fashion. Tests that need to be run in this fashion can use the result |
| of |
| the TestUtil.getHostName() method instead.</li> |
| <li>Tests can only use the ping option of the NetworkServerControl |
| class.</li> |
| <li>Tests cannot be dependent on test- or suite-specific policy files.<br> |
| </li> |
| </ol> |
| New .exclude files for the current frameworks (i.e. |
| DerbyNetRemote.exclude and DerbyNetClientRemote.exclude) have been |
| created to skip some tests that cannot be run against a remote server |
| because of those implications. <br> |
| <h3><a name="encoding"></a></h3> |
| <h3><a name="4.15_Encoding_issues"></a>4.15 Encoding issues</h3> |
| <p>There are three aspects to dealing with encoding issues, how to run |
| a test with specific encoding, what to do to ensure a test can be run |
| without relying on a certain encoding, and how to generate test output |
| in fixed encoding.<br> |
| </p> |
| <ul> |
| <li> |
| <h4>Run a test with a specific encoding</h4> |
| </li> |
| </ul> |
| <p style="margin-left: 40px;">To force a specific test to run with a |
| specific |
| encoding one can use the property derbyTesting.encoding. This is |
| currently only supported with jdk15. For instance, |
| to run a specific .java test one time only with |
| -DderbyTesting.encoding: |
| </p> |
| <table |
| style="margin-left: auto; margin-right: auto; text-align: left; height: 32px; width: 760px;" |
| border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td style="vertical-align: top;">java |
| -DderbyTesting.encoding=UTF-16 |
| org.apache.derbyTesting.functionTests.harness.RunTest |
| jdbcapi/lobStreams.java<small><br> |
| </small></td> |
| </tr> |
| </tbody> |
| </table> |
| <p> The test |
| encodingTests contains tests that run with derbyTesting.encoding=UTF-16 |
| in the encodingTests.properties file:</p> |
| <table style="width: 50%; text-align: left; margin-left: 40px;" |
| border="1" cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td style="vertical-align: top;">derbyTesting.encoding=UTF-16</td> |
| </tr> |
| </tbody> |
| </table> |
| <ul> |
| <li> |
| <h4>ensure running on non-ISO-8859 systems is possible</h4> |
| </li> |
| </ul> |
| <p style="margin-left: 40px;">Special care is required to ensure that |
| individual tests can be run successfully on non-ISO-8859 systems. This |
| includes:<br> |
| </p> |
| <ul style="margin-left: 40px;"> |
| <li> |
| <p>Ensure reading of files is done in an appropriate manner. The |
| test harness will copy supportfiles with extension .*sql, .multi, |
| .view, .txt, .properties, and .policy in local encoding. This means, |
| that accessing such a file should be done with an appropriate |
| mechanism; i.e. from within ij (i.e. a .sql test) use 'run', and from |
| within a .java test use FileReader. The test harness will copy other |
| supportfiles in fixed encoding, i.e. using UTF-8. To access these files |
| and you will have to use an input mechanism that supports specifying |
| of the encoding, such as FileInputStream and BufferedReader. |
| Various .sql tests that run other .sql scripts use the test-specific |
| mechanismn in ij to access files from the classpath: 'run resource'. |
| This construct assumes the files listed are in the classpath - the |
| classes dir or derbyTesting.jar - and they are assumed to be in UTF-8.</p> |
| </li> |
| <li> |
| <p>Avoid usage of non-encoding safe constructors for Strings. i.e. |
| Avoid String(byte[]) and String(byte[],int,int) to create a string.</p> |
| </li> |
| </ul> |
| <p style="margin-left: 40px;"> </p> |
| <ul> |
| <li> |
| <h4>Generating master files on non-ISO-8859 systems</h4> |
| </li> |
| </ul> |
| <ul style="margin-left: 40px;"> |
| <li> |
| <p>Special care is required when developing tests and running them |
| on non-ISO-8859 systems. The output files cannot be checked into a svn |
| tree which is assumed to be ISO-8859 compatible. To avoid surprises, |
| use the property -DgenerateUTF8Out which will create a .utf8out file |
| which is a copy of the test output converted into UTF-8. Be careful not |
| to let a file transfer program modify this file (for instance. ftp in |
| ascii mode may try to convert the file).<br> |
| </p> |
| <table style="width: 90%; text-align: left;" border="1" |
| cellpadding="2" cellspacing="2"> |
| <tbody> |
| <tr> |
| <td style="vertical-align: top;">java |
| -DgenerateUTF8Out=true |
| org.apache.derbyTesting.functionTests.harness.RunTest |
| lang/supersimple.sql</td> |
| </tr> |
| </tbody> |
| </table> |
| </li> |
| </ul> |
| <h2>Notes</h2> |
| <p style="margin-bottom: 0in;"><a name="Note1:"></a><font size="2"><br> |
| Note1:</font></p> |
| <p style="margin-left: 0.42in; margin-bottom: 0in;"><font size="2">There |
| is one more suite included: the j9derbynetmats suite is a |
| modification of the derbynetmats suite. It was intended to test the |
| network server with the jvm available with IBM's WCTME (Workplace |
| Client Technology, Micro Edition; formerly WSDD). |
| Its use has been discontinued. |
| Note that the setup for running the j9derbynetmats tests is very |
| specific to the test harness, not even using the WCTME files in their |
| normal location.</font> <font size="2"><br> |
| The j9derbynetmats suite is |
| included to serve as an example of splitting the network server |
| process to run with a different jvm than the test client. The |
| j9derbynetmats suite will run with another jvm as client (as defined |
| in the suite properties), but start up network server with the 'j9' |
| jvm files (the reference to 'j9' is based on the executable, j9.exe), |
| based on the property 'serverJvm'. Running this suite requires |
| providing the property bootcp, which is interpreted from |
| the test harness class j9_13. See also section on adding a new <a |
| href="#ov9">jvm |
| setup</a>. </font> |
| </p> |
| <p style="margin-bottom: 0in;"><a name="Note2:"></a><br> |
| <font size="2">Note2:</font></p> |
| <p style="margin-left: 0.42in; margin-bottom: 0in;">O<font size="2">ccasionally, |
| cleanup is unsuccessful. This does not constitute a problem in any |
| way, as the harness for most suites starts with a clean database, and |
| clean copies of all files. However, you will see something like this |
| in the output:</font><br> |
| <font size="2">Warning: Cleanup failed on |
| baseDir: /local/myrun1/DerbyNet/supersimple.</font></p> |
| <p style="margin-bottom: 0in;"><a name="Note3:_"></a><br> |
| <font size="2">Note3: |
| </font></p> |
| <p style="margin-left: 0.42in; margin-bottom: 0in;"><font size="2">NetServer |
| also has a configuration for connecting to DB2 via jcc - the IBM |
| Universal Driver - and via the older DB2 driver. But there are |
| currently no tests to exercise these settings.</font></p> |
| <p style="margin-bottom: 0in;"><br> |
| </p> |
| </body> |
| </html> |