test/docs/TestOrganization.txt - xmlbeans - Git at Google

 Best practices of writing tests

 1)  TESTS:
     Tests are written under version they apply to. For e.g. all tests related to
     v2 should be written under v2/test

     The test folder is broken down into
     a) src
     b) cases
     c) tools

     Below are the uses for each of the folders.
     a) Src:

     This is where the actual test cases (in this case junit driver) live. The src
     folder is divided by functional areas at the highest level.

     Each functional area is further sub divided by smaller functional areas (if
     required). If the division is not required then functional area is divided
     by importance of the tests. Importance are of 2 types

     a) Checkin
     b) Detailed.

     As the names suggest checkin tests are really important tests for a particular
     area while Detailed tests aim is to test more of edge cases/error condition/
     various kinds of input etc.

     Not each top-level functional area needs to have checkin folder. If that area
     is not that widely used in the product there might just be detailed folder.

     Since checkin tests have to be run by all developer every time they checkin
     the goal is to include the most crucial tests for each functional area of
     the product.

     Also each top level/sub functional area can have a Common folder. The main
     goal here is group smaller utilities used to test functional area in one
     place so that both the Checkin/Detailed tests can use. If you feel the tools
     you are writing can be share across Cross-functional areas then use the Tools
     folder (more to come later in the doc). Any class which is not itself a Junit
     test should go into that folder, even if it is a superclass for a Junit test
     in the checkin/detailed folder. Same goes for all interfaces.


     Below is an e.g. of how one of the top-level functional area's directory
     structure may look like.

     xmlcursor/
             |-checkin/
             |-detailed/
             |-common/
             |-saver/
             |     |-detailed/
             |     |-common/
             |-xpath/
             |     |-checkin/
             |     |-detailed/
             |     |-common/


     In this e.g. the xmlcursor area is broken down by checkin/detailed tests for
     smaller functionality that is cursor specific. For functionality in the
     cursor that is bigger sub folders are created with its own checkin/detailed/
     common areas. Again saver does not have a checkin folder since the tests in
     this area are not too important.

     Overall when writing a test please consider which areas they fit and only
     create sub folder/top level folders only if necessary.

 2) CASES:
    This contains schema files/xml instances belonging to those schemas and schema
    independent xml instances which can be shared within the functional areas.

    Cases is further divided into top level areas matching src folder and all
    schemas/xml instances used by functional areas are placed within this.

    All schemas are compiled with scomp during the build process before compiling
    the tests. If you have added a top level folder to src then the schemas & xml
    instances for those tests should have their top-level folder within cases/xbean.
    If these schemas need to be compiled by scomp you will need to edit the
    testbuild.xml file to include the new folder in the build. Add this folder
    to the variable "schema.dirs". (More details below)


    If you are just adding schemas to existing folders within xbean/cases then
    these will be automatically picked up in the build process.

    If you have schemas that you don't want to pre compile in the build process
    please rename the extension from .xsd to .xsddonotcompile. In this way scomp
    will not pick up pre compile the schema.

    The best practice for the cases folder is to try to reuse schemas within the
    functional areas. So please check the schemas we have before adding more schemas.

    Since currently the directory structure is such that process sharing of
    schemas exists within top level functional areas (e.g. all tests with
    xmlcursor can reuse the schemas) there is some amount of duplication since
    schemas are not shared among top level functional areas. (for e.g. xmlcursor
    & xmlobject will not have schemas shared among them). Though we recognize this
    and are making attempts to revise the directory structure this has
    implications on how the tests run so for the immediate future this might stay
    as it is.


 3) TOOLS:
     The tools area is meant for storing tools/utilities which help running the
     tests and reporting results. If you feel you have written a tool which could
     be used across all functional areas please put the source within tools/src.
     However before adding a tool please check the existing tools which we already
     have.

 4) BUILD FILE CHANGES:
     Depending on how tests and cases are added, the build file (testbuild.xml)
     will need to be updated.

     a) New folder containing schemas added to cases:
        Append this folder to the property schema.dirs
         e.g:
         Old value:
         <property name="schema.dirs"
                 value="${build.dir.test.schemas}/xbean/xmlcursor"/>

         New value:
         <property name="schema.dirs"
                 value="${build.dir.test.schemas}/xbean/xmlcursor,
                        ${build.dir.test.schemas}/xbean/NEWAREA"/>
         If your schemas are needed in order to run checkintests please update the
         "checkin.schema.dirs" property as well.


     b) New top-level area added to test (src/NEWAREA)
        Add this folder to the list of areas for incremental compile
        * Append the area name to the property area.names
        * Add a new target with the name "build.area.NEWAREA"

          Properties that need to be set for this target to work
              * includes.for.compile - includes spec for javac

          Optionally set the following properties
              * this.schema.dirs - list of folders for scomp
              * this.schema.standalone.dirs - list of individual schemas for scomp
              * testcasesjar.update - set to true if testcases.jar should be
                                      updated instead of overwriting

          Add a call to target "unit.build" at the end of the target.

          e.g:
          <target name="build.area.xmlcursor">
             <property name="this.schema.dirs"
                 value="${build.dir.test.schemas}/xbean/xmlcursor"/>
             <property name="includes.for.compile"
                 value="**/xmlcursor/**/*.java"/>
             <property name="testcasejar.update" value="true"/>
             <antcall target="unit.build"/>
         </target>
	Best practices of writing tests

	1) TESTS:
	Tests are written under version they apply to. For e.g. all tests related to
	v2 should be written under v2/test

	The test folder is broken down into
	a) src
	b) cases
	c) tools

	Below are the uses for each of the folders.
	a) Src:

	This is where the actual test cases (in this case junit driver) live. The src
	folder is divided by functional areas at the highest level.

	Each functional area is further sub divided by smaller functional areas (if
	required). If the division is not required then functional area is divided
	by importance of the tests. Importance are of 2 types

	a) Checkin
	b) Detailed.

	As the names suggest checkin tests are really important tests for a particular
	area while Detailed tests aim is to test more of edge cases/error condition/
	various kinds of input etc.

	Not each top-level functional area needs to have checkin folder. If that area
	is not that widely used in the product there might just be detailed folder.

	Since checkin tests have to be run by all developer every time they checkin
	the goal is to include the most crucial tests for each functional area of
	the product.

	Also each top level/sub functional area can have a Common folder. The main
	goal here is group smaller utilities used to test functional area in one
	place so that both the Checkin/Detailed tests can use. If you feel the tools
	you are writing can be share across Cross-functional areas then use the Tools
	folder (more to come later in the doc). Any class which is not itself a Junit
	test should go into that folder, even if it is a superclass for a Junit test
	in the checkin/detailed folder. Same goes for all interfaces.


	Below is an e.g. of how one of the top-level functional area's directory
	structure may look like.

	xmlcursor/
	\|-checkin/
	\|-detailed/
	\|-common/
	\|-saver/
	\| \|-detailed/
	\| \|-common/
	\|-xpath/
	\| \|-checkin/
	\| \|-detailed/
	\| \|-common/


	In this e.g. the xmlcursor area is broken down by checkin/detailed tests for
	smaller functionality that is cursor specific. For functionality in the
	cursor that is bigger sub folders are created with its own checkin/detailed/
	common areas. Again saver does not have a checkin folder since the tests in
	this area are not too important.

	Overall when writing a test please consider which areas they fit and only
	create sub folder/top level folders only if necessary.

	2) CASES:
	This contains schema files/xml instances belonging to those schemas and schema
	independent xml instances which can be shared within the functional areas.

	Cases is further divided into top level areas matching src folder and all
	schemas/xml instances used by functional areas are placed within this.

	All schemas are compiled with scomp during the build process before compiling
	the tests. If you have added a top level folder to src then the schemas & xml
	instances for those tests should have their top-level folder within cases/xbean.
	If these schemas need to be compiled by scomp you will need to edit the
	testbuild.xml file to include the new folder in the build. Add this folder
	to the variable "schema.dirs". (More details below)



	If you are just adding schemas to existing folders within xbean/cases then
	these will be automatically picked up in the build process.

	If you have schemas that you don't want to pre compile in the build process
	please rename the extension from .xsd to .xsddonotcompile. In this way scomp
	will not pick up pre compile the schema.

	The best practice for the cases folder is to try to reuse schemas within the
	functional areas. So please check the schemas we have before adding more schemas.

	Since currently the directory structure is such that process sharing of
	schemas exists within top level functional areas (e.g. all tests with
	xmlcursor can reuse the schemas) there is some amount of duplication since
	schemas are not shared among top level functional areas. (for e.g. xmlcursor
	& xmlobject will not have schemas shared among them). Though we recognize this
	and are making attempts to revise the directory structure this has
	implications on how the tests run so for the immediate future this might stay
	as it is.


	3) TOOLS:
	The tools area is meant for storing tools/utilities which help running the
	tests and reporting results. If you feel you have written a tool which could
	be used across all functional areas please put the source within tools/src.
	However before adding a tool please check the existing tools which we already
	have.

	4) BUILD FILE CHANGES:
	Depending on how tests and cases are added, the build file (testbuild.xml)
	will need to be updated.

	a) New folder containing schemas added to cases:
	Append this folder to the property schema.dirs
	e.g:
	Old value:
	<property name="schema.dirs"
	value="${build.dir.test.schemas}/xbean/xmlcursor"/>

	New value:
	<property name="schema.dirs"
	value="${build.dir.test.schemas}/xbean/xmlcursor,
	${build.dir.test.schemas}/xbean/NEWAREA"/>
	If your schemas are needed in order to run checkintests please update the
	"checkin.schema.dirs" property as well.


	b) New top-level area added to test (src/NEWAREA)
	Add this folder to the list of areas for incremental compile
	* Append the area name to the property area.names
	* Add a new target with the name "build.area.NEWAREA"

	Properties that need to be set for this target to work
	* includes.for.compile - includes spec for javac

	Optionally set the following properties
	* this.schema.dirs - list of folders for scomp
	* this.schema.standalone.dirs - list of individual schemas for scomp
	* testcasesjar.update - set to true if testcases.jar should be
	updated instead of overwriting

	Add a call to target "unit.build" at the end of the target.

	e.g:
	<target name="build.area.xmlcursor">
	<property name="this.schema.dirs"
	value="${build.dir.test.schemas}/xbean/xmlcursor"/>
	<property name="includes.for.compile"
	value="/xmlcursor//*.java"/>
	<property name="testcasejar.update" value="true"/>
	<antcall target="unit.build"/>
	</target>