= guts - GPDB Unit Testing System =

The following documentation should help developing unit tests for GPDB using 
the cmockery framework.

Unit testing is an approach to test a small part of a program isolated from the 
test. This type of testing is very common in Java community. The benefits are 
a) that problems are found very early in the development process, 
b) it simplifies the integration of different modules because unit testing 
established a basic trust in the small units themselves, c) it serves as 
documentation about the expected behavior, d) enables easier code reuse because 
code is already tested via two different code paths [Wikipedia].

The unit test code is stored in src/test/unit. The directory contains the 
modified cmockery library, all mock source files currently in use (mock) and a 
directory test, which contains all test code.

The different test executables can be found in the subdirectories of the 
directory test. By the design the used test library cmockery works, a test 
executable can only focus on a set of source files. We call this portion the 
System Under Test (SUT). We will talk more about the System Under Test later. 
For now, it is important to know that the SUT is the port of code tested in a 
test and that each SUT has its own test executable. Usually a SUT consists of a 
single file, e.g. heapam.c, but it can also consist of multiple related files.
 
== Test Development Process ==

The development of a new test consists of multiple steps:

1.	Selecting an appropriate System Under Test. Often the SUT consists of the 
    source file of the function to test. But exceptions from this rule of thumb 
    are possible. There are already tests with the given SUT, the new test can 
    be added to the existing executable. Step 2 can be skipped in this case.
    
2.	Creating the SUT test executable. If the SUT does not exist, a new test 
    executable needs to be created:

	- Create a new test executable directory, usually named after the SUT source 
	  file(s) and add a Makefile. The directory template contains template files
	  for the Makefile and the README file.
	  
	  > BIN=SOMETHING_test
	  > include ../Makefile.all
      > include ${CMOCKERY_DIR}/Makefile
      > include ${MOCK_DIR}/Makefile

      > TEST_OBJS=SOMETHING_test.o
      > SUT_OBJS=$(CDB_BACKEND_DIR)/SOMETHING.o
      > MOCK_OBJS=
	
      > include ../Makefile.all2
	  
	- Change the BIN, TEST_OBJ, SUT_OBJS and MOCK_OBJS variables in the mock 
	  file. BIN, TEST_OBJ, and SUT_OBJS are usually straightforward to set. 
	  
	  MOCK_OBJS needs to contain the mock function object files for all function 
	  calls that are done from the SUT to functions not contained in the SUT. 
	  The easiest way is to create the Makefile with TEST_OBJ, SUT_OBJECT set
	  and MOCK_OBTS empty. Then the following call helps locating all the needed 
	  mock files. 
	  
	  make 2>&1 | python ../../find_mock_objs.py
	  
	  Here is a sample output:
	  
	  > Mock Source Files
	  > aset_mock.c
	  [...]
	  > transam_mock.c
	  > tuptoaster_mock.c
	  > xact_mock.c
	  > xlog_mock.c
	  > xlogutils_mock.c
      > 
	  > Prepared for Makefile
	  > $(MOCK_DIR)/aset_mock.o \
	  [...]
	  > 	$(MOCK_DIR)/transam_mock.o \
	  > 	$(MOCK_DIR)/tuptoaster_mock.o \
	  > 	$(MOCK_DIR)/xact_mock.o \
	  > 	$(MOCK_DIR)/xlog_mock.o \
	  > 	$(MOCK_DIR)/xlogutils_mock.o
	  
	  The output is a list of mock object files ready to be copied into the 
	  Makefile. The output also may contain "Skipped References". Usually, these 
	  are references for which no much functions are generated.
	  
	  The Makefile could look like this in the end:
	  
	  > BIN=foo_test
	  > include ../Makefile.all
      > include ${CMOCKERY_DIR}/Makefile
      > include ${MOCK_DIR}/Makefile
      > 
      > TEST_OBJS=foo_test.o
      > SUT_OBJS=$(CDB_BACKEND_DIR)/foo/foo.o
      > MOCK_OBJS=$(MOCK_DIR)/aset_mock.o \
	  [...]
	  > 	$(MOCK_DIR)/transam_mock.o \
	  > 	$(MOCK_DIR)/tuptoaster_mock.o \
	  > 	$(MOCK_DIR)/xact_mock.o \
	  > 	$(MOCK_DIR)/xlog_mock.o \
	  > 	$(MOCK_DIR)/xlogutils_mock.o
      > 	
      > include ../Makefile.all2	  
	  
	- If there are functions calls for which currently no mock function exists, 
	  we need to generate the mock function. For this, the filename (only the 
	  last filename, not the full path) of the mock function should be added to 
	  mock/mock_autogen/mock_source and generate_mocks.py should be called. 
	  After this the mock function should exists in the mock directory.
	  
	- Create a new test source file, also usually named after the SUT. An 
	  example is heapam_test.c in the heapam SUT directory. In the beginning 
	  the test source file consists only of the main function.
	  
	  Here is an example:
	  
      > #include <stdarg.h>
      > #include <stddef.h>
      > #include <setjmp.h>
      > #include "cmockery.h"
      > 
      > #include "postgres.h"
      > 
      > void test_foo_bar(void** state)
      > {
      > 	assert_int_equal(0, 1); /* Do not do this. Provide a real test */
      > }
      > 
      > int main(int argc, char* argv[])
      > {
      > 	cmockery_parse_arguments(argc, argv);
      > 
      > 	const UnitTest tests[] = {
      > 			unit_test(test_foo_bar)
      > 	};
      > 	return run_tests(tests);
      > }

	  
3.	Insert new test case functions and register them in the main function. A 
	test function usually consists of three steps:
    
	- Describe the interaction of the called functions from the SUT with the 
	  environment. This is only via expect_- and will_-functions from the 
	  cmockery library. Will_ functions are used to define which mocked 
	  functions are expected to be called by the test code. While the ordering 
	  will_-function calls for different functions names don't need to be 
	  strictly ordered, it is recommended to keep a strict ordering here. 
	  Will-function calls also define the return values, assignments to 
	  out-going parameters and side effects. This way we are able to define the 
	  exact state a tested function can observe.
	  
      In addition to the will_-functions, the expect_-functions are used to 
      describe our expectations about the values of parameters in the mocked 
      function calls. Details and a full list of the functions are showed later 
      in this document. 

    - Call a function or a series of functions from the System Under Test. This 
      function or functions are the unit testing with a specific test case.

    - The return values and the state after each test function call, can be 
      validated by assert_ calls. A list of all available assert_ calls is 
      presented later in this document.

== CMockery Usage ==

A cmockery test executable usually starts similar to this example from 
heapam_test.c:

int main(int argc, char* argv[]) {
    cmockery_parse_arguments(argc, argv);

    const UnitTest tests[] = {
    		unit_test(test_heap_xlog_newpage)
    };
    return run_tests(tests);
}

The test framework accepts command line arguments:

- --cmockery_abort_at_missing_check
  In certain situations, is helpful to call the test executable with 
  --cmockery_abort_at_missing_check. When this option is activated or if the 
  cmockery_abort_at_missing_check() function is called, cmockery aborts instead 
  of failing a test when a check is missing. 
  
  If used with a debugger attached or when generating core files, this helps to 
  identify where in the SUT code unexpected function calls have been made.
  However, usually checked in code should not contain this 
  abort_at_missing_check call since otherwise all following tests in the test 
  executable are not executed at all.
  
- --cmockery_generate_suppression
  In certain situations, it is helpful to call the test executable with 
  --cmockery_generate_suppression. When this option is activated or if the 
  cmockery_enable_generate_suppression() function is called, cmockery outputs
  example expect_ and will_ lines that are currently missing in the test 
  executable.
  
  While a test is pretty useless if it solely consists of generated expect_ 
  and will_ calls, it is helpful to setup the test environment. This is 
  especially true, if the test only deals with a certain line in a very long 
  function.
  
- --run_disabled_tests
  Certain test cases can be disabled with the disable_unit_test() function. 
  If cmockery is called with --run_disabled_tests, these tests are executed 
  even when they are disabled.

A test function like test_heap_xlog_newpage in the example has the 
signature "void test_heap_xlog_newpage(void** state)". The parameter state is 
used by cmockery internally and should not be used or modified by the test 
developer.

The following functions are provided by cmockery to describe the interaction of 
a SUT with its environment:

=== Assertions ===
- assert_true(c): Assert that the given expression is true. 
  If the expression is not true, the test fails.
  
- assert_false(c): Asserts that the given expression is false. 
  If the expression is true, the test fails.
  
- assert_int_equal(a, b): Asserts that two integral values are equal.

- assert_int_not_equal(a, b): Asserts that two integral values are not equal.

- assert_string_equal(a, b): Asserts that the contents of two strings are equal.

- assert_string_not_equal(a, b): Asserts that the contents of two strings are 
  different.

- assert_memory_equal(a, b, size): Asserts that two memory areas of a given 
  size have the same contents.
  
- assert_memory_not_equal(a, b, size): Asserts that two memory areas of a given 
  size have different contents.
  
- assert_in_range(value, min, max): Asserts that the value of an expression lies 
  in a given range.
  
- assert_not_in_range(value, min, max): Similar to assert_in_range, but negates.

- assert_in_set(value, values, number_of_values): Asserts that a value is in a 
  set of values. In contrast to except_in_set, values can be a pointer to an 
  array here. The parameter number_of_values determines the length of the array.
  
- assert_not_in_set(value, values, number_of_values): Similar to assert_in_set, 
  but negates.

=== Function call expectations ===

- will_return(function_name, value): Adds an expectation that a function with 
  the given name will be called once. If the function is called, the value 
  provided by will_return is returned to the SUT. 
  
- will_be_called(function_name): Adds an expectation that a function 
  with the given name will be called once. This function should be used for 
  functions that do not return values.
  
- will_return_with_sideeffect(function_name, value, sideeffect_function, 
      sideeffect_data): 
  This function is similar to will_return, but also executes a callback 
  function. The callback can be used to perform actions on global variables. 
  
- will_be_called_with_sideeffect(function_name, sideeffect_function, 
      sideeffect_data): 
  This function is similar to will_be_called, but also executes a callback 
  function. The callback can be used to perform actions on global variables.
  
=== Basic function parameter expectations ===

- expect_value(function_name, parameter, value): Adds an expectations that, 
  when a function is called, a parameter has a given value. If the function is 
  called, and the parameter has a different value the test fails.
  
- expect_value_count(function_name, parameter, value, count): Similar to 
  expect_value, but expect that the function is consecutively called count 
  number of times with the parameter having the given value. If count is -1, 
  the function establishes that for all later calls of the function, the 
  parameter has to have the given value. Similar functions having a _count 
  suffix exist for all expect_ functions. 
  We will omit them for the other functions. 
  
- expect_not_value(function_name, parameter, value): Adds an expectation that a 
  parameter has not a given value.
  
- expect_string(function_name, parameter, string): Adds an expectation that a 
  string parameter has a value equal to the given string. 

- expect_not_string(function_name, parameter, string): Adds an expectation that 
  a string parameter has a value not equal to the given string.
  
- expect_any(function_name, parameter): Declares that we have no expectation on 
  a given parameter during a specific function call. This expectation never 
  fails.
  
=== Advanced parameter expectations ===

- expect_memory(function_name, parameter, memory, size): Adds an expectation 
  that the memory a pointer parameter points is equal to the memory parameter 
  for the next size bytes. The data behind the memory parameter is copied and 
  can be safely modified or freed with out changing the expectation.
  
- expect_not_memory(function_name, parameter, memory, size): 
  Similar to expect_memory, but negated. 
  
- expect_check(function_name, parameter, check_function, check_data): 
  Adds an expectation that when a function is called, a parameter fulfills a 
  certain property.
   
  The check_function has the signature int(const LongestIntegralValue value, 
  const LongtestIntegralValue check_data). 
  If the function returns 1 the value fulfills the property. 0 indicates an 
  error. The value is the value of the parameter during the function call 
  casted to the type LongestIntegralValue. Similarly, check_data is the data 
  from the check_data parameter of the expect_check call.
  Usually this function is not used directly. 
  All other expect_ calls are implemented via expect_check. However, it can be 
  helpful for certain custom expectations. 
  
- expect_in_range(function, parameter, min, max): 
  Adds an expectation that a parameter value lies in a range.
  
- expect_not_in_range(function, parameter, min, max): 
  Similar to expect_in_range, but negated. The parameter value should not be in 
  the given range.
  
- expect_in_set(function, parameter, value_array): 
  Adds an expectation that a parameter value is in the set of values specified 
  by the value array. Note that the value array needs to be an actual array. 
  A pointer to values is not working.
  
- expect_not_in_set(function, parameter, value_array): 
  Similar to expect_in_set, but negates. The parameter value should not be in 
  the set.
  
- will_assign_value(function, parameter, value): 
  When the function is called, the given value will be assigned to the 
  dereferenced parameter. This function is used to assign 
  specific parameters to out-going parameters. 
  
- will_assign_string(function, parameter, string): 
  When the function is called, the given string will be assigned to the 
  out-going (aka non-const) string parameter.
  
- will_assign_memory(function, parameter, memory, size): 
  When the function is called, the memory block is copied to the memory 
  the out-going (aka non-const) pointer parameter points to.
  
=== Other cmockery features

- If disable_unit_test() is called after the start of a test case function, the
  execution of the test is skipped. While it is generally discouraged to skip a 
  test case because the test keeps failing, it is handy in certain situations.
  
== Basic Tips ==
- A unit test should be automated. It should not contain manual input. 
  If a test was successful or failed needs to be determined by assert_ and 
  expect_ calls and not be the "correct" output on screen.

== Modification on cmockery ==

The underlying unit testing/mocking library cmockery is open sourced under
the Apache 2.0 license. To better suit our requirements, we have made the 
following modifications

- Add will_be_called for void functions
- Add will_assign_/optional_assignment to support out-going parameters
- Adds the option to abort on an missing expectation
- Adds the option to generate expect_/will_ statements on missing expectations
- Adds a better and colored console output
- Add the parsing of command line parameters

== Further information ==

- cmockery Project Wiki:
  http://code.google.com/p/cmockery/wiki/Cmockery_Unit_Testing_Framework
  
- Wikipedia: 
  http://en.wikipedia.org/wiki/Unit_testing