gunittest: more detailed documentation of running tests including shell scripts

git-svn-id: https://svn.osgeo.org/grass/grass/trunk@64302 15284696-431f-4ddb-bdfa-cd5b030d7da7

lib/python/docs/src/gunittest_running_tests.rst

@@ -2,7 +2,68 @@ Running the test framework of GRASS GIS
 =======================================
 
 This is an advanced guide to running tests of GRASS GIS using GRASS
-testing framework (`gunittest`).
+testing framework (`gunittest`). For introduction to this topic,
+go to :ref:`test-general`.
+
+
+Running tests and creating report
+---------------------------------
+
+To test before commit, run all tests using testing framework.
+First start GRASS GIS session and go to the root directory of your
+GRASS GIS source code copy::
+
+    cd my/grass/source/code/root
+
+Then execute::
+
+    python -m grass.gunittest.main --location locname --location-type nc
+
+where ``locname`` is a name of location in current GRASS GIS data(base) directory
+(GISDBASE) and ``nc`` is a location specified by individual test files
+(the later is not yet fully implemented, so just put there ``nc`` every time).
+
+``grass.gunittest.main`` writes a text summary to standard output and
+it creates an HTML report from all tests in all ``testsuite`` directories inside
+the directory tree. The report is placed in ``testreport`` by default.
+Open file ``testreport/index.html`` in you web browser to inspect it.
+
+To execute just part of the tests when fixing something, ``cd`` into some
+subdirectory, e.g. ``lib`` and execute the same command as above. 
+gain, it will execute all tests in all ``testsuite`` subdirectories and
+create a report.
+
+For changing GRASS GIS data(base) directory and for other parameters, see
+help for ``grass.gunittest.main`` module::
+
+    python -m grass.gunittest.main --help
+
+
+Running individual test files
+-----------------------------
+
+To run a single test file, start GRASS session in the Location and Mapset
+suitable for testing and go to the directory where the test file is.
+Then run the file as a Python script::
+
+    python test_something.py
+
+If the file is a ``gunittest``-based or ``unittest``-based test,
+you will receive a textual output with failed individual tests (test methods).
+If the file is a general Python scriptyou need to examine the output carefully
+as well as source code itself to see what is expected behavior.
+
+The same as for general Python scripts, applies also to Shell scripts,
+so you should examine the output carefully.
+You should execute scripts using::
+
+    sh -e -x test_topology_vgeneralize.sh
+
+The ``-x`` is just to see which commands are executed but the ``-e`` flag
+is crucial because this is how the GRASS testing framework runs the Shell
+scripts. The flag causes execution to stop once some command gives a non-zero
+return code.
+
 
 Example Bash script to run be used as a cron job
 ------------------------------------------------

lib/python/docs/src/gunittest_testing.rst

@@ -111,6 +111,10 @@ for a complete guide.
 Building blocks and terminology
 -------------------------------
 
+.. note::
+    Some parts of the terminology should be revised to ensure understanding and
+    acceptance.
+
 test function and test method
     A *test function* is a test of one particular feature or a test of
     one particular result.
@@ -211,6 +215,7 @@ test failure and test error
     i.e. when exception is risen for example preparation code or
     a test method itself.
 
+.. _test-general:
 
 Testing with gunittest package in general
 -----------------------------------------
@@ -495,9 +500,50 @@ However, do not use use doctest for tests of edge cases, for tests which require
 generate complex data first, etc. In these cases use `gunittest`.
 
 
+.. _test-as-scripts:
+
+Tests as general scripts
+------------------------
+
+GRASS testing framework supports also general Python or Shell scripts
+to be used as test files. This is strongly discouraged because it
+is not using standard ``gnunittest`` assert functions which only leads
+to reimplementing the functionality, relying on a person examining the data,
+or improper tests such as mere testing
+if the module executed without an error without looking at the actual results.
+Moreover, the testing framework does not have a control over what is
+executed and how which limits potential usage and features of testing
+framework. Doing this also prevents testing framework from creating a
+detailed report and thus better understanding of what is tested and what is
+failing. Shell scripts are also harder to execute on MS Windows where the
+interpreter might not be available or might not be on path.
+
+The testing framework uses Shell interpreter with ``-e`` flag when executing
+the tests, so the tests does not have to use ``set -e`` and can rely on it being
+set from outside. The flag ensures that if some command fails, i.e. ends with
+non-zero return code (exit status), the execution of the script ends too.
+The testing framework also uses ``-x`` flag to print the executed commands
+which usually makes examining of the test output easier.
+
+If multiple test files are executed using ``grass.gunittest.main`` module,
+the testing framework creates a temporary Mapset for the general Python and
+Shell scripts in the same way as it does for ``gunittest``-based test files.
+When the tests are executed separately, the clean up in current Mapset
+and current working directory must be ensured by the user or the script itself
+(which is generally true for all test files).
+
+.. warning::
+    This is a bad practice which prevents creation of detailed reports and
+    usage of advanced ``gunittest`` features, so you should avoid it
+    whenever possible.
+
+
 Data
 ----
 
+.. note::
+    Both the section and the practice itself are under development.
+
 Most of the tests requires some input data. However, it is good to write
 a test in the way that it is independent on the available data.
 In case of GRASS, we have we can have tests of functions where