gunittest_testing.rst 31 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723
  1. Testing GRASS GIS source code and modules
  2. =========================================
  3. If you are already familiar with the basic concepts
  4. of GRASS testing framework, you might want to skip to one of:
  5. * :ref:`test-module` section
  6. * :ref:`test-c` section
  7. * :ref:`test-python` section
  8. * :ref:`test-doctest` section
  9. * :class:`~gunittest.case.TestCase` class
  10. * :ref:`running-tests-report` section
  11. Introduction
  12. ------------
  13. For the testing in GRASS GIS, we are using a `gunittest` package and
  14. we usually refer to the system of writing and running tests
  15. as to a *GRASS testing framework*.
  16. The framework is based on Python `unittest`_ package with a large number
  17. of GRASS-specific improvements, extensions and changes. These include
  18. things such as creation of GRASS-aware HTML test reports,
  19. or running of test in the way that process terminations potentially
  20. caused by C library functions does not influence the main testing process.
  21. Some tests will run without any data but many tests require
  22. the small (basic) version of GRASS GIS sample Location for North Carolina
  23. (see `GRASS GIS sample data`).
  24. Basic example
  25. -------------
  26. If you are writing a test of a GRASS module,
  27. create a Python script with the content derived from the example below.
  28. When using existing existing maps, suppose you are in North Carolina SPM
  29. GRASS sample location.
  30. The file can contain one or more test case classes. Each class
  31. can contain one or more test methods (functions).
  32. Here we create one test case class with one test method.
  33. The other two methods are class methods ensuring the right environment
  34. for all test methods inside a test case class.
  35. When a test file becomes part of source code (which is the usual case)
  36. it must be placed into a directory named ``testsuite``.
  37. .. code-block:: python
  38. from grass.gunittest.case import TestCase
  39. from grass.gunittest.main import test
  40. # test case class must be derived from grass.gunittest.TestCase
  41. class TestSlopeAspect(TestCase):
  42. @classmethod
  43. def setUpClass(cls):
  44. """Ensures expected computational region"""
  45. # to not override mapset's region (which might be used by other tests)
  46. cls.use_temp_region()
  47. # cls.runModule or self.runModule is used for general module calls
  48. cls.runModule('g.region', raster='elevation')
  49. # note that the region set by default for NC location is the same as
  50. # the elevation raster map, this is an example shows what to do
  51. # in the general case
  52. @classmethod
  53. def tearDownClass(cls):
  54. cls.del_temp_region()
  55. # test method must start with test_
  56. def test_limits(self):
  57. """Test that slope and aspect are in expected limits"""
  58. # we don't have to delete (g.remove) the maps
  59. # but we need to use unique names within one test file
  60. slope = 'limits_slope'
  61. aspect = 'limits_aspect'
  62. # self.assertModule is used to call module which we test
  63. # we expect module to finish successfully
  64. self.assertModule('r.slope.aspect', elevation='elevation',
  65. slope=slope, aspect=aspect)
  66. # function tests if map's min and max are within expected interval
  67. self.assertRasterMinMax(map=slope, refmin=0, refmax=90,
  68. msg="Slope in degrees must be between 0 and 90")
  69. self.assertRasterMinMax(map=aspect, refmin=0, refmax=360,
  70. msg="Aspect in degrees must be between 0 and 360")
  71. if __name__ == '__main__':
  72. test()
  73. In the example we have used only two assert methods, one to check that
  74. module runs and end successfully and the other to test that map values are
  75. within an expect interval. There is a much larger selection of assert methods
  76. in :class:`~gunittest.case.TestCase` class documentation
  77. and also in Python `unittest`_ package documentation.
  78. To run the test, run GRASS GIS, use NC SPM sample location and create
  79. a separate mapset (name it ``test`` for example). Then go to the directory
  80. with the test file and run it:
  81. .. code-block:: sh
  82. python some_test_file.py
  83. The output goes to the terminal in this case. Read further to see
  84. also more advanced ways of invoking the tests.
  85. We have shown a test of a GRASS module using NC sample location.
  86. However, tests can be written also for C and Python library and also
  87. for internal functions in modules. See the rests of this document
  88. for a complete guide.
  89. Building blocks and terminology
  90. -------------------------------
  91. .. note::
  92. Some parts of the terminology should be revised to ensure understanding and
  93. acceptance.
  94. test function and test method
  95. A *test function* is a test of one particular feature or a test of
  96. one particular result.
  97. A *test function* is referred to as *test method*, *individual test*
  98. or just *test*.
  99. assert function and assert method
  100. An *assert function* (or *assert method*) refers to a function
  101. which checks that some predicate is fulfilled. For example,
  102. predicate can be that two raster maps does not differ from each
  103. other or that module run ends with successfully.
  104. test case
  105. The test methods testing one particular topic or feature are in one
  106. test case class.
  107. From another point of view, one test case class contains all tests
  108. which require the same preparation and cleanup steps.
  109. In other words, a *test case* class contains all tests which are
  110. using the same *test fixture*.
  111. There is also a general :class:`~gunittest.case.TestCase` class which
  112. all concrete test case classes should inherit from to get all
  113. GRASS-specific testing functionality and also to be found
  114. by the testing framework.
  115. test suite
  116. A *test suite*, or also *testsuite*, is a set of tests focused on one
  117. topic, functionality or unit (similarly to test case).
  118. In GRASS GIS, it is a set of files in one ``testsuite`` directory.
  119. The test files in one ``testsuite``
  120. directory are expected to test what is in the parent directory
  121. of a given ``testsuite`` directory. This is used to organize
  122. tests in the source code and also to generate test reports.
  123. The term *test suite* may also refer to ``TestSuite`` class
  124. which is part of Python `unittest`_ test invocation mechanism
  125. used by `gunittest` internally.
  126. More generally, a *test suite* is a group of test cases or any tests
  127. (test methods, test cases and other test suites) in one or more files.
  128. test file
  129. A *test file* is a Python script executable as a standalone process.
  130. It does not set up any special environment and runs where it was invoked.
  131. The testing framework does not rely on the file to end in a standard
  132. way which means that if one file ends with segmentation fault
  133. the testing framework can continue in testing of other test files.
  134. Test files are central part `gunittest` system and are also the biggest
  135. difference from Python `unittest`_. Test file name should be unique
  136. but does not have to contain all parent directory names, for example
  137. it can consist from a simplified name of a module plus a word or two
  138. describing which functionality is tested. The name should not contain
  139. dots (except for the ``.py`` suffix).
  140. Alternatively, a test file could be called *test script* or
  141. *test module* (both in Python and GRASS sense) but note that
  142. none of these is used.
  143. test runner and test invoker
  144. Both *test runner* and *test invoker* refer to classes, functions or
  145. scripts used to run (invoke) tests or test files. One of the terms may
  146. fade of in the future (probably *invoke* because it is not used by
  147. Python `unittest`_).
  148. test fixture (test set up and tear down)
  149. The preparation of the test is called *setup* or *set up* and the cleaning
  150. after the test is called *teardown* or *tear down*. A *test fixture* refers
  151. to these two steps and also to the environment where the test or tests
  152. are executed.
  153. Each test case class can define ``setUp``, ``setUpClass``, ``tearDown``
  154. and ``tearDownClass`` methods to implement preparation and cleanup
  155. steps for tests it contains. The methods ending with ``Class`` are
  156. class methods (in Python terminology) and should be defined using
  157. ``@classmethod`` decorator and with ``cls`` as first argument. These
  158. methods are executed once for the whole class while the methods
  159. without ``Class`` are executed for each test method.
  160. In GRASS GIS, the preparation may, but does not have to, contain imports
  161. of maps, using temporary region, setting computational region,
  162. or generating random maps. The cleanup step should remove temporary
  163. region as well as remove all created maps and files.
  164. test report
  165. A *test report* is a document or set of documents with results of
  166. all executed tests together with additional information such as output
  167. of test.
  168. Note that also *test result* is used also used in similar context
  169. because the class responsible for representing or creating the report
  170. in Python `unittest`_ package is called ``TestResult``.
  171. test failure and test error
  172. A *test failure* occurs when a assert fails, e.g. value of
  173. a parameter given to ``assertTrue()`` function is ``False``.
  174. A *test error* occurs when something what is not tested fails,
  175. i.e. when exception is risen for example preparation code or
  176. a test method itself.
  177. .. _test-general:
  178. Testing with gunittest package in general
  179. -----------------------------------------
  180. The tests should be in files in a ``testsuite`` directory which is a subdirectory
  181. of the directory with tested files (module, package, library). Each test file
  182. (testing file) can have can have several test cases (testing classes).
  183. All test file names should have pattern ``test*.py`` or ``*.py``
  184. if another naming convention seems more appropriate.
  185. GRASS GIS `gunittest` package and testing framework is similar to the standard
  186. Python ``unittest`` package, so the ways to build tests are very similar.
  187. Test methods are in a test test case class and each test method tests one
  188. think using one or more assert methods.
  189. ::
  190. from grass.gunittest.case import TestCase
  191. from grass.gunittest.main import test
  192. class TestPython(TestCase):
  193. def test_counting(self):
  194. """Test that Python can count to two"""
  195. self.assertEqual(1 + 1, 2)
  196. if __name__ == '__main__':
  197. test()
  198. Each test file should be able to run by itself accept certain set of command
  199. line parameters (currently none). This is done using
  200. ``if __name__ == '__main__'`` and ``gunittest.test()`` function.
  201. To run a test file, start GRASS session in the location and mapset suitable for
  202. testing (typically, NC sample location) and go to the test file's directory
  203. (it will be usually some ``testsuite`` directory in the source code)
  204. and run it as a Python script::
  205. python test_something.py
  206. When running individual test files, it is advisable to be in a separate
  207. mapset, so for example when using NC sample location, you should use
  208. a new mapset of arbitrary name but not one of the predefined mapsets).
  209. To run all tests in the source tree, you have to be in the source code
  210. directory where you want to find tests, also you need to be inside
  211. a GRASS session and use command similar to this one::
  212. python -m grass.gunittest.main --location nc_spm_grass7 --location-type nc
  213. All test files in all ``testsuite`` directories will be executed and
  214. a report will be created in a newly created ``testreport`` directory.
  215. Open the file ``testreport/index.html`` to browse though the results.
  216. Note that again you need to be in GRASS session to run the tests in this way.
  217. The ``--location-type`` parameter serves to filter tests according to data
  218. they can run successfully with. It is ignored for tests which does not have
  219. this specified.
  220. In this case each running test file gets its own mapset and
  221. current working directory but all run are in one location.
  222. .. warning::
  223. The current location is ignored but you should not run tests
  224. in the location which is precious to you for the case that something fails
  225. and current location is used for tests.
  226. When your are writing tests you can rely on having maps which are present
  227. in the NC sample location, or you can generate random maps. You can also
  228. import your data which you store inside ``data`` directory inside the
  229. given ``testsuite`` directory (for maps, ASCII formats are usually used).
  230. If you can create tests independent on location projection and location data
  231. it is much better then relying on given data but it is not at all required
  232. and all approaches are encouraged.
  233. Whenever possible it is advantageous to use available assert methods.
  234. GRASS-specific assert methods are in :class:`gunittest.case.TestCase` class.
  235. For general assert methods refer to Python `unittest`_ package documentation.
  236. Both are used in the same way; they are methods of a given test case class.
  237. In cases (which should be rare) when no assert methods fits the purpose,
  238. you can use your own checking finalized with a call of ``assertTrue()``
  239. or ``assertFalse()`` method with the ``msg`` parameter parameter set
  240. to an informative message.
  241. When you are using multiple assert methods in one test method, you must
  242. carefully consider what assert methods are testing and in which order
  243. you should put them. Consider the following example::
  244. # incorrect order
  245. def test_map_in_list_wrong(self):
  246. maps = get_list_of_maps()
  247. self.assertIn('elevation', maps)
  248. # there is no point in testing that
  249. # if list (or string) was empty or None execution of test ended
  250. # at the line with assertIn
  251. self.assertTrue(maps)
  252. # correct order
  253. def test_map_in_list_correct(self):
  254. maps = get_list_of_maps()
  255. # see if list (or string) is not empty (or None)
  256. self.assertTrue(maps)
  257. # and then see if the list fulfills more advanced conditions
  258. self.assertIn('elevation', maps)
  259. If you are not sure when you would use multiple asserts consider the case
  260. when using only ``assertIn()`` function::
  261. def test_map_in_list_short(self):
  262. maps = get_list_of_maps()
  263. self.assertIn('elevation', maps)
  264. If the list (or string) is empty, the test fails and the message says
  265. something about ``elevation''`` not being in the ``maps`` but
  266. it might be much more useful if it would tell us that the list ``maps``
  267. does not contain any items. In case of ``maps`` being ``None``, the situation
  268. is more complicated since we using ``assertIn`` with ``None`` will
  269. cause test error (not only failure). We must consider what is
  270. expected behavior of ``get_list_of_maps()`` function and what
  271. we are actually testing. For example, if we would be testing function
  272. interface, we probably should test separately different possibilities
  273. using ``assertIsNotNone()`` and then ``assertTrue()`` and then anything else.
  274. Another reason for using multiple assert methods is that we may want to
  275. test different qualities of a result. Following the previous example,
  276. we can test that a list contains some map and does not contain some other.
  277. If you are testing a lot of things and they don't have any clear order
  278. or dependencies, it might be more advantageous to split
  279. testing into several testing methods and do the preparation (creating a list
  280. in our example) in ``setUpClass()`` or ``setUp()`` method.
  281. .. _test-module:
  282. Tests of GRASS modules
  283. ----------------------
  284. This is applicable for both GRASS modules written in C or C++ and
  285. GRASS modules written in Python since we are testing the whole module
  286. (which is invoked as a subprocess).
  287. ::
  288. def test_elevation(self):
  289. self.assertModule('r.info', map='elevation', flags='g')
  290. ...
  291. Use method ``assertRasterMinMax()`` to test that a result is within
  292. expected range. This is a very general test which checks the basic
  293. correctness of the result and can be used with different maps
  294. in different locations.
  295. ::
  296. def test_slope_limits(self):
  297. slope = 'limits_slope'
  298. self.assertModule('r.slope.aspect', elevation='elevation',
  299. slope=slope)
  300. self.assertRasterMinMax(map=slope, refmin=0, refmax=90,
  301. msg="Slope in degrees must be between 0 and 90")
  302. .. todo::
  303. Add example of assertions of key-value results.
  304. Especially if a module module has a lot of different parameters allowed
  305. in different combinations, you should test the if the wrong ones are really
  306. disallowed and proper error messages are provided (in addition, you can
  307. test things such as creation and removal of maps in error states).
  308. ::
  309. from grass.gunittest.gmodules import SimpleModule
  310. class TestRInfoParameterHandling(TestCase):
  311. """Test r.info handling of wrong input of parameters."""
  312. def test_rinfo_wrong_map(self):
  313. """Test input of map which does not exist."""
  314. map_name = 'does_not_exist'
  315. # create a module instance suitable for testing
  316. rinfo = SimpleModule('r.info', map=map_name, flags='g')
  317. # test that module fails (ends with non-zero return code)
  318. self.assertModuleFail(rinfo)
  319. # test that error output is not empty
  320. self.assertTrue(rinfo.outputs.stderr)
  321. # test that the right map is mentioned in the error message
  322. self.assertIn(map_name, stderr)
  323. In some cases it might be advantageous to create a module instance
  324. in `setUp()` method and then modify it in test methods.
  325. .. note:
  326. Test should be (natural) language, i.e. locale, independent
  327. to allow testing the functionality under different locale settings.
  328. So, if you are testing content of messages (which should be usually
  329. translated), use `assertIn()` method (regular expression might be
  330. applicable in some cases but in most cases `in` is exactly the
  331. operation needed).
  332. .. _test-c:
  333. Tests of C and C++ code
  334. -----------------------
  335. There are basically two possibilities how to test C and C++ code.
  336. If you are testing GRASS library code the functions which are part of API
  337. these functions are exposed through Python ``ctypes`` and thus can be tested
  338. in Python. See section :ref:`test-python` for reference.
  339. However, more advantageous and more preferable (although sometimes
  340. more complicated) solution is to write a special program, preferably
  341. GRASS module (i.e., using ``G_parser``). The dedicated program can
  342. provide more direct interface to C and C++ functions used by
  343. a GRASS module then the module and can also serve for doing benchmarks
  344. which are not part of the testing.
  345. This can approach can be applied to both
  346. See the example in ``lib/raster3d`` GRASS source code directory
  347. to create a proper Makefiles. A ``main()`` function should be written
  348. in the same way as for a standard module.
  349. Having a GRASS module for the purpose of testing you can write test
  350. as if it would be standard GRASS module.
  351. .. _test-python:
  352. Tests of Python code
  353. --------------------
  354. For testing of Python code contained in some package, use
  355. `gunittest` in the same way as `unittest`_ would be used.
  356. This basically means that if you will write tests of Python functions
  357. and C functions exposed to Python
  358. through ``ctypes`` API, you might want to focus more on `unittest`_
  359. documentation since you will perhaps need the more standard
  360. assert functions rather then the GRASS-specific ones.
  361. .. _test-doctest:
  362. Testing Python code with doctest
  363. --------------------------------
  364. .. note::
  365. The primary use of ``doctest`` is to ensure that the documentation
  366. for functions and classes is valid. Additionally, it can increase
  367. the number of tests when executed together with other tests.
  368. In Python, the easiest thing to test are functions which performs some
  369. computations or string manipulations, i.e. they have some numbers or strings
  370. on the input and some other numbers or strings on the output.
  371. At the beginning you can use doctest for this purpose. The syntax is as follows::
  372. def sum_list(list_to_sum):
  373. """Here is some documentation in docstring.
  374. And here is the test::
  375. >>> sum_list([2, 5, 3])
  376. 10
  377. """
  378. In case of GRASS modules which are Python scripts, you can add something like
  379. this to your script::
  380. if __name__ == "__main__":
  381. if len(sys.argv) == 2 and sys.argv[1] == '--doctest':
  382. import doctest
  383. doctest.testmod()
  384. else:
  385. main()
  386. No output means that everything was successful. Note that you cannot use all
  387. the ways of running doctest since doctest will fail don the module file due
  388. to the dot or dots in the file name. Moreover, it is sometimes required that
  389. the file is accessible through sys.path which is not true for case of GRASS modules.
  390. However, do not use use doctest for tests of edge cases, for tests which require
  391. generate complex data first, etc. In these cases use `gunittest`.
  392. .. _test-as-scripts:
  393. Tests as general scripts
  394. ------------------------
  395. GRASS testing framework supports also general Python or Shell scripts
  396. to be used as test files. This is strongly discouraged because it
  397. is not using standard ``gnunittest`` assert functions which only leads
  398. to reimplementing the functionality, relying on a person examining the data,
  399. or improper tests such as mere testing
  400. if the module executed without an error without looking at the actual results.
  401. Moreover, the testing framework does not have a control over what is
  402. executed and how which limits potential usage and features of testing
  403. framework. Doing this also prevents testing framework from creating a
  404. detailed report and thus better understanding of what is tested and what is
  405. failing. Shell scripts are also harder to execute on MS Windows where the
  406. interpreter might not be available or might not be on path.
  407. The testing framework uses Shell interpreter with ``-e`` flag when executing
  408. the tests, so the tests does not have to use ``set -e`` and can rely on it being
  409. set from outside. The flag ensures that if some command fails, i.e. ends with
  410. non-zero return code (exit status), the execution of the script ends too.
  411. The testing framework also uses ``-x`` flag to print the executed commands
  412. which usually makes examining of the test output easier.
  413. If multiple test files are executed using ``grass.gunittest.main`` module,
  414. the testing framework creates a temporary Mapset for the general Python and
  415. Shell scripts in the same way as it does for ``gunittest``-based test files.
  416. When the tests are executed separately, the clean up in current Mapset
  417. and current working directory must be ensured by the user or the script itself
  418. (which is generally true for all test files).
  419. .. warning::
  420. This is a bad practice which prevents creation of detailed reports and
  421. usage of advanced ``gunittest`` features, so you should avoid it
  422. whenever possible.
  423. Data
  424. ----
  425. .. note::
  426. Both the section and the practice itself are under development.
  427. Most of the tests requires some input data. However, it is good to write
  428. a test in the way that it is independent on the available data.
  429. In case of GRASS, we have we can have tests of functions where
  430. some numbers or strings are input and some numbers or string are output.
  431. These tests does not require any data to be provided since the numbers
  432. can be part of the test. Then we have another category of tests, typically
  433. tests of GRASS modules, which require some maps to be on the input
  434. and thus the output (and test) depends on the specific data.
  435. Again, it it best to have tests which does not require any special data
  436. or generally environment settings (e.g. geographic projection)
  437. but it is much easier to write good tests with a given set of data.
  438. So, an compromises must be made and tests of different types should be written.
  439. In the GRASS testing framework, each test file should be marked according to
  440. category it belongs to. Each category corresponds to GRASS location or locations
  441. where the test file can run successfully.
  442. Universal tests
  443. First category is *universal*. The tests in this category use some some
  444. hard coded constants, generated data, random data, or their own imported
  445. data as in input to function and GRASS modules. All the tests, input data
  446. and reference results should be projection independent. These tests will
  447. runs always regardless of available locations.
  448. Standard names tests
  449. Second category are tests using *standard names*. Tests rely on a
  450. certain set of maps with particular names to be present in the location.
  451. Moreover, the tests can rely also on the (semantic) meaning of the
  452. names, i.e. raster map named elevation will always contain some kind of
  453. digital elevation model of some area, so raster map elevation can be
  454. used to compute aspect. In other words, these tests should be able to
  455. (successfully) run in any location with a maps named in the same way as
  456. in the standard testing location(s).
  457. Standard data tests
  458. Third category of tests rely on *standard data*. These tests expect that the
  459. GRASS location they run in not only contains the maps with particular names
  460. as in the *standard names* but the tests rely also on the data being the
  461. same as in the standard testing location(s). However, the (geographic)
  462. projection or data storage can be different. This is expected to be the
  463. most common case but it is much better if the tests is one of the previous
  464. categories (*universal* or *standard names*). If it is possible the
  465. functions or modules with tests in this category should have also tests
  466. which will fit into one of the previous categories, even though these
  467. additional tests will not be as precise as the other tests.
  468. Location specific tests
  469. Finally, there are tests which requires certain concrete location. There
  470. is (or will be) a set of standard testing locations each will have the same
  471. data (maps) but the projections and data storage types will be different.
  472. The suggested locations are: NC sample location in SPM projection,
  473. NC in SPF, NC in LL, NC in XY, and perhaps NC in UTM, and NC in some
  474. custom projection (in case of strange not-fitting projection, there is
  475. a danger that the results of analyses can differer significantly).
  476. Moreover, the set can be extened by GRASS locations which are using
  477. different storage backends, e.g. PostGIS for vectors and PostgreSQL for
  478. temporal database. Tests can specify one or preferably more of these
  479. standard locations.
  480. Specialized location tests
  481. Additionally, an specialized location with a collection of strange,
  482. incorrect, or generally extreme data will be provided. In theory, more
  483. than one location like this can be created if the data cannot be
  484. together in one location or if the location itself is somehow special,
  485. e.g. because of projection.
  486. Each category, or perhaps each location (will) have a set of external data
  487. available for import or other purposes. The standardization of this data
  488. is in question and thus this may be specific to each location or this
  489. can be a separate resource common to all tests using one of the standardized
  490. locations, or alternatively this data can be associated with the location
  491. with special data.
  492. .. note::
  493. The more general category you choose for your tests the more testing data
  494. can applied to your tests and the more different circumstances can be tried
  495. with your tests.
  496. Data specific to one test
  497. -------------------------
  498. If the data required by the test are not part of standard location
  499. and cannot be part of the test file itself, this data should be stored
  500. in files in ``data`` subdirectory of ``testsuite`` directory.
  501. The test should access the data using a relative path from its location,
  502. i.e. all data will be accessed using ``data/...``. This ``data`` directory
  503. might be used directly when running test file directly in the directory
  504. in the source code or might be copied to the test current working directory
  505. when running tests by the main test invoking tool.
  506. Tests creating separate Mapsets, Locations and GRASS Databases
  507. --------------------------------------------------------------
  508. If test is creating a custom Mapset or Mapsets, it can create them in
  509. the current Location or create a custom GRASS Database in the current
  510. directory. In any case, test has to take care of cleaning up (deleting)
  511. the created directories and it has to use names which will be unique
  512. enough (name of the test case class or the file is probably a good choice
  513. but completely unique identifier is probably much better).
  514. If test needs custom Location or it tests something related to GRASS Database,
  515. it must always create a new GRASS Database in the current directory.
  516. In any case, the author must try the tests cautiously and several times
  517. in the same Location to see if everything works as expected. Testing
  518. framework is using Mapsets to separate the tests and the functions
  519. does not explicitly check for the case where a test is using different
  520. Mapset then the one which has been given to it by the framework.
  521. Analyzing quality of source code
  522. --------------------------------
  523. Besides testing, you can also use some tools to check the quality of your code
  524. according to various standards and occurrence of certain code patterns.
  525. For C/C++ code use third party solution `Coverity Scan`_ where GRASS GIS
  526. is registered as project number `1038`_. Also you can use `Cppcheck`_
  527. which will show a lot of errors which compilers do not check.
  528. In any case, set your compiler to high error and warning levels,
  529. check them and fix them in your code.
  530. For Python, we recommend pylint and then for style issues pep8 tool
  531. (and perhaps also pep257 tool). However, there is more tools available
  532. you can use them together with the recommend ones.
  533. To provide a way to evaluate the Python source code in the whole GRASS source
  534. tree there is a Python script ``grass_py_static_check.py`` which uses
  535. pylint and pep8 with GRASS-specific settings. Run the tool in GRASS session
  536. in the source code root directory. A HTML report will be created in
  537. ``pylint_report`` directory.
  538. ::
  539. grass_py_static_check.py
  540. .. note::
  541. ``grass_py_static_check.py`` is available in `sandbox`_.
  542. Additionally, if you are invoking your Python code manually using python command,
  543. e.g. when testing, use parameters::
  544. python -Qwarn -tt -3 some_module.py
  545. This will warn you about usage of old division semantics for integers
  546. and about incompatibilities with Python 3 (if you are using Python 2)
  547. which 2to3 tool cannot fix. Finally, it will issue errors if are using tabs
  548. for indentation inconsistently (note that you should not use tabs for
  549. indentation at all).
  550. Further reading
  551. ---------------
  552. .. toctree::
  553. :maxdepth: 2
  554. gunittest
  555. gunittest_running_tests
  556. .. _unittest: https://docs.python.org/2/library/unittest.html
  557. .. _doctest: https://docs.python.org/2/library/doctest.html
  558. .. _Coverity Scan: https://scan.coverity.com/
  559. .. _1038: https://scan.coverity.com/projects/1038
  560. .. _Cppcheck: http://cppcheck.sourceforge.net/
  561. .. _sandbox: https://svn.osgeo.org/grass/sandbox/wenzeslaus/grass_py_static_check.py
  562. .. _GRASS GIS sample data: https://grass.osgeo.org/download/sample-data