Add GitHub Actions with build, test, and Python static code analysis (#583)

This mostly cherry picks 4bb3a3a, 39415e8, and 80b51d7 from master and then adds several adjustments for 7.8 branch.

Add GitHub Actions to build, run tests, and Python static code analysis (#525):

This adds a CI workflow with build and test running on Ubuntu 16.04 and 18.04. The build job is not parallelized and is meant for clear and relatively fast check of compilation and building in general. This way it is clear if the compilation failed when the test job failed without examining the output. (Duplicating what is currently running on Travis.)

The test job needs to build first, but the main focus is to run tests, so the compilation is parallelized (depending on nproc) and thus potentially less readable. This runs the whole test suite. Failing tests (file names) are displayed.

The test job requires minimal success rate using the (backported) --min-success flag. Test uses nc_spm_full_v2alpha2 location from fatra.cnr.ncsu.edu for tests.

Additional work flow adds a static code analysis/code quality check/linting using Flake8 for Python code in lib/python, gui/wxpython, scripts and temporal directories (separately). As an experiment, who approaches are employed in dealing with current errors. lib/python uses a configuration file which ignores code in testsuite directories and ignores a lot of errors specified in the configuration. The other directories use the default settings and the failure is ignored using GitHub Actions configuration.

Helper files are placed into .github/workflows (possibly to be moved if general enough).

Ignore current errors from wxGUI, scripts, and temporal (#537):

Python code errors from Flake8 are now ignored gui/wxpython, scripts, and temporal
in the same way as in lib/python. Unique set of errors is used for each directory
to keep ignored errors at minimum.

Serious errors are sorted at the top. Whitespace at the bottom.
Line length is set to 88, but ignored.

Further:

Add numpy and other Python pkgs to CI (#585)

Push the min required success to maximum to 90%. The current tests are running with 90-91% success rate in 7.8 branch, so let's test for that.

It changes the grass command from grass79 to grass78.

.github/workflows/apt.txt

@@ -0,0 +1,22 @@
+bison
+build-essential
+flex
+libbz2-dev
+libcairo-dev
+libfftw3-dev
+libgdal-dev
+libgl1-mesa-dev
+libglu1-mesa-dev
+libnetcdf-dev
+libopenblas-dev
+libpng-dev
+libproj-dev
+libreadline-dev
+libzstd-dev
+python3-dateutil
+python3-numpy
+python3-ply
+python3-six
+python3-termcolor
+sqlite3
+zlib1g-dev

.github/workflows/build.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+# The make step requires something like:
+# export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$PREFIX/lib"
+# further steps additionally require:
+# export PATH="$PATH:$PREFIX/bin"
+
+# fail on non-zero return code from a subprocess
+set -e
+
+# print commands
+set -x
+
+if [ -z "$1" ]
+then
+    echo "Usage: $0 PREFIX"
+    exit 1
+fi
+
+# non-existent variables as an errors
+set -u
+
+export INSTALL_PREFIX=$1
+
+./configure \
+    --prefix="$INSTALL_PREFIX/" \
+    --enable-largefile \
+    --with-cxx \
+    --with-zstd \
+    --with-bzlib \
+    --with-blas \
+    --with-lapack \
+    --with-readline \
+    --with-openmp \
+    --with-pthread \
+    --with-tiff \
+    --with-freetype \
+    --with-freetype-includes="/usr/include/freetype2/" \
+    --with-proj-share=/usr/share/proj \
+    --with-geos \
+    --with-sqlite \
+    --with-fftw \
+    --with-netcdf
+
+make
+make install

.github/workflows/ci.yml

@@ -0,0 +1,75 @@
+name: CI
+
+on:
+- push
+- pull_request
+
+jobs:
+  build:
+    name: ${{ matrix.os }} build
+
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os:
+        - ubuntu-16.04
+        - ubuntu-18.04
+      fail-fast: false
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Get dependencies
+      run: |
+        sudo apt-get update -y
+        sudo apt-get install -y wget git gawk findutils
+        xargs -a <(awk '! /^ *(#|$)/' ".github/workflows/apt.txt") -r -- \
+            sudo apt-get install -y --no-install-recommends --no-install-suggests
+    - name: Create installation directory
+      run: |
+        mkdir $HOME/install
+    - name: Ensure one core for compilation
+      run: |
+        echo "::set-env name=MAKEFLAGS::-j1"
+    - name: Set LD_LIBRARY_PATH for compilation
+      run: |
+        echo "::set-env name=LD_LIBRARY_PATH::$HOME/install/lib"
+    - name: Build
+      run: .github/workflows/build.sh $HOME/install
+
+  test:
+    name: ${{ matrix.os }} tests
+
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os:
+        - ubuntu-16.04
+        - ubuntu-18.04
+      fail-fast: false
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Get dependencies
+      run: |
+        sudo apt-get update -y
+        sudo apt-get install -y wget git gawk findutils
+        xargs -a <(awk '! /^ *(#|$)/' ".github/workflows/apt.txt") -r -- \
+            sudo apt-get install -y --no-install-recommends --no-install-suggests
+    - name: Create installation directory
+      run: |
+        mkdir $HOME/install
+    - name: Set number of cores for compilation
+      run: |
+        echo "::set-env name=MAKEFLAGS::-j$(nproc)"
+    - name: Set LD_LIBRARY_PATH for compilation
+      run: |
+        echo "::set-env name=LD_LIBRARY_PATH::$HOME/install/lib"
+    - name: Build
+      run: .github/workflows/build.sh $HOME/install
+    - name: Add the bin directory to PATH
+      run: |
+        echo "::add-path::$HOME/install/bin"
+    - name: Test executing of the grass command
+      run: .github/workflows/test_simple.sh
+    - name: Run tests
+      run: .github/workflows/test_thorough.sh

.github/workflows/flake8.yml

@@ -0,0 +1,85 @@
+name: Code quality check
+
+on:
+- push
+- pull_request
+
+jobs:
+  flake8-lib-python:
+
+    runs-on: ubuntu-18.04
+
+    steps:
+    - uses: actions/checkout@v1
+    - name: Set up Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.8
+    - name: Install
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8
+    - name: Run Flake8
+      run: |
+        cd lib/python
+        flake8 --count --statistics --show-source --jobs=$(nproc) .
+
+  flake8-wxgui:
+
+    continue-on-error: true
+    runs-on: ubuntu-18.04
+
+    steps:
+    - uses: actions/checkout@v1
+    - name: Set up Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.8
+    - name: Install
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8
+    - name: Run Flake8
+      run: |
+        cd gui/wxpython
+        flake8 --count --statistics --show-source --jobs=$(nproc) .
+
+  flake8-scripts:
+
+    continue-on-error: true
+    runs-on: ubuntu-18.04
+
+    steps:
+    - uses: actions/checkout@v1
+    - name: Set up Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.8
+    - name: Install
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8
+    - name: Run Flake8
+      run: |
+        cd scripts
+        flake8 --count --statistics --show-source --jobs=$(nproc) .
+
+  flake8-temporal-modules:
+
+    continue-on-error: true
+    runs-on: ubuntu-18.04
+
+    steps:
+    - uses: actions/checkout@v1
+    - name: Set up Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.8
+    - name: Install
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8
+    - name: Run Flake8
+      run: |
+        cd temporal
+        flake8 --count --statistics --show-source --jobs=$(nproc) .

.github/workflows/test_simple.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+# fail on non-zero return code from a subprocess
+set -e
+
+grass78 --tmp-location EPSG:4326 --exec g.region res=0.1 -p

.github/workflows/test_thorough.sh

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+
+# fail on non-zero return code from a subprocess
+set -e
+
+grass78 --tmp-location XY --exec \
+    g.extension g.download.location
+grass78 --tmp-location XY --exec \
+    g.download.location url=http://fatra.cnr.ncsu.edu/data/nc_spm_full_v2alpha2.tar.gz dbase=$HOME
+
+grass78 --tmp-location XY --exec \
+    python3 -m grass.gunittest.main \
+        --grassdata $HOME --location nc_spm_full_v2alpha2 --location-type nc \
+        --min-success 90

gui/wxpython/.flake8

@@ -0,0 +1,68 @@
+[flake8]
+ignore =
+    E114, # indentation is not a multiple of four (comment)
+    E115, # expected an indented block (comment)
+    E262, # inline comment should start with '# '
+    E265, # block comment should start with '# '
+    E266, # too many leading '#' for block comment
+    W605, # invalid escape sequence '\.'
+    E402, # module level import not at top of file
+    E502, # the backslash is redundant between brackets
+    E712, # comparison to False should be 'if cond is False:' or 'if not cond:'
+    E713, # test for membership should be 'not in'
+    E722, # do not use bare 'except'
+    E731, # do not assign a lambda expression, use a def
+    E741, # ambiguous variable name 'l'
+    E999, # SyntaxError: EOL while scanning string literal
+    F401, # 'animation.utils.getCpuCount' imported but unused
+    F403, # 'from gmodeler.model import *' used; unable to detect undefined names
+    F405, # '_' may be undefined, or defined from star imports: gmodeler.model
+    F811, # redefinition of unused 'wx' from line 106
+    F821, # undefined name '_'
+    F841, # local variable 'dc' is assigned to but never used
+    E117, # over-indented
+    E122, # continuation line missing indentation or outdented
+    E123, # closing bracket does not match indentation of opening bracket's line
+    E124, # closing bracket does not match visual indentation
+    E125, # continuation line with same indent as next logical line
+    E126, # continuation line over-indented for hanging indent
+    E127, # continuation line over-indented for visual indent
+    E128, # continuation line under-indented for visual indent
+    E131, # continuation line unaligned for hanging indent
+    E202, # whitespace before '}'
+    E203, # whitespace before ':'
+    E211, # whitespace before '('
+    E222, # multiple spaces after operator
+    E225, # missing whitespace around operator
+    E226, # missing whitespace around arithmetic operator
+    E231, # missing whitespace after ','
+    E241, # multiple spaces after ','
+    E261, # at least two spaces before inline comment
+    E271, # multiple spaces after keyword
+    E272, # multiple spaces before keyword
+    E301, # expected 1 blank line, found 0
+    E302, # expected 2 blank lines, found 1
+    E303, # too many blank lines (3)
+    E305, # expected 2 blank lines after class or function definition, found 1
+    E306, # expected 1 blank line before a nested definition, found 0
+    E401, # multiple imports on one line
+    E501, # line too long (96 > 79 characters)
+    W291, # trailing whitespace
+    W293, # blank line contains whitespace
+    W503, # line break before binary operator
+    W504, # line break after binary operator
+
+max-line-length = 88
+exclude =
+    .git,
+    __pycache__,
+    .env,
+    .venv,
+    env,
+    venv,
+    ENV,
+    env.bak,
+    venv.bak,
+    ctypes,
+    pydispatch,
+    testsuite,

lib/python/.flake8

@@ -0,0 +1,83 @@
+[flake8]
+ignore =
+    E101, # indentation contains mixed spaces and tabs
+    W191, # indentation contains tabs
+    E111, # indentation is not a multiple of four
+    E114, # indentation is not a multiple of four (comment)
+    E116, # unexpected indentation (comment)
+    E242, # tab after ','
+    E251, # unexpected spaces around keyword / parameter equals
+    E262, # inline comment should start with '# '
+    E265, # block comment should start with '# '
+    E266, # too many leading '#' for block comment
+    E402, # module level import not at top of file
+    E501, # line too long (183 > 150 characters)
+    E502, # the backslash is redundant between brackets
+    E701, # multiple statements on one line (colon)
+    E702, # multiple statements on one line (semicolon)
+    E703, # statement ends with a semicolon
+    E711, # comparison to None should be 'if cond is None:'
+    E712, # comparison to True should be 'if cond is True:' or 'if cond:'
+    E713, # test for membership should be 'not in'
+    E721, # do not compare types, use 'isinstance()'
+    E722, # do not use bare 'except'
+    E731, # do not assign a lambda expression, use a def
+    E741, # ambiguous variable name 'l'
+    F401, # '.reader.BandReferenceReader' imported but unused
+    F403, # 'from ctypes import *' used; unable to detect undefined names
+    F405, # 'RasterRow' may be undefined, or defined from star imports: ctypes, grass.pygrass.raster, grass.pygrass.vector
+    F811, # redefinition of unused 'utils' from line 26
+    F821, # undefined name '_'
+    F841, # local variable 't0' is assigned to but never used
+    F901, # 'raise NotImplemented' should be 'raise NotImplementedError'
+    W605, # invalid escape sequence '\_'
+    W291, # trailing whitespace
+    W292, # no newline at end of file
+    W293, # blank line contains whitespace
+    W391, # blank line at end of file
+    W503, # line break before binary operator
+    W504, # line break after binary operator
+    E117, # over-indented
+    E121, # continuation line under-indented for hanging indent
+    E122, # continuation line missing indentation or outdented
+    E123, # closing bracket does not match indentation of opening bracket's line
+    E124, # closing bracket does not match visual indentation
+    E125, # continuation line with same indent as next logical line
+    E126, # continuation line over-indented for hanging indent
+    E127, # continuation line over-indented for visual indent
+    E128, # continuation line under-indented for visual indent
+    E129, # visually indented line with same indent as next logical line
+    E201, # whitespace after '{'
+    E202, # whitespace before ')'
+    E203, # whitespace before ':'
+    E211, # whitespace before '('
+    E221, # multiple spaces before operator
+    E222, # multiple spaces after operator
+    E225, # missing whitespace around operator
+    E226, # missing whitespace around arithmetic operator
+    E228, # missing whitespace around modulo operator
+    E231, # missing whitespace after ':'
+    E241, # multiple spaces after ','
+    E261, # at least two spaces before inline comment
+    E271, # multiple spaces after keyword
+    E272, # multiple spaces before keyword
+    E301, # expected 1 blank line, found 0
+    E302, # expected 2 blank lines, found 1
+    E303, # too many blank lines (3)
+    E305, # expected 2 blank lines after class or function definition, found 1
+    E401, # multiple imports on one line
+
+max-line-length = 88
+exclude =
+    .git,
+    __pycache__,
+    .env,
+    .venv,
+    env,
+    venv,
+    ENV,
+    env.bak,
+    venv.bak,
+    ctypes,
+    pydispatch,
+    testsuite,

scripts/.flake8

@@ -0,0 +1,60 @@
+[flake8]
+ignore =
+    E265, # block comment should start with '# '
+    E266, # too many leading '#' for block comment
+    E402, # module level import not at top of file
+    E711, # comparison to None should be 'if cond is None:'
+    E712, # comparison to False should be 'if cond is False:' or 'if not cond:'
+    E713, # test for membership should be 'not in'
+    E722, # do not use bare 'except'
+    E731, # do not assign a lambda expression, use a def
+    E741, # ambiguous variable name 'l'
+    F401, # 'grass.script.core.gisenv' imported but unused
+    F632, # use ==/!= to compare str, bytes, and int literals
+    F821, # undefined name '_'
+    F841, # local variable 'center' is assigned to but never used
+    E111, # indentation is not a multiple of four
+    E711, # comparison to None should be 'if cond is None:'
+    F632, # use ==/!= to compare str, bytes, and int literals
+    W391, # blank line at end of file
+    E111, # indentation is not a multiple of four
+    E121, # continuation line under-indented for hanging indent
+    E125, # continuation line with same indent as next logical line
+    E127, # continuation line over-indented for visual indent
+    E128, # continuation line under-indented for visual indent
+    E202, # whitespace before ')'
+    E211, # whitespace before '['
+    E221, # multiple spaces before operator
+    E225, # missing whitespace around operator
+    E226, # missing whitespace around arithmetic operator
+    E231, # missing whitespace after ','
+    E251, # unexpected spaces around keyword / parameter equals
+    E261, # at least two spaces before inline comment
+    E271, # multiple spaces after keyword
+    E272, # multiple spaces before keyword
+    E302, # expected 2 blank lines, found 1
+    E303, # too many blank lines (2)
+    E305, # expected 2 blank lines after class or function definition, found 1
+    E501, # line too long (161 > 150 characters)
+    W293, # blank line contains whitespace
+    W391, # blank line at end of file
+    W503, # line break before binary operator
+    W504, # line break after binary operator
+    E121, # continuation line under-indented for hanging indent
+    E221, # multiple spaces before operator
+    E251, # unexpected spaces around keyword / parameter equals
+
+max-line-length = 88
+exclude =
+    .git,
+    __pycache__,
+    .env,
+    .venv,
+    env,
+    venv,
+    ENV,
+    env.bak,
+    venv.bak,
+    ctypes,
+    pydispatch,
+    testsuite,

temporal/.flake8

@@ -0,0 +1,48 @@
+[flake8]
+ignore =
+    E265, # block comment should start with '# '
+    E266, # too many leading '#' for block comment
+    E502, # the backslash is redundant between brackets
+    E711, # comparison to None should be 'if cond is not None:'
+    E712, # comparison to False should be 'if cond is False:' or 'if not cond:'
+    E722, # do not use bare 'except'
+    F401, # 'ply.lex' imported but unused
+    F821, # undefined name '_'
+    F841, # local variable 'column' is assigned to but never used
+    W291, # trailing whitespace
+    W293, # blank line contains whitespace
+    W391, # blank line at end of file
+    W504, # line break after binary operator
+    E117, # over-indented
+    E125, # continuation line with same indent as next logical line
+    E126, # continuation line over-indented for hanging indent
+    E127, # continuation line over-indented for visual indent
+    E128, # continuation line under-indented for visual indent
+    E129, # visually indented line with same indent as next logical line
+    E222, # multiple spaces after operator
+    E225, # missing whitespace around operator
+    E226, # missing whitespace around arithmetic operator
+    E228, # missing whitespace around modulo operator
+    E231, # missing whitespace after ','
+    E241, # multiple spaces after ','
+    E251, # unexpected spaces around keyword / parameter equals
+    E271, # multiple spaces after keyword
+    E302, # expected 2 blank lines, found 1
+    E303, # too many blank lines (2)
+    E305, # expected 2 blank lines after class or function definition, found 1
+    E501, # line too long (179 > 150 characters)
+  
+max-line-length = 88
+exclude =
+    .git,
+    __pycache__,
+    .env,
+    .venv,
+    env,
+    venv,
+    ENV,
+    env.bak,
+    venv.bak,
+    ctypes,
+    pydispatch,
+    testsuite,