v.db.select: Handle all formats in option, add CSV, fix JSON (#1121)

Instead of separate flags for JSON and vertical output format, use format option,
so -j is replaced by format=json, etc. The formats are no longer spread among
the other flags and a written command is more readble. It is also more explicit
since the default format is now represented in the interface and vertical output
clearly stands out as yet another alternative.

The default format is named plain, alternative names considered and rejected were text, ascii and default.
It is meant to be read by humans, but now it can be also newly safely parsed when set as TSV,
however, this feature is questionable and the idea is that plain is not used for parsing.

The use of an option does not change anything major in the code in terms of output
except for use of an enum instead of directly accessing the flags' answers.
The parameter handling changed significantly. The standard exclude function
cannot be used and custom checks and messages are needed to disallow combinations
such as JSON with custom null value.

The significant clean-up related changes to existing formats are:

* The JSON output of NULL values is fixed, writing JSON literal null.
* JSON outputs are now always Object (dictionary) so theoretically they can be combined or enhanced in the future. 
* Different default separator for different formats and outputs (pipe for plain, comma for CSV).
* Separator for -e can now be customized.
* All required JSON escapes are escaped.
* Tab is escaped when escaping is enabled (so you could theoretically write TSV with escapes using plain).
* For all formats, JSON-style escapes except for double quotes are applied when escaping is enabled.
* Vertical output does not add a newline when newline is vertical separator (no more two newlines).

This adds another format, CSV, as format=csv, i.e., without need to add another
flag (e.g., -c and -v were already taken). Most of the code was already there
as basic structure was the original plain default output and quoting was already
done for the JSON output. The original format was not a proper CSV because it
lacked handling quoting of fields which for common CSV parsers takes care of new line
characters and separators in values/cells. It was not completely safe even with escaping
because it did does not escape to the separator (to be delimiter separated values format
with escapes). Delimiter for CSV can be customized, but the quoting cannot.

Now in CSV, all non-numeric columns are quoted (with two quotes representing a quote in the text).
The quotes used are double quotes, the separator is derived by the horizontal/field
separator option. By default is is a comma (a different separator is set for each format by default).
CSV output is available even for the -r flag (minimal region/bbox).

vector/v.db.select/main.c

@@ -7,10 +7,11 @@
  *               OGR support by Martin Landa <landa.martin gmail.com>
  *               -e, -j, and -f flags by Huidae Cho <grass4u gmail.com>
  *               group option by Luca Delucchi <lucadeluge gmail.com>
+ *               CSV and format option by Vaclav Petras <wenzeslaus gmail com>
  *
  * PURPOSE:      Print vector attributes
  *
- * COPYRIGHT:    (C) 2005-2020 by the GRASS Development Team
+ * COPYRIGHT:    (C) 2005-2021 by the GRASS Development Team
  *
  *               This program is free software under the GNU General
  *               Public License (>=v2). Read the file COPYING that
@@ -29,6 +30,33 @@
 #include <grass/vector.h>
 #include <grass/dbmi.h>
 
+enum OutputFormat {
+    PLAIN,
+    JSON,
+    CSV,
+    VERTICAL
+};
+
+void fatal_error_option_value_excludes_flag(struct Option *option,
+                                            struct Flag *excluded,
+                                            const char *because)
+{
+    if (!excluded->answer)
+        return;
+    G_fatal_error(_("The flag -%c is not allowed with %s=%s. %s"),
+                  excluded->key, option->key, option->answer, because);
+}
+
+void fatal_error_option_value_excludes_option(struct Option *option,
+                                              struct Option *excluded,
+                                              const char *because)
+{
+    if (!excluded->answer)
+        return;
+    G_fatal_error(_("The option %s is not allowed with %s=%s. %s"),
+                  excluded->key, option->key, option->answer, because);
+}
+
 int main(int argc, char **argv)
 {
     struct GModule *module;
@@ -36,6 +64,7 @@ int main(int argc, char **argv)
     {
         struct Option *map;
         struct Option *field;
+        struct Option *format;
         struct Option *fsep;
         struct Option *vsep;
         struct Option *nullval;
@@ -50,7 +79,6 @@ int main(int argc, char **argv)
         struct Flag *colnames;
         struct Flag *vertical;
         struct Flag *escape;
-        struct Flag *json;
         struct Flag *features;
     } flags;
     dbDriver *driver;
@@ -60,19 +88,24 @@ int main(int argc, char **argv)
     dbColumn *column;
     dbValue *value;
     struct field_info *Fi;
-    int ncols, col, more, first_rec;
+    int ncols, col, more;
+    bool first_rec;
     struct Map_info Map;
     char query[DB_SQL_MAX];
     struct ilist *list_lines;
     char *fsep, *vsep;
     struct bound_box *min_box, *line_box;
-    int i, line, area, init_box, cat, field_number;
+    int i, line, area, cat, field_number;
+    bool init_box;
+    enum OutputFormat format;
+    bool vsep_needs_newline;
 
     module = G_define_module();
     G_add_keyword(_("vector"));
     G_add_keyword(_("attribute table"));
     G_add_keyword(_("database"));
     G_add_keyword(_("SQL"));
+    G_add_keyword(_("export"));
     module->description = _("Prints vector map attributes.");
 
     options.map = G_define_standard_option(G_OPT_V_MAP);
@@ -94,8 +127,23 @@ int main(int argc, char **argv)
         _("GROUP BY conditions of SQL statement without 'group by' keyword");
     options.group->guisection = _("Selection");
 
+    options.format = G_define_option();
+    options.format->key = "format";
+    options.format->type = TYPE_STRING;
+    options.format->required = YES;
+    options.format->label = _("Output format");
+    options.format->options = "plain,csv,json,vertical";
+    options.format->descriptions =
+        "plain;Configurable plain text output;"
+        "csv;CSV (Comma Separated Values);"
+        "json;JSON (JavaScript Object Notation);"
+        "vertical;Plain text vertical output (instead of horizontal)";
+    options.format->answer = "plain";
+    options.format->guisection = _("Format");
+
     options.fsep = G_define_standard_option(G_OPT_F_SEP);
-    options.fsep->guisection = _("Main");
+    options.fsep->answer = NULL;
+    options.fsep->guisection = _("Format");
 
     options.vsep = G_define_standard_option(G_OPT_F_SEP);
     options.vsep->key = "vertical_separator";
@@ -124,22 +172,11 @@ int main(int argc, char **argv)
     flags.colnames->description = _("Do not include column names in output");
     flags.colnames->guisection = _("Format");
 
-    flags.vertical = G_define_flag();
-    flags.vertical->key = 'v';
-    flags.vertical->description =
-        _("Vertical output (instead of horizontal)");
-    flags.vertical->guisection = _("Format");
-
     flags.escape = G_define_flag();
     flags.escape->key = 'e';
     flags.escape->description = _("Escape newline and backslash characters");
     flags.escape->guisection = _("Format");
 
-    flags.json = G_define_flag();
-    flags.json->key = 'j';
-    flags.json->description = _("JSON output");
-    flags.json->guisection = _("Format");
-
     flags.features = G_define_flag();
     flags.features->key = 'f';
     flags.features->description =
@@ -148,9 +185,6 @@ int main(int argc, char **argv)
 
     G_gisinit(argv[0]);
 
-    G_option_excludes(flags.json, flags.colnames, flags.vertical,
-                      flags.escape, NULL);
-
     if (G_parser(argc, argv))
         exit(EXIT_FAILURE);
 
@@ -161,6 +195,29 @@ int main(int argc, char **argv)
                           options.file->answer);
     }
 
+    if (strcmp(options.format->answer, "csv") == 0)
+        format = CSV;
+    else if (strcmp(options.format->answer, "json") == 0)
+        format = JSON;
+    else if (strcmp(options.format->answer, "vertical") == 0)
+        format = VERTICAL;
+    else
+        format = PLAIN;
+    if (format == JSON) {
+        fatal_error_option_value_excludes_flag(options.format, flags.escape,
+                                               _("Escaping is based on the format"));
+        fatal_error_option_value_excludes_flag(options.format, flags.colnames,
+                                               _("Column names are always included"));
+        fatal_error_option_value_excludes_option(options.format, options.fsep,
+                                                 _("Separator is part of the format"));
+        fatal_error_option_value_excludes_option(options.format, options.nullval,
+                                                 _("Null value is part of the format"));
+    }
+    if (format != VERTICAL) {
+        fatal_error_option_value_excludes_option(options.format, options.vsep,
+                                                 _("Only vertical output can use vertical separator"));
+    }
+
     min_box = line_box = NULL;
     list_lines = NULL;
 
@@ -175,11 +232,30 @@ int main(int argc, char **argv)
         list_lines = Vect_new_list();
 
     /* the field separator */
-    fsep = G_option_to_separator(options.fsep);
+    if (options.fsep->answer) {
+        fsep = G_option_to_separator(options.fsep);
+    }
+    else {
+        /* A different separator is needed to for each format and output. */
+        if (format == CSV) {
+            fsep = G_store(",");
+        }
+        else if (format == PLAIN || format == VERTICAL) {
+            if (flags.region->answer)
+               fsep = G_store("=");
+            else
+               fsep = G_store("|");
+        }
+        else
+            fsep = NULL;  /* Something like a separator is part of the format. */
+    }
     if (options.vsep->answer)
         vsep = G_option_to_separator(options.vsep);
     else
         vsep = NULL;
+    vsep_needs_newline = true;
+    if (vsep && !strcmp(vsep, "\n"))
+        vsep_needs_newline = false;
 
     db_init_string(&sql);
     db_init_string(&value_string);
@@ -248,9 +324,9 @@ int main(int argc, char **argv)
     table = db_get_cursor_table(&cursor);
     ncols = db_get_table_number_of_columns(table);
 
-    /* column names if horizontal output (ignore for -r, -c, -v, -j) */
+    /* column names if horizontal output (ignore for -r, -c, JSON, vertical) */
     if (!flags.region->answer && !flags.colnames->answer &&
-        !flags.vertical->answer && !flags.json->answer) {
+        format != JSON && format != VERTICAL) {
         for (col = 0; col < ncols; col++) {
             column = db_get_table_column(table, col);
             if (col)
@@ -260,11 +336,15 @@ int main(int argc, char **argv)
         fprintf(stdout, "\n");
     }
 
-    init_box = TRUE;
-    first_rec = TRUE;
+    init_box = true;
+    first_rec = true;
 
-    if (!flags.region->answer && flags.json->answer)
-        fprintf(stdout, "[");
+    if (format == JSON) {
+        if (flags.region->answer)
+            fprintf(stdout, "{\"extent\":\n");
+        else
+            fprintf(stdout, "{\"records\":[\n");
+    }
 
     /* fetch the data */
     while (1) {
@@ -276,8 +356,8 @@ int main(int argc, char **argv)
             break;
 
         if (first_rec)
-            first_rec = FALSE;
-        else if (!flags.region->answer && flags.json->answer)
+            first_rec = false;
+        else if (!flags.region->answer && format == JSON)
             fprintf(stdout, ",\n");
 
         cat = -1;
@@ -305,44 +385,60 @@ int main(int argc, char **argv)
 
             db_convert_column_value_to_string(column, &value_string);
 
-            if (!flags.colnames->answer && flags.vertical->answer)
+            if (!flags.colnames->answer && format == VERTICAL)
                 fprintf(stdout, "%s%s", db_get_column_name(column), fsep);
 
-            if (col && !flags.vertical->answer && !flags.json->answer)
+            if (col && format != JSON && format != VERTICAL)
                 fprintf(stdout, "%s", fsep);
 
-            if (flags.json->answer) {
+            if (format == JSON) {
                 if (!col)
                     fprintf(stdout, "{");
                 fprintf(stdout, "\"%s\":", db_get_column_name(column));
             }
 
-            if (options.nullval->answer && db_test_value_isnull(value)) {
-                if (flags.json->answer)
-                    fprintf(stdout, "\"%s\"", db_get_column_name(column));
-                else
+            if (db_test_value_isnull(value)) {
+                if (format == JSON)
+                    fprintf(stdout, "null");
+                else if (options.nullval->answer)
                     fprintf(stdout, "%s", options.nullval->answer);
             }
             else {
                 char *str = db_get_string(&value_string);
 
-                if (flags.escape->answer || flags.json->answer) {
+                /* Escaped charcters in different formats
+                 * JSON (mandatory): \" \\ \r \n \t \f \b
+                 * CSV (usually none, here optional): \\ \r \n \t \f \b
+                 * Plain, vertical (optional): v7: \\ \r \n, v8 also: \t \f \b
+                 */
+                if (flags.escape->answer || format == JSON) {
                     if (strchr(str, '\\'))
                         str = G_str_replace(str, "\\", "\\\\");
                     if (strchr(str, '\r'))
                         str = G_str_replace(str, "\r", "\\r");
                     if (strchr(str, '\n'))
                         str = G_str_replace(str, "\n", "\\n");
-                    if (flags.json->answer && strchr(str, '"'))
+                    if (strchr(str, '\t'))
+                        str = G_str_replace(str, "\t", "\\t");
+                    if (format == JSON && strchr(str, '"'))
                         str = G_str_replace(str, "\"", "\\\"");
+                    if (strchr(str, '\f'))  /* form feed, somewhat unlikely */
+                        str = G_str_replace(str, "\f", "\\f");
+                    if (strchr(str, '\b'))  /* backspace, quite unlikely */
+                        str = G_str_replace(str, "\b", "\\b");
+                }
+                /* Common CSV does not escape, but doubles quotes (and we quote all
+                 * text fields which takes care of a separator character in text). */
+                if (format == CSV && strchr(str, '"')) {
+                    str = G_str_replace(str, "\"", "\"\"");
                 }
 
-                if (flags.json->answer) {
-                    int sqltype = db_get_column_sqltype(column);
+                if (format == JSON || format == CSV) {
+                    int type =
+                        db_sqltype_to_Ctype(db_get_column_sqltype(column));
 
-                    if (sqltype == DB_SQL_TYPE_INTEGER ||
-                        sqltype == DB_SQL_TYPE_DOUBLE_PRECISION ||
-                        sqltype == DB_SQL_TYPE_REAL)
+                    /* Don't quote numbers, quote text and datetime. */
+                    if (type == DB_C_TYPE_INT || type == DB_C_TYPE_DOUBLE)
                         fprintf(stdout, "%s", str);
                     else
                         fprintf(stdout, "\"%s\"", str);
@@ -351,9 +447,9 @@ int main(int argc, char **argv)
                     fprintf(stdout, "%s", str);
             }
 
-            if (flags.vertical->answer)
+            if (format == VERTICAL)
                 fprintf(stdout, "\n");
-            else if (flags.json->answer) {
+            else if (format == JSON) {
                 if (col < ncols - 1)
                     fprintf(stdout, ",");
                 else
@@ -380,44 +476,62 @@ int main(int argc, char **argv)
                                   line);
                 if (init_box) {
                     Vect_box_copy(min_box, line_box);
-                    init_box = FALSE;
+                    init_box = false;
                 }
                 else
                     Vect_box_extend(min_box, line_box);
             }
         }
         else {
-            if (!flags.vertical->answer && !flags.json->answer)
+            /* End of record in attribute printing */
+            if (format != JSON && format != VERTICAL)
                 fprintf(stdout, "\n");
-            else if (vsep)
-                fprintf(stdout, "%s\n", vsep);
+            else if (vsep) {
+                if (vsep_needs_newline)
+                    fprintf(stdout, "%s\n", vsep);
+                else
+                    fprintf(stdout, "%s", vsep);
+            }
         }
     }
 
-    if (!flags.region->answer && flags.json->answer)
-        fprintf(stdout, "]\n");
+    if (!flags.region->answer && format == JSON)
+        fprintf(stdout, "\n]}\n");
 
     if (flags.region->answer) {
-        if (flags.json->answer) {
+        if (format == CSV) {
+            fprintf(stdout, "n%ss%sw%se", fsep, fsep, fsep);
+            if (Vect_is_3d(&Map)) {
+                fprintf(stdout, "%st%sb", fsep, fsep);
+            }
+            fprintf(stdout, "\n");
+            fprintf(stdout, "%f%s%f%s%f%s%f", min_box->N, fsep, min_box->S,
+                    fsep, min_box->W, fsep, min_box->E);
+            if (Vect_is_3d(&Map)) {
+                fprintf(stdout, "%s%f%s%f", fsep, min_box->T, fsep, min_box->B);
+            }
+            fprintf(stdout, "\n");
+        }
+        else if (format == JSON) {
             fprintf(stdout, "{");
             fprintf(stdout, "\"n\":%f,", min_box->N);
             fprintf(stdout, "\"s\":%f,", min_box->S);
             fprintf(stdout, "\"w\":%f,", min_box->W);
             fprintf(stdout, "\"e\":%f", min_box->E);
             if (Vect_is_3d(&Map)) {
-                fprintf(stdout, ",\"t\":%f,\n", min_box->T);
-                fprintf(stdout, "\"b\":%f\n", min_box->B);
+                fprintf(stdout, ",\"t\":%f,", min_box->T);
+                fprintf(stdout, "\"b\":%f", min_box->B);
             }
-            fprintf(stdout, "}\n");
+            fprintf(stdout, "\n}}\n");
         }
         else {
-            fprintf(stdout, "n=%f\n", min_box->N);
-            fprintf(stdout, "s=%f\n", min_box->S);
-            fprintf(stdout, "w=%f\n", min_box->W);
-            fprintf(stdout, "e=%f\n", min_box->E);
+            fprintf(stdout, "n%s%f\n", fsep, min_box->N);
+            fprintf(stdout, "s%s%f\n", fsep, min_box->S);
+            fprintf(stdout, "w%s%f\n", fsep, min_box->W);
+            fprintf(stdout, "e%s%f\n", fsep, min_box->E);
             if (Vect_is_3d(&Map)) {
-                fprintf(stdout, "t=%f\n", min_box->T);
-                fprintf(stdout, "b=%f\n", min_box->B);
+                fprintf(stdout, "t%s%f\n", fsep, min_box->T);
+                fprintf(stdout, "b%s%f\n", fsep, min_box->B);
             }
         }
         fflush(stdout);

vector/v.db.select/testsuite/test_v_db_select.py

@@ -163,7 +163,8 @@ out_sep = """1076,366545504,324050.96875,1077,1076,Zwe,366545512.376,324050.9723
 1290,63600420,109186.835938,1291,1290,Zwe,63600422.4739,109186.832069
 """
 
-out_json = """[{"cat":1,"onemap_pro":963738.75,"PERIMETER":4083.97998,"GEOL250_":2,"GEOL250_ID":1,"GEO_NAME":"Zml","SHAPE_area":963738.608571,"SHAPE_len":4083.979839},
+out_json = """{"records":[
+{"cat":1,"onemap_pro":963738.75,"PERIMETER":4083.97998,"GEOL250_":2,"GEOL250_ID":1,"GEO_NAME":"Zml","SHAPE_area":963738.608571,"SHAPE_len":4083.979839},
 {"cat":2,"onemap_pro":22189124,"PERIMETER":26628.261719,"GEOL250_":3,"GEOL250_ID":2,"GEO_NAME":"Zmf","SHAPE_area":22189123.2296,"SHAPE_len":26628.261112},
 {"cat":3,"onemap_pro":579286.875,"PERIMETER":3335.55835,"GEOL250_":4,"GEOL250_ID":3,"GEO_NAME":"Zml","SHAPE_area":579286.829631,"SHAPE_len":3335.557182},
 {"cat":4,"onemap_pro":20225526,"PERIMETER":33253,"GEOL250_":5,"GEOL250_ID":4,"GEO_NAME":"Zml","SHAPE_area":20225526.6368,"SHAPE_len":33253.000508},
@@ -172,7 +173,8 @@ out_json = """[{"cat":1,"onemap_pro":963738.75,"PERIMETER":4083.97998,"GEOL250_"
 {"cat":7,"onemap_pro":969311.1875,"PERIMETER":4096.096191,"GEOL250_":8,"GEOL250_ID":7,"GEO_NAME":"Ybgg","SHAPE_area":969311.720138,"SHAPE_len":4096.098584},
 {"cat":8,"onemap_pro":537840.625,"PERIMETER":3213.40625,"GEOL250_":9,"GEOL250_ID":8,"GEO_NAME":"Zmf","SHAPE_area":537840.953797,"SHAPE_len":3213.407197},
 {"cat":9,"onemap_pro":3389078.25,"PERIMETER":16346.604492,"GEOL250_":10,"GEOL250_ID":9,"GEO_NAME":"Zml","SHAPE_area":3389077.17888,"SHAPE_len":16346.604884},
-{"cat":10,"onemap_pro":906132.375,"PERIMETER":4319.162109,"GEOL250_":11,"GEOL250_ID":10,"GEO_NAME":"Zml","SHAPE_area":906132.945012,"SHAPE_len":4319.163379}]"""
+{"cat":10,"onemap_pro":906132.375,"PERIMETER":4319.162109,"GEOL250_":11,"GEOL250_ID":10,"GEO_NAME":"Zml","SHAPE_area":906132.945012,"SHAPE_len":4319.163379}
+]}"""
 
 
 class SelectTest(TestCase):
@@ -185,8 +187,13 @@ class SelectTest(TestCase):
     cat = 10
 
     def testRun(self):
-        """Basic test of v.db.select"""
-        self.assertModule("v.db.select", map=self.invect)
+        """Module runs with minimal parameters and give output."""
+        sel = SimpleModule(
+            "v.db.select",
+            map=self.invect,
+        )
+        sel.run()
+        self.assertTrue(sel.outputs.stdout, msg="There should be some output")
 
     def testFileExists(self):
         """This function checks if the output file is written correctly"""
@@ -237,11 +244,12 @@ class SelectTest(TestCase):
         self.assertLooksLike(reference=out_sep, actual=sel.outputs.stdout)
 
     def testJSON(self):
+        """Test that JSON can be decoded and formatted exactly as expected"""
         import json
 
         sel = SimpleModule(
             "v.db.select",
-            flags="j",
+            format="json",
             map=self.invect,
             where="cat<={cat}".format(cat=self.cat),
         )

vector/v.db.select/testsuite/test_v_db_select_json_csv.py

@@ -0,0 +1,184 @@
+############################################################################
+#
+# MODULE:       Test of v.db.select
+# AUTHOR(S):    Vaclav Petras <wenzeslaus gmail com>
+# PURPOSE:      Test parsing and structure of CSV and JSON outputs
+# COPYRIGHT:    (C) 2021 by Vaclav Petras the GRASS Development Team
+#
+#               This program is free software under the GNU General Public
+#               License (>=v2). Read the file COPYING that comes with GRASS
+#               for details.
+#
+#############################################################################
+
+"""Test parsing and structure of CSV and JSON outputs from v.db.select"""
+
+import json
+import csv
+import itertools
+import io
+
+import grass.script as gs
+
+from grass.gunittest.case import TestCase
+from grass.gunittest.main import test
+
+POINTS = """\
+17.46938776,18.67346939,143,1,Big Hill,2
+20.93877551,17.44897959,125,2,Small Hill,2
+18.89795918,14.18367347,130,3,1,3
+18.89795918,14.18367347,130,4,1,3
+15.91836735,10.67346939,126,5,1,3
+15.91836735,10.67346939,126,6,1,3
+15.91836735,10.67346939,126,7,1,3
+15.91836735,10.67346939,126,8,1,3
+15.91836735,10.67346939,126,9,1,3
+"""
+
+
+class DifficultValueTest(TestCase):
+    """Test case for CSV and JSON parsing and structure with difficult values.
+
+    Tests that CSV and JSON can be loaded properly by CSV/JSON readers
+    and have expected structure.
+    Standard grass.script is used for testing to mimic actual use.
+    Several hard to swallow texts are generated to test the escaping and quoting.
+    """
+
+    # Setup variables to be used for outputs
+    vector_points = "points"
+
+    @classmethod
+    def setUpClass(cls):
+        """Create points with difficult attribute values"""
+        cls.runModule(
+            "v.in.ascii",
+            input="-",
+            stdin_=POINTS,
+            flags="z",
+            z=3,
+            cat=0,
+            separator="comma",
+            output=cls.vector_points,
+            columns="x double precision, y double precision,"
+            " z double precision, owner_id integer,"
+            " place_name text, num_buildings integer",
+        )
+        # Ensure presence of some challenging characters in a safe and transparent way.
+        # This assumes that v.db.update value processing is little more straightforward
+        # than v.in.ascii parsing.
+        # NULL value
+        cls.runModule(
+            "v.db.update",
+            map=cls.vector_points,
+            layer=1,
+            column="place_name",
+            query_column="NULL",
+            where="owner_id = 3",
+        )
+        # Double quotes pair
+        cls.runModule(
+            "v.db.update",
+            map=cls.vector_points,
+            layer=1,
+            column="place_name",
+            value='The "Great" Place',
+            where="owner_id = 4",
+        )
+        # Single single quote (used as apostrophe)
+        cls.runModule(
+            "v.db.update",
+            map=cls.vector_points,
+            layer=1,
+            column="place_name",
+            query_column='"Joan\'s Place"',
+            where="owner_id = 5",
+        )
+        # Pipe
+        cls.runModule(
+            "v.db.update",
+            map=cls.vector_points,
+            layer=1,
+            column="place_name",
+            value="Bright Hall|BLDG209",
+            where="owner_id = 6",
+        )
+        # Comma
+        cls.runModule(
+            "v.db.update",
+            map=cls.vector_points,
+            layer=1,
+            column="place_name",
+            value="Raleigh, NC, USA",
+            where="owner_id = 7",
+        )
+        # Colon and square brackets
+        cls.runModule(
+            "v.db.update",
+            map=cls.vector_points,
+            layer=1,
+            column="place_name",
+            value="Building: GeoLab[5]",
+            where="owner_id = 8",
+        )
+        # Newline
+        cls.runModule(
+            "v.db.update",
+            map=cls.vector_points,
+            layer=1,
+            column="place_name",
+            query_column="'892 Long Street' || char(10) || 'Raleigh NC 29401'",
+            where="owner_id = 9",
+        )
+
+    @classmethod
+    def tearDownClass(cls):
+        """Remove the test data"""
+        cls.runModule("g.remove", flags="f", type="vector", name=cls.vector_points)
+
+    def test_csv_loads(self):
+        """Load CSV with difficult values with many separators"""
+        for delimeter, null_value in itertools.product(
+            [None, ",", ";", "\t", "|"], [None, "NULL"]
+        ):
+            text = gs.read_command(
+                "v.db.select",
+                map=self.vector_points,
+                format="csv",
+                separator=delimeter,
+                null_value=null_value,
+            )
+            # This covers the defaults for v.db.select.
+            if delimeter is None:
+                delimeter = ","
+            if null_value is None:
+                null_value = ""
+            io_string = io.StringIO(text)
+            reader = csv.DictReader(
+                io_string,
+                delimiter=delimeter,
+                quotechar='"',
+                doublequote=True,
+                lineterminator="\n",
+                strict=True,
+            )
+            data = list(reader)
+            self.assertEqual(data[2]["place_name"], null_value)
+            self.assertEqual(data[3]["place_name"], 'The "Great" Place')
+            self.assertEqual(data[5]["place_name"], "Bright Hall|BLDG209")
+            self.assertEqual(data[6]["place_name"], "Raleigh, NC, USA")
+            self.assertEqual(data[8]["place_name"], "892 Long Street\nRaleigh NC 29401")
+
+    def test_json_loads(self):
+        """Load JSON with difficult values"""
+        text = gs.read_command("v.db.select", map=self.vector_points, format="json")
+        data = json.loads(text)
+        data = data["records"]
+        self.assertIsNone(data[2]["place_name"])
+        self.assertEqual(data[3]["place_name"], 'The "Great" Place')
+        self.assertEqual(data[7]["place_name"], "Building: GeoLab[5]")
+        self.assertEqual(data[8]["place_name"], "892 Long Street\nRaleigh NC 29401")
+
+
+if __name__ == "__main__":
+    test()

vector/v.db.select/v.db.select.html

@@ -3,6 +3,187 @@
 <em>v.db.select</em> prints attributes of a vector map from one or several
 user selected attribute table columns.
 
+<h3>Output formats</h3>
+
+Four different formats can be used depending on the circumstances
+using the <b>format</b> option: plain text, CSV, JSON, and vertical
+plain text.
+
+<h4>Plain text</h4>
+
+The plain text is the default output which is most suitable for reading by humans,
+e.g., when working in the command line or obtaining specific values from the attribute
+table using the <em>v.db.select</em> GUI dialog.
+
+<p>
+The individual fields (attribute values) are separated by a pipe (<tt>|</tt>)
+which can be customized using the <b>separator</b> option.
+The records (rows) are separated by newlines.
+
+<p>
+Example with a pipe as a separator (the default):
+
+<div class="code"><pre>
+cat|road_name|multilane|year|length
+1|NC-50|no|2001|4825.369405
+2|NC-50|no|2002|14392.589058
+3|NC-98|no|2003|3212.981242
+4|NC-50|no|2004|13391.907552
+</pre></div>
+
+When escaping is enabled, the following characters in the fields are escaped:
+backslash (<tt>\\</tt>), carriage return (<tt>\r</tt>), line feed (<tt>\n</tt>),
+tabulator (<tt>\t</tt>), form feed (<tt>\f</tt>), and backslash (<tt>\b</tt>).
+
+<p>
+No quoting or escaping is performed by default, so if these characters are in
+the output, they look just like the separators.
+This is usually not a problem for humans looking at the output to get a general idea
+about query result or attribute table content.
+
+<p>
+Consequently, this format is not recommended for computers, e.g., for reading attribute
+data in Python scripts.
+It works for further parsing in limited cases when the values don't contain separators
+or when the separators are set to one of the escaped characters.
+
+<h4>CSV</h4>
+
+CSV (comma-separated values) has many variations. This module by default produces
+CSV with comma (<tt>,</tt>) as the field separator (delimiter). All text fields
+(based on the type) are quoted with double quotes. Double quotes in fields are
+represented as two double quotes. Newline characters in the fields are present
+as-is in the output. Header is included by default containing column names.
+
+<p>
+All full CSV parsers such as the ones in LibreOffice or Python are able to parse this
+format when configured to the above specification.
+
+<p>
+Example with default settings:
+
+<div class="code"><pre>
+cat,road_name,multilane,year,length
+1,"NC-50","no",2001,4825.369405
+2,"NC-50","no",2002,14392.589058
+3,"NC-98","no",2003,3212.981242
+4,"NC-50","no",2004,13391.907552
+</pre></div>
+
+<p>
+If desired, the separator can be customized and escaping can be enabled
+with the same characters being escaped as for the plain text.
+Notably, newlines and tabs are escaped, double quotes are not, and the separator
+is not escaped either (unless it is a tab).
+However, the format is guaranteed only for the commonly used separators
+such as comma, semicolon, pipe, and tab.
+
+<p>
+Note that using multi-character separator is allowed, but not recommended
+as it is not generally supported by CSV readers.
+
+<p>
+CSV is the recommended format for further use in another analytical applications,
+especially for use with spreadsheet applications. For scripting, it is advantageous
+when tabular data is needed (rather than key-value pairs).
+
+<h4>JSON</h4>
+
+JSON (JavaScript Object Notation) format is produced according to
+the specification so it is readily readable by JSON parsers.
+The standard JSON escapes are performed (backslash, carriage return, line feed,
+tabulator, form feed, backslash, and double quote) for string values.
+Numbers in the database such as integers and doubles are represented as numbers,
+while texts (TEXT, VARCHAR, etc.) and dates in the database are represented
+as strings in JSON. NULL values in database are represented as JSON <tt>null</tt>.
+Indentation and newlines in the output are minimal and not guaranteed.
+
+<p>
+Records which are the result of the query are stored under key <tt>records</tt>
+as an array (list) of objects (collections of key-value pairs).
+The keys for attributes are lowercase or uppercase depending on how
+the columns were defined in the database.
+
+<p>
+Example with added indentation (note that booleans are not directly supported;
+here, an attribute is a string with value <tt>no</tt>):
+
+<div class="code"><pre>
+{
+  "records": [
+    {
+      "cat": 1,
+      "road_name": "NC-50",
+      "multilane": "no",
+      "year": 2001,
+      "length": 4825.369405
+    },
+    {
+      "cat": 2,
+      "road_name": "NC-50",
+      "multilane": "no",
+      "year": 2002,
+      "length": 14392.589058
+    }
+  ]
+}
+</pre></div>
+
+<p>
+JSON is the recommended format for reading the data in Python
+and for any uses and environments where convenient access to individual values
+is desired and JSON parser is available.
+
+<h4>Vertical plain text</h4>
+
+In the vertical plain text format, each value is on a single line
+and is preceded by the name of the attribute (column) which is
+separated by separator. The individual records can be separated by
+the vertical separator (<b>vertical_separator</b> option).
+
+<p>
+Example with (horizontal) separator <tt>=</tt> and vertical separator <tt>newline</tt>:
+
+<div class="code"><pre>
+cat=1
+road_name=NC-50
+multilane=no
+year=2001
+length=4825.369405
+
+cat=2
+road_name=NC-50
+multilane=no
+year=2002
+length=14392.589058
+</pre></div>
+
+Newline is automatically added after a vertical separator unless it is a newline
+which allows for separating the records, e.g., by multiple dashes.
+
+The escaping (<b>-e</b>) need to should be enabled in case the output
+is meant for reading by a computer rather than just as a data overview
+for humans. Escaping will ensure that values with newlines will be
+contained to a single line.
+
+This format is for special uses in scripting, for example, in combination
+with <b>columns</b> option set to one column only and escaping (<b>-e</b>)
+and no column names flags (<b>-c</b>). It is also advantageous when you
+need implement the parsing yourself.
+
+<h2>NOTES</h2>
+
+<ul>
+    <li>
+        CSV and JSON were added in version 8.0 as new primary formats for further
+        consumption by scripts and other applications.
+    </li>
+    <li>
+        Escaping of plain and vertical formats was extended from just backslash
+        and newlines to all escapes from JSON except for double quote character.
+    </li>
+</ul>
+
 <h2>EXAMPLES</h2>
 
 All examples are based on the North Carolina sample dataset.
@@ -125,6 +306,21 @@ US-64|yes
 US-70|yes
 </pre></div>
 
+<h3>Read results in Python</h3>
+
+The <em>json</em> package in the standard Python library can load
+a JSON string obtained as output from the <em>v.db.select</em> module
+through the <em>read_command</em> function:
+
+<div class="code"><pre>
+import json
+import grass.script as gs
+
+text = gs.read_command("v.db.select", map="roadsmajor", format="json")
+data = json.loads(text)
+for row in data["records"]:
+    print(row["ROAD_NAME"])
+</pre></div>
 
 <h2>SEE ALSO</h2>
 
@@ -136,9 +332,11 @@ US-70|yes
 
 Radim Blazek, ITC-Irst, Trento, Italy<br>
 Minimal region extent added by Martin Landa,
-FBK-irst (formerly ITC-irst), Trento, Italy (2008/08)<br>
+FBK-irst (formerly ITC-irst), Trento, Italy<br>
 Group option added by Luca Delucchi,
-Fondazione Edmund Mach, Trento, Italy (2015/12)
+Fondazione Edmund Mach, Trento, Italy<br>
+Huidae Cho (JSON output, escaping and features-only flags)<br>
+Vaclav Petras (true CSV output, format option and documentation)
 
 <!--
 <p>